Observability

NimbleBrain emits vendor-neutral telemetry: OpenTelemetry traces over OTLP, structured JSON logs, and a Prometheus metrics endpoint. None of it requires infrastructure to run — with no collector configured, traces are simply not exported and trace ids still flow through logs for correlation. This page covers what to wire up when you run NimbleBrain in production.

Tracing (OpenTelemetry)

The runtime wraps its hot paths in OpenTelemetry spans and exports them over OTLP/HTTP. It depends only on the open OTel and W3C tracecontext primitives, so you point it at any OTLP-compatible collector (the OpenTelemetry Collector, Grafana Tempo, Jaeger, Honeycomb, and so on).

Spans

The runtime produces these spans today:

Span	Covers
agent.turn	One engine run (a full agent turn)
llm.call	A single model stream
tool.dispatch	An MCP tool dispatch
HTTP server span	The inbound request; continues an upstream traceparent if present

Spans nest automatically and carry the verified request identity (user_id, workspace_id, conversation_id) plus the boot-time tenant_id. They never carry the user’s display name, email, secrets, prompts, tool arguments or results, or file contents.

Enabling export

Set OTEL_EXPORTER_OTLP_ENDPOINT to your collector’s base URL. When it is unset, nothing is exported (the default — local bun run dev and OSS checkouts need no infrastructure).

docker-compose.yml (platform service)

environment:
  - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
  - NB_SERVICE_NAME=nimblebrain-runtime
  - NB_TENANT_ID=acme

Variable	Default	Purpose
OTEL_EXPORTER_OTLP_ENDPOINT	unset	Collector base URL. Unset = no export; trace ids still exist for log correlation
NB_SERVICE_NAME	nimblebrain-runtime	Service name stamped on every span and log line
NB_TENANT_ID	unset	This deployment’s tenant. A boot-time constant stamped on every span and log line — never read from a request header

Note

NB_TENANT_ID is set once at process start (in multi-tenant deployments the chart injects it per pod). It is structurally unspoofable: it rides on the OTel Resource and the logger reads it once at boot, so no request can forge another tenant’s id.

Propagation

Outbound calls that should extend a trace — minting a service token, fetching an authenticated remote MCP server — inject the active W3C traceparent so the trace continues across the hop. Inbound, the HTTP middleware continues any upstream traceparent it receives. No configuration is required; this is on whenever a span is active.

Structured logs

By default the runtime logs pretty, colored lines to stderr for the local dev loop. Set NB_LOG_FORMAT=json to switch to one structured JSON object per line — this is what you want for deployed pods (the platform chart sets it for you).

docker-compose.yml (platform service)

environment:
  - NB_LOG_FORMAT=json

Each JSON line is auto-enriched with:

Field	Source
service	NB_SERVICE_NAME
tenant_id	NB_TENANT_ID (boot-time, omitted in dev)
correlation_id	The active trace id — pivot a log straight to its trace
user_id / workspace_id / conversation_id	The verified request identity (omitted when not in a request)

Identity comes only from the verified request context, never the wire, and never includes the human display name. Logs stay on stderr, so Kubernetes captures them alongside stdout and stdout stays clean for JSON-RPC and piped output.

Error reporting (Sentry, optional)

The OpenTelemetry pipeline above is the system of record for traces and logs. Sentry is an optional, additive error sink on top of it — turn it on when you want uncaught exceptions and unhandled promise rejections grouped, deduplicated, and alertable with full stack traces, without scraping logs for them.

It is off by default and switched explicitly — set NB_SENTRY_ENABLED=true (with NB_SENTRY_DSN) to turn it on. Enablement is never inferred from DSN presence, so a stray SENTRY_DSN in the environment can’t silently wire a process into Sentry. When off, the SDK is never even loaded (the OSS, local-dev, and test default). In multi-tenant deployments the chart sets NB_SENTRY_ENABLED per tenant from runtime.config.sentry.enabled, so one tenant can run with Sentry on and another off. These NB_SENTRY_* vars mirror the web client — our own knobs, not Sentry’s auto-read SENTRY_*.

docker-compose.yml (platform service)

environment:
  - NB_SENTRY_ENABLED=true
  - NB_SENTRY_DSN=https://<key>@<org>.ingest.sentry.io/<project>
  - NB_SENTRY_ENV=production

Variable	Default	Purpose
NB_SENTRY_ENABLED	false	Explicit on/off switch. Must be true to enable — never inferred from the DSN
NB_SENTRY_DSN	unset	The project DSN (required when enabled)
NB_SENTRY_ENV	unset	Environment tag (e.g. staging, production) — one project, discriminated by this tag
NB_SENTRY_TRACES_SAMPLE_RATE	0	Sentry performance tracing. 0 = errors only (recommended)

Release is the runtime’s existing NB_VERSION → NB_BUILD_SHA build identity (the same values /v1/health reports), so events group by deploy with no extra var.

Note

Sentry here is initialized as an out-of-kernel preload and does not take over OpenTelemetry — the runtime keeps owning the global tracer provider, so your OTLP → Tempo/Grafana traces are untouched. Sentry captures errors; OTel owns tracing. Performance tracing (NB_SENTRY_TRACES_SAMPLE_RATE > 0) drives Sentry’s own transactions independently and is off by default.

Caution

The same trust boundary as traces and logs applies: sendDefaultPii is off and events are scrubbed before send — no request headers or cookies (auth tokens), no request bodies, no query strings (which can carry prompts and ids). The breadcrumb trail is scrubbed the same way: console breadcrumbs are dropped (dependency logs can carry prompts) and URL query strings are stripped from request breadcrumbs (outbound LLM/MCP/OAuth calls). The only identity on a backend event is the boot-time tenant_id tag — unlike the web client, the runtime does not attach per-request workspace_id / user_id (the kernel can’t import Sentry, so that identity stays on the OTel spans). Never widen this to capture request content.

Prometheus metrics

The runtime exposes a Prometheus exposition endpoint at the bare path GET /metrics:

curl http://platform:27247/metrics

A few properties worth knowing:

• It is served at the bare /metrics path, not under /v1. The bundled nimblebrain-web Caddy proxy only forwards /v1/*, so /metrics is not reachable through the public web container — it is scraped in-cluster.
• Like /v1/health, it is unauthenticated. It carries no per-user data and is meant to be reachable only inside the cluster.

In Kubernetes, point a ServiceMonitor (or your scrape config) at the platform service’s /metrics path. Do not expose it to the internet.

Health and readiness probes

The platform serves a liveness/readiness endpoint at GET /v1/health. It is unauthenticated and returns the build identity plus the state of each installed bundle:

curl http://platform:27247/v1/health

{
  "status": "ok",
  "version": "1.2.3",
  "buildSha": "abc1234",
  "bundles": [
    { "name": "@nimblebraininc/ipinfo", "state": "running" }
  ]
}

Field	Meaning
status	"ok" once the server is serving
version	The platform version
buildSha	The build commit SHA (null if not stamped)
bundles	Each installed bundle and its lifecycle state

The Docker image already wires this into its HEALTHCHECK (curl -f http://localhost:27247/v1/health), and Compose gates the web service on it via depends_on: condition: service_healthy. In Kubernetes, use the same path for both livenessProbe and readinessProbe.

Note

There is no uptime field. Use version / buildSha to confirm which build is live and bundles[].state to confirm bundles started cleanly.

Multi-replica session storage

The /mcp endpoint allocates a stateful session per client (Mcp-Session-Id). At a single replica these live in process memory. To run more than one platform replica, the session metadata must be shared across pods through a registry, and requests must be routed back to the pod holding a session’s transport.

Session store

Select the session store with the sessionStore config block (or the env knobs below):

Store	When
memory	Single replica (default). Sessions live in process memory only
redis	Multiple replicas. Session metadata is shared via Redis so any pod can recognize a session id

nimblebrain.json

{
  "sessionStore": {
    "type": "redis",
    "url": "redis://nb-redis:6379"
  }
}

The registry stores only session metadata (sessionId, identity, workspace, timestamps) — never the live transport, which is process-bound. Routing a request to the pod that owns a session’s transport is the load balancer’s job (sticky routing on Mcp-Session-Id), not the registry’s.

Session limits

These apply regardless of store and tune the per-process transport map:

Variable	Default	Purpose
MCP_MAX_SESSIONS	100	Maximum concurrent MCP sessions per process. At the cap, a new initialize evicts the least-recently-used session rather than returning an error
MCP_SESSION_TTL_SECONDS	28800 (8h)	Idle session TTL. Drives both the per-process idle sweep and the registry TTL

Caution

sessionStore: "redis" is necessary but not sufficient for replicas > 1. You also need session-affinity routing on Mcp-Session-Id (ALB lb_cookie stickiness or header-hash routing) and shared (RWX) storage for workspace data. Until those are in place, keep the platform at a single replica.