Skip to content

Observability

nimblebrain-server-sdk[observability] makes a server observable the same way as every other NimbleBrain service, in one call.

Terminal window
uv add 'nimblebrain-server-sdk[observability]'
observable_server.py
"""A server with metrics, JSON logs and traces, set up in one call."""
from fastmcp import FastMCP
from nimblebrain_server_sdk import ADDS
from nimblebrain_server_sdk.observability import (
counter,
get_logger,
instrument_app,
service_version,
)
mcp = FastMCP("notes", version=service_version()) # SERVICE_VERSION, else "dev"
log = get_logger(__name__)
NOTES_ADDED = counter("notes_added_total", "Notes added.") # on the same registry as the rest
@mcp.tool(annotations=ADDS)
async def add_note(text: str) -> dict[str, bool]:
"""Add a note."""
NOTES_ADDED.inc()
log.info("note_added", length=len(text)) # one JSON line, with the request's identity
return {"saved": True}
app = mcp.http_app()
instrument_app(app, service="notes") # GET /metrics, request metrics, logs, trace propagation
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="127.0.0.1", port=8000)

instrument_app takes the server’s ASGI app (mcp.http_app(), or a FastAPI or Starlette app) and the service’s name. Call it once.

GET /metrics serves Prometheus text. When METRICS_PORT is set, the registry is also served on that metrics-only port, so a scraper never needs to reach the port that serves /mcp.

Metric Labels
http_requests_total method, route, status
http_request_duration_seconds method, route
http_in_flight_requests method
service_errors_total service, code, upstream_status
  • route is the route template, never the raw path.
  • service_errors_total counts every error surfaced through the error contract.
  • counter, gauge and histogram declare a server’s own metrics on the same registry.
  • Identity is never a metric label. Tenant and workspace belong in logs.

One JSON object per line on stdout, with event, level, timestamp (ISO 8601 UTC) and service. Inside a request each line also carries the tenant_id and workspace_id from the edge’s identity headers and the active span’s trace_id.

  • get_logger(__name__) returns a structured logger: log.info("note_added", length=12).
  • Standard library logging is routed through the same renderer, extra fields included.
  • Request URLs from the httpx family are held at WARNING, so a key in a query string never reaches a log.
  • A logged exception that is, or was caused by, a database error (SQLAlchemy, asyncpg, psycopg) keeps its frames and classes but no message, because Postgres and SQLAlchemy write row values into the message. The line adds db_error with the SQLSTATE and the constraint, table and column the driver reported.
  • The inbound traceparent header is honoured.
  • Spans export over OTLP to OTEL_EXPORTER_OTLP_ENDPOINT when it is set. A recorded database error is exported with its type only: no message and no stacktrace, since a message line can imitate a frame. Its frames are on the log line.
  • On a FastMCP app the request span is FastMCP’s own; on a FastAPI app it is the FastAPI instrumentation’s.
  • inject_trace_headers(headers) carries the trace onto an outbound call.
  • Each response carries x-correlation-id: the inbound trace id when there is one.
test_observable_server.py
"""The metrics endpoint serves the request metrics and the server's own counter."""
from observable_server import app
from starlette.testclient import TestClient
def test_metrics_are_served_in_prometheus_text() -> None:
with TestClient(app) as client:
response = client.get("/metrics")
assert response.status_code == 200
assert "notes_added_total" in response.text
assert "http_requests_total" in response.text