Skip to content

observability

nimblebrain_server_sdk.observability · install nimblebrain-server-sdk[observability] · version 0.7.2

from nimblebrain_server_sdk.observability import CapacityRefusal, ErrorCode, ServiceError

Observability for a server: RED metrics, structured JSON logs and W3C trace propagation.

Needs the [observability] extra (pip install 'nimblebrain-server-sdk[observability]'); imported without it, this package raises an ImportError that names the extra.

A server becomes observable by adopting this package rather than hand-rolling instrumentation. Wiring is three lines:

from nimblebrain_server_sdk.observability import instrument_app, get_logger
app = create_app()
instrument_app(app, service="notes")
log = get_logger(__name__)

After that: GET /metrics serves the registry, every log line is JSON stamped with the verified tenant_id / workspace_id and the active OTel trace_id, and traceparent is extracted inbound and exported over OTLP (when an endpoint is configured). Domain metrics ride the same registry via counter(); cross-boundary calls propagate the trace via inject_trace_headers().

The error contract (ServiceError, ErrorCode, to_tool_error, rate_limited_tool_error, CapacityRefusal) lives in nimblebrain_server_sdk.errors and is re-exported here as the same objects.

def counter(name: str, documentation: str, labelnames: tuple[str, ...] = ()) -> Counter

Declare a domain counter on the shared registry (e.g. tasks_created_total).

def gauge(name: str, documentation: str, labelnames: tuple[str, ...] = ()) -> Gauge

Declare a domain gauge on the shared registry (e.g. a current-state level like downstream_resolvable{service,dep}).

Use for a value that can go up and down — a level to read now, not a monotonic total (that’s counter). Rides the same /metrics scrape.

def get_logger(name: str | None = None) -> structlog.stdlib.BoundLogger

A structured logger. Identity/correlation are injected automatically.

def histogram(name: str, documentation: str, labelnames: tuple[str, ...] = (), buckets: tuple[float, ...] = _DURATION_BUCKETS) -> Histogram

Declare a domain histogram on the shared registry (e.g. forward_seconds).

For a distribution a counter cannot express — how long a domain operation took, not how many there were. Defaults to the same agent/LLM-shaped duration buckets the RED histogram uses, so a domain latency and its route latency are readable on one axis; pass buckets for a non-duration distribution.

def inject_trace_headers(headers: dict[str, str] | None = None) -> dict[str, str]

Inject traceparent (+ baggage) into an outbound header dict.

Use at trust boundaries the auto-instrumentation doesn’t cover so the trace survives the hop:

httpx.post(url, headers=inject_trace_headers({"authorization": token}))
def instrument_app(app: App, *, service: str, metrics_path: str = '/metrics', log_level: str = 'INFO', caller_resolver: CallerResolver | None = None) -> App

Make a FastAPI/Starlette app observable. Call once after creating the app.

  • configures JSON-to-stdout logging stamped with request context
  • installs the OTLP tracer provider + FastAPI auto-instrumentation
  • adds the RED-metrics + context-binding middleware
  • mounts GET {metrics_path} serving the Prometheus registry
  • if METRICS_PORT is set, ALSO serves the registry on that dedicated port

The dedicated METRICS_PORT is for edge-fronted servers whose app port also serves /mcp: Prometheus scrapes the metrics-only port (admitted by a network policy), so /mcp is never exposed to the monitoring namespace.

caller_resolver is the infra-plane opt-in: a service with no tenant is attributed by its caller. Pass a fn(get_header) -> caller and every log line in the request scope carries caller. Tenant/fleet services omit it — their logs are unchanged. The resolver must bound cardinality (collapse unknowns to one bucket).

def rate_limited_tool_error(refusal: CapacityRefusal) -> Any

The rate_limited tool error for a call refused for capacity, to return.

Classified, counted and logged like any ServiceError. The host sees the refusal’s own message and what to do next; the code is retryable, and the call ran nothing.

def service_version() -> str

The running service’s own version, for the MCP serverInfo.version field.

Read from SERVICE_VERSION, which the deployment sets to the deployed version (0.2.0 for a release, a short SHA for a staging build), so a server reports exactly what’s deployed. Falls back to "dev" for a local run with nothing injected.

Pass the result as FastMCP(name, version=service_version()). A server that omits version= reports the FastMCP library version instead, which tells a connecting client nothing about the service.

def to_tool_error(err: ServiceError) -> Any

A real fastmcp.ToolResult(is_error=True) to return from a tool.

Returns the FastMCP result object — NOT a dict. Returning a plain dict does NOT work: FastMCP scores a returned dict as is_error=False (a success with the flag buried in structured data), which reproduces the very in-band-error footgun this contract exists to prevent. The sanitized message is the text block; structured_content.error carries the envelope.

Imports fastmcp when called, so importing this module does not.

class CapacityRefusal(Protocol)

A call refused for capacity before it ran anything, carrying host-safe text.

nimblebrain_server_sdk.db.Overloaded is one; this module names the shape rather than the class, so it takes no database dependency.

class ErrorCode(StrEnum)

The closed set of cross-cutting error codes.

class ServiceError(code: ErrorCode, message: str | None = None, *, upstream_status: int | None = None, retryable: bool | None = None, detail: str | None = None)

A classified, host-safe failure crossing a service boundary.

message crosses the trust boundary to the host — keep it generic; NEVER put a vendor name/URL, raw upstream body, secret, or tenant data in it. detail is operator-only (logs); it never reaches the host envelope.

Domain-specific failures subclass this and override metric_code to roll up to one of the cross-cutting codes for the metric label (so the service_errors_total{code=...} label stays the closed set).

def from_upstream(status: int, message: str | None = None, *, detail: str | None = None) -> ServiceError

Build from a raw upstream HTTP status (the common provider-error path).

The bare status number is stamped as host-safe upstream_status; the vendor’s prose belongs in detail (logs only), never in message. Omit message to use the classified code’s host-safe default.

metric_code: str

The closed-set code used as the service_errors_total label.

Cross-cutting errors are their own code; domain subclasses override this to roll up, keeping the metric’s label set bounded.

def to_envelope() -> dict[str, object]

The host-safe error payload (HTTP body / MCP structuredContent.error).