SmithDB metrics reference

SmithDB components expose Prometheus metrics at /metrics on each pod’s HTTP port. This page lists the metrics worth watching for each component and how to read them. For scrape and export configuration, see Configure SmithDB observability. Names are as they appear on /metrics. If your collector adds a namespace or prefix, adjust accordingly. For Datadog and Grafana dashboards built on these metrics, with the scrape annotations each one expects, see the SmithDB observability examples in the LangSmith Helm chart repository.

This page covers metrics SmithDB emits. Kubernetes signals such as OOM kills, container restarts, CPU and memory against limits, and cache disk usage are worth alerting on but come from your infrastructure monitoring, not from SmithDB.

Baseline metrics

If the full reference is more than you need, the following metrics answer whether SmithDB is healthy. All other metrics provide additional detail.

Question	Metric	Component
Are queries succeeding, and fast?	`server_requests_duration`	Query
Are writes landing, and fast?	`ingest_batch_latency`	Ingestion
Is ingestion keeping up with arrivals?	`ingest_pending_segment_entry_count`	Ingestion
Is compaction keeping up?	`compaction_queue_job_count`	Compaction
Is compaction succeeding?	`compaction_job_completion`	Compaction

Compaction has two entries because the failure modes are separate: jobs can fail repeatedly while the queue length stays flat, and the queue can grow while every job that runs succeeds.

Ingestion

Metric	Type	Description
`ingest_batch_latency`	Histogram	End-to-end latency from batch received to flush completed.
`ingest_flush_count`	Counter	Flushes to object storage. A flat rate while traffic arrives means writes are not landing.
`ingest_flush_rows`	Histogram	Rows per flush, by level. Falling rows at a steady flush rate suggests premature flushing.
`ingest_flush_duration`	Histogram	How long flushes take. Rising durations point at object store latency or undersized ingestion.
`ingest_flush_reason`	Counter	What triggered each flush, labeled `reason` (`size`, `time`, or `shutdown`) and `level` (`L0` or `L1`). L1 flushes on accumulated run count, or a timeout.
`ingest_pending_segment_entry_count`	Gauge	Entries not yet flushed. Sustained growth means flushing is backlogged.
`semaphore_permits_in_use`	Gauge	Concurrency permits held, against `semaphore_max_permits`. Sustained saturation means writes are queuing.
`server_requests_duration`	Histogram	Ingestion request latency. Split by `status` for success and failure counts.

Query

Metric	Type	Description
`server_requests_duration`	Histogram	Query latency. Split by `status` for success and failure counts, where `status` is the gRPC code (`0` is success) or the HTTP code, and by `path` to find the slow endpoint.
`query_fanout_partitions`	Histogram	Active scan partitions per query. Rising fanout drives both latency and memory.
`query_fanout_limit_exceeded_total`	Counter	Queries rejected for excessive fanout. Any sustained rate means queries are failing outright.
`query_execution_max_mem_used_bytes`	Histogram	Peak memory for the largest operator in a query plan. Approaching the configured limit precedes termination.

Compaction

Metric	Type	Description
`compaction_queue_job_count`	Gauge	Jobs waiting in the compaction queue. Sustained growth means compaction is falling behind.
`compaction_jobs_scheduled`	Counter	Work created by the pipeline. Compare against jobs executed.
`compaction_jobs_executed`	Counter	Work consumed. Persistently below scheduled means under-provisioned workers.
`compaction_job_completion`	Counter	Job outcomes, labeled `status` (`success` or `failure`) and `job_kind`. A `job_kind` dropping to zero means that job type has stopped running.
`compaction_queue_jobs_skipped_capacity`	Counter	Jobs skipped for exceeding a worker’s remaining capacity. A sustained rate means worker capacity is too small for the jobs being produced.
`compaction_queue_oldest_pending_created_at_seconds`	Gauge	Unix timestamp when the oldest pending job was enqueued, or zero when the queue is empty. Compute age as `time() - metric`, and exclude the zero case.
`compaction_job_latency_seconds`	Histogram	Job creation to completion. Includes queue wait, so it rises when workers saturate as well as when jobs are slow.
`compaction_worker_capacity_used`	Gauge	In-flight capacity cost by `job_kind`, against `compaction_worker_capacity_limit`. Sustained use near the limit explains skipped jobs and a growing queue.
`compaction_worker_running_tasks`	Gauge	Tasks running by `job_kind`. Zero while the queue is non-empty means workers are stalled, not busy.

Migration

Migration Job pods emit metrics during a historical migration. How you aggregate a metric across pods depends on its type:

Gauges: Report totals for the whole migration. The values come from TaskDB and refresh every two minutes. Every pod reports the same values, so aggregate with max, not sum.
Counters: Track the work each pod does. Sum their rates across pods.

Migration Job pods emit the following metrics:

Metric	Type	Description
`migration_tasks`	Gauge	Migration tasks, labeled `kind` (`run` or `feedback`) and `status` (`pending`, `running`, `completed`, or `failed`). Progress is the `completed` count against the total across statuses.
`migration_jobs`	Gauge	Migration jobs, labeled `kind` and `status`. Run jobs finish as `promoted` and feedback jobs as `validated`. `failed` and `validation_failed` are failures, and every other status means the job is still in progress.
`migration_run_tasks_completed_total`	Counter	Run tasks completed by the pod. The rate is migration throughput in tasks.
`migration_run_task_planned_rows_migrated_total`	Counter	Rows in the run tasks the pod completed, as counted in ClickHouse when the tasks were planned. The rate is migration throughput in rows.

Migration does not retry failed tasks or jobs on its own. Any failed task, or a failed or validation_failed job count, keeps the migration Job from reaching Complete. The Job keeps running until you resolve the failure. See Migration Job failures.

All components

Metric	Type	Description
`object_store_op_duration`	Histogram	Object store operation latency.
`sys_jemalloc_resident_bytes`	Gauge	Resident process memory. Track against the pod memory limit.

LangSmith ingestion path

Emitted by LangSmith rather than by SmithDB, and labeled store="clickhouse|smithdb", so the two stores can be compared directly during dual ingestion.

Metric	Type	Description
`langsmith_ingestion_e2e_latency_seconds`	Histogram	API receipt to store write ack, per run. The user-facing number; compare `store="smithdb"` against `store="clickhouse"`.
`langsmith_ingestion_api_to_worker_latency_seconds`	Histogram	Queue wait before the worker starts. Rising here is a queue problem, not a SmithDB problem.
`langsmith_ingestion_worker_to_store_latency_seconds`	Histogram	Store write time alone. Isolates SmithDB from queue delay.
`langsmith_asynq_ingestion_queue_pending`	Gauge	Tasks waiting in the LangSmith ingestion queue before a worker picks them up. Sustained growth means the queue is not keeping up with arrivals, upstream of SmithDB.

Connect these docs to your agent of choice via MCP for real-time answers.

Edit this page on GitHub or file an issue.

Get started by cloud provider

Deploy with Terraform

Setup guides

Configuration

Connect external services

Platform auth & access control

SmithDB

Self-hosted observability

Hybrid

Scripts

Reference

Baseline metrics

Ingestion

Query

Compaction

Migration

All components

LangSmith ingestion path

​Baseline metrics

​Ingestion

​Query

​Compaction

​Migration

​All components

​LangSmith ingestion path

Baseline metrics

Ingestion

Query

Compaction

Migration

All components

LangSmith ingestion path