DocsOperateMonitoring

Monitoring

Prometheus metrics, a ready-to-import Grafana dashboard, alerting rules for the four "page-the-operator" conditions, structured log forwarding, and a health-check endpoint for Kubernetes probes and external uptime monitors.

Overview

docsMonitoring.overviewIntro

/etc/solidus/config.toml

[telemetry]
prometheus_enabled = true
prometheus_path = "/metrics"
log_format = "json"           # required for log forwarders (Loki, ES, Datadog)
log_level = "info"

docsMonitoring.overviewCalloutText

Prometheus metrics

12 metric series are exposed across 5 subsystems. Filter below to find what you need.

docsMonitoring.table.metric	docsMonitoring.table.type	docsMonitoring.table.exampleValue
docsMonitoring.group.nodesolidus_block_height Current block height as seen by this node.	gauge	4827392
docsMonitoring.group.p2psolidus_peer_count Number of currently-connected peers.	gauge	47
docsMonitoring.group.nodesolidus_uptime_seconds Seconds since the node process started. Resets on restart.	counter	1842301
docsMonitoring.group.nodesolidus_rss_bytes Resident set size — process memory consumption in bytes.	gauge	1342177280
docsMonitoring.group.consensussolidus_consensus_blocks_produced_total Blocks this validator has produced as proposer.	counter	14287
docsMonitoring.group.consensussolidus_consensus_blocks_missed_total Blocks where this validator was elected proposer but failed to produce.	counter	3
docsMonitoring.group.consensussolidus_consensus_view_changes_total View changes observed by this node (BFT timeout escalations).	counter	142
docsMonitoring.group.validatorsolidus_validator_reputation Reputation score 0-1000 for this validator DID.	gauge	847
docsMonitoring.group.validatorsolidus_validator_violation_points Current violation point count 0-3. Threshold = 3 triggers slash.	gauge	0
docsMonitoring.group.p2psolidus_p2p_bandwidth_in_bytes Cumulative inbound peer traffic in bytes.	counter	8847291002
docsMonitoring.group.p2psolidus_p2p_bandwidth_out_bytes Cumulative outbound peer traffic in bytes.	counter	6219384172
docsMonitoring.group.storagesolidus_storage_disk_used_bytes Disk space consumed by the Solidus data directory.	gauge	699035811840

Scrape config for your Prometheus server:

prometheus.yml

scrape_configs:
  - job_name: 'solidus-node'
    scrape_interval: 15s
    static_configs:
      - targets: ['localhost:9652']
        labels:
          environment: 'mainnet'
          tier: 'subnet'

Grafana dashboard

A pre-built Grafana dashboard JSON ships with every release. It mirrors the operator dashboard's overview screen — same panels, same color palette — so the two interfaces feel like the same product.

Solidus Node · OverviewLast 1h · refresh 15s

Block height

solidus_block_height

4,827,392

Peer count

solidus_peer_count

RSS memory

solidus_rss_bytes

1.24 GB

TPS

consensus

1,247

Uptime 30d

solidus_uptime_seconds

99.97%

Violation points

solidus_validator_violation_points

0 / 3

CLEAN

Solidus Node Dashboard

12 panels · works with any Grafana 9+ instance

Dashboard JSON ships with v0.1.0

To import:

1docsMonitoring.grafanaImportStep1
2docsMonitoring.grafanaImportStep2
3Select your Prometheus data source from the dropdown.
4docsMonitoring.grafanaImportStep4

Alerting rules

Drop these into your Prometheus alertmanager config. They cover the four conditions worth waking the operator for — anything else should fall through to a daily digest.

alertmanager/rules.yml

groups:
  - name: solidus-node-critical
    rules:
      - alert: NodeDown
        expr: up{job="solidus-node"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Solidus node {{ $labels.instance }} is down"

      - alert: LowPeerCount
        expr: solidus_peer_count < 3
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low peer count: {{ $value }} (expected >= 3)"

      - alert: DiskCritical
        expr: (solidus_storage_disk_used_bytes
               / node_filesystem_size_bytes{mountpoint="/var/lib/solidus"})
              > 0.90
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Disk usage > 90% on {{ $labels.instance }}"

      - alert: SyncStalled
        expr: rate(solidus_block_height[5m]) == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Block height has not advanced for 5 minutes"

Why these four?

They each map directly to a distinct failure mode where the operator must act manually: crashed process (NodeDown), network partition (LowPeerCount), storage exhaustion (DiskCritical), or stuck consensus (SyncStalled). The remaining metrics are useful for capacity planning, not for paging.

Log forwarding

docsMonitoring.logForwardingIntro

JSON log line format

example output

{
  "ts":        "2026-03-17T09:41:22.487Z",
  "level":    "INFO",
  "component": "consensus",
  "msg":       "Block finalized",
  "fields": {
    "block_height": 4827392,
    "tx_count":     247,
    "latency_ms":   1832,
    "proposer":     "did:sol:b3d1f8c4a7f2"
  }
}

Loki / Promtail

promtail-config.yaml

scrape_configs:
  - job_name: solidus-node
    journal:
      max_age: 12h
      labels:
        job: solidus-node
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        regex: 'solidus-node.service'
        action: keep
    pipeline_stages:
      - json:
          expressions:
            level:     level
            component: component
      - labels:
          level:
          component:

Elasticsearch / Filebeat

filebeat.yml

filebeat.inputs:
  - type: journald
    id: solidus-node
    include_matches.match:
      - _SYSTEMD_UNIT=solidus-node.service
processors:
  - decode_json_fields:
      fields: ["message"]
      target: "solidus"
      overwrite_keys: true
output.elasticsearch:
  hosts: ["https://es.example.internal:9200"]
  index: "solidus-node-%{+yyyy.MM.dd}"

Health checks

docsMonitoring.healthChecksDescription

health check

# Manual check
curl -s http://127.0.0.1:9652/health | jq

# Healthy → 200
# Syncing → 503 (don't route traffic yet)
# Degraded → 200 (still functional, but flagged)
# Critical → 503

Example response — healthy

GET /health

{
  "status":         "healthy",
  "block_height":   4827392,
  "sync_lag":       0,
  "peer_count":     47,
  "uptime_seconds": 1842301,
  "violation_points": 0,
  "checks": {
    "consensus":  "ok",
    "p2p":        "ok",
    "storage":    "ok",
    "validator":  "ok"
  }
}

Kubernetes probes

deployment.yaml

livenessProbe:
  httpGet:
    path: /health
    port: 9652
  initialDelaySeconds: 30
  periodSeconds: 30
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /health
    port: 9652
  initialDelaySeconds: 60
  periodSeconds: 10
  failureThreshold: 2  # 503 during sync is normal — give it room

Don't restart on a 503 during sync

docsMonitoring.healthCheckWarningText

Up next: Slashing reference · Open operator dashboard →