DocsOperateMonitoring

Monitoring

Prometheus metrics, a ready-to-import Grafana dashboard, alerting rules for the four "page-the-operator" conditions, structured log forwarding, and a health-check endpoint for Kubernetes probes and external uptime monitors.

Overview

Solidus Node exposes a Prometheus-compatible metrics endpoint on the same RPC port (default 9652), under the path /metrics. It's enabled by default in v0.1.0+; verify the config block below.

/etc/solidus/config.toml

[telemetry]
prometheus_enabled = true
prometheus_path = "/metrics"
log_format = "json"           # required for log forwarders (Loki, ES, Datadog)
log_level = "info"

The /metrics endpoint binds to 127.0.0.1 only. Reach it from another host via SSH tunnel or a Prometheus instance scraping over a private network.

Prometheus metrics

12 metric series are exposed across 5 subsystems. Filter below to find what you need.

Metric	Type	Example value
nodesolidus_block_height Current block height as seen by this node.	gauge	4827392
p2psolidus_peer_count Number of currently-connected peers.	gauge	47
nodesolidus_uptime_seconds Seconds since the node process started. Resets on restart.	counter	1842301
nodesolidus_rss_bytes Resident set size — process memory consumption in bytes.	gauge	1342177280
consensussolidus_consensus_blocks_produced_total Blocks this validator has produced as proposer.	counter	14287
consensussolidus_consensus_blocks_missed_total Blocks where this validator was elected proposer but failed to produce.	counter	3
consensussolidus_consensus_view_changes_total View changes observed by this node (BFT timeout escalations).	counter	142
validatorsolidus_validator_reputation Reputation score 0-1000 for this validator DID.	gauge	847
validatorsolidus_validator_violation_points Current violation point count 0-3. Threshold = 3 triggers slash.	gauge	0
p2psolidus_p2p_bandwidth_in_bytes Cumulative inbound peer traffic in bytes.	counter	8847291002
p2psolidus_p2p_bandwidth_out_bytes Cumulative outbound peer traffic in bytes.	counter	6219384172
storagesolidus_storage_disk_used_bytes Disk space consumed by the Solidus data directory.	gauge	699035811840

Scrape config for your Prometheus server:

prometheus.yml

scrape_configs:
  - job_name: 'solidus-node'
    scrape_interval: 15s
    static_configs:
      - targets: ['localhost:9652']
        labels:
          environment: 'mainnet'
          tier: 'subnet'

Grafana dashboard

A pre-built Grafana dashboard JSON ships with every release. It mirrors the operator dashboard's overview screen — same panels, same color palette — so the two interfaces feel like the same product.

Solidus Node · OverviewLast 1h · refresh 15s

Block height

solidus_block_height

4,827,392

Peer count

solidus_peer_count

RSS memory

solidus_rss_bytes

1.24 GB

TPS

consensus

1,247

Uptime 30d

solidus_uptime_seconds

99.97%

Violation points

solidus_validator_violation_points

0 / 3

CLEAN

Solidus Node Dashboard

12 panels · works with any Grafana 9+ instance

Dashboard JSON ships with v0.1.0

To import:

1In Grafana, open Dashboards → Import.
2Upload solidus-node-dashboard.json or paste its contents.
3Select your Prometheus data source from the dropdown.
4Click Import. Panels populate within one scrape interval.

Alerting rules

Drop these into your Prometheus alertmanager config. They cover the four conditions worth waking the operator for — anything else should fall through to a daily digest.

alertmanager/rules.yml

groups:
  - name: solidus-node-critical
    rules:
      - alert: NodeDown
        expr: up{job="solidus-node"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Solidus node {{ $labels.instance }} is down"

      - alert: LowPeerCount
        expr: solidus_peer_count < 3
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low peer count: {{ $value }} (expected >= 3)"

      - alert: DiskCritical
        expr: (solidus_storage_disk_used_bytes
               / node_filesystem_size_bytes{mountpoint="/var/lib/solidus"})
              > 0.90
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Disk usage > 90% on {{ $labels.instance }}"

      - alert: SyncStalled
        expr: rate(solidus_block_height[5m]) == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Block height has not advanced for 5 minutes"

Why these four?

They each map directly to a distinct failure mode where the operator must act manually: crashed process (NodeDown), network partition (LowPeerCount), storage exhaustion (DiskCritical), or stuck consensus (SyncStalled). The remaining metrics are useful for capacity planning, not for paging.

Log forwarding

With log_format = "json" in your config, every log line is one JSON object. Pipe to Loki, Elasticsearch, Datadog, or anywhere else that parses JSON.

JSON log line format

example output

{
  "ts":        "2026-03-17T09:41:22.487Z",
  "level":    "INFO",
  "component": "consensus",
  "msg":       "Block finalized",
  "fields": {
    "block_height": 4827392,
    "tx_count":     247,
    "latency_ms":   1832,
    "proposer":     "did:sol:b3d1f8c4a7f2"
  }
}

Loki / Promtail

promtail-config.yaml

scrape_configs:
  - job_name: solidus-node
    journal:
      max_age: 12h
      labels:
        job: solidus-node
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        regex: 'solidus-node.service'
        action: keep
    pipeline_stages:
      - json:
          expressions:
            level:     level
            component: component
      - labels:
          level:
          component:

Elasticsearch / Filebeat

filebeat.yml

filebeat.inputs:
  - type: journald
    id: solidus-node
    include_matches.match:
      - _SYSTEMD_UNIT=solidus-node.service
processors:
  - decode_json_fields:
      fields: ["message"]
      target: "solidus"
      overwrite_keys: true
output.elasticsearch:
  hosts: ["https://es.example.internal:9200"]
  index: "solidus-node-%{+yyyy.MM.dd}"

Health checks

A dedicated /health endpoint on the RPC port reports node health in a Kubernetes-probe-friendly format. Status codes drive the readiness decision; the body is for human eyes.

health check

# Manual check
curl -s http://127.0.0.1:9652/health | jq

# Healthy → 200
# Syncing → 503 (don't route traffic yet)
# Degraded → 200 (still functional, but flagged)
# Critical → 503

Example response — healthy

GET /health

{
  "status":         "healthy",
  "block_height":   4827392,
  "sync_lag":       0,
  "peer_count":     47,
  "uptime_seconds": 1842301,
  "violation_points": 0,
  "checks": {
    "consensus":  "ok",
    "p2p":        "ok",
    "storage":    "ok",
    "validator":  "ok"
  }
}

Kubernetes probes

deployment.yaml

livenessProbe:
  httpGet:
    path: /health
    port: 9652
  initialDelaySeconds: 30
  periodSeconds: 30
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /health
    port: 9652
  initialDelaySeconds: 60
  periodSeconds: 10
  failureThreshold: 2  # 503 during sync is normal — give it room

Don't restart on a 503 during sync

A freshly-restored or recently-restarted node will report syncing (HTTP 503) for the first 5-30 minutes. Set initialDelaySeconds generously on the liveness probe — restarting mid-sync wastes the catchup work.

Up next: Slashing reference · Open operator dashboard →