Telemetry & Monitoring

Prerequisites: Quick Start with telemetry_enable: true.

Endpoints

Endpoint	Description
`GET /health`	Structured health check with 200/503 status
`GET /api/v1/self_info`	Gateway identity, peers, config, and pairing mode
`GET /metrics`	Prometheus metrics

/api/v1/version is removed in RC12. Version info is available in /health and /api/v1/self_info.

Health Endpoint

GET /health returns 200 (healthy) or 503 (degraded) based on four checks:

json

{
  "status": "healthy",
  "gateway_id": "optimum-dev-hoodi-kubernetes-validator-lighthouse",
  "version": "v0.0.1-rc12",
  "commit_hash": "a0b2bc1",
  "uptime_seconds": 1639,
  "checks": {
    "cl_peers": {"status": "ok", "value": 1},
    "last_block_age_sec": {"status": "ok", "value": 1},
    "mump2p_peers": {"status": "ok", "value": 13},
    "subscribed_topics": {"status": "ok", "value": 65}
  }
}

Check	Passes when
`cl_peers`	≥ 1 CL peer connected
`mump2p_peers`	≥ 1 mump2p peer connected
`subscribed_topics`	≥ 1 topic subscribed
`last_block_age_sec`	Last block received < 60s ago

If any check fails, status becomes "degraded" and the failing checks are listed in "failing".

Self Info

GET /api/v1/self_info returns gateway identity and peer information for CL client connection and operational visibility.

Example response:

json

{
  "ab_testing_enable": true,
  "chain": "hoodi",
  "commit_hash": "a0b2bc1",
  "fork_digest": "c6ecb76c",
  "gateway_cluster_id": "optimum_hoodi_v0_2",
  "gateway_id": "optimum-dev-hoodi-kubernetes-validator-lighthouse",
  "paired_with": "partner",
  "remote_url": "dev-bootstrap.getoptimum.io",
  "version": "v0.0.1-rc12",
  "skip_messages_from_self": true,
  "peer_id": "12D3KooWNKZuPvVw5Sfnbq3nvyukxmhBBPZUXHeqzwcehmmwnKcR",
  "libp2p": {
    "direct_peers": {
      "16Uiu2HAmDv5wLBz48dbi7atvUhcF7sefwQjwKLSrMPqLCYZceYCV": {
        "ID": "16Uiu2HAmDv5wLBz48dbi7atvUhcF7sefwQjwKLSrMPqLCYZceYCV",
        "Addrs": ["/ip4/10.2.12.23/tcp/9000"]
      }
    },
    "multiaddrs": ["/ip4/10.0.0.10/tcp/33212", "/ip4/35.237.206.17/tcp/33212"],
    "peer_ids": ["16Uiu2HAmDv5wLBz48dbi7atvUhcF7sefwQjwKLSrMPqLCYZceYCV"],
    "peers_per_topic": {
      "/eth2/c6ecb76c/beacon_block/ssz_snappy": 1,
      "/eth2/c6ecb76c/beacon_attestation_0/ssz_snappy": 1
    },
    "total_peers": 1
  },
  "mump2p": {
    "peer_ids": ["12D3KooWQKjdLGDYu1b2p4NJ39iuFsVArXmkrA77co7a2gDD94uv", "..."],
    "peers_per_topic": {
      "/eth2/c6ecb76c/beacon_block/ssz_snappy": 5,
      "mump2p_aggregated_messages": 5
    },
    "total_peers": 13
  },
  "rlnc_config": {
    "forward_shard_threshold": 0.75,
    "publisher_shard_multiplier": 1.2,
    "random_message_size_bytes": 512,
    "rlnc_shard_factor": 4
  }
}

Use a multiaddr from libp2p.multiaddrs that is reachable from your CL host and peer_id when configuring your CL client: --peer=/ip4/YOUR_IP/tcp/33212/p2p/YOUR_PEER_ID.

Gateway Metrics

Endpoint: GET /metrics

Metrics are labeled with gateway_id and gateway_cluster_id. See Metrics Reference for the full list.

CL connected: When a CL client connects, optp2p_gateway_optimum_gateway_cl_peers goes from 0 to ≥1. Messages flow on both block and attestation topics.

Logs

Gateway logs are JSON lines. Use docker logs optimum-gateway to inspect them. Key fields: fork_digest (e.g. c6ecb76c for hoodi) appears in startup, bootstrap update, and topic names.

Setting Up the Monitoring Dashboard

This section walks through deploying a local Prometheus + Grafana stack that scrapes your gateway and loads the Partner Dashboard automatically.

Prerequisites

Docker and Docker Compose installed
Optimum Gateway running with telemetry_enable: true
Ports 3000 (Grafana) and 9090 (Prometheus) available

Step 1: Create Monitoring Directory

bash

mkdir -p optimum-monitoring/{prometheus,grafana-provisioning/datasources,grafana-provisioning/dashboards,grafana-dashboards}
cd optimum-monitoring

Your final structure:

text

optimum-monitoring/
├── docker-compose.yml
├── prometheus/
│   ├── prometheus.yml
│   └── targets.json
├── grafana-provisioning/
│   ├── datasources/
│   │   └── prometheus.yaml
│   └── dashboards/
│       └── dashboards.yml
└── grafana-dashboards/
    └── partner-dashboard.json

Step 2: Docker Compose

Create docker-compose.yml:

yaml

services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus:/etc/prometheus
      - prometheus-data:/prometheus
    ports:
      - "9090:9090"
    restart: unless-stopped
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--storage.tsdb.retention.time=1h"
      - "--storage.tsdb.retention.size=2GB"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9090/-/healthy"]
      interval: 30s
      timeout: 10s
      retries: 3
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    volumes:
      - ./grafana-provisioning:/etc/grafana/provisioning:ro
      - ./grafana-dashboards:/var/lib/grafana/dashboards:ro
      - grafana-data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
    restart: unless-stopped
    depends_on:
      - prometheus
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/api/health"]
      interval: 30s
      timeout: 10s
      retries: 3
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

volumes:
  prometheus-data:
  grafana-data:

Step 3: Prometheus Configuration

Create prometheus/prometheus.yml:

yaml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'gateway'
    file_sd_configs:
      - files:
          - /etc/prometheus/targets.json

Create prometheus/targets.json (choose your platform):

Docker Desktop (macOS / Windows):

json

[
  {
    "targets": ["host.docker.internal:48123"],
    "labels": { "job": "gateway" }
  }
]

Linux Docker:

json

[
  {
    "targets": ["172.17.0.1:48123"],
    "labels": { "job": "gateway" }
  }
]

If you run Hoodi and Mainnet gateways at the same time, add one scrape entry per gateway in targets.json (different ports); use the dashboard Network dropdown to switch between them.

Step 4: Grafana Provisioning

Create grafana-provisioning/datasources/prometheus.yaml:

yaml

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    access: proxy
    isDefault: true

Create grafana-provisioning/dashboards/dashboards.yml:

yaml

apiVersion: 1

providers:
  - name: 'optimum-gateway'
    orgId: 1
    folder: 'Default'
    type: file
    disableDeletion: false
    updateIntervalSeconds: 10
    allowUiUpdates: true
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true

Step 5: Add the Partner Dashboard

Copy the JSON below into grafana-dashboards/partner-dashboard.json:

Click to expand: Partner Dashboard JSON (v0.0.1-rc12)

json

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "grafana",
          "uid": "-- Grafana --"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "description": "Partner-facing dashboard for Optimum Gateway v0.0.1-rc12. All metrics sourced from the gateway /metrics endpoint. Select Network (Hoodi or Mainnet), then your gateway.",
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 1,
  "id": null,
  "links": [],
  "panels": [
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "id": 1,
      "panels": [],
      "title": "Gateway Info",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${ds_prometheus}"
      },
      "description": "ON when both CL and mump2p peers are connected to the gateway.",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [
            {
              "options": {
                "0": {
                  "color": "red",
                  "index": 1,
                  "text": "OFF"
                },
                "1": {
                  "color": "green",
                  "index": 0,
                  "text": "ON"
                }
              },
              "type": "value"
            }
          ],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "red",
                "value": 0
              },
              {
                "color": "green",
                "value": 1
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 5,
        "x": 0,
        "y": 1
      },
      "id": 2,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "auto",
        "percentChangeColorMode": "standard",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "auto",
        "wideLayout": true
      },
      "pluginVersion": "12.3.1",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${ds_prometheus}"
          },
          "editorMode": "code",
          "expr": "(max by(gateway_id) (last_over_time(optp2p_gateway_optimum_gateway_cl_peers{gateway_id=\"$gateway\"}[$__rate_interval])) > bool 0) * (max by(gateway_id) (last_over_time(optp2p_gateway_optimum_gateway_mump2p_peers{gateway_id=\"$gateway\"}[$__rate_interval])) > bool 0)",
          "instant": true,
          "legendFormat": "__auto",
          "range": false,
          "refId": "A"
        }
      ],
      "title": "Status",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${ds_prometheus}"
      },
      "description": "peers connected to the gateway.",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "red",
                "value": 0
              },
              {
                "color": "green",
                "value": 1
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 5,
        "x": 0,
        "y": 5
      },
      "id": 3,
      "options": {
        "colorMode": "value",
        "graphMode": "area",
        "justifyMode": "auto",
        "orientation": "auto",
        "percentChangeColorMode": "standard",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "value_and_name",
        "wideLayout": true
      },
      "pluginVersion": "12.3.1",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${ds_prometheus}"
          },
          "editorMode": "code",
          "expr": "optp2p_gateway_optimum_gateway_cl_peers{gateway_id=\"$gateway\"}",
          "legendFormat": "CL Peers",
          "range": true,
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${ds_prometheus}"
          },
          "editorMode": "code",
          "expr": "optp2p_gateway_optimum_gateway_mump2p_peers{gateway_id=\"$gateway\"}",
          "hide": false,
          "instant": false,
          "legendFormat": "mump2p Peers",
          "range": true,
          "refId": "B"
        }
      ],
      "title": "Peers",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${ds_prometheus}"
      },
      "description": "Gateway version, commit, Go version, and public IP.",
      "fieldConfig": {
        "defaults": {
          "color": {
            "fixedColor": "text",
            "mode": "fixed"
          },
          "custom": {
            "align": "auto",
            "cellOptions": {
              "type": "auto"
            },
            "filterable": false,
            "footer": {
              "reducers": []
            },
            "inspect": false
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 7,
        "x": 5,
        "y": 1
      },
      "id": 5,
      "options": {
        "cellHeight": "sm",
        "frameIndex": 0,
        "showHeader": false
      },
      "pluginVersion": "12.3.1",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${ds_prometheus}"
          },
          "editorMode": "code",
          "expr": "optp2p_gateway_optimum_gateway_app_build_info{gateway_id=\"$gateway\"}",
          "instant": true,
          "legendFormat": "__auto",
          "range": false,
          "refId": "A"
        }
      ],
      "title": "Build Info",
      "transformations": [
        {
          "id": "labelsToFields",
          "options": {
            "keepLabels": [
              "version",
              "commit",
              "go",
              "public_ip"
            ],
            "mode": "rows"
          }
        }
      ],
      "type": "table"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${ds_prometheus}"
      },
      "description": "Number of active libp2p and mump2p topics this gateway is subscribed to.",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "red",
                "value": 0
              },
              {
                "color": "green",
                "value": 1
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 4,
        "x": 12,
        "y": 1
      },
      "id": 10,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "center",
        "orientation": "auto",
        "percentChangeColorMode": "standard",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "value_and_name",
        "wideLayout": true
      },
      "pluginVersion": "12.3.1",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${ds_prometheus}"
          },
          "editorMode": "code",
          "expr": "sum(optimum_p2p_active_topics{gateway_id=\"$gateway\"})",
          "legendFormat": "Active Topics",
          "refId": "A"
        }
      ],
      "title": "Subscribed Topics",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${ds_prometheus}"
      },
      "description": "CL and mump2p peer count over time.",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "peers",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "smooth",
            "lineWidth": 2,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "showValues": false,
            "spanNulls": true,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              }
            ]
          },
          "unit": "short"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "CL peers"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "green",
                  "mode": "fixed"
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "mump2p peers"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "blue",
                  "mode": "fixed"
                }
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 8,
        "w": 8,
        "x": 16,
        "y": 1
      },
      "id": 11,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "multi",
          "sort": "desc"
        }
      },
      "pluginVersion": "12.3.1",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${ds_prometheus}"
          },
          "editorMode": "code",
          "expr": "optp2p_gateway_optimum_gateway_cl_peers{gateway_id=\"$gateway\"}",
          "legendFormat": "CL peers",
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${ds_prometheus}"
          },
          "editorMode": "code",
          "expr": "optp2p_gateway_optimum_gateway_mump2p_peers{gateway_id=\"$gateway\"}",
          "hide": false,
          "legendFormat": "mump2p peers",
          "refId": "B"
        }
      ],
      "title": "Peer Count over Time",
      "type": "timeseries"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 9
      },
      "id": 100,
      "panels": [],
      "title": "Block Arrival Performance",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${ds_prometheus}"
      },
      "description": "Percentage of blocks first seen via mump2p vs libp2p over the last 5 minutes.",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "max": 100,
          "min": 0,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "yellow",
                "value": 0
              },
              {
                "color": "green",
                "value": 60
              },
              {
                "color": "dark-green",
                "value": 90
              }
            ]
          },
          "unit": "percent"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 6,
        "w": 8,
        "x": 0,
        "y": 10
      },
      "id": 101,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "auto",
        "percentChangeColorMode": "standard",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "value",
        "wideLayout": true
      },
      "pluginVersion": "12.3.1",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${ds_prometheus}"
          },
          "editorMode": "code",
          "expr": "100 * sum(rate(optp2p_gateway_optimum_gateway_blocks_first_seen_mump2p_total{gateway_id=\"$gateway\"}[$prom_interval]))\n/\n(\n  sum(rate(optp2p_gateway_optimum_gateway_blocks_first_seen_mump2p_total{gateway_id=\"$gateway\"}[$prom_interval]))\n  +\n  sum(rate(optp2p_gateway_optimum_gateway_blocks_first_seen_libp2p_total{gateway_id=\"$gateway\"}[$prom_interval]))\n)",
          "legendFormat": "mump2p first",
          "refId": "A"
        }
      ],
      "title": "Accelerated slots (last 5m)",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${ds_prometheus}"
      },
      "description": "Percentage of slots where mump2p was strictly faster than libp2p over time for this gateway. Greener = higher share.",
      "fieldConfig": {
        "defaults": {
          "custom": {
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "scaleDistribution": {
              "type": "linear"
            }
          },
          "unit": "percent"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 6,
        "w": 16,
        "x": 8,
        "y": 10
      },
      "id": 105,
      "interval": "30s",
      "options": {
        "calculate": false,
        "cellGap": 1,
        "color": {
          "exponent": 0.5,
          "fill": "dark-orange",
          "max": 100,
          "min": 0,
          "mode": "scheme",
          "reverse": true,
          "scale": "exponential",
          "scheme": "Greens",
          "steps": 64
        },
        "exemplars": {
          "color": "rgba(255,0,255,0.7)"
        },
        "filterValues": {
          "le": 1e-9
        },
        "legend": {
          "show": true
        },
        "rowsFrame": {
          "layout": "auto"
        },
        "tooltip": {
          "mode": "single",
          "showColorScale": false,
          "yHistogram": false
        },
        "yAxis": {
          "axisPlacement": "left",
          "reverse": false
        }
      },
      "pluginVersion": "12.3.1",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${ds_prometheus}"
          },
          "editorMode": "code",
          "expr": "100 * sum(rate(optp2p_gateway_optimum_gateway_blocks_first_seen_mump2p_total{gateway_id=\"$gateway\"}[$prom_interval])) / (sum(rate(optp2p_gateway_optimum_gateway_blocks_first_seen_mump2p_total{gateway_id=\"$gateway\"}[$prom_interval])) + sum(rate(optp2p_gateway_optimum_gateway_blocks_first_seen_libp2p_total{gateway_id=\"$gateway\"}[$prom_interval])))",
          "legendFormat": "mump2p boost",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Accelerated slots over time",
      "type": "heatmap"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${ds_prometheus}"
      },
      "description": "Median block arrival time via mump2p from slot start for this gateway over time.",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "smooth",
            "lineWidth": 2,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "showValues": false,
            "spanNulls": true,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              }
            ]
          },
          "unit": "ms"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "mump2p"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "#73BF69",
                  "mode": "fixed"
                }
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 8,
        "w": 24,
        "x": 0,
        "y": 16
      },
      "id": 104,
      "options": {
        "legend": {
          "calcs": [
            "mean",
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "multi",
          "sort": "desc"
        }
      },
      "pluginVersion": "12.3.1",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${ds_prometheus}"
          },
          "editorMode": "code",
          "expr": "histogram_quantile(0.5, sum by(le) (rate(optp2p_gateway_optimum_gateway_block_arrival_mump2p_ms_bucket{gateway_id=\"$gateway\"}[$prom_interval])))",
          "legendFormat": "mump2p",
          "refId": "A"
        }
      ],
      "title": "mump2p arrival over time (median)",
      "type": "timeseries"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 24
      },
      "id": 300,
      "panels": [],
      "title": "Attestation Performance: Gateway Level",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${ds_prometheus}"
      },
      "description": "Percentage of attestations delivered via the selected gateway within the 8-second slot deadline.",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "fixed"
          },
          "mappings": [],
          "max": 100,
          "min": 0,
          "noValue": "no data",
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              }
            ]
          },
          "unit": "percent"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "at gateway"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "green",
                  "mode": "fixed"
                }
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 7,
        "w": 8,
        "x": 0,
        "y": 25
      },
      "id": 504,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "center",
        "orientation": "horizontal",
        "percentChangeColorMode": "standard",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "value_and_name",
        "wideLayout": true
      },
      "pluginVersion": "12.3.1",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${ds_prometheus}"
          },
          "editorMode": "code",
          "expr": "100 * (\n  sum(rate(optp2p_gateway_optimum_gateway_attestation_arrival_in_slot_mump2p_ms_bucket{le=\"8000\", gateway_id=\"$gateway\", gateway_cluster_id=~\"$network\"}[5m]))\n  /\n  sum(rate(optp2p_gateway_optimum_gateway_attestation_arrival_in_slot_mump2p_ms_count{gateway_id=\"$gateway\", gateway_cluster_id=~\"$network\"}[5m]))\n)",
          "legendFormat": "at gateway",
          "range": true,
          "refId": "B"
        }
      ],
      "title": "Attestations received before 8s (from slot start)",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${ds_prometheus}"
      },
      "description": "End-to-end attestation propagation latency across the mump2p network over time. Measures time from sender gateway emit to receiver gateway decode.",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "ms",
            "axisPlacement": "left",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "smooth",
            "lineWidth": 2,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "showValues": false,
            "spanNulls": true,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              }
            ]
          },
          "unit": "ms"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "p50"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "green",
                  "mode": "fixed"
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "p95"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "yellow",
                  "mode": "fixed"
                }
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 7,
        "w": 16,
        "x": 8,
        "y": 25
      },
      "id": 310,
      "options": {
        "legend": {
          "calcs": [
            "mean",
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "multi",
          "sort": "desc"
        }
      },
      "pluginVersion": "12.3.1",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${ds_prometheus}"
          },
          "editorMode": "code",
          "expr": "histogram_quantile(0.5, sum by(le) (rate(optp2p_gateway_optimum_gateway_attestation_propagation_latency_ms_bucket{gateway_id=\"$gateway\"}[5m])))",
          "legendFormat": "p50",
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${ds_prometheus}"
          },
          "editorMode": "code",
          "expr": "histogram_quantile(0.95, sum by(le) (rate(optp2p_gateway_optimum_gateway_attestation_propagation_latency_ms_bucket{gateway_id=\"$gateway\"}[5m])))",
          "hide": false,
          "legendFormat": "p95",
          "refId": "B"
        }
      ],
      "title": "Attestation propagation at $gateway",
      "type": "timeseries"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 32
      },
      "id": 507,
      "panels": [],
      "title": "Attestation Performance: Fleet Level",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${ds_prometheus}"
      },
      "description": "Percentage of attestations delivered before the 8s mark from the beginning of the slot.",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "fixed"
          },
          "mappings": [],
          "max": 100,
          "min": 0,
          "noValue": "no data",
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              }
            ]
          },
          "unit": "percent"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "in mump2p network"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "green",
                  "mode": "fixed"
                }
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 7,
        "w": 8,
        "x": 0,
        "y": 33
      },
      "id": 505,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "center",
        "orientation": "horizontal",
        "percentChangeColorMode": "standard",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "value_and_name",
        "wideLayout": true
      },
      "pluginVersion": "12.3.1",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${ds_prometheus}"
          },
          "editorMode": "code",
          "expr": "100 * (\n  sum(rate(optp2p_gateway_optimum_gateway_attestation_arrival_in_slot_mump2p_ms_bucket{le=\"8000\", gateway_cluster_id=~\"$network\"}[5m]))\n  /\n  sum(rate(optp2p_gateway_optimum_gateway_attestation_arrival_in_slot_mump2p_ms_count{gateway_cluster_id=~\"$network\"}[5m]))\n)",
          "legendFormat": "in mump2p network",
          "range": true,
          "refId": "B"
        }
      ],
      "title": "Attestations delivered before 8s (from slot start)",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${ds_prometheus}"
      },
      "description": "How long it takes for attestations to reach 50% and 95% of the nodes in mump2p once they are injected in the network. ",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "ms",
            "axisPlacement": "left",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "smooth",
            "lineWidth": 2,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "showValues": false,
            "spanNulls": true,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              }
            ]
          },
          "unit": "ms"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "p50"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "green",
                  "mode": "fixed"
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "p95"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "yellow",
                  "mode": "fixed"
                }
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 7,
        "w": 16,
        "x": 8,
        "y": 33
      },
      "id": 506,
      "options": {
        "legend": {
          "calcs": [
            "mean",
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "multi",
          "sort": "desc"
        }
      },
      "pluginVersion": "12.3.1",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${ds_prometheus}"
          },
          "editorMode": "code",
          "expr": "histogram_quantile(0.5, sum by(le) (rate(optp2p_gateway_optimum_gateway_attestation_propagation_latency_ms_bucket{gateway_cluster_id=~\"$network\"}[5m])))",
          "legendFormat": "p50",
          "range": true,
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${ds_prometheus}"
          },
          "editorMode": "code",
          "expr": "histogram_quantile(0.95, sum by(le) (rate(optp2p_gateway_optimum_gateway_attestation_propagation_latency_ms_bucket{gateway_cluster_id=~\"$network\"}[5m])))",
          "hide": false,
          "legendFormat": "p95",
          "range": true,
          "refId": "B"
        }
      ],
      "title": "mump2p network attestation propagation",
      "type": "timeseries"
    }
  ],
  "preload": false,
  "refresh": "5s",
  "schemaVersion": 42,
  "tags": [
    "optimum",
    "partner",
    "v0.0.1-rc12"
  ],
  "templating": {
    "list": [
      {
        "current": {},
        "includeAll": false,
        "label": "Prometheus",
        "name": "ds_prometheus",
        "options": [],
        "query": "prometheus",
        "refresh": 1,
        "regex": "",
        "type": "datasource",
        "hide": 0
      },
      {
        "current": {
          "text": "Hoodi",
          "value": "optimum_hoodi_.*"
        },
        "includeAll": false,
        "label": "Network",
        "name": "network",
        "options": [
          {
            "selected": true,
            "text": "Hoodi",
            "value": "optimum_hoodi_.*"
          },
          {
            "selected": false,
            "text": "Mainnet",
            "value": "optimum_ethereum_mainnet_.*"
          }
        ],
        "query": "Hoodi : optimum_hoodi_.*, Mainnet : optimum_ethereum_mainnet_.*",
        "type": "custom"
      },
      {
        "current": {},
        "datasource": {
          "type": "prometheus",
          "uid": "${ds_prometheus}"
        },
        "definition": "label_values(optp2p_gateway_optimum_gateway_app_build_info{gateway_cluster_id=~\"$network\"}, gateway_id)",
        "includeAll": false,
        "label": "Gateway",
        "name": "gateway",
        "options": [],
        "query": {
          "qryType": 1,
          "query": "label_values(optp2p_gateway_optimum_gateway_app_build_info{gateway_cluster_id=~\"$network\"}, gateway_id)",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
        "refresh": 2,
        "regex": "",
        "sort": 1,
        "type": "query"
      },
      {
        "current": {
          "text": "5m",
          "value": "5m"
        },
        "hide": 2,
        "label": "Prometheus Interval",
        "name": "prom_interval",
        "query": "5m",
        "skipUrlSync": true,
        "type": "constant"
      }
    ]
  },
  "time": {
    "from": "now-15m",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "browser",
  "title": "Optimum Gateway - Partner Dashboard (v0.0.1-rc12)",
  "uid": "partner-gateway-rc12"
}

Step 6: Start the Stack

bash

docker compose up -d
docker compose ps

You should see both prometheus and grafana with status Up.

Step 7: Access Grafana

Open http://localhost:3000
Login: admin / admin (skip password change)
Go to Dashboards > Default > select the partner dashboard

The dashboard auto-selects your Prometheus datasource and discovers gateway(s) via the gateway_id label.

Dashboard Panels

The Partner Dashboard includes the following sections:

Gateway Info

Status - ON/OFF based on CL + mump2p peer connectivity
CL Peers / mump2p Peers - current peer counts
Hoodi Slot - live slot number from Hoodi genesis
Hoodi Epoch - epoch index (32 slots per epoch)
Build Info - version, commit, Go, public IP
Subscribed Topics - number of subscribed topics

Block Arrival Performance

Arrival via mump2p (median) - median block arrival time via mump2p from slot start
Accelerated slots - percentage of slots where mump2p delivered the block strictly before libp2p
Accelerated slots over time - heatmap of accelerated slot percentage over time
mump2p arrival over time (median) - arrival time trend

Attestation Performance

Accelerated attestations - percentage of attestations where mump2p delivered first
Accelerated attestations over time - trend of attestation race wins
Attestations delivered before 8s deadline - percentage of mump2p attestations arriving within the 8-second slot deadline

Prometheus Queries (Quick Reference)

All queries use gateway-local metrics from the /metrics endpoint.

What	PromQL
CL peers	`optp2p_gateway_optimum_gateway_cl_peers{gateway_id="$gateway"}`
mump2p peers	`optp2p_gateway_optimum_gateway_mump2p_peers{gateway_id="$gateway"}`
Block arrival p50 via mump2p	`histogram_quantile(0.50, sum by(le) (rate(optp2p_gateway_optimum_gateway_block_arrival_mump2p_ms_bucket{gateway_id="$gateway"}[5m])))`
Accelerated slots %	`rate(blocks_first_seen_mump2p_total[5m]) / (rate(blocks_first_seen_mump2p_total[5m]) + rate(blocks_first_seen_libp2p_total[5m])) * 100`
Attestation arrival p50 via mump2p	`histogram_quantile(0.50, sum by(le) (rate(optp2p_gateway_optimum_gateway_attestation_arrival_in_slot_mump2p_ms_bucket{gateway_id="$gateway"}[5m])))`
Attestation first-seen mump2p	`rate(optp2p_gateway_optimum_gateway_attestation_first_seen_mump2p_total{gateway_id="$gateway"}[5m])`

Stopping the Stack

bash

docker compose down        # stop, keep data
docker compose down -v     # stop, delete all data

References

Metrics Reference - metric names and types
Metrics Methodology - why and how we measure
Configuration - gateway settings
Troubleshooting - fixing dashboard / gateway issues

Telemetry & Monitoring ​

Endpoints ​

Health Endpoint ​

Self Info ​

Gateway Metrics ​

Logs ​

Setting Up the Monitoring Dashboard ​

Prerequisites ​

Step 1: Create Monitoring Directory ​

Step 2: Docker Compose ​

Step 3: Prometheus Configuration ​

Step 4: Grafana Provisioning ​

Step 5: Add the Partner Dashboard ​

Step 6: Start the Stack ​

Step 7: Access Grafana ​

Dashboard Panels ​

Gateway Info ​

Block Arrival Performance ​

Attestation Performance ​

Prometheus Queries (Quick Reference) ​

Stopping the Stack ​

References ​

Telemetry & Monitoring

Endpoints

Health Endpoint

Self Info

Gateway Metrics

Logs

Setting Up the Monitoring Dashboard

Prerequisites

Step 1: Create Monitoring Directory

Step 2: Docker Compose

Step 3: Prometheus Configuration

Step 4: Grafana Provisioning

Step 5: Add the Partner Dashboard

Step 6: Start the Stack

Step 7: Access Grafana

Dashboard Panels

Gateway Info

Block Arrival Performance

Attestation Performance

Prometheus Queries (Quick Reference)

Stopping the Stack

References