Telemetry & Monitoring

Prerequisites: Complete the Quick Start guide and ensure your gateway is running with telemetry_enable: true in your configuration.

The Optimum Gateway exposes Prometheus metrics that provide insights into gateway performance, network connectivity, and message processing on the Ethereum Hoodi testnet. This guide shows you how to access these metrics and set up a Grafana dashboard for monitoring.

This setup uses industry-standard monitoring tools:

Grafana: Open-source analytics and interactive visualization web application
Prometheus: Open-source monitoring and alerting toolkit

Gateway Metrics

When telemetry is enabled in your gateway configuration, metrics are exposed at the HTTP endpoint:

Metrics Endpoint: http://localhost:48123/metrics

The gateway exports metrics in Prometheus format, including:

Gateway health and uptime status
Network peer connections (libp2p and mumP2P)
Message throughput and processing latency
Topic subscription status
Connection quality metrics

Accessing Basic Metrics

You can view raw metrics directly from the gateway:

Check if metrics are available:

bash

curl http://localhost:48123/metrics | head -10

Output:

text

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.

View gateway identification:

bash

curl http://localhost:48123/metrics | grep gateway_id

Output:

text

gateway_id{gateway_id="partner_validator_hoodi_001"} 1

Check peer connections:

bash

curl http://localhost:48123/metrics | grep peers_connected

Output (when peers are connected):

text

optimum_gateway_peers_connected{type="libp2p"} 8
optimum_gateway_peers_connected{type="optp2p"} 3

Note: If no output appears, it means no peers are currently connected or the gateway is still initializing.

Gateway Information APIs

The gateway provides REST API endpoints for retrieving gateway information:

Version Endpoint: http://localhost:48123/api/v1/version

bash

curl -s http://localhost:48123/api/v1/version

Output:

json

{
  "version": "v0.0.1-rc1",
  "commit": "abc123d"
}

Self Info Endpoint: http://localhost:48123/api/v1/self_info

This endpoint returns the gateway's peer identity and network addresses:

bash

curl -s http://localhost:48123/api/v1/self_info

Output:

json

{"multiaddrs":["/ip4/172.17.0.4/tcp/33212"],"peer_id":"16Uiu2HAmUmoQs3h574s1uyeBYpkHwNHAGV68YjdUfZKgAG7sdTLL"}

Peer Address Construction:

The full peer address for connecting to this gateway is constructed as:

text

multiaddrs[0] + "/" + "p2p" + "/" + peer_id

For the example above:

text

/ip4/192.168.1.223/tcp/33212/p2p/16Uiu2HAkzmBdRzsvyUBCJqv7Ux2LqbfKHbNi7FqVLbKvem8UF6bT

This information is useful for:

Identifying your gateway in network logs
Sharing connection details with other validators
Troubleshooting peer-to-peer connectivity issues

Gateway Operation & Expected Behavior

Normal Gateway Operation

The gateway runs continuously and will show regular activity in the logs:

text

{"timestamp":1756834302,"_level":"info","short_message":"started finding peers","commit":"(devel)","pkg":"optimum_p2p","ns":"optimum-disc"}
{"timestamp":1756834302,"_level":"info","short_message":"discovered peer","commit":"(devel)","pkg":"optimum_p2p","ns":"optimum-disc","peer":"12D3KooWBkMTKHH5gUoc41GgqhkserMq9yQPjnffckny7HBGye8S"}
{"timestamp":1756834335,"_level":"info","short_message":"service statistic","commit":"(devel)","peers_libp2p":"0","peers_optp2p":"9","address":"16Uiu2HAmMLL4VarycXv1eoqjnzqx23KSdMvMBWaroBuYzfoRLJZN","hosts":"[/ip4/172.17.0.4/tcp/33212]","topics":"0","bad_messages_to_opt":"0","bad_messages_to_cl":"0"}
{"timestamp":1756834336,"_level":"info","short_message":"dumping state","commit":"(devel)","pkg":"optimum_p2p","cluster_id":"optimum_hoodi_v0_1","host_id":"12D3KooWDmgRirMxNjhW29DbSEoPATJRMGroCDFtBHXCydMd1rNf","peers":"9"}

Key indicators of healthy operation:

✅ Peer Discovery: "discovered peer" messages show network connectivity
✅ Service Statistics: Regular stats every ~30 seconds with peer counts
✅ mumP2P Peers: "peers_optp2p":"9" shows successful connections
✅ Gateway Address: Your unique peer ID and multiaddr are logged
⚠️ Connection Errors: "failed to connect to peer" are normal for some remote peers

Gateway Termination Conditions

Normal termination scenarios:

Manual shutdown: Gateway stops cleanly when you press Ctrl+C or stop the container
Consensus Layer timeout: Gateway will terminate if the Consensus Layer (CL) doesn't send any messages for 1 minute
Configuration errors: Invalid settings in app_conf.yml will cause startup failure

Important: If you're running a local test setup without a real Consensus Layer client, the gateway may exit after 1 minute due to no CL message activity. This is expected behavior.

Setting Up Monitoring Dashboard

For comprehensive monitoring and visualization, you can deploy a Grafana dashboard with Prometheus. This provides real-time charts and alerts for your gateway.

Prerequisites

Docker and Docker Compose installed
Optimum Gateway running with telemetry_enable: true
Ports 3000 (Grafana) and 9090 (Prometheus) available

Step 1: Create Monitoring Directory Structure

Create project directory:

bash

mkdir optimum-monitoring
cd optimum-monitoring

Create the required folder structure:

bash

# Create all required directories
mkdir -p prometheus
mkdir -p grafana-provisioning/datasources
mkdir -p grafana-provisioning/dashboards
mkdir -p grafana-dashboards

Your final directory structure will look like this after completing steps 2-5:

text

optimum-monitoring/
├── docker-compose.yml
├── prometheus/
│   ├── prometheus.yml
│   └── targets.json
├── grafana-provisioning/
│   ├── datasources/
│   │   └── prometheus.yaml
│   └── dashboards/
│       └── dashboards.yml
└── grafana-dashboards/
    └── gateway-dashboard.json

What each folder contains:

prometheus/ - Prometheus configuration and target definitions
grafana-provisioning/datasources/ - Automatic datasource configuration
grafana-provisioning/dashboards/ - Dashboard loading configuration
grafana-dashboards/ - Actual dashboard JSON files

Step 2: Create Docker Compose Configuration

Create docker-compose.yml:

yaml

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus:/etc/prometheus
      - prometheus-data:/prometheus
    ports:
      - "9090:9090"
    restart: unless-stopped
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--storage.tsdb.retention.time=1h"
      - "--storage.tsdb.retention.size=2GB"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9090/-/healthy"]
      interval: 30s
      timeout: 10s
      retries: 3
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    volumes:
      - ./grafana-provisioning:/etc/grafana/provisioning:ro
      - ./grafana-dashboards:/var/lib/grafana/dashboards:ro
      - grafana-data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
    restart: unless-stopped
    depends_on:
      - prometheus
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/api/health"]
      interval: 30s
      timeout: 10s
      retries: 3
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

volumes:
  prometheus-data:
  grafana-data:

Step 3: Configure Prometheus

Create prometheus/prometheus.yml:

yaml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'servers'
    file_sd_configs:
      - files:
          - /etc/prometheus/targets.json

Create prometheus/targets.json:

For Docker Desktop (macOS/Windows):

json

[
  {
    "targets": [
      "host.docker.internal:48123"
    ],
    "labels": {
      "job": "servers"
    }
  }
]

For Linux Docker:

json

[
  {
    "targets": [
      "172.17.0.1:48123"
    ],
    "labels": {
      "job": "servers"
    }
  }
]

For host networking (recommended):

json

[
  {
    "targets": [
      "localhost:48123"
    ],
    "labels": {
      "job": "servers"
    }
  }
]

Choose the appropriate configuration based on:

Docker Desktop (Mac/Windows): Use host.docker.internal:48123
Linux Docker: Use 172.17.0.1:48123 or your Docker bridge IP
Host networking: Use localhost:48123 (simplest, recommended)

Step 4: Configure Grafana Provisioning

Create grafana-provisioning/datasources/prometheus.yaml:

yaml

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    access: proxy
    isDefault: true
    uid: PBFA97CFB590B2093

Create grafana-provisioning/dashboards/dashboards.yml:

yaml

apiVersion: 1

providers:
  - name: 'optimum-gateway'
    orgId: 1
    folder: 'Default'
    type: file
    disableDeletion: false
    updateIntervalSeconds: 10
    allowUiUpdates: true
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true

Step 5: Add Gateway Dashboard

Create the gateway dashboard file grafana-dashboards/gateway-dashboard.json:

Click to expand: Complete Gateway Dashboard JSON (Copy this exactly)

json

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "grafana",
          "uid": "-- Grafana --"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": 6,
  "links": [],
  "panels": [
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "id": 17,
      "panels": [],
      "title": "eth_latency",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "PBFA97CFB590B2093"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "ms"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 10,
        "w": 24,
        "x": 0,
        "y": 1
      },
      "id": 16,
      "maxPerRow": 3,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "table",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "single",
          "sort": "none"
        }
      },
      "pluginVersion": "12.1.0",
      "repeat": "gateway_id",
      "repeatDirection": "h",
      "targets": [
        {
          "disableTextWrap": false,
          "editorMode": "builder",
          "expr": "last_over_time(optp2p_gateway_optimum_gateway_eth_latency{gateway_id=\"$gateway_id\"}[1m])",
          "fullMetaSearch": false,
          "includeNullMetadata": true,
          "legendFormat": "{{instance}} - eth latency",
          "range": true,
          "refId": "A",
          "useBackend": false
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
          "disableTextWrap": false,
          "editorMode": "code",
          "expr": "last_over_time(optp2p_gateway_optimum_gateway_libp2p_propagation_latency{gateway_id=\"$gateway_id\", topic=\"/eth2/82556a32/beacon_block/ssz_snappy\"}[1m])",
          "fullMetaSearch": false,
          "hide": false,
          "includeNullMetadata": true,
          "instant": false,
          "legendFormat": "{{instance}} - mumP2P latency",
          "range": true,
          "refId": "B",
          "useBackend": false
        }
      ],
      "title": "latency per node ${gateway_id}",
      "type": "timeseries"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 25
      },
      "id": 13,
      "panels": [],
      "title": "message size",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "PBFA97CFB590B2093"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "decbytes"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 12,
        "w": 24,
        "x": 0,
        "y": 26
      },
      "id": 14,
      "maxPerRow": 3,
      "options": {
        "legend": {
          "calcs": [
            "last"
          ],
          "displayMode": "table",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "single",
          "sort": "none"
        }
      },
      "pluginVersion": "12.1.0",
      "repeat": "gateway_id",
      "repeatDirection": "h",
      "targets": [
        {
          "disableTextWrap": false,
          "editorMode": "builder",
          "expr": "max by(topic) (optp2p_gateway_optimum_gateway_message_size_max{gateway_id=\"$gateway_id\"})",
          "fullMetaSearch": false,
          "includeNullMetadata": true,
          "legendFormat": "{{topic}} - max",
          "range": true,
          "refId": "A",
          "useBackend": false
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
          "disableTextWrap": false,
          "editorMode": "builder",
          "exemplar": false,
          "expr": "max by(topic) (optp2p_gateway_optimum_gateway_message_size_min{gateway_id=\"$gateway_id\"})",
          "format": "time_series",
          "fullMetaSearch": false,
          "hide": false,
          "includeNullMetadata": true,
          "instant": false,
          "legendFormat": "{{topic}} - min",
          "range": true,
          "refId": "B",
          "useBackend": false
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
          "disableTextWrap": false,
          "editorMode": "code",
          "exemplar": false,
          "expr": "avg_over_time(optp2p_gateway_optimum_gateway_message_size_avg{gateway_id=\"$gateway_id\"}[5m])",
          "fullMetaSearch": false,
          "hide": false,
          "includeNullMetadata": true,
          "instant": false,
          "legendFormat": "{{topic}} - avg",
          "range": true,
          "refId": "C",
          "useBackend": false
        }
      ],
      "title": "message size",
      "type": "timeseries"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 38
      },
      "id": 10,
      "panels": [],
      "title": "aggregation",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "PBFA97CFB590B2093"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 11,
        "w": 24,
        "x": 0,
        "y": 39
      },
      "id": 12,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "single",
          "sort": "none"
        }
      },
      "pluginVersion": "12.1.0",
      "targets": [
        {
          "disableTextWrap": false,
          "editorMode": "builder",
          "expr": "sum by(gateway_id) (rate(optp2p_gateway_optimum_gateway_aggregation_included_total[1m]))",
          "fullMetaSearch": false,
          "includeNullMetadata": true,
          "legendFormat": "{{instance}}",
          "range": true,
          "refId": "A",
          "useBackend": false
        }
      ],
      "title": "aggregated messages count in one",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "PBFA97CFB590B2093"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 0,
        "w": 24,
        "x": 0,
        "y": 50
      },
      "id": 11,
      "maxPerRow": 3,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "single",
          "sort": "none"
        }
      },
      "pluginVersion": "12.1.0",
      "repeat": "gateway_id",
      "repeatDirection": "h",
      "targets": [
        {
          "disableTextWrap": false,
          "editorMode": "builder",
          "expr": "optp2p_gateway_optimum_gateway_aggregation_included_total{gateway_id=\"$gateway_id\"}",
          "fullMetaSearch": false,
          "includeNullMetadata": true,
          "legendFormat": "__auto",
          "range": true,
          "refId": "A",
          "useBackend": false
        }
      ],
      "title": "aggregate messages ${gateway_id}",
      "type": "timeseries"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 50
      },
      "id": 9,
      "panels": [],
      "title": "latency",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "PBFA97CFB590B2093"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "ms"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 13,
        "w": 24,
        "x": 0,
        "y": 51
      },
      "id": 8,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "table",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "single",
          "sort": "none"
        }
      },
      "pluginVersion": "12.1.0",
      "repeat": "gateway_id",
      "repeatDirection": "h",
      "targets": [
        {
          "disableTextWrap": false,
          "editorMode": "code",
          "expr": "histogram_quantile(0.95, rate(optp2p_gateway_optimum_gateway_libp2p_propagation_latency_milliseconds_bucket{gateway_id=\"$gateway_id\"}[1m]))",
          "fullMetaSearch": false,
          "includeNullMetadata": true,
          "legendFormat": "{{topic}} - p95",
          "range": true,
          "refId": "A",
          "useBackend": false
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
          "disableTextWrap": false,
          "editorMode": "builder",
          "expr": "histogram_quantile(0.75, rate(optp2p_gateway_optimum_gateway_libp2p_propagation_latency_milliseconds_bucket{gateway_id=\"$gateway_id\"}[1m]))",
          "fullMetaSearch": false,
          "hide": false,
          "includeNullMetadata": true,
          "instant": false,
          "legendFormat": "{{topic}} - p75",
          "range": true,
          "refId": "B",
          "useBackend": false
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
          "expr": "sum by (topic) (\n  rate(optp2p_gateway_optimum_gateway_libp2p_propagation_latency_milliseconds_sum{gateway_id=\"$gateway_id\"}[1m])\n)\n/\nsum by (topic) (\n  rate(optp2p_gateway_optimum_gateway_libp2p_propagation_latency_milliseconds_count{gateway_id=\"$gateway_id\"}[1m])\n)",
          "hide": false,
          "instant": false,
          "legendFormat": "{{topic}} - avg",
          "range": true,
          "refId": "C"
        }
      ],
      "title": "mumP2P latency ${gateway_id}",
      "type": "timeseries"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 64
      },
      "id": 2,
      "panels": [],
      "repeat": "gateway_id",
      "title": "gateway_stat ${gateway_id}",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "PBFA97CFB590B2093"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 10,
        "w": 8,
        "x": 0,
        "y": 65
      },
      "id": 1,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "single",
          "sort": "none"
        }
      },
      "pluginVersion": "12.1.0",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
          "disableTextWrap": false,
          "editorMode": "builder",
          "expr": "rate(optp2p_gateway_optimum_gateway_libp2p_total_messages{gateway_id=\"$gateway_id\"}[1m])",
          "fullMetaSearch": false,
          "includeNullMetadata": true,
          "legendFormat": "{{instance}} pass to CL total messages",
          "range": true,
          "refId": "A",
          "useBackend": false
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
          "disableTextWrap": false,
          "editorMode": "builder",
          "expr": "rate(optp2p_gateway_optimum_gateway_optimum_total_messages{gateway_id=\"$gateway_id\"}[1m])",
          "fullMetaSearch": false,
          "hide": false,
          "includeNullMetadata": true,
          "instant": false,
          "legendFormat": "{{instance}} pass to Optimum network cluster total messages",
          "range": true,
          "refId": "B",
          "useBackend": false
        }
      ],
      "title": "total messages per protocol",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "PBFA97CFB590B2093"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 10,
        "w": 8,
        "x": 8,
        "y": 65
      },
      "id": 4,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "single",
          "sort": "none"
        }
      },
      "pluginVersion": "12.1.0",
      "targets": [
        {
          "disableTextWrap": false,
          "editorMode": "builder",
          "expr": "rate(optp2p_gateway_optimum_gateway_libp2p_published_messages_per_topic_total{gateway_id=\"$gateway_id\"}[1m])",
          "fullMetaSearch": false,
          "includeNullMetadata": true,
          "legendFormat": "{{topic}}",
          "range": true,
          "refId": "A",
          "useBackend": false
        }
      ],
      "title": "messages published into CL per topic",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "PBFA97CFB590B2093"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 10,
        "w": 8,
        "x": 16,
        "y": 65
      },
      "id": 3,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "single",
          "sort": "none"
        }
      },
      "pluginVersion": "12.1.0",
      "targets": [
        {
          "disableTextWrap": false,
          "editorMode": "builder",
          "expr": "rate(optp2p_gateway_optimum_gateway_optimum_published_messages_per_topic_total{gateway_id=\"$gateway_id\"}[1m])",
          "fullMetaSearch": false,
          "includeNullMetadata": true,
          "legendFormat": "{{topic}}",
          "range": true,
          "refId": "A",
          "useBackend": false
        }
      ],
      "title": "messages published into mumP2P protocol per topic",
      "type": "timeseries"
    }
  ],
  "preload": false,
  "refresh": "5s",
  "schemaVersion": 41,
  "tags": [
    "ethereum_hoodi_testnet"
  ],
  "templating": {
    "list": [
      {
        "current": {
          "text": [
            "34_87_46_171",
            "34_159_112_5",
            "146_148_100_154"
          ],
          "value": [
            "34_87_46_171",
            "34_159_112_5",
            "146_148_100_154"
          ]
        },
        "definition": "label_values(gateway_id)",
        "includeAll": true,
        "label": "gateway_id",
        "multi": true,
        "name": "gateway_id",
        "options": [],
        "query": {
          "qryType": 1,
          "query": "label_values(gateway_id)",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
        "refresh": 1,
        "regex": "",
        "type": "query"
      },
      {
        "allowCustomValue": false,
        "current": {
          "text": "",
          "value": ""
        },
        "definition": "label_values(relay_id)",
        "hide": 1,
        "includeAll": false,
        "label": "prysm_ip",
        "name": "prysm_ip",
        "options": [],
        "query": {
          "qryType": 1,
          "query": "label_values(relay_id)",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
        "refresh": 1,
        "regex": "/prysm_ip_name=\"(?<text>[^\"]+)|prysm_ip=\"(?<value>[^\"]+)/g",
        "type": "query"
      }
    ]
  },
  "time": {
    "from": "now-15m",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "browser",
  "title": "Optimum Gateway",
  "uid": "beswatavaqe4gc",
  "version": 163
}

Step 6: Start Monitoring Stack

bash

docker-compose up -d

Output:

text

[+] Running 2/2
 ✔ Container optimum-monitoring-prometheus-1  Started                    0.5s 
 ✔ Container optimum-monitoring-grafana-1     Started                    0.6s

Verify services are running:

bash

docker-compose ps

Output:

text

NAME                                IMAGE                    COMMAND                  SERVICE      CREATED         STATUS                             PORTS
optimum-monitoring-grafana-1      grafana/grafana:latest   "/run.sh"                grafana      42 seconds ago   Up 3 seconds (health: starting)    0.0.0.0:3000->3000/tcp
optimum-monitoring-prometheus-1   prom/prometheus:latest   "/bin/prometheus --c…"   prometheus   42 seconds ago   Up 37 seconds (health: starting)   0.0.0.0:9090->9090/tcp

Important startup timing:

Prometheus starts first and begins scraping gateway metrics immediately
Grafana starts second and automatically connects to Prometheus
Initial data appears within 15-30 seconds of startup
Dashboard loading may take 1-2 minutes for all panels to populate

Accessing the Dashboard

Grafana Web Interface

Open your browser and navigate to: http://localhost:3000
Login credentials:
- Username: admin
- Password: admin
You will be prompted to change the password on first login (you can skip this step).
Navigate to dashboard:
- Click Dashboards in the left sidebar
- Look for the Gateway Dashboard
- Click to open the monitoring dashboard

After logging in, navigate to the dashboard by clicking Dashboards in the left sidebar and selecting Gateway Dashboard.

View live metrics - You'll see exactly this:

Ethereum Hoodi Testnet Latency Panel:

eth latency: 100-800ms with spikes (blue/green lines)
mumP2P latency: Consistently lower ~50ms (yellow line)
Clear performance advantage of mumP2P protocol visible immediately on Hoodi testnet

Message Size Panel:

Hoodi testnet beacon block messages: Consistently ~17.0 kB
Real-time spikes: Up to 120kB during high activity
Multiple topics: /eth2/82556a32/beacon_block/ssz_snappy and others

Multi-Gateway Dashboard:

Template variables: Dropdown showing all gateway IDs (146_148_100_154, 34_159_112_5, 34_87_46_171)
Individual panels: Each gateway gets separate latency and message panels
Peer composition charts: Colorful pie charts showing connected peer diversity

Aggregation Metrics:

Message aggregation: Real count of messages bundled together
Performance comparison: Hoodi vs Hoodi+MUMP2P speedup (5.7x-6.2x improvement)
Live latency tracking: MUMP2P latency around 155-175ms

Prometheus Web Interface

Open your browser and navigate to: http://localhost:9090
Verify gateway targets - You'll see exactly this:
- Go to Status → Targets
- 6 targets total: All showing green "UP" status
- Gateway endpoints: Like http://34.159.112.5:48123/metrics, http://34.87.46.171:48123/metrics
- Scrape intervals: 169ms-978ms (very fast, real-time)
- Last scrape times: Recent timestamps like "620ms ago", "3.671s ago"
What this means:
- Multiple Hoodi testnet gateways are being monitored simultaneously
- All connections are healthy (green UP status)
- Data is being collected in near real-time
- Each gateway exposes metrics on port 48123
Query metrics:
- Go to Graph tab
- Try sample queries:
  - up - Shows which targets are available
  - optimum_gateway_peers_connected - Shows peer connections
  - rate(optimum_gateway_messages_total[5m]) - Message rate

Dashboard Panels Overview

The gateway dashboard provides several monitoring panels:

Gateway Status

Gateway Health: Shows if the gateway is running and responding
Version Information: Displays gateway version and build details
Uptime: How long the gateway has been running

Network Connectivity

Peer Connections: Number of connected libp2p and mumP2P peers
Connection Quality: Latency and stability metrics
Network Throughput: Data sent and received rates

Message Processing

Message Rates: Messages processed per second by topic
Processing Latency: Time taken to process different message types
Topic Subscriptions: Status of Ethereum topic subscriptions

Performance Metrics

Resource Usage: Memory and CPU utilization
Error Rates: Failed connections and processing errors
Alert Conditions: Configured thresholds for monitoring

Stopping the Monitoring Stack

To stop the monitoring services:

bash

docker-compose down

To remove all data and start fresh:

bash

docker-compose down -v

Important: This will delete all historical metrics data stored in Prometheus and Grafana configurations.

The monitoring setup provides comprehensive insights into your gateway's operation and helps ensure optimal performance for validator operations.

Next Steps

Monitoring is now set up! Here's what to do next:

Troubleshooting - Quick fixes if your dashboard shows no data or errors
Configuration - Adjust gateway settings for optimal performance
Quick Start - Return to basics if you need to restart your gateway

Dashboard showing no data? Check that your gateway is running and see our Troubleshooting Guide for solutions.

Telemetry & Monitoring ​

Gateway Metrics ​

Accessing Basic Metrics ​

Gateway Information APIs ​

Gateway Operation & Expected Behavior ​

Normal Gateway Operation ​

Gateway Termination Conditions ​

Setting Up Monitoring Dashboard ​

Prerequisites ​

Step 1: Create Monitoring Directory Structure ​

Step 2: Create Docker Compose Configuration ​

Step 3: Configure Prometheus ​

Step 4: Configure Grafana Provisioning ​

Step 5: Add Gateway Dashboard ​

Step 6: Start Monitoring Stack ​

Accessing the Dashboard ​

Grafana Web Interface ​

Prometheus Web Interface ​

Dashboard Panels Overview ​

Gateway Status ​

Network Connectivity ​

Message Processing ​

Performance Metrics ​

Stopping the Monitoring Stack ​

Next Steps ​

Telemetry & Monitoring

Gateway Metrics

Accessing Basic Metrics

Gateway Information APIs

Gateway Operation & Expected Behavior

Normal Gateway Operation

Gateway Termination Conditions

Setting Up Monitoring Dashboard

Prerequisites

Step 1: Create Monitoring Directory Structure

Step 2: Create Docker Compose Configuration

Step 3: Configure Prometheus

Step 4: Configure Grafana Provisioning

Step 5: Add Gateway Dashboard

Step 6: Start Monitoring Stack

Accessing the Dashboard

Grafana Web Interface

Prometheus Web Interface

Dashboard Panels Overview

Gateway Status

Network Connectivity

Message Processing

Performance Metrics

Stopping the Monitoring Stack

Next Steps