Troubleshooting

Prerequisites: Quick Start and Configuration.

Is My Gateway Running?

curl http://localhost:48123/health
docker logs optimum-gateway --tail=10

Gateway Startup Issues

Container Won't Start

lsof -i :33211 -i :33212 -i :43213 -i :48123
python -c "import yaml; yaml.safe_load(open('config/app_conf.yml'))"
docker logs optimum-gateway --timestamps

Causes: Port conflicts, invalid YAML, missing config, wrong volume mount.

Config File Issues

ls -la config/app_conf.yml
docker inspect optimum-gateway | grep -A 5 "Mounts"

Identity Directory Issues

docker exec optimum-gateway ls -la /tmp/libp2p /tmp/optp2p
mkdir -p data/libp2p data/mump2p

Mount identity dirs as volumes to persist across restarts.

Normal Errors (Ignore These)

text

failed to connect to bootstrap node... i/o timeout
failed to send handshake for peer... connection closed
failed to load topics data... file does not exist

Expected during startup or on first run; does not affect functionality.

Real Connection Issues

Missing gateway_id:

yaml

gateway_id: "your_unique_identifier"

Verify gateway is running:

curl -s http://localhost:48123/health

CL Client Connection Issues

Verify CL Connection

curl -s http://localhost:48123/health | jq '.checks.cl_peers'

When CL is connected: cl_peers.value ≥ 1, subscribed_topics.value = 65.

Lighthouse Not Reconnecting After Gateway Restart

Lighthouse does not re-dial peers automatically. If the gateway restarts, Lighthouse stays disconnected.

Fix: Add your Lighthouse node as a direct_cl_peer in the gateway config:

yaml

direct_cl_peers:
  - /ip4/YOUR_LIGHTHOUSE_IP/tcp/9000/p2p/YOUR_LIGHTHOUSE_PEER_ID

The gateway will run a background goroutine that keeps retrying the connection. See Configuration – Direct CL Peers.

Prysm

curl -s "http://localhost:3500/eth/v1/node/peers" | grep -c "connected"

Fixes: Correct --peer multiaddr; gateway and Prysm reachable on 33212.

Nimbus

Connect to gateway:

nimbus_beacon_node \
  --direct-peer=/ip4/YOUR_GATEWAY_IP/tcp/33212/p2p/YOUR_GATEWAY_PEER_ID \
  --netkey-file=/data/netkey \
  ...

Use --netkey-file with a stable key (not random). Multiaddr is enough; ENR is optional for --direct-peer.

Gateway → Nimbus: Add your Nimbus multiaddr to direct_cl_peers (public IP and --tcp-port, e.g. with --nat=extip: on host networking). See Configuration – Direct CL Peers.

Verify Nimbus sees the gateway:

curl -s http://localhost:9596/eth/v1/node/peers/YOUR_GATEWAY_PEER_ID | jq '.data | {state, agent}'

Expected: "state": "connected" and agent containing optimum-gateway. A one-off "disconnected" can be normal churn — recheck and confirm curl -s http://localhost:48123/health | jq '.checks.cl_peers' shows cl_peers ≥ 1.

Symptom	Cause	Fix
`Adding privileged direct peer` in logs but no connection	Config parsed only; firewall or wrong IP/port	Open 33212 (Nimbus -> gateway) and your Nimbus P2P port (gateway -> Nimbus); use public IP in `direct_cl_peers`
Gateway peer always `disconnected` in REST	Missing `direct_cl_peers` or unstable netkey	Add gateway `direct_cl_peers`; use `--netkey-file`
`restAllowedOrigin: some("\'*\'")` in logs	Quoted wildcard in compose	Use `--rest-allow-origin=*` (no extra quotes); REST/CORS only, not P2P

Lighthouse v8.x PeerDAS Configuration (Important)

Lighthouse v8.x (post-Fulu/PeerDAS) may disconnect from the gateway. Add:

yaml

command:
  - lighthouse
  - beacon_node
  - --network=hoodi
  - --semi-supernode
  - --target-peers=500
  - --libp2p-addresses=/ip4/YOUR_GATEWAY_IP/tcp/33212
  - --trusted-peers=YOUR_GATEWAY_PEER_ID

Why: PeerDAS changes peer requirements; --semi-supernode and --target-peers=500 prevent pruning.

Lighthouse v8.0.0 Release Notes

Teku PeerDAS Configuration (Important)

Minimum version: Teku v26.4.0 or later. Older versions have a known bug (NoSuchElementException at MetadataDasPeerCustodyTracker.onPeerMetadataUpdate) that causes recurring errors when exchanging metadata with peers.

Use --p2p-direct-peers (not just --p2p-static-peers): --p2p-static-peers only attempts an initial connection — Teku can prune the gateway during peer management. --p2p-direct-peers maintains a persistent, protected connection that won't be dropped.

teku \
  --network=hoodi \
  --p2p-direct-peers=/ip4/YOUR_GATEWAY_IP/tcp/33212/p2p/YOUR_GATEWAY_PEER_ID \
  --p2p-static-peers=/ip4/YOUR_GATEWAY_IP/tcp/33212/p2p/YOUR_GATEWAY_PEER_ID \
  --p2p-subscribe-all-subnets-enabled \
  --p2p-subscribe-all-custody-subnets-enabled \
  --checkpoint-sync-url=https://hoodi.beaconstate.info \
  --ee-endpoint=http://YOUR_EL_HOST:8551 \
  --ee-jwt-secret-file=/path/to/jwt.hex \
  --rest-api-enabled \
  --rest-api-host-allowlist=* \
  --metrics-enabled \
  --metrics-host-allowlist=* \
  --validators-builder-registration-default-enabled \
  --validators-proposer-default-fee-recipient=YOUR_FEE_RECIPIENT

Verify Teku is connected:

curl -s http://localhost:48123/health | jq '.checks.cl_peers'

Expected: cl_peers.value ≥ 1. In Teku logs, look for Connected peers: 1 (or more) in the sync status lines.

Common Teku + Gateway issues:

Symptom	Cause	Fix
`NoSuchElementException` at `MetadataDasPeerCustodyTracker`	Teku version too old	Upgrade to v26.4.0+
Gateway peer keeps disconnecting	Using `--p2p-static-peers` only	Add `--p2p-direct-peers` flag
`Payload marked as invalid by Execution Client`	EL (Geth/Nethermind) still syncing	Wait for EL to finish syncing
Sync stalled at "Waiting for execution layer sync"	EL not caught up or incompatible Engine API	Ensure EL supports Fulu Engine API (e.g., `engine_getPayloadV5`)

Getting Peer Information

curl -s http://localhost:48123/api/v1/self_info | jq -r '.peer_id'

Network & Ports

Port	Direction	Usage	Open in firewall
33211	Inbound	Sidecar gRPC (mump2p tracer)	Yes
33212	Inbound	libp2p listener for CL peers	Yes
48123	Inbound	Telemetry API (`/health`, `/api/v1/self_info`, `/metrics`)	Yes
43213	Outbound	mump2p egress to Optimum network	No (outbound only)

sudo lsof -i :33211 -i :33212 -i :43213 -i :48123
sudo ufw allow 33211/tcp
sudo ufw allow 33212/tcp
sudo ufw allow 48123/tcp

Only 33211, 33212, and 48123 need UFW allow rules; 43213 is outbound-only.

Docker: Use --network host for simplest setup.

Fork Digest

Fork digest is resolved automatically. Use short topic names only:

yaml

eth_topics_subscribe:
  - beacon_block
  - beacon_attestation_0
  # ... through beacon_attestation_63

If topics fail, verify chain matches your network (mainnet vs hoodi).

Mainnet: set chain: mainnet and restart — if chain stays hoodi, use /api/v1/self_info to confirm values after fixing config.

Telemetry & Monitoring

No data in Grafana:

curl http://localhost:48123/metrics | grep optp2p_gateway

Ensure telemetry_enable: true and Prometheus can reach the gateway.

Health degraded:

curl http://localhost:48123/health | jq '.failing'

Check which check is failing and address accordingly.

Quick Commands

docker restart optimum-gateway
docker logs optimum-gateway -f
curl http://localhost:48123/health
curl http://localhost:48123/metrics | grep gateway_id

Troubleshooting ​

Is My Gateway Running? ​

Gateway Startup Issues ​

Container Won't Start ​

Config File Issues ​

Identity Directory Issues ​

Normal Errors (Ignore These) ​

Real Connection Issues ​

CL Client Connection Issues ​

Verify CL Connection ​

Lighthouse Not Reconnecting After Gateway Restart ​

Prysm ​

Nimbus ​

Lighthouse v8.x PeerDAS Configuration (Important) ​

Teku PeerDAS Configuration (Important) ​

Getting Peer Information ​

Network & Ports ​

Fork Digest ​

Telemetry & Monitoring ​

Quick Commands ​

Troubleshooting

Is My Gateway Running?

Gateway Startup Issues

Container Won't Start

Config File Issues

Identity Directory Issues

Normal Errors (Ignore These)

Real Connection Issues

CL Client Connection Issues

Verify CL Connection

Lighthouse Not Reconnecting After Gateway Restart

Prysm

Nimbus

Lighthouse v8.x PeerDAS Configuration (Important)

Teku PeerDAS Configuration (Important)

Getting Peer Information

Network & Ports

Fork Digest

Telemetry & Monitoring

Quick Commands