Monitoring - NATS Server

Overview

NATS Server provides a comprehensive set of built-in HTTP monitoring endpoints that expose real-time server metrics, connection details, and operational statistics. These endpoints enable deep observability into your NATS infrastructure without requiring external agents.

All monitoring endpoints are exposed via HTTP on a configurable port (default: 8222) and return JSON-formatted data.

HTTP Monitoring Port

Configuration

Enable monitoring by configuring the HTTP port:

monitoring.conf

http_port: 8222

# Optional: Bind to specific host
http: "0.0.0.0:8222"

# Optional: HTTPS monitoring
https_port: 8223

Command line flags:

CLI Flags

# Enable HTTP monitoring
nats-server -m 8222

# Enable HTTPS monitoring
nats-server -ms 8223

From server.go:3037-3046, the following paths are available:

/varz - Server information and statistics
/connz - Connection details
/routez - Route connection information
/subsz - Subscription details
/jsz - JetStream statistics

HTTPS Configuration

Secure monitoring endpoints with TLS:

https-monitoring.conf

https_port: 8223

tls {
    cert_file: "/path/to/monitoring-cert.pem"
    key_file: "/path/to/monitoring-key.pem"
}

Monitoring endpoints expose sensitive information. Always secure them with TLS and authentication in production.

Monitoring Endpoints

/varz - Server Information

Provides comprehensive server runtime information and statistics.

Basic Usage

curl http://localhost:8222/varz

curl http://localhost:8222/varz | jq

{
  "server_id": "NDJWE4VZOJFLN6BVWSQBQBQM3OSFHZL2DZPPFVCBQDVS6G3HGPVV7F6W",
  "server_name": "nats-1",
  "version": "2.10.0",
  "proto": 1,
  "go": "go1.21.0",
  "host": "0.0.0.0",
  "port": 4222,
  "max_connections": 65536,
  "ping_interval": 120000000000,
  "ping_max": 2,
  "http_port": 8222,
  "auth_required": false,
  "tls_required": false,
  "start": "2024-01-15T10:30:00Z",
  "now": "2024-01-15T12:45:30Z",
  "uptime": "2h15m30s",
  "mem": 12582912,
  "cores": 8,
  "cpu": 2.5,
  "connections": 142,
  "total_connections": 1523,
  "routes": 2,
  "remotes": 2,
  "leafnodes": 3,
  "in_msgs": 1523945,
  "out_msgs": 3047890,
  "in_bytes": 152394500,
  "out_bytes": 304789000,
  "slow_consumers": 0,
  "subscriptions": 2145,
  "http_req_stats": {
    "/varz": 142,
    "/connz": 28,
    "/routez": 15,
    "/subsz": 8,
    "/jsz": 35
  },
  "jetstream": {
    "config": {
      "max_memory": 1073741824,
      "max_storage": 10737418240,
      "store_dir": "/var/lib/nats/jetstream"
    },
    "stats": {
      "memory": 52428800,
      "storage": 1048576000,
      "accounts": 3,
      "api": {
        "total": 5234,
        "errors": 12
      }
    }
  }
}

Key Fields (from monitor.go:1211-1283):

string

Unique server ID generated at start

string

NATS Server version

int

Current number of active connections

uint64

Total connections since server start

int64

Total inbound messages

int64

Total outbound messages

int64

Total inbound bytes

int64

Total outbound bytes

int64

Number of slow consumers detected

uint32

Current active subscriptions

float64

Current CPU usage percentage

int64

Current memory usage in bytes

object

JetStream configuration and statistics (if enabled)

/connz - Connection Details

Provides detailed information about client connections.

Basic Usage

curl http://localhost:8222/connz

Query Parameters:

string

default:"cid"

Sort connections by: cid, start, subs, pending, msgs_to, msgs_from, bytes_to, bytes_from, last, idle, uptime, stop, reason, rtt

bool

default:"false"

Include username in connection details

bool

default:"false"

Include subscription subjects

bool

default:"false"

Include detailed subscription information

int

default:"0"

Pagination offset

int

default:"1024"

Maximum connections to return

uint64

Filter by specific connection ID

string

Filter by state: open, closed, all

string

Filter by MQTT client ID

curl "http://localhost:8222/connz?state=open&limit=10"

curl "http://localhost:8222/connz?subs=true&limit=5"

curl "http://localhost:8222/connz?sort=pending"

curl "http://localhost:8222/connz?cid=42"

Response Fields (from monitor.go:47-154):

Connection Info

{
  "server_id": "NDJWE4...",
  "now": "2024-01-15T12:45:30Z",
  "num_connections": 142,
  "total": 142,
  "offset": 0,
  "limit": 1024,
  "connections": [
    {
      "cid": 42,
      "kind": "Client",
      "type": "nats",
      "ip": "192.168.1.100",
      "port": 54321,
      "start": "2024-01-15T10:30:00Z",
      "last_activity": "2024-01-15T12:45:29Z",
      "rtt": "2ms",
      "uptime": "2h15m30s",
      "idle": "1s",
      "pending_bytes": 0,
      "in_msgs": 15234,
      "out_msgs": 30468,
      "in_bytes": 1523400,
      "out_bytes": 3046800,
      "subscriptions": 12,
      "name": "my-service",
      "lang": "go",
      "version": "1.31.0",
      "tls_version": "1.3",
      "tls_cipher_suite": "TLS_AES_128_GCM_SHA256",
      "authorized_user": "app-user",
      "account": "ORDERS"
    }
  ]
}

/routez - Route Information

Details about route (server-to-server) connections in a cluster.

Basic Usage

curl http://localhost:8222/routez

Query Parameters:

bool

default:"false"

Include subscription details for routes

bool

default:"false"

Include detailed subscription information

curl http://localhost:8222/routez | jq

{
  "server_id": "NDJWE4...",
  "server_name": "nats-1",
  "now": "2024-01-15T12:45:30Z",
  "num_routes": 2,
  "routes": [
    {
      "rid": 1,
      "remote_id": "NCBXXX...",
      "remote_name": "nats-2",
      "did_solicit": true,
      "is_configured": true,
      "ip": "192.168.1.101",
      "port": 6222,
      "start": "2024-01-15T10:00:00Z",
      "last_activity": "2024-01-15T12:45:29Z",
      "rtt": "500µs",
      "uptime": "2h45m30s",
      "idle": "1s",
      "pending_size": 0,
      "in_msgs": 152394,
      "out_msgs": 152395,
      "in_bytes": 15239400,
      "out_bytes": 15239500,
      "subscriptions": 2145
    }
  ]
}

/subsz - Subscription Details

Information about active subscriptions across all connections.

Basic Usage

curl http://localhost:8222/subsz

Query Parameters:

bool

default:"false"

Include detailed subscription list

int

default:"0"

Pagination offset

int

default:"1024"

Maximum subscriptions to return

string

Test which subscriptions match a publish subject

curl http://localhost:8222/subsz

curl "http://localhost:8222/subsz?test=orders.new&subs=true"

{
  "server_id": "NDJWE4...",
  "now": "2024-01-15T12:45:30Z",
  "num_subscriptions": 2145,
  "num_cache": 0,
  "num_inserts": 12534,
  "num_removes": 10389,
  "num_matches": 1523945,
  "cache_hit_rate": 0.95,
  "max_fanout": 45,
  "avg_fanout": 1.5,
  "total": 2145,
  "offset": 0,
  "limit": 1024,
  "subscriptions_list": [
    {
      "subject": "orders.>",
      "qgroup": "workers",
      "sid": "1",
      "msgs": 15234,
      "cid": 42,
      "account": "ORDERS"
    }
  ]
}

/jsz - JetStream Statistics

Detailed JetStream metrics including streams, consumers, and storage usage.

Basic Usage

curl http://localhost:8222/jsz

Query Parameters:

bool

default:"false"

Include per-account JetStream details

bool

default:"false"

Include stream details

bool

default:"false"

Include consumer details

bool

default:"false"

Include configuration details

bool

default:"false"

Only return data if server is meta-leader

int

default:"0"

Pagination offset for accounts

int

default:"1024"

Maximum accounts to return

curl http://localhost:8222/jsz

curl "http://localhost:8222/jsz?accounts=true&streams=true&consumers=true"

{
  "server_id": "NDJWE4...",
  "now": "2024-01-15T12:45:30Z",
  "config": {
    "max_memory": 1073741824,
    "max_storage": 10737418240,
    "store_dir": "/var/lib/nats/jetstream",
    "domain": "production"
  },
  "memory": 52428800,
  "storage": 1048576000,
  "reserved_memory": 268435456,
  "reserved_storage": 2147483648,
  "accounts": 3,
  "ha_assets": 15,
  "api": {
    "total": 5234,
    "errors": 12
  },
  "streams": 8,
  "consumers": 24,
  "messages": 1523945,
  "bytes": 1048576000,
  "meta_cluster": {
    "name": "prod-cluster",
    "leader": "nats-1",
    "replicas": [
      {
        "name": "nats-1",
        "current": true,
        "active": true
      },
      {
        "name": "nats-2",
        "current": false,
        "active": true
      },
      {
        "name": "nats-3",
        "current": false,
        "active": true
      }
    ]
  },
  "account_details": [
    {
      "name": "ORDERS",
      "id": "ACBJXX...",
      "memory": 26214400,
      "storage": 524288000,
      "streams": 4,
      "consumers": 12,
      "stream_detail": [
        {
          "name": "ORDERS",
          "created": "2024-01-15T10:00:00Z",
          "state": {
            "messages": 152394,
            "bytes": 152394000,
            "first_seq": 1,
            "last_seq": 152394,
            "consumer_count": 3
          },
          "cluster": {
            "name": "prod-cluster",
            "leader": "nats-1",
            "replicas": [
              {"name": "nats-1", "current": true},
              {"name": "nats-2", "current": false},
              {"name": "nats-3", "current": false}
            ]
          }
        }
      ]
    }
  ]
}

Integration with Monitoring Tools

Prometheus

Use the NATS Prometheus Exporter:

prometheus.yml

scrape_configs:
  - job_name: 'nats'
    static_configs:
      - targets: ['localhost:7777']  # Prometheus exporter port

Run Exporter

nats-exporter -varz http://localhost:8222/varz \
              -connz http://localhost:8222/connz \
              -routez http://localhost:8222/routez \
              -subz http://localhost:8222/subsz \
              -jsz=http://localhost:8222/jsz

Grafana

Import official NATS Grafana dashboards:

NATS Server Dashboard: General server metrics
JetStream Dashboard: Stream and consumer metrics
Connection Dashboard: Client connection details

DataDog

Use the DataDog NATS integration:

datadog.yaml

instances:
  - host: localhost
    port: 8222
    tags:
      - env:production
      - cluster:us-east

Custom Monitoring

Build custom monitoring with polling:

Custom Monitor

import requests
import time

def monitor_nats():
    while True:
        varz = requests.get('http://localhost:8222/varz').json()
        
        print(f"Connections: {varz['connections']}")
        print(f"Messages/sec: {varz['in_msgs']}")
        print(f"CPU: {varz['cpu']}%")
        print(f"Memory: {varz['mem'] / 1024 / 1024:.2f} MB")
        
        if varz.get('jetstream'):
            js = varz['jetstream']['stats']
            print(f"JetStream Memory: {js['memory'] / 1024 / 1024:.2f} MB")
            print(f"JetStream Storage: {js['storage'] / 1024 / 1024 / 1024:.2f} GB")
        
        time.sleep(10)

monitor_nats()

Metrics and Observability

Key Metrics to Monitor

Connection Count

Monitor connections and total_connections for client activity trends.

Message Throughput

Track in_msgs, out_msgs, in_bytes, out_bytes for throughput.

Slow Consumers

Alert on slow_consumers increasing to detect backpressure.

Memory Usage

Monitor mem and JetStream memory to prevent OOM.

CPU Usage

Watch cpu percentage for performance issues.

Subscription Count

Track subscriptions for interest patterns.

JetStream Storage

Monitor JetStream storage vs max_storage limits.

API Errors

Alert on JetStream API errors increasing.

Health Checks

Use /healthz for health monitoring:

Health Check

curl http://localhost:8222/healthz

# Returns:
# - HTTP 200 if healthy
# - HTTP 503 if unhealthy

Configure health check options:

JetStream Health

# Check if JetStream enabled
curl "http://localhost:8222/healthz?js-enabled-only=true"

# Check specific stream
curl "http://localhost:8222/healthz?stream=ORDERS&account=ORDERS"

Real-Time Monitoring Example

Live Dashboard

watch -n 1 'curl -s http://localhost:8222/varz | jq "{connections, in_msgs, out_msgs, mem, cpu}"'

# Output:
# {
#   "connections": 142,
#   "in_msgs": 1523945,
#   "out_msgs": 3047890,
#   "mem": 12582912,
#   "cpu": 2.5
# }

Security Considerations

Monitoring endpoints expose sensitive operational data. Always secure them in production.

Authentication

Restrict monitoring endpoint access:

Secured Monitoring

http_port: 8222

authorization {
    users = [
        {
            user: "monitor"
            password: "$MONITOR_PASSWORD"
            permissions = {
                # Allow monitoring only
            }
        }
    ]
}

Network Restrictions

Bind to localhost or internal network:

Restricted Access

# Only allow localhost
http: "127.0.0.1:8222"

# Or specific internal IP
http: "10.0.0.1:8222"

Use firewall rules to restrict access to monitoring ports.

Next Steps

JetStream

Monitor JetStream streams and consumers

Configuration

Advanced monitoring configuration

Operations

Operational best practices

Prometheus Exporter

Official Prometheus integration

​Overview

​HTTP Monitoring Port

​Configuration

​HTTPS Configuration

​Monitoring Endpoints

​/varz - Server Information

​/connz - Connection Details

​/routez - Route Information

​/subsz - Subscription Details

​/jsz - JetStream Statistics

​Integration with Monitoring Tools

​Prometheus

​Grafana

​DataDog

​Custom Monitoring

​Metrics and Observability

​Key Metrics to Monitor

Connection Count

Message Throughput

Slow Consumers

Memory Usage

CPU Usage

Subscription Count

JetStream Storage

API Errors

​Health Checks

​Real-Time Monitoring Example

​Security Considerations

​Authentication

​Network Restrictions

​Next Steps

JetStream

Configuration

Operations

Prometheus Exporter

Overview

HTTP Monitoring Port

Configuration

HTTPS Configuration

Monitoring Endpoints

/varz - Server Information

/connz - Connection Details

/routez - Route Information

/subsz - Subscription Details

/jsz - JetStream Statistics

Integration with Monitoring Tools

Prometheus

Grafana

DataDog

Custom Monitoring

Metrics and Observability

Key Metrics to Monitor

Health Checks

Real-Time Monitoring Example

Security Considerations

Authentication

Network Restrictions

Next Steps