CIS Prometheus Benchmark

This recommendation verifies that basic authentication is enabled for the Prometheus web UI and API on the Prometheus monitoring system. Implementing this control strengthens the overall security configuration and reduces exposure to potential threats.

Failure to implement this control may leave the Prometheus monitoring system vulnerable to attack or non-compliant with organizational security policies. This control helps establish a consistent, hardened configuration baseline.

# Check if authentication is enabled:
curl -s -o /dev/null -w '%{http_code}' http://prometheus.example.com:9090/api/v1/query?query=up

# Check web config for basic auth:
grep -r 'basic_auth' /etc/prometheus/web.yml 2>/dev/null

# Configure basic authentication:
apt install -y apache2-utils
htpasswd -nBC 12 admin > /tmp/hash

cat > /etc/prometheus/web.yml << 'EOF'
basic_auth_users:
  admin: $2y$12$<HTPASSWD_HASH>
EOF

# Start Prometheus with web config:
prometheus --config.file=/etc/prometheus/prometheus.yml \
  --web.config.file=/etc/prometheus/web.yml

This recommendation verifies that TLS is configured with strong cipher suites on the Prometheus monitoring system. Implementing this control strengthens the overall security configuration and reduces exposure to potential threats.

Failure to implement this control may leave the Prometheus monitoring system vulnerable to attack or non-compliant with organizational security policies. This control helps establish a consistent, hardened configuration baseline.

# Check TLS configuration:
grep -A5 'tls' /etc/prometheus/web.yml 2>/dev/null

# Verify HTTPS:
curl -sv https://prometheus.example.com:9090 2>&1 | grep -E 'SSL|TLS|subject'

# Configure TLS:
cat > /etc/prometheus/web.yml << 'EOF'
tls_server_config:
  cert_file: /etc/prometheus/certs/server.crt
  key_file: /etc/prometheus/certs/server.key
  min_version: TLS12
  cipher_suites:
  - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

basic_auth_users:
  admin: $2y$12$<HASH>
EOF

systemctl restart prometheus

This recommendation verifies that scrape targets use HTTPS with mutual TLS authentication on the Prometheus monitoring system. Implementing this control strengthens the overall security configuration and reduces exposure to potential threats.

Failure to implement this control may leave the Prometheus monitoring system vulnerable to attack or non-compliant with organizational security policies. This control helps establish a consistent, hardened configuration baseline.

# Check scrape configurations:
grep -A20 'scrape_configs' /etc/prometheus/prometheus.yml | head -40

# Verify scrape TLS:
grep -B2 -A5 'scheme.*https\|tls_config' /etc/prometheus/prometheus.yml

# Configure secure scraping:
cat >> /etc/prometheus/prometheus.yml << 'EOF'
scrape_configs:
  - job_name: 'node-exporters'
    scheme: https
    tls_config:
      ca_file: /etc/prometheus/certs/ca.crt
      cert_file: /etc/prometheus/certs/client.crt
      key_file: /etc/prometheus/certs/client.key
    basic_auth:
      username: prometheus
      password_file: /etc/prometheus/scrape-password
    static_configs:
      - targets: ['node1:9100', 'node2:9100']
EOF

This recommendation verifies that service discovery uses authenticated and filtered configs on the Prometheus monitoring system. Implementing this control strengthens the overall security configuration and reduces exposure to potential threats.

Failure to implement this control may leave the Prometheus monitoring system vulnerable to attack or non-compliant with organizational security policies. This control helps establish a consistent, hardened configuration baseline.

# Check service discovery configs:
grep -E 'sd_configs|_sd_config' /etc/prometheus/prometheus.yml

# Verify relabel configs:
grep -A10 'relabel_configs' /etc/prometheus/prometheus.yml | head -20

# Use secure service discovery:
cat >> /etc/prometheus/prometheus.yml << 'EOF'
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
        api_server: https://k8s-api.example.com:6443
        tls_config:
          ca_file: /etc/prometheus/certs/k8s-ca.crt
        bearer_token_file: /etc/prometheus/k8s-token
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_namespace]
        action: drop
        regex: kube-system|kube-public
EOF

This recommendation verifies that security-focused alerting rules are defined and validated on the Prometheus monitoring system. Implementing this control strengthens the overall security configuration and reduces exposure to potential threats.

Failure to implement this control may leave the Prometheus monitoring system vulnerable to attack or non-compliant with organizational security policies. This control helps establish a consistent, hardened configuration baseline.

# Check alerting rules:
ls -la /etc/prometheus/rules/
promtool check rules /etc/prometheus/rules/*.yml 2>&1

# Verify Alertmanager integration:
grep -A5 'alerting' /etc/prometheus/prometheus.yml

# Create security alerting rules:
cat > /etc/prometheus/rules/security.yml << 'EOF'
groups:
- name: security
  rules:
  - alert: HighErrorRate
    expr: rate(http_requests_total{code=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "High error rate on {{ $labels.instance }}"
  - alert: NodeExporterDown
    expr: up{job="node-exporters"} == 0
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "Node exporter down on {{ $labels.instance }}"
  - alert: DiskSpaceLow
    expr: (node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 < 10
    for: 5m
    labels:
      severity: critical
EOF

promtool check rules /etc/prometheus/rules/security.yml

This recommendation verifies that Alertmanager uses TLS SMTP with inhibition and routing on the Prometheus monitoring system. Implementing this control strengthens the overall security configuration and reduces exposure to potential threats.

Failure to implement this control may leave the Prometheus monitoring system vulnerable to attack or non-compliant with organizational security policies. This control helps establish a consistent, hardened configuration baseline.

# Check Alertmanager config:
cat /etc/alertmanager/alertmanager.yml | grep -v '^#' | head -30

# Verify notification channels:
amtool config show 2>/dev/null

# Configure Alertmanager with inhibition and routing:
cat > /etc/alertmanager/alertmanager.yml << 'EOF'
global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: 'alerts@example.com'
  smtp_auth_username: 'alerts@example.com'
  smtp_auth_password: '<PASSWORD>'
  smtp_require_tls: true

route:
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'team-slack'
  routes:
  - match:
      severity: critical
    receiver: 'pagerduty'

receivers:
- name: 'team-slack'
  slack_configs:
  - api_url: 'https://hooks.slack.com/services/...' 
    channel: '#alerts'
- name: 'pagerduty'
  pagerduty_configs:
  - service_key: '<KEY>'

inhibit_rules:
- source_match:
    severity: critical
  target_match:
    severity: warning
  equal: ['alertname', 'instance']
EOF

This recommendation verifies that TSDB retention time and size limits are configured on the Prometheus monitoring system. Implementing this control strengthens the overall security configuration and reduces exposure to potential threats.

Failure to implement this control may leave the Prometheus monitoring system vulnerable to attack or non-compliant with organizational security policies. This control helps establish a consistent, hardened configuration baseline.

# Check TSDB retention settings:
grep 'storage' /etc/prometheus/prometheus.yml
ps aux | grep prometheus | grep -o 'storage[^ ]*'

# Check disk usage:
du -sh /var/lib/prometheus/

# Configure retention:
# Start Prometheus with retention flags:
prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/ \
  --storage.tsdb.retention.time=90d \
  --storage.tsdb.retention.size=50GB \
  --storage.tsdb.wal-compression

# In systemd unit:
# ExecStart=/usr/local/bin/prometheus \
#   --storage.tsdb.retention.time=90d \
#   --storage.tsdb.retention.size=50GB

This recommendation verifies that remote write with TLS is configured for long-term storage on the Prometheus monitoring system. Implementing this control strengthens the overall security configuration and reduces exposure to potential threats.

Failure to implement this control may leave the Prometheus monitoring system vulnerable to attack or non-compliant with organizational security policies. This control helps establish a consistent, hardened configuration baseline.

# Check remote write configuration:
grep -A10 'remote_write' /etc/prometheus/prometheus.yml

# Verify Thanos/Cortex integration:
ps aux | grep -E 'thanos|cortex|mimir'

# Configure remote write for long-term storage:
cat >> /etc/prometheus/prometheus.yml << 'EOF'
remote_write:
  - url: "https://thanos-receive.example.com:19291/api/v1/receive"
    tls_config:
      cert_file: /etc/prometheus/certs/client.crt
      key_file: /etc/prometheus/certs/client.key
      ca_file: /etc/prometheus/certs/ca.crt
    queue_config:
      max_samples_per_send: 5000
      batch_send_deadline: 30s
      max_shards: 10
EOF

This recommendation verifies that recording rules are used for expensive query optimization on the Prometheus monitoring system. Implementing this control strengthens the overall security configuration and reduces exposure to potential threats.

Failure to implement this control may leave the Prometheus monitoring system vulnerable to attack or non-compliant with organizational security policies. This control helps establish a consistent, hardened configuration baseline.

# Check recording rules:
find /etc/prometheus/rules/ -name '*.yml' -exec grep -l 'record:' {} \;

# Verify rule efficiency:
promtool check rules /etc/prometheus/rules/*.yml

# Create recording rules for performance:
cat > /etc/prometheus/rules/recording.yml << 'EOF'
groups:
- name: recording_rules
  interval: 30s
  rules:
  - record: job:http_requests_total:rate5m
    expr: rate(http_requests_total[5m])
  - record: job:http_request_duration_seconds:p99
    expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))
  - record: instance:node_cpu_utilisation:rate5m
    expr: 1 - avg without(cpu) (rate(node_cpu_seconds_total{mode="idle"}[5m]))
EOF

This recommendation verifies that federation is authenticated and filtered for multi-cluster on the Prometheus monitoring system. Implementing this control strengthens the overall security configuration and reduces exposure to potential threats.

Failure to implement this control may leave the Prometheus monitoring system vulnerable to attack or non-compliant with organizational security policies. This control helps establish a consistent, hardened configuration baseline.

# Check federation configuration:
grep -A10 'federation' /etc/prometheus/prometheus.yml
grep 'honor_labels' /etc/prometheus/prometheus.yml

# Configure federation for multi-cluster:
cat >> /etc/prometheus/prometheus.yml << 'EOF'
  - job_name: 'federate'
    scrape_interval: 30s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job=~".+"}'
        - '{__name__=~"job:.*"}'
    static_configs:
      - targets:
        - 'prometheus-dc1.example.com:9090'
        - 'prometheus-dc2.example.com:9090'
    tls_config:
      ca_file: /etc/prometheus/certs/ca.crt
    basic_auth:
      username: federation
      password_file: /etc/prometheus/federation-password
EOF

This recommendation verifies that Prometheus is running a current supported version on the Prometheus monitoring system. Implementing this control strengthens the overall security configuration and reduces exposure to potential threats.

Failure to implement this control may leave the Prometheus monitoring system vulnerable to attack or non-compliant with organizational security policies. This control helps establish a consistent, hardened configuration baseline.

# Check Prometheus version:
prometheus --version

# Check for known vulnerabilities:
prometheus --version 2>&1 | head -1

# Upgrade Prometheus:
VER=$(curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest | grep tag_name | cut -d'"' -f4 | tr -d 'v')
wget https://github.com/prometheus/prometheus/releases/download/v${VER}/prometheus-${VER}.linux-amd64.tar.gz
tar xzf prometheus-${VER}.linux-amd64.tar.gz
systemctl stop prometheus
cp prometheus-${VER}.linux-amd64/prometheus /usr/local/bin/
cp prometheus-${VER}.linux-amd64/promtool /usr/local/bin/
systemctl start prometheus

This setting ensures that Prometheus runs as a dedicated non-root user with restricted paths on the Prometheus monitoring system. Restricting this capability limits potential abuse and enforces the principle of least privilege across the environment.

Unrestricted access to this capability could allow unauthorized users or processes to perform actions beyond their intended scope. Applying least-privilege principles to the Prometheus monitoring system is essential for defense in depth.

# Check Prometheus user:
ps aux | grep prometheus | grep -v grep | awk '{print $1}'

# Verify file permissions:
ls -la /etc/prometheus/
ls -la /var/lib/prometheus/

# Run Prometheus as dedicated user:
useradd --no-create-home --shell /bin/false prometheus
chown -R prometheus:prometheus /etc/prometheus/ /var/lib/prometheus/
chmod 640 /etc/prometheus/prometheus.yml
chmod 600 /etc/prometheus/web.yml

# Systemd unit:
cat > /etc/systemd/system/prometheus.service << 'EOF'
[Unit]
Description=Prometheus
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --web.config.file=/etc/prometheus/web.yml \
  --storage.tsdb.path=/var/lib/prometheus/
Restart=always
RestartSec=5
NoNewPrivileges=yes
ProtectHome=yes
ProtectSystem=strict
ReadWritePaths=/var/lib/prometheus/

[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload && systemctl enable --now prometheus

This recommendation verifies that automated TSDB snapshot backups are configured on the Prometheus monitoring system. Implementing this control strengthens the overall security configuration and reduces exposure to potential threats.

Failure to implement this control may leave the Prometheus monitoring system vulnerable to attack or non-compliant with organizational security policies. This control helps establish a consistent, hardened configuration baseline.

# Check TSDB snapshots:
curl -s -XPOST http://localhost:9090/api/v1/admin/tsdb/snapshot 2>/dev/null | python3 -m json.tool

# List snapshots:
ls -la /var/lib/prometheus/snapshots/

# Create automated backup script:
cat > /usr/local/bin/prometheus-backup.sh << 'SCRIPT'
#!/bin/bash
BACKUP_DIR=/opt/prometheus-backups
mkdir -p $BACKUP_DIR

# Create TSDB snapshot:
SNAP=$(curl -s -u admin:$PROM_PASSWORD -XPOST \
  https://localhost:9090/api/v1/admin/tsdb/snapshot | python3 -c 'import sys,json;print(json.load(sys.stdin)["data"]["name"])')

tar czf $BACKUP_DIR/prom-snap-$(date +%Y%m%d).tar.gz \
  /var/lib/prometheus/snapshots/$SNAP/

# Backup config:
tar czf $BACKUP_DIR/prom-config-$(date +%Y%m%d).tar.gz \
  /etc/prometheus/

find $BACKUP_DIR -mtime +14 -delete
SCRIPT
chmod +x /usr/local/bin/prometheus-backup.sh

This recommendation verifies that admin and lifecycle APIs are disabled in production on the Prometheus monitoring system. Implementing this control strengthens the overall security configuration and reduces exposure to potential threats.

Failure to implement this control may leave the Prometheus monitoring system vulnerable to attack or non-compliant with organizational security policies. This control helps establish a consistent, hardened configuration baseline.

# Check Prometheus API lifecycle flags:
ps aux | grep prometheus | grep -o 'lifecycle\|admin'

# Disable admin APIs in production:
prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --web.enable-lifecycle=false \
  --web.enable-admin-api=false

# If lifecycle API needed for config reload, use signal instead:
kill -HUP $(pidof prometheus)

# Or with systemd:
systemctl reload prometheus