CIS Grafana Loki Benchmark

This recommendation ensures that authentication is enforced via reverse proxy or built-in auth on the Loki log aggregation system. Enforcing this requirement establishes a minimum security standard and prevents insecure configurations.

Without this enforcement, the Loki log aggregation system may accept insecure configurations that weaken overall security posture. Mandating this control ensures consistent protection against common attack vectors.

# Check Loki authentication:
curl -s -o /dev/null -w '%{http_code}' http://loki.example.com:3100/ready

# Verify auth is required:
curl -s -o /dev/null -w '%{http_code}' http://loki.example.com:3100/loki/api/v1/labels

# Configure authentication via reverse proxy (nginx):
cat > /etc/nginx/sites-available/loki << 'EOF'
server {
    listen 443 ssl;
    server_name loki.example.com;
    ssl_certificate /etc/ssl/loki.crt;
    ssl_certificate_key /etc/ssl/loki.key;

    location / {
        auth_basic "Loki";
        auth_basic_user_file /etc/nginx/.htpasswd;
        proxy_pass http://127.0.0.1:3100;
    }
}
EOF

# Or use Loki's built-in multi-tenancy:
# auth_enabled: true in loki.yaml

This recommendation verifies that multi-tenancy is enabled with tenant isolation on the Loki log aggregation system. Implementing this control strengthens the overall security configuration and reduces exposure to potential threats.

Failure to implement this control may leave the Loki log aggregation system vulnerable to attack or non-compliant with organizational security policies. This control helps establish a consistent, hardened configuration baseline.

# Check multi-tenancy:
grep 'auth_enabled' /etc/loki/loki.yaml

# Verify tenant header required:
curl -s -H 'X-Scope-OrgID: test' http://localhost:3100/loki/api/v1/labels

# Enable multi-tenancy:
cat > /etc/loki/loki.yaml << 'EOF'
auth_enabled: true

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  log_level: warn

common:
  path_prefix: /var/loki
  storage:
    filesystem:
      chunks_directory: /var/loki/chunks
      rules_directory: /var/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory
EOF

# Clients must include X-Scope-OrgID header

This recommendation verifies that TLS is configured for HTTP and gRPC with mutual authentication on the Loki log aggregation system. Implementing this control strengthens the overall security configuration and reduces exposure to potential threats.

Failure to implement this control may leave the Loki log aggregation system vulnerable to attack or non-compliant with organizational security policies. This control helps establish a consistent, hardened configuration baseline.

# Check TLS configuration:
grep -A5 'tls' /etc/loki/loki.yaml 2>/dev/null

# Verify HTTPS:
curl -sv https://loki.example.com:3100/ready 2>&1 | grep -E 'SSL|TLS'

# Configure TLS in Loki:
cat >> /etc/loki/loki.yaml << 'EOF'
server:
  http_tls_config:
    cert_file: /etc/loki/certs/server.crt
    key_file: /etc/loki/certs/server.key
    client_ca_file: /etc/loki/certs/ca.crt
    client_auth_type: RequireAndVerifyClientCert
  grpc_tls_config:
    cert_file: /etc/loki/certs/server.crt
    key_file: /etc/loki/certs/server.key
    client_ca_file: /etc/loki/certs/ca.crt
    client_auth_type: RequireAndVerifyClientCert
EOF

This recommendation verifies that storage backend uses server-side encryption on the Loki log aggregation system. Implementing this control strengthens the overall security configuration and reduces exposure to potential threats.

Failure to implement this control may leave the Loki log aggregation system vulnerable to attack or non-compliant with organizational security policies. This control helps establish a consistent, hardened configuration baseline.

# Check storage backend encryption:
grep -A20 'storage_config' /etc/loki/loki.yaml

# Verify S3 encryption:
grep -i 'sse\|encrypt' /etc/loki/loki.yaml

# Configure encrypted S3 storage backend:
cat > /etc/loki/loki.yaml << 'EOF'
auth_enabled: true

storage_config:
  aws:
    s3: s3://us-east-1/loki-logs
    bucketnames: loki-chunks
    endpoint: s3.us-east-1.amazonaws.com
    region: us-east-1
    access_key_id: ${AWS_ACCESS_KEY_ID}
    secret_access_key: ${AWS_SECRET_ACCESS_KEY}
    sse:
      type: SSE-S3
    insecure: false
    http_config:
      idle_conn_timeout: 90s
  boltdb_shipper:
    active_index_directory: /var/loki/boltdb-shipper-active
    cache_location: /var/loki/boltdb-shipper-cache
    shared_store: s3
EOF

This recommendation verifies that retention policies and compaction are configured on the Loki log aggregation system. Implementing this control strengthens the overall security configuration and reduces exposure to potential threats.

Failure to implement this control may leave the Loki log aggregation system vulnerable to attack or non-compliant with organizational security policies. This control helps establish a consistent, hardened configuration baseline.

# Check retention configuration:
grep -A10 'limits_config\|retention' /etc/loki/loki.yaml

# Verify compactor settings:
grep -A10 'compactor' /etc/loki/loki.yaml

# Configure retention:
cat >> /etc/loki/loki.yaml << 'EOF'
limits_config:
  retention_period: 90d
  max_query_length: 721h
  max_query_parallelism: 16
  max_entries_limit_per_query: 10000
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20
  per_stream_rate_limit: 3MB
  per_stream_rate_limit_burst: 15MB

compactor:
  working_directory: /var/loki/compactor
  shared_store: s3
  retention_enabled: true
  retention_delete_delay: 2h
  compaction_interval: 10m
EOF

This recommendation verifies that per-tenant ingestion rate limits and stream limits are set on the Loki log aggregation system. Implementing this control strengthens the overall security configuration and reduces exposure to potential threats.

Failure to implement this control may leave the Loki log aggregation system vulnerable to attack or non-compliant with organizational security policies. This control helps establish a consistent, hardened configuration baseline.

# Check per-tenant limits:
grep -A20 'overrides' /etc/loki/loki.yaml

# Verify rate limiting:
grep -i 'rate_limit\|ingestion_rate' /etc/loki/loki.yaml

# Configure per-tenant rate limits:
cat >> /etc/loki/loki.yaml << 'EOF'
overrides:
  tenant-production:
    ingestion_rate_mb: 20
    ingestion_burst_size_mb: 40
    max_streams_per_user: 10000
    max_entries_limit_per_query: 50000
    retention_period: 180d
  tenant-dev:
    ingestion_rate_mb: 5
    ingestion_burst_size_mb: 10
    max_streams_per_user: 1000
    retention_period: 30d
EOF

This recommendation verifies that Promtail uses TLS with pipeline stages for sensitive data redaction on the Loki log aggregation system. Implementing this control strengthens the overall security configuration and reduces exposure to potential threats.

Failure to implement this control may leave the Loki log aggregation system vulnerable to attack or non-compliant with organizational security policies. This control helps establish a consistent, hardened configuration baseline.

# Check Promtail configuration:
grep -A20 'scrape_configs' /etc/promtail/promtail.yaml | head -30

# Verify pipeline stages:
grep -A10 'pipeline_stages' /etc/promtail/promtail.yaml

# Configure Promtail with pipeline filtering:
cat > /etc/promtail/promtail.yaml << 'EOF'
server:
  http_listen_port: 9080

positions:
  filename: /var/lib/promtail/positions.yaml

clients:
  - url: https://loki.example.com:3100/loki/api/v1/push
    tenant_id: production
    tls_config:
      ca_file: /etc/promtail/certs/ca.crt
    basic_auth:
      username: promtail
      password_file: /etc/promtail/password

scrape_configs:
  - job_name: system
    static_configs:
      - targets: [localhost]
        labels:
          job: syslog
          __path__: /var/log/syslog
    pipeline_stages:
      - regex:
          expression: '(?P<ip>\d+\.\d+\.\d+\.\d+)'
      - replace:
          expression: '(\d+\.\d+\.\d+\.)\d+'
          replace: '${1}xxx'
EOF

This setting ensures that label cardinality is controlled to prevent resource exhaustion on the Loki log aggregation system. Restricting this capability limits potential abuse and enforces the principle of least privilege across the environment.

Unrestricted access to this capability could allow unauthorized users or processes to perform actions beyond their intended scope. Applying least-privilege principles to the Loki log aggregation system is essential for defense in depth.

# Check log label cardinality:
curl -s http://localhost:3100/loki/api/v1/labels | python3 -m json.tool

# Check stream count:
curl -s 'http://localhost:3100/loki/api/v1/series' -G --data-urlencode 'match[]={job=~".+"}' | python3 -c 'import sys,json;d=json.load(sys.stdin);print(f"{len(d.get("data",[]))} streams")'

# Limit label cardinality in Promtail:
# Drop high-cardinality labels:
# pipeline_stages:
#   - labeldrop:
#     - request_id
#     - trace_id
#     - session_id

# Set structured metadata instead of labels:
# pipeline_stages:
#   - structured_metadata:
#       request_id:

# Monitor cardinality:
# Alert on: loki_ingester_streams_created_total rate > threshold

This recommendation verifies that Loki ruler alerting rules are configured for security events on the Loki log aggregation system. Implementing this control strengthens the overall security configuration and reduces exposure to potential threats.

Failure to implement this control may leave the Loki log aggregation system vulnerable to attack or non-compliant with organizational security policies. This control helps establish a consistent, hardened configuration baseline.

# Check Loki ruler configuration:
grep -A15 'ruler' /etc/loki/loki.yaml

# List recording rules:
curl -s -H 'X-Scope-OrgID: production' http://localhost:3100/loki/api/v1/rules | python3 -m json.tool | head -20

# Configure Loki alerting rules:
mkdir -p /etc/loki/rules/production
cat > /etc/loki/rules/production/alerts.yml << 'EOF'
groups:
- name: security-logs
  rules:
  - alert: HighAuthFailureRate
    expr: sum(rate({job="auth"} |= "authentication failed" [5m])) > 10
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "High authentication failure rate"
  - alert: SSHBruteForce
    expr: sum(rate({job="syslog"} |= "Failed password" [5m])) by (host) > 5
    for: 2m
    labels:
      severity: warning
EOF

# Add ruler config to loki.yaml:
# ruler:
#   storage:
#     type: local
#     local:
#       directory: /etc/loki/rules
#   alertmanager_url: http://alertmanager:9093

This recommendation verifies that query audit logging is enabled for compliance on the Loki log aggregation system. Implementing this control strengthens the overall security configuration and reduces exposure to potential threats.

Failure to implement this control may leave the Loki log aggregation system vulnerable to attack or non-compliant with organizational security policies. This control helps establish a consistent, hardened configuration baseline.

# Check query audit logging:
grep -i 'query_logging\|audit' /etc/loki/loki.yaml

# Check query frontend config:
grep -A10 'frontend' /etc/loki/loki.yaml

# Enable query logging:
# In loki.yaml:
# analytics:
#   reporting_enabled: false
# query_scheduler:
#   max_outstanding_requests_per_tenant: 256
# frontend:
#   log_queries_longer_than: 10s
#   max_outstanding_per_tenant: 256
#   compress_responses: true

# Log all queries for audit:
# server:
#   log_level: info
#   log_format: json

This recommendation verifies that Loki is running a current supported version on the Loki log aggregation system. Implementing this control strengthens the overall security configuration and reduces exposure to potential threats.

Failure to implement this control may leave the Loki log aggregation system vulnerable to attack or non-compliant with organizational security policies. This control helps establish a consistent, hardened configuration baseline.

# Check Loki version:
loki -version 2>&1 | head -1

# Or via API:
curl -s http://localhost:3100/loki/api/v1/status/buildinfo | python3 -m json.tool

# Upgrade Loki:
VER=2.9.4
wget https://github.com/grafana/loki/releases/download/v${VER}/loki-linux-amd64.zip
unzip loki-linux-amd64.zip
chmod +x loki-linux-amd64
systemctl stop loki
mv loki-linux-amd64 /usr/local/bin/loki
systemctl start loki

This setting ensures that Loki runs as a dedicated non-root user with restricted paths on the Loki log aggregation system. Restricting this capability limits potential abuse and enforces the principle of least privilege across the environment.

Unrestricted access to this capability could allow unauthorized users or processes to perform actions beyond their intended scope. Applying least-privilege principles to the Loki log aggregation system is essential for defense in depth.

# Check Loki process user:
ps aux | grep loki | grep -v grep | awk '{print $1}'

# Verify file permissions:
ls -la /etc/loki/ /var/loki/

# Run Loki as dedicated user:
useradd --no-create-home --shell /bin/false loki
mkdir -p /var/loki /etc/loki
chown -R loki:loki /var/loki /etc/loki
chmod 640 /etc/loki/loki.yaml

cat > /etc/systemd/system/loki.service << 'EOF'
[Unit]
Description=Grafana Loki
After=network-online.target

[Service]
Type=simple
User=loki
Group=loki
ExecStart=/usr/local/bin/loki -config.file=/etc/loki/loki.yaml
Restart=on-failure
RestartSec=5
NoNewPrivileges=yes
ProtectHome=yes
ProtectSystem=strict
ReadWritePaths=/var/loki

[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload && systemctl enable --now loki

This recommendation verifies that memberlist clustering with replication is configured on the Loki log aggregation system. Implementing this control strengthens the overall security configuration and reduces exposure to potential threats.

Failure to implement this control may leave the Loki log aggregation system vulnerable to attack or non-compliant with organizational security policies. This control helps establish a consistent, hardened configuration baseline.

# Check Loki memberlist (clustering):
grep -A10 'memberlist' /etc/loki/loki.yaml

# Verify cluster health:
curl -s http://localhost:3100/memberlist | head -20

# Configure Loki clustering:
cat >> /etc/loki/loki.yaml << 'EOF'
memberlist:
  join_members:
    - loki-1.example.com:7946
    - loki-2.example.com:7946
    - loki-3.example.com:7946
  dead_node_reclaim_time: 30s
  gossip_to_dead_nodes_time: 15s
  left_ingesters_timeout: 30s

ingester:
  lifecycler:
    ring:
      replication_factor: 3
      kvstore:
        store: memberlist
EOF

This recommendation verifies that configuration and index backups are automated on the Loki log aggregation system. Implementing this control strengthens the overall security configuration and reduces exposure to potential threats.

Failure to implement this control may leave the Loki log aggregation system vulnerable to attack or non-compliant with organizational security policies. This control helps establish a consistent, hardened configuration baseline.

# Check backup configuration:
ls -la /opt/loki-backups/ 2>/dev/null

# Verify BoltDB index backups:
ls -la /var/loki/boltdb-shipper-active/

# Create backup script:
cat > /usr/local/bin/loki-backup.sh << 'SCRIPT'
#!/bin/bash
BACKUP_DIR=/opt/loki-backups
mkdir -p $BACKUP_DIR
DATE=$(date +%Y%m%d)

# Backup configuration:
tar czf $BACKUP_DIR/loki-config-${DATE}.tar.gz /etc/loki/

# Backup BoltDB index:
tar czf $BACKUP_DIR/loki-index-${DATE}.tar.gz /var/loki/boltdb-shipper-active/

# Backup rules:
tar czf $BACKUP_DIR/loki-rules-${DATE}.tar.gz /etc/loki/rules/ 2>/dev/null

find $BACKUP_DIR -mtime +14 -delete
SCRIPT
chmod +x /usr/local/bin/loki-backup.sh
echo '0 3 * * * root /usr/local/bin/loki-backup.sh' > /etc/cron.d/loki-backup