Master Loki: Deploy, Configure, and Query Logs Efficiently

Introduction

When designing a container‑cloud logging solution, traditional ELK/EFK stacks can be heavyweight and often provide more search capabilities than needed. Loki, an open‑source project from Grafana Labs, offers a lightweight, highly available, multi‑tenant log aggregation system optimized for Prometheus and Kubernetes users.

Project URL: https://github.com/grafana/loki/

Key Features of Loki

Loki does not perform full‑text indexing of log lines; it stores compressed logs and indexes only metadata, reducing storage and operational costs.

Logs are indexed and grouped using the same label model as Prometheus, enabling seamless integration with Alertmanager.

Optimized for Kubernetes pod logs; pod labels are automatically indexed.

Native Grafana support eliminates the need to switch between Grafana and Kibana.

Deployment

1. Install Promtail and Loki

<code>wget https://github.com/grafana/loki/releases/download/v2.2.1/loki-linux-amd64.zip</code>
<code>wget https://github.com/grafana/loki/releases/download/v2.2.1/promtail-linux-amd64.zip</code>

2. Install Promtail

<code># Create directories
mkdir -pv /opt/app/{promtail,loki}

# Create promtail configuration
cat <<EOF > /opt/app/promtail/promtail.yaml
server:
  http_listen_port: 9080
  grpc_listen_port: 0
positions:
  filename: /var/log/positions.yaml
client:
  url: http://localhost:3100/loki/api/v1/push
scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          host: yourhost
          __path__: /var/log/*.log
EOF

# Unzip and install
unzip promtail-linux-amd64.zip
mv promtail-linux-amd64 /opt/app/promtail/promtail

# Systemd service file
cat <<EOF > /etc/systemd/system/promtail.service
[Unit]
Description=Promtail server
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/opt/app/promtail/promtail -config.file=/opt/app/promtail/promtail.yaml
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=promtail

[Install]
WantedBy=default.target
EOF

systemctl daemon-reload
systemctl restart promtail
systemctl status promtail
</code>

3. Install Loki

<code># Create directories
mkdir -pv /opt/app/{promtail,loki}

# Loki configuration
cat <<EOF > /opt/app/loki/loki.yaml
auth_enabled: false
server:
  http_listen_port: 3100
  grpc_listen_port: 9096
ingester:
  wal:
    enabled: true
    dir: /opt/app/loki/wal
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
  chunk_idle_period: 1h
  max_chunk_age: 1h
  chunk_target_size: 1048576
  chunk_retain_period: 30s
  max_transfer_retries: 0
schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h
storage_config:
  boltdb_shipper:
    active_index_directory: /opt/app/loki/boltdb-shipper-active
    cache_location: /opt/app/loki/boltdb-shipper-cache
    cache_ttl: 24h
    shared_store: filesystem
  filesystem:
    directory: /opt/app/loki/chunks
compactor:
  working_directory: /opt/app/loki/boltdb-shipper-compactor
  shared_store: filesystem
limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h
chunk_store_config:
  max_look_back_period: 0s
table_manager:
  retention_deletes_enabled: false
  retention_period: 0s
ruler:
  storage:
    type: local
    local:
      directory: /opt/app/loki/rules
  rule_path: /opt/app/loki/rules-temp
  alertmanager_url: http://localhost:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true
EOF

# Unzip and install
unzip loki-linux-amd64.zip
mv loki-linux-amd64 /opt/app/loki/loki

# Systemd service file
cat <<EOF > /etc/systemd/system/loki.service
[Unit]
Description=loki server
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/opt/app/loki/loki -config.file=/opt/app/loki/loki.yaml
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=loki

[Install]
WantedBy=default.target
EOF

systemctl daemon-reload
systemctl restart loki
systemctl status loki
</code>

Using Loki in Grafana

In Grafana, add a new data source of type Loki and set the URL to

http://loki:3100

. After saving, go to Explore to view logs, select Log labels to see available label keys, and filter logs using those labels.

Label‑Based Queries

Loki stores logs in streams identified by a set of labels. Queries first resolve the label hash to locate relevant chunks, then filter the log lines inside those chunks. This approach yields low overhead and fast query performance.

<code>{job="message"} |= "kubelet" [1m]</code>

Dynamic and High‑Cardinality Labels

Dynamic labels have values that change frequently; high‑cardinality labels have a very large number of possible values (e.g., IP addresses). Using such labels indiscriminately can create millions of streams, exhausting resources.

Example: extracting

action

and

status_code

from Apache access logs with a regex stage creates separate streams for each combination, which can quickly multiply.

<code>scrape_configs:
  - job_name: system
    pipeline_stages:
      - regex:
          expression: "^(?P<ip>\\S+) (?P<identd>\\S+) (?P<user>\\S+) \\[(?P<timestamp>[\\w:/]+\\s[+\\-]\\d{4})\\] \"(?P<action>\\S+)\\s?(?P<path>\\S+)?\\s?(?P<protocol>\\S+)?\" (?P<status_code>\\d{3}|-) (?P<size>\\d+|-)"
    static_configs:
      - targets: ["localhost"]
        labels:
          job: apache
          __path__: /var/log/apache.log
</code>

Best Practices

Keep the number of labels low for small log volumes; add labels only when they improve query selectivity.

Avoid high‑cardinality labels such as per‑IP unless absolutely necessary.

Configure

chunk_target_size

and

max_chunk_age

to balance chunk size and query latency.

Ensure logs are ingested in chronological order; Loki rejects out‑of‑order data for performance reasons.

Conclusion

Loki provides a cost‑effective, label‑centric logging solution that integrates tightly with Prometheus and Grafana. By understanding its architecture, deployment steps, and labeling strategies, operators can build scalable log pipelines that deliver fast, reliable observability without the overhead of full‑text indexing.