Collect Kubernetes Logs with OpenTelemetry and Loki Using Helm

Previously we introduced how to collect Kubernetes metrics using the OpenTelemetry Collector; now we explore gathering log data.

Install Loki

Deploy Loki with a Helm chart, without any log collector, as the OpenTelemetry Collector will forward logs to Loki.

<code>$ helm repo add grafana https://grafana.github.io/helm-chart
$ helm repo update</code>

Create a

loki-values.yaml

file to configure the Loki Helm chart:

<code># loki-values.yaml
loki:
  commonConfig:
    replication_factor: 1
  auth_enabled: false
  storage:
    type: "filesystem"
singleBinary:
  replicas: 1
  persistence:
    enabled: true
    size: 10Gi
    storageClass: cfsauto
monitoring:
  lokiCanary:
    enabled: false
  selfMonitoring:
    grafanaAgent:
      installOperator: false
test:
  enabled: false
gateway:
  ingress:
    enabled: true
    ingressClassName: nginx
    tls: []
    hosts:
    - host: loki.k8s.local
      paths:
      - path: /
        pathType: Prefix</code>

Deploy Loki with a single command:

<code>$ helm upgrade --install loki grafana/loki -f loki-values.yaml --namespace kube-otel
$ kubectl get pods -n kube-otel -l app.kubernetes.io/instance=loki
$ kubectl get ingress -n kube-otel</code>

Enable filelog Receiver

Update

otel-collector-ds-values.yaml

to add a Loki exporter and enable the

filelogreceiver

:

<code># otel-collector-ds-values.yaml
mode: daemonset
presets:
  hostMetrics:
    enabled: true
  kubernetesAttributes:
    enabled: true
  kubeletMetrics:
    enabled: true
  # Enable filelogreceiver
  logsCollection:
    enabled: true
config:
  exporters:
    loki:
      endpoint: http://loki-gateway/loki/api/v1/push
      timeout: 10s
      read_buffer_size: 200
      write_buffer_size: 100
      retry_on_failure:
        enabled: true
        initial_interval: 10s
        max_interval: 60s
        max_elapsed_time: 10m
      default_labels_enabled: true
  processors:
    resource:
      attributes:
      - action: insert
        key: loki.resource.labels
        value: k8s.namespace.name,k8s.pod.name,k8s.container.name
  service:
    pipelines:
      logs:
        exporters: [loki]
        processors: [memory_limiter, k8sattributes, resource, batch]
        receivers: [otlp, filelog]
</code>

Apply the updated DaemonSet:

<code>$ helm upgrade --install opentelemetry-collector ./opentelemetry-collector -f otel-ds-values.yaml --namespace kube-otel --create-namespace</code>

Verify the configuration:

<code>kubectl get cm -n opentelemetry-collector-agent -oyaml</code>

Loki Exporter

The exporter sends data to Loki via HTTP. Key settings include:

endpoint

: Loki HTTP endpoint, e.g.,

http://loki:3100/loki/api/v1/push

default_labels_enabled

: Controls which default labels (job, instance, exporter, level) are added; disabling all without adding custom labels will drop logs.

If all default labels are disabled and no other labels are added, logs are discarded because Loki requires at least one label.

Use the

attributes

processor to map OTLP attributes to Loki labels, for example mapping

event.domain

to a label and

service.name

to another.

<code>processors:
  attributes:
    actions:
    - action: insert
      key: loki.attribute.labels
      value: event.domain
  resource:
    attributes:
    - action: insert
      key: loki.resource.labels
      value: service.name
</code>

Buffer sizes (

read_buffer_size

and

write_buffer_size

) control how much data is cached before sending, affecting throughput and reliability.

filelog Receiver

The receiver reads and parses log files, then forwards them to the collector.

<code>filelog:
  exclude:
  - /var/log/pods/kube-otel_opentelemetry-collector*_*/opentelemetry-collector/*.log
  include:
  - /var/log/pods/*/*/*.log
  include_file_name: false
  include_file_path: true
  operators:
  - id: get-format
    routes:
    - expr: body matches "^\\{"
      output: parser-docker
    - expr: body matches "^[^ Z]+ "
      output: parser-crio
    - expr: body matches "^[^ Z]+Z"
      output: parser-containerd
    type: router
  - id: parser-crio
    regex: ^(?P<time>[^ Z]+) (?P<stream>stdout|stderr)(?P<logtag>[^]*)?(?P<log>.*)$
    timestamp:
      layout: 2006-01-02T15:04:05.999999999Z07:00
      layout_type: gotime
      parse_from: attributes.time
    type: regex_parser
  - combine_field: attributes.log
    combine_with: ""
    id: crio-recombine
    is_last_entry: attributes.logtag == 'F'
    max_log_size: 102400
    output: extract_metadata_from_filepath
    source_identifier: attributes["log.file.path"]
    type: recombine
  - id: parser-containerd
    regex: ^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr)(?P<logtag>[^]*)?(?P<log>.*)$
    timestamp:
      layout: "%Y-%m-%dT%H:%M:%S.%LZ"
      parse_from: attributes.time
    type: regex_parser
  - combine_field: attributes.log
    combine_with: ""
    id: containerd-recombine
    is_last_entry: attributes.logtag == 'F'
    max_log_size: 102400
    output: extract_metadata_from_filepath
    source_identifier: attributes["log.file.path"]
    type: recombine
  - id: parser-docker
    output: extract_metadata_from_filepath
    timestamp:
      layout: "%Y-%m-%dT%H:%M:%S.%LZ"
      parse_from: attributes.time
    type: json_parser
  - id: extract_metadata_from_filepath
    parse_from: attributes["log.file.path"]
    regex: ^.*/(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_(?P<uid>[a-f0-9\-]+)/(?P<container_name>[^\._]+)/(?P<restart_count>\d+)\.log$
    type: regex_parser
  - from: attributes.stream
    to: attributes["log.iostream"]
    type: move
  - from: attributes.container_name
    to: resource["k8s.container.name"]
    type: move
  - from: attributes.namespace
    to: resource["k8s.namespace.name"]
    type: move
  - from: attributes.pod_name
    to: resource["k8s.pod.name"]
    type: move
  - from: attributes.restart_count
    to: resource["k8s.container.restart_count"]
    type: move
  - from: attributes.uid
    to: resource["k8s.pod.uid"]
    type: move
  - from: attributes.log
    to: body
    type: move
  start_at: beginning
</code>

The configuration excludes unwanted logs, includes desired files, adds file path metadata, and uses a series of operators (router, regex parsers, recombine, move) to extract timestamps, streams, tags, and Kubernetes metadata before sending to Loki.

Enable k8sobject Receiver

For a collector in deployment mode, enable the

k8sobject

receiver to collect Kubernetes Events.

<code># otel-collector-deploy-values.yaml
mode: deployment
replicaCount: 1
presets:
  clusterMetrics:
    enabled: true
  kubernetesEvents:
    enabled: true
config:
  exporters:
    loki:
      endpoint: http://loki-gateway/loki/api/v1/push
      timeout: 10s
      read_buffer_size: 200
      write_buffer_size: 100
      retry_on_failure:
        enabled: true
        initial_interval: 10s
        max_interval: 60s
        max_elapsed_time: 10m
  service:
    pipelines:
      logs:
        exporters: [loki]
</code>

Deploy the updated collector:

<code>$ helm upgrade --install opentelemetry-collector-cluster ./opentelemetry-collector -f otel-collector-deploy-values.yaml --namespace kube-otel --create-namespace</code>

Configure the

k8sobjects

receiver to watch events:

<code>k8sobjects:
  objects:
  - group: events.k8s.io
    mode: watch
    name: events
</code>

After deployment, Kubernetes event logs become visible in Loki via Grafana.

Loki data source added to Grafana.

Explore logs in the Grafana Loki Explorer.