Integrating OpenTelemetry Metrics into Apache Airflow with Prometheus and Grafana

Configuring Your Airflow Environment

To enable OpenTelemetry in an existing Airflow deployment, install the otel extra package and set several environment variables as described in the Airflow documentation. An OTel collector and an observability platform (Prometheus in this guide) are required to store and visualize the metrics.

OTel Collector

The OpenTelemetry Collector provides a vendor‑agnostic way to receive, process, and export telemetry data. It aggregates all Airflow metrics and forwards them to Prometheus. For configuration details, refer to the OpenTelemetry Collector quick‑start guide and the Docker‑Compose files bundled with Airflow’s Breeze development environment.

If Airflow and the collector run in local Docker containers, you can view raw Prometheus‑format metrics at localhost:28889/metrics . Each metric appears as three lines: HELP (description), TYPE (counter, gauge, or timer), and the metric name with labels and current value.

Prometheus

Prometheus serves as the monitoring and storage solution. After starting the recommended Docker‑Compose setup, you can see active targets at localhost:29090/targets . The Targets page shows the connection to the otel-collector .

Grafana

Grafana provides a flexible dashboard for exploring and sharing data. Using the free open‑source version in another Docker container, you can access the login page at localhost:23000 . Configuration files in the Breeze repository help set up data sources and a default dashboard.

Creating Your First Grafana Dashboard

Deploy a sample DAG that runs every minute and sleeps for a random duration between 1 and 10 seconds. Place the following Python code in the DAG folder and let it run for several cycles to generate metric data:

import time

from airflow import DAG
from airflow.decorators import task
from airflow.utils.timezone import datetime
from datetime import timedelta
from random import randint

@task
def task1():
    time.sleep(randint(1, 10))

with DAG(
    dag_id='sleep_random',
    start_date=datetime(2021, 1, 1),
    schedule_interval=timedelta(minutes=1),
    catchup=False
) as dag:
    task1()

After the DAG has produced data, create a new Grafana panel, set the data source to Prometheus , and open the Metrics Browser to explore available metrics such as airflow_task_instance_count , airflow_task_duration_success , and airflow_dagrun_duration_success_sleep_random . Filter metrics by the airflow_ prefix to narrow the list.

Configure the panel to display the task duration as a bar chart with seconds on the Y‑axis, set the range from 0 to 12 seconds, and adjust the resolution if needed. The resulting dashboard shows the random sleep times and the additional system overhead incurred during task execution.

What’s Next?

Further Learning

For deeper Grafana knowledge, start with the Grafana Fundamentals tutorial. To discuss Airflow topics, join the Airflow community Slack.

Future Airflow and OpenTelemetry Features

The next major addition will be OpenTelemetry tracing, which will reveal the full execution path of pipeline tasks and enable Gantt‑style visualizations to identify bottlenecks.

Appendix 1 – Brief Overview of Metrics

Counters

Counters are monotonically increasing integers, such as the total number of tasks executed since Airflow started.

Gauges

Gauges represent instantaneous values that can go up or down, like the current number of DAGs in the system.

Timers

Timers report the duration of an event, for example how long a task runs.

Deep Dive

See the Airflow documentation for a full list of metric types and the OpenTelemetry documentation for details on supported metric signals.