Deploying Thanos on Kubernetes: Architecture, Deployment Options, and Practical Guide

The author, a member of the Tencent Cloud Container Service (TKE) team, continues the series on building a cloud‑native large‑scale distributed monitoring system by focusing on the practical deployment of Thanos.

Deployment methods : Thanos can be deployed on Kubernetes using three approaches – the prometheus-operator CRD, community Helm charts, or the official kube-thanos JSONnet/YAML templates. The guide recommends using raw YAML for greater transparency and customisation.

Choosing Sidecar or Receiver : The Sidecar model is mature and keeps data local to each Prometheus instance, while the experimental Receiver model centralises data via remote‑write. Sidecar is preferred unless queries span many data‑center‑distributed sidecars or you need a push‑only ingestion path.

Object storage configuration : Long‑term storage requires configuring an object‑storage secret (e.g., Tencent COS or AliYun OSS). Example secret YAML is provided and must be mounted into all Thanos components via --objstore.config or --objstore.config-file .

Prometheus with Sidecar :

apiVersion: v1
kind: Secret
metadata:
  name: thanos-objectstorage
  namespace: thanos
type: Opaque
stringData:
  objectstorage.yaml: |
    type: COS
    config:
      bucket: "thanos"
      region: "ap-singapore"
      app_id: "12*******5"
      secret_key: "tsY***************************Edm"
      secret_id: "AKI******************************gEY"

kind: Service
apiVersion: v1
metadata:
  name: prometheus-headless
  namespace: thanos
  labels:
    app.kubernetes.io/name: prometheus
spec:
  type: ClusterIP
  clusterIP: None
  selector:
    app.kubernetes.io/name: prometheus
  ports:
  - name: web
    protocol: TCP
    port: 9090
    targetPort: web
  - name: grpc
    port: 10901
    targetPort: grpc
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: prometheus
  namespace: thanos
  labels:
    app.kubernetes.io/name: prometheus
spec:
  serviceName: prometheus-headless
  podManagementPolicy: Parallel
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: prometheus
  template:
    metadata:
      labels:
        app.kubernetes.io/name: prometheus
    spec:
      serviceAccountName: prometheus
      containers:
      - name: prometheus
        image: quay.io/prometheus/prometheus:v2.15.2
        args:
        - --config.file=/etc/prometheus/config_out/prometheus.yaml
        - --storage.tsdb.path=/prometheus
        - --storage.tsdb.retention.time=10d
        ports:
        - containerPort: 9090
          name: web
      - name: thanos
        image: quay.io/thanos/thanos:v0.11.0
        args:
        - sidecar
        - --log.level=debug
        - --tsdb.path=/prometheus
        - --prometheus.url=http://127.0.0.1:9090
        - --objstore.config-file=/etc/thanos/objectstorage.yaml
        ports:
        - name: http-sidecar
          containerPort: 10902
        - name: grpc
          containerPort: 10901

Thanos Query (stateless Deployment with three replicas, headless service not required):

apiVersion: v1
kind: Service
metadata:
  name: thanos-query
  namespace: thanos
  labels:
    app.kubernetes.io/name: thanos-query
spec:
  ports:
  - name: grpc
    port: 10901
    targetPort: grpc
  - name: http
    port: 9090
    targetPort: http
  selector:
    app.kubernetes.io/name: thanos-query
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: thanos-query
  namespace: thanos
  labels:
    app.kubernetes.io/name: thanos-query
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: thanos-query
  template:
    metadata:
      labels:
        app.kubernetes.io/name: thanos-query
    spec:
      containers:
      - name: thanos-query
        image: thanosio/thanos:v0.11.0
        args:
        - query
        - --log.level=debug
        - --query.auto-downsampling
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:9090
        - --store=dnssrv+_grpc._tcp.prometheus-headless.thanos.svc.cluster.local
        - --store=dnssrv+_grpc._tcp.thanos-rule.thanos.svc.cluster.local
        - --store=dnssrv+_grpc._tcp.thanos-store.thanos.svc.cluster.local

Store Gateway (StatefulSet with two replicas, headless service, uses object storage for indexing):

apiVersion: v1
kind: Service
metadata:
  name: thanos-store
  namespace: thanos
  labels:
    app.kubernetes.io/name: thanos-store
spec:
  clusterIP: None
  ports:
  - name: grpc
    port: 10901
    targetPort: 10901
  - name: http
    port: 10902
    targetPort: 10902
  selector:
    app.kubernetes.io/name: thanos-store
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: thanos-store
  namespace: thanos
  labels:
    app.kubernetes.io/name: thanos-store
spec:
  replicas: 2
  serviceName: thanos-store
  podManagementPolicy: Parallel
  template:
    metadata:
      labels:
        app.kubernetes.io/name: thanos-store
    spec:
      containers:
      - name: thanos-store
        image: thanosio/thanos:v0.11.0
        args:
        - store
        - --log.level=debug
        - --data-dir=/var/thanos/store
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:10902
        - --objstore.config-file=/etc/thanos/objectstorage.yaml

Ruler (StatefulSet, high‑availability via --label=rule_replica , uploads results to object storage):

apiVersion: v1
kind: ConfigMap
metadata:
  name: thanos-rules
  namespace: thanos
data:
  record.rules.yaml: |- ...
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: thanos-ruler
  namespace: thanos
spec:
  replicas: 2
  serviceName: thanos-ruler
  template:
    spec:
      containers:
      - name: thanos-ruler
        image: thanosio/thanos:v0.11.0
        args:
        - rule
        - --log.level=debug
        - --objstore.config-file=/etc/thanos/objectstorage.yaml
        - --rule-file=/etc/thanos/rules/record.rules.yaml
        - --query=dnssrv+_grpc._tcp.thanos-query.thanos.svc.cluster.local:10901

Compact (single‑replica StatefulSet, continuously compresses and down‑samples data from object storage):

apiVersion: v1
kind: Service
metadata:
  name: thanos-compact
  namespace: thanos
spec:
  ports:
  - name: http
    port: 10902
    targetPort: http
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: thanos-compact
  namespace: thanos
spec:
  replicas: 1
  serviceName: thanos-compact
  template:
    spec:
      containers:
      - name: thanos-compact
        image: thanosio/thanos:v0.11.0
        args:
        - compact
        - --wait
        - --objstore.config-file=/etc/thanos/objectstorage.yaml
        - --data-dir=/var/thanos/compact
        - --retention.resolution-raw=90d
        - --retention.resolution-5m=180d
        - --retention.resolution-1h=360d

Receiver (experimental) : Deploy three replicas with a hashring configuration, expose remote_write endpoint, and let Prometheus push data directly to it, removing the need for a Sidecar.

apiVersion: v1
kind: ConfigMap
metadata:
  name: thanos-receive-hashrings
  namespace: thanos
data:
  thanos-receive-hashrings.json: |
    [{"hashring":"soft-tenants","endpoints":["thanos-receive-0.thanos-receive.kube-system.svc.cluster.local:10901","thanos-receive-1.thanos-receive.kube-system.svc.cluster.local:10901","thanos-receive-2.thanos-receive.kube-system.svc.cluster.local:10901"]}]
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: thanos-receive
  namespace: thanos
spec:
  replicas: 3
  serviceName: thanos-receive
  template:
    spec:
      containers:
      - name: thanos-receive
        image: thanosio/thanos:v0.11.0
        args:
        - receive
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:10902
        - --remote-write.address=0.0.0.0:19291
        - --objstore.config-file=/etc/thanos/objectstorage.yaml
        - --tsdb.path=/var/thanos/receive
        - --tsdb.retention=12h
        - --label=receive_replica="$(NAME)"
        - --receive.hashrings-file=/etc/thanos/thanos-receive-hashrings.json

Finally, the article advises configuring Grafana (or other front‑ends) to use the Thanos Query service as the Prometheus data source, completing the end‑to‑end monitoring stack.

Conclusion : By following the detailed steps and YAML examples, readers should be able to build and operate a robust, cloud‑native, large‑scale monitoring system based on Thanos and Prometheus.