How to Achieve Accurate Container Resource Metrics with Lxcfs and Kubernetes Admission Webhook

Why Visual Isolation for Containers?

Container technology offers isolation different from traditional VMs, but many monitoring commands like top and free still show host-level data because /proc and /sys are not virtualized inside containers.

Use Cases for Container Resource View Isolation

In production, teams accustomed to checking resources on physical or virtual machines see host data inside containers, leading to confusion.

JVM‑based Java applications read host memory limits to set heap and stack sizes, causing startup failures when container quotas are lower.

CPU‑intensive services (e.g., Nginx) read host CPU info, which can misconfigure thread counts and degrade performance.

Solution Overview

The combination of Lxcfs and a Kubernetes mutating admission webhook provides per‑container resource view isolation.

Lxcfs Details

Lxcfs is a small FUSE filesystem that virtualizes procfs files to make containers appear more like VMs. It currently virtualizes /proc but not /sys/devices/system/cpu/online in version 3.1.2; the latter is merged into the master branch and requires custom compilation for full isolation.

Deploy Lxcfs on every node in the Kubernetes cluster. Use systemd to ensure the service restarts automatically and to run a remount script after crashes.

<code>[Unit]
Description=FUSE filesystem for LXC
ConditionVirtualization=!container
Before=lxc.service
Documentation=man:lxcfs(1)

[Service]
ExecStart=/usr/bin/lxcfs -l /var/lib/lxc/lxcfs/
KillMode=process
Restart=always
Delegate=yes
ExecStopPost=-/bin/fusermount -u /var/lib/lxc/lxcfs
ExecReload=/bin/kill -USR1 $MAINPID
# add remount script
ExecStartPost=/usr/local/bin/container_remount_lxcfs.sh

[Install]
WantedBy=multi-user.target</code>

Remount script (executed after Lxcfs restarts):

<code>#! /bin/bash
PATH=$PATH:/bin
LXCFS="/var/lib/lxc/lxcfs"
LXCFS_ROOT_PATH="/var/lib/lxc"
containers=$(docker ps | grep -v pause | grep -v calico | awk '{print $1}' | grep -v CONTAINE)
for container in $containers; do
  mountpoint=$(docker inspect --format '{{ range .Mounts }}{{ if eq .Destination "/var/lib/lxc" }}{{ .Source }}{{ end }}{{ end }}' $container)
  if [ "$mountpoint" = "$LXCFS_ROOT_PATH" ]; then
    echo "remount $container"
    PID=$(docker inspect --format '{{.State.Pid}}' $container)
    for file in meminfo cpuinfo loadavg stat diskstats swaps uptime; do
      nsenter --target $PID --mount -- mount -B "$LXCFS/proc/$file" "/proc/$file"
    done
    for file in online; do
      nsenter --target $PID --mount -- mount -B "$LXCFS/sys/devices/system/cpu/$file" "/sys/devices/system/cpu/$file"
    done
  fi
done</code>

Admission Webhook Integration

The mutating admission webhook intercepts POD creation requests, mounts the Lxcfs procfs into the pod, and then forwards the request to the regular handler, persisting changes to etcd.

Configuration steps include:

Kubernetes version >= 1.9.

Enable MutatingAdmissionWebhook and ValidatingAdmissionWebhook in the kube‑apiserver admission control flags.

If kube‑proxy is absent, add --enable-aggregator-routing=true to the apiserver.

Upgrade runc on all nodes if the current version does not allow mounting procfs.

Result

After deployment, container‑level commands like top and free report the container’s own resource limits, and applications such as Java or Nginx receive correct CPU and memory information, eliminating startup failures and performance issues.