How to Develop a Custom Protocol for a Service Mesh Using Aeraki and MetaProtocol Proxy

How to Develop a Custom Protocol for a Service Mesh

Aeraki Mesh, a CNCF sandbox project, enables management of any L7 protocol within a service mesh. This tutorial shows how to use Aeraki and MetaProtocol Proxy to develop a custom protocol, providing L7 traffic routing, local and global rate limiting, and full Istio integration.

Implementing the Codec Interface

The Aeraki example awesomerpc (https://github.com/aeraki-mesh/meta-protocol-awesomerpc) provides a framework for a custom protocol. The key files are src/application_protocols/awesomerpc/awesomerpc_codec.h and awesomerpc_codec.cc , which define a codec that inherits from MetaProtocolProxy::Codec . Implement the three required methods: decode , encode , and onError .

git clone https://github.com/aeraki-mesh/meta-protocol-awesomerpc.git my-protocol-proxy

Focus on the src/application_protocols/awesomerpc/ directory. The codec header defines the class:

/**
 * Codec for Awesomerpc protocol.
 */
class AwesomerpcCodec : public MetaProtocolProxy::Codec,
                  public Logger::Loggable
{
public:
  AwesomerpcCodec() {}
  ~AwesomerpcCodec() override = default;

  // Decode the buffer into metadata for routing, rate‑limiting, etc.
  MetaProtocolProxy::DecodeStatus decode(Buffer::Instance& buffer,
                                         MetaProtocolProxy::Metadata& metadata) override;

  // Encode mutations back into the buffer.
  void encode(const MetaProtocolProxy::Metadata& metadata,
              const MetaProtocolProxy::Mutation& mutation,
              Buffer::Instance& buffer) override;

  // Encode error information into the buffer.
  void onError(const MetaProtocolProxy::Metadata& metadata,
               const MetaProtocolProxy::Error& error,
               Buffer::Instance& buffer) override;

  ...
};

Reference implementations for Dubbo and Thrift codecs can be found in the MetaProtocol Proxy repository.

Configuring the WORKSPACE

In the repository root, edit the WORKSPACE file to add dependencies for MetaProtocol, Envoy, and Istio‑Proxy. Use the appropriate Git commit hashes and ensure version compatibility between Aeraki, MetaProtocol, and Istio.

Building the Proxy

Use Ubuntu 18.04 as the build environment. Install Bazel (via bazelisk) and required packages:

sudo wget -O /usr/local/bin/bazel https://github.com/bazelbuild/bazelisk/releases/latest/download/bazelisk-linux-$( [ $(uname -m) = "aarch64" ] && echo "arm64" || echo "amd64" )
sudo chmod +x /usr/local/bin/bazel

sudo apt-get install \
autoconf \
automake \
cmake \
curl \
libtool \
make \
ninja-build \
patch \
python3-pip \
unzip \
virtualenv

sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt update
sudo apt-get install llvm-10 lldb-10 llvm-10-dev libllvm10 llvm-10-runtime clang-10 clang++-10 lld-10 gcc-10 g++-10

Compile the binary with:

./build.sh

Defining an ApplicationProtocol CRD

Create an Aeraki ApplicationProtocol custom resource so Istio can recognize the new protocol:

apiVersion: metaprotocol.aeraki.io/v1alpha1
kind: ApplicationProtocol
metadata:
  name: my-protocol
  namespace: istio-system
spec:
  protocol: my-protocol
  codec: aeraki.meta_protocol.codec.my_protocol

Protocol Selection and Service Definition

Service ports must follow the naming pattern tcp-metaprotocol-{application‑protocol}-xxx . Example ServiceEntry for a Dubbo service:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: dubbo-demoservice
  namespace: meta-dubbo
  annotations:
    interface: org.apache.dubbo.samples.basic.api.DemoService
spec:
  hosts:
    - org.apache.dubbo.samples.basic.api.demoservice
  ports:
    - number: 20880
      name: tcp-metaprotocol-dubbo
      protocol: TCP
  workloadSelector:
    labels:
      app: dubbo-sample-provider
  resolution: STATIC

Example Service for a Thrift service:

apiVersion: v1
kind: Service
metadata:
  name: thrift-sample-server
spec:
  selector:
    app: thrift-sample-server
  ports:
    - name: tcp-metaprotocol-thrift-hello-server
      protocol: TCP
      port: 9090
      targetPort: 9090

Note: The port name must include the "tcp" prefix so Istio treats it as a TCP service, after which Aeraki extracts the application protocol for L7 processing.

For further community support, join the cloud‑native technical groups via the provided QR code.