AoE (AI on Edge): Design Principles, Architecture, and Integration Guide

Background – With the rapid development of artificial‑intelligence technologies, many inference frameworks have emerged for edge devices. While they give developers more choices, they also increase the cost of deploying AI to terminals. Besides selecting a framework, developers must consider data pre‑/post‑processing stability, model package size, and data security.

What is AoE? AoE (AI on Edge) is an open‑source AI integration runtime environment (IRE) released by Didi. It follows the design principles of stability, ease‑of‑use, and security, helping developers deploy deep‑learning models from various frameworks onto edge devices efficiently.

The project’s GitHub repository is https://github.com/didi/aoe .

Why a dedicated edge AI runtime?

Proliferation of terminal‑side inference frameworks raises deployment complexity and cost.

Directly integrating a framework involves dynamic library loading, resource management, pre‑/post‑processing, model upgrades, and stability guarantees.

AoE has already been used in Didi’s bank‑card OCR product.

Supported inference frameworks – AoE currently integrates NCNN and TensorFlow Lite. It abstracts the five common stages of any inference framework: initialization, pre‑processing, inference execution, post‑processing, and resource release.

Core abstraction

AoE defines two key interfaces:

/**
 * Model translation component
 */
interface InterpreterComponent
extends Component {
    /**
     * Initialize and load model resources
     * @param context   Context for service binding
     * @param modelOptions List of model configuration options
     * @return true if the model is loaded successfully
     */
    boolean init(@NonNull Context context, @NonNull List
modelOptions);

    /** Execute inference */
    @Nullable TOutput run(@NonNull TInput input);

    /** Release resources */
    void release();

    /** Check if model is ready */
    boolean isReady();
}

interface Convertor
{
    /** Convert business input to model input */
    @Nullable TModelInput preProcess(@NonNull TInput input);

    /** Convert model output to business output */
    @Nullable TOutput postProcess(@Nullable TModelOutput modelOutput);
}

Stability guarantees on Android – To avoid crashes caused by native operations, AoE runs native code in a separate process. This isolates inference crashes from the main app process.

The implementation consists of three parts:

Register a remote service in AndroidManifest.xml : <application> <service android:name=".AoeProcessService" android:exported="false" android:process=":aoeProcessor" /> </application>

Re‑bind the service when it crashes: @Override public Object run(@NonNull Object input) { if (isServiceRunning()) { // ... execute inference } else { bindService(); // restart remote process } return null; }

Optimize cross‑process communication. AoE uses the Kryo library for serialization because it accounts for ~90% of the IPC overhead.

MNIST integration example

To add a new model, determine its underlying framework (e.g., TensorFlow Lite) and extend the corresponding InterpreterComponent :

public class MnistInterpreter extends TensorFlowLiteInterpreter
{
    @Nullable
    @Override
    public float[] preProcess(@NonNull float[] input) {
        return input;
    }

    @Nullable
    @Override
    public Integer postProcess(@Nullable float[][] modelOutput) {
        if (modelOutput != null && modelOutput.length == 1) {
            for (int i = 0; i < modelOutput[0].length; i++) {
                if (Float.compare(modelOutput[0][i], 1f) == 0) {
                    return i;
                }
            }
        }
        return null;
    }
}

Configure the client: mClient = new AoeClient(requireContext(), "mnist", new AoeClient.Options() .setInterpreter(MnistInterpreter.class)/*.useRemoteService(false)*/, "mnist"); Run inference and release resources: // Initialize int resultCode = mClient.init(); // Inference Object result = mClient.process(mSketchModel.getPixelData()); if (result instanceof Integer) { int num = (int) result; Log.d(TAG, "num: " + num); mResultTextView.setText(num == -1 ? "Not recognized." : String.valueOf(num)); } // Release if (mClient != null) { mClient.release(); } AoE currently provides SDKs for Android and iOS, with Linux support slated for release soon. Call to action – The project invites developers to join the AoE community, contribute code, or provide feedback. Stars on the GitHub repository are welcomed.