Model Deployment Challenges and a Seldon‑Based Cloud‑Native Solution

Model deployment is often the "last mile" of algorithm engineering, bringing high operational complexity such as load balancing, fault tolerance, scaling, resource isolation, rate limiting, and metric monitoring, which are typically outside the expertise of algorithm teams.

The original in‑house platform, Sunfish, provides a deployment module called ABox that consists of three components: master (routing requests via Zookeeper and executing custom user‑defined libraries), worker (registering heartbeats, pulling models, and running TensorFlow‑Serving), and manager (registering servers, creating models, handling UDL updates, and managing clusters and services). While functional for TensorFlow CPU models, ABox suffers from several pain points.

Pain points include heavy operational effort (manual scaling, limited instance count tied to worker nodes, manual URL registration, OOM risk in containerized TF‑Serving), load imbalance across workers, lack of resource isolation, fragmented management of TensorFlow and other frameworks, and no GPU deployment capability.

To address these issues, the team introduced Seldon , an open‑source, cloud‑native model‑deployment platform built on Kubernetes. Seldon registers a custom resource definition (CRD) called SeldonDeployment and provides a controller that creates, updates, and deletes the underlying Deployments, Services, and VirtualServices. It also integrates with Prometheus and Jaeger for monitoring and tracing.

Seldon supports two kinds of model servers: Reusable Model Servers (e.g., TensorFlow‑Serving, Triton) that fetch models from external storage, and Non‑Reusable Model Servers that embed the model in a custom container. An example of a TensorFlow model deployment is shown below:

apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  name: tfserving
spec:
  name: mnist
  predictors:
  - graph:
      children: []
      implementation: TENSORFLOW_SERVER
      modelUri: gs://seldon-models/tfserving/mnist-model
      name: mnist-model
      parameters:
        - name: signature_name
          type: STRING
          value: predict_images
        - name: model_name
          type: STRING
          value: mnist-model
    name: default
    replicas: 1

The redesign kept the ABox master as the Dubbo entry point, replaced the original ingress controller with an Nginx Ingress, added an HDFS‑Initializer for reusable model servers, adopted Tencent Cloud's GpuManager for GPU sharing, integrated a Kubernetes client into the algorithm platform for CRUD operations on Seldon deployments, and built custom logging (Filebeat → Kafka → log‑server) and resource‑monitoring dashboards.

Key implementation details include:

Ingress replacement: using Nginx instead of Istio/Ambassador to simplify operations.

Reusable Model Server initialization: custom HDFS‑Initializer to pull models from HDFS.

GPU solution: selected GpuManager over vGPU due to compatibility and operational considerations, accepting a ~5% performance overhead.

Log management: containers embed Filebeat to ship logs to Kafka, which are consumed by a proprietary log‑server.

Resource monitoring: periodic collection of CPU/memory per pod, with real‑time dashboards.

Service migration: staged rollout across QA, pre‑release, and production environments using traffic‑splitting switches.

Additional notes highlight that TensorFlow‑Serving requires AVX/AVX2 instructions; on VMs lacking these, the container crashes with an "Illegal instruction" error. The following command reproduces the failure:

/usr/bin/tensorflow_model_server --port=9000 --model_name=xxx --model_base_path=/path/to/model

In conclusion, the first phase of the migration has delivered a unified, cloud‑native model‑deployment capability that reduces operational burden, yet further work remains—such as supporting inference graphs, advanced rollout strategies, automatic custom image building, and additional model initializers.