How We Built a Scalable Apache APISIX Gateway on the 360 Platform

API Gateway Selection

In October 2019 our team set out to redesign the gateway layer of the 360 basic operation platform, evaluating popular open‑source gateways such as Kong, Orange, and Apache APISIX, and ultimately selecting APISIX because its etcd‑based storage matched our requirements.

Online Running Status

We now manage close to 900 APIs with a daily PV of about 10 million; monitoring shows both the gateway and all micro‑services are operating smoothly.

Gateway POD monitoring and micro‑service load graphs confirm stable performance.

Architecture Diagram

The final architecture places the gateway service on our container cloud, while the etcd cluster runs on three virtual machines.

Containerized Development and Deployment

We built a custom OpenResty image that includes required APISIX dependencies, installed APISIX 0.9, and then demonstrated the full deployment process.

<code>docker run -itd -p 9080:9080 -p9443:9443 --name myapisix hulklab/openresty:0.0.1</code>

<code>docker exec -it myapisix bash</code>

<code>luarocks install apisix 0.9-0</code>

<code># apisix 0.9-0 is now installed in /usr/local (license: Apache License 2.0)</code>

<code>cd /usr/local/apisix</code>

<code>ls -l</code>

<code>drwxr-xr-x 3 root   root 4096 Dec 18 11:52 conf</code>

<code>drwxr-xr-x 2 root   root 4096 Dec 18 11:37 logs</code>

<code>apisix start</code>

<code>ps aux|grep openresty</code>

If apisix start fails, start etcd first and adjust etcd.host in /usr/local/apisix/conf/config.yaml .

<code>#config.yaml:69</code>

<code>etcd:</code>

<code>  host: "http://172.17.0.1:2379"   # etcd address</code>

Plugin Development Overview

Our project separates business code from APISIX core code, allowing independent upgrades. The lua/apisix/plugins directory holds custom plugins, while lua/libs stores shared libraries. The following plugins are used:

ip-restriction (access_by_lua)

basic-auth (access_by_lua, custom)

web-auth (access_by_lua, custom)

limit-rate (access_by_lua, custom)

proxy-rewrite (access_by_lua, balancer_by_lua, custom)

log (log_by_lua, custom)

alarm (log_by_lua, custom)

Basic‑Auth Plugin Sample

The plugin defines a single enable parameter to allow fine‑grained control when the plugin is bound to a service.

<code>local plugin_name = "odin-basic-auth"</code>

<code>local schema = {</code>

<code>    type = "object",</code>

<code>    properties = {</code>

<code>        enable = { type = "boolean", default = true, enum = { true, false } },</code>

<code>    },</code>

<code>}</code>

<code>local _M = {</code>

<code>    version = 0.1,</code>

<code>    priority = 2802,</code>

<code>    name = plugin_name,</code>

<code>    schema = schema,</code>

<code>}</code>

The enable flag is necessary because APISIX plugins can be bound to services, making it impossible to disable them per route without such a parameter.

<code>function _M.check_schema(conf)</code>

<code>    local ok, err = core.schema.check(schema, conf)</code>

<code>    if not ok then</code>

<code>        return false, err</code>

<code>    end</code>

<code>    return true</code>

<code>end</code>

<code>function _M.access(conf, ctx)</code>

<code>    if not conf.enable then return end</code>

<code>    local headers = ngx.req.get_headers()</code>

<code>    if not headers.Authorization then</code>

<code>        return 401, { message = "authorization is required" }</code>

<code>    end</code>

<code>    local username, password, err = extract_auth_header(headers.Authorization)</code>

<code>    if err then return 401, { message = err } end</code>

<code>    local res = authorizations_etcd:get(username)</code>

<code>    if res == nil then return 401, { message = "failed to find authorization from etcd" } end</code>

<code>    if not res.value or not res.value.id then return 401, { message = "user is not found" } end</code>

<code>    if res.value.password ~= password then return 401, { message = "password is error" } end</code>

<code>end</code>

Etcd Cache Object

The plugin creates an etcd cache object during the init phase, enabling fast in‑memory reads while keeping data up‑to‑date via etcd watch.

<code>local authorizations_etcd</code>

<code>local appkey_scheme = {</code>

<code>    type = "object",</code>

<code>    properties = {</code>

<code>        username = { description = "username", type = "string" },</code>

<code>        password = { type = "string" }</code>

<code>    }</code>

<code>}</code>

<code>function _M.init()</code>

<code>    authorizations_etcd, err = core.config.new("/authorizations", { automatic = true, item_schema = appkey_scheme })</code>

<code>    if not authorizations_etcd then error("failed to create etcd instance for fetching authorizations: " .. err) end</code>

<code>end</code>

Plugin API Definition

The plugin exposes a custom API to add or update authorizations in etcd.

<code>function _M.api()</code>

<code>    return {</code>

<code>        { methods = { "POST", "PUT" }, uri = "/apisix/plugin/basic-auth/set", handler = set_auth }</code>

<code>    }</code>

<code>end</code>

<code>local function set_auth()</code>

<code>    local username = req.get_str("username")</code>

<code>    local password = req.get_str("password")</code>

<code>    local key = "/authorizations/" .. username</code>

<code>    local res, err = core.etcd.set(key, { username = username, password = password })</code>

<code>    if not res then core.response.exit(500, err) end</code>

<code>    core.response.exit(res.status, res.body)</code>

<code>end</code>

<code>curl -i -X PUT 'http://127.0.0.1:9080/apisix/plugin/basic-auth/set' -d username=zhangsan -d password=hao123 -d user_id=3 -d action_ids=,1,2,3,</code>

Post‑Deployment Issues

Log files grew beyond the 50 GB quota, so we added cron and logrotate to the image and started the cron daemon in the container entrypoint to clean daily logs.

Acknowledgements

We thank the Apache APISIX contributors; see the official website and GitHub repository for more information.