From Sharding‑JDBC to ShardingSphere: A Year‑Long Open‑Source Journey and Apache Incubator Success

Opportunity – Joining JD.com

Before joining JD.com, the author led architecture at another internet company, open‑sourcing Elastic‑Job and Sharding‑JDBC, gaining a solid reputation in the open‑source community.

He also contributed to other projects such as DubboX, Apache Mesos documentation, and early promotion of Apache SkyWalking, believing open‑source should transcend company, race, and belief boundaries.

Seeking broader horizons, he chose JD.com to nurture Sharding‑JDBC into an Apache‑level project, despite other offers.

Expansion – Rapid Growth

Upon arrival, Sharding‑JDBC lacked cloud deployment capability and distributed transaction support. The author expanded its scope by adding a cloud‑ready proxy, leveraging his Netty experience to develop a MySQL protocol proxy within two weeks.

He integrated the proxy with backend sharding, built a Service Mesh‑compatible sidecar concept, and published an InfoQ article on Database Mesh.

The project was renamed ShardingSphere, comprising Sharding‑Proxy, Sharding‑Sidecar, and the original driver, forming an ecosystem illustrated by the included diagram.

Pursuit – Consolidation

Preparing the Apache Incubator proposal, the author received feedback that his personal commit ratio was too high, prompting team formation.

The newly built team accelerated development of core sharding, proxy, and distributed transaction features, collaborating with Apache ServiceComb (Saga) and Apache SkyWalking for APM integration.

Harvest – Achievements

ShardingSphere quickly integrated with two China‑originated Apache projects, deepening its understanding of the Apache Way.

In October 2018, after successful discussions with Apache mentors, Roman became Champion, with Craig, Jiang Ning, and Benjamin as mentors, leading to ShardingSphere’s acceptance into the Apache Incubator on November 10, 2018.

New Challenges – Scaling Up

ShardingSphere now serves a massive JD.com business with hundreds of thousands of data nodes, moving from framework‑based sharding to a dedicated middleware, handling high concurrency and large‑scale data.

Performance tests show near‑native JDBC speed with stable GC and resource usage, positioning ShardingSphere for further growth in 2019.

Conclusion

The past year brought setbacks and successes; the team’s nomination for a technology award validates their effort. The author thanks the ShardingSphere community and looks forward to continued innovation and contribution to JD.com and the broader industry.