Graceful Shutdown in Go: Designing Robust Service Termination with the GS Library
This article describes a real‑world incident where rapid pod scaling caused order‑submission failures in a serverless e‑commerce platform, analyzes the root causes, and presents a Go‑based graceful‑shutdown solution—including ASyncClose, SyncClose, and ForceSyncClose modes—implemented in the open‑source GS library to help developers reliably terminate services.
During a May Day holiday the author, a veteran of 17 years in IT, was forced to work overtime because a mis‑configured pressure threshold caused a serverless merchant's core pods to scale up and down erratically, leading to frequent order‑submission failures.
Even though a 5‑minute scale‑down suppression was set, the rapid scaling of Pod instances (sometimes 10, sometimes 70) overwhelmed the system, and the shutdown logic failed to close the HTTP server first, leaving dangling database connections.
The post introduces the concept of "graceful shutdown" and provides a simple Go program that listens for SIGINT or SIGTERM and prints "Shutting down..." before exiting.
It then discusses the pain points of repetitive shutdown code, difficulty ordering module termination, lack of multi‑service handling, and poor extensibility.
Three shutdown strategies are presented:
All asynchronous shutdown : each service's Close() is invoked via asyncClose() and a final asyncWait() waits for completion.
All synchronous shutdown : each service's Close() is invoked via syncClose() followed by syncWait() .
Mixed async/sync shutdown : some services close asynchronously while others close synchronously, with appropriate wait calls.
The GS library is then refactored to introduce a CloseType enum ( ASyncClose , SyncClose , ForceSyncClose ) and a unified close method that handles the different modes using sync.Once , a wait group, and context cancellation.
Key functions include:
func (s *TerminateSignal) close(closeMode CloseType, wg *sync.WaitGroup) { ... }
func (s *TerminateSignal) Close(wg *sync.WaitGroup) { s.close(ASyncClose, wg) }
func (s *TerminateSignal) SyncClose(wg *sync.WaitGroup) { s.close(SyncClose, wg) }A waiting helper demonstrates how to react to OS signals and apply the chosen CloseType across multiple TerminateSignal instances, using either goroutine‑based asynchronous closing or sequential synchronous closing.
The article concludes with a quick‑start guide for the GS library, showing how to create a TerminateSignal , register cleanup callbacks, and wait for graceful termination using WaitForAsync , WaitForSync , or WaitForForceSync .
Overall, the piece provides a practical, code‑driven approach to improving service reliability by ensuring orderly shutdown of internal modules in Go‑based backend systems.
Rare Earth Juejin Tech Community
Juejin, a tech community that helps developers grow.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.