Data‑Driven Decision Optimization: Challenges and Advances in Offline Reinforcement Learning

The talk begins with an overview of the difficulties faced when deploying data‑driven decision optimization in practice, highlighting the gap between ideal AI deployment and the constraints of real‑world environments, such as limited interaction, safety constraints, and imperfect data coverage.

It then introduces offline reinforcement learning (Offline RL) as a solution that removes the need for online environment interaction by learning policies directly from static datasets. The presentation covers the evolution of Offline RL, the shortcomings of naïve off‑policy methods, and the importance of handling distribution shift, out‑of‑distribution (OOD) actions, and over‑conservative behavior.

Several algorithmic strategies are described, including policy‑constraint techniques, value‑function regularization, state‑conditioned distance functions, and the DOGE (Data‑driven Offline Gradient‑based Exploration) method, which combines policy constraints with geometry‑aware learning to improve generalization and reduce conservatism.

Practical applications are showcased, notably the deployment of Offline RL for combustion‑airflow optimization in coal‑fired power plants, achieving measurable efficiency gains, and the use of mixed offline‑online approaches (e.g., H2O) that fuse real‑world data with imperfect simulators to correct dynamics bias.

The session concludes with a Q&A discussing the distinction between sequential decision making and traditional PID control, the challenges of scaling across heterogeneous power‑plant configurations, and future directions such as stitching trajectories, imitation‑learning hybrids, and broader industrial adoption.