Stability-Aware Reinforcement Learning

All research & projects

RESEARCH CASE STUDY

Stability-Aware Reinforcement Learning

Developed an RL-first control architecture in which TD3 proposes continuous coolant and monomer flow commands and a Lyapunov-guided GART-LMPC layer evaluates each move before it is applied to a simulated polymerization reactor.

Collaboration

Linde

Role

Designed and integrated the TD3 policy, MPC-based actor pretraining, one-step action evaluation, recovery controller, and repeated-run and new-setpoint analysis.

Methods

TD3 · Behavioral cloning · Lyapunov-guided control · GART-LMPC

System architecture

A left-to-right flow from measurements and target through TD3, one-step prediction, and a Lyapunov acceptance test. The accepted path goes to the applied input; the rejected path goes to GART-LMPC fallback before joining the applied-input path. Process feedback closes the loop, while reward, replay, and policy updates appear in a separate training-only lane.

Stability-aware TD3 execution and learning workflow. TD3 proposes a candidate process input; the identified model and accepted GART target define a one-step Lyapunov check; accepted proposals or GART-LMPC replacement moves are applied to the process.

Contribution and implementation

  • TD3 receives an observer-based process state, requested viscosity and temperature targets, and previous inputs, then proposes bounded coolant- and monomer-flow commands.

  • An identified model and accepted Guarded Admissible Reachable Target (GART) evaluate each proposal against model, input, move, and one-step Lyapunov checks. A failed proposal is replaced by a GART-LMPC move; if no usable replacement exists, the implementation clips and holds the previous input without treating that contingency as automatically certified.

  • MPC first-move examples initialize the TD3 actor before online learning. Twin critics, delayed actor updates, target-policy smoothing, and mixed replay learn from the control action that was actually applied rather than from a rejected proposal.

Implementation: Python · PyTorch · CVXPY

Results and evaluation

Initial adaptation, eight retained simulation runs per method: gated MPC-pretrained TD3 achieved viscosity RMSE of 0.132 +/- 0.002 L/g, temperature RMSE of 0.317 +/- 0.016 K, and 2.47 +/- 0.36% actual interventions, compared with 0.142 +/- 0.002 L/g, 0.374 +/- 0.014 K, and 3.62 +/- 0.52% for gated cold-start TD3.

Continuation, eight retained simulation runs per method: gated MPC-pretrained TD3 achieved viscosity RMSE of 0.125 +/- 0.002 L/g, temperature RMSE of 0.274 +/- 0.014 K, and 2.14 +/- 0.50% actual interventions, compared with 0.129 +/- 0.003 L/g, 0.334 +/- 0.019 K, and 3.54 +/- 0.48% for gated cold-start TD3.

In one continued-online-learning new-setpoint test, final-cycle gated-TD3 RMSE was 0.0965 L/g for viscosity and 0.512 K for temperature, compared with 0.1169 L/g and 0.566 K without the gate and 0.2287 L/g and 1.094 K for deterministic GART-LMPC. Actual gated interventions occurred on 2.18% of samples during the first 100 cycles and 0.125% during the final cycle; the ungated trajectory failed the same diagnostic condition on 12.09% and 18.75% of samples.

A two-column matrix compares four TD3 configurations over initial adaptation and continuation. Each cell reports viscosity RMSE, temperature RMSE, and an authority-event rate, with separate labels for diagnostic failures and actual gate interventions.

Tracking and authority metrics during initial adaptation and continuation. Values are mean plus or minus one standard deviation over eight retained runs per TD3 configuration; ungated rows show diagnostic certificate failures and gated rows show actual interventions.

Three panels compare final-cycle composite RMSE for gated TD3, ungated TD3, and GART-LMPC, then compare outside-setpoint-neighborhood event rates for gated and ungated TD3 over the first 100 cycles and the final cycle. The figure notes that learning continued and the result is single-run.

Single-run new-setpoint online-adaptation summary. The gated TD3 trajectory has lower final-cycle tracking error than the same starting TD3 policy without the gate and deterministic GART-LMPC; authority-event types remain explicitly separated.

Limitations and evidence maturity

  • The evidence is limited to a nonlinear polymer-CSTR simulation and a local, model-based, one-step acceptance condition; it does not establish laboratory or plant deployment, universal physical safety, global nonlinear stability, recursive feasibility at every step, or production readiness.

  • Eight-run results are descriptive means and standard deviations without inferential statistics. The new-setpoint comparison uses one continued-online-learning trajectory per case, and its practical neighborhood is a post-processing threshold rather than a controller rule or theorem.