MPC-Pretrained Reinforcement Learning

All research & projects

RESEARCH CASE STUDY

MPC-Pretrained Reinforcement Learning

Developed an RL-centered workflow that initializes Twin Delayed Deep Deterministic Policy Gradient (TD3) from model predictive control demonstrations, then gives the neural policy direct control and online adaptation across polymer-reactor and Aspen Dynamics C2-splitter simulations.

Collaboration

Linde

Role

Designed and evaluated the demonstration generator, actor and critic pretraining, staged online fine-tuning, reward and replay logic, and controller comparisons across the two process simulations.

Methods

TD3 · Behavioral cloning · Online adaptation · Mixed experience replay · MPC demonstrations

System architecture

Four-stage diagram showing offline MPC demonstration generation, behavioral cloning and offline critic training, online TD3 control with bounded actions and mixed replay, and final evaluation without exploration or network updates. MPC is used to initialize the RL policy and is not the online controller.

MPC supplies offline expert demonstrations that initialize a TD3 actor and twin critics. The pretrained policy then controls the process, stores executed transitions, and adapts online through offset-aware rewards and mixed replay.

Contribution and implementation

  • An offline MPC expert generated about three million one-step transitions; behavioral cloning trained the bounded TD3 actor, while twin critics were initialized from the same offline experience.

  • TD3 observes the estimated process state, output target, and previous input, then directly generates bounded continuous commands for the polymer reactor or C2 splitter without runtime MPC control.

  • Online transfer uses replay warm-up, critic-only refinement, and joint actor-critic updates, followed by offset-aware reward shaping and a replay mixture of prioritized, recent, and uniform experience.

Implementation: Python · PyTorch · Aspen Dynamics

Results and evaluation

Across five independent polymer-reactor simulation runs, RL2 achieved a lower mean final absolute steady-state error in all eight comparisons; at nominal SP(a), viscosity error decreased from 0.034 plus or minus 0.024 to 0.001 plus or minus 0.001 L/g.

RL2 achieved a lower final absolute steady-state error in all 12 tabulated C2-splitter simulation comparisons; across the six ethane-composition cases, recomputed reductions ranged from 89.69% to 99.63%, with run count and uncertainty not stated in the source table.

The Aspen Dynamics loop required approximately 1 second per online control step, including learning operations, compared with a 10-minute control interval; the polymer study’s complete 200-episode online run required approximately 20 minutes.

Two grouped horizontal bar charts compare RL1 with RL2 final errors for intrinsic viscosity and temperature under nominal and operational-change scenarios at two setpoints. RL2 bars are shorter in all eight comparisons, with exact mean and standard-deviation values shown beside each bar.

Mean final absolute steady-state errors for RL1 and RL2 across five independent polymer-reactor runs. RL2 reduced the mean final error in every nominal and operation-change comparison; values show mean and standard deviation.

Two horizontal bar panels show six composition and six temperature error reductions across nominal, fluctuating-feed, and gradual-ramp C2-splitter simulations. Composition reductions range from 89.69 to 99.63 percent, and temperature reductions range from 35.78 to 97.77 percent.

Recomputed RL2 reductions in final absolute steady-state error relative to RL1 for the Aspen Dynamics C2 splitter. The source table does not report run count or uncertainty.

Limitations and evidence maturity

  • All reported evaluations are simulation-based; the work does not demonstrate laboratory or plant deployment, and the C2 result table does not report a run count or uncertainty.

  • The reported simulations enforced actuator bounds only; policy pretraining is not a formal guarantee of stability, safety, or broader constraint satisfaction, and reward shaping and mixed replay were not separated in a complete ablation.