Multi-Channel RL-Assisted MPC

All research & projects

RESEARCH CASE STUDY

Multi-Channel RL-Assisted MPC

Designed a four-channel RL-assisted MPC architecture in which DQN and TD3 agents propose bounded controller adjustments and channel-specific critic gates decide whether to use each learned proposal or a conservative supervisor action.

Collaboration

Linde

Role

Researcher who designed and evaluated the multi-channel RL-assisted MPC workflow in closed-loop process simulations.

Methods

MPC · DQN · TD3

System architecture

A vertical controller pipeline connects measurements and targets to horizon and weight configuration, dynamic-matrix correction, constrained MPC, residual projection, and a simulated process. A DQN assistant and three TD3 assistants each pass through a separate critic gate, with a dashed simulation-learning path below.

Four specialized RL assistants propose bounded changes around an MPC calculation; channel-specific critic gates can retain a conservative supervisor action.

Contribution and implementation

  • The horizon channel uses DQN for discrete prediction/control-horizon selection, while TD3 agents continuously adjust objective weights, correct the dynamic matrix, and propose a constrained residual-input correction around the MPC backbone.

  • Each channel has its own critic-based selector. A learned proposal must exceed its supervisor alternative by a configured advantage margin; ties, non-finite scores, and policies that are not ready default to the supervisor.

  • Simulation training uses the executed actions and a common control reward with mixed prioritized, recent, and uniform replay. Learned proposals are released only after data collection and critic-learning phases, and the final evaluation episode disables exploration and network updates.

Implementation: Python · PyTorch

Results and evaluation

Polymerization Scenario 2, one saved deterministic evaluation trajectory: combined gated assistance reduced viscosity RMSE from 0.2396 to 0.0957 L/g and temperature RMSE from 1.160 to 0.848 K relative to offset-free MPC. Band-normalized error fell from 11.415 to 4.561, while normalized input movement increased from 1.683% to 4.675%.

Distillation Scenario 2, one saved deterministic evaluation trajectory: combined gated assistance reduced composition RMSE from 0.00427 to 0.00320 and temperature RMSE from 0.4022 to 0.1252 degrees C relative to offset-free MPC. Joint acceptable-band occupancy increased from 76.25% to 91.75%, while normalized input movement increased from 0.418% to 0.625%.

Distillation Scenario 1 descriptive saved-run ensembles: mean worst post-handoff reward for combined assistance was -14.14 +/- 14.06 with gating (n=6) versus -521.30 +/- 42.93 without gating (n=4). These values are descriptive means and one sample standard deviation, not confidence intervals or a significance result.

Four metric cards compare offset-free MPC with combined gated assistance in one saved polymerization Scenario 2 trajectory. Viscosity RMSE falls from 0.2396 to 0.0957 liters per gram, temperature RMSE from 1.160 to 0.848 kelvin, and band-normalized error from 11.415 to 4.561, while normalized input movement rises from 1.683 to 4.675 percent.

In the saved changed-condition polymer simulation, combined gated assistance improved both tracking outputs, with increased input movement.

A horizontal dot plot compares mean worst post-handoff reward with and without gating for dynamic-matrix, residual, and combined assistance. Gated values are less negative in each descriptive saved-run ensemble, with unequal run counts and one sample standard deviations shown.

Across the saved distillation runs, critic gating limited the worst reward drops after learned policies were released.

Limitations and evidence maturity

  • The evidence is limited to nonlinear polymerization and Aspen Dynamics distillation simulations; it does not establish laboratory, pilot-plant, or production deployment, formal closed-loop stability, recursive feasibility, or physical safety.

  • Scenario 2 uses one saved trajectory per configuration. Repeated Scenario 1 studies have unequal run counts and incomplete seed and commit provenance, and no leave-one-channel-out combined ablation establishes the causal contribution of each assistance channel.