Reinforcement-Learning pH Control for BioSMB

All research & projects

RESEARCH CASE STUDY

Reinforcement-Learning pH Control for BioSMB

A simulation-pretrained TD3 policy generated bounded acid/base flow decisions during a 4.55-hour BioSMB laboratory run across a changing pH sequence.

Collaboration

Sartorius

Role

Defined the chemistry-aware reinforcement-learning formulation, trained and exported the TD3 actor, integrated it into the laboratory application, designed scheduled-target operation, and audited the laboratory time series.

Methods

TD3 · simulation pretraining · time-series analysis

System architecture

Diagram showing outlet pH flowing into a five-value state, then a pretrained TD3 actor, a ratio-first action mapper, and an inline acetate-buffer mixing process with acid, acetate, and water streams returning the next pH measurement.

A reliable outlet pH measurement and target form the TD3 state; the pretrained actor produces two normalized decisions that a chemistry-aware mapper converts into bounded acid and acetate flow commands.

Four-stage diagram separating offline TD3 training, actor export, demonstrated pretrained inference in the laboratory, and a later online-adaptation extension marked as implemented but not shown in the lab data.

TD3 pretraining, actor export, demonstrated laboratory inference, and the later implemented-only online-adaptation extension remain separate evidence stages.

Contribution and implementation

  • The TD3 actor was pretrained for 500,000 steps in an ideal chemistry simulation and exported with a fixed five-state, two-action deployment contract before laboratory application.

  • A ratio-first bounded action mapper converts the TD3 policy outputs into feasible acid and acetate flow commands using the measured outlet pH, target, error, and previous actions.

  • Laboratory analysis preserves the full transient and settled-block views; the later replay-based online-adaptation path is implemented software but is not demonstrated by this stored run.

Implementation: Python · PyTorch

Results and evaluation

The stored hardware-connected experiment ran for 4.5517 hours and contains 14,426 valid outlet-pH measurements and 236 controller-action changes.

With targets reconstructed from the supplied schedule, the transient-inclusive raw MAE was 0.02376 pH and the mean absolute error of the eight final-five-minute block means was 0.0088 pH.

As a run-level diagnostic rather than an inline-flow measurement, one-minute reservoir-mass-derived and commanded flows had correlations of 0.99846 for acid and 0.99932 for sodium acetate.

Line chart with eight scheduled blocks. Reconstructed targets rise from 3.9 to 5.5 and return to 4.3, while measured final-five-minute pH means closely follow the same path with small standard-deviation bars.

Final-five-minute measured pH means follow the reconstructed rising-and-falling target schedule across eight blocks; mean absolute block-mean error is 0.0088 pH.

Three-bar chart comparing final-five-minute block-mean pH error for the startup transition, upward target changes, and downward target changes, with a note that this is not a calibrated uncertainty or robustness study.

Stored-run operating-direction comparison for startup, upward, and downward target changes; this is not a formal uncertainty study.

Limitations and evidence maturity

  • This is one stored laboratory run; the active target and exact policy identity were not embedded in the CSV, so target-dependent metrics are reconstructed rather than directly logged.

  • The run does not demonstrate the later online-gradient adaptation path, formal robustness or safety guarantees, production deployment, or same-protocol superiority over an established controller.