Multi-scale integrated management and operation of chemical/energy systems combining reinforcement learning and mathematical programming

Abstract

Uncertainty on the information essential to decision-making and inconsistencies between different decision levels are the main contributors to interrupt sustainable operation of chemical manufacturing systems. Thus a new discipline is required beyond the existing deterministic and reactive approaches to introduce a globally optimal solution at each time. Therefore, the purpose of this study is to introduce a smart and sustainable management and operation strategy for energy/chemical manufacturing system through data and model based decision supporting tools representing (projecting) an uncertain real world. In particular, the study focuses on the linkage between the high-level (planning) and the low-level (operation) decision layers. The resulting expansion of the boundary of decision-making process can provide more robust and flexible management and operation strategies by resolving inconsistency between different levels.

For this, I develop a multi-scale decision-making model that combines Markov decision process and mathematical programming in a complementary way. To support the integration of the decision hierarchy, a data-driven uncertainty prediction model is suggested which is valid across all time scales considered. To obtain a computationally feasible solution of the proposed model, reinforcement learning is introduced and knowledge necessary for decision-making is learned through optimization-embedded simulation. Specifically, function approximation techniques and Bayesian inference are performed to learn the value function that quantifies the long-term value of the current system state and to statistically improve the beliefs on the incomplete model parameters. The usefulness of the developed multi-scale decision-making model is proven through the following case studies that are motivated by real industrial problems: 1) integrated procurement planning and scheduling considering trade-offs between multiple suppliers and uncertainties in supply and demand, 2) operational planning and optimal sizing of microgrid considering multi-scale uncertainty in wind supply, and 3) crude procurement and refinery operational planning considering variations in crude/product price and randomness in crude quality.