Multi-timescale, multi-period decision-making model development by combining reinforcement learning and mathematical programming

Abstract

This study focuses on the linkage between decision layers that have different time scales. The resulting expansion of the boundary of decision-making process can provide more robust and flexible management and operation strategies by resolving inconsistencies between different levels. For this, we develop a multi-timescale decision-making model that combines Markov decision process (MDP) and mathematical programming (MP) in a complementary way and introduce a computationally tractable solution algorithm based on reinforcement learning (RL) to solve the MP-embedded MDP problem. To support the integration of the decision hierarchy, a data-driven uncertainty prediction model is suggested which is valid across all time scales considered. A practical example of refinery procurement and production planning is presented to illustrate the proposed method, along with numerical results of a benchmark case study. (C) 2018 Elsevier Ltd. All rights reserved.