This study focuses on the linkage between decision layers that have different time scales. The resulting expansion of the boundary of decision-making process can provide more robust and flexible management and operation strategies by resolving inconsistencies between different levels. For this, we develop a multi-timescale decision-making model that combines Markov decision process (MDP) and mathematical programming (MP) in a complementary way and introduce a computationally tractable solution algorithm based on reinforcement learning (RL) to solve the MP-embedded MDP problem. To support the integration of the decision hierarchy, a data-driven uncertainty prediction model is suggested which is valid across all time scales considered. A practical example of refinery procurement and production planning is presented to illustrate the proposed method, along with numerical results of a benchmark case study. (C) 2018 Elsevier Ltd. All rights reserved.