We consider a 5G small cell model where the cells are deployed in a way to cover users in the neighbor cells in order to introduce cell active-sleep control, so-called inter-cell power management (IPM). Using a Markov decision process (MDP), one can obtain an optimal policy for IPM, but with limited scalability. This paper proposes a modular IPM strategy which is a suboptimal policy to group only a few neighbor cells. The modular IPM strategy can mitigate the computational complexity of MDP while it achieves a near-optimal solution to save the energy usage in small cells. It is verified to be a feasible solution through the memory usage estimation and simulation study. The modular IPM strategy with fine-step quantized states can save even more power than the non-modular strategy with coarsestep quantized states. Consequently, the proposed modular IPM strategy becomes much favored in the base station management for power saving in a large small-cell cluster.