In this paper, we propose a novel energy efficient solution for TDM Passive Optical Networks (TDM-PONs), which is a widely used access network technology, to facilitate local customer internetworking. Here, our prime objectives are to ensure traffic latency among different customers connected through TDM-PON and maximize energy saving performance of Optical Network Units (ONUs), which are TDM-PON equipment situated at customer premises. To do so, we devise a novel sleep management protocols for ONUs. In our proposal, when some ONUs need to exchange traffic, they maintain sleep mode as a group in order to reduce traffic latency among the customers attached with those ONUs. To date, considering traffic forwarding between access and core network entities, researchers have come up with several solutions to manage sleep mode in ONUs in order to improve PON energy saving. However, to the best of our knowledge, how ONUs can manage sleep mode and at the same time satisfy latency requirements while managing data flows among them has been paid little attention. The effectiveness of our solution is validated and compared with other alternative approach by simulation. Simulation results demonstrate that proposed solution reduces inter-ONU traffic delay noticeably compared to other solution while providing satisfactory energy saving performance of ONUs.