We proposed a method of label-free segmentation of cell nuclei by exploiting a deep learning (DL) framework. Over the years, fluorescent proteins and staining agents have been widely used to identify cell nuclei. However, the use of exogenous agents inevitably prevents from long-term imaging of live cells and rapid analysis and even interferes with intrinsic physiological conditions. Without any agents, the proposed method was applied to label-free optical diffraction tomography (ODT) of human breast cancer cells. A novel architecture with optimized training strategies was validated through cross-modality and cross-laboratory experiments. The nucleus volumes from the DL-based label-free ODT segmentation accurately agreed with those from fluorescent-based. Furthermore, the 4D cell nucleus segmentation was successfully performed for the time-lapse ODT images. The proposed method would bring out broad and immediate biomedical applications with our framework publicly available.