Establishing early warning systems for anthrax attacks is crucial in biodefense. Despite numerous studies for decades, the limited sensitivity of conventional biochemical methods essentially requires preprocessing steps and thus has limitations to be used in realistic settings of biological warfare. We present an optical method for rapid and label-free screening of Bacillus anthracis spores through the synergistic application of holographic microscopy and deep learning. A deep convolutional neural network is designed to classify holographic images of unlabeled living cells. After training, the network outperforms previous techniques in all accuracy measures, achieving single-spore sensitivity and subgenus specificity. The unique "representation learning" capability of deep learning enables direct training from raw images instead of manually extracted features. The method automatically recognizes key biological traits encoded in the images and exploits them as fingerprints. This remarkable learning ability makes the proposed method readily applicable to classifying various single cells in addition to B. anthracis, as demonstrated for the diagnosis of Listeria monocytogenes, without any modification. We believe that our strategy will make holographic microscopy more accessible to medical doctors and biomedical scientists for easy, rapid, and accurate point-of-care diagnosis of pathogens.