Diffraction imaging of cells allows rapid phenotyping by the response of intracellular molecules to coherent illumination. However, its ability to distinguish numerous types of human leukocytes remains to be investigated. Here, we show that accurate classification of three lymphocyte subtypes can be achieved with features extracted from cross-polarized diffraction image (p-DI) pairs. A deep neural network (DNN) of DINet-PS has been developed for feature extraction from and filtering of, in the angular frequency domain, p-DI pairs acquired from live lymphocytes isolated from human spleen tissues. We built the network in a dual-channel structure and incorporated two adaptive spectral filter blocks to actively suppress extracted features related to the noise component of light in p-DI pairs. The DINet-PS was trained with p-DI pairs acquired from 5311 CD4+ T, 3819 CD8+ T, and 4054 CD19+ B cells after preprocessing and rebelling of manually derived secondary labels and classification accuracy of 96.6 ± 0.40% has been achieved in hold-out test data sets among the three subtypes. Our results show the power of DNN to extract cell-related features from p-DI pairs and the potential of polarization diffraction imaging flow cytometry for accurate and label-free classification of lymphocyte subtypes in particular and leukocytes in general.