Precise evaluation of breast tumor malignancy based on tissue calcifications has important practical value in the disease diagnosis, as well as the understanding of tumor development. Traditional X-ray mammography provides the overall morphologies of the calcifications but lacks intrinsic chemical information. In contrast, spontaneous Raman spectroscopy offers detailed chemical analysis but lacks the spatial profiles. Here, we applied hyperspectral stimulated Raman scattering (SRS) microscopy to extract both the chemical and morphological features of the microcalcifications, based on the spectral and spatial domain analysis. A total of 211 calcification sites from 23 patients were imaged with SRS, and the results were analyzed with a support vector machine (SVM) based classification algorithm. With optimized combinations of chemical and geometrical features of microcalcifications, we were able to reach a precision of 98.21% and recall of 100.00% for classifying benign and malignant cases, significantly improved from the pure spectroscopy or imaging based methods. Our findings may provide a rapid means to accurately evaluate breast tumor malignancy based on fresh tissue biopsies.