Tumor invasion into draining lymph nodes, especially sentinel lymph nodes (SLNs), is a key determinant of prognosis and treatment in breast cancer as part of the TNM staging system. Using multicolor histology and quantitative image analysis, we quantified immune cells within SLNs from a discovery cohort of 76 breast cancer patients. We found statistically more in situ CD3+ T cells in tumor negative vs. tumor positive nodes (mean of 8878 vs. 6704, respectively, p = 0.006), but no statistical difference in CD20+ B cells or CD1a+ dendritic cells. In univariate analysis, a reduced hazard was seen with a unit increase in log CD3 with HR 0.49 (95% CI 0.30-0.80) and log CD20 with HR 0.37 (95% CI 0.22-0.62). In multivariate analysis, log CD20 remained significant with HR 0.42 (95% CI 0.25-0.69). When restricted to SLN tumor negative patients, increased log CD20 was still associated with improved DFS (HR = 0.26, 95% CI 0.08-0.90). The CD20 results were validated in a separate cohort of 21 patients (n = 11 good outcome, n = 10 poor outcome) with SLN negative triple-negative breast cancer (TNBC) ("good" mean of 7011 vs. "poor" mean of 4656, p = 0.002). Our study demonstrates that analysis of immune cells within SLNs, regardless of tumor invasion status, may provide additional prognostic information, and highlights B cells within SLNs as important in preventing future recurrence.