Brown adipose tissue can expend large amounts of energy, and therefore increasing its size or activity is a promising therapeutic approach to combat metabolic disease. In humans, major deposits of brown fat cells are found intimately associated with large blood vessels, corresponding to perivascular adipose tissue (PVAT). However, the cellular origins of PVAT are poorly understood. Here, we determine the identity of perivascular adipocyte progenitors in mice and humans. In mice, thoracic PVAT develops from a fibroblastic lineage, consisting of progenitor cells (Pdgfra+, Ly6a+ and Pparg-) and preadipocytes (Pdgfra+, Ly6a+ and Pparg+), which share transcriptional similarity with analogous cell types in white adipose tissue. Interestingly, the aortic adventitia of adult animals contains a population of adipogenic smooth muscle cells (Myh11+, Pdgfra- and Pparg+) that contribute to perivascular adipocyte formation. Similarly, human PVAT contains presumptive fibroblastic and smooth muscle-like adipocyte progenitor cells, as revealed by single-nucleus RNA sequencing. Together, these studies define distinct populations of progenitor cells for thermogenic PVAT, providing a foundation for developing strategies to augment brown fat activity.