Fetal growth plays a role in programming of adult cardiometabolic disorders, which in men, are associated with lowered testosterone levels. Fetal growth and fetal androgen exposure can also predetermine testosterone levels in men, although how is unknown, because the adult Leydig cells (ALCs) that produce testosterone do not differentiate until puberty. To explain this conundrum, we hypothesized that stem cells for ALCs must be present in the fetal testis and might be susceptible to programming by fetal androgen exposure during masculinization. To address this hypothesis, we used ALC ablation/regeneration to identify that, in rats, ALCs derive from stem/progenitor cells that express chicken ovalbumin upstream promoter transcription factor II. These stem cells are abundant in the fetal testis of humans and rodents, and lineage tracing in mice shows that they develop into ALCs. The stem cells also express androgen receptors (ARs). Reduction in fetal androgen action through AR KO in mice or dibutyl phthalate (DBP) -induced reduction in intratesticular testosterone in rats reduced ALC stem cell number by ∼40% at birth to adulthood and induced compensated ALC failure (low/normal testosterone and elevated luteinizing hormone). In DBP-exposed males, this failure was probably explained by reduced testicular steroidogenic acute regulatory protein expression, which is associated with increased histone methylation (H3K27me3) in the proximal promoter. Accordingly, ALCs and ALC stem cells immunoexpressed increased H3K27me3, a change that was also evident in ALC stem cells in fetal testes. These studies highlight how a key component of male reproductive development can fundamentally reprogram adult hormone production (through an epigenetic change), which might affect lifetime disease risk.