Understanding the impact of genetic alterations on epigenomic phenotypes during breast cancer progression is challenging with unimodal measurements. Here, we report wellDA-seq, the first high-genomic resolution, high-throughput method that can simultaneously measure the whole genome and chromatin accessibility profiles of thousands of single cells. Using wellDA-seq, we profiled 22,123 single cells from 2 normal and 9 tumors breast tissues. By directly mapping the epigenomic phenotypes to genetic lineages across cancer subclones, we found evidence of both genetic hardwiring and epigenetic plasticity. In 6 estrogen-receptor positive breast cancers, we directly identified the ancestral cancer cells, and found that their epithelial cell-of-origin was Luminal Hormone Responsive cells. We also identified cell types with copy number aberrations (CNA) in normal breast tissues and discovered non-epithelial cell types in the microenvironment with CNAs in breast cancers. These data provide insights into the complex relationship between genetic alterations and epigenomic phenotypes during breast tumor evolution.