At the kilo- to megabase pair scales, eukaryotic genomes are partitioned into self-interacting modules or topologically associated domains (TADs) that associate to form nuclear compartments. Here, we combine high-content super-resolution microscopies with state-of-the-art DNA-labeling methods to reveal the variability in the multiscale organization of the Drosophila genome. We find that association frequencies within TADs and between TAD borders are below ~10%, independently of TAD size, epigenetic state, or cell type. Critically, despite this large heterogeneity, we are able to visualize nanometer-sized epigenetic domains at the single-cell level. In addition, absolute contact frequencies within and between TADs are to a large extent defined by genomic distance, higher-order chromosome architecture, and epigenetic identity. We propose that TADs and compartments are organized by multiple, small-frequency, yet specific interactions that are regulated by epigenetics and transcriptional state.