Heterochromatic loci can exhibit different transcriptional states in genetically identical cells. A popular model posits that the inheritance of modified histones is sufficient for inheritance of the silenced state. However, silencing inheritance requires silencers and therefore cannot be driven by the inheritance of modified histones alone. To address these observations, we determined the chromatin architectures produced by strong and weak silencers in Saccharomyces. Strong silencers recruited Sir proteins and silenced the locus in all cells. Strikingly, weakening these silencers reduced Sir protein recruitment and stably silenced the locus in some cells; however, this silenced state could probabilistically convert to an expressed state that lacked Sir protein recruitment. Additionally, changes in the constellation of silencer-bound proteins or the concentration of a structural Sir protein modulated the probability that a locus exhibited the silenced or expressed state. These findings argued that distinct silencer states generate epigenetic states and regulate their dynamics.