Retinal development requires precise temporal and spatial coordination of cell cycle exit, cell fate specification, cell migration and differentiation. When this process is disrupted, retinoblastoma, a developmental tumor of the retina, can form. Epigenetic modulators are central to precisely coordinating developmental events, and many epigenetic processes have been implicated in cancer. Studying epigenetic mechanisms in development is challenging because they often regulate multiple cellular processes; therefore, elucidating the primary molecular mechanisms involved can be difficult. Here we explore the role of Brg1 (Smarca4) in retinal development and retinoblastoma in mice using molecular and cellular approaches. Brg1 was found to regulate retinal size by controlling cell cycle length, cell cycle exit and cell survival during development. Brg1 was not required for cell fate specification but was required for photoreceptor differentiation and cell adhesion/polarity programs that contribute to proper retinal lamination during development. The combination of defective cell differentiation and lamination led to retinal degeneration in Brg1-deficient retinae. Despite the hypocellularity, premature cell cycle exit, increased cell death and extended cell cycle length, retinal progenitor cells persisted in Brg1-deficient retinae, making them more susceptible to retinoblastoma. ChIP-Seq analysis suggests that Brg1 might regulate gene expression through multiple mechanisms.