Sperm chromatin retains small amounts of histones, and chromatin states of sperm mirror gene expression programs of the next generation. However, it remains largely unknown how paternal epigenetic information is transmitted through sperm chromatin. Here, we present a novel mouse model of paternal epigenetic inheritance, in which deposition of Polycomb repressive complex 2 (PRC2) mediated-repressive H3K27me3 is attenuated in the paternal germline. By applying modified methods of assisted reproductive technology using testicular sperm, we rescued infertility of mice missing Polycomb protein SCML2, which regulates germline gene expression by establishing H3K27me3 on bivalent promoters with other active marks H3K4me2/3. We profiled epigenomic states (H3K27me3 and H3K4me3) of testicular sperm and epididymal sperm, demonstrating that the epididymal pattern of the sperm epigenome is already established in testicular sperm and that SCML2 is required for this process. In F1 males of X-linked Scml2-knockout mice, which have a wild-type genotype, gene expression is dysregulated in the male germline during spermiogenesis. These dysregulated genes are targets of SCML2-mediated H3K27me3 in F0 sperm. Further, dysregulation of gene expression was observed in the mutant-derived wild-type F1 preimplantation embryos. Together, we present functional evidence that the classic epigenetic regulator Polycomb mediates paternal epigenetic inheritance through sperm chromatin.