The nuclear receptor RORγ is a major driver of autoimmune diseases and certain types of cancer due to its aberrant function in T helper 17 (Th17) cell differentiation and tumor cholesterol metabolism, respectively. Compound screening using the classic receptor-coactivator interaction perturbation scheme led to identification of many small-molecule modulators of RORγ(t). We report here that inverse agonists/antagonists of RORγ such as VTP-43742 derivative VTP-23 and TAK828F, which can potently inhibit the inflammatory gene program in Th17 cells, unexpectedly lack high potency in inhibiting the growth of TNBC tumor cells. In contrast, antagonists such as XY018 and GSK805 that strongly suppress tumor cell growth and survival display only modest activities in reducing Th17-related cytokine expression. Unexpectedly, we found that VTP-23 significantly induces the cholesterol biosynthesis program in TNBC cells. Our further mechanistic analyses revealed that the VTP inhibitor enhances the local chromatin accessibility, H3K27ac mark and the cholesterol master regulator SREBP2 recruitment at the RORγ binding sites, whereas XY018 exerts the opposite activities, despite their similar effects on circadian rhythm program. Similar distinctions between TAK828F and SR2211 in their effects on local chromatin structure at Il17 genes were also observed. Together, our study shows for the first-time that structurally distinct RORγ antagonists possess different or even contrasting activities in tissue/cell-specific manner. Our findings also highlight that the activities at natural chromatin are key determinants of RORγ modulators' tissue selectivity.