Persistent organic pollutants (POPs) can disrupt the thyroid hormone system in humans. We assessed the associations of several POPs with serum thyroid hormones (T3 and T4) and thyroid-stimulating hormone, and investigated the modulating effects of sex, menopausal status, and age on these associations, in a subgroup of the adult population (n = 1250) from the Korean National Environmental Health Survey. PCB105 and PCB118 were negatively associated with total T4 in premenopausal females and males aged <50, whereas the associations were insignificant in other groups. PCB180, p,p'-DDE, and p,p'-DDT showed positive associations with total T3 in postmenopausal females; however, among males aged ≥50, PCB118, PCB138, and p,p'-DDE showed negative associations with total T3. The effects of exposure to multiple POPs were examined in multi-factor analyses. Factor 2 comprised PCB52, hexachlorobenzene, and BDE-47 was associated with an increase in free T4 in premenopausal females (β = 0.015, p = 0.024), while Factor 1, which contained most POPs, was associated with a change in total T3 in postmenopausal females (β = 0.032, p = 0.040) and males aged ≥50 (β = -0.039, p = 0.023). Changes in total T4 or total T3 could be explained by differences in thyroxine-binding globulin (TBG) and peripheral deiodinase activity (G_D). Negative associations of TBG with PCB105 in premenopausal females and PCB153 in males aged <50 may mediate the effect of decreasing total T4. PCB180, p,p'-DDE, p,p'-DDT, and Factor 1 were positively associated with G_D, which is consistent with an increased total T3 in postmenopausal females. PCB118 was negatively associated with G_D and total T3 in males aged ≥50. BDE-47 and β-hexachlorocyclohexane were associated with thyroid autoantibodies in premenopausal females and males aged <50. Our observations suggest that the thyroid-disrupting effects of POPs may differ by sex, sex hormonal status, and age, and may be mediated by TBG and G_D.