Leisure time physical activity (LTPA) is strongly associated with socioeconomic position (SEP). Few studies have investigated if demanding occupational physical activity (OPA) could impede a physically active lifestyle in low SEP groups. The aim of this study was to investigate the association between OPA and LTPA among low SEP men and women. We used cross-sectional data from 895 low SEP workers who wore accelerometers for 1⁻5 consecutive workdays. The associations between the relative importance of activities performed during work and leisure time were assessed using compositional regression models stratified on sex. Compositional isotemporal substitution models were used to assess the implication of increasing occupational walking, standing, or sitting on LTPA. We found dissimilarity in LTPA between the sexes, with men spending more waking leisure time sedentary than women (men ~67%, women ~61%), suggesting women performed more household tasks. In men, the associations between OPA and LTPA were weak. In women, the strongest association was observed between the relative importance of occupational walking and leisure time standing (β = -0.16; p = 0.01), where reallocating 15 min work time to occupational walking showed an expected decrease in leisure time standing of 7 min. If this time was spent on additional sedentary leisure time, it could have adverse health consequences.