Clocking up 20 hours a week of TV time appears to be detrimental, the US authors from Harvard say in the British Journal of Sports Medicine.
Yet 15 hours or more of exercise a week boosts semen quality, according to the results in nearly 200 college students.
The researchers said more studies were needed to explore the possible causes.
And some experts say men wanting to conceive need to be selective about the sport they do as some types may harm sperm.
Too much time riding a bike or doing long-distance running in tight clothing may not be good, other studies suggest.
Similarly, wearing tight underwear rather than boxer shorts has been linked with lower sperm levels.
In the latest study, the researchers from the Harvard School of Public Health asked 189 young men who were students at a university in New York to record how many hours they had been spending doing physical activity and watching TV in a typical week.
The volunteers, all aged between 18 and 22, were also asked to provide a sperm sample for lab analysis.
When the researchers compared the survey findings with the sperm test results they found the link between sedentary lifestyle and low sperm count.
Men who were the most physically active, doing 15 hours or more of moderate to vigorous exercise each week by playing football, baseball or basketball for example, had sperm counts which were 73% higher than those who were least physically active.
Those who spent lots of time watching TV or DVDs - at least 20 hours a week - had a sperm count that was 44% lower than men who spent little time in front of the box.
None of the men had sperm counts so low that doctors would classify them as sub-fertile.
Sperm motility (how well it swims) and shape were unaffected.
The researchers say their findings are not conclusive but suggest that men who want to improve their fertility might want to look at increasing the amount of physical exercise they do if they currently do little.
It is not clear why sitting on the sofa watching TV might lower sperm count. It might be down to temperature - sperm prefer cooler conditions and production halts if the scrotum gets too hot.
Obesity may also affect fertility - although most of the men were not overweight.
Dr Audrey Jane Gaskins and colleagues say more studies are needed to confirm and explore the possible causes behind their findings.
Dr Allan Pacey, senior lecturer in andrology at the University of Sheffield, said: "It remains to be seen if coaxing a TV-watching couch potato into doing some regular exercise could actually improve his sperm count. Or whether there exists an unknown fundamental difference between men who like exercise and those who do not which might account for the findings.
"This should be a relatively easy study to perform, but before all worried men hunt for their sports bag it's important to note that other research suggests that doing too much exercise can be harmful to sperm production.
"My advice would be everything in moderation - and that includes time in the gym as well as watching TV."
Men who do little exercise and spend much of their spare time watching TV have lower sperm counts than more active men, a study suggests.