I'm running a python file every minute using a cron job. It queries a site and gather's information, but it has to load through 4-5 pages before it gets to the data I need.
The execution time is around 5-10s per query.
I'm wondering if there's a difference in server load if the file is being run congruently multiple times verses having 3 different files assigned to load sections.
Example:
test1.py loads information between A-H
test2.py loads information between I-Q
test3.py loads information between R-Z
If someone requests information about a "B", "M", and "S" topic each file would run and return the results, verses one file test.py running a loop to return all three results.
P.S. I'm asking because I'm expecting in the future that people will request information about 2-6 topics, and that's just one person. So I don't want one file running for 60 seconds straight. I'm wondering if it'll alleviate load to spread it across multiple files.
P.P.S. Also I'm wondering the implications of using python vs php.
Using multiple files to fetch smaller parts probably won't make a difference in server load (well, in fact it'd make the load x times bigger for x times shorter period of time, but the overall result is the same), but it should fetch the data faster (thanks to multithreading and paralleling the requests) therefore reducing your response times.
Server load comparison