I have a python code which is requesting a Rest API. The API has more than 5000+ pages so i tried to request it but always i am getting error at 2000th request.
The error is: "df = pd.json_normalize(json_data["items"])
KeyError: 'items'"
How can i solve this problem ?
P.S. In locally, the code is working clearly.
I found a solution for this problem. Like @elad said, this is not a airflow error. Due to airflow web service and scheduler the system working a little bit slower. So, my token expired while airflow tasks running. I reorganized my code, I generated token in loop with specific conditions such as try-except. Thanks for everything !
Why airflow is returning error while requesting Rest API?