Google Oauth2 Contacts API returns Invalid token: Stateless token expired after an hour
See, when you are logging into the website, you are probably using cookies. So basically you might be using the same session and actually the api is not called.
The time when you are logging in incognito mode or in a diffrent browser, that cookie cannot be used, so this time api is called. For this reason, the token is getting changed. 
For example, if after few users have signed up with google, you change the scope of the app, what happens is, if the user has enabled cookies and it has not expired, when he visits your site, it simply logs him in. It does not asks for permissions (that you added recently to scope). But when he logs out and logs in again, then it asks for the additional permission and th token also gets changed.
What you should do is, you should go through th codes of django-allauth and clear it out how they are using the token. You must also know that for getting refresh token, you must have offline access enabled in your configuration.
What's wrong with my setup?
I am using django-allauth for social signup and recently i added contacts to it's scope. Things are working fine. It now asks for permission to manage contacts and I am able to get contact details of users through the API.
But once i make a request to get contacts of a user(I am not saving any refresh token or accss token at that time), after an hour when i make the request again with same token, It shows this error "Invalid token: Stateless token expired".
However I can still login into the website and the token does not change. However when I logout and login again the token changes and i can again get the contacts using that token for one hour.
What's the issue? What am I missing?