Title: To comply with the new European legislation many websites put a GDPR &#x2F; cookie consent notice in front of their websites. There are different implementations of this. While some are only implemented as modal covering the website or bar on the bottom of the screens (in both cases right next to the original content), other implementations redirect the user to a totally different (sub-)domain or even hijack the request and show the consent form instead of the requested content (on the same URL with a 200 status code).<p>The latter ones present a issue to my crawler. I cannot access the content of the page without accepting those notices.<p>Things I&#x27;m considering to bypass those notices:<p>* US IP address (easy to implement, but some websites also display those notices to US IP&#x27;s)<p>* Heuristics to detect those notices and accept them programatically (takes some time to implement - while a couple of vendors (i.e. OneTrust) offer off-the-shelf solutions which are easy to identify and automate, there are also many custom made solutions, so the system would need understand the concept of a consent form and how to bypass it - some forms only require the press of the right button, others involve checkboxes&#x2F;radio buttons). To collect test data one solution might be to visit a set of websites once with an US IP, once with an EU IP and&#x2F;or with different user agents (browser or googlebot).<p>Do you have any ideas how to approach this problem? Or are you even utilizing some techniques already and are willing to share them?
Upvote: