title: Web scrape an embedded table in google tag manager's datalayer?
body: <p>I am trying to web scrape table from <a href="https://data.oecd.org/migration/foreign-born-population.htm" rel="nofollow noreferrer">OECD Data</a>. I am learning to use Python and BeautifulSoup in Jupyter to achieve this.</p>
  <p>The closest similar post that I've found is <a href="https://stackoverflow.com/questions/58791812/how-to-web-scrape-tables-embedded-in-websites-using-python">How to web scrape tables embedded in websites using Python</a>. In my case, the scraped object has a nested JSON object that doesn't match with what I inspect from the page.</p>
  <h2>Scraped object</h2>
  <pre><code>&lt;!DOCTYPE html&gt;
  &lt;html lang=&quot;en&quot;&gt;&lt;head&gt;
      &lt;script&gt;
          dataLayer = [{
              'siteName':'data',
              'siteEnvironment':'live',
              'pageLanguage':'en',
              'pageTopic':'',
              'pageSubTopic':'',
              'pageCategory':'oecdChart',
              'indicatorCode':'FBORNPOP'
          }];
      &lt;/script&gt;
      &lt;title&gt;Foreign-born population, Total, % of population, 2013 – 2018&lt;/title&gt;
      &lt;meta content=&quot;IE=edge&quot; http-equiv=&quot;X-UA-Compatible&quot;/&gt;
      &lt;meta content=&quot;noindex&quot; name=&quot;robots&quot;/&gt;
      &lt;link href=&quot;//data.oecd.org/media/dataportal/stylesheets/main.min.css&quot; rel=&quot;stylesheet&quot; type=&quot;text/css&quot;/&gt;
      &lt;link href=&quot;//data.oecd.org/media/oecdorg/styleassets/css/responsive/bootstrap.min.css&quot; rel=&quot;stylesheet&quot; type=&quot;text/css&quot;/&gt;
      &lt;meta content=&quot;summary_large_image&quot; name=&quot;twitter:card&quot;/&gt;
      &lt;meta content=&quot;@OECD&quot; name=&quot;twitter:site&quot;/&gt;
      &lt;meta content=&quot;@OECD&quot; name=&quot;twitter:creator&quot;/&gt;
      &lt;meta content=&quot;Interactive charts by the OECD&quot; name=&quot;twitter:title&quot;/&gt;
      &lt;meta content=&quot;Foreign-born population, Total, % of population, 2013 – 2018&quot; name=&quot;twitter:description&quot;/&gt;
      &lt;meta content=&quot;https://data.oecd.org/custom/share/txqy.png&quot; name=&quot;twitter:image:src&quot;/&gt;
      &lt;meta content=&quot;website&quot; property=&quot;og:type&quot;/&gt;
      &lt;meta content=&quot;OECD Data&quot; property=&quot;og:site_name&quot;/&gt;
      &lt;meta content=&quot;https://data.oecd.org/chart/5Rmq&quot; property=&quot;og:url&quot;/&gt;
      &lt;meta content=&quot;Interactive charts by the OECD&quot; property=&quot;og:title&quot;/&gt;
      &lt;meta content=&quot;Foreign-born population, Total, % of population, 2013 – 2018&quot; property=&quot;og:description&quot;/&gt;
      &lt;meta content=&quot;https://data.oecd.org/custom/share/txqy.png&quot; property=&quot;og:image&quot;/&gt;
      &lt;meta content=&quot;en_GB&quot; property=&quot;og:locale&quot;/&gt;
      &lt;meta content=&quot;fr_FR&quot; property=&quot;og:locale:alternate&quot;/&gt;
      &lt;meta content=&quot;embedded&quot; itemprop=&quot;context&quot;/&gt;
  &lt;/head&gt;
  &lt;body&gt;
      &lt;!-- New Google Tag Manager --&gt;
      &lt;noscript&gt;&lt;iframe height=&quot;0&quot; src=&quot;https://www.googletagmanager.com/ns.html?id=GTM-P5JSM4P&quot; style=&quot;display:none;visibility:hidden&quot; width=&quot;0&quot;&gt;&lt;/iframe&gt;&lt;/noscript&gt;
      &lt;script&gt;
      (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':
              new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],
              j=d.createElement(s),dl=l!='dataLayer'?'&amp;l='+l:'';j.async=true;j.src=
              'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);
      })(window, document, 'script', 'dataLayer', 'GTM-P5JSM4P');
      &lt;/script&gt;
      &lt;!-- End New Google Tag Manager --&gt;
  
  &lt;!-- Google Tag Manager --&gt;
  &lt;noscript&gt;
      &lt;iframe height=&quot;0&quot; src=&quot;//www.googletagmanager.com/ns.html?id=GTM-WZ3XJR&quot; style=&quot;display: none; visibility: hidden&quot; width=&quot;0&quot;&gt;&lt;/iframe&gt;
  &lt;/noscript&gt;
  &lt;script&gt;
      (function(w, d, s, l, i) {
          w[l] = w[l] || [];
          w[l].push({
              'gtm.start':
                  new Date().getTime(),
              event: 'gtm.js'
          });
          var f = d.getElementsByTagName(s)[0],
              j = d.createElement(s),
              dl = l != 'dataLayer' ? '&amp;l=' + l : '';
          j.async = true;
          j.src =
              '//www.googletagmanager.com/gtm.js?id=' + i + dl;
          f.parentNode.insertBefore(j, f);
      })(window, document, 'script', 'dataLayer', 'GTM-WZ3XJR');
  &lt;/script&gt;
  &lt;!-- End Google Tag Manager --&gt;
  &lt;div class=&quot;ddp-root&quot;&gt;
      &lt;div class=&quot;embedded-chart&quot;&gt;
          &lt;div class=&quot;ddp-chart&quot; data-baseline-comparison=&quot;&quot; data-compact-header=&quot;false&quot; data-data='{&quot;header&quot;:{&quot;id&quot;:&quot;6bda3584-cfa5-46cb-afd3-d2512d0d52f8&quot;,&quot;test&quot;:false,&quot;prepared&quot;:&quot;2020-03-01T09:16:26.3887278Z&quot;,&quot;sender&quot;:{&quot;id&quot;:&quot;OECD&quot;,&quot;name&quot;:&quot;Organisation for Economic Co-operation and Development&quot;},&quot;links&quot;:[{&quot;href&quot;:&quot;https://stats.oecd.org:443/sdmx-json/data/DP_LIVE/.FBORNPOP.TOT.PC_POP.A/OECD?json-lang=en&amp;amp;dimensionAtObservation=allDimensions&amp;amp;startPeriod=2013&amp;amp;endPeriod=2018&quot;,&quot;rel&quot;:&quot;request&quot;}]},&quot;dataSets&quot;:[{&quot;action&quot;:&quot;Information&quot;,&quot;observations&quot;:{&quot;0:0:0:0:0:0&quot;:[27.54,0,null],&quot;0:0:0:0:0:1&quot;:[27.814,0,null],&quot;0:0:0:0:0:2&quot;:[28.077,0,null],&quot;0:0:0:0:0:3&quot;:[28.65,0,null],&quot;0:0:0:0:0:4&quot;:[29.2,0,null],&quot;0:0:0:0:0:5&quot;:[29.641,0,null],&quot;1:0:0:0:0:0&quot;:[15.676,0,null],&quot;1:0:0:0:0:1&quot;:[15.816,0,null],&quot;1:0:0:0:0:2&quot;:[15.807,0,null],&quot;1:0:0:0:0:3&quot;:[16.281,0,null],&quot;1:0:0:0:0:4&quot;:[16.566,0,null],&quot;1:0:0:0:0:5&quot;:[16.809,0,null],&quot;2:0:0:0:0:0&quot;:[19.952,0,null],&quot;2:0:0:0:0:1&quot;:[20.108,0,null],&quot;2:0:0:0:0:2&quot;:[20.275,0,null],&quot;2:0:0:0:0:3&quot;:[20.779,0,null],&quot;3:0:0:0:0:0&quot;:[8.115,0,null],&quot;3:0:0:0:0:1&quot;:[8.43,0,null],&quot;3:0:0:0:0:2&quot;:[8.838,0,null],&quot;3:0:0:0:0:3&quot;:[9.463,0,null],&quot;3:0:0:0:0:4&quot;:[9.952,0,null],&quot;3:0:0:0:0:5&quot;:[10.282,0,null],&quot;4:0:0:0:0:0&quot;:[5.235,0,null],&quot;4:0:0:0:0:1&quot;:[5.553,0,null],&quot;4:0:0:0:0:2&quot;:[5.85,0,null],&quot;4:0:0:0:0:3&quot;:[6.127,0,null],&quot;4:0:0:0:0:4&quot;:[6.473,0,null],&quot;4:0:0:0:0:5&quot;:[6.726,0,null],&quot;5:0:0:0:0:0&quot;:[12.539,0,null],&quot;5:0:0:0:0:1&quot;:[12.976,0,null],&quot;5:0:0:0:0:2&quot;:[13.45,0,null],&quot;5:0:0:0:0:3&quot;:[13.982,0,null],&quot;5:0:0:0:0:4&quot;:[15.513,0,null],&quot;5:0:0:0:0:5&quot;:[16.006,0,null],&quot;6:0:0:0:0:0&quot;:[4.274,0,null],&quot;6:0:0:0:0:1&quot;:[4.527,0,null],&quot;6:0:0:0:0:2&quot;:[4.831,0,null],&quot;6:0:0:0:0:3&quot;:[5.171,0,null],&quot;6:0:0:0:0:4&quot;:[5.289,0,null],&quot;6:0:0:0:0:5&quot;:[5.534,0,null],&quot;7:0:0:0:0:0&quot;:[41.495,0,null],&quot;7:0:0:0:0:1&quot;:[42.707,0,null],&quot;7:0:0:0:0:2&quot;:[43.887,0,null],&quot;7:0:0:0:0:3&quot;:[45.258,0,null],&quot;7:0:0:0:0:4&quot;:[46.404,0,null],&quot;7:0:0:0:0:5&quot;:[47.563,0,null],&quot;8:0:0:0:0:0&quot;:[0.787,0,null],&quot;8:0:0:0:0:1&quot;:[0.79,0,null],&quot;8:0:0:0:0:2&quot;:[0.74,0,null],&quot;8:0:0:0:0:3&quot;:[0.79,0,null],&quot;9:0:0:0:0:0&quot;:[11.468,0,null],&quot;9:0:0:0:0:1&quot;:[11.581,0,null],&quot;9:0:0:0:0:2&quot;:[11.795,0,null],&quot;9:0:0:0:0:3&quot;:[12.106,0,null],&quot;9:0:0:0:0:4&quot;:[12.545,0,null],&quot;9:0:0:0:0:5&quot;:[12.97,0,null],&quot;10:0:0:0:0:0&quot;:[13.059,0,null],&quot;10:0:0:0:0:1&quot;:[13.685,0,null],&quot;10:0:0:0:0:2&quot;:[14.236,0,null],&quot;10:0:0:0:0:3&quot;:[14.701,0,null],&quot;10:0:0:0:0:4&quot;:[15.075,0,null],&quot;10:0:0:0:0:5&quot;:[15.362,0,null],&quot;11:0:0:0:0:0&quot;:[8.628,0,null],&quot;11:0:0:0:0:1&quot;:[8.455,0,null],&quot;11:0:0:0:0:2&quot;:[8.555,0,null],&quot;11:0:0:0:0:3&quot;:[8.613,0,null],&quot;12:0:0:0:0:0&quot;:[15.308,0,null],&quot;12:0:0:0:0:1&quot;:[15.804,0,null],&quot;12:0:0:0:0:2&quot;:[16.397,0,null],&quot;12:0:0:0:0:3&quot;:[17.039,0,null],&quot;12:0:0:0:0:4&quot;:[18.006,0,null],&quot;12:0:0:0:0:5&quot;:[18.803,0,null],&quot;13:0:0:0:0:0&quot;:[27.325,0,null],&quot;13:0:0:0:0:1&quot;:[27.883,0,null],&quot;13:0:0:0:0:2&quot;:[28.376,0,null],&quot;13:0:0:0:0:3&quot;:[28.761,0,null],&quot;13:0:0:0:0:4&quot;:[29.259,0,null],&quot;13:0:0:0:0:5&quot;:[29.483,0,null],&quot;14:0:0:0:0:0&quot;:[12.29,0,null],&quot;14:0:0:0:0:1&quot;:[12.535,0,null],&quot;14:0:0:0:0:2&quot;:[13.107,0,null],&quot;14:0:0:0:0:3&quot;:[13.662,0,null],&quot;14:0:0:0:0:4&quot;:[14.157,0,null],&quot;14:0:0:0:0:5&quot;:[13.794,0,null],&quot;15:0:0:0:0:0&quot;:[12.846,0,null],&quot;15:0:0:0:0:1&quot;:[12.942,0,null],&quot;15:0:0:0:0:2&quot;:[13.174,0,null],&quot;15:0:0:0:0:3&quot;:[13.436,0,null],&quot;15:0:0:0:0:4&quot;:[13.481,0,null],&quot;15:0:0:0:0:5&quot;:[13.626,0,null],&quot;16:0:0:0:0:0&quot;:[2.364,0,null],&quot;16:0:0:0:0:1&quot;:[2.486,0,null],&quot;16:0:0:0:0:2&quot;:[2.593,0,null],&quot;17:0:0:0:0:0&quot;:[6.603,0,null],&quot;17:0:0:0:0:1&quot;:[0.0,0,null],&quot;17:0:0:0:0:2&quot;:[6.641,0,null],&quot;17:0:0:0:0:4&quot;:[5.811,0,null],&quot;18:0:0:0:0:0&quot;:[16.039,0,null],&quot;18:0:0:0:0:1&quot;:[16.132,0,null],&quot;18:0:0:0:0:2&quot;:[16.376,0,null],&quot;18:0:0:0:0:3&quot;:[17.148,0,null],&quot;19:0:0:0:0:0&quot;:[22.435,0,null],&quot;19:0:0:0:0:1&quot;:[23.36,0,null],&quot;19:0:0:0:0:2&quot;:[24.478,0,null],&quot;19:0:0:0:0:3&quot;:[25.077,0,null],&quot;20:0:0:0:0:0&quot;:[15.031,0,null],&quot;20:0:0:0:0:1&quot;:[14.934,0,null],&quot;20:0:0:0:0:2&quot;:[14.831,0,null],&quot;20:0:0:0:0:3&quot;:[14.767,0,null],&quot;20:0:0:0:0:4&quot;:[14.701,0,null],&quot;20:0:0:0:0:5&quot;:[15.014,0,null],&quot;21:0:0:0:0:0&quot;:[11.89,0,null],&quot;21:0:0:0:0:1&quot;:[12.032,0,null],&quot;21:0:0:0:0:2&quot;:[12.186,0,null],&quot;21:0:0:0:0:3&quot;:[12.276,0,null],&quot;21:0:0:0:0:4&quot;:[12.476,0,null],&quot;21:0:0:0:0:5&quot;:[12.486,0,null],&quot;22:0:0:0:0:0&quot;:[9.529,0,null],&quot;22:0:0:0:0:1&quot;:[9.596,0,null],&quot;22:0:0:0:0:2&quot;:[9.708,0,null],&quot;22:0:0:0:0:3&quot;:[9.94,0,null],&quot;22:0:0:0:0:4&quot;:[10.199,0,null],&quot;22:0:0:0:0:5&quot;:[10.415,0,null],&quot;23:0:0:0:0:0&quot;:[14.515,0,null],&quot;23:0:0:0:0:1&quot;:[16.023,0,null],&quot;23:0:0:0:0:2&quot;:[16.504,0,null],&quot;23:0:0:0:0:3&quot;:[16.378,0,null],&quot;23:0:0:0:0:4&quot;:[16.844,0,null],&quot;23:0:0:0:0:5&quot;:[17.593,0,null],&quot;24:0:0:0:0:0&quot;:[16.081,0,null],&quot;24:0:0:0:0:1&quot;:[16.61,0,null],&quot;24:0:0:0:0:2&quot;:[17.375,0,null],&quot;24:0:0:0:0:3&quot;:[18.305,0,null],&quot;24:0:0:0:0:4&quot;:[18.96,0,null],&quot;24:0:0:0:0:5&quot;:[19.392,0,null],&quot;25:0:0:0:0:0&quot;:[7.063,0,null],&quot;25:0:0:0:0:1&quot;:[7.161,0,null],&quot;25:0:0:0:0:2&quot;:[7.3,0,null],&quot;25:0:0:0:0:3&quot;:[7.523,0,null],&quot;26:0:0:0:0:0&quot;:[13.292,0,null],&quot;26:0:0:0:0:1&quot;:[12.88,0,null],&quot;26:0:0:0:0:2&quot;:[12.773,0,null],&quot;26:0:0:0:0:3&quot;:[12.769,0,null],&quot;26:0:0:0:0:4&quot;:[12.997,0,null],&quot;26:0:0:0:0:5&quot;:[13.365,0,null],&quot;27:0:0:0:0:0&quot;:[0.0,0,null],&quot;27:0:0:0:0:1&quot;:[1.883,0,null],&quot;27:0:0:0:0:2&quot;:[2.024,0,null],&quot;27:0:0:0:0:3&quot;:[2.235,0,null],&quot;27:0:0:0:0:4&quot;:[2.383,0,null],&quot;27:0:0:0:0:5&quot;:[2.781,0,null],&quot;28:0:0:0:0:0&quot;:[0.0,0,null],&quot;28:0:0:0:0:1&quot;:[0.0,0,null],&quot;28:0:0:0:0:2&quot;:[0.0,0,null],&quot;29:0:0:0:0:0&quot;:[10.891,0,null],&quot;29:0:0:0:0:1&quot;:[11.374,0,null],&quot;29:0:0:0:0:2&quot;:[11.906,0,null],&quot;29:0:0:0:0:3&quot;:[12.639,0,null],&quot;29:0:0:0:0:4&quot;:[13.884,0,null],&quot;29:0:0:0:0:5&quot;:[16.162,0,null],&quot;30:0:0:0:0:0&quot;:[3.186,0,null],&quot;30:0:0:0:0:1&quot;:[3.225,0,null],&quot;30:0:0:0:0:2&quot;:[3.273,0,null],&quot;30:0:0:0:0:3&quot;:[3.336,0,null],&quot;30:0:0:0:0:4&quot;:[3.418,0,null],&quot;30:0:0:0:0:5&quot;:[3.492,0,null],&quot;31:0:0:0:0:0&quot;:[23.472,0,null],&quot;31:0:0:0:0:1&quot;:[22.936,0,null],&quot;31:0:0:0:0:2&quot;:[22.532,0,null],&quot;31:0:0:0:0:3&quot;:[22.187,0,null],&quot;31:0:0:0:0:4&quot;:[21.78,0,null],&quot;32:0:0:0:0:0&quot;:[13.879,0,null],&quot;32:0:0:0:0:1&quot;:[13.629,0,null],&quot;32:0:0:0:0:2&quot;:[13.47,0,null],&quot;32:0:0:0:0:3&quot;:[13.138,0,null],&quot;32:0:0:0:0:4&quot;:[12.898,0,null],&quot;32:0:0:0:0:5&quot;:[12.749,0,null],&quot;33:0:0:0:0:1&quot;:[4.807,0,null],&quot;33:0:0:0:0:2&quot;:[9.251,0,null],&quot;33:0:0:0:0:3&quot;:[4.463,0,null],&quot;33:0:0:0:0:4&quot;:[4.404,0,null],&quot;33:0:0:0:0:5&quot;:[4.554,0,null]}}],&quot;structure&quot;:{&quot;links&quot;:[{&quot;href&quot;:&quot;https://stats.oecd.org/sdmx-json/dataflow/DP_LIVE/OECD&quot;,&quot;rel&quot;:&quot;dataflow&quot;}],&quot;name&quot;:&quot;OECD Data Live dataset&quot;,&quot;description&quot;:&quot;OECD Data Live dataset&quot;,&quot;dimensions&quot;:{&quot;observation&quot;:[{&quot;keyPosition&quot;:0,&quot;id&quot;:&quot;LOCATION&quot;,&quot;name&quot;:&quot;Country&quot;,&quot;values&quot;:[{&quot;id&quot;:&quot;AUS&quot;,&quot;name&quot;:&quot;Australia&quot;},{&quot;id&quot;:&quot;BEL&quot;,&quot;name&quot;:&quot;Belgium&quot;},{&quot;id&quot;:&quot;CAN&quot;,&quot;name&quot;:&quot;Canada&quot;},{&quot;id&quot;:&quot;DNK&quot;,&quot;name&quot;:&quot;Denmark&quot;},{&quot;id&quot;:&quot;FIN&quot;,&quot;name&quot;:&quot;Finland&quot;},{&quot;id&quot;:&quot;DEU&quot;,&quot;name&quot;:&quot;Germany&quot;},{&quot;id&quot;:&quot;HUN&quot;,&quot;name&quot;:&quot;Hungary&quot;},{&quot;id&quot;:&quot;LUX&quot;,&quot;name&quot;:&quot;Luxembourg&quot;},{&quot;id&quot;:&quot;MEX&quot;,&quot;name&quot;:&quot;Mexico&quot;},{&quot;id&quot;:&quot;NLD&quot;,&quot;name&quot;:&quot;Netherlands&quot;},{&quot;id&quot;:&quot;NOR&quot;,&quot;name&quot;:&quot;Norway&quot;},{&quot;id&quot;:&quot;PRT&quot;,&quot;name&quot;:&quot;Portugal&quot;},{&quot;id&quot;:&quot;SWE&quot;,&quot;name&quot;:&quot;Sweden&quot;},{&quot;id&quot;:&quot;CHE&quot;,&quot;name&quot;:&quot;Switzerland&quot;},{&quot;id&quot;:&quot;GBR&quot;,&quot;name&quot;:&quot;United Kingdom&quot;},{&quot;id&quot;:&quot;USA&quot;,&quot;name&quot;:&quot;United States&quot;},{&quot;id&quot;:&quot;CHL&quot;,&quot;name&quot;:&quot;Chile&quot;},{&quot;id&quot;:&quot;GRC&quot;,&quot;name&quot;:&quot;Greece&quot;},{&quot;id&quot;:&quot;IRL&quot;,&quot;name&quot;:&quot;Ireland&quot;},{&quot;id&quot;:&quot;NZL&quot;,&quot;name&quot;:&quot;New Zealand&quot;},{&quot;id&quot;:&quot;EST&quot;,&quot;name&quot;:&quot;Estonia&quot;},{&quot;id&quot;:&quot;FRA&quot;,&quot;name&quot;:&quot;France&quot;},{&quot;id&quot;:&quot;ITA&quot;,&quot;name&quot;:&quot;Italy&quot;},{&quot;id&quot;:&quot;SVN&quot;,&quot;name&quot;:&quot;Slovenia&quot;},{&quot;id&quot;:&quot;AUT&quot;,&quot;name&quot;:&quot;Austria&quot;},{&quot;id&quot;:&quot;CZE&quot;,&quot;name&quot;:&quot;Czech Republic&quot;},{&quot;id&quot;:&quot;ESP&quot;,&quot;name&quot;:&quot;Spain&quot;},{&quot;id&quot;:&quot;TUR&quot;,&quot;name&quot;:&quot;Turkey&quot;},{&quot;id&quot;:&quot;POL&quot;,&quot;name&quot;:&quot;Poland&quot;},{&quot;id&quot;:&quot;ISL&quot;,&quot;name&quot;:&quot;Iceland&quot;},{&quot;id&quot;:&quot;SVK&quot;,&quot;name&quot;:&quot;Slovak Republic&quot;},{&quot;id&quot;:&quot;ISR&quot;,&quot;name&quot;:&quot;Israel&quot;},{&quot;id&quot;:&quot;LVA&quot;,&quot;name&quot;:&quot;Latvia&quot;},{&quot;id&quot;:&quot;LTU&quot;,&quot;name&quot;:&quot;Lithuania&quot;}],&quot;role&quot;:&quot;REF_AREA&quot;},{&quot;keyPosition&quot;:1,&quot;id&quot;:&quot;INDICATOR&quot;,&quot;name&quot;:&quot;Indicator&quot;,&quot;values&quot;:[{&quot;id&quot;:&quot;FBORNPOP&quot;,&quot;name&quot;:&quot;Foreign-born population&quot;}]},{&quot;keyPosition&quot;:2,&quot;id&quot;:&quot;SUBJECT&quot;,&quot;name&quot;:&quot;Subject&quot;,&quot;values&quot;:[{&quot;id&quot;:&quot;TOT&quot;,&quot;name&quot;:&quot;Total&quot;}]},{&quot;keyPosition&quot;:3,&quot;id&quot;:&quot;MEASURE&quot;,&quot;name&quot;:&quot;Measure&quot;,&quot;values&quot;:[{&quot;id&quot;:&quot;PC_POP&quot;,&quot;name&quot;:&quot;% of population&quot;}]},{&quot;keyPosition&quot;:4,&quot;id&quot;:&quot;FREQUENCY&quot;,&quot;name&quot;:&quot;Frequency&quot;,&quot;values&quot;:[{&quot;id&quot;:&quot;A&quot;,&quot;name&quot;:&quot;Annual&quot;}],&quot;role&quot;:&quot;FREQ&quot;},{&quot;id&quot;:&quot;TIME_PERIOD&quot;,&quot;name&quot;:&quot;Time&quot;,&quot;values&quot;:[{&quot;id&quot;:&quot;2013&quot;,&quot;name&quot;:&quot;2013&quot;},{&quot;id&quot;:&quot;2014&quot;,&quot;name&quot;:&quot;2014&quot;},{&quot;id&quot;:&quot;2015&quot;,&quot;name&quot;:&quot;2015&quot;},{&quot;id&quot;:&quot;2016&quot;,&quot;name&quot;:&quot;2016&quot;},{&quot;id&quot;:&quot;2017&quot;,&quot;name&quot;:&quot;2017&quot;},{&quot;id&quot;:&quot;2018&quot;,&quot;name&quot;:&quot;2018&quot;}],&quot;role&quot;:&quot;TIME_PERIOD&quot;}]},&quot;attributes&quot;:{&quot;dataSet&quot;:[],&quot;series&quot;:[],&quot;observation&quot;:[{&quot;id&quot;:&quot;TIME_FORMAT&quot;,&quot;name&quot;:&quot;Time Format&quot;,&quot;values&quot;:[{&quot;id&quot;:&quot;P1Y&quot;,&quot;name&quot;:&quot;Annual&quot;}]},{&quot;id&quot;:&quot;OBS_STATUS&quot;,&quot;name&quot;:&quot;Observation Status&quot;,&quot;values&quot;:[]}]},&quot;annotations&quot;:[{&quot;title&quot;:&quot;Copyright OECD - All rights reserved&quot;,&quot;uri&quot;:&quot;&quot;,&quot;text&quot;:&quot;&quot;},{&quot;title&quot;:&quot;Terms and Conditions&quot;,&quot;uri&quot;:&quot;http://www.oecd.org/termsandconditions/&quot;,&quot;text&quot;:&quot;&quot;},{&quot;title&quot;:&quot;Privacy Policy&quot;,&quot;uri&quot;:&quot;http://www.oecd.org/privacy/&quot;,&quot;text&quot;:&quot;&quot;},{&quot;title&quot;:&quot;MyOECD&quot;,&quot;uri&quot;:&quot;https://www.oecd.org/login&quot;,&quot;text&quot;:&quot;&quot;},{&quot;title&quot;:&quot;Contact Us&quot;,&quot;uri&quot;:&quot;http://www.oecd.org/contact/&quot;,&quot;text&quot;:&quot;&quot;}]}}' data-editable=&quot;false&quot; data-embedded=&quot;true&quot; data-end-time=&quot;2018&quot; data-external-footer-id=&quot;embedded-chart-footer&quot; data-fixed-height=&quot;true&quot; data-frequency=&quot;A&quot; data-highlighted-locations=&quot;&quot; data-indicator=&quot;FBORNPOP&quot; data-language=&quot;en&quot; data-locations=&quot;&quot; data-measure=&quot;PC_POP&quot; data-referrer=&quot;//data.oecd.org/migration/foreign-born-population.htm&quot; data-show-baseline=&quot;False&quot; data-show-non-baseline=&quot;True&quot; data-source=&quot;OECD International Migration Statistics: International migration database&quot; data-start-time=&quot;2013&quot; data-stored-configuration=&quot;impose&quot; data-subject=&quot;TOT&quot; data-table-note=&quot;Information on data for Israel: http://oe.cd/israel-disclaimer&quot; data-type=&quot;table&quot; data-use-latest-data=&quot;False&quot;&gt;
      &lt;/div&gt;
      &lt;footer class=&quot;embedded-chart-footer&quot; id=&quot;embedded-chart-footer&quot;&gt;
          &lt;div class=&quot;copyright-tooltip info-tooltip&quot; tabindex=&quot;0&quot;&gt;
              &lt;i class=&quot;icon-ddp-copyright&quot;&gt;&lt;/i&gt;
              &lt;p class=&quot;info-tooltip-content&quot;&gt;
                  © OECD
                  &lt;span class=&quot;separator&quot;&gt;·&lt;/span&gt;
                  &lt;a href=&quot;//www.oecd.org/termsandconditions/&quot; target=&quot;_top&quot;&gt;Terms &amp;amp; Conditions&lt;/a&gt;
              &lt;/p&gt;
          &lt;/div&gt;
          &lt;p class=&quot;explore&quot;&gt;Compare countries on &lt;a class=&quot;chart-open-referrer&quot; href=&quot;//data.oecd.org/migration/foreign-born-population.htm&quot; target=&quot;_top&quot;&gt;data.oecd.org&lt;/a&gt;&lt;/p&gt;
          &lt;p class=&quot;logo&quot;&gt;&lt;a class=&quot;chart-open-referrer&quot; href=&quot;//data.oecd.org/migration/foreign-born-population.htm&quot; target=&quot;_top&quot; title=&quot;OECD Data&quot;&gt;OECD Data&lt;/a&gt;&lt;/p&gt;
      &lt;/footer&gt;
  &lt;/div&gt;
  &lt;/div&gt;
  &lt;script src=&quot;//data.oecd.org/media/dataportal/scripts/ddp-chart-config.js&quot;&gt;&lt;/script&gt;
  &lt;script src=&quot;//data.oecd.org/media/dataportal/scripts/jquery.min.js&quot;&gt;&lt;/script&gt;
  &lt;script src=&quot;//data.oecd.org/media/dataportal/scripts/application.js&quot;&gt;&lt;/script&gt;
  
  
  &lt;/body&gt;&lt;/html&gt;
  </code></pre>
  <p>Am I scraping a cached webpage?</p>
  <h2>Edited</h2>
  <p>See below for the code I use and above for the response.</p>
  <pre><code>from bs4 import BeautifulSoup
  from requests_html import HTMLSession
  
  session = HTMLSession()
  oecd_url = &quot;https://data.oecd.org/chart/5Rmq&quot;
  response = session.get(oecd_url)
  soup = BeautifulSoup(response.html.html,&quot;html5lib&quot;)
  soup
  </code></pre>