Cached Chrome Top Million Websites
If you are interested in the research on technologies used on the Internet, I recommend playing with the &quot;Minicrawl&quot; dataset.<p>It contains data about ~7 million top websites, and for every website, it also contains: - the full content of the main page; - the verbose output of curl, containing various timing info; the HTTP headers, protocol info...<p>Using this dataset, you can build a service similar to <a href="https:&#x2F;&#x2F;builtwith.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;builtwith.com&#x2F;</a> for your research.<p>Data: <a href="https:&#x2F;&#x2F;clickhouse-public-datasets.s3.amazonaws.com&#x2F;minicrawl&#x2F;data.native.zst" rel="nofollow">https:&#x2F;&#x2F;clickhouse-public-datasets.s3.amazonaws.com&#x2F;minicraw...</a> (129 GB compressed, ~1 TB uncompressed).<p>Description: <a href="https:&#x2F;&#x2F;github.com&#x2F;ClickHouse&#x2F;ClickHouse&#x2F;issues&#x2F;18842">https:&#x2F;&#x2F;github.com&#x2F;ClickHouse&#x2F;ClickHouse&#x2F;issues&#x2F;18842</a><p>You can easily try it with clickhouse-local without downloading:<p><pre><code>  $ curl https:&#x2F;&#x2F;clickhouse.com&#x2F; | sh

  $ .&#x2F;clickhouse local 
    ClickHouse local version 22.13.1.294 (official build).

    milovidov-desktop :) DESCRIBE url(&#x27;https:&#x2F;&#x2F;clickhouse-public-datasets.s3.amazonaws.com&#x2F;minicrawl&#x2F;data.native.zst&#x27;)

    DESCRIBE TABLE url(&#x27;https:&#x2F;&#x2F;clickhouse-public-datasets.s3.amazonaws.com&#x2F;minicrawl&#x2F;data.native.zst&#x27;)

    Query id: 6746232f-7f5f-4c5a-ac68-d749d949a2dc

    ┌─name────┬─type───┬─default_type─┬─default_expression─┬─comment─┬─codec_expression─┬─ttl_expression─┐
    │ rank    │ UInt32 │              │                    │         │                  │                │
    │ domain  │ String │              │                    │         │                  │                │
    │ log     │ String │              │                    │         │                  │                │
    │ content │ String │              │                    │         │                  │                │
    └─────────┴────────┴──────────────┴────────────────────┴─────────┴──────────────────┴────────────────┘

    4 rows in set. Elapsed: 1.390 sec. 

    milovidov-desktop :) SELECT rank, domain, log, substringUTF8(content, 1, 100) FROM url(&#x27;https:&#x2F;&#x2F;clickhouse-public-datasets.s3.amazonaws.com&#x2F;minicrawl&#x2F;data.native.zst&#x27;) LIMIT 1 FORMAT Vertical

    SELECT
        rank,
        domain,
        log,
        substringUTF8(content, 1, 100)
    FROM url(&#x27;https:&#x2F;&#x2F;clickhouse-public-datasets.s3.amazonaws.com&#x2F;minicrawl&#x2F;data.native.zst&#x27;)
    LIMIT 1
    FORMAT Vertical

    Query id: 8dba6976-0bf6-4ce8-a0f1-aa579c828175

    Row 1:
    ──────
    rank:                           1907977
    domain:                         0--0.uk
    log:                            *   Trying 213.32.47.30:80...
    * Connected to 0--0.uk (213.32.47.30) port 80 (#0)
    &gt; GET &#x2F; HTTP&#x2F;1.1
    &gt; Host: 0--0.uk
    &gt; Accept: *&#x2F;*
    &gt; User-Agent: Mozilla&#x2F;5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko&#x2F;20100101 Firefox&#x2F;84.0
    &gt; 
    * Mark bundle as not supporting multiuse
    &lt; HTTP&#x2F;1.1 302 Moved Temporarily
    &lt; Server: nginx
    &lt; Date: Sun, 29 May 2022 06:27:14 GMT
    &lt; Content-Type: text&#x2F;html
    &lt; Content-Length: 154
    &lt; Connection: keep-alive
    &lt; Location: https:&#x2F;&#x2F;0--0.uk&#x2F;
    &lt; 
    * Ignoring the response-body
    { [154 bytes data]
    * Connection #0 to host 0--0.uk left intact
    * Issue another request to this URL: &#x27;https:&#x2F;&#x2F;0--0.uk&#x2F;&#x27;
    *   Trying 213.32.47.30:443...
    * Connected to 0--0.uk (213.32.47.30) port 443 (#1)
    * ALPN, offering h2
    * ALPN, offering http&#x2F;1.1
    *  CAfile: &#x2F;etc&#x2F;ssl&#x2F;certs&#x2F;ca-certificates.crt
    *  CApath: &#x2F;etc&#x2F;ssl&#x2F;certs
    * TLSv1.0 (OUT), TLS header, Certificate Status (22):
    } [5 bytes data]
    * TLSv1.3 (OUT), TLS handshake, Client hello (1):
    } [512 bytes data]
    * TLSv1.2 (IN), TLS header, Certificate Status (22):
    { [5 bytes data]
    * TLSv1.3 (IN), TLS handshake, Server hello (2):
    { [108 bytes data]
    * TLSv1.2 (IN), TLS header, Certificate Status (22):
    { [5 bytes data]
    * TLSv1.2 (IN), TLS handshake, Certificate (11):
    { [4150 bytes data]
    * TLSv1.2 (IN), TLS header, Certificate Status (22):
    { [5 bytes data]
    * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
    { [333 bytes data]
    * TLSv1.2 (IN), TLS header, Certificate Status (22):
    { [5 bytes data]
    * TLSv1.2 (IN), TLS handshake, Server finished (14):
    { [4 bytes data]
    * TLSv1.2 (OUT), TLS header, Certificate Status (22):
    } [5 bytes data]
    * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
    } [70 bytes data]
    * TLSv1.2 (OUT), TLS header, Finished (20):
    } [5 bytes data]
    * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
    } [1 bytes data]
    * TLSv1.2 (OUT), TLS header, Certificate Status (22):
    } [5 bytes data]
    * TLSv1.2 (OUT), TLS handshake, Finished (20):
    } [16 bytes data]
    * TLSv1.2 (IN), TLS header, Finished (20):
    { [5 bytes data]
    * TLSv1.2 (IN), TLS header, Certificate Status (22):
    { [5 bytes data]
    * TLSv1.2 (IN), TLS handshake, Finished (20):
    { [16 bytes data]
    * SSL connection using TLSv1.2 &#x2F; ECDHE-RSA-AES128-GCM-SHA256
    * ALPN, server accepted to use http&#x2F;1.1
    * Server certificate:
    *  subject: CN=mail.htservices.co.uk
    *  start date: May 15 18:36:37 2022 GMT
    *  expire date: Aug 13 18:36:36 2022 GMT
    *  subjectAltName: host &quot;0--0.uk&quot; matched cert&#x27;s &quot;0--0.uk&quot;
    *  issuer: C=US; O=Let&#x27;s Encrypt; CN=R3
    *  SSL certificate verify ok.
    * TLSv1.2 (OUT), TLS header, Supplemental data (23):
    } [5 bytes data]
    &gt; GET &#x2F; HTTP&#x2F;1.1
    &gt; Host: 0--0.uk
    &gt; Accept: *&#x2F;*
    &gt; User-Agent: Mozilla&#x2F;5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko&#x2F;20100101 Firefox&#x2F;84.0
    &gt; 
    * TLSv1.2 (IN), TLS header, Supplemental data (23):
    { [5 bytes data]
    * Mark bundle as not supporting multiuse
    &lt; HTTP&#x2F;1.1 200 OK
    &lt; Server: nginx
    &lt; Date: Sun, 29 May 2022 06:27:15 GMT
    &lt; Content-Type: text&#x2F;html;charset=utf-8
    &lt; Transfer-Encoding: chunked
    &lt; Connection: keep-alive
    &lt; X-Frame-Options: SAMEORIGIN
    &lt; Expires: -1
    &lt; Cache-Control: no-store, no-cache, must-revalidate, max-age=0
    &lt; Pragma: no-cache
    &lt; Content-Language: en-US
    &lt; Set-Cookie: ZM_TEST=true;Secure
    &lt; Set-Cookie: ZM_LOGIN_CSRF=b2dda010-d795-4759-a9c3-80349f3b46ed;Secure;HttpOnly
    &lt; Vary: User-Agent
    &lt; X-UA-Compatible: IE=edge
    &lt; Vary: Accept-Encoding, User-Agent
    &lt; 
    { [13068 bytes data]
    * Connection #1 to host 0--0.uk left intact

    substringUTF8(content, 1, 100): &lt;!DOCTYPE html&gt;
    &lt;!-- set this class so CSS definitions that now use REM size, would work relative to

    1 row in set. Elapsed: 0.539 sec. Processed 4.60 thousand rows, 273.86 MB (8.54 thousand rows&#x2F;s., 508.28 MB&#x2F;s.)</code></pre>