Question: I would like to know the size of the documents hosted on a website.

A solution could be to download all contents of this website with tools such as `wget -r`. However, this specific website hosts many huge files and the amount of data probably exceed 3To. I cannot afford to use such bandwidth and host so many data.

A second solution could be to connect by ftp to this website. But I don't have ftp access to this server.

Is there another solution to my problem? To sum up, I want to know how big is a directory hosted on a website and I have only read and browse permissions.

P-S: I'm using Linux (Debian) and welcome in particular every solution compatible with it.

Answer: 
As noted, you *could* use the `Content-Length` item from an HTTP header. Various tools can get the header information, e.g.,

```
$ lynx -dump -head http://localhost
HTTP/1.1 200 OK
Date: Wed, 18 Nov 2015 11:21:40 GMT
Server: Apache
Last-Modified: Sun, 10 Nov 2013 20:02:38 GMT
ETag: "bf2-4ead818dfd380"
Accept-Ranges: bytes
Content-Length: 3058
MS-Author-Via: DAV
Content-Type: text/html
Connection: close

```

However, you will still have to retrieve a given web page to extract URLs from it. Lynx gives a list at the end of a dump, e.g,.

```
$ lynx -dump http://localhost
...
References

   1. http://localhost/NetBeans-ROR-plugin/updates/updates.xml
   2. http://localhost/webcal
   3. http://localhost/changepassword
   4. http://localhost/profilemanager
   5. http://www.apple.com/server/

```

Putting those together, it would be possible to write a script that retrieved URLs from a page, then walked through the list of referenced URLs, first doing a "head" (probably faster than reading the whole referenced page), inspecting both the `Content-Length` and `Content-Type` to see if that information is available, and if so, whether it points to another web page to crawl through.

For reference:

* [HTTP/1.1 Header Field Definitions](http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html)
* [Quick reference to HTTP headers](https://www.cs.tut.fi/~jkorpela/http.html)
* [Lynx user's guide](http://lynx.invisible-island.net/lynx_help/Lynx_users_guide.html)
* [cURL manual](http://curl.haxx.se/docs/manual.html)
* [wget manual](https://www.gnu.org/software/wget/manual/wget.html)

Since you only want *sizes*, your script could simply use the header information when it points to non-HTML items where the content-length is known.
Use curl with `-I` option to get only the response header (i.e. without file downloading)

```
$ curl -sI REMOTE_FILE_LOCATION | grep Content-Length

```

or if you want to extract only the number

```
$ curl -sI REMOTE_FILE_LOCATION | grep Content-Length | cut -d' ' -f2

```

`curl` is usually installed by default on Linux, so you don't have to install it. In order to use `lynx` from the previous answer, you'll have to download and install it.