### ----------------------------------------------------
### -- cro
### -- Licenses: Artistic-2.0
### -- Authors: Jonathan Worthington <jnthn@jnthn.net>
### -- File: docs/reference/cro-http-client.md
### ----------------------------------------------------
# Cro::HTTP::Client

The `Cro::HTTP::Client` class provides a flexible asynchronous HTTP and
HTTPS client, scaling from simple to more complex use cases. It can be
consumed in two ways:

* By making calls on the type object (`Cro::HTTP::Client.get($url)`). This
  is good for one-off requests, but does not provide connection re-use when
  making multiple requests to the same server (such as by using HTTP/1.1
  persistent connections or HTTP/2.0 mutliplexing).
* By making an instance of `Cro::HTTP::Client`. By default, this enables
  re-use of a pool of connections (HTTP/1.1) or multiplexing (HTTP/2.0). It
  may also be configured with a default base URL, default authorization data
  to pass along, and even middleware to insert into the request/response
  processing pipeline. An instance of `Cro::HTTP::Client` may be used
  concurrently.

In general, if you are going to make a one-off request, just use the type
object. If you are going to make many requests to the same server or set of
servers, make an instance.

By default, a HTTPS request will use ALPN to negotiate whether to do HTTP/2 or
HTTP/1.1, and a HTTP request will always use HTTP/1.1.

## Making basic requests

The `get`, `post`, `put`, `delete`, `patch` and `head` methods may be called on either
the type object or an instance of `Cro::HTTP::Client`. They will all return a
`Promise`, which will be kept if the request is successful or broken if an
error occurs.

    my $resp = await Cro::HTTP::Client.get('https://www.raku.org/');

The response will be provided as a `Cro::HTTP::Response` object. It will be
produced as soon as the request headers are available; the body may not yet
have been received.

If making an instance of `Cro::HTTP::Client`, a base URI may be specified.
The URI passed to request methods will be appended to the base URI using
the relative URI resolution algorithm.

    my $client = Cro::HTTP::Client.new(base-uri => "http://foo.com/some/path/");
    await $client.get('added');             # http://foo.com/some/path/added
    await $client.get('/rooted');           # http://foo.com/rooted
    await $client.get('http://bar.com/');   # http://bar.com

## Error handling

By default, error responses (4xx and 5xx status codes) will result in an
exception that does the `X::Cro::HTTP::Error` role. Such exceptions have a
`response` property containing the `Cro::HTTP::Response` object.

    my $resp = await Cro::HTTP::Client.delete($product-url);
    CATCH {
        when X::Cro::HTTP::Error {
            if .response.status == 404 {
                say "Product not found!";
            }
            else {
                say "Unexpected error: $_";
            }
        }
    }

The actual exception type will be either `X::Cro::HTTP::Error::Client` for
4xx errors, and `X::Cro::HTTP::Error::Server` for 5xx errors (which is useful
when setting up retries that should distinguish server errors from client
errors).

The exception also has a `request` property, which provides access to the
`Cro::HTTP::Request` that was sent.

    my $resp = await Cro::HTTP::Client.get($url);
    CATCH {
        when X::Cro::HTTP::Error {
            say "Problem fetching " ~ .request.target;
        }
    }

This method simply delegates to `.response.request`, since each response
object has the request that was sent attached to it. In the event of a
redirect, the request object will be that of the redirected request, not the
originally sent request.

## Setting the user agent

By default, `Cro::HTTP::Client` sends a `User-agent` header with the value `Cro`.
This can be done at the request level:

    my $resp = await Cro::HTTP::Client.get: 'example.com',
        user-agent => 'MyCrawler v42';

Or set at construction time when making an instance of the client, in which case
it will be used for all requests (unless overridden in a specific request):

    my $client = Cro::HTTP::Client.new:
        user-agent => 'MyCrawler v42';

To suppress sending a `User-agent` header, pass either `False`, `Nil`, or the
empty string.

## Adding extra request headers

One or more headers can be set for a request by passing an array to the
`headers` named argument. It may contain either `Pair` objects, instances
of `Cro::HTTP::Header`, or a mix of the two.

    my $resp = await Cro::HTTP::Client.get: 'example.com',
        headers => [
            referer => 'http://anotherexample.com',
            Cro::HTTP::Header.new(
                name => 'X-MyCustomHeader',
                value => 'pancake'
            )
        ];

If the headers should be added to all requests, they can be set by default at
construction time:

    my $client = Cro::HTTP::Client.new:
        headers => [
            X-MyCustomHeader => 'strudel'
        ];

## Adding query string parameters

Query string parameters can be added to the request by passing the `query` named
argument. It can be either a list of `Pair`s or a `Hash`. Both keys and values
will be encoded.

    my $resp = await Cro::HTTP::Client.get: 'we.love.pand.as/pandas/search',
        query => { location => 'Chengdu', maxweight => 90 };

## Setting the request body

To give the request a body, pass the `body` named argument. The `content-type`
named argument should typically be passed too, to indicate the type of the
body. For example, a request with a JSON body can be sent as:

    my %panda = name => 'Bao Bao', eats => 'bamboo';
    my $resp = await Cro::HTTP::Client.post: 'we.love.pand.as/pandas',
        content-type => 'application/json',
        body => %panda;

If writing a client for a JSON API, it may become tedious to set the content
type on every request. In this case, it can be set when constructing an
instance of the client, and used by default (note that it will only be used
if a body is set):

```
# Configure with JSON content type.
my $client = Cro::HTTP::Client.new: content-type => 'application/json';

# And later get it added by default.
my %panda = name => 'Bao Bao', eats => 'bamboo';
my $resp = await $client.post: 'we.love.pand.as/pandas', body => %panda;
```

The `Cro::HTTP::Client` class uses a `Cro::BodySerializer` in order to
serialize request bodies for sending. Besides JSON, there are body parsers
encoding and sending a `Str`:

    my $resp = await Cro::HTTP::Client.post: 'we.love.pand.as/facts',
        content-type => 'text/plain; charset=UTF-8',
        body => "99% of a Panda's diet consists of bamboo";

A `Blob`:

    my $resp = await Cro::HTTP::Client.put: 'we.love.pand.as/images/baobao.jpg',
        content-type => 'image/jpeg',
        body => slurp('baobao.jpg', :bin);

Form data formatted according to `application/x-www-form-urlencoded` (this is
the default in a web browser):

    my $resp = await Cro::HTTP::Client.post: 'we.love.pand.as/pandas',
        content-type => 'application/x-www-form-urlencoded',
        # Can use a Hash; an Array of Pair allows multiple values per name
        body => [
            name => 'Bao Bao',
            eats => 'bamboo'
        ];

Or form data formatted according to `multipart/form-data` (this is used in web
browsers for forms that contain file uploads):

    my $resp = await Cro::HTTP::Client.post: 'we.love.pand.as/pandas',
        content-type => 'multipart/form-data',
        body => [
            # Simple pairs for simple form values
            name => 'Bao Bao',
            eats => 'bamboo',
            # For file uploads, make a part object
            Cro::HTTP::Body::MultiPartFormData::Part.new(
                headers => [Cro::HTTP::Header.new(
                    name => 'Content-type',
                    value => 'image/jpeg'
                )],
                name => 'photo',
                filename => 'baobao.jpg',
                body-blob => slurp('baobao.jpg', :bin)
            )
        ];

To replace the set of body serializers that a client will use, pass an array
of them when constructing an instance of `Cro::HTTP::Client` using the
`body-serializers` named argument:

    use Cro::HTTP::BodySerializers;
    
    my $client = Cro::HTTP::Client.new:
        body-serializers => [
            Cro::HTTP::BodySerializer::JSON,
            My::BodySerializer::XML
        ];

To instead retain the existing set of body serializers and add some new ones
(which will have higher precedence), use `add-body-serializers`:

    my $client = Cro::HTTP::Client.new:
        add-body-serializers => [ My::BodySerializer::XML ];

It is also possible to have the body come from a stream of bytes by passing a
`Supply` to `body-byte-stream`.

    my $resp = await Cro::HTTP::Client.post: 'example.com/incoming',
        content-type => 'application/octet-stream',
        body-byte-stream => $supply;

The `body` and `body-byte-stream` arguments cannot be used together; trying to
do so will result in a `X::Cro::HTTP::Client::BodyAlreadySet` exception.

## Getting the response body

The response body is always provided asynchronously, either by a `Promise` (if
requesting the enitre body) or a `Supply` (when the body is to be delivered as
it arrives).

The `body` method returns a `Promise` that will be kept when the body has
been received and parsed.

```
my $resp = await Cro::HTTP::Client.get($some-json-api-url);
my $json = await $resp.body;
```

The `body` method will offer the response to each available body parser, and
returns a `Promise` that will be kept when the first applicable body parser has
completely parsed the body. The default body parsers available are:

* JSON, which will be used when the `Content-type` header is either
  `application/json` or uses the `+json` suffix. `JSON::Fast` will be used to
  perform the parsing.
* String fallback, which is used when the `Content-type` type is `text/*`. A
  `Str` will be returned.
* Blob fallback, which is used in all other cases and returns a `Blob` with
  the body.

A `Cro::HTTP::Client` instance can be configured either with a replacement set
of body parsers by passing the `body-parsers` argument:

    use Cro::HTTP::BodyParsers;
    
    my $client = Cro::HTTP::Client.new:
        body-parsers => [
            Cro::HTTP::BodyParser::JSON,
            My::BodyParser::XML
        ];

Or to prepend extra body parsers to the default set, use `add-body-parsers`:

    my $client = Cro::HTTP::Client.new:
        add-body-parsers => [ My::BodyParser::XML ];

To get the response body as a `Supply` that will emit the bytes as they
arrive over the network, use the `body-byte-stream` method:

    react {
        whenever $resp.body-byte-stream -> $chunk {
            say "Got chunk: $chunk.gist()";
        }
    }

To get the entire response body as a `Blob`, use the `body-blob` method:

    my Blob $body = await $resp.body-blob();

To get the entire response body as a `Str`, use the `body-text` method:

    my Str $body = await $resp.body-text();

This method will look at the `Content-type` header to see if a `charset` is
specified, and decode the body using that. Otherwise, it will see if the body
starts with a [BOM](https://en.wikipedia.org/wiki/Byte_order_mark) and rely on
that. If it is not passed, the a heuristic will be used: if the body can be
decoded as `utf-8` then it will be deemed to be `utf-8`, and failing that it
will be decoded as `latin-1` (which can never fail as all bytes are valid).

## Cookies

By default, cookies in the response are ignored. However, constructing a
`Cro::HTTP::Client` with the `:cookie-jar` option (that is, passing `True`)
will create an instance of `Cro::HTTP::Client::CookieJar`. This will be used
to store all cookies set in responses. Relevant cookies will automatically be
included in follow-up requests.

    my $client = Cro::HTTP::Client.new(:cookie-jar);

Cookie relevance is determiend by considering host, path, and the `Secure`
extension. Cookies that have passed their expiration date for maximum age will
automatically be removed from the cookie jar.

It is also possible to pass in an instance of `Cro::HTTP::Client::CookieJar`,
which makes it possible to share one cookie jar amongst several instances of
the client (or to pass in a subclass that adds extra features).

    my $jar = Cro::HTTP::Client::CookieJar.new;
    my $client = Cro::HTTP::Client.new(cookie-jar => $jar);
    my $json-client = Cro::HTTP::Client.new:
        cookie-jar => $jar,
        content-type => 'application/json';

To include a particular set of cookies with a request, pass them in a hash
using the `cookies` named argument when making a reuqest:

    my $resp = await $client.get: 'http://somesite.com/',
        cookies => {
            session => $fake-session-id
        };

Cookies passed in this way will *override* any cookies from a cookie jar.

To get the cookies set by a response, use the `cookies` method on the
`Cro::HTTP::Response` object, which returns a `List` of `Cro::HTTP::Cookie`
objects.

## Following redirects

By default, `Cro::HTTP::Client` will follow HTTP redirect responses, with a
limit of 5 redirects being enforced in order to avoid circular redirects. If
there are more than 5 redirections, `X::Cro::HTTP::Client::TooManyRedirects`
will be thrown.

This behavior can be configured when constructing a new `Cro::HTTP::Client` or
on a per-request basis, with the per-request setting overriding the behavior
configured at construction time. In either case, it is done using the `follow`
named argument.

    :follow         # follow redirects (up to 5 times per request)
    :!follow        # never follow redirects
    :follow(2)      # follow redirects (up to 2 times per request)
    :follow(10)     # follow redirects (up to 10 times per request)

The 301, 307 and 308 redirects are treated identically at this time; no
caching of permanent redirects takes place. They retain the original request
method. 302 and 303 instead cause a `GET` request to be issued, regardless of
the original request method.

The `.request` property of the response object will refer to the final request
that was issued after a redirect (or sequence of redirects). Therefore, to get
the URI that a request was eventually redirected to, use the `.uri` property
of that request object.

    my $resp = await Cro::HTTP::Client.get($some-url);
    say "Eventually requested $resp.request.uri()";

## Authentication

Both basic authentication and bearer authentication are supported directly by
`Cro::HTTP::Client`. These can be configured when instantiating the client, or
per request (which will override that configured on the instance).

For basic authentication, pass the `auth` option with a hash containing a
username and a password.

    auth => {
        username => $user,
        password => $password
    }

For bearer authentication, pass the `auth` option with a hash containing a
bearer:

    auth => { bearer => $jwt }

Failing to pass precisely either `username` and `password` *or* `bearer` will
result in an `X::Cro::Client::InvalidAuth` exception.

In both cases, the authentication information will be sent immediately with
the request. In order to only have it sent if the server responds to the
initial request with a 401 response, set the `if-asked` option to `True`.

    auth => {
        username => $user,
        password => $password,
        if-asked => True
    }

## TLS configuration

The `ca` argument, passed either at construction time or to a request method,
is used to provide TLS configuration. Its primary use is for providing a
custom CA certificate:

    my $client = Cro::HTTP::Client.new:
        ca => { ca-file => 't/certs-and-keys/ca-crt.pem' };

However, the hash may contain any arguments that the `connect` method of
[the TLS module](https://github.com/jnthn/p6-io-socket-async-ssl) accepts. Of
note, one can disable certificate checking by passing the `insecure` option:

    my $client = await Cro::HTTP::Client.get: 'https://badly-configur.ed/',
        ca => { :insecure };

As the name suggests, this is not a secure configuration; transmissions are
encrypted, but there's no checking that the server is who it claims to be.

## Proxying

By default, `Cro::HTTP::Client` will honor the `HTTP_PROXY`, `HTTPS_PROXY`
and `NO_PROXY` enrivonment variables. It is also possible to pass the
`http-proxy` and/or `https-proxy` named arguments when constructing
`Cro::HTTP::Client`; these will be used for all requests made with that
instance (and take preference over any proxy found via the environment,
and furthermore cause `NO_PROXY` to be disregarded).

It is not possible to override the proxy at a per-request level. There is no
mechanism to ignore the `HTTP_PROXY` or `HTTPS_PROXY` environment variables,
however one could delete them from `%*ENV` if needed.

## Persistent connections

An instance of `Cro::HTTP::Client` will use persistent connections by default.
When many requests are being made to the same server, this can enable better
throughput by not requiring a new connection to be established each time. To
not use persisted connections, pass `:!persistent` to the constructor. When
using the type object (for example, `Cro::HTTP::Client.get($url)`, then no
persistent connection cache will be used.

## Timeouts

By default, `Cro::HTTP::Client` enforces:

* A 60s timeout on establishing a connection to the target server
* A 60s timeout on receiving the response headers once a connection has been
  established and the request sent
* No timeout on receiving the entire response body
* No overall bounding timeout for the entire HTTP request/response

These can be configured by passing the `timeout` setting, either at an instance
or per-request level. One may pass:

* A `Real` value, which will be interpreted as the total number of seconds for
  the entire HTTP request/response (including the body being downloaded). This
  will not increase the default connection and headers timeouts, however they
  will be clipped to the total time budget if it is smaller.
* A hash with the keys `connection`, `headers`, `body`, and `total` (any not
  provided will have the default values of 60, 60, `Inf`, and `Inf`)
  respectively.
* An object that does the `Cro::Policy::Timeout` role, should one wish to
  implement a more complex scheme.

In the case persistent connections are being used:

* For HTTP/1.1, the connection will be closed in the event of any timeout
* For HTTP/2.0, in the event of a body timeout, only the individual stream
  will be reset, and the connection left intact

## HTTP version

The `:http` option can be passed, either at construction or per request, to
control which versions of HTTP should be used. It can be passed a single item
or list. Valid options are `1.1` (which will implicitly handle HTTP/1.0 too)
and `2`.

    :http<1.1>      # HTTP/1.1 only
    :http<2>        # HTTP/2 only
    :http<1.1 2>    # HTTP/1.1 and HTTP/2 (HTTPS only; selected by ALPN)

The default is `:http<1.1>` for a HTTP request, and `:http<1.1 2>` for a HTTPS
request. It is not legal to use `:http<1.1 2>` with a HTTP connection, as ALPN
is the only supported mechanism for deciding which protocol to use.

## Push promises

HTTP/2.0 provides push promises, which allow the server to push extra resources
to the client as part of the response. By default, `Cro::HTTP::Client` will
instruct the remote server to **not** send push promises. To opt in to this
feature, either:

* If making an instance of `Cro::HTTP::Client`, pass `:push-promises` to the
  constructor to enable them for all requests made with the client instance
* Otherwise, pass `:push-promises` when making a request (for example, to
  the `get` method). However, when using HTTP/2.0, it's usually wise to make
  an instance and re-use the connection for many requests.

Push promises are obtained by calling the `push-promises` method of the
`Cro::HTTP::Response` object that the request produces. This returns a `Supply`
that emits an instance of `Cro::HTTP::PushPromise` for each push promise the
server sends. Each of those in turn has a `response` property that returns a
`Promise` that will be kept with a `Cro::HTTP::Response` object when the push
promise is fulfilled.

Making a request and obtaining all push promises can therefore be achieved as
follows:

```
react {
    my $client = Cro::HTTP::Client.new(:push-promises);
    my $response = await $client.get($url);
    whenever $response.push-promises -> $prom {
        whenever $prom.response -> $resp {
            say "Push promise for $prom.target() had status $resp.status()";
        }
    }
}
```

## Custom HTTP methods

The `get`, `post`, `put`, `delete`, `patch` and `head` methods are convenience
forms of the more general `request` method, which takes the HTTP request
method as a first argument. The `request` method can be used to make requests
with other HTTP methods. For example, making a request with the `LINK` method
can be achieved using:

```
my $resp = await Cro::HTTP::Client.request('LINK', $url);
```

This may also be useful if the request method to use is held in a variable.

## Tracing

To debug problems with the Cro HTTP client, or to understand in more detail
exactly what is being sent and received, set `CRO_TRACE=1` in the environment.
(Note that this turns on tracing for all Cro components, not just the client.)

Long binary blobs (such as dumps of the TCP packets arriving) will be
truncated in the debug output. To raise the limit, put something like
`CRO_TRACE_MAX_BINARY_DUMP=8192` in the environment (you may wish to pick a
higher value).


