txt: ## dist_zef-raku-community-modules-HTTP-UserAgent.md
  [![Actions Status](https://github.com/raku-community-modules/HTTP-UserAgent/actions/workflows/linux.yml/badge.svg)](https://github.com/raku-community-modules/HTTP-UserAgent/actions) [![Actions Status](https://github.com/raku-community-modules/HTTP-UserAgent/actions/workflows/macos.yml/badge.svg)](https://github.com/raku-community-modules/HTTP-UserAgent/actions)
  
  # NAME
  
  HTTP::UserAgent - Web user agent class
  
  # SYNOPSIS
  
  ```
  use HTTP::UserAgent;
  
  my $ua = HTTP::UserAgent.new;
  $ua.timeout = 10;
  
  my $response = $ua.get("URL");
  
  if $response.is-success {
      say $response.content;
  }
  else {
      die $response.status-line;
  }
  
  ```
  
  # DESCRIPTION
  
  This module provides functionality to crawling the web witha handling cookies and correct User-Agent value.
  
  It has TLS/SSL support.
  
  # METHODS
  
  ## method new
  
  ```
  method new(HTTP::UserAgent:U: :$!useragent, Bool :$!throw-exceptions, :$!max-redirects = 5, :$!debug)
  
  ```
  
  Default constructor.
  
  There are four optional named arguments:
  
  * useragent
  
  A string that specifies what will be provided in the `User-Agent` header in the request. A number of standard user agents are described in [HTTP::UserAgent::Common](http::UserAgent::Common), but a string that is not specified there will be used verbatim.
  
  * throw-exceptions
  
  By default the `request` method will not throw an exception if the response from the server indicates that the request was unsuccesful, in this case you should check `is-success` to determine the status of the [HTTP::Response](http::Response) returned. If this is specified then an exception will be thrown if the request was not a success, however you can still retrieve the response from the `response` attribute of the exception object.
  
  * max-redirects
  
  This is the maximum number of redirects allowed for a single request, if this is exceeded then an exception will be thrown (this is not covered by `no-exceptions` above and will always be throw,) the default value is 5.
  
  * debug
  
  It can etheir be a Bool like simply `:debug` or you can pass it a IO::Handle or a file name. Eg `:debug($*ERR)` will ouput on stderr `:debug("mylog.txt")` will ouput on the file.
  
  ## method auth
  
  ```
  method auth(HTTP::UserAgent:, Str $login, Str $password)
  
  ```
  
  Sets username and password needed to HTTP Auth.
  
  ## method get
  
  ```
  multi method get(Str $url is copy, :bin?, *%headers)
  multi method get(URI $uri, :bin?, *%headers)
  
  ```
  
  Requests the $url site, returns HTTP::Response, except if throw-exceptions is set as described above whereby an exception will be thrown if the response indicates that the request wasn't successfull.
  
  If the Content-Type of the response indicates that the content is text the `content` of the Response will be a decoded string, otherwise it will be left as a <Blob>.
  
  If the ':bin' adverb is supplied this will force the response `content` to always be an undecoded <Blob>
  
  Any additional named arguments will be applied as headers in the request.
  
  ## method post
  
  ```
  multi method post(URI $uri, %form, *%header ) -> HTTP::Response
  multi method post(Str $uri, %form, *%header ) -> HTTP::Response
  
  ```
  
  Make a POST request to the specified uri, with the provided Hash of %form data in the body encoded as "application/x-www-form-urlencoded" content. Any additional named style arguments will be applied as headers in the request.
  
  An [HTTP::Response](http::Response) will be returned, except if throw-exceptions has been set and the response indicates the request was not successfull.
  
  If the Content-Type of the response indicates that the content is text the `content` of the Response will be a decoded string, otherwise it will be left as a <Blob>.
  
  If the ':bin' adverb is supplied this will force the response `content` to always be an undecoded <Blob>
  
  If greater control over the content of the request is required you should create an [HTTP::Request](http::Request) directly and populate it as needed,
  
  ## method request
  
  ```
  method request(HTTP::Request $request, :bin?)
  
  ```
  
  Performs the request described by the supplied [HTTP::Request](http::Request), returns a [HTTP::Response](http::Response), except if throw-exceptions is set as described above whereby an exception will be thrown if the response indicates that the request wasn't successful.
  
  If the response has a 'Content-Encoding' header that indicates that the content was compressed, then it will attempt to inflate the data using Compress::Zlib, if the module is not installed then an exception will be thrown. If you do not have or do not want to install Compress::Zlib then you should be able to send an 'Accept-Encoding' header with a value of 'identity' which should cause a well behaved server to send the content verbatim if it is able to.
  
  If the Content-Type of the response indicates that the content is text the `content` of the Response will be a decoded string, otherwise it will be left as a <Blob>. The content-types that are always considered to be binary (and thus left as a <Blob> ) are those with the major-types of 'image','audio' and 'video', certain 'application' types are considered to be 'text' (e.g. 'xml', 'javascript', 'json').
  
  If the ':bin' adverb is supplied this will force the response `content` to always be an undecoded <Blob>
  
  You can use the helper subroutines defined in [HTTP::Request::Common](http::Request::Common) to create the [HTTP::Request](http::Request) for you or create it yourself if you have more complex requirements.
  
  ## routine get :simple
  
  ```
  sub get(Str $url) returns Str is export(:simple)
  
  ```
  
  Like method get, but returns decoded content of the response.
  
  ## routine head :simple
  
  ```
  sub head(Str $url) returns Parcel is export(:simple)
  
  ```
  
  Returns values of following header fields:
  
  * Content-Type
  * Content-Length
  * Last-Modified
  * Expires
  * Server
  
  ## routine getstore :simple
  
  ```
  sub getstore(Str $url, Str $file) is export(:simple)
  
  ```
  
  Like routine get but writes the content to a file.
  
  ## routine getprint :simple
  
  ```
  sub getprint(Str $url) is export(:simple)
  
  ```
  
  Like routine get but prints the content and returns the response code.
  
  # SUPPORT MODULES
  
  ## HTTP::Cookie - HTTP cookie class
  
  This module encapsulates single HTTP Cookie.
  
  ```
  use HTTP::Cookie;
  
  my $cookie = HTTP::Cookie.new(:name<test_name>, :value<test_value>);
  say ~$cookie;
  
  ```
  
  The following methods are provided:
  
  ### method new
  
  ```
  my $c = HTTP::Cookie.new(:name<a_cookie>, :value<a_value>, :secure, fields => (a => b));
  
  ```
  
  A constructor, it takes these named arguments:
  
  | key | description |
  | --- | --- |
  | name | name of a cookie |
  | value | value of a cookie |
  | secure | Secure param |
  | httponly | HttpOnly param |
  | fields | list of field Pairs (field => value) |
  
  ### method Str
  
  Returns a cookie as a string in readable (RFC2109) form.
  
  ## HTTP::Cookies - HTTP cookie jars
  
  This module provides a bunch of methods to manage HTTP cookies.
  
  ```
  use HTTP::Cookies;
  my $cookies = HTTP::Cookies.new(
    :file<./cookies>,
    :autosave
  );
  $cookies.load;
  
  ```
  
  ### method new
  
  ```
  my $cookies = HTTP::Cookies.new(
    :file<./cookies.here>
    :autosave,
  );
  
  Constructor, takes named arguments:
  
  ```
  
  | key | description |
  | --- | --- |
  | file | where to write cookies |
  | autosave | save automatically after every operation on cookies or not |
  
  ### method set-cookie
  
  ```
  my $cookies = HTTP::Cookies.new;
  $cookies.set-cookie('Set-Cookie: name1=value1; HttpOnly');
  
  ```
  
  Adds a cookie (passed as an argument $str of type Str) to the list of cookies.
  
  ### method save
  
  ```
  my $cookies = HTTP::Cookies.new;
  $cookies.set-cookie('Set-Cookie: name1=value1; HttpOnly');
  $cookies.save;
  
  ```
  
  Saves cookies to the file ($.file).
  
  ### method load
  
  ```
  my $cookies = HTTP::Cookies.new;
  $cookies.load;
  
  ```
  
  Loads cookies from file specified at instantiation ($.file).
  
  ### method extract-cookies
  
  ```
  my $cookies = HTTP::Cookies.new;
  my $response = HTTP::Response.new(Set-Cookie => "name1=value; Secure");
  $cookies.extract-cookies($response);
  
  ```
  
  Gets cookies ('Set-Cookie: ' lines) from the HTTP Response and adds it to the list of cookies.
  
  ### method add-cookie-header
  
  ```
  my $cookies = HTTP::Cookies.new;
  my $request = HTTP::Request.new;
  $cookies.load;
  $cookies.add-cookie-header($request);
  
  ```
  
  Adds cookies fields ('Cookie: ' lines) to the HTTP Request.
  
  ### method clear-expired
  
  ```
  my $cookies = HTTP::Cookies.new;
  $cookies.set-cookie('Set-Cookie: name1=value1; Secure');
  $cookies.set-cookie('Set-Cookie: name2=value2; Expires=Wed, 09 Jun 2021 10:18:14 GMT');
  $cookies.clear-expired; # contains 'name1' cookie only
  
  ```
  
  Removes expired cookies.
  
  ### method clear
  
  ```
  my $cookies = HTTP::Cookies.new;
  $cookies.load;   # contains something
  $cookies.clear;  # will be empty after this action
  
  ```
  
  Removes all cookies.
  
  ### method push-cookie
  
  ```
  my $c = HTTP::Cookie.new(:name<a>, :value<b>, :httponly);
  my $cookies = HTTP::Cookies.new;
  $cookies.push-cookie: $c;
  
  ```
  
  Pushes cookies (passed as an argument $c of type HTTP::Cookie) to the list of cookies.
  
  ### method Str
  
  Returns all cookies in human (and server) readable form.
  
  ## HTTP::UserAgent::Common - the most commonly used User-Agents
  
  This module provides a list of the most commonly used User-Agents.
  
  ```
  use HTTP::UserAgent::Common;
  say get-ua('chrome_linux');
  
  ```
  
  ### routine get-ua
  
  ```
  say get-ua('chrome_linux');
  
  ```
  
  Returns correct UserAgent or unchaged passed argument if UserAgent could not be found.
  
  Available UserAgents:
  
  ```
  chrome_w7_64 firefox_w7_64 ie_w7_64 chrome_w81_64 firefox_w81_64 mob_safari_osx
  safari_osx chrome_osx firefox_linux chrome_linux
  
  ```
  
  ## HTTP::Header - class encapsulating HTTP message header
  
  This module provides a class with a set of methods making us able to easily handle HTTP message headers.
  
  ```
  use HTTP::Header;
  my $h = HTTP::Header.new;
  $h.field(Accept => 'text/plain');
  say $h.field('Accept');
  $h.remove-field('Accept');
  
  ```
  
  ### method new
  
  ```
  my $head = HTTP::Header.new(:h1<v1>, :h2<v2>);
  
  ```
  
  A constructor. Takes name => value pairs as arguments.
  
  ### method header
  
  ```
  my $head = HTTP::Header.new(:h1<v1>, :h2<v2>);
  say $head.header('h1');
  
  my $head = HTTP::Header.new(:h1<v1>, :h2<v2>);
  $head.header(:h3<v3>);
  
  ```
  
  Gets/sets header field.
  
  ### method init-field
  
  ```
  my $head = HTTP::Header.new;
  $head.header(:h1<v1>);
  $head.init-header(:h1<v2>, :h2<v2>); # it doesn't change the value of 'h1'
  say ~$head;
  
  ```
  
  Initializes a header field: adds a field only if it does not exist yet.
  
  ### method push-header
  
  ```
  my $head = HTTP::Header.new;
  $head.push-header( HTTP::Header::Field.new(:name<n1>, :value<v1>) );
  say ~$head;
  
  ```
  
  Pushes a new field. Does not check if exists.
  
  ### method remove-header
  
  ```
  my $head = HTTP::Header.new;
  $head.header(:h1<v1>);
  $head.remove-header('h1');
  
  ```
  
  Removes a field of name $field.
  
  ### method header-field-names
  
  ```
  my $head = HTTP::Header.new(:h1<v1>, :h2<v2>);
  my @names = $head.header-field-names;
  say @names; # h1, h2
  
  ```
  
  Returns a list of names of all fields.
  
  ### method clear
  
  ```
  my $head = HTTP::Header.new(:h1<v1>, :h2<v2>);
  $head.clear;
  
  ```
  
  Removes all fields.
  
  ### method Str
  
  Returns readable form of the whole header section.
  
  ### method parse
  
  ```
  my $head = HTTP::Header.new.parse("h1: v1\r\nh2: v2\r\n");
  say $head.raku;
  
  ```
  
  Parses the whole header section.
  
  ## HTTP::Header::Field
  
  This module provides a class encapsulating HTTP Message header field.
  
  ```
  use HTTP::Header::Field;
  my $header = HTTP::Header::Field.new(:name<Date>, values => (123, 456));
  
  ```
  
  ### method new
  
  Constructor. Takes these named arguments:
  
  | key | description |
  | --- | --- |
  | name | name of a header field |
  | values | array of values of a header field |
  
  ### method Str
  
  Stringifies an HTTP::Header::Field object. Returns a header field in a human (and server) readable form.
  
  ## HTTP::Request - class encapsulating HTTP request message
  
  Module provides functionality to easily manage HTTP requests.
  
  ```
  use HTTP::Request;
  my $request = HTTP::Request.new(GET => 'http://www.example.com/');
  
  ```
  
  ### method new
  
  A constructor, the first form takes parameters like:
  
  * method => URL, where method can be POST, GET ... etc.
  * field => values, header fields
  
  ```
  my $req = HTTP::Request.new(:GET<example.com>, :h1<v1>);
  
  ```
  
  The second form takes the key arguments as simple positional parameters and is designed for use in places where for example the request method may be calculated and the headers pre-populated.
  
  ### method set-method
  
  ```
  my $req = HTTP::Request.new;
  $req.set-method: 'POST';
  
  ```
  
  Sets a method of the request.
  
  ### method uri
  
  ```
  my $req = HTTP::Request.new;
  $req.uri: 'example.com';
  
  ```
  
  Sets URL to request.
  
  ### method add-cookies
  
  ```
  method add-cookies(HTTP::Cookies $cookies)
  
  ```
  
  This will cause the appropriate cookie headers to be added from the supplied HTTP::Cookies object.
  
  ### method add-form-data
  
  ```
  multi method add-form-data(%data, :$multipart)
  multi method add-form-data(:$multipart, *%data);
  multi method add-form-data(Array $data, :$multipart)
  
  ```
  
  Adds the form data, supplied either as a `Hash`, an `Array` of `Pair`s, or in a named parameter style, to the POST request (it doesn't make sense on most other request types).
  
  The default is to use 'application/x-www-form-urlencoded' and 'multipart/form-data' can be used by providing the ':multipart' named argument. Alternatively a previously applied "content-type" header of either 'application/x-www-form-urlencoded' or 'multipart/form-data' will be respected and in the latter case any applied boundary marker will be retained.
  
  As a special case for multipart data if the value for some key in the data is an `Array` of at least one item then it is taken to be a description of a file to be "uploaded" where the first item is the path to the file to be inserted, the second (optional) an alternative name to be used in the content disposition header and the third an optional `Array` of `Pair`s that will provide additional header lines for the part.
  
  ### method Str
  
  Returns stringified object.
  
  ### method parse
  
  ```
  method parse(Str $raw_request --> HTTP::Request:D)
  
  ```
  
  Parses raw HTTP request. See `HTTP::Message`
  
  ## HTTP::Request::Common - Construct common HTTP::Request objects
  
  ```
  use HTTP::Request::Common;
  
  my $ua       = HTTP::UserAgent.new;
  my $response = $ua.request(GET 'http://google.com/');
  
  ```
  
  This module provide functions that return newly created `HTTP::Request` objects. These functions are usually more convenient to use than the standard `HTTP::Request` constructor for the most common requests. The following functions are provided:
  
  ### GET $url, Header => Value...
  
  The `GET` function returns an `HTTP::Request` object initialized with the "GET" method and the specified URL.
  
  ### HEAD $url, Header => Value,...
  
  Like `GET` but the method in the request is "HEAD".
  
  ### DELETE $url, Header => Value,...
  
  Like `GET` but the method in the request is "DELETE".
  
  ### `PUT $url, Header =` Value,..., content => $content>
  
  Like `GET` but the method in the request is "PUT".
  
  ## HTTP::Response - class encapsulating HTTP response message
  
  ```
  use HTTP::Response;
  my $response = HTTP::Response.new(200);
  say $response.is-success; # it is
  
  ```
  
  Module provides functionality to easily manage HTTP responses.
  
  Response object is returned by the .get() method of [HTTP::UserAgent](http::UserAgent).
  
  ### method new
  
  ```
  my $response = HTTP::Response.new(200, :h1<v1>);
  
  ```
  
  A constructor, takes named arguments:
  
  | key | description |
  | --- | --- |
  | code | code of the response |
  | fields | header fields (field\_name => values) |
  
  ### method is-success
  
  ```
  my $response = HTTP::Response.new(200);
  say 'YAY' if $response.is-success;
  
  ```
  
  Returns True if response is successful (status == 2xx), False otherwise.
  
  ## method set-code
  
  ```
  my $response = HTTP::Response.new;
  $response.set-code: 200;
  
  ```
  
  Sets code of the response.
  
  ### method Str
  
  Returns stringified object.
  
  ### method parse
  
  See `HTTP::Message`.
  
  ## HTTP::Message - class encapsulating HTTP message
  
  ```
  use HTTP::Message;
  my $raw_msg = "GET / HTTP/1.1\r\nHost: somehost\r\n\r\n";
  my $mess = HTTP::Message.new.parse($raw_msg);
  say $mess;
  
  ```
  
  This module provides a bunch of methods to easily manage HTTP message.
  
  ### method new
  
  ```
  my $msg = HTTP::Message.new('content', :field<value>);
  
  ```
  
  A constructor, takes these named arguments:
  
  | key | description |
  | --- | --- |
  | content | content of the message (optional) |
  | fields | fields of the header section |
  
  ### method add-content
  
  ```
  my $msg = HTTP::Message.new('content', :field<value>);
  $msg.add-content: 's';
  say $msg.content; # says 'contents'
  
  ```
  
  Adds HTTP message content. It does not remove the existing value, it concats to the existing content.
  
  ### method decoded-content
  
  ```
  my $msg = HTTP::Message.new();
  say $msg.decoded-content;
  
  ```
  
  Returns decoded content of the message (using <Encode> module to decode).
  
  ### method field
  
  See `HTTP::Header`.
  
  ### method init-field
  
  See `HTTP::Header`.
  
  ### method push-field
  
  See `HTTP::Header`.
  
  ### method remove-field
  
  See `HTTP::Header`.
  
  ### method clear
  
  ```
  my $msg = HTTP::Message.new('content', :field<value>);
  $msg.clear;
  say ~$msg; # says nothing
  
  ```
  
  Removes the whole message, both header and content section.
  
  ### method parse
  
  ```
  my $msg = HTTP::Message.new.parse("GET / HTTP/1.1\r\nHost: example\r\ncontent\r\n");
  say $msg.raku;
  
  ```
  
  Parses the whole HTTP message.
  
  It takes the HTTP message (with \r\n as a line separator) and obtains the header and content sections, creates a `HTTP::Header` object.
  
  ### method Str
  
  Returns HTTP message in a readable form.
  
  # AUTHOR
  
  * Filip Sergot
  
  Source can be located at: <https://github.com/raku-community-modules/HTTP-UserAgent> . Comments and Pull Requests are welcome.
  
  # COPYRIGHT AND LICENSE
  
  Copyright 2014 - 2022 Filip Sergot
  
  Copyright 2023 - 2025 The Raku Community
  
  This library is free software; you can redistribute it and/or modify it under the MIT License.
  