txt: ## dist_github-gfldex-Shell-Piping.md
  # Shell::Piping
  
  [![Build Status](https://travis-ci.org/gfldex/raku-shell-piping.svg?branch=master)](https://travis-ci.org/gfldex/raku-shell-piping)
  
  Shell pipes without a shell but Raku.
  
  ## SYNOPSIS
  
  ```
  use v6.d;
  use Shell::Piping;
  
  my int $exitcode = 0;
  my &RED = $*OUT.t ?? { „\e[31m$_\e[0m“ } !! { $_ };
  
  sub MAIN(Str $where = ‚/tmp/.‘) {
      my @result;
      my @err;
  
      px«find $where» |» { /a/ ?? $_ !! Nil } |» px<sort -r> |» @result :stderr(@err) :done({$exitcode ⚛= 1 if .exitcodes});
  
      .say for @result.head(10);
  
      if $exitcode {
          $*ERR.put: @err».&RED.join(„\n“);
      }
  
      exit $exitcode;
  }
  
  ```
  
  ## USAGE
  
  This module provides the operator `|»` (aliased to `|>>`) to implement shell-like
  piping using `Proc::Async` objects, `Code` objects, `Channel`, `Supply` and
  custom objects. A quote construct like operator `px` is provided to create
  `Proc::Async` instances.
  
  ### `px<>`, `px«»`, `px{}`
  
  These operators take a single argument without a space between `px` and the
  argument. It will then split the argument on white spaces. The first element is
  considered a command and the remaining elements arguments to that command.
  If a command does not contain a directory separator, `%*ENV<PATH>` will be searched
  for that command and the first hit used to create a `Proc::Async`. If a
  directory separator is used the first argument is assumed to be a `IO::Path`.
  In both cases the resulting file is checked for existence and filesystem access
  rights to execute it. The exceptions `X::Shell::CommandNotFound` and
  `X::Shell::CommandNoAccess` will be thrown when those tests fail. Please note
  that the file might be deleted between this check and the actual execution of
  the command. The semantics of the provided argument follow general Raku
  subscript rules. As such `px<foo bar>` and `px«foo $bar»` will generate an
  argument list automatically. While the code inside `px{foo, bar}` has to return
  that list by your effort.
  
  ```
  my $proc = px<foo $not-interpolated>; # no interpolation, $PATH is queried
  my $var = "42";
  $proc = px«/usr/bin/meaning $var»; # interpolation and $PATH is not queried
  $proc = px{ 'C:/WINDOWS/SYSTEM32/VIOLATE-PRIVACY'.subst('/', '\') ~ '.exe', secrets.txt };
  await $proc.start;
  
  ```
  
  It is not the resposibility of `px` to actually do anything with the resulting
  `Proc::Async` instance.
  
  The adverb `:timeout(Numerical)` takes seconds with fractions and kills the
  spawned process when at least this time has passed. If the adverb is used
  `Proc::Async::Timeout` will be returned instead of `Proc::Async`. When the
  timeout is hit `X::Proc::Async::Timeout` will be thrown.
  
  ```
  my @a;
  
  loop {
      (px<curl https://www.raku.org>:timeout(60)) |» @a;
      last;
      CATCH {
          when X::Proc::Async::Timeout {
              put „{.command} timed out. Trying again and again and again.“;
          }
      }
  }
  
  ```
  
  ### `multi infix:<|»>` and `multi infix:«|>>»`
  
  The MMD candidates of this operator take two arguments and return a
  `Shell::Pipe` object. This object implements `.sink` and `.start`, whereby the
  first will call the latter. When `.sink` is called all members of a pipe will
  be wired up, started in the right order and `await`ed. In sink context the
  whole pipe will block until the last `Proc::Async` returned from its `.start`
  method.
  
  Members of a pipe can be `Proc::Async`, `Code` objects, `Channel`, `Supply`,
  `IO::Handle`, `IO::Path` and `Array`-like objects. The latter are identified by
  a subset.
  
  ```
  subset Arrayish of Any where { !.isa('Code') && .^can(‚push‘) && .^can(‚list‘) }
  
  ```
  
  `Proc::Async` has its STDOUT fed line-by-line to the next element in the pipe.
  If it is a RHS argument to `|»` its STDIN is written to with the output of the
  LHS. STDERR is left untouched unless the adverbs `:quiet` or `:stderr` are used.
  
  ```
  px<find /tmp> |» px<sort> :quiet; # equivalent to `find /tmp 2>/dev/null | sort 2>/dev/null`;
  
  ```
  
  To capture STDOUT and output to the terminal at the same time, set `$*echo` to
  `on`.
  
  `Code` objects can be used at any place in a pipe. The semantics however vary.
  At the beginning of a pipe the object has to return an Iterable or implement
  `.list`. It will be called once and iterated over its return value. As such we
  support `gather/take`, sequence operators and many buildins. Each value returned
  from an iteration will be added a newline, encoded as utf8 and fed to the next
  member of the pipe. If a code object is in the middle of a pipe it will be
  called each time a line of text is produced to its left and its return value
  fed to the right. If `Nil` is returned this value will be skipped. At the end
  of the pipe the code object is called with each line produced by its left
  neighbour.
  
  ```
  my @a;
  { 2,4,8 … 2**32 } |» px<sha256sum> |» @a;
  px<find /tmp> |» { /a/ ?? .lc !! Nil } |» px<sort>;
  px<find /tmp> |» { .say } :quiet;
  
  ```
  
  `Channel` and `Supplier`/`Supply` can be used at the start and end of a pipe.
  If they are closed, the entire pipe will have STDIN/STDOUT closed. This allows
  a pipe to be controlled from the outside. Any case to complex for a `Code`
  object should therefore be handled with a `Channel`.
  
  ```
  my $c = Channel.new;
  my $sort = px<sort>;
  start {
      await $sort.ready; # this line is optional
      for 1..∞ {
          $c.send: $^a;
      }
  }
  
  Promise.in(60).then: { $c.close }; # a timeout
  $c |» $sort |» px<uniq> |» { .say };
  
  ```
  
  `IO::Path` objects are opened for reading at the begin of a pipe and for writing
  at the end. `IO::Handle` objects are expected to be open already and must be
  open for writing at the end. File handles will not be closed by the pipe.
  
  ```
  px<find /tmp> |» px<sort> |» { .uc  } |» ‚/tmp/sorted.txt‘.IO :quiet;
  
  ```
  
  `Array`-like objects can be used at both ends of a pipe. If used as a first
  element its `.list` method will be called and iterated. At the end of a pipe
  the `.push` method is called. That means lines from a LHS are always added to
  this object.
  
  ```
  class Custom {
      has @.buffer;
      method push -> \v { @.buffer.push: v; @.buffer.shift if +@.buffer > 100; self }
      method list { @.buffer.list }
  }
  
  my $c = Custom.new;
  px<find /usr -iname *.txt> |» $c;
  $c |» px<sort> |» { .say };
  
  ```
  
  ### Slurpy list deconstruction
  
  List deconstruction is supported on the right hand side of `|»`, whereby a
  `Whatever` is required as a first or last element of the declarator list. Used
  as a first element `Whatever` indicated that elements at the beginning of the
  output are to be skipped. With `Whatever` as the last element, only the first
  n-elements are to be kept. This is useful if you are only interested in the
  first or last few elements of the output of a script.
  
  ```
  px<find /usr -iname *.txt> |» my (Whatever, $second-last, $last-line);
  px<find /usr -iname *.txt> |» out ($first-line, $, $third-line, Whatever);
  px<find /usr -iname *.txt> |» my $first-line-only;
  
  ```
  
  That declarator lists don't accept `*` is a limitation of Raku (for now).
  
  A single scalar that is not `Arrayish` will have the first ouput line assigned.
  If there are more lines we `warn`.
  
  To capture STDOUT and output to the terminal at the same time, set `$*echo` to
  `on`.
  
  ## Adverbs
  
  ### `:done(&c(Shell::Pipe $pipe))`
  
  Will be called after the last command of a pipe has exited and before
  `X::Shell::NonZeroExitcode` will be thrown. The argument `$pipe` can be used
  for error handling via `.exitcodes` and introspection via `.pipees`.
  
  ### `:stderr(Arrayish|Code|Channel|IO::Path|IO::Handle|Capture)`
  
  This adverb redirects all STDERR into objects similar to what ‚|»‘ accepts.
  Error text is processed line by line and forwarded as a pair of `(Int $index, Str $text)`. Whereby `$index` is the position of the pipee producing the text
  starting with 0.
  
  ```
  px<find /usr> |» px<sort> |» @a :stderr(@err) :done({.exitcodes});
  for @err.grep({.head == 0}) {
      say ‚find warned about: ‘, .Str;
  }
  
  ```
  
  To log to a file `:stderr()` takes an `IO::Handle` that is open for writing or a
  `IO::Path` that will be opened for writing. To close the handle call
  `.stderr.close` in the `:done()` callback.
  
  Multiple targets for the STDERR stream can be provides with a `Junction`. For
  now only `&` junctions are supported. All targets will receive the same lines
  of text. Whereby no particular order should be assumed.
  
  ```
  px<find /usr> |» px<sort> |» @a :stderr('logfile.txt'.IO & @err & Capture);
  
  ```
  
  The value `Capture` can have an `Int` mixed in to limit capturing to the last n
  lines.
  
  ```
  my $n = 10; # at most $n lines of STDERR are captured
  px<find /usr> |» px<sort> |» @a :stderr(Capture but $n);
  
  ```
  
  ### `:quiet`
  
  The adverb `:quiet` will gobble up all STDERR streams and discard them. This
  can be made the default by setting `$*quiet` to the exported symbol `on`.
  
  ```
  my $*quiet = on;
  my @a;
  px<find -iname *.raku> |» @a;
  
  ```
  
  ## Error handling
  
  When any `Proc::Async` in a pipe finished with a non-zero exitcode the pipe
  returns a `Failure` of `X::Shell::NonZeroExitcode`. Calling `.exitcode` on the
  pipe will mark this `Failure` as handled. The callback in `:done()` is called
  before the Failure can throw. Handling exitcodes by hand has to go there.
  Individual exitcodes of pipe commands are stored in an Array with an index that
  corresponds to the commands position in the pipe. If STDERR output is captured
  with `:stderr(Capture)`. The text per command is available, again as a list of
  `($idx, $text)`. This can be made the default by setting `$*capture-stderr` to
  the exported symbol `on`.
  
  ```
  sub error-handler($pipe) {
      my @a = $pipe.exitcodes;
      for @a {
          .command.say;
          .exitcode.say;
          .STDERR.say;
      }
  }
  px«find /usr» |» px«sort» :done(&error-handler) :stderr(Capture);
  
  ```
  
  The class `Shell::Pipe::Exitcode` supports smartmatching against `Int`, `Str`
  and `Regex`. This can be used for handling exceptions.
  
  ```
  px«find /usr» |» px«sort» :stderr(Capture);
  
  CATCH {
      when X::Shell::NonZeroExitcode { 
          for .pipe.exitcodes {
              when ‚find‘ & 1 & /‘(<![‘]>+)‘: Permission denied/ {
                  say „did not look in $0“;
              }
          }
      }
  
  }
  
  ```
  
  ## Exceptions
  
  ```
  CATCH {
      when X::Shell::CommandNotFound {
          say .cmd ~ ‚was not found‘;
      }
      when X::Shell::CommandNoAccess {
          say .cmd ~ ‚was unaccessable‘;
      }
      when X::Shell::NonZeroExitcode {
          for .pipe.exitcodes {
              say .command, .exitcode, .pipe.stderr ~~ Capture ?? .STDERR !! ‚‘;
              when ‚find‘ & 1 & /‘(<![‘]>+)‘: Permission denied/ {
                  say „did not look in $0“;
              }
          }
      }
      when X::Shell::NoExitcodeYet {
          say .^name, „\n“, .message;
      }
  }
  
  ```
  
  ### Refining Exceptions
  
  The exceptions `X::Shell::CommandNotFound` and `X::Shell::CommandNoAccess` are
  refinable. This means the error message can be tweaked with the class method
  `.refine`. This method takes two `Callable`s. When `.message` is called with
  the exception instance and expected to return `Bool`. On `True` the 2nd
  callback is called with the exception instance and supposed to return a text.
  This text will be used instead of the default text and returned from
  `.message`. Replacing this message will act on the class and even on created
  but yet to be thrown exceptions.
  
  ```
  X::Shell::CommandNotFound.refine(
      (my &b = {.cmd eq ‚raku‘}),
      { ‚Please install Rakudo with `apt install rakudo`.‘ }
  );
  X::Shell::CommandNotFound.refine(&b, :revert);
  X::Shell::CommandNotFound.refine(:revert-all);
  
  ```
  
  The method `.revert` also takes one `Callable` and the adverb `:revert` to
  remove one refinement or all refinements with `:revert-all`.
  
  ### X::Shell::CommandNotFound
  
  Will be thrown by `px«»` or when the pipe is started if the file used as a
  command is not found. The meaning of "not found" depends on the OS. If the
  command was searched for in `%*ENV<PATH>`, that path will be shown in the
  exception message. This exception also checks for dangling symlinks and
  provides an alternate error message for this case.
  
  ### X::Shell::CommandNoAccess
  
  Will be thrown by `px«»` or when the pipe is started if the file used as a
  command exists but can not be executed. Filesystem access rights depend on the
  OS.
  
  ### X::Shell::NonZeroExitcode
  
  This will be thrown after the last pipee exits and holds a `Shell::Pipe` in
  `.pipe`. If `:stderr(Capture)` is used the exception message contains all error
  text grouped by the shell command names. When an `Int` is mixed in, only that
  many lines will be captured.
  
  The command line will be clipped at 180 characters. This limit can be changed
  by setting `$*max-exitcode-command` to any `Int` or `Inf`.
  
  ### X::Shell::NoExitcodeYet
  
  Will be thrown if `.exitcodes` is accessed before the pipe finished. Please
  note that filling the underlying Array is not atomic. When or after `.done` is
  called using `.exitcodes` is fine.
  
  ### Colour control
  
  Exceptions will print their error messges in red to STDERR if send to a
  terminal. This can be controlled by `%*ENV<SHELLPIPINGNOCOLOR>` and
  `$*colored-exceptions`. The environment variable can be set to any value. The
  dynamic variable to the exported symbols `on` and `off` whereby `on` is the
  default when the variable is not declared by any caller.
  
  ```
  use Shell::Piping;
  use Shell::Piping::Whereceptions;
  
  sub s(IO(Str) $f where &it-is-a-file) {
  }
  
  my $*colored-exceptions = off;
  s('/foo/bar');
  
  my @a;
  px<find /tmp/> |» @a;
  
  ```
  
  # Wherecetions
  
  Are subs to be used in where clauses to test for conditions that would throw
  later on. Whereceptions will output to STDERR in red unless
  `%*ENV<SHELLPIPINGNOCOLOR>` is set to any value. When sensible there will be
  checks for dangling symlinks and an alternate error message will be returned by
  the exceptions. All exceptions are subclasses of `X::IO::Whereception`.
  
  ## SYNOPSIS
  
  ```
  sub works-with-files(IO::Path(Str) $file where &it-is-a-file) {
      say ‚answer‘ for $file.lines.grep(42);
  }
  
  sub works-with-directories(IO::Path(Str) $dir where &it-is-a-directory) {
      for $dir {
          .&works-with-files when .IO.f;
          .IO.dir()».&?BLOCK when .IO.d;
      }
  }
  
  }
  
  sub will-shell-out(IO::Path(Str) $file where &it-is-executable) {
      px<find -iname '42'> |» px«$file» |» (my @stdout);
  }
  
  ```
  
  ### `sub it-is-a-file(IO() $f)`
  
  Will call `.e` and `.f` and throw `X::IO::FileNotFound`.
  
  ### `sub it-is-a-directory(IO() $d)`
  
  Will call `.d` and throw `X::IO::DirectoryNotFound`.
  
  ### `sub it-is-executable(IO() $exec)`
  
  Will call `.x` and throw `X::IO::FileNotExecutable`.
  
  ## LICENSE
  
  All files (unless noted otherwise) can be used, modified and redistributed
  under the terms of the Artistic License Version 2. Examples (in the
  documentation, in tests or distributed as separate files) can be considered
  public domain.
  
  ⓒ2020 Wenzel P. P. Peppmeyer
  