## structures.md
Data structures

How Raku deals with data structures and what we can expect from them

# [Scalar structures](#Data_structures "go to top of document")[§](#Scalar_structures "direct link")

Some classes do not have any *internal* structure and to access parts of them, specific methods have to be used. Numbers, strings, and some other monolithic classes are included in that class. They use the `$` sigil, although complex data structures can also use it.

```raku
my $just-a-number = 7;
my $just-a-string = "8";
```

There is a [`Scalar`](/type/Scalar) class, which is used internally to assign a default value to variables declared with the `$` sigil.

```raku
my $just-a-number = 333;
say $just-a-number.VAR.^name; # OUTPUT: «Scalar␤»
```

Any complex data structure can be *scalarized* by using the [item contextualizer `$`](/type/Any#index-entry-%24_(item_contextualizer)):

```raku
(1, 2, 3, $(4, 5))[3].VAR.^name.say; # OUTPUT: «Scalar␤»
```

However, this means that it will be treated as such in the context they are. You can still access its internal structure.

```raku
(1, 2, 3, $(4, 5))[3][0].say; # OUTPUT: «4␤»
```

An interesting side effect, or maybe intended feature, is that scalarization conserves identity of complex structures.

```raku
for ^2 {
     my @list = (1, 1);
     say @list.WHICH;
} # OUTPUT: «Array|93947995146096␤Array|93947995700032␤»
```

Every time `(1, 1)` is assigned, the variable created is going to be different in the sense that `===` will say it is; as it is shown, different values of the internal pointer representation are printed. However

```raku
for ^2 {
  my $list = (1, 1);
  say $list.WHICH
} # OUTPUT: «List|94674814008432␤List|94674814008432␤»
```

In this case, `$list` is using the Scalar sigil and thus will be a [`Scalar`](/type/Scalar). Any scalar with the same value will be exactly the same, as shown when printing the pointers.

# [Complex data structures](#Data_structures "go to top of document")[§](#Complex_data_structures "direct link")

Complex data structures fall in two different broad categories: [`Positional`](/type/Positional), or list-like and [`Associative`](/type/Associative), or key-value pair like, according to how you access its first-level elements. In general, complex data structures, including objects, will be a combination of both, with object properties assimilated to key-value pairs. While all objects subclass [`Mu`](/type/Mu), in general complex objects are instances of subclasses of [`Any`](/type/Any). While it is theoretically possible to mix in [`Positional`](/type/Positional) or [`Associative`](/type/Associative) without doing so, most methods applicable to complex data structures are implemented in [`Any`](/type/Any).

Navigating these complex data structures is a challenge, but Raku provides a couple of functions that can be used on them: [`deepmap`](/routine/deepmap) and [`duckmap`](/routine/duckmap). While the former will go to every single element, in order, and do whatever the block passed requires,

```raku
say [[1, 2, [3, 4]],[[5, 6, [7, 8]]]].deepmap( *.elems );
# OUTPUT: «[[1 1 [1 1]] [1 1 [1 1]]]␤»
```

which returns `1` because it goes to the deeper level and applies `elems` to them, `duckmap` can perform more complicated operations:

```raku
say [[1, 2, [3, 4]], [[5, 6, [7, 8]]]].duckmap:
   -> $array where .elems == 2 { $array.elems };
# OUTPUT: «[[1 2 2] [5 6 2]]␤»
```

In this case, it dives into the structure, but returns the element itself if it does not meet the condition in the block (`1, 2`), returning the number of elements of the array if it does (the two `2`s at the end of each subarray).

Since `deepmap` and `duckmap` are [`Any`](/type/Any) methods, they also apply to Associative arrays:

```raku
say %( first => [1, 2], second => [3,4] ).deepmap( *.elems );
# OUTPUT: «{first => [1 1], second => [1 1]}␤»
```

Only in this case, they will be applied to every list or array that is a value, leaving the keys alone.

[`Positional`](/type/Positional) and [`Associative`](/type/Associative) can be turned into each other.

```raku
say %( first => [1, 2], second => [3,4] ).list[0];
# OUTPUT: «second => [3 4]␤»
```

However, in this case, and for Rakudo >= 2018.05, it will return a different value every time it runs. A hash will be turned into a list of the key-value pairs, but it is guaranteed to be disordered. You can also do the operation in the opposite direction, as long as the list has an even number of elements (odd number will result in an error):

```raku
say <a b c d>.Hash # OUTPUT: «{a => b, c => d}␤»
```

But

```raku
say <a b c d>.Hash.kv # OUTPUT: «(c d a b)␤»
```

will obtain a different value every time you run it; [`kv`](/type/Pair#method_kv) turns every [`Pair`](/type/Pair) into a list.

Complex data structures are also generally [`Iterable`](/type/Iterable). Generating an [iterator](/routine/iterator) out of them will allow the program to visit the first level of the structure, one by one:

```raku
.say for 'א'..'ס'; # OUTPUT: «א␤ב␤ג␤ד␤ה␤ו␤ז␤ח␤ט␤י␤ך␤כ␤ל␤ם␤מ␤ן␤נ␤ס␤»
```

`'א'..'ס'` is a [`Range`](/type/Range), a complex data structure, and with `for` in front it will iterate until the list is exhausted. You can use `for` on your complex data structures by overriding the [iterator](/routine/iterator) method (from role [`Iterable`](/type/Iterable)):

```raku
class SortedArray is Array {
  method iterator() {
    self.sort.iterator
  }
};
my @thing := SortedArray.new([3,2,1,4]);
.say for @thing; # OUTPUT: «1␤2␤3␤4␤»
```

`for` calls directly the `iterator` method on `@thing` making it return the elements of the array in order. Much more on [iterating on the page devoted to it](/language/iterating).

# [Functional structures](#Data_structures "go to top of document")[§](#Functional_structures "direct link")

Raku is a functional language and, as such, functions are first-class *data* structures. Functions follow the [`Callable`](/type/Callable) role, which is the 4th element in the quartet of fundamental roles. [`Callable`](/type/Callable) goes with the `&` sigil, although in most cases it is elided for the sake of simplicity; this sigil elimination is always allowed in the case of [`Callable`](/type/Callable)s.

```raku
my &a-func= { (^($^þ)).Seq };
say a-func(3), a-func(7); # OUTPUT: «(0 1 2)(0 1 2 3 4 5 6)␤»
```

[`Block`](/type/Block)s are the simplest callable structures, since [`Callable`](/type/Callable)s cannot be instantiated. In this case we implement a block that logs events and can retrieve them:

```raku
my $logger = -> $event, $key = Nil  {
  state %store;
  if ( $event ) {
    %store{ DateTime.new( now ) } = $event;
  } else {
    %store.keys.grep( /$key/ )
  }
}
$logger( "Stuff" );
$logger( "More stuff" );
say $logger( Nil, "2018-05-28" ); # OUTPUT: «(Stuff More stuff)␤»
```

A [`Block`](/type/Block) has a [`Signature`](/type/Signature), in this case two arguments, the first of which is the event that is going to be logged, and the second is the key to retrieve the events. They will be used in an independent way, but its intention is to showcase the use of a [state variable](/syntax/state) that is kept from every invocation to the next. This state variable is encapsulated within the block, and cannot be accessed from outside except by using the simple API the block provides: calling the block with a second argument. The two first invocations log two events, the third invocation at the bottom of the example use this second type of call to retrieve the stored values. [`Block`](/type/Block)s can be cloned:

```raku
my $clogger = $logger.clone;
$clogger( "Clone stuff" );
$clogger( "More clone stuff" );
say $clogger( Nil, "2018-05-28" );
# OUTPUT: «(Clone stuff More clone stuff)␤»

```

And cloning will reset the state variable; instead of cloning, we can create *façades* that change the API. For instance, eliminate the need to use [`Nil`](/type/Nil) as first argument to retrieve the log for a certain date:

```raku
my $gets-logs = $logger.assuming( Nil, * );
$logger( %(changing => "Logs") );
say $gets-logs( "2018-05-28" );
# OUTPUT: «({changing => Logs} Stuff More stuff)␤»

```

[`assuming`](/type/Code#method_assuming) wraps around a block call, giving a value (in this case, [`Nil`](/type/Nil)) to the arguments we need, and passing on the arguments to the other arguments we represent using `*`. In fact, this corresponds to the natural language statement "We are calling `$logger` *assuming* the first argument is [`Nil`](/type/Nil)". We can slightly change the appearance of these two Blocks to clarify they are actually acting on the same block:

```raku
my $Logger = $logger.clone;
my $Logger::logs = $Logger.assuming( *, Nil );
my $Logger::get = $Logger.assuming( Nil, * );
$Logger::logs( <an array> );
$Logger::logs( %(key => 42) );
say $Logger::get( "2018-05-28" );

```

Although `::` is generally used for invocation of class methods, it is actually a valid part of the name of a variable. In this case we use them conventionally to simply indicate `$Logger::logs` and `$Logger::get` are actually calling `$Logger`, which we have capitalized to use a class-like appearance. The point of this tutorial is that using functions as first-class citizens, together with the use of state variables, allows the use of certain interesting design patterns such as this one.

As such first class data structures, callables can be used anywhere another type of data can.

```raku
my @regex-check = ( /<alnum>/, /<alpha>/, /<punct>/ );
say @regex-check.map: "33af" ~~ *;
# OUTPUT: «(｢3｣␤ alnum => ｢3｣ ｢a｣␤ alpha => ｢a｣ Nil)␤»
```

Regexes are actually a type of callable:

```raku
say /regex/.does( Callable ); # OUTPUT: «True␤»
```

And in the example above we are calling regexes stored in an array, and applying them to a string literal.

Callables are composed by using the [function composition operator ∘](/language/operators#infix_o,_infix_%E2%88%98):

```raku
my $typer = -> $thing { $thing.^name ~ ' → ' ~ $thing };
my $Logger::withtype = $Logger::logs ∘ $typer;
$Logger::withtype( Pair.new( 'left', 'right' ) );
$Logger::withtype( ¾ );
say $Logger::get( "2018-05-28" );
# OUTPUT: «(Pair → left right Rat → 0.75)␤»

```

We are composing `$typer` with the `$Logger::logs` function defined above, obtaining a function that logs an object preceded by its type, which can be useful for filtering, for instance. `$Logger::withtype` is, in fact, a complex data structure composed of two functions which are applied in a serial way, but every one of the callables composed can keep state, thus creating complex transformative callables, in a design pattern that is similar to object composition in the object oriented realm. You will have to choose, in every particular case, what is the programming style which is most suitable for your problem.

# [Defining and constraining data structures](#Data_structures "go to top of document")[§](#Defining_and_constraining_data_structures "direct link")

Raku has different ways of defining data structures, but also many ways to constrain them so that you can create the most adequate data structure for every problem domain. [`but`](/routine/but), for example, mixes roles or values into a value or a variable:

```raku
my %not-scalar := %(2 => 3) but Associative[Int, Int];
say %not-scalar.^name; # OUTPUT: «Hash+{Associative[Int, Int]}␤»
say %not-scalar.of;    # OUTPUT: «Associative[Int, Int]␤»
%not-scalar{3} = 4;
%not-scalar<thing> = 3;
say %not-scalar;       # OUTPUT: «{2 => 3, 3 => 4, thing => 3}␤»

```

In this case, `but` is mixing in the `Associative[Int, Int]` role; please note that we are using binding so that the type of the variable is the one defined, and not the one imposed by the `%` sigil; this mixed-in role shows in the `name` surrounded by curly braces. What does that really mean? That role includes two methods, `of` and `keyof`; by mixing the role in, the new `of` will be called (the old `of` would return [`Mu`](/type/Mu), which is the default value type for Hashes). However, that is all it does. It is not really changing the type of the variable, as you can see since we are using any kind of key and values in the next few statements.

However, we can provide new functionality to a variable using this type of mixin:

```raku
role Lastable {
  method last() {
    self.sort.reverse[0]
  }
}
my %hash-plus := %( 3 => 33, 4 => 44) but Lastable;
say %hash-plus.sort[0]; # OUTPUT: «3 => 33␤»
say %hash-plus.last;    # OUTPUT: «4 => 44␤»

```

In `Lastable` we use the universal `self` variable to refer to whatever object this particular role is mixed in; in this case it will contain the hash it is mixed in with; it will contain something else (and possibly work some other way) in another case. This role will provide the `last` method to any variable it's mixed with, providing new, attachable, functionalities to *regular* variables. Roles can even be [added to existing variables using the `does` keyword](/language/objects#Mixins_of_roles).

[Subsets](/language/typesystem#subset) can also be used to constrain the possible values a variable might hold; they are Raku attempt at [gradual typing](https://en.wikipedia.org/wiki/Gradual_typing); it is not a full attempt, because subsets are not really types in a strict sense, but they allow runtime type checking. It adds type-checking functionality to regular types, so it helps create a richer type system, allowing things like the one shown in this code:

```raku
subset OneOver where (1/$_).Int == 1/$_;
my OneOver $one-fraction = ⅓;
say $one-fraction; # OUTPUT: «0.333333␤»

```

On the other hand, `my OneOver $ = ⅔;` will cause a type-check error. Subsets can use [`Whatever`](/type/Whatever), that is, `*`, to refer to the argument; but this will be instantiated every time you use it to a different argument, so if we use it twice in the definition we would get an error. In this case we are using the topic single variable, `$_`, to check the instantiation. Subsetting can be done directly, without the need of declaring it, in [signatures](/language/typesystem#subset).

# [Infinite structures and laziness](#Data_structures "go to top of document")[§](#Infinite_structures_and_laziness "direct link")

It might be assumed that all the data contained in a data structure is actually *there*. That is not necessarily the case: in many cases, for efficiency reasons or simply because it is not possible, the elements contained in a data structure only jump into existence when they are actually needed. This computation of items as they are needed is called [reification](/language/glossary#Reify) or vivification.

```raku
# A list containing infinite number of un-reified Fibonacci numbers:
my @fibonacci = 1, 1, * + * … ∞;

# We reify 10 of them, looking up the first 10 of them with array index:
say @fibonacci[^10]; # OUTPUT: «(1 1 2 3 5 8 13 21 34 55)␤»

# We reify 5 more: 10 we already reified on previous line, and we need to
# reify 5 more to get the 15th element at index 14. Even though we need only
# the 15th element, the original Seq still has to reify all previous elements:
say @fibonacci[14]; # OUTPUT: «987␤»

```

Above we were reifying a [`Seq`](/type/Seq) we created with the [sequence operator](/language/operators#infix_...), but other data structures use the concept as well. For example, an un-reified [`Range`](/type/Range) is just the two end points. In some languages, calculating the sum of a huge range is a lengthy and memory-consuming process, but Raku calculates it instantly:

```raku
say sum 1 .. 9_999_999_999_999; # OUTPUT: «49999999999995000000000000␤»
```

Why? Because the sum can be calculated *without* vivifying the Range; that is, without figuring out all the elements it contains. This is why this feature exists. You can even make your own things reify-on-demand, using [`gather` and `take`](/syntax/gather%20take):

```raku
my $seq = gather {
    say "About to make 1st element"; take 1;
    say "About to make 2nd element"; take 2;
}
say "Let's reify an element!";
say $seq[0];
say "Let's reify more!";
say $seq[1];
say "Both are reified now!";
say $seq[^2];

# OUTPUT:
# Let's reify an element!
# About to make 1st element
# 1
# Let's reify more!
# About to make 2nd element
# 2
# Both are reified now!
# (1 2)
```

Following the output above, you can see the print statements *inside* the `gather` got executed only when we reified the individual elements while looking up an element. Also note that the elements got reified just once. When we printed the same elements again on the last line of the example, the messages inside `gather` was no longer printed. This is because the construct used already-reified elements from the [`Seq`](/type/Seq)'s cache.

Note that above we assigned the `gather` to a [`Scalar`](/type/Scalar) container (the `$` sigil), not the [`Positional`](/type/Positional) one (the `@` sigil). The reason is that the `@`-sigiled variables are *mostly eager*. What this means is they *reify the stuff assigned to them* right away *most of the time*. The only time they don't do it is when the items are known to be [`is-lazy`](/routine/is-lazy), like our sequence generated with infinity as the end point. Were we to assign the `gather` to an `@`-variable, the `say` statements inside of it would've been printed right away.

Another way to vivify the list fully is by calling [`.elems`](/routine/elems) on it. This is the reason why checking whether a list contains any items is best done by using `.Bool` method (or just using `if @array { … }`), since you don't need to reify *all* the elements to find out if there are `any` of them.

There are times where you *do* want to fully-reify a list before doing something. For example, the [`IO::Handle.lines`](/type/IO/Handle#routine_lines) returns a [`Seq`](/type/Seq). The following code contains a bug; keeping reification in mind, try to spot it:

```raku
my $fh = "/tmp/bar".IO.open;
my $lines = $fh.lines;
close $fh;
say $lines[0];

```

We open a [filehandle](/type/IO/Handle), then assign return of [`.lines`](/type/IO/Handle#routine_lines) to a [`Scalar`](/type/Scalar) variable, so the returned [`Seq`](/type/Seq) does not get reified right away. We then [`close`](/routine/close) the filehandle, and try to print an element from `$lines`.

The bug in the code is by the time we reify the `$lines` [`Seq`](/type/Seq) on the last line, we've *already closed* the filehandle. When the [`Seq`](/type/Seq)'s iterator tries to generate the item we've requested, it results in the error about attempting to read from a closed handle. So, to fix the bug we can either assign to an `@`-sigiled variable or call [`.elems`](/routine/elems) on `$lines` before closing the handle:

```raku
my $fh = "/tmp/bar".IO.open;
my @lines = $fh.lines;
close $fh;
say @lines[0]; # no problem!

```

We can also use any function whose side effect is reification, like `.elems` mentioned above:

```raku
my $fh = "/tmp/bar".IO.open;
my $lines = $fh.lines;
say "Read $lines.elems() lines"; # reifying before closing handle
close $fh;
say $lines[0]; # no problem!

```

Using [eager](/routine/eager) will also reify the whole sequence:

```raku
my $fh = "/tmp/bar".IO.open;
my $lines = eager $fh.lines; # Uses eager for reification.
close $fh;
say $lines[0];

```

# [Introspection](#Data_structures "go to top of document")[§](#Introspection "direct link")

Languages that allow [introspection](https://en.wikipedia.org/wiki/Type_introspection) like Raku have functionalities attached to the type system that let the developer access container and value metadata. This metadata can be used in a program to carry out different actions depending on their value. As it is obvious from the name, metadata are extracted from a value or container via the metaclass.

```raku
my $any-object = "random object";
my $metadata = $any-object.HOW;
say $metadata.^mro;                   # OUTPUT: «((ClassHOW) (Any) (Mu))␤»
say $metadata.can( $metadata, "uc" ); # OUTPUT: «(uc uc)␤»
```

With the first `say` we show the class hierarchy of the metamodel class, which in this case is [`Metamodel::ClassHOW`](/type/Metamodel/ClassHOW). It inherits directly from [`Any`](/type/Any), meaning any method there can be used; it also mixes in several roles which can give you information about the class structure and functions. But one of the methods of that particular class is [`can`](/type/Metamodel/ClassHOW#method_can), which we can use to look up whether the object can use the `uc` (uppercase) method, which it obviously can. However, it might not be so obvious in some other cases, when roles are mixed in directly into a variable. For instance, in the [case of `%hash-plus` defined above](#Defining_and_constraining_data_structures):

```raku
say %hash-plus.^can("last"); # OUTPUT: «(last)␤»

```

In this case we are using the *syntactic sugar* for `HOW.method`, `^method`, to check if your data structure responds to that method; the output, which shows the name of the methods that match, certifies that we can use it.

See also [this article on class introspection](https://perl6advent.wordpress.com/2015/12/19/day-19-introspection/) on how to access class properties and methods, and use it to generate test data for a class; this [Advent Calendar article describes the metaobject protocol](https://perl6advent.wordpress.com/2010/12/22/day-22-the-meta-object-protocol/) extensively.


## math.md
Doing math with Raku

Different mathematical paradigms and how they are implemented in this language

# [Arithmetic](#Doing_math_with_Raku "go to top of document")[§](#Arithmetic "direct link")

Raku can do arithmetic using different data types. [`Num`](/type/Num), [`Rat`](/type/Rat) and [`Complex`](/type/Complex) can all operate as a [field under the operations of addition, subtraction, multiplication and division](https://en.wikipedia.org/wiki/Field_(mathematics)) (technically, it should be noted that data types dealing with floating point number representations are not a field in the mathematical sense due to the inherent imprecisions of their arithmetic. However, they constitute an approximate enough, computer friendly version of such mathematical objects for most of the cases). The equivalent mathematical fields are:

| Raku class | Field |
| --- | --- |
| Rat | ℚ |
| Num | ℝ |
| Complex | ℂ |

The integers ([`Int`](/type/Int)), or ℤ, as they're usually called in mathematics, are not a mathematical field but rather a ring, since they are not closed under multiplicative inverses. However, if the integer division `div` is used, their operations will always yield other integers; if `/` is used, on the other hand, in general the result will be a [`Rat`](/type/Rat).

Besides, [`Int`](/type/Int) can do infinite-precision arithmetic (or at least infinite as memory allows; `Numeric overflow` can still occur), without falling back to [`Num`](/type/Num) if the number is too big:

```raku
my @powers = 2, 2 ** * ... Inf; say @powers[4].chars; # OUTPUT: «19729␤»
```

Also strictly speaking, the Rational class that behaves like a mathematical field is [`FatRat`](/type/FatRat). For efficiency reasons, operating with [`Rat`](/type/Rat)s will fall back to [`Num`](/type/Num) when the numbers are big enough or when there is a big difference between numerator and denominator. [`FatRat`](/type/FatRat) can work with arbitrary precision, the same as the default [`Int`](/type/Int) class.

Some modules in the ecosystem can work with additional data types mathematically:

* [`Math::Vector`](https://github.com/colomon/Math-Vector) basic operations for [vectors](https://en.wikipedia.org/wiki/Coordinate_vector).
* [`Math::Matrix`](https://github.com/pierre-vigier/Perl6-Math-Matrix) operates on [matrices rings over numeric rings](https://en.wikipedia.org/wiki/Matrix_(mathematics)).
* [`Math::Quaternion`](https://github.com/Util/Perl6-Math-Quaternion) operates on the [quaternion algebra, ℍ](https://en.wikipedia.org/wiki/Quaternion), which are a generalization of complex numbers.
* [`Math::Polynomial`](https://github.com/colomon/Math-Polynomial) works with polynomials, and is able to do simple arithmetic with them.
* [`Math::Symbolic`](https://github.com/raydiak/Math-Symbolic), for symbolic math.

Numbers are duck-typed automatically to the numeric class they actually represent:

```raku
.^name.say for (4, ⅗, 1e-9, 3+.1i); # OUTPUT: «Int␤Rat␤Num␤Complex␤»
```

Arithmetic operations are performed by taking into account the type of operands:

```raku
say .33-.22-.11 == 0; # OUTPUT: «True␤»
```

In this case, all numbers are interpreted as [`Rat`](/type/Rat)s, which makes the operation exact. In general, most other languages would interpret them as floating point numbers, which can also be achieved in Raku if needed:

```raku
say .33.Num -.22.Num - .11.Num; # OUTPUT: «1.3877787807814457e-17␤»
```

For cases such as this, Raku also includes an `approximately equal` operator, [≅](/language/operators#infix_=~=)

```raku
say .33.Num -.22.Num - .11.Num ≅ 0; # OUTPUT: «True␤»
```

# [Sets](#Doing_math_with_Raku "go to top of document")[§](#Sets "direct link")

Raku includes the [`Set`](/type/Set) data type, as well as support for [most set operations](/language/setbagmix#Operators_with_set_semantics). [Union and intersection](https://en.wikipedia.org/wiki/Algebra_of_sets) are not only native operations, they use their *natural* symbols, ∩ and ∪. For instance, this code would check the fundamental laws of the arithmetic of sets for a limited number of sets:

```raku
my @arbitrary-numbers = ^100;
my \U = @arbitrary-numbers.Set;

my @sets;

@sets.push: Set.new( @arbitrary-numbers.pick( @arbitrary-numbers.elems.rand)) for @arbitrary-numbers;

my (@union, @intersection);

for @sets -> $set {
    @union.push: $set ∩ $set === $set;
    @intersection.push: $set ∪ $set === $set;
}

say "Idempotent union is ", so @union.all;
# OUTPUT: «Idempotent union is True␤»
say "Idempotent intersection is ", so @intersection.all;
# OUTPUT: «Idempotent intersection is True␤»
my (@universe, @empty-set, @id-universe, @id-empty);

for @sets -> \A {
    @universe.push: A ∪ U === U;
    @id-universe.push: A ∩ U === A;
    @empty-set.push: A ∩ ∅ === ∅;
    @id-empty.push: A ∪ ∅ === A;
}

say "Universe dominates ", so @universe.all;    # OUTPUT: «Universe dominates True␤»
say "Empty set dominates ", so @empty-set.all;  # OUTPUT: «Empty set dominates True␤»

say "Identity with U ", so @id-universe.all;    # OUTPUT: «Identity with U True␤»
say "Identity with ∅ ", so @id-empty.all;       # OUTPUT: «Identity with ∅ True»

```

In this code, which uses the [empty set](/language/setbagmix#term_%E2%88%85) which is already defined by Raku, not only do we check if the equalities in the algebra of sets hold, we also use, via [sigilless variables](/language/variables#index-entry-\_(sigilless_variables)) and the Unicode form of the set operators, expressions that are as close as possible to the original form; `A ∪ U === U`, for example, except for the use of the [value identity operator `===`](/routine/===) is very close to the actual mathematical expression in the [Wikipedia entry](https://en.wikipedia.org/wiki/Algebra_of_sets).

We can even test De Morgan's law, as in the code below:

```raku
my @alphabet = 'a'..'z';
my \U = @alphabet.Set;
sub postfix:<⁻>(Set $a) { U ⊖ $a }
my @sets;
@sets.push: Set.new( @alphabet.pick( @alphabet.elems.rand)) for @alphabet;
my ($de-Morgan1,$de-Morgan2) = (True,True);
for @sets X @sets -> (\A, \B){
    $de-Morgan1 &&= (A ∪ B)⁻  === A⁻ ∩ B⁻;
    $de-Morgan2 &&= (A ∩ B)⁻  === A⁻ ∪ B⁻;
}
say "1st De Morgan is ", $de-Morgan1;
say "2nd De Morgan is ", $de-Morgan2;

```

We declare `⁻` as the *complement* operation, which computes the symmetrical difference ⊖ between the Universal set `U` and our set. Once that is declared, it is relatively easy to express operations such as the complementary of the union of A and B, `(A ∪ B)⁻`, with a notation that is very close to the original mathematical notation.

# [Sequences](#Doing_math_with_Raku "go to top of document")[§](#Sequences "direct link")

A [sequence](https://en.wikipedia.org/wiki/Sequence) is an *enumerated* collection of objects in which repetitions are allowed, and also a first-class data type in Raku called [`Seq`](/type/Seq). [`Seq`](/type/Seq) is able to represent infinite sequences, like the natural numbers:

```raku
my \𝕟 = 1,2 … ∞;
say 𝕟[3];         # OUTPUT: «4␤»
```

Infinite sequences use ∞, `Inf` or `*` (Whatever) as terminator. […](/language/operators#infix_...) is the list generator, which in fact can understand arithmetic and geometric progression sequences as long as you insert the first numbers:

```raku
say 1,5,9 … * > 100;
# OUTPUT: «(1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101)␤»
say 1,3,9 … * > 337; # OUTPUT: «(1 3 9 27 81 243 729)␤»
```

The first sequence will be terminated when the generated number is bigger than 100; the second sequence, which is a geometric progression, when it is bigger than 337.

The fact that an arbitrary generator can be used makes easy to generate sequences such as [Fibonacci numbers](https://en.wikipedia.org/wiki/Fibonacci_number):

```raku
say 1,1, * + * … * > 50;#  OUTPUT: «(1 1 2 3 5 8 13 21 34 55)␤»
```

We can, in fact, compute the approximation to the [golden ratio](https://en.wikipedia.org/wiki/Golden_ratio) this way:

```raku
my @phis = (2.FatRat, 1 + 1 / * ... *);
my @otherphi = (1 - @phis[200], 1 + 1 / * ... *);
say @otherphi[^10, |(20, 30 ... 100)];  # OUTPUT:
# «((-0.61803398874989484820458683436563811772030918
# -0.61803398874989484820458683436563811772030918
# -0.61803398874989484820458683436563811772030918
# -0.61803398874989484820458683436563811772030918
# -0.61803398874989484820458683436563811772030918
# -0.618033…»
```

The [Math::Sequences](https://github.com/ajs/perl6-Math-Sequences) module includes many mathematical sequences, already defined for you. It has many [sequences from the encyclopedia](https://oeis.org/), some of them with their original name, such as ℤ.

Some set operators also operate on sequences, and they can be used to find out if an object is part of it:

```raku
say 876 ∈ (7,14 … * > 1000) ; # OUTPUT: «False␤»
```

In this particular case, we can find out if `876` is a multiple of 7 straight away, but the same principle holds for other sequences using complicated generators. And we can use set inclusion operators too:

```raku
say (55,89).Set ⊂ (1,1, * + * … * > 200); # OUTPUT: «True␤»
```

That said, it does not take into account if it is effectively a subsequence, just the presence of the two elements here. Sets have no order, and even if you don't explicitly cast the subsequence into a Set or explicitly cast it into a [`Seq`](/type/Seq) it will be coerced into such for the application of the inclusion operator.

# [Mathematical constants](#Doing_math_with_Raku "go to top of document")[§](#Mathematical_constants "direct link")

Raku includes a set of mathematical constants:

```raku
say π; # OUTPUT: «3.141592653589793␤»
say τ; # Equivalent to 2π; OUTPUT: «6.283185307179586␤»
say 𝑒; # OUTPUT: «2.718281828459045␤»
```

These constants are also available through [ASCII equivalents](/language/unicode_ascii): `e`, `pi` and `tau`.

The [Math::Constants](https://github.com/JJ/p6-math-constants) module includes an additional series of physical and mathematical constants such as the previously mentioned golden ratio φ or the Planck's constant ℎ.

Since Raku allows for definition of variables that use Unicode graphemes, and also variable and constant names without any kind of sigil, it is considered a good practice to use the actual mathematical name of concepts to denominate them wherever possible.

# [Numerical integration of ordinary differential equations](#Doing_math_with_Raku "go to top of document")[§](#Numerical_integration_of_ordinary_differential_equations "direct link")

Raku is an amazing programming language, and you can do a lot of cool math with it. A great amount of work during an applied mathematician's work is to simulate the models they create. For this reason, in every coding language, a numerical integrator is a must-have. Learning how to do this in Raku can be very useful.

## [Requirements](#Doing_math_with_Raku "go to top of document")[§](#Requirements "direct link")

In Raku there are some modules in the ecosystem that can make it easier:

* [`Math::Model`](https://github.com/moritz/Math-Model) which lets you write mathematical and physical models in an easy and natural way.
* [`Math::RungeKutta`](https://github.com/moritz/Math-RungeKutta) Runge-Kutta integration for systems of ordinary, linear differential equations.

For this example we are going to use [`Math::Model`](https://github.com/moritz/Math-Model) for its useful syntax, but remember that this module requires [`Math::RungeKutta`](https://github.com/moritz/Math-RungeKutta) as well. Simply install them with *zef* before using these examples.

## [Malthus model](#Doing_math_with_Raku "go to top of document")[§](#Malthus_model "direct link")

Let's start with the *'Hello World'* of mathematical Ecology: [Malthusian growth model](https://en.wikipedia.org/wiki/Malthusian_growth_model). A Malthusian growth model, sometimes called a simple exponential growth model, is essentially exponential growth based on the idea of the function being proportional to the speed to which the function grows. The equation, then, looks like this:

*dx/dt = g\*x*

*x(0) = x\_0*

Where *g* is the population growth rate, sometimes called Malthusian parameter.

How can we translate that into Raku? Well Math::Model brings some help with a very understandable way to do that:

```raku
use Math::Model;

my $m = Math::Model.new(
    derivatives => {
        velocity => 'x',
    },
    variables   => {
        velocity           => { $:growth_constant * $:x },
        growth_constant    => { 1 }, # basal growth rate
    },
    initials    => {
        x       => 3,
    },
    captures    => ('x'),
);

$m.integrate(:from(0), :to(8), :min-resolution(0.5));
$m.render-svg('population growth malthus.svg', :title('population growth'));

```

To fully understand what is going on, let's go through it step by step.

### [Step by step explanation](#Doing_math_with_Raku "go to top of document")[§](#Step_by_step_explanation "direct link")

* First we load the module that make the calculations: [`Math::Model`](https://github.com/moritz/Math-Model).

  Raku highlighting

  ```
  use Math::Model;

  ```
* We create the model to add all the information in it.

  Raku highlighting

  ```
  my $m = Math::Model.new(

  ```
* We declare the derivatives that are in our model. In this case, if you remember our equation, we have our variable *x* and its derivative *x'* (usually know as the velocity).

  Raku highlighting

  ```
  derivatives => {
      velocity => 'x',
  },

  ```
* After that, we declare how our models evolve. We just need formulas for the derivatives that are not also integration variables (in this case, only *x*), and for other variables we use in the formulas (the growth rate).

  Raku highlighting

  ```
  variables   => {
      velocity           => { $:growth_constant * $:x},
      growth_constant    => { 1 }, # basal growth rate
  },

  ```
* Finally we declare our initial conditions and use *captures* to tell [`Math::Model`](https://github.com/moritz/Math-Model) which variable or variables to record while the simulation is running.

  Raku highlighting

  ```
  initials    => {
      x       => 3,
  },
  captures    => ('x'),

  ```

At this point our model is set. We need to run the simulation and render a cool plot about our results:

```raku
$m.integrate(:from(0), :to(8), :min-resolution(0.5));
$m.render-svg('population growth malthus.svg', :title('population growth'));

```

There, we select our time limits and the resolution. Next, we generate a plot. All understood? Well, let's see our result!

[link to the image](https://rawgit.com/thebooort/perl-6-math-model-tutorial/master/population%20growth%20malthus.svg)

Looks great! But to be honest, it is quite unrepresentative. Let's explore other examples from more complex situations!

## [Logistic model](#Doing_math_with_Raku "go to top of document")[§](#Logistic_model "direct link")

Resources aren't infinite and our population is not going to grow forever. P-F Verhulst thought the same thing, so he presented the [logistic model](https://en.wikipedia.org/wiki/Logistic_function). This model is a common model of population growth, where the rate of reproduction is proportional to both the existing population and the amount of available resources, all else being equal. It looks like this:

*dx/dt = g\*x\*(1-x/k)*

*x(0)=x\_0*

where the constant g defines the growth rate and k is the carrying capacity. Modifying the above code we can simulate its behavior in time:

```raku
use Math::Model;

my $m = Math::Model.new(
    derivatives => {
        velocity => 'x',
    },
    variables   => {
        velocity           => { $:growth_constant * $:x - $:growth_constant * $:x * $:x / $:k },
        growth_constant    => { 1 },   # basal growth rate
        k                  => { 100 }, # carrying capacity
    },
    initials    => {
        x       => 3,
    },
    captures    => ('x'),
);

$m.integrate(:from(0), :to(8), :min-resolution(0.5));
$m.render-svg('population growth logistic.svg', :title('population growth'));

```

Let's look at our cool plot: [link to the image](https://rawgit.com/thebooort/perl-6-math-model-tutorial/master/population%20growth%20logistic.svg)

As you can see population growths till a maximum.

## [Strong Allee Effect](#Doing_math_with_Raku "go to top of document")[§](#Strong_Allee_Effect "direct link")

Interesting, isn't it? Even if these equations seem basic they are linked to a lot of behaviors in our world, like tumor growth. But, before end, let me show you a curious case. Logistic model could be accurate but... What happens when, from a certain threshold, the population size is so small that the survival rate and / or the reproductive rate drops due to the individual's inability to find other ones?

Well, this is an interesting phenomenon described by W.C.Allee, usually known as [Allee effect](https://en.wikipedia.org/wiki/Allee_effect). A simple way to obtain different behaviors associated with this effect is to use the logistic model as a departure, add some terms, and get a cubic growth model like this one:

*dx/dt=r\*x\*(x/a-1)\*(1-x/k)*

where all the constants are the same as before and A is called *critical point*.

Our code would be:

```raku
use Math::Model;

my $m = Math::Model.new(
    derivatives => {
        velocity_x    => 'x',
    },
    variables   => {
        velocity_x           => { $:growth_constant * $:x *($:x/$:a -1)*(1- $:x/$:k) },
        growth_constant    => { 0.7 },   # basal growth rate
        k                  => { 100 }, # carrying capacity
        a                  => { 15 },  # critical point
    },
    initials    => {
        x       => 15,
    },
    captures    => ('x'),
);

$m.integrate(:from(0), :to(100), :min-resolution(0.5));
$m.render-svg('population growth allee.svg', :title('population growth'));

```

Try to execute this one yourself to see the different behaviors that arise when you change the initial condition around the critical point!

## [Weak Allee Effect](#Doing_math_with_Raku "go to top of document")[§](#Weak_Allee_Effect "direct link")

Whilst the strong Allee effect is a demographic Allee effect with a critical population size or density, the weak Allee effect is a demographic Allee effect without a critical population size or density, i.e., a population exhibiting a weak Allee effect will possess a reduced per capita growth rate at lower population density or size. However, even at this low population size or density, the population will always exhibit a positive per capita growth rate. This model differs lightly from the strong one, getting a new formula:

*dx/dt=r\*x\*(1-x/k)\*(x/a)\*\*n*, with n>0

Our code would be:

```raku
use Math::Model;

my $m = Math::Model.new(
    derivatives => {
        velocity_x => 'x',
    },
    variables   => {
        velocity_x           => { $:growth_constant * $:x *(1- $:x/$:k)*($:x/$:a)**$:n },
        growth_constant    => { 0.7 },   # basal growth rate
        k                  => { 100 }, # carrying capacity
        a                  => { 15 },  # critical point
        n                  => { 4  }
    },
    initials    => {
        x       => 15,
    },
    captures    => ('x'),
);

$m.integrate(:from(0), :to(100), :min-resolution(0.5));
$m.render-svg('population growth allee.svg', :title('population growth'));

```

## [Extra info](#Doing_math_with_Raku "go to top of document")[§](#Extra_info "direct link")

Do you like physics? Check the original post by the creator of the modules that have been used, Moritz Lenz: <https://perlgeek.de/blog-en/perl-6/physical-modelling.html>.


## dist_cpan-CTILMES-Libarchive.md
# Libarchive - Multi-format archive and compression

[libarchive](https://www.libarchive.org/) is a multi-format archive
and compression libarary. This module provides a very composable high
level interface to the library for reading, processing and writing
archives of files.

See [PHLPM Talk](http://curt.tilmes.org/2019-PHLPM-Libarchive) for
some more description and examples of usage (mostly duplicates what is
here).

## Simple, streaming archive reading

```
use Libarchive::Simple;

.put for archive-read 'myfile.tar.gz';               # Print listing

.extract for archive-read $*IN;                      # Extract all files

# Print a custom listing, using field accessors
for archive-read($*IN) {
    put "dir: {.pathname}" if .is-dir;
    put "file: {.pathname} {.human-size}" if .is-file;
}

for archive-read('this.tar.gz') {
   .content.put if .pathname eq 'README'
}

archive-read('this.zip'.IO)           # Process Seq in normal ways
    .grep({ .pathname ~~ /README/ })  # with for, grep, map, etc.
    .map: { .extract :verbose };      # print listing to STDERR as extract

# Many extract options to customize, either in object or extract()
for archive-read('dvd.iso', :extract-no-overwrite,
                 destpath => '/somewhere') {
    next unless .pathname eq 'the-file-i-want';
    .extract(perm => 0o600);
}

```

Can read from filename, IO::Path, IO::Handle, Memory Buf, Supply of Blobs,
Channel of Blobs

`archive-read()` is just short-hand for `Libarchive::Read.new()`

## Simple, streaming archive writing

```
use Libarchive::Simple;

with archive-write('foo.zip')
{
    .add: 'afile';         # Add a file from the filesystem to the archive
    .add: 'somedir';       # Add a directory, but not contents
    .add: dir('somedir');  # Add every file in a directory
    .add: 'thisdir', dir('thisdir');  # Add directory and contents

    .write: 'afile', "Some content\n";      # Create a file from a Str
    .write: 'bfile', buf8.new(1,2,3,4);     # or from a Blob
    .write: 'bigrandomfile',                # or an IO::Handle
            '/dev/urandom'.IO.open(:bin),
            size => 100_000;                # override size

    .mkdir: 'adir';                               # Create a directory
    .mkdir: 'bdir', perm => 0o700;                # Override perm
    .write: 'cdir'.IO.add('another'), "this\n";   # IO::Path is fine too

    .symlink: 'linked', 'adir/anotherfile';       # Create a symlink
    .symlink: 'anotherlink' => 'adir/yetanother'; # Pair symlink is ok

    .close;                                       # Always close!
}

```

Can write to filename, IO::Path, IO::Handle, Memory Buf, Supplier of Blobs,
Channel of Blobs. Must specify format (optionally filters) unless filename:

```
archive-write($*OUT, format => 'zip');  # Send zip file to STDOUT

```

`archive-write()` is just shorthand for `Libarchive::Write.new()`

## Simple, Slurping all content into memory:

```
use Libarchive::Simple;

my $archive := archive-slurp 'this.tar';
say $archive;                                   # Print listing
put $archive<README>;                           # content of a file
$archive<afile>.content = "Change content\n";   # change existing file
$archive<adir/bad>:delete;                      # Remove file
$archive.spurt: 'foo.zip';                      # Dump archive back to disk

```

`archive-slurp()` is just shorthand for `Libarchive::Archive.new()`

It creates an object that is both `Iterable` just like `archive-read`,
and also `Associative`, including all the data/content from the
archive instead of reading it out of the stream as it goes, so you can
use hyper processing in parallel without worry. The keys are paths,
not just filenames. If the archive has two files with exactly the
same path, you'll just get one. (Why would you do that anyway?)

## Processing Archives in a pipeline

`Libarchive::Read` (and `archive-read`) produces a Seq of
`Libarchive::Entry`s. You can use the `.copy` method to copy them into
an `Libarchive::Write`.

For example, you could hook up a reader to a writer to convert a tar
file to a zip file (or ISO or whatever):

```
use Libarchive::Simple;

with archive-write($*OUT, format => 'zip')
{
    .copy: archive-read($*IN, format => 'tar')
    .close;
}

```

Or even process the contents in various ways as they go:

```
use Libarchive::Simple;

with archive-write($*OUT, format => 'zip')
{
    .write: 'NEWREADME', "This is my README\n";      # Add some extra files
    .write: 'LICENSE', "Special license file\n";
    .copy: archive-read($*IN, format => 'tar')
           .grep({ .pathname ~~ /good/})             # Only pass good files
           .map({ .pathname(.pathname.uc) })         # Uppercase filenames
           .map({ .uname('fred').perm(0o600)});      # Change owner and perm
    .close;
}

```

When streaming, make sure you keep the sequence lazy, otherwise the
stream with the data will be past before the copy occurs. If you want
random access, use `Libarchive::Archive` or `archive-slurp`.

# Filtering without an Archive, format 'raw'

`libarchive` supports a special format 'raw' that works on a single
virtual file, passing it through the specified filters. This can be
used to `compress`, `gzip`, `bzip2` etc.

The manual process is something like this:

```
with archive-write($dest, format => 'raw', filter => 'gzip')
{
    .write('ignore-filename', $source, size => ...);
    .close
}

```

or

```
with archive-read($source, format => 'raw')
{
    my $header = .read;  # Read and ignore the archive header
    while my $buf = .read-data(<blocksize>)
    {
        ...do something with $buf...
    }
}

```

These constructs have been packaged up into `Libarchive::Filter` with
two subroutines `archive-encode` and `archive-decode`. Each take a
`$source`, and a `$destination` that can be most of the normal things.
`archive-encode`, of course, must include 1 or more filters to be
useful.

For example, you can read/write files:

```
use Libarchive::Filter;
archive-encode('Some content', 'file.gz', filter => 'gzip');
my $content = archive-decode('file.gz');
... $content eq 'Some content';

```

or just use a memory buffer:

```
use Libarchive::Filter;
my $buf = archive-encode('Some content', filter => 'gzip');
...encoded into $buf...

my $content = archive-decode($buf);
...$content eq 'Some content'

```

`archive-encode` sources can be anything that `archive-write` will write:
content in a `Str` or `Buf`, or a filename `IO::Path`, an
`IO::Handle`, a `Supply` or `Channel` of `Blob`s.

`archive-encode` destinations can be anything that `archive-write`
will produce: `Buf`, `IO::Handle`, `Supplier`, `Channel`, or a `Str`
or `IO::Path` filename.

`archive-decode` sources can be anything that `archive-read` will read:
filename in a `Str` or `IO::Path`, `Blob`, `Supply`, `IO::Handle`, or
`Channel`.

`archive-decode` destinations can be `Blob`, `IO::Handle`, `IO::Path`,
`Supplier`, `Channel`. If you don't set a destination, a `Str` with
the content is returned.

Note that the `Str` *into* `archive-encode` or *out* of
`archive-decode` is the content itself, but `Str` *out* of
`archive-encode` or *into* `archive-decode` are filenames. You can
always use `IO::Path` for a filename.

A number of shortcuts for various filters have also been defined:

```
use Libarchive::Filter :gzip;

my $buf = gzip('Some content');
my $content = gunzip($buf);

```

These include:

* `:gzip` -> `gzip()` and `gunzip()`
* `:compress` -> `compress()` and `uncompress()`
* `:bzip2` -> `bzip2()` and `bunzip2()`
* `:lz4` -> `lz4()` and `unlz4()`
* `:uuencode` -> `uuencode()` and `uudecode()`
* `:lzma` -> `lzma()` and `unlzma()`

You can also specify `use Libarchive::Filter :all` to get all the
shortcut routines.

These all take the same options that `archive-encode()` and
`archive-decode()` do and go to/from files, IO::Handles, Supplies,
Channels, etc.

## Formats and Filters

Valid read formats:

'7zip', 'ar', 'cab', 'cpio', 'empty', 'gnutar', 'iso9660', 'lha',
'mtree', 'rar', 'raw', 'tar', 'warc', 'xar', 'zip', 'zip-streamable',
'zip-seekable'

Valid read filters:

'bzip2', 'compress', 'gzip', 'grzip', 'lrzip', 'lz4', 'lzip', 'lzma',
'lzop', 'none', 'rpm', 'uu', 'xz', 'zstd'

You can specify a list of multiple formats/filters to consider if you
want to limit which types you support. You can also specify 'all' for
either format or filter, which is the default.

Valid write formats:

'7zip', 'ar', 'arbsd', 'argnu', 'arsvr4', 'bsdtar', 'cd9660', 'cpio',
'gnutar', 'iso', 'iso9660', 'mtree', 'mtree-classic', 'newc', 'odc',
'oldtar', 'pax', 'paxr', 'posix', 'raw', 'rpax', 'shar', 'shardump',
'ustar', 'v7tar', 'v7', 'warc', 'xar', 'zip'

Valid write filters:

'b64encode', 'bzip2', 'compress', 'grzip', 'gzip', 'lrzip', 'lz4',
'lzip', 'lzma', 'lzop', 'uuencode', 'xz', 'zstd'

By default, if you write to a file, the extension of the filename will
be used to set the format (and possibly filter):

You can override by explicitly specifying a format and/or filters:

```
Libarchive::Write.new('myfile.tar.gz', format => 'zip');

```

will create a zip file named 'myfile.tar.gz' (but don't do that).

If you are writing to a stream, you **must** specify a format:

```
Libarchive::Write.new($*OUT, format => 'zip');

```

You can optionally specify one or more filters to use while writing.

```
Libarchive::Write.new('myfile', format => 'gnutar',
                             filter => <gzip b64encode>);

```

Multiple filters are built into a pipeline, so the order they are
listed is significant.

For more details on the specific way that libarchive handles each
format, including some limitations, see the man page:
[libarchive-formats.5](https://github.com/libarchive/libarchive/wiki/ManPageLibarchiveFormats5)
and the
[libarchive wiki](https://github.com/libarchive/libarchive/wiki/LibarchiveFormats).

## Libarchive Entry methods

An `Libarchive::Entry` is sort of like a super-stat, holding all of the
information about a file system component.

`Str` and `gist` return a single line summary of the archive entry,
kind of like an 'ls -l' or 'tar t' listing.

The other methods can query and/or set various information about the
entry:

`pathname`, `size`, `uid`, `gid`, `uname`, `gname`, `fflags`

`perm` - Integer permissions, for new files, defaults to `0o644`, for new
directories, defaults to `0o755`.

`atime`, `mtime`, `ctime`, `birthtime` - Various times, returned as
`DateTime`s. Depending on the archive format, these might not be set.

`symlink` - for a symbolic link, this is what it points to

`strmode` - Read only unixish string for filetype/permissions
(like `-rw-r--r--` or `drwxr-x-r-x`)

`mode` - file mode, better to use perm and/or filetype

`human-size` - uses [`Number::Bytes::Human`](https://github.com/dugword/Number-Bytes-Human) to process the size, so you get values like "15M", "25K" or "96B" for the
size of a file.

`filetype` - returns an `Libarchive::Filetype` object that numifys to
the Unix/C filetype bits and stringifys to: REG, LINK, SOCK, CHAR,
BLOCK, DIR, FIFO. You can pass in `:dir` to set filetype to DIR (or
just use '.mkdir');

`is-file` - Bool shortcut to query for filetype REG

`is-dir` - Bool shortcut to query for filetype DIR

## Libarchive Entry Extraction

A `Libarchive::Read` produces `Libarchive::Entry::Read` objects that
are `Libarchive::Entry`s with several additional methods:

`data` reads the content of the entry from the data stream and returns
it as a `Buf`.

`content` - same as `data`, but `decode`s the `Buf` into a Str
(encoding `utf-8` -- if you want other encodings, just call `decode`
on data).

`extract` - extracts the entry into a filesystem entity (file,
directory, symlink, socket, fifo, etc.)

You can change the `pathname` to rename or move the file around. You
can also pass in `:destpath` either to the main object on creation, or
to `extract()` and it will be prepended to the `pathname`.

You can also pass in extract flags, either to the main object, or to
individual `extract` calls to control the extraction:

## Extract flags:

Extract flags can be specified to `Libarchive::Read.new()`, or to the
`.open()`, or to `.extract()`. Flags to `.new()` and `.open()` are
sticky, and will affect all future `.open`s as well. Flags to
`.extract()` are not -- they affect only the specific extract.

`:extract-owner` - The user and group IDs should be set on the restored
file. By default, the user and group IDs are not restored.

`:extract-perm` - Full permissions (including SGID, SUID, and sticky
bits) should be restored exactly as specified, without obeying the
current umask. Note that SUID and SGID bits can only be restored if
the user and group ID of the object on disk are correct. If
`:extract_owner` is not specified, then SUID and SGID bits will only be
restored if the default user and group IDs of newly-created objects on
disk happen to match those specified in the archive entry. By default,
only basic permissions are restored, and umask is obeyed.

`:extract-time` - The timestamps (mtime, ctime, and atime) should be
restored. By default, they are ignored. Note that restoring of atime
is not currently supported.

`:extract-no-overwrite` - Existing files on disk will not be
overwritten. By default, existing regular files are truncated and
overwritten; existing directories will have their permissions updated;
other pre-existing objects are unlinked and recreated from scratch.

`:extract-unlink` - Existing files on disk will be unlinked before any
attempt to create them. In some cases, this can prove to be a
significant performance improvement. By default, existing files are
truncated and rewritten, but the file is not recreated. In particular,
the default behavior does not break existing hard links.

`:extract-acl` - Attempt to restore ACLs. By default, extended ACLs are
ignored.

`:extract-fflags` - Attempt to restore extended file flags. By default,
file flags are ignored.

`:extract-xattr` - Attempt to restore POSIX.1e extended attributes. By
default, they are ignored.

`:extract-secure-symlinks` - Refuse to extract any object whose final
location would be altered by a symlink on disk. This is intended to
help guard against a variety of mischief caused by archives that
(deliberately or otherwise) extract files outside of the current
directory. The default is not to perform this check. If
`:extract-unlink` is specified together with this option, the library
will remove any intermediate symlinks it finds and return an error
only if such symlink could not be removed.

`:extract-secure-nodotdot` - Refuse to extract a path that contains a
.. element anywhere within it. The default is to not refuse such
paths. Note that paths ending in .. always cause an error, regardless
of this flag.

`:extract-secure-noabsolutepaths` - Refuse to extract an absolute
path. The default is to not refuse such paths.

`:extract-sparse` - Scan data for blocks of NUL bytes and try to
recreate them with holes. This results in sparse files, independent of
whether the archive format supports or uses them.

`:extract-clear-nochange-fflags` - Before removing a file system object
prior to replacing it, clear platform-specific file flags which might
prevent its removal.

## Creating a new archive

## Writing to an archive

Using either `Libarchive::Write.new()` or `archive-write()`, there are
a number of methods for adding/creating filesystem entities.

### Add existing filesystem entitities:

`add()` adds existing entities by filename or `IO::Path`.

You may find ecosystem modules such as
[`File::Find`](https://github.com/tadzik/File-Find) or
[`Concurrent::File::Find`](https://github.com/gfldex/perl6-concurrent-file-find)
useful for generating lists of files:

```
    use Libarchive::Simple;
    use Concurrent::File::Find;

    with archive-write('somefile.tar.gz')
    {
        .add '/somedir', find('/somedir'); # Recursively add files
    }

```

If you add files within a directory, don't forget to add the directory
itself if you want it to be created on extraction too.

### Create new files

`write($filename, $content)` will create a new file

`$filename` can be a `Str` or something that will convert to a `Str`,
like an `IO::Path`. `$content` can be a `Str`, a `Blob`, an
`IO::Handle` from which the content will be read, or an `IO::Path`
from which the content will be read.

### Create directories

`mkdir($pathname)` will add a new directory to the archive

### Create new symbolic links

`symlink($pathname, $symlink)` or
`symlink($pathname => $symlink)`

### Add a sequence of `Archive::Entry`s

Use `copy()` to read from an `archive-read()` or `archive-slurp()`
sequence into a new archive.

## LICENSE

Copyright © 2019 United States Government as represented by the
Administrator, National Aeronautics and Space Administration. No
Copyright is claimed in the United States under Title 17, U.S. Code.
All Other Rights Reserved.


## has.md
has

Combined from primary sources listed below.

# [In Variables](#___top "go to top of document")[§](#(Variables)_declarator_has "direct link")

See primary documentation
[in context](/language/variables#The_has_declarator)
for **The has declarator**.

`has` scopes attributes to instances of a class or role, and methods to classes or roles. `has` is implied for methods, so `has method x() {}` and `method x() {}` do the same thing.

See [object orientation](/language/objects) for more documentation and some examples.


## missinginitializer.md
class X::Syntax::Term::MissingInitializer

Compilation error due to declaring a term without initialization

```raku
class X::Syntax::Term::MissingInitializer does X::Syntax { }
```

Syntax error when a term (a backslash variable) is declared without initialization assignment.

For example

```raku
my \foo;

```

dies with

｢text｣ without highlighting

```

```

===SORRY!===
Term definition requires an initializer

```

```

Valid code would be

```raku
my \foo = 42;

```


## dist_zef-antononcube-ML-TriesWithFrequencies.md
# Raku ML::TriesWithFrequencies

[![MacOS](https://github.com/antononcube/Raku-ML-TriesWithFrequencies/actions/workflows/macos.yml/badge.svg)](https://github.com/antononcube/Raku-ML-TriesWithFrequencies/actions/workflows/macos.yml)
[![Linux](https://github.com/antononcube/Raku-ML-TriesWithFrequencies/actions/workflows/linux.yml/badge.svg)](https://github.com/antononcube/Raku-ML-TriesWithFrequencies/actions/workflows/linux.yml)
[![Win64](https://github.com/antononcube/Raku-ML-TriesWithFrequencies/actions/workflows/windows.yml/badge.svg)](https://github.com/antononcube/Raku-ML-TriesWithFrequencies/actions/workflows/windows.yml)
[![](https://img.shields.io/badge/License-Artistic%202.0-0298c3.svg)](https://opensource.org/licenses/Artistic-2.0)
[![](https://raku.land/zef:antononcube/ML::TriesWithFrequencies/badges/version)](https://raku.land/zef:antononcube/ML::TriesWithFrequencies)
[![](https://raku.land/zef:antononcube/ML::TriesWithFrequencies/badges/downloads)](https://raku.land/zef:antononcube/ML::TriesWithFrequencies)

This Raku package has functions for creation and manipulation of
[Tries (Prefix trees)](https://en.wikipedia.org/wiki/Trie)
with frequencies.

The package provides Machine Learning (ML) functionalities,
not "just" a Trie data structure.

This Raku implementation closely follows the Java implementation [AAp3].

The subset of functions with the prefix "trie-" follows the one used in the Mathematica package [AAp2].
That is the "top-level" sub-system of function names; the sub-system is follows the typical Object-Oriented Programming (OOP)
Raku style.

**Remark:** Below Mathematica and Wolfram Language (WL) are used as synonyms.

**Remark:** There is a Raku package with an alternative implementation, [AAp6],
made mostly for comparison studies. (See the implementation notes below.)
The package in this repository, `ML::TriesWithFrequencies`, is my *primary*
Tries-with-frequencies package.

---

## Installation

Via zef-ecosystem:

```
zef install ML::TriesWithFrequencies

```

From GitHub:

```
zef install https://github.com/antononcube/Raku-ML-TriesWithFrequencies

```

---

## Usage

Consider a trie (prefix tree) created over a list of words:

```
use ML::TriesWithFrequencies;
my $tr = trie-create-by-split( <bar bark bars balm cert cell> );
trie-say($tr);

```

```
# TRIEROOT => 6
# ├─b => 4
# │ └─a => 4
# │   ├─l => 1
# │   │ └─m => 1
# │   └─r => 3
# │     ├─k => 1
# │     └─s => 1
# └─c => 2
#   └─e => 2
#     ├─l => 1
#     │ └─l => 1
#     └─r => 1
#       └─t => 1

```

Here we convert the trie with frequencies above into a trie with probabilities:

```
my $ptr = trie-node-probabilities( $tr );
trie-say($ptr);

```

```
# TRIEROOT => 1
# ├─b => 0.6666666666666666
# │ └─a => 1
# │   ├─l => 0.25
# │   │ └─m => 1
# │   └─r => 0.75
# │     ├─k => 0.3333333333333333
# │     └─s => 0.3333333333333333
# └─c => 0.3333333333333333
#   └─e => 1
#     ├─l => 0.5
#     │ └─l => 1
#     └─r => 0.5
#       └─t => 1

```

Here we shrink the trie with probabilities above:

```
trie-say(trie-shrink($ptr));

```

```
# TRIEROOT => 1
# ├─ba => 0.6666666666666666
# │ ├─lm => 0.25
# │ └─r => 0.75
# │   ├─k => 0.3333333333333333
# │   └─s => 0.3333333333333333
# └─ce => 0.3333333333333333
#   ├─ll => 0.5
#   └─rt => 0.5

```

Here we retrieve a sub-trie with a key:

```
trie-say(trie-retrieve($ptr, 'bar'.comb))

```

```
# r => 0.75
# ├─k => 0.3333333333333333
# └─s => 0.3333333333333333

```

Here is a "dot-pipeline" that combines the steps above:

```
<bar bark bars balm cert cell>.&trie-create-by-split
.node-probabilities
.shrink
.retrieve(<ba r>)
.form

```

```
# r => 0.75
# ├─k => 0.3333333333333333
# └─s => 0.3333333333333333

```

**Remark:** In the pipeline above we retrieve with `<ba r>`, not with `<b a r>`,
because the trie is already shrunk.

The package provides a fair amount of functions in order to facilitate ML applications.
In support of that statement, here are the methods of `ML::TriesWithFrequencies::Trie`:

```
ML::TriesWithFrequencies::Trie.^method_names

```

```
# (clone make merge insert create create-by-split node-probabilities leaf-probabilities leafQ position retrieve has-complete-match contains is-key shrink node-counts remove-by-threshold remove-by-pareto-fraction remove-by-regex select-by-threshold select-by-pareto-fraction select-by-regex root-to-leaf-paths words words-with-probabilities classify echo echo-function form trieRootLabel trieValueLabel getKey getValue getChildren setKey setValue setChildren to-map-format hash WL XML JSON Str random-choice from-map-format from-json-map-format new gist key value children BUILDALL)

```

Generate random words using trie, make a new trie, and visualize it:

```
my @randomWords = $ptr.random-choice(200):drop-root;
my $ptrRandom = trie-create(@randomWords).node-probabilities;
$ptrRandom.form;

```

```
# TRIEROOT => 1
# ├─b => 0.675
# │ └─a => 1
# │   ├─l => 0.2
# │   │ └─m => 1
# │   └─r => 0.8
# │     ├─k => 0.4351851851851852
# │     └─s => 0.46296296296296297
# └─c => 0.325
#   └─e => 1
#     ├─l => 0.5384615384615384
#     │ └─l => 1
#     └─r => 0.46153846153846156
#       └─t => 1

```

Compare with the original one:

```
$ptr.form

```

```
# TRIEROOT => 1
# ├─b => 0.6666666666666666
# │ └─a => 1
# │   ├─l => 0.25
# │   │ └─m => 1
# │   └─r => 0.75
# │     ├─k => 0.3333333333333333
# │     └─s => 0.3333333333333333
# └─c => 0.3333333333333333
#   └─e => 1
#     ├─l => 0.5
#     │ └─l => 1
#     └─r => 0.5
#       └─t => 1

```

**Remark:** It is expected with large numbers of generated words to get frequencies
very close to those of the original trie.

---

## Representation

Each trie is a tree of objects of the class `ML::TriesWithFrequencies::Trie`.
Such trees can be nicely represented as hash-maps. For example:

```
my $tr = trie-shrink(trie-create-by-split(<core cort>));
say $tr.gist;

```

```
# {TRIEROOT => {TRIEVALUE => 2, cor => {TRIEVALUE => 2, e => {TRIEVALUE => 1}, t => {TRIEVALUE => 1}}}}

```

The function `trie-say` uses that Hash-representation:

```
trie-say($tr)

```

```
# TRIEROOT => 2
# └─cor => 2
#   ├─e => 1
#   └─t => 1

```

### JSON

The JSON-representation follows the inherent object-tree
representation with `ML::TriesWithFrequencies::Trie`:

```
say $tr.JSON;

```

```
# {"key":"TRIEROOT", "value":2, "children":[{"key":"cor", "value":2, "children":[{"key":"e", "value":1, "children":[]}, {"key":"t", "value":1, "children":[]}]}]}

```

### XML

The XML-representation follows (resembles) the Hash-representation
(and output from `trie-say`):

```
say $tr.XML;

```

```
# <TRIEROOT>
#  <TRIEVALUE>2</TRIEVALUE>
#  <cor>
#   <TRIEVALUE>2</TRIEVALUE>
#   <e>
#    <TRIEVALUE>1</TRIEVALUE>
#   </e>
#   <t>
#    <TRIEVALUE>1</TRIEVALUE>
#   </t>
#  </cor>
# </TRIEROOT>

```

Using the XML representation allows for
[XPath](https://www.w3schools.com/xml/xml_xpath.asp)
searches, say, using the package
[`XML::XPath`](https://github.com/ufobat/p6-XML-XPath).
Here is an example:

```
use XML::XPath;
my $tr0 = trie-create-by-split(<bell best>);
trie-say($tr0);

```

```
# TRIEROOT => 2
# └─b => 2
#   └─e => 2
#     ├─l => 1
#     │ └─l => 1
#     └─s => 1
#       └─t => 1

```

Convert to XML:

```
say $tr0.XML;

```

```
# <TRIEROOT>
#  <TRIEVALUE>2</TRIEVALUE>
#  <b>
#   <TRIEVALUE>2</TRIEVALUE>
#   <e>
#    <TRIEVALUE>2</TRIEVALUE>
#    <l>
#     <TRIEVALUE>1</TRIEVALUE>
#     <l>
#      <TRIEVALUE>1</TRIEVALUE>
#     </l>
#    </l>
#    <s>
#     <TRIEVALUE>1</TRIEVALUE>
#     <t>
#      <TRIEVALUE>1</TRIEVALUE>
#     </t>
#    </s>
#   </e>
#  </b>
# </TRIEROOT>

```

Search for `<b e l>`:

```
say XML::XPath.new(xml=>$tr0.XML).find('//b/e/l');

```

```
# <l>
#     <TRIEVALUE>1</TRIEVALUE>
#     <l>
#      <TRIEVALUE>1</TRIEVALUE>
#     </l>
#    </l>

```

### WL

The Hash-representation is used in the Mathematica package [AAp2].
Hence, such WL format is provided by the Raku package:

```
say $tr.WL;

```

```
# <|$TrieRoot -> <|$TrieValue -> 2, "cor" -> <|$TrieValue -> 2, "e" -> <|$TrieValue -> 1|>, "t" -> <|$TrieValue -> 1|>|>|>|>

```

---

## Cloning

All `trie-*` functions and `ML::TriesWithFrequencies::Trie` methods that manipulate tries produce trie clones.

For performance reasons I considered having in-place trie manipulations, but that, of course, confuses reasoning
in development, testing, and usage. Hence, ubiquitous cloning.

---

## Two stiles of pipelining

As it was mentioned above the package was initially developed to have the functional programming design
of the Mathematica package [AAp2]. With that design and using the
[feed operator `==>`](https://docs.raku.org/language/operators#infix_==%3E)
we can construct pipelines like this one:

```
my @words2 = <bar barman bask bell belly>;
my @words3 = <call car cast>;

trie-create-by-split(@words2)==>
trie-merge(trie-create-by-split(@words3))==>
trie-node-probabilities==>
trie-shrink==>
trie-say

```

```
# TRIEROOT => 1
# ├─b => 0.625
# │ ├─a => 0.6
# │ │ ├─r => 0.6666666666666666
# │ │ │ └─man => 0.5
# │ │ └─sk => 0.3333333333333333
# │ └─ell => 0.4
# │   └─y => 0.5
# └─ca => 0.375
#   ├─ll => 0.3333333333333333
#   ├─r => 0.3333333333333333
#   └─st => 0.3333333333333333

```

The package also supports "dot pipelining" through chaining of methods:

```
@words2.&trie-create-by-split
        .merge(@words3.&trie-create-by-split)
        .node-probabilities
        .shrink
        .form

```

```
# TRIEROOT => 1
# ├─b => 0.625
# │ ├─a => 0.6
# │ │ ├─r => 0.6666666666666666
# │ │ │ └─man => 0.5
# │ │ └─sk => 0.3333333333333333
# │ └─ell => 0.4
# │   └─y => 0.5
# └─ca => 0.375
#   ├─ll => 0.3333333333333333
#   ├─r => 0.3333333333333333
#   └─st => 0.3333333333333333

```

**Remark:** The `trie-*` functions are implemented through the methods of `ML::TriesWithFrequencies::Trie`.
Given the method the corresponding function is derived by adding the prefix `trie-`.
(For example, `$tr.shrink` vs `trie-shrink($tr)`.)

Here is the previous pipeline re-written to use only methods of `ML::TriesWithFrequencies::Trie`:

```
ML::TriesWithFrequencies::Trie.create-by-split(@words2)
        .merge(ML::TriesWithFrequencies::Trie.create-by-split(@words3))
        .node-probabilities
        .shrink
        .form

```

---

## Implementation notes

### UML diagram

Here is a UML diagram that shows package's structure:

```
use UML::Translators;
to-uml-spec('ML::TriesWithFrequencies', format => 'mermaid')

```

```
classDiagram
class ML_TriesWithFrequencies_PathsGatherer {
  +$!ulp
  +@!tracedPaths
  +BUILDALL()
  +new()
  +trace()
  +tracedPaths()
  +trie-trace()
  +ulp()
}

class TRIEROOT {
  <<constant>>
}
TRIEROOT --|> Stringy

class TRIEVALUE {
  <<constant>>
}
TRIEVALUE --|> Stringy

class ML_TriesWithFrequencies_ChildRandomChooser {
  +$!ulp
  +$!weighted
  +@!tracedPaths
  +BUILDALL()
  +new()
  +trace()
  +tracedPaths()
  +trie-trace()
  +ulp()
  +weighted()
}

class ML_TriesWithFrequencies_Trieish {
  <<role>>
  +$!key
  +$!value
  +%!children
  +JSON()
  +Str()
  +WL()
  +XML()
  +clone()
  +getChildren()
  +getKey()
  +getValue()
  +hash()
  +setChildren()
  +setKey()
  +setValue()
  +to-map-format()
  +trieRootLabel()
  +trieValueLabel()
}

class ML_TriesWithFrequencies_ParetoBasedRemover {
  +$!pareto-fraction
  +$!postfix
  +$!remove-bottom
  +BUILDALL()
  +new()
  +pareto-fraction()
  +postfix()
  +remove()
  +remove-bottom()
  +trie-map()
  +trie-pareto-remove()
}
ML_TriesWithFrequencies_ParetoBasedRemover --|> ML_TriesWithFrequencies_TrieTraverse

class ML_TriesWithFrequencies_ThresholdBasedRemover {
  +$!below-threshold
  +$!postfix
  +$!threshold
  +BUILDALL()
  +below-threshold()
  +new()
  +postfix()
  +remove()
  +threshold()
  +trie-map()
  +trie-threshold-remove()
}
ML_TriesWithFrequencies_ThresholdBasedRemover --|> ML_TriesWithFrequencies_TrieTraverse

class ML_TriesWithFrequencies_RegexBasedRemover {
  +$!invert
  +$!key-pattern
  +$!postfix
  +BUILDALL()
  +invert()
  +key-pattern()
  +new()
  +postfix()
  +remove()
  +trie-map()
  +trie-regex-remove()
}
ML_TriesWithFrequencies_RegexBasedRemover --|> ML_TriesWithFrequencies_TrieTraverse

class ML_TriesWithFrequencies_TrieTraverse {
  <<role>>
}

class ML_TriesWithFrequencies_Trie {
  +$!key
  +$!value
  +%!children
  +BUILDALL()
  +JSON()
  +Str()
  +WL()
  +XML()
  +children()
  +classify()
  +clone()
  +contains()
  +create()
  +create-by-split()
  +echo()
  +echo-function()
  +form()
  +from-json-map-format()
  +from-map-format()
  +getChildren()
  +getKey()
  +getValue()
  +gist()
  +has-complete-match()
  +hash()
  +insert()
  +is-key()
  +key()
  +leaf-probabilities()
  +leafQ()
  +make()
  +merge()
  +new()
  +node-counts()
  +node-probabilities()
  +position()
  +random-choice()
  +remove-by-pareto-fraction()
  +remove-by-regex()
  +remove-by-threshold()
  +retrieve()
  +root-to-leaf-paths()
  +select-by-pareto-fraction()
  +select-by-regex()
  +select-by-threshold()
  +setChildren()
  +setKey()
  +setValue()
  +shrink()
  +to-map-format()
  +trieRootLabel()
  +trieValueLabel()
  +value()
  +words()
  +words-with-probabilities()
}
ML_TriesWithFrequencies_Trie --|> ML_TriesWithFrequencies_Trieish

class ML_TriesWithFrequencies_LeafProbabilitiesGatherer {
  +$!counts-trie
  +$!ulp
  +BUILDALL()
  +counts-trie()
  +new()
  +trace()
  +trie-trace()
  +ulp()
}

```

**Remark:** The function `to-uml-spec` is provided by the package ["UML::Translators"](https://raku.land/zef:antononcube/UML::Translators), [AAp7].

### Performance

This package is a Raku re-implementation of the Java Trie package [AAp3].

The initial implementation was:

* ≈ 5-6 times slower than the Mathematica implementation [AAp2]
* ≈ 100 times slower than the Java implementation [AAp3]

The initial implementation used:

* General types for Trie nodes, i.e. `Str` for the key and `Numeric` for the value
* Argument type verification with `where` statements in the signatures of the `trie-*` functions

After reading [RAC1] I refactored the code to use native types (`num`, `str`)
and moved the `where` verifications inside the functions.

I also refactored the function `trie-merge` to use less copying of data and
to take into account which of the two tries has smaller number of children.

After those changes the current Raku implementation is:

* ≈ 2.5 times slower than the Mathematica implementation [AAp2]
* ≈ 40 times slower than the Java implementation [AAp3]

After the (monumental) work on
[the new MoarVM dispatch mechanism](https://6guts.wordpress.com/2021/09/29/the-new-moarvm-dispatch-mechanism-is-here/),
[JW1], was incorporated in standard Rakudo releases (September/October 2021)
additional 20% speed-up was obtained. Currently this package is:

* ≈ 2.0 times slower than the Mathematica implementation [AAp2]
* ≈ 30 times slower than the Java implementation [AAp3]

These speed improvements are definitely not satisfactory. I strongly consider:

1. Re-implementing in Raku the Mathematica package [AAp2], i.e. to move into Tries that are hashes.

   * (It turned out option 1 does not produce better results; see [AAp6].)
2. Re-implementing in C or C++ the Java package [AAp3] and hooking it up to Raku.

### Moving from FP design and OOP design

The initial versions of the package -- up to version 0.5.0 -- had exported functions only
in the namespace `ML::TriesWithFrequencies` with the prefix `trie-`.
Those functions came from a purely Functional Programming (FP) design.

In order to get chains of OOP methods application that
are typical in Raku programming the package versions after version 0.6.0 and later have trie
manipulation transformation methods in the class `ML::TriesWithFrequencies::Trie`.

In order to get trie-class methods a fairly fundamental code refactoring was required.
Here are the steps:

1. The old class `ML::TriesWithFrequencies::Trie` was made into the role
   `ML::TriesWithFrequencies::Trieish`.
2. The traversal and remover classes were made to use `ML::TriesWithFrequencies::Trieish` type
   instead of `ML::TriesWithFrequencies::Trie`.
3. The trie functions implementations -- with the prefix "trie-" --
   of `ML::TriesWithFrequencies` were moved as methods implementations in `ML::TriesWithFrequencies::Trie`.
4. The trie functions in `ML::TriesWithFrequencies` were reimplemented using the methods
   of `ML::TriesWithFrequencies::Trie`.

**Remark:** See the section "Two stiles of pipelining" above for illustrations of the two approaches.

---

## TODO

In the following list the most important items are placed first.

* DONE Implement "get words" and "get root-to-leaf paths" functions.

  * See `trie-words` and `trie-root-to-leaf-paths`.
* DONE Convert most of the WL unit tests in [AAp5] into Raku tests.
* DONE Implement Trie traversal functions.

  * The general `trie-map` function is in a separate role.
  * A concrete traversal functionality is a class that does the role
    and provides additional context.
* DONE Implement (sub-)trie removal functions.

  * DONE By threshold (below and above)
  * DONE By Pareto principle adherence (top and bottom)
  * DONE By regex over the keys
* TODO Implement optional ULP spec argument for relevant functions:

  * DONE `trie-root-to-leaf-paths`
  * DONE `trie-words`
  * TODO Membership test functions?
* DONE Design and code refactoring so trie objects to have OOP interface.

  * Instead of just having `trie-words($tr, <c>)` we should be also able to say `$tr.trie-words(<c>)`.
* TODO Implement `trie-prune` function.
* DONE Implement Trie-based classification.
* DONE Create trie from hash representation.
* TODO Investigate faster implementations.

  * DONE Re-implement the Trie functionalities using hash representation (instead of a tree of Trie-node objects.)

    * See [AAp6].
  * TODO Make a C or C++ implementation and hook it up to Raku.
* DONE Program a trie-form visualization that is "wide", i.e. places the children nodes horizontally.

  * Using "Pretty::Table".
  * Using the function `to-pretty-table` of "Data::Reshapers". (Also based on "Pretty::Table".)
* TODO Document examples of doing Trie-based text mining or data-mining.

---

## References

### Articles

[AA1] Anton Antonov,
["Tries with frequencies for data mining"](https://mathematicaforprediction.wordpress.com/2013/12/06/tries-with-frequencies-for-data-mining/),
(2013),
[MathematicaForPrediction at WordPress](https://mathematicaforprediction.wordpress.com).

[AA2] Anton Antonov,
["Removal of sub-trees in tries"](https://mathematicaforprediction.wordpress.com/2014/10/12/removal-of-sub-trees-in-tries/),
(2013),
[MathematicaForPrediction at WordPress](https://mathematicaforprediction.wordpress.com).

[AA3] Anton Antonov,
["Tries with frequencies in Java"](https://mathematicaforprediction.wordpress.com/2017/01/31/tries-with-frequencies-in-java/),
(2017),
[MathematicaForPrediction at WordPress](https://mathematicaforprediction.wordpress.com).
[GitHub Markdown](https://github.com/antononcube/MathematicaForPrediction).

[JW1] Jonathan Worthington,
["The new MoarVM dispatch mechanism is here!"](https://6guts.wordpress.com/2021/09/29/the-new-moarvm-dispatch-mechanism-is-here/),
(2021),
[6guts at WordPress](https://6guts.wordpress.com).

[RAC1] Tib,
["Day 10: My 10 commandments for Raku performances"](https://raku-advent.blog/2020/12/10/day-10-my-10-commandments-for-raku-performances/),
(2020),
[Raku Advent Calendar](https://raku-advent.blog).

[WK1] Wikipedia entry, [Trie](https://en.wikipedia.org/wiki/Trie).

### Packages

[AAp1] Anton Antonov,
[Tries with frequencies Mathematica Version 9.0 package](https://github.com/antononcube/MathematicaForPrediction/blob/master/TriesWithFrequenciesV9.m),
(2013),
[MathematicaForPrediction at GitHub](https://github.com/antononcube/MathematicaForPrediction).

[AAp2] Anton Antonov,
[Tries with frequencies Mathematica package](https://github.com/antononcube/MathematicaForPrediction/blob/master/TriesWithFrequencies.m),
(2013-2018),
[MathematicaForPrediction at GitHub](https://github.com/antononcube/MathematicaForPrediction).

[AAp3] Anton Antonov,
[Tries with frequencies in Java](https://github.com/antononcube/MathematicaForPrediction/tree/master/Java/TriesWithFrequencies),
(2017),
[MathematicaForPrediction at GitHub](https://github.com/antononcube/MathematicaForPrediction).

[AAp4] Anton Antonov,
[Java tries with frequencies Mathematica package](https://github.com/antononcube/MathematicaForPrediction/blob/master/JavaTriesWithFrequencies.m),
(2017),
[MathematicaForPrediction at GitHub](https://github.com/antononcube/MathematicaForPrediction).

[AAp5] Anton Antonov,
[Java tries with frequencies Mathematica unit tests](https://github.com/antononcube/MathematicaForPrediction/blob/master/UnitTests/JavaTriesWithFrequencies-Unit-Tests.wlt),
(2017),
[MathematicaForPrediction at GitHub](https://github.com/antononcube/MathematicaForPrediction).

[AAp6] Anton Antonov,
[ML::HashTriesWithFrequencies Raku package](https://github.com/antononcube/Raku-ML-HashTriesWithFrequencies),
(2021),
[GitHub/antononcube](https://github.com/antononcube).

[AAp7] Anton Antonov,
[UML::Translators Raku package](https://raku.land/zef:antononcube/UML::Translators),
(2022),
[GitHub/antononcube](https://github.com/antononcube).

### Videos

[AAv1] Anton Antonov,
["Prefix Trees with Frequencies for Data Analysis and Machine Learning"](https://www.youtube.com/watch?v=MdVp7t8xQbQ),
(2017),
Wolfram Technology Conference 2017,
[Wolfram channel at YouTube](https://www.youtube.com/channel/UCJekgf6k62CQHdENWf2NgAQ).


## reserved.md
class X::Syntax::Reserved

Compilation error due to use of syntax reserved for future use

```raku
class X::Syntax::Reserved does X::Syntax { }
```

Syntax error thrown when a syntax is used which is reserved for future expansion.

For example

```raku
my @a();

```

dies with

｢text｣ without highlighting

```

```

===SORRY!===
The () shape syntax in array declarations is reserved

```

```

# [Methods](#class_X::Syntax::Reserved "go to top of document")[§](#Methods "direct link")

## [method reserved](#class_X::Syntax::Reserved "go to top of document")[§](#method_reserved "direct link")

```raku
method reserved(--> Str:D)
```

Returns a text description of the reserved syntax.

## [method instead](#class_X::Syntax::Reserved "go to top of document")[§](#method_instead "direct link")

```raku
method instead(--> Str)
```

Describes what to use instead of the reserved syntax (if anything).


## dist_zef-jonathanstowe-Lumberjack-Config-JSON.md
# Lumberjack::Config::JSON

Configure the Lumberjack logging framework from JSON

![Build Status](https://github.com/jonathanstowe/Lumberjack-Config-JSON/workflows/CI/badge.svg)

## Synopsis

```

use Lumberjack::Config::JSON;

my $config = Lumberjack::Config::JSON.from-json(q:to/EOJ/);
{
   "dispatchers" : [
      {
         "dispatcher" : "Lumberjack::Dispatcher::Console",
         "levels": [
            "Debug", "Info", "Warn", "Error"
         ],
         "classes" : [
            "My::Class"
         ]
      }
   ],
   "levels" : [
      {
         "class" : "My::Class",
         "level" : "All"
      }
   ]
}
EOJ

```

## Description

This provides a mechanism to configure [Lumberjack](https://github.com/jonathanstowe/Lumberjack)
from a description in `JSON`.

This is a fairly simple generic configuration, if you want more sophistication you may want to
compose your configuration in code.

Somewhat more detail can be found in the [Documentation](Documentation.md).

## Installation

Assuming you have a working Rakudo installation you should be able to install this with *zef* :

zef install Lumberjack::Config::JSON

## Support

## Licence and Copyright

This is free software, please see the <LICENCE> for details.

© Jonathan Stowe 2021-


## dist_zef-lizmat-ForwardIterables.md
[![Actions Status](https://github.com/lizmat/ForwardIterables/actions/workflows/linux.yml/badge.svg)](https://github.com/lizmat/ForwardIterables/actions) [![Actions Status](https://github.com/lizmat/ForwardIterables/actions/workflows/macos.yml/badge.svg)](https://github.com/lizmat/ForwardIterables/actions) [![Actions Status](https://github.com/lizmat/ForwardIterables/actions/workflows/windows.yml/badge.svg)](https://github.com/lizmat/ForwardIterables/actions)

# NAME

ForwardIterables - turn two or more Iterables into a single iterator

# SYNOPSIS

```
use ForwardIterables;

my @a = ^5;
my @b = <a b c d e>;
my @c = ForwardIterables.new(@a,@b).Seq;
say @c;  # [0 1 2 3 4 a b c d e]

```

# DESCRIPTION

The `ForwardIterables` distribution provides a `ForwardIterables` class that creates a single `Iterator` from any number `Iterables` that will **lazily** produce values in a "forward" order.

This functionality is similar to the [`flat`](https://docs.raku.org/routine/flat) method, but with the important distinction that it does **NOT** look at the containerization of the arguments. So any iterable such as a `Array` or `List` inside a `Hash` or an `Array`, **will** produce all of its values.

And it also does **not** recurse into any `Iterable` values that it encounters, so in that aspect it is **NOT** like `flat` at all.

It also provides a `Seq` method, to directly produce a `Seq` object from the iterator, so it can be used in expressions.

# PROBABLY NOT USER FACING

This module is probably more useful for module developers, than for people writing direct Raku production code.

## PERFORMANCE

Depending on the situation, the use of this iterator can be anywhere from 1.5x to 3x as fast as the equivalent code using `.flat`.

# AUTHOR

Elizabeth Mattijsen [liz@raku.rocks](mailto:liz@raku.rocks)

Source can be located at: <https://github.com/lizmat/OneSeq> . Comments and Pull Requests are welcome.

If you like this module, or what I'm doing more generally, committing to a [small sponsorship](https://github.com/sponsors/lizmat/) would mean a great deal to me!

# COPYRIGHT AND LICENSE

Copyright 2025 Elizabeth Mattijsen

This library is free software; you can redistribute it and/or modify it under the Artistic License 2.0.


## dist_github-cygx-Image-PNG-Inflated.md
# Image::PNG::Inflated [build status](https://travis-ci.org/cygx/p6-image-png-inflated)

Creates PNG images from raw RGBA data of depth 8, chunking the data into
non-compressed 64k blocks.

## Synopsis

```
use Image::PNG::Inflated;
use Image::RGBA::Fun;

# grayscale gradient
my $grad = blob8.new(map { |($_, $_, $_, 255) }, ^256);
my $blob = [~] $grad xx 64;             # ^--- a opaque pixel

spurt 'grayscale.png', to-png($blob, 256, 64);
                            # width --^    ^-- height

# our most beloved butterfly
my $img = load-rgba-from-textfile('camelia.txt');
spurt 'camelia.png', to-png($img);

```

The only exported symbol is the `&to-png` subroutine

```
sub to-png($img, $w = $img.width, $h = $img.height) { ... }

```

If the `$img` argument is not of type `blob8` itself, such an object must
be returned from the call `$img.blob8`.

## Bugs and Development

Development happens [at GitHub](https://github.com/cygx/p6-image-png-inflated). If you found a bug or have a feature
request, use the [issue tracker](https://github.com/cygx/p6-image-png-inflated/issues) over there.

## Copyright and License

Copyright (C) 2015, 2017, 2019 by cygx <[cygx@cpan.org](mailto:cygx@cpan.org)>

Distributed under the [Boost Software License, Version 1.0](http://www.boost.org/LICENSE_1_0.txt)


## dist_zef-grizzlysmit-RegexUtils.md
## Table of Contents

* [NAME](#name)
* [AUTHOR](#author)
* [VERSION](#version)
* [TITLE](#title)
* [SUBTITLE](#subtitle)
* [COPYRIGHT](#copyright)
* [Introduction](#introduction)

  * [Motivations](#motivations)
* [Grammars](#grammars)

  * [grammar FlagsPerl5Modern & action class FlagsPerl5ModernActions](#grammar-flagsperl5modern--action-class-flagsperl5modernactions)
  * [grammar FlagsEMCA262Modern & action class FlagsEMCA262ModernActions](#grammar-flagsemca262modern--action-class-flagsemca262modernactions)
  * [grammar FlagsPerl5Old & action class FlagsPerl5OldActions](#grammar-flagsperl5old--action-class-flagsperl5oldactions)
  * [grammar FlagsEMCA262Old & action class FlagsEMCA262OldActions](#grammar-flagsemca262old--action-class-flagsemca262oldactions)
* [Methods](#methods)

  * [CreatePerlRegex(…)](#createperlregex)
  * [CreateEMCA262Regex(…)](#createemca262regex)
  * [get-modern-perl5-flags(…)](#get-modern-perl5-flags)
  * [get-old-perl5-flags(…)](#get-old-perl5-flags)
  * [get-modern-emca262-flags(…)](#get-modern-emca262-flags)
  * [get-old-emca262-flags(…)](#get-old-emca262-flags)

# NAME

Gzz::Text::Utils

# AUTHOR

Francis Grizzly Smit ([grizzly@smit.id.au](mailto:grizzly@smit.id.au))

# VERSION

v0.1.5

# TITLE

RegexUtils

# SUBTITLE

A Raku class/module that provides helpers for Regex stuff both Perl5 and EMCA262Regex

# COPYRIGHT

LGPL V3.0+ [LICENSE](https://github.com/grizzlysmit/RegexUtils/blob/main/LICENSE)

# Introduction

A **Raku** class/module that provides helpers for **`Regex`** stuff both **`Perl5`** and **`EMCA262Regex`**.

[Top of Document](#table-of-contents)

## Motivations

I have a program which exists as a **`Mod_Perl`** web program,and a **Raku** command line version. The **`Mod_Perl`** version also uses **JavaScript** and <JavaScript/EMCA> regexes I found that there was an all most perfect solution for the use of these regexes in the **`EMCA262Regex`** module, but I still needed to convert the strings to regexes as I wanted to add options, **`RegexUtils::CreateEMCA262Regex`** is my solution to this problem I then added **`RegexUtils::CreatePerlRegex`** for doing the same for **Perl5** regexes.

[Top of Document](#able-of-contents)

## Grammars

### grammar FlagsPerl5Modern & action class FlagsPerl5ModernActions

Support for using a limited range of flags for use with Perl 5 style regexes, using **Raku's** style of regex flags.

```
grammar FlagsPerl5Modern {
    token TOP   { <flags> }
    token flags { <flag>+ % ':' }
    token flag  {  \w+ }
}

class FlagsPerl5ModernActions {
    method TOP($/) { make $<flags>.made }
    method flag($/) {
        given ~$/ {
            when 'i'          {
                                  make ':ignorecase';
                              }
            when 'ignorecase' {
                                  make ':ignorecase';
                              }
            when 'm'          { make ':ignoremark'; }
            when 'ignoremark' { make ':ignoremark'; }
            when 's'          { make ':sigspace';   }
            when 'sigspace'   { make ':sigspace';   }
            default:          { make '';            }
        }
    }
    method flags($/) {
        my $made-elts = $<flag>».made.join('');
        make $made-elts;
    }
}

```

[Top of Document](#table-of-contents)

### grammar FlagsEMCA262Modern & action class FlagsEMCA262ModernActions

A grammar and action pair to parse modern style **Raku** flags and get rid of ones that do not fit with ECMA regexes.

```
grammar FlagsEMCA262Modern {
    token TOP   { <flags> }
    token flags { <flag>+ % ':' }
    token flag  {  \w+ }
}

class FlagsEMCA262ModernActions {
    method TOP($/) { make $<flags>.made }
    method flag($/) {
        given ~$/ {
            when 'i'          {
                                  make ':ignorecase';
                              }
            when 'ignorecase' {
                                  make ':ignorecase';
                              }
            when 'g'          { make ':global';     }
            when 'global'     { make ':global';     }
            default:          { make '';            }
        }
    }
    method flags($/) {
        my $made-elts = $<flag>».made.join('');
        make $made-elts;
    }
}

```

[Top of Document](#table-of-contents)

### grammar FlagsPerl5Old & action class FlagsPerl5OldActions

A grammar and action pair to parse old style **Perl** flags into modern **Raku** flags.

```
grammar FlagsPerl5Old {
    token TOP   { <flags> }
    token flags { <flag>+ % <ww> }
    token flag  { \w }
}

class FlagsPerl5OldActions {
    method TOP($/) { make $<flags>.made }
    method flag($/) {
        given ~$/ {
            when 'i'          { make ':ignorecase'; }
            when 'm'          { make ':ignoremark'; }
            when 'g'          { make ':global';     }
            when 'x'          { make ':sigspace';   }
            default:          { make '';            }
        }
    }
    method flags($/) {
        my $made-elts = $/<flag>».made.join('');
        make $made-elts;
    }
}

```

[Top of Document](#table-of-contents)

### grammar FlagsEMCA262Old & action class FlagsEMCA262OldActions

A grammar and action pair to convert **ECMA262Regex** flags into **Raku** ones

```
grammar FlagsEMCA262Old {
    token TOP   { <flags> }
    token flags { <flag>+ % <ww> }
    token flag  { \w }
}

class FlagsEMCA262OldActions {
    method TOP($/) { make $<flags>.made }
    method flag($/) {
        given ~$/ {
            when 'i'          { make ':ignorecase'; }
            when 'g'          { make ':global';   }
            default:          { make '';            }
        }
    }
    method flags($/) {
        my $made-elts = $/<flag>».made.join('');
        make $made-elts;
    }
}

```

[Top of Document](#table-of-contents)

## Methods

### CreatePerlRegex(…)

Creates a Perl5 regex from a string with some options suported.

**Note:** not all raku flags are supported due to diffrences between the flags in the two languages.

Here is some exaple use.

```
[0] > use RegexUtils;
Nil
[1] > my $regex-perl5 = RegexUtils.CreatePerlRegex('^fo+\n$', ':ignorecase');
rx:Perl5:ignorecase/^fo+\n$/
[2] > so 'Fooo' ~~ $regex-perl5;
False
[3] > so "Fooo\n" ~~ $regex-perl5;
True
[4] > say $regex-perl5;
rx:Perl5:ignorecase/^fo+\n$/

```

[Top of Document](#table-of-contents)

### CreateEMCA262Regex(…)

Creates a regex from a string containing a EMCA262Regex with some options suported.

**`CreateEMCA262Regex`** converts a **EMCA262 Regex** to a raku one. using the [EMCA262Regex](https://modules.raku.org/dist/ECMA262Regex:zef:zef:jnthn) package and applies flags to it.

Here is some exaple use.

```
[0] > use RegexUtils;
Nil
[1] > my $regex-emca = RegexUtils.CreateEMCA262Regex('^[f][o]+\n', ':ignorecase');
rx:ignorecase/^<[f]><[o]>+\n/
[2] > so "Fooo\n" ~~ $regex-emca;
True
[3] > so " Fooo\n" ~~ $regex-emca;
False

```

**Note:** `ignorecase` only works for cases like below due to how **`EMCA262Regex`** translates the charater constants, this is a problem that needs solving.

e.g.

```
[0] > use RegexUtils;
Nil
[1] > my $regex-emca = RegexUtils.CreateEMCA262Regex('^fo+\n', ':ignorecase');
rx:ignorecase/^\x66\x6F+\n/
[2] > so "Fooo\n" ~~ $regex-emca;
False

```

[Top of Document](#table-of-contents)

### get-modern-perl5-flags(…)

A method to apply the **`RegexUtils::FlagsPerl5Modern`** grammar to a string, in order to filter flags.

```
method get-modern-perl5-flags(Str:D $flags --> Str:D) {
    my $actions = RegexUtils::FlagsPerl5ModernActions;
    my $match   = RegexUtils::FlagsPerl5Modern.parse($flags.substr(1), :$actions);
    without $match {
        die "invalid flags";
    }
    return $match.made;
} # method get-modern-perl5-flags(Str:D $flags --> Str:D) #

```

### get-old-perl5-flags(…)

Translate old style **Perl** flags into **Raku** ones using the grammar action pair

```
method get-old-perl5-flags(Str:D $flags --> Str:D) {
    my $actions = RegexUtils::FlagsPerl5OldActions;
    my $match   = RegexUtils::FlagsPerl5Old.parse($flags, :$actions);
    without $match {
        die "invalid flags $flags";
    }
    return $match.made;
} # method get-old-perl5-flags(Str:D $flags --> Str:D) #

```

[Top of Document](#table-of-contents)

### get-modern-emca262-flags(…)

Filter modern flags for use with **EMCA262Regex's**.

```
method get-modern-emca262-flags(Str:D $flags --> Str:D) {
    my $actions = RegexUtils::FlagsEMCA262ModernActions;
    my $match   = RegexUtils::FlagsEMCA262Modern.parse($flags.substr(1), :$actions);
    without $match {
        die "invalid flags";
    }
    return $match.made;
} # method get-modern-emca262-flags(Str:D $flags --> Str:D) #

```

### get-old-emca262-flags(…)

Translate old style **EMCA262Regex** flags into **Raku** ones.

```
method get-old-emca262-flags(Str:D $flags --> Str:D) {
    my $actions = RegexUtils::FlagsEMCA262OldActions;
    my $match   = RegexUtils::FlagsEMCA262Old.parse($flags, :$actions);
    without $match {
        die "invalid flags $flags";
    }
    return $match.made;
} # method get-old-emca262-flags(Str:D $flags --> Str:D) #

```

[Top of Document](#table-of-contents)


## dist_zef-rawleyfowler-Sort-Fast.md
# Sort::Fast

Sort lists and arrays of numeric values very fast.

## How to use

```
use Sort::Fast;

my @foo = [1,82,3,9,10,200,30,4];

@foo = @foo.sort-fast;

say @foo; # [1, 3, 4, 9, 10, 30, 82, 200]

```

## License

This project is available under the `Artistic 2.0` license, please view it at the root of the project for more information.