### ----------------------------------------------------
### -- Collection
### -- Licenses: Artistic-2.0
### -- Authors: Richard Hainsworth, aka finanalyst
### -- File: README.md
### ----------------------------------------------------
### -- Chunk 2 of 2
**

This path must be prepended to any sources added (see below) to the cache, otherwise they will not be added to the destination file.

*  **$mode-root**

Likewise for the mode sources.

*  **%plugin-options**

Has the values of plugin options that over-ride a plugin's own defaults. See [Plugin level configuration](Plugin level configuration.md) for more information.

*  **%options**

Has the values of 'collection-info' and 'no-status' flags.

### Render
The Collection plugin-manager calls the `ProcessedPod.add-plugin` method with the config keys and the path modified to the plugin's subdirectory. The ProcessPod instance is only modified by `Render` plugins, so if there is plugin configuration data that is needed by another Milestone callable, the plugin should call a blank callable, with empty block and templates.

If the `render` key is True, no callable is provided, and the plugin name will be added via the **.add-plugin** method of the `ProcessedPod` object. See `ProcessedPod` documentation.

If the `render` key is a Str, then it is the filename of a Raku callable of the form

```
sub ( $pr, %options --> Array ) {...}
```
where

*  **$pr** is a <ProcessedPod> object,

*  **%options** is the same as for Setup, and

*  the callable **returns** a list of triples, with the form (to, from-plug, file)

	*  **to** is the destination under the `%config<destination> ` directory where the asset will be looked for, eg., an image file to be served.

	*  **from-plug** is the name of the plugin in whose directory the asset is contained, where the value `myself` means the path of the plugin calling the render callable. Actually, 'myself' is the value of Collection::MYSELF.

	*  **file** is the filename local to the source plugin's subdirectory that is to be copied to the destination. This may contain a path relative to the plugin's subdirectory.

	*  For example for an HTML file, this would be the relative URL for the `src` field. Eg., `to = 'asset/image'`, `file = 'new-image.png'`, `from-plug = 'myself'` and the html would be `<img src="asset/image/new-image.png" /> `.

Since a render plugin is to be added using the `ProcessedPod` interface, it must have the `custom-raku` and `template-raku` keys defined, even if they evaluate to blank (eg. `:custom-raku()` ).

So the config file must have:

*  render (True | name of callable)

*  custom-raku => a Raku program that evaluates to an array of custom blocks (must be set to `()` if no Raku program )

*  template-raku => a Raku program that evaluates to a hash of RakuClosure templates (must be set to `()` if no Raku program)

It is possible to specify `path` but it must be relative to the plugin's sub-directory.

### Compilation
Note that the structure files are rendered after the `compilation` stage, BUT the information for rendering the structure files, that is the custom blocks and the templates must accompany a `render ` plugin. Compilation plugin callables are to process the data accumulated during the rendering of the content files, and to make it available for the custom blocks / templates that will be invoked when the structure documents are rendered.

The `compilation` key must point to a Raku program that delivers a sub object

```
sub ( $pr, %processed, %options ) { ... }
```
> **$pr**  
is the ProcessedPod object rendering the content files.

> **%processed**  
is a hash whose keys are source file names with a hash values containing TOC, Glossary, Links, Metadata, Footnotes, Templates-used structures produced by B<ProcessedPod>.

> **%options**  
as for setup

If the return value of the callable is an Array of triplets (as for a Render callable), then assets are transferred from the plugin directory. Any other type of return value is ignored.

### Transfer
The `transfer` key points to a Raku file that evaluates to a

```
sub ($pr, %processed, %options --> Array ) {...}
```
> **%processed**  
as in Compilation

> **$pr**  
as in Compilation

> **%options**  
as for Setup

> **return object**  
as for the compilation plugin

## Report
The `report` key points to a Raku file that evaluates to a

```
sub (%processed, @plugins-used, $pr, %options --> Array ) {...}
```
> **%processed**  
as in Compilation

> **@plugins-used**  
is an array of Pairs whose key is the milestone and value is a hash of the plugins used and their config parameters.

> **$pr**  
as in Compilation

> **%options**  
as for Setup

The plugin should return an Array of Pair, where for each Pair .key = (path/)name of the report file with extension, and .value is the text of the report in the appropriate format

The `collect` sub will write the file to the correct directory.

### Completion
The `completion` key points to a Raku file that evaluates to a

```
sub ($destination, $landing-place, $output-ext, %plugin-options, %options) {...}
```
*  **$destination**

is the name of the output path from the mode directory (defined in the mode configuration)

*  **$landing-place**

is the first file to be processed since, eg., for a website, order is not sufficient. name is relative to the destination directory.

*  **$output-ext**

is the output extension.

*  **%plugin-options**

As for Setup

*  **%options**

As for Setup

There is no return value specified for this plugin type.

# Plugin development
There is a separate development distribution `raku-collection-plugin-development`, which contains several tools for adding and testing plugins. However, a single plugin can be tested using the module `Collection::TestPlugin`, which is included in this distribution.

## Collection plugin specification
All Collection plugins must conform to the following rules

*  The plugin name must:

	*  start with a letter

	*  followed by at least one \w or \-

	*  not contain \_ or \.

	*  thus a name matches / <alpha> <[\w] + [\-] - [\_]>+ /

*  The plugin directory contains

	*  `config.raku`, which is a file that evaluates to a Raku hash.

	*  README.rakudoc

	*  t/01-basic.rakutest

*  The `config.raku` file must contain the following keys

	*  `name`. This is the released name of the plugin. It will be possible for a new plugin to have the same functionality as another, while extending or changing the output. For more detail, see [Collection plugin management system](Collection plugin management system.md). Typically, the name of the plugin will match the name of the sub-directory it is in.

	*  `version`. This point to a Str in the format \d+\.\d+\.\d+ which matches the semantic version conventions.

	*  `auth`. This points to a Str that is consistent with the Raku 'auth' conventions.

	*  `license`. Points to a SPDX license type.

	*  `authors`. This points to a sequence of names of people who are the authors of the plugin.

	*  one or more of `render setup report compilation completion`

		*  If render then

			*  the render key may be a boolean

			*  or the render key is a Str which must

				*  be a filename in the directory

				*  be a raku program that evaluated to a callable

				*  the callable has a signature defined for render callables

			*  the key `custom-raku`

				*  must exist

				*  must either be empty, viz. `custom-raku()`

				*  or must have a Str value

				*  if it has a Str value and the key `:information` does contain `custom-raku` then it is treated as if `custom-raku` is empty

				*  if it has a Str value and the key `:information` does NOT contain `custom-raku` then the Str value should

					*  point to a file name in the current directory

					*  and the filename should contain a Raku program that evaluates to an Array.

			*  the key `template-raku`

				*  must exist

				*  must either be empty, viz. `template-raku()`

				*  or must have a Str value

				*  if it has a Str value and the key `:information` does contain `template-raku` then it is treated as if `template-raku` is empty

				*  if it has a Str value and the key `:information` does NOT contain `template-raku` then the Str value should

					*  point to a file name in the current directory

					*  and the filename should contain a Raku program that evaluates to a Hash.

		*  If not render, then the value must point to a Raku program and evaluate to a callable.

	*  _Other key names_. If other keys are present, they must point to filenames in the current directory.

	*  `information`. This key does not need to exist.

		*  If it exists, then it must contain the names of other keys.

		*  If a key named in the `:information` list contains a Str, the filename will NOT exist in the plugin directory, but will be generated by the plugin itself, or is used as information by the plugin.

		*  This config key is intended only for plugin testing purposes.

## Collection plugin tests
This distribution contains the module `Collection::TestPlugin` with a single exported subroutine `plugin-ok`. This subroutine verifies that the plugin rules are kept for the plugin.

Additional plugin specific tests should be included.

# Plugin updating
The local computer may contain

*  More than one collection, eg. a personal website and a Raku documentation collection

*  Each Collection may have more than one Mode, eg., a HTML website, an epub producer.

*  A collection/mode combination may rely on a previous API of a plugin, which may be broken by a later API.

*  A new plugin may have been written as a drop-in replacement for an older version, and the new plugin may have a different name, or auth, or version.

In order to implement this flexibility, the following are specified:

*  There is a released plugins directory (see [Released plugins directory](Released plugins directory.md)) to contain all Collection plugins.

*  The semantic versioning scheme is mandated for Collection plugins. A version is `v<major>.<minor>,<patch>`. Changes at the `<patch> ` level do not affect the plugin's functionality. Changes at the `<minor> ` level introduce new functionality, but are backward compatible. Changes at the `<major> ` level break backward compatibility.

*  Each distributed plugin is contained in the release directory in a subdirectory named by the plugin name, the auth field, and the major version number (minor and patch numbers are not included because they should not by definition affect the API).

*  Each Mode configuration only contains the name of the plugin (without the auth, or version names).

*  The developer may define which name/version/auth of a released plugin is to be mapped to the plugin required in the Mode configuration. Thus

	*  changes in API can be frozen to earlier versions of the plugin for some concrete Mode.

	*  different plugin versions can be used for different collection/modes

	*  a differently named plugin can be mapped to a plugin required by a specific collection/mode.

		*  **Note** an alternate plugin, as given by `name` may also have a non-default `auth`, so `auth` may need to be added to `plugins.rakuon` as well.

*  Consequently, all released plugins are defined for

	*  a **format** (eg. html)

	*  a **major** version

	*  an **auth** name

*  The mapping information from a released plugin to a Mode is contained in a file in the root of a Collection.

*  When the plugins are updated

	*  all the latest versions for each Format/Name/Version/Auth are downloaded.

	*  a symlink is generated (or if the OS does not allow symlink, the whole directory is copied) from the released version to the directory where each mode expects its plugins to be located.

*  Each Collection root directory (the directory containing the topmost `config.raku` file) will contain the file `plugins.rakuon`.

*  The plugin management tool (PMT)

	*  checks if a `plugins.rakuon` exists. If not, it generates a minimal one.

	*  runs through the plugins-required of each **Mode** in the collection.

	*  for each distinct plugin verifies whether

		*  the plugin has an entry in `plugins.rakuon`, in which case

			*  the PMT maps (or remaps if the constraint is new) the released plugin name/auth/ver to the plugin-required name using the rules of `plugins.rakuon` as given in [Mapping released plugins to mode directories](Mapping released plugins to mode directories.md)

		*  the plugin does not have an entry, which means it has not yet been mapped, and there are no constraints on the plugin, so the default name/auth/version are used.

## Mapping released plugins to mode directories
The file `plugins.rakuon` contains a hash with the following keys:

*  `_metadata_`. Contains a hash with data for the `refresh` functionality.

	*  `collection-plugin-root` This contains the name of a directory reachable from the Collection root directory with the released plugins are downloaded.

	*  `update-behaviour` Defines what happens when a new released plugin has an increased Major number. Possible values are:

		*  _auto_ All relevant plugin names are updated to the new version, a message is issued

		*  _conserve_ Plugins are not updated to the new version, a warning is issued, updating is then plugin by plugin with a `-force` flag set.

		*  **Note** The update behaviour is not initially implemented.

*  Every other toplevel key that meets the plugin naming convenstion is interpreted as a Mode. This means a mode cannot be named `_metadata_`.

*  The Mode key will point to a hash with the keys:

	*  `_mode_format` Each mode may only contain plugins from one Format, eg., _html_.

	*  By the plugin naming rules, a _plugin_ may not be named `_mode_format`.

	*  Every other key in a mode hash must be a plugin name contained in the Mode's plugins-required configuration.

	*  There may be zero plugin keys

	*  If a plugin in the plugins-required configuration list does not have an entry at this level, then it has not been mapped to a sub-directory of the `released-directory`.

	*  A plugin key that exists must point to a Hash, which must at least contain:

		*  mapped => the name of the released plugin

	*  The plugin hash may also point to one or more of:

		*  name => the name of the alternate released plugin

			*  the default name (if the key is absent) is the plugin-required's name

			*  if a different name is given, a released plugin is mapped to the required directory in the mode sub-directory

		*  major => the major number preceeded by 'v'

			*  the default value (if the key is absent) is the greatest released major value

			*  A major value is the maximum major value of the full version permitted, thus freezing at that version

		*  auth => the auth value

			*  the default value (if the key is absent) is 'collection'

	*  If there is no distributed plugin for a specified `auth | major | name `, then an error is thrown.

Some examples:

*  The Raku-Collection-Raku-Documentation, Website mode, requires the plugin `Camelia`. The plugin exists as a HTML format. It has version 1.0.0, and an auth 'collection'. It is distributed as `html/camelia_v1_auth_collection`. Suppose a version with a new API is created. Then two versions of the plugin will be distributed, including the new one `html/camelia_v2_auth_collection`.

If the key for camelia in the hash for mode Website only contains an empty `version` key, then the defaults will be implied and a link (or copy) will be made between the released directory `html/camelia_v2_auth_collection` and `Website/plugins/camelia`

*  If plugins.rakuon contains the following: `Website =` %( :FORMA"> \{\{\{ contents }}}

, :camelia( %( :major(1), ) ) > then the link will be between `html/camelia_v1_auth_collection` and `Website/plugins/camelia`

*  Suppose there is another valid `auth` string **raku-dev** and there is a distributed plugin _html/camelia_v2_auth_raku-dev_, and suppose `plugins.rakuon` contains the following: `Website =` %( :FORMA"> \{\{\{ contents }}}

, :camelia( %( :auth<raku-dev>, ) ) > then the link will be made between `html/camelia_v2_auth_raku-dev` and `Website/plugins/camelia`

*  Suppose a different icon is developed called `new-camelia` by `auth` **raku-dev**, then `plugins.rakuon` may contain the following: `Website =` %( :FORMA"> \{\{\{ contents }}}

, camelia( %( :name<new-camelia>, :auth<raku-dev>, ) ) > then a link (copy) is made between `html/new-camelia_v2_auth_raku-dev` and `Website/plugins/camelia`

	*  Note how the auth must be given for a renaming if there is not a `collection` version of the plugin

## Released plugins directory
When Collection is installed, a directory called (by default) `$*HOME/.local/share/Collection` is created (on a Linux system this will be the same as `~/.local/share/Collection`).

If the Environment variable `PluginPath` is set to a valid path name upon installation, then that will be used instead. But when `refresh-collection-plugins` is first used, then the non-default name must be supplied.

The directory is initialised to point at the `https://github.com/finanalyst/collection-plugins` repo and the `manifest.rakuon` file is downloaded.

If another release directory location is desired, then

*  it can be specified when `refresh-collection-plugins` is first run for each collection, in which case the file `plugin.rakuon` is given the value for the `:collection-plugin-root`.

*  a custom Raku program for refreshing Collection files can be written, eg., the github repo is to be located in `~/.my_own_collection`

```
use v6.d;
use Collection::RefreshPlugins;
sub MAIN(|c) {
    $Collection::RefreshPlugins::release-dir = "$*HOME/.my_own_collection";
    refresh( |c );
}

```
## Refresh process
The intent is for the released plugins to be held in a single directory (called the _released plugins_ directory), and for the references in a Collection-Mode `plugins` directory to be links (OS dependent) to the _released plugins_ directory.

The _released plugins_ directory is a Github repository, so doing a `git pull` will pull in the most recent versions of the plugins. Consequently, each Collection-Mode plugin reference will automatically be updated.

A `git pull` is therefore one form of a refresh. (TODO if an OS does not have directory links, then this form of refresh will need to be enhanced with a copy operation).

Refresh needs to deal with other situations

*  A new plugin name is added to the Collection-Mode's `plugins-required` list.

	*  A link needs to be added to the Collections-Mode's `plugins` directory.

*  A new entry exists for the plugin name in the `plugins.rakuon`

	*  An entry changes the release name associated with the Collection-Mode plugin.

		*  If the desired released plugin does not exist, then an Exception is thrown. Default behaviour might lead to an infinite loop.

*  A released plugin major version has increased since the last refresh.

	*  The default is for a Collection-Mode plugin initially to be linked to the most recent version of the plugin

	*  Several behaviours are possible:

		*  The default _update-behaviour_ is given in the `_metadata_` hash of `plugins.rakuon`

		*  _force_ Leave the existing links in place, issue a warning, update to latest only when forced. Include suggestion to change plugins.rakuon file to suppress warnings.

		*  _auto_ (default) Automatically update to the latest version, issuing a message

## CLI Plugin Management System
Collection contains the utility `collection-refresh-plugins` as a (PMS). It is called as follows:

```
collection-refresh-plugins [-collection=<path>] [-collections='path1 path2 path3']
```
`-collection` is the path to a Collection directory. By default it is the Current Working Directory.

`-collections` is a space delimited list of paths to Collections

When a Collection directory contains a file `plugins.rakuon`, then the utility will inspect the release directory, updates it, and maps (copy) the most recent plugins according this file. See below for more detail about the specification of `plugins.rakuon`.

When a Collection does not contain a file <plugins.rakuon>, it generates one from the `plugin-required` key of each of the `config.raku` files in each Mode. During this process, the user is prompted for the directory name (relative to the current working directory) of released plugin directory.

# Problems and TODO items
## Archiving and Minor Changes
In principle, if a small change is made in a source file of a Collection, only the rendered version of that file should be changed, and the Collection pages (eg., the index and the glossaries) updated. The archiving method chosen here is based on `Archive::Libarchive` and a `.7z` format. It works in tests where a small quantity of data is stored.

**However**, when there are many source files (eg., the Raku documentation), the process of restoring state information is **significantly** longer than re-rendering all the cached files. Consequently, the option `no-preserve-state` prevents the archiving of processed state. (TODO understanding and optimising the de-archiving process.)

## Dump file formatting
The aim is to use `PrettyDump` instead of <.raku> to transform information into text. However, <PrettyDump> does not handle `Bags` properly.

# Post-cache methods
Post-cache is a role added to a `Pod::From::Load` object so that Setup plugin callables can act on Cache'd content by adding pod files to the Cache (perhaps pre-processing primary source files) that will be rendered, masking primary pod files so that they are not rendered, or aliasing primary pod files.

If a secondary source file in the Cache is given a name that is the same as a primary source file, then if the underlying cache object should remain visible, another name (alias) should be given to the file in the Post-cache database.

The Post-cache methods `sources`, `list-files`, and `pod` have the same function and semantics as `Pod::From::Cache` except that the post-cache database is searched first, and if contents are found there, the contents are returned (which is why post-cache file names hide primary file names). If there is no name in the Post-cache database, then it is passed on to the underlying cache.

## multi method add(Str $fn, Array $p --&gt; Pod::From::Cache )
Adds the filename $fn to the cache. $p is expected to be an array of Pod::Blocks, but no check is made. This is intentional to allow the developer flexibility, but then a call to `pod( $fn )` will yield an array that is not POD6, which might not be expected.

The invocant is returned, thus allowing add to be chained with mask and alias.

## multi method mask(Str $fn --&gt; Pod::From::Cache)
This will add only a filename to the database, and thus mask any existing filename in the underlying cache.

Can be chained.

## multi method add-alias(Str $fn, Str :alias! --&gt; Pod::From::Cache)
This will add a filename to the database, with the value of a key in the underlying cache. Chain with mask to prevent the original spelling of the filename in the underlying cache being visible.

Can be chained.

If the alias is already taken, an exception is thrown. This will even occur if the same alias is used for the same cached content file.

## method behind-alias(Str $fn --&gt; Str )
Returns the original name of the cached content file, if an alias has been created, otherwise returns the same filename.

## method pod(Str $fn)
Will return

*  an array of Pod::Block (or other content - beware of adding other content) if the underlying Cache or database have content,

*  the array of Pod::Block in an underlying filename, spelt differently

*  `Nil` if there is no content (masking an underlying file in Cache)

*  throw a NoPodInCache Exception if there is no pod associated with either the database or the underlying cache. If the original filename is used after an alias have been generated, the Exception will also be thrown.

## multi method last-version( @version-data )
The `@version-data` contains an array of strings to be given to a `run` sub. Typically it is a call to `git revparse`. The return value is the result of the command to the Operating System.

# head

multi method last-version( @per-file-version, $fn, $doc-source, :debug)

The `@per-file-version` array of strings is appended by the value of $fn, changed to the underlying source if it is an alias.

Typically, the string is a git command. However, the git command in particular runs on a directory, but the source file may be in a sub-directory, which needs to be stripped, so `$doc-source` is provided to enable the correct filename.

`:debug` generates information about the `run` command, if true.

# Asset-cache methods
Asset-cache handles content that is not in Pod6 form. The instance of the Asset-cache class is passed via the plugin-data interface of `ProcessedPod`, so it is available to all plugin callables after the setup milestone, for example in the plugin callable:

```
sub render-plugin( $pp ) {
    my $image-manager = $pp.get-data('image-manager');
    ...
    $pp.add-data('custom-block', $image-manager);
}
```
By creating a name-space in the plugin data section and assigning it the value of the image-manager, the plugin callable can make the image-manager available to templates that get that data, which is a property in parameters called by the name-space.

`ProcessedPod` provides data from the name-space of a Block, if it exists, as a parameter to the template called for the Block. Note that the default name-space for a block is all lower-case, unless a `name-space` config option is provided with the Pod Block in the content file.

If a plugin provides an asset (eg., image, jquery script), it needs to provide a `render` callable that returns the triple so that Collect moves the asset from the plugin directory to the output directory where it can be served. This needs to be done separately if a CSS contains a url for local image.

The basename for the assets is set in the Top level configuration in the option `asset-basename`

`$image-manager` is of type Asset-cache, which has the following methods:

```
    #| the directory base, not included in filenames
    has Str $.basename is rw;
    #| the name of the file being rendered
    Str $.current-file
    #| asset-sources provides a list of all the items in the cache
    method asset-sources
    #| asset-used-list provides a list of all the items that referenced by Content files
    method asset-used-list
    #| asset-add adds an item to the data-base, for example, a transformed image
    method asset-add( $name, $object, :$by = (), :$type = 'image' )
    #| returns name / type / by information in database (not the object blob)
    method asset-db
    #| remove the named asset, and return its metadata
    method asset-delete( $name --> Hash )
    #| returns the type of the asset
    method asset-type( $name --> Str )
    #| if an asset with name and type exists in the database, then it is marked as used by the current file
    #| returns true with success, and false if not.
    method asset-is-used( $asset, $type --> Bool )
    #| brings all assets in directory with given extensions and with type
    #| these are set in the configuration
    multi method asset-slurp( $directory,  @extensions, $type )
    #| this just takes the value of the config key in the top-level configuration
    multi method asset-slurp( %asset-paths )
    #| with type 'all', all the assets are sent to the same output directory
    multi method asset-spurt( $directory, $type = 'all' )
    #| the value of the config key in the mode configuration
    multi method asset-spurt( %asset-paths )

```
# Copyright and License
(c) Copyright, 2021-2022 Richard Hainsworth

**LICENSE** Artistic-2.0







----
Rendered from README at 2024-04-22T12:38:14Z


