= Artifact definition format and style guide

:toc:
:toclevels: 4
:icons:

:numbered!:
[abstract]
== Summary

This guide contains a description of the forensics artifacts definitions.
The artifacts definitions are
link:http://www.yaml.org/spec/1.2/spec.html[YAML]-based. The format is
currently still under development and is likely to undergo some change. One of
the goals of this guide is to ensure consistency and readbility of the
artifacts definitions.

[preface]
== Revision history

[cols="1,1,1,5",options="header"]
|===
| Version | Author | Date | Comments
| 0.0.1 | G. Castle | November 2014 | Initial version.
| 0.0.2 | G. Castle | December 2014 | Minor format changes.
| 0.0.3 | J.B. Metz | April 2015 | Merged style guide and artifact definitions wiki page.
| 0.0.3 | J.B. Metz | September 2015 | Additional label.
| 0.0.4 | J.B. Metz | July 2016 | Added information about a naming convention.
| 0.0.5 | J.B. Metz | February 2019 | Removed returned_types as keyword and format changes.
|===

:numbered:
== Background

The first version of the artifact definitions originated from the
https://github.com/google/grr[GRR project], where it is used to describe and
quickly collect data of interest, e.g. specific files or Windows Registry keys.
The goal of the format is to provide a way to describe the majority of forensic
artifacts in a language that is readable by humans and machines.

The format is designed to be simple and straight forward, so that a digital
forensic analysist is able to quickly write artifact definitions during an
investigation without having to rely on complex standards or tooling.

The format is intended to describe forensically-relevant data on a machine,
while being tool agnostic. In particular we intentionally avoided adding
IOC-like logic, or describing how the data should be collected since this
various between tools.

=== Terminology

The term artifact (or artefact) is widely used within computer (or digital)
forensics, though there is no official definition of this term.

The definition closest to the meaning of the word within computer forensics is
that of the word artifact within
http://en.wikipedia.org/wiki/Artifact_(archaeology)[archaeology]. The term
should not be confused with the word artifact used within
http://en.wikipedia.org/wiki/Artifact_(software_development)[software development].

If archaeology defines an artifact as:
```
something made or given shape by man, such as a tool or
a work of art, esp an object of archaeological interest
```

The definition of artifact within computer forensics could be:
```
An object of digital archaeological interest.
```

Where digital archaeology roughly refers to computer forensics without the
forensic (legal) context.

== The artifact definition

The best way to show what an artifact definition is, is by example. The
following example is the artifact definition for the Windows EVTX System Event
Logs.

[source,yaml]
----
name: WindowsSystemEventLogEvtx
doc: Windows System Event log for Vista or later systems.
sources:
- type: FILE
  attributes: {paths: ['%%environ_systemroot%%\System32\winevt\Logs\System.evtx']}
conditions: [os_major_version >= 6]
labels: [Logs]
supported_os: [Windows]
urls: ['http://www.forensicswiki.org/wiki/Windows_XML_Event_Log_(EVTX)']
----

The artifact definition can have the following values:

[cols="1,5",options="header"]
|===
| Value | Description
| name | The name. An unique string that identifies the artifact definition. +
Also see section: <<artifact_name,Name>>.
| doc | The description (or documentation). A human readable string that describes the artifact definition. +
*Style note*: Typically one line description of the artifact, mentioning important caveats. +
If more description is necessary, use the <<artifact_long_docs,Long docs form>>.
| sources | A list of source definitions. +
See section: <<sources,Sources>>.
| conditions | Optional list of conditions that describe when the artifact definition should apply. +
See section: <<conditions,Conditions>>.
| labels | Optional list of predefined labels.
See section: <<labels,Labels>>.
| provides | Optional list of *TODO*
| supported_os | Optional list that indicates which operating systems the artifact definition applies to.
See section: <<supported_os,Supported operating system>>.
| urls | Optional list of URLs with more contextual information. +
Ideally the artifact definition links to an article that discusses the artificat in more depth e.g. on http://forensicswiki.org[Forensics Wiki]
|===

=== [[artifact_name]]Name

*Style note*: The name of an artifact defintion should be in CamelCase name without spaces.

As of July 2016 we are migrating to the following naming convention:

* Prefix platform specific artifact definitions with the name of the operating system using "Linux", "MacOS" or "Windows"
* If not platform specific:
** prefix with the application name, for example "ChromeHistory".
** prefix with the name of the subsystem, for example "WMIComputerSystemProduct".

*Style note*: If the sole source of the artifact definition for example are
files use "BrowserHistoryFiles" instead of "BrowserHistory" to reduce ambiguity.

=== [[artifact_long_docs]]Long docs form

Multi-line documentation should use the YAML Literal Style as indicated by the |
character.

[source,yaml]
----
doc: |
  The Windows run keys.

  Note users.sid will currently only expand to SIDs with profiles on the system,
  not all SIDs.
----

*Style note*: the short description (first line) and the longer portion are
separated by an empty line.

*Style note*: explicit newlines (\n) should not be used.

== [[sources]]Sources

Every source definition starts with a `type` followed by arguments e.g.

[source,yaml]
----
sources:
- type: COMMAND
  attributes:
    args: [-qa]
    cmd: /bin/rpm
----

[source,yaml]
----
sources:
- type: FILE
  attributes:
    paths:
      - /root/.bashrc
      - /root/.cshrc
      - /root/.ksh
      - /root/.logout
      - /root/.profile
      - /root/.tcsh
      - /root/.zlogin
      - /root/.zlogout
      - /root/.zprofile
      - /root/.zprofile
----

*Style note*: where sources take a single argument with a single value, the one-line {}
form should be used to save on line breaks as below:

[source,yaml]
----
- type: FILE
  attributes: {paths: ['%%environ_systemroot%%\System32\winevt\Logs\System.evtx']}
----

[cols="1,5",options="header"]
|===
| Value | Description
| attributes | A dictionary of keyword attributes specific to the type of source definition.
| type | The source type.
| conditions | Optional list of conditions to when the artifact definition should apply. +
See section: <<conditions,Conditions>>.
| supported_os | Optional list that indicates which operating systems the artifact definition applies to. +
See section: <<supported_os,Supported operating system>>.
|===

=== Source types

Currently the following different source types are defined:

[cols="1,5",options="header"]
|===
| Value | Description
| ARTIFACT_GROUP | A source that consists of a group of other artifacts.
| COMMAND | A source that consists of the output of a command.
| FILE | A source that consists of the contents of files.
| PATH | A source that consists of the contents of paths.
| REGISTRY_KEY | A source that consists of the contents of Windows Registry keys.
| REGISTRY_VALUE | A source that consists of the contents of Windows Registry values.
| WMI | A source that consists of the output of Windows Management Instrumentation (WMI) queries.
|===

The sources types are defined in
link:https://github.com/ForensicArtifacts/artifacts/blob/master/artifacts/definitions.py[definitions.py].
as TYPE_INDICATOR constants.

=== Artifact group source

The artifact group source is a source that consists of a group of other artifacts e.g.

[source,yaml]
----
- type: ARTIFACT_GROUP
  attributes:
    names: [WindowsRunKeys, WindowsServices]
----

Where `attributes` can contain the following values:

[cols="1,5",options="header"]
|===
| Value | Description
| names | A list of artifact definition names that make up this "composite" artifact. +
This can also be used to group multiple artifact definitions into one for convenience.
|===

=== Command source

The command source is a source that consists of the output of a command e.g.

[source,yaml]
----
- type: COMMAND
  attributes:
    args: [-qa]
    cmd: /bin/rpm
----

Where `attributes` can contain the following values:

[cols="1,5",options="header"]
|===
| Value | Description
| args | A list arguments to pass to the command.
| cmd | The path of the command.
|===

=== File source

The file source is a source that consists of the contents of files e.g.

[source,yaml]
----
- type: FILE
  attributes:
    paths: ['%%environ_systemroot%%\System32\winevt\Logs\System.evtx']
----

Where `attributes` can contain the following values:

[cols="1,5",options="header"]
|===
| Value | Description
| paths | A list of file paths that can potentially be collected. +
The paths can use parameter expansion e.g. `%%environ_systemroot%%`. +
See section: <<parameter_expansion,Parameter expansion and globs>>
| separator | Optional path segment seperator e.g. '\' for Windows systems. +
When not specified the default path segment separator is '/'.
|===

=== Path source

The path source is a source that consists of the contents of paths e.g.

[source,yaml]
----
- type: PATH
  attributes:
    paths: ['\Program Files']
    separator: '\'
----

Where `attributes` can contain the following values:

[cols="1,5",options="header"]
|===
| Value | Description
| paths | A list of file paths that can potentially be collected. +
The paths can use parameter expansion e.g. `%%environ_systemroot%%`. +
See section: <<parameter_expansion,Parameter expansion and globs>>
| separator | Optional path segment seperator e.g. '\' for Windows systems. +
When not specified the default path segment separator is '/'.
|===

=== Windows Registry key source

The Windows Registry key source is a source that consists of the contents of
Windows Registry keys e.g.

[source,yaml]
----
sources:
- type: REGISTRY_KEY
  attributes:
    keys:
    - 'HKEY_USERS\%%users.sid%%\Software\Microsoft\Internet Explorer\TypedURLs\*'
----

Where `attributes` can contain the following values:

[cols="1,5",options="header"]
|===
| Value | Description
| keys | A list of Windows Registry key paths that can potentially be collected. +
The paths can use parameter expansion e.g. `%%users.sid%%`. +
See section: <<parameter_expansion,Parameter expansion and globs>>
|===

=== Windows Registry value source

The Windows Registry value source is a source that consists of the contents of
Windows Registry values e.g.

[source,yaml]
----
- type: REGISTRY_VALUE
  attributes:
    key_value_pairs:
      - {key: 'HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Explorer\WindowsUpdate', value: 'CISCNF4654'}
----

Where `attributes` can contain the following values:

[cols="1,5",options="header"]
|===
| Value | Description
| key_value_pairs | A list of Windows Registry key paths and value names that can potentially be collected. +
The key path can use parameter expansion e.g. `%%users.sid%%`. +
See section: <<parameter_expansion,Parameter expansion and globs>>
|===

=== Windows Management Instrumentation (WMI) query source

The  Windows Management Instrumentation (WMI) query source is a source that
consists of the output of Windows Management Instrumentation (WMI) queries e.g.

[source,yaml]
----
- type: WMI
  attributes:
    query: SELECT * FROM Win32_UserAccount WHERE name='%%users.username%%'
----

Where `attributes` can contain the following values:

[cols="1,5",options="header"]
|===
| Value | Description
| base_object | Optional WMI base object e.g. `winmgmts:\root\SecurityCenter2`
| query | The Windows Management Instrumentation (WMI) query. +
The query can use parameter expansion e.g. `%%users.username%%`. +
See section: <<parameter_expansion,Parameter expansion and globs>>
|===

== [[conditions]]Conditions

*TODO: work is in progress to move this out of GRR into something more portable.*

Artifact conditions are currently implemented using the
link:https://github.com/google/objectfilter[objectfilter] system that allows
you to apply complex conditions to the attributes of an object. Artifacts can
apply conditions to any of the Knowledge Base object attributes as defined in
the GRR link:https://github.com/google/grr/blob/master/proto/knowledge_base.proto[knowledge_base.proto].

*Style note*: single quotes should be used for strings when writing conditions.

[source,yaml]
----
conditions: [os_major_version >= 6 and time_zone == 'America/Los_Angeles']
----

=== [[supported_os]]Supported operating system

Since operating system (OS) conditions are a very common constraint, this has
been provided as a separate option "supported_os" to simplify syntax. For
supported_os no quotes are required. The currently supported operating systems
are:

* Darwin (also used for Mac OS X)
* Linux
* Windows

[source,yaml]
----
supported_os: [Darwin, Linux, Windows]
----

This can be translated to objectfilter as:

[source,yaml]
----
["os =='Darwin'" OR "os=='Linux'" OR "os == 'Windows'"]
----

== [[labels]]Labels

Currently the following different labels are defined:

[cols="1,5",options="header"]
|===
| Value | Description
| Antivirus | Antivirus related artifacts, e.g. quarantine files.
| Authentication | Authentication artifacts.
| Browser | Web Browser artifacts.
| Cloud Storage | Cloud Storage artifacts.
| Configuration Files | Configuration files artifacts.
| Execution | Contain execution events.
| External Media | Contain external media data or events e.g. USB drives.
| KnowledgeBase | Artifacts used in knowledge base generation.
| Logs | Contain log files.
| Memory | Artifacts retrieved from memory.
| Network | Describe networking state.
| Processes | Describe running processes.
| Software | Installed software.
| System | Core system artifacts.
| Users | Information about users.
| Rekall | Artifacts using the Rekall memory forensics framework.
|===

The labes are defined in
link:https://github.com/ForensicArtifacts/artifacts/blob/master/artifacts/definitions.py[definitions.py].

== Style notes

=== Artifact definition YAML files

Artifact definition YAML filenames should be of the form:
....
$FILENAME.yaml
....

Where $FILENAME is name of the file e.g. windows.yaml.

Each defintion file should have a comment at the top of the file with a
one-line summary describing the type of artifact definitions contained in the
file e.g.

[source,yaml]
----
# Windows specific artifacts.
----

=== Lists

Generally use the short [] format for single-item lists that fit inside 80
characters to save on unnecessary line breaks:

[source,yaml]
----
labels: [Logs]
supported_os: [Windows]
urls: ['http://www.forensicswiki.org/wiki/Windows_XML_Event_Log_(EVTX)']
----

and the bulleted list form for multi-item lists or long lines:

[source,yaml]
----
paths:
  - 'HKEY_USERS\%%users.sid%%\Software\Microsoft\Windows\CurrentVersion\Run\*'
  - 'HKEY_USERS\%%users.sid%%\Software\Microsoft\Windows\CurrentVersion\RunOnce\*'
  - 'HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Run\*'
  - 'HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnce\*'
  - 'HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnceEx\*'
----

=== Quotes

Quotes should not be used for doc strings, artifact names, and simple lists
like labels and supported_os.

Paths and URLs should use single quotes to avoid the need for manual escaping.

[source,yaml]
----
paths: ['%%environ_temp%%\*.exe']
urls: ['http://www.forensicswiki.org/wiki/Windows_XML_Event_Log_(EVTX)']
----

Double quotes should be used where escaping causes problems, such as
regular expressions:

[source,yaml]
----
content_regex_list: ["^%%users.username%%:[^:]*\n"]
----

=== Minimize the number of definitions by using multiple sources

To minimize the number of artifacts in the list, combine them using the
supported_os and conditions attributes where it makes sense. e.g. rather than
having FirefoxHistoryWindows, FirefoxHistoryLinux, FirefoxHistoryDarwin, do:

[source,yaml]
----
name: FirefoxHistory
doc: Firefox places.sqlite files.
sources:
- type: FILE
  attributes:
    paths:
      - %%users.localappdata%%\Mozilla\Firefox\Profiles\*\places.sqlite
      - %%users.appdata%%\Mozilla\Firefox\Profiles\*\places.sqlite
  supported_os: [Windows]
- type: FILE
  attributes:
    paths: [%%users.homedir%%/Library/Application Support/Firefox/Profiles/*/places.sqlite]
  supported_os: [Darwin]
- type: FILE
  attributes:
    paths: ['%%users.homedir%%/.mozilla/firefox/*/places.sqlite']
  supported_os: [Linux]
labels: [Browser]
supported_os: [Windows, Linux, Darwin]
----

== [[parameter_expansion]]Parameter expansion and globs

*TODO*

