\chapter{Managing resources}
\label{sec:resourcesManagement}

This chapter focuses on the resource storage architecture of \emph{HyCube} and describes individual DHT operations. Possible operations on the resources in \emph{HyCube} include inserting resources to the DHT (PUT) and retrieving resources (GET). Additionally, \emph{HyCube} supports refreshing resources (REFRESH) - updating resource expiration times, removing resources from the DHT (DELETE), and resource replication among nodes. In \emph{HyCube}, the nature of resources, as well as the way of calculating the resource key depends at the application - \emph{HyCube} provides generic support for any resource type. The key (hash) is expected to be within the node identifier space - $(d \cdot l)$-bit number ($d$ - the number of dimensions, $l$ - the number of levels of the hierarchical hypercube), and the node responsible for resources with the hash value $key$ is the closest node to $key$ (Euclidean metric) among all nodes existing in the system. However, to increase the availability and provide load balancing, the resource may be replicated to other nodes (certain number of closest nodes to $key$).

There are two main approaches for storing resources - one is based on storing the actual resources (for example files) by the DHT nodes to which the resource keys are mapped, while the second approach is based on storing pointers by the DHT nodes - the information where individual resources are stored (for example from which nodes particular files may be downloaded). The technique adopted by \emph{HyCube} is universal and the choice which approach is used is made at the application level. The resource entries in \emph{HyCube} (stored by nodes) consist of a resource descriptor (metadata) and resource data. However, the data may be empty, and the pointer to the actual data might be stored within the resource descriptor, or within the data field. The resource descriptor consists of a certain number of key-value pairs (both keys and values are text strings). There are four predefined keys for resource descriptors:

\pagebreak

\begin{itemize}
	\renewcommand{\labelitemi}{$\bullet$}
	\item \emph{\textbf{resourceId}}	- should be a unique identifier of the resource, allowing to unambiguously distinguish it from any other resource
	\item \emph{\textbf{resourceUrl}}	- determines the location of the resource, allowing to unambiguously locate the resource copy among possible multiple copies/instances of the same resource (having the same value of resourceId)
	\item \emph{\textbf{resourceName}}	- the name of the resource
	\item \emph{\textbf{resourceType}}	- resource type (possible values, as well as the way the type is used, are application-specific)
\end{itemize}

Specifying values for \emph{resourceId} and \emph{resourceUrl} is mandatory, because these values are used by the resource management algorithms (storing, refreshing, getting, deleting and replication). A resource key may theoretically be the same for multiple different resources (usually it is the result of a hash function on the resource name or content), and \emph{resourceId} is essential to unambiguously distinguish the resources. If the resource entries stored in the DHT are used only as references to nodes storing actual resources, \emph{resourceUrl} value should make it possible to uniquely locate the resource copy that the entry points. Depending on a system parameter value, nodes may store multiple copies of the same resource if the \emph{resourceUrl} values are different.

In addition to four predefined resource descriptor keys, at the application level, additional keys may be defined. The resource metadata might then be used to apply additional query criteria on the resources returned by nodes - only resources matching all criteria specified in queries (GET) would be returned to the requesting node. The criteria might also be used when deleting resources from the DHT.

The following sections describe the operations supported by \emph{HyCube} DHT. The discussion focuses on operations on resource entries and assumes that they contain the resource data. If the data is, in fact, located somewhere else, and resource entries are only pointers, it is assumed that the actual data insertion, retrieval and deletion is handled by the external application, beyond the scope of \emph{HyCube}.

\section{Storing resources in the DHT}

The PUT operation allows a node to store a resource in the distributed hash table. A node initiating a PUT operation routes a PUT message containing the resource descriptor (and data) specifying the resource key as the message recipient. The message is routed to the closest node to the specified key (until no further routing is possible), which saves the resource in its local storage and sends the status of the operation (PUT\_REPLY message containing the information whether the resource was saved in the node's local storage or not) directly to the requesting node. Before storing the resource, the closest node checks whether it is one of the $k_{store}$ closest nodes to the specified key ($k_{store}$ is the system parameter). Based on this check, the node decides whether the resource should be accepted or not. The check is made based on analyzing the local neighborhood set. For that purpose, a local estimation of the network ``density'' $\rho$ (the author's definition) is calculated as follows:

\begin{equation}
\rho = \frac{\sum_{i=0}^{t} \rho_i}{t+1}
\end{equation}

\noindent
where $\rho_i$ is defined as follows:

\begin{equation}
\rho_i = \frac{\left|\{N \in NS : dist(N) \leq dist(N_i)\}\right|}{dist(N_i)^d}
\end{equation}

\noindent
where $NS$ is the neighborhood set, $dist(N)$ denotes the distance to node $N$, $d$ is the number of dimensions of the hierarchical hypercube, and $t \in [0, |NS|-1]$ is the node index (in the neighborhood set ordered ascending by distances) calculated as follows:

\begin{equation}
t = \max \big(0, \mathop{\mathrm{round}}(\varphi \cdot |NS|) - 1 \big)
\end{equation}

\noindent
where $\varphi \in (0,1]$ is a system parameter - the quantile function threshold, determining what number of closest neighborhood set nodes should be taken into account for calculating the density. Each density $\rho_i$ represents the number of nodes in a unit $d$-sphere calculated based on $(i+1)$-th node (the value $dist(N_i)^d$ in the denominator is proportional to the volume ($d$-dimensional volume) of a unit $d$-sphere, so $\rho_i$ is proportional to the real local density). $\rho$ is the average density - based on all $t+1$ nodes. Calculating the average value makes the density estimation less vulnerable to fluctuations of single node distances.

Based on $\rho$, it is possible to estimate the number of nodes in the system in a $d$-sphere of any given radius $r$:

\begin{equation}
n = \rho \cdot r^d
\end{equation}

\noindent
Thus, the estimated radius of the $d$-sphere containing $k$ closest nodes to any given key may be calculated as:

\begin{equation}
r_k = \left(\frac{k}{\rho}\right)^{\frac{1}{d}}
\end{equation}

Having calculated $r_k$ for $k = k_{store}$, it is possible to estimate whether the node is one of the $k_{store}$ closest nodes to any given key - the node should check if the distance to the key is less than or equal to $r_k$. However, such en estimation might not be accurate enough. The radius used for the density calculation is the minimum radius covering the number of nodes considered (distance to $N_i$), which may cause nodes to incorrectly estimate the density, resulting in the number of nodes accepting resources being smaller than $k_{store}$. Taking this problem into account, it is necessary to scale the estimated value $r_k$, and the node should rather check whether the following condition is met:

\begin{equation}
dist(key) \leq r_k \cdot \xi
\end{equation}

\noindent
$\xi$ is the system parameter - estimated distance coefficient. The value should be adjusted for the chosen value of $\varphi$. The higher is the value of this parameter, the more nodes will accept resources, considering themselves as one of $k_{store}$ closest nodes to the resource key. The value should be large enough to ensure that resources will be accepted by at least $k_{store}$ nodes in the system, regardless of their key. Because for certain nodes, the locally calculated density may be smaller than the average density, for certain keys, more nodes than $k_{store}$ could potentially accept the resource. This is however not a serious issue, meaning that the resource will be more likely to be accepted in less dense areas, which would have positive impact on replication under churn (replication is discussed in Section \ref{sec:replication}).

When the neighborhood set nodes selection forces the diversity of directions in which the nodes are located, the most distant neighbors might not reflect the real closest existing nodes (omitting some close nodes). In such a case the real density could be higher than estimated. Performed simulations showed that, in such a case, the numbers of nodes accepting resources start growing rapidly for $\varphi > 0.5$. Because forcing uniform distribution of neighborhood set nodes is the default behavior in \emph{HyCube}, $\varphi$ should be given a value smaller than 1 to achieve better accuracy (skipping the most distant nodes). However, too small values of $\varphi$ would cause the density to be calculated taking only a very small number of nodes into account. Although the density estimation may skip some close nodes even for smaller values of $\varphi$, the performed simulations indicated that the value $\varphi=0.5$ provides good and relatively stable density estimation (relatively small variance of the numbers of accepting nodes).

The simulation results indicated that, for $\varphi=0.5$, the values of $\xi \geq 1.1$ allow resources to be accepted by at least $k_{store}$ nodes for great majority of randomly generated keys. For values $\xi \geq 1.2$, almost for all tests, at least 8 nodes accepted the resources. $\xi$ should be given a higher value if the nodes should be accepted by more than $k_{store}$ nodes, for example to force accepting resources by each of $k_{store}$ closest nodes to the key (which is a different condition than accepting resources by any $k_{store}$ nodes\footnote{Because the local density may be different in the proximity of the estimating node, and the resource key, certain more distant nodes may accept the resource, while some close nodes may reject the resource.}). It may, however, cause resources to be potentially accepted by a larger number of nodes. When resources are required to be accepted by a certain number of closest nodes (by all of the closest nodes) with high probability, it may also be ensured by increasing the value of $k_{store}$, leaving the value $\xi$ equal to 1.1 or 1.2, which would also cause the resources to be accepted by more nodes, at the same time, increasing the probability of accepting resources by the closest nodes to the resource keys. The values of these parameters should be adjusted to actual needs, depending on the application.

When a node accepts a resource, it stores the resource in its local storage, making it accessible for other nodes to retrieve (GET). Because potentially, there might be many resource entries with the same key, but not representing the same resource (different \emph{resourceId}), such resources are considered separate entities and are stored separately. Furthermore, it is possible for a node (depending on the value of a system parameter) to store multiple resources with the same \emph{resourceId}, but having different values of \emph{resourceUrl}. Such resource entries represent the same resource located physically in different locations. The maximum number of resource entries stored for any \emph{resourceId} is determined by the value of another system parameter.

When resources are being stored by nodes, they are assigned expiration times (the resource validity time is determined by a system parameter). After the expiration time elapses for any resource, the resource is deleted from node's local storage.

\section{Refreshing resources in the DHT}

As described in the previous section, every resource is given an expiration time, and the resource entry contains the last refresh time. After the expiration time (\emph{refreshTime} + \emph{expirationTime}), the resource is deleted from the storing node's local storage. To prevent deleting resources, the node placing the resource in the DHT may send a REFRESH\_PUT message to the node storing the resource, which would renew its refresh time. The new refresh time is sent within the refresh message. However, if the refresh time received is greater than the current local time, the new refresh time is given the value of the current local time. To avoid problems caused by two nodes potentially located in different time zones, the refresh time should always be converted to the UTC (Coordinated Universal Time). The REFRESH\_PUT message is routed towards the resource key, and when no next hop is found (closest node was reached), a check is made whether the resource with the specified \emph{resourceId} and \emph{resourceUrl} is present in the local storage and whether the node is one of the $k_{store}$ closest nodes to the resource key. If both conditions are met, the node updates the refresh time for the resource entry and sends a REFRESH\_PUT\_REPLY message directly to the requesting node. The REFRESH\_PUT\_REPLY contains the status of the operation. If at least one of the checks fails, a negative response is sent.

This mechanism is supposed to prevent maintaining resources inserted to the DHT when they are no longer current. Nodes publishing resources in the DHT should therefore periodically refresh their validity time, and the refresh interval should be smaller than the resource expiration time.

\section{Retrieving resources from the DHT}

In order to retrieve (GET) resources from the DHT, a node sends a GET message containing the requested key and additional criteria, specifying the requested key as the message recipient. Depending on the value of the request parameter \emph{GetFromClosest}, the message is either routed to the node closest to the resource key, which searches its local storage for the resources according to the specified criteria, or the resource(s) are returned by the first node on the route that is able to return the resource(s). The returning node sends back (directly) a GET\_REPLY message containing the results, or an empty result, depending whether the resources were found or not. If \emph{GetFromClosest} equals \emph{false} (the resource is supposed to be returned by the first node on the route storing it), every node on the route (including the initiating node itself) first makes a check, whether the node is one of $k_{store}$ closest nodes to the resource key, and whether the resource requested is stored in the local storage. If both conditions are met, the result is immediately returned directly to the requesting node. Otherwise, the node tries to route the message. If, however, the message cannot be routed because no next hop is found in the routing tables (the closest node is reached), the node returns the results found locally (if any), even it is not one of the $k_{store}$ closest nodes to the resource key. This method of retrieving resources yields very good properties in terms of load balancing if the resource is replicated to multiple nodes (Section \ref{sec:replication}). When the message reaches one of the $k_{store}$ closest nodes, this node is likely to contain all published copies of the resource, while if the route is broken before one of the nodes maintaining the resource (considering itself as one of the $k_{store}$ closest nodes) is reached, this node may return only partial result or not return anything at all (resources might have been stored in that node in the past, and, in some cases, resources may be replicated to more than $k_{store}$ nodes).

Every GET request, in addition to the resource key, contains additional resource search criteria (key-value pairs). The criteria key set should be a subset of the resource descriptor key set. Before returning the GET result, nodes filter out the resources, and only resources matching the criteria are returned (matching resource descriptor values for all the keys specified in the criteria).

\section{Resource removal from the DHT}

In certain applications, it would be desirable to allow nodes to remove (DELETE) resources that have been earlier inserted into the distributed hash table. \emph{HyCube} provides such a possibility. A node willing to delete a resource should send a DELETE message addressed to that node (or to the resource key, in which case the message would be routed to the node closest to the resource key), specifying the exact criteria for deletion - key-value pairs (the key set should be a subset of the resource descriptor key set: \emph{resourceId}, \emph{resourceUrl}, \ldots). The resource entry should be deleted only when all key-value pairs specified within the criteria match those of the resource entry descriptor. For instance, a node might decide to delete a resource copy published by itself, while still leaving other copies of the resource accessible. In this case, the node would specify the \emph{resourceId} and \emph{resourceUrl}, which unambiguously identifies that particular instance of the resource. Every node processing a DELETE message is supposed to send a DELETE\_REPLY message directly to the requestor, containing the status of the operation (\emph{true} is the resource was deleted, \emph{false} otherwise).

When a resource is removed from one node's local storage, it may still be accessible through other nodes, and may be again inserted to the node that removed it through the replication mechanism. That is why the resource being removed should be deleted from all nodes storing the resource at the same time. It is however never guaranteed that the resource is entirely deleted from the DHT, as it is difficult to track to which nodes the resource was replicated. All resources are however assigned the expiration time (mentioned earlier), which causes resources to be deleted after certain amount of time if they are not refreshed. This mechanism is able to eventually remove outdated resources from the DHT completely.

\section{Storage access control}

By default, in \emph{HyCube}, it is possible to perform PUT, REFRESH\_PUT, GET and DELETE operations on any resource by any node. In many cases, this behavior might not be desirable. The design of \emph{HyCube} allows introducing an additional (application defined) component that would control which operations are allowed and which are not, based on the resource descriptor and the node performing the operation. For example, such a mechanism might be enabled to allow REFRESH\_PUT and DELETE operations to be performed only by the nodes that originally put the resource in the DHT. Such a check could be based for example on the \emph{resourceUrl} value. Another example could be limiting access for resources for certain nodes based on the resource metadata.

%\section{Exact versus routed resource operations}
\section{Routing requests vs sending requests directly to nodes}

As described in the previous sections, PUT, REFRESH\_PUT, GET and DELETE messages are routed towards the resource key, and the responses are returned directly to the requesting node. However, another approach may be taken if the request should be sent to a certain exact node (known to the requesting node). In such a case, this exact node's ID should be set as the recipient of the message and the message should be sent directly to the recipient. The message would then not be routed further, even if it is not the closest node to the resource key. Specifying the exact recipient for resource operation messages allows placing resources in multiple nodes, updating the refresh time (REFRESH) for resources stored in multiple nodes, and retrieving the resources from several nodes at a time. Usually, to increase resource availability, the resource is stored in several nodes - the closest nodes to the resource key. Moreover, as already mentioned, the DELETE operation would successfully remove a resource from the DHT only if the resource is removed from all the nodes storing it. To determine the set of closest nodes, the search procedure may be performed, and the messages may then be sent directly to the nodes found. Depending on the application, other techniques may be employed - for instance, if the resource should be put to only one node (for example allocation of subtasks in a distributed computing system), a node lookup might be used to find the node closest to a certain identifier.

In majority of applications, the resources are stored by multiple nodes, and the PUT operation should usually be preceded by the search locating the closest nodes, to which the resources are then inserted. However, retrieving the resources from the DHT, in most cases, may be done more efficiently, without the use of the search procedure. In the routing approach (towards the resource key), it is possible to limit the number of messages exchanged between nodes to the minimum, finding the resource with high probability in the closest node to the resource key. If additionally, the value of the option \emph{GetFromClosest} is set to \emph{false}, the check is made by all nodes on the route, increasing the probability of finding the resource requested. Furthermore, such a way of retrieving resources would provide simple, but effective load balancing - the first node containing the resource would be different depending on the path (direction from which the request comes).

\section{Resource replication and load balancing}
\label{sec:replication}

It was already mentioned that, to increase availability of resources, they may me replicated to multiple nodes. In \emph{HyCube}, resources are replicated among a certain number of nodes (let us denote this number by $k_{rep}$) that are the closest ones to the resource keys ($k_{rep}$ is a system parameter). However, in dynamic systems, the set of $k_{rep}$ closest nodes may change very quickly, making resources difficult to locate. At one extreme, all nodes responsible for some resource key may leave the system, making the resources unavailable. Therefore, it is crucial that the resources get constantly replicated among $k_{rep}$ currently closest nodes. $k_{rep}$ may be larger than $k_{store}$, but in this case, some nodes may not return the replica when receiving a GET message (with the \emph{GetFromClosest} option set to \emph{false}) - instead they would route the message further towards the resource key.

The replication procedure is run periodically by all nodes. Every node sends information about all resources it maintains to all the nodes in its neighborhood set (REPLICATE messages). To keep the message sizes at reasonable levels, the resources stored for different keys should be sent in separate REPLICATE messages. The number of nodes to which the replication information is sent may be limited to a certain maximum number of nodes by setting the value of the system parameter \emph{maxReplicationNSNodesNum}, in which case the REPLICATE messages will be sent only to \emph{maxReplicationNSNodesNum} closest nodes in the neighborhood set. However, to allow resources to be replicated towards all directions in the Euclidean space, the value of this parameter should not be too low. The replication information contains only the resource descriptors (metadata) and the last resource refresh time. Upon receiving a REPLICATE message, for every replicated resource, the receiving node should check if it is one of the $k_{rep}$ closest nodes to the resource key. If the node is one of the closest nodes, for each resource corresponding to that key, the node should check if it maintains this resource in its local storage. If the resource is already stored (\emph{resourceId} and \emph{resourceUrl} are compared), the refresh time of the resource should be updated (if the refresh time for the resource in the replication message is greater than the local refresh time). If the resource is, however, not stored in the local storage, the node should perform the GET operation (GET messages routed towards the resource key, the value of \emph{GetFromClosest} flag used in replication is set by a system parameter), and, upon receiving the GET\_REPLY, insert the resource to its local storage, setting the refresh time to the one received in the REPLICATE message. However, if the refresh time received is greater than the current local time, the refresh time is set to the current time.

The replication mechanism of \emph{HyCube}, in addition to increasing resource availability, also provides load balancing for most popular resources. \emph{HyCube} is designed to realize this approach by allowing an external (application-specific) module, implementing an appropriate interface, to be registered - a replication spread manager, which may analyze all PUT, REFRESH\_PUT, GET and DELETE operations and calculate the replication spread factor $\psi$ for each resource stored. Based on $\psi$, the node performing the replication calculates to how many nodes the resource should be replicated:

\begin{equation}
k_{rep} = \psi \cdot k_{rep}
\end{equation}

\noindent
where $k_{bal}$ is the default number of replication nodes. The value of $k_{bal}$ for every resource is included in the REPLICATE messages sent to nodes. The nodes receiving replication information, accept the replicas only when they determine that they are within $k_{bal}$ closest nodes to the resource key. For more popular resources, the replication spread manager may increase the value of $\psi$, which would cause the resource to be replicated to more nodes. When combined with retrieving (getting) resources with \emph{GetFromClosest} option set to \emph{false}, such a replication mechanism provides very good load balancing - in a multidimensional space, the increase of the replication radius would cause more nodes handling GET requests coming from different directions. As already mentioned, the value of $k_{rep}$ (and thus $k_{bal}$) may be greater than the system parameter $k_{store}$. However, in this case, some nodes, although accepting the replica, would not return the resource when receiving a GET message with the \emph{GetFromClosest} option set to \emph{false}. The implementation of the replication spread manager is application-dependent and is beyond the scope of \emph{HyCube}. The default implementation returns the value $\psi = 1$ for every resource.

One more remark should be made regarding replication and updating refresh times for resources. Whenever there is a time difference between any two nodes, the refresh time sent within a REPLICATE message may be inadequate when the message is received by another node. That could potentially cause the refresh time to be automatically increased just by the replication process, which is not a desired behavior. However, the new refresh time assigned to resources is always the smaller of the two values - the refresh time received and the current local time, meaning that the refresh time will never be increased by exchanging REPLICATE messages to a larger value than the current local time. The refresh time may also be increased as a result of receiving a REFRESH\_PUT message, when a larger refresh time value is specified by the refreshing node. Although there is also a possibility that the refreshing node's time is shifted (relatively to the node receiving the refresh request), the refresh time will never be set to a greater value than the receiving node's local time. Thus, the only potentially problematic situation may occur when both, the refreshing and the refreshed node have their local clocks going faster. In such a case, the refreshed node would potentially send a wrong refresh time in replicate messages (potentially far in future). However, as other nodes would then set the refresh time to their current local time value, and the replicating node (the one with the clock going faster) would stop replicating the resource after it is outdated according to its local time, the excess storing time at the whole DHT level would, at one extreme, be equal to the resource expiration time (the worst case), and the resource would then be considered outdated and removed by all other nodes.

The replication mechanism employed by \emph{HyCube} is able to maintain replica availability in the presence of many node failures, as well as under high rates of churn. Although in such situations, it may happen that certain resources are replicated to more nodes than needed, this anomaly would be only temporary - the excess replicas would be removed after their expiration time (the resources stored by the excess nodes would not be refreshed).

Although replicating resources is performed every certain time interval, the algorithm does not introduce much overhead, because only a small number of REPLICATE messages are exchanged, and nodes physically transfer the resources only when it is needed.

% ex: set tabstop=4 shiftwidth=4 softtabstop=4 noexpandtab fileformat=unix filetype=tex encoding=utf-8 fileencodings= fenc= spelllang=pl,en spell:


\section{UTxO}
\label{sec:utxo}

A key constraint that must always be satisfied as a result and precondition of
a valid ledger state transition is called the \textit{general accounting
property}, or the \textit{preservation of value} condition. Every piece of
software that is a part of the implementation of the
Cardano cryptocurrency must function in such a way as to not result in
a violation of this rule.
If this condition is not satisfied, it is an indicator of
incorrect accounting, potentially due to
malicious disruption or a bug.

The preservation of value is expressed as an equality that uses values in
the ledger state and the environment, as well as the values in the body of
the signal transaction.
We have defined the rules of the delegation protocol in a way that should
consistently satisfy the preservation of value. In the future, we hope to
give a formally-verified proof that every \textit{valid} ledger state satisfies
this property.

In this section, we discuss the relevant accounting that needs to be done
as a result of processing a transaction, i.e.~the deposits for all certificates,
transaction fees, transaction withdrawals and refunds for individual
deregistration, so that we may keep track of whether the preservation of
value is satisfied. Stake pool retirement refunds are not triggered by a
transaction (but rather, happen at the epoch boundary) and are therefore
not considered in our state change rules invoked due to a signal transaction.

Note that when a transaction is issued by a wallet to be applied to the ledger
state (i.e.~processed), the rules in this section are defined in such a way that it is impossible to
apply only some parts of a transaction (e.g.~only certain certificates).
Every part of the transaction must be valid and it must be live, otherwise
it is ignored entirely. It is the wallet's responsibility to inform the user
that a transaction failed to be processed.

\subsection{UTxO Transitions}
\label{sec:utxo-trans}

Figure~\ref{fig:functions:utxo} defines functions needed for the UTxO transition system.
See Figure~\ref{fig:defs:utxo-shelley} for most of the definitions used in the transition system.

\begin{itemize}

  \item
    The function $\fun{outs}$ creates unspent outputs generated by a transaction, so that
    they can be added to the ledger state.  For each output in the transaction,
    $\fun{outs}$ maps the transaction id and output index to the output.

  \item
    The $\fun{ubalance}$ function calculates sum total of all the coin in a given UTxO.
  \item
    The $\fun{wbalance}$ function calculates the total sum of all the reward withdrawals in a
    transaction.

  \item The calculation $\fun{consumed}$ gives the value consumed by the
    transaction $\var{tx}$ in the context of the protocol parameters, the
    current UTxO on the ledger and the registered stake credentials. This
    calculation is a sum of all coin in the inputs of $\var{tx}$, reward
    withdrawals and stake credential deposit refunds. Some of the definitions
    used in this function will be defined in Section~\ref{sec:deps-refunds}. In particular,
    $\fun{keyRefunds}$ is defined in Figure~\ref{fig:functions:deposits-refunds}.

  \item The calculation $\fun{produced}$ gives the value produced by the transaction $\var{tx}$
    in the context of the protocol parameters and the registered stake pools.
    This calculation is a sum of all coin in the outputs of $\var{tx}$,
    the transaction fee and all needed deposits.
    Some of the definitions used in this function will be defined in
    Section~\ref{sec:deps-refunds}.
    In particular, $\fun{totalDeposits}$ is defined in Figure~\ref{fig:functions:deposits-refunds}.
\end{itemize}

For a transaction and a given ledger state, the preservation of value property holds
exactly when the results of $\fun{consumed}$ equal the results of $\fun{produced}$.
Moreover, when the property holds, value is only moved between transaction outputs,
the reward accounts, the fee pot and the deposit pot.

Note that the $\fun{produced}$ function takes the registered stake pools ($\var{poolParams}$)
as a parameter only in order to determine which pool registration certificates are
new (and thus require a deposit) and which ones are updates.
Registration will be discussed more in Section~\ref{sec:delegation-shelley}.

\begin{figure}[htb]
  \begin{align*}
    & \fun{outs} \in \TxBody \to \UTxO
    & \text{tx outputs as UTxO} \\
    & \fun{outs} ~ \var{tx} =
        \left\{
          (\fun{txid} ~ \var{tx}, \var{ix}) \mapsto \var{txout} ~
          \middle|
          \var{ix} \mapsto \var{txout} \in \txouts{tx}
        \right\}
    \nextdef
    & \fun{ubalance} \in \UTxO \to \Coin
    & \text{UTxO balance} \\
    & \fun{ubalance} ~ utxo = \sum_{(~\wcard ~ \mapsto (\wcard, ~c)) \in \var{utxo}} c
    \nextdef
    & \fun{wbalance} \in \Wdrl \to \Coin
    & \text{withdrawal balance} \\
    & \fun{wbalance} ~ ws = \sum_{(\wcard\mapsto c)\in\var{ws}} c
    \nextdef
    & \fun{consumed} \in \PParams \to \UTxO \to \TxBody \to \Coin
    & \text{value consumed} \\
    & \consumed{pp}{utxo}{tx} = \\
    & ~~\ubalance{(\txins{tx} \restrictdom \var{utxo})} +
        \fun{wbalance}~(\fun{txwdrls}~{tx}) \\
    & ~~ + \keyRefunds{pp}{tx} \\
    \nextdef
    & \fun{produced} \in \PParams \to (\KeyHash_{pool}\mapsto\PoolParam) \to \TxBody \to \Coin
    & \text{value produced} \\
    & \fun{produced}~\var{pp}~\var{poolParams}~\var{tx} = \\
    &~~\ubalance{(\outs{tx})}
    + \txfee{tx} + \totalDeposits{pp}{poolParams}{(\txcerts{tx})}\\
  \end{align*}

  \caption{Functions used in UTxO rules}
  \label{fig:functions:utxo}
\end{figure}

\clearpage

The types for the UTxO transition are given in Figure~\ref{fig:ts-types:utxo-shelley}.
The environment, $\UTxOEnv$, consists of:

\begin{itemize}
  \item The current slot.
  \item The protocol parameters.
  \item The registered stake pools
    (also explained in Section~\ref{sec:delegation-shelley},
    Figure~\ref{fig:delegation-defs}).
  \item The genesis key delegation mapping.
\end{itemize}
The current slot and registrations are needed for the refund calculations
described in Section~\ref{sec:deps-refunds}.

The state needed for the UTxO transition $\UTxOState$, consists of:

\begin{itemize}
  \item The current UTxO.
  \item The deposit pot.
  \item The fee pot.
  \item Proposed updates (see Section~\ref{sec:update}).
\end{itemize}
The signal for the UTxO transition is a transaction.

\begin{figure}[htb]
  \emph{UTxO environment}
  \begin{equation*}
    \UTxOEnv =
    \left(
      \begin{array}{r@{~\in~}lr}
        \var{slot} & \Slot & \text{current slot}\\
        \var{pp} & \PParams & \text{protocol parameters}\\
        \var{poolParams} & \KeyHash_{pool}\mapsto\PoolParam & \text{stake pools}\\
        \var{genDelegs} & \GenesisDelegation & \text{genesis key delegations} \\
      \end{array}
    \right)
  \end{equation*}
  %
  \emph{UTxO States}
  \begin{equation*}
    \UTxOState =
    \left(
      \begin{array}{r@{~\in~}lr}
        \var{utxo} & \UTxO & \text{UTxO}\\
        \var{deposits} & \Coin & \text{deposits pot}\\
        \var{fees} & \Coin & \text{fee pot}\\
        \var{pup} & \PPUpdateState & \text{proposed updates}\\
      \end{array}
    \right)
  \end{equation*}
  %
  \emph{UTxO transitions}
  \begin{equation*}
    \_ \vdash
    \var{\_} \trans{utxo}{\_} \var{\_}
    \subseteq \powerset (\UTxOEnv \times \UTxOState \times \Tx \times \UTxOState)
  \end{equation*}
  %
  \caption{UTxO transition-system types}
  \label{fig:ts-types:utxo-shelley}
\end{figure}

The UTxO transition system is given in Figure~\ref{fig:rules:utxo-shelley}.
Rule~\ref{eq:utxo-inductive-shelley} specifies the conditions under which a transaction can
be applied to a particular $\UTxOState$ in environment $\UTxOEnv$:

The transition contains the following predicates:

\begin{itemize}
  \item
    The transaction is live (the current slot is less than its time to live).
  \item
    The transaction has at least one input.
    The global uniqueness of transaction inputs prevents replay attacks.
    By requiring that all transactions spend at least one input,
    the entire transaction is safe from such attacks.
    A delegation certificate by itself, for example, does not have this property.
  \item
    The fee paid by the transaction has to be greater than or equal to the minimum fee,
    which is based on the size of the transaction.
    A user or wallet might choose to create a fee larger than necessary
    in exchange for a faster processing time.
  \item
    Each input spent in the transaction must be in the set of unspent
    outputs.
  \item
    The \textit{preservation of value} property must hold.
    In other words, the amount of value produced by the transaction must be the same as
    the amount consumed.
  \item
    The $\mathsf{PPUP}$ transition is successful.
  \item
    The coin value of each new output must be at least as large as the
    minimum value specified by the protocol parameter $\var{minUTxOValue}$.
  \item
    The transaction size must be below the allowed maximum.
    Note that there is an implicit max transaction size given by the max block size,
    and that if we wished to allow a transaction to be as large as will fit in a block, this
    check would not be needed.
    Being able to limit the size below that of the block, however, gives us some
    control over how transactions will be packed into the blocks.
\end{itemize}
If all the predicates are satisfied, the state is updated as follows:

\begin{itemize}
  \item Update the UTxO:
    \begin{itemize}
      \item Remove from the UTxO all the $(\var{txin}, \var{txout})$ pairs
        associated with the $\var{txins}$'s in the $\var{inputs}$ list of
        the transaction body $\var{txb}$.
      \item Add all the $\var{outputs}$ of $\var{tx}$ to the
        UTxO, associated with the $\fun{txid}$ of the transaction body $\var{txb}$
    \end{itemize}
  \item Add all new deposits to the deposit pot and subtract all deposit refunds.
  \item Add the transaction fee to the fee pot.
  \item Update the current update proposals.
\end{itemize}

The accounting for the reward withdrawals is not done in this transition system.
The rewards are tracked with the delegation state and will
be removed in the final delegation transition, see ~\ref{eq:delegs-base}.

Note here that output entries for both the deposit refunds and the rewards
withdrawals must be included in the body of the transaction
carrying the deregistration certificates (requesting these refunds) and the
reward requests. It is the job
of the wallet to calculate the value of these refunds and withdrawals and
generate the correct outputs to include in the outputs list of $\var{tx}$ such
that applying this transaction results in a
valid ledger update adding correct amounts of coin to the right addresses.

The approach of including refunds and rewards directly in the $outputs$ gives
great flexibility to the management of the coin value obtained from these
accounts, i.e.~it can be directed to any address. However, it means there is no
direct link between the $wdrls$ requests (similarly for the key deregistration
certificate addresses and refund amounts) and the $outputs$. We verify that
the included outputs are correct and authorized through the preservation of value condition
and witnessing the transaction. The combination of the
preservation of value and witnessing, described in Section~\ref{sec:witnesses-shelley},
assures that the ledger state is updated correctly.

The main difference, however, in how rewards and refunds work is that refunds
come from a $\var{deposits}$ pool, which is a single coin value indicating
the total decayed amount of all the deposits ever made, while rewards come from individual
accounts where a reward is accumulated to a specific address.

\begin{figure}[htb]
  \begin{equation}\label{eq:utxo-inductive-shelley}
    \inference[UTxO-inductive]
    { \var{txb}\leteq\txbody{tx}
      & \txttl txb \geq \var{slot}
      \\ \txins{txb} \neq \emptyset
      & \minfee{pp}{tx} \leq \txfee{txb}
      & \txins{txb} \subseteq \dom \var{utxo}
      \\
      \consumed{pp}{utxo}~{txb} = \produced{pp}{poolParams}~{txb}
      \\
      ~
      \\
      {
        \begin{array}{r}
          \var{slot} \\
          \var{pp} \\
          \var{genDelegs} \\
        \end{array}
      }
      \vdash \var{pup} \trans{\hyperref[fig:rules:pp-update]{ppup}}{\fun{txup}~\var{tx}} \var{pup'}
      \\
      ~
      \\
      \forall (\wcard\mapsto (\wcard,~c)) \in \txouts{txb}, c \geq (\fun{minUTxOValue}~\var{pp})
      \\
      \forall (\wcard\mapsto (a,~\wcard)) \in \txouts{txb}, a \in \AddrBS \to \fun{bootstrapAttrsSize}~a \leq 64
      \\
      \forall (\wcard\mapsto (a,~\wcard)) \in \txouts{txb}, \fun{netId}~a =\NetworkId
      \\
      \forall (a\mapsto\wcard) \in \txwdrls{txb}, \fun{netId}~a =\NetworkId
      \\
      \fun{txsize}~{tx}\leq\fun{maxTxSize}~\var{pp}
      \\
      ~
      \\
      \var{refunded} \leteq \keyRefunds{pp}{txb}
      \\
      \var{depositChange} \leteq
        \totalDeposits{pp}{poolParams}{(\txcerts{txb})} - \var{refunded}
    }
    {
      \begin{array}{r}
        \var{slot}\\
        \var{pp}\\
        \var{poolParams}\\
        \var{genDelegs}\\
      \end{array}
      \vdash
      \left(
      \begin{array}{r}
        \var{utxo} \\
        \var{deposits} \\
        \var{fees} \\
        \var{pup}\\
      \end{array}
      \right)
      \trans{utxo}{tx}
      \left(
      \begin{array}{r}
        \varUpdate{(\txins{txb} \subtractdom \var{utxo}) \cup \outs{txb}}  \\
        \varUpdate{\var{deposits} + \var{depositChange}} \\
        \varUpdate{\var{fees} + \txfee{txb}} \\
        \varUpdate{pup'}\\
      \end{array}
      \right)
    }
  \end{equation}
  \caption{UTxO inference rules}
  \label{fig:rules:utxo-shelley}
\end{figure}

The UTXO rule has ten predicate failures:
\begin{itemize}
\item In the case of any input not being valid, there is a \emph{BadInput}
  failure.
\item In the case of the current slot being greater than the time-to-live of the
  current transaction, there is an \emph{Expired} failure.
\item In the case that the transaction is too large,
  there is a \emph{MaxTxSize} failure.
\item In the case of an empty input set, there is a \emph{InputSetEmpty} failure,
  in order to prevent replay attacks.
\item If the fees are smaller than the minimal transaction fees, there is a
  \emph{FeeTooSmall} failure.
\item If the transaction does not correctly conserve the balance, there is a
  \emph{ValueNotConserved} failure.
\item If the transaction creates any outputs with the wrong network ID,
  there is a \emph{WrongNetwork} failure.
\item If the transaction contains any withdrawals with the wrong network ID,
  there is a \emph{WrongNetworkWithdrawal} failure.
\item If the transaction creates an output below the allowed minimum value,
  there is a \emph{OutputTooSmall} failure.
\item If the transaction creates any boostrap outputs whose attributes have
  size bigger than 64, there is a \emph{OutputBootAddrAttrsTooBig} failure.
\end{itemize}

\clearpage

\subsection{Deposits and Refunds}
\label{sec:deps-refunds}

Deposits are described in appendix B.2 of the delegation design document
\cite{delegation_design}.  These deposit functions were used above in the UTxO
transition,~\ref{sec:utxo-trans}. Deposits are used for stake credential
registration and pool registration certificates, which will be explained in
Section~\ref{sec:delegation-shelley}.  In particular, the function
$\cwitness{}$, which gets the certificate witness from a certificate, will be
defined later.  Figure~\ref{fig:functions:deposits-refunds} defines the deposit
and refund functions.
\begin{itemize}
  \item The function $\fun{totalDeposits}$ returns the total deposits that have to be
    made by a transaction.  This calculation uses two protocol parameters, namely
    the key deposit value and the pool deposit value.
    Note that those certificates which are
    updates of stake pool parameters of already registered pool keys should not
    (and are, in fact, not allowed to) make a deposit.
  \item The function $\fun{keyRefunds}$, calculates the total amount of returned
    deposits from stake key deregistration certificates.

    Note that $\fun{keyRefunds}$ uses the \textit{current} protocol parameters.
    This means that any deposits made prior to a change in the deposit values
    will be refunded with the current value, not the one originally paid.

    The protocol parameters are not
    expected to change often and using the current ones for the calculation
    is a deliberate simplification choice, which does not introduce any inconsistencies
    into the system rules or properties. In particular, the general accounting
    property is not violated.
\end{itemize}

\begin{figure}[htb]
  \begin{align*}
    & \fun{totalDeposits} \in \PParams \to (\KeyHash_{pool}\mapsto\PoolParam) \\
    & ~~~~\to \seqof{\DCert} \to \Coin
    & \text{total deposits for transaction} \\
    & \totalDeposits{pp}{poolParams}{certs} = \\
    &  ~~~ (\fun{keyDeposit}~pp)\cdot|\var{certs}\cap\DCertRegKey| \\
    &  ~~~~~ + (\fun{poolDeposit}~pp)\cdot|\var{newPools}| \\
    & ~~~\where \\
    &  ~~~~~~~ \var{newPools} = \{\var{certs}\cap\DCertRegPool ~\mid~ \cwitness{c}\notin \var{poolParams}\}
    \nextdef
      & \fun{keyRefunds} \in \PParams \to \TxBody \to \Coin
      & \text{key refunds for a transaction} \\
      & \keyRefunds{pp}{tx} = (\fun{keyDeposit}~\var{pp})\cdot|\txcerts{tx} \cap \DCertDeRegKey|\\
      \end{align*}
      \caption{Functions used in Deposits - Refunds}
      \label{fig:functions:deposits-refunds}
  \end{figure}

\clearpage

\subsection{Witnesses}
\label{sec:witnesses-shelley}

The purpose of witnessing is make sure that the intended action is authorized by
the holder of the signing key, providing replay protection as a consequence.
Replay prevention is an inherent property of UTxO type accounting
since transaction IDs are unique and we require all transaction to
consume at least one input.

A transaction is witnessed by a signature and a verification key corresponding
to this signature.  The witnesses, together with the transaction body, form a
full transaction.  Every witness in a transaction signs the transaction body.
Moreover, the witnesses are represented as finite maps from verification keys to
signatures, so that any key that is required to sign a transaction only provides
a single witness.  This means that, for example, a transaction which includes a
delegation certificate and a reward withdrawal corresponding to the same stake
credential still only includes one signature.

Figure~\ref{fig:functions-witnesses} defines the function which
gathers all the (hashes of) verification keys needed to witness a given transaction.
This consists of:
\begin{itemize}
  \item payment keys for outputs being spent
  \item stake credentials for reward withdrawals
  \item stake credentials for delegation certificates (except \DCertMir{} and \DCertRegKey{})
  \item delegates of the genesis keys for any protocol parameter updates
  \item stake credentials for the pool owners in a pool registration certificate
\end{itemize}

\begin{figure}[htb]
  \begin{align*}
    & \fun{propWits} \in \Update \to \GenesisDelegation \to \powerset{\KeyHash}
    & \text{hashkeys for proposals} \\
    & \fun{propWits}~(\var{pup},~\wcard)~\var{genDelegs} = \\
    & ~~\left\{
      \var{kh}
      \mid
      \var{gkh}\mapsto(\var{kh},~\wcard)\in
      \left(\dom{\var{pup}}\restrictdom\var{genDelegs}\right)
      \right\}
  \end{align*}

  \begin{align*}
    & \hspace{-0.8cm}\fun{certWitsNeeded} \Tx \to \powerset{\Credential}
    & \text{certificates with witnesses} \\
    &  \hspace{-0.8cm}\fun{certWitsNeeded}~\var{tx} = \\
    & \bigcup\{\cwitness{c} \mid c \in \txcerts{tx} \setminus (\DCertRegKey\cup\DCertMir)\}
  \end{align*}

  \begin{align*}
    & \hspace{-0.8cm}\fun{witsVKeyNeeded} \in \UTxO \to \Tx \to (\KeyHashGen\mapsto\VKey) \to
      \powerset{\KeyHash}
    & \text{required key hashes} \\
    &  \hspace{-0.8cm}\fun{witsVKeyNeeded}~\var{utxo}~\var{tx}~\var{genDelegs} = \\
    & ~~\{ \fun{paymentHK}~a \mid i \mapsto (a, \wcard) \in \var{utxo},~i\in\fun{txinsVKey}~{tx} \} \\
    \cup & ~~
           \{\fun{stakeCred_r}~a\mid a\mapsto \wcard \in \AddrRWDVKey
      \restrictdom \txwdrls{tx}\}\\
    \cup & ~~(\AddrVKey \cap \fun{certWitsNeeded}~{tx}) \\
    \cup & ~~\fun{propWits}~(\fun{txup}~\var{tx})~\var{genDelegs} \\
    \cup & ~~\bigcup_{\substack{c \in \txcerts{tx} \\ ~c \in\DCertRegPool}} \fun{poolOwners}~{c}
  \end{align*}
  \begin{align*}
    & \hspace{-1cm}\fun{scriptsNeeded} \in \UTxO \to \Tx \to
      \powerset{\HashScr}
    & \text{required script hashes} \\
    &  \hspace{-1cm}\fun{scriptsNeeded}~\var{utxo}~\var{tx} = \\
    & ~~\{ \fun{validatorHash}~a \mid i \mapsto (a, \wcard) \in \var{utxo},\\
    & ~~~~~i\in\fun{txinsScript}~{(\fun{txins~\var{tx}})}~{utxo}\} \\
    \cup & ~~\{ \fun{stakeCred_{r}}~\var{a} \mid a \in \dom (\AddrRWDScr
           \restrictdom \fun{txwdrls}~\var{tx}) \} \\
      \cup & ~~(\AddrScr \cap \fun{certWitsNeeded}~{tx})
  \end{align*}
  \caption{Functions used in witness rule}
  \label{fig:functions-witnesses}
\end{figure}

The UTxOW transition system adds witnessing to the previous UTxO transition system.
Figure~\ref{fig:ts-types:utxow-shelley} defines the type for this transition.

\begin{figure}
  \emph{UTxO with witness transitions}
  \begin{equation*}
    \var{\_} \vdash
    \var{\_} \trans{utxow}{\_} \var{\_}
    \subseteq \powerset (\UTxOEnv \times \UTxOState \times \Tx \times \UTxOState)
  \end{equation*}
  \caption{UTxO with witness transition-system types}
  \label{fig:ts-types:utxow-shelley}
\end{figure}

Figure~\ref{fig:rules:utxow-shelley} defines UTxOW transition.
It has six predicates:
\begin{itemize}
  \item Every signature in the transaction is a valid signature of the transaction body.
  \item The set of (hashes of) verification keys given by the transaction is a subset of
    the set of needed (hashes of) verification keys.
  \item Every multisignature script is valid.
  \item The set of scripts given by the transaction is equal to the set of required scripts.
  \item Any instantaneous reward certificates have quorum agreement from the genesis nodes delegates.
  \item Either the transaction metadata hash and the transaction metadata are both absent,
    or the hash present in the body is actually equal to the hash of the metadata.
\end{itemize}
If the predicates are satisfied, the state is transitioned by the UTxO transition rule.

\begin{figure}
  \begin{equation}
    \label{eq:utxo-witness-inductive-shelley}
    \inference[UTxO-wit]
    {
      (utxo, \wcard, \wcard, \wcard) \leteq \var{utxoSt} \\
      \var{witsKeyHashes} \leteq \{\fun{hashKey}~\var{vk} \vert \var{vk} \in
      \dom (\txwitsVKey{tx}) \}\\~\\
      \forall \var{hs} \mapsto \var{validator} \in \fun{txwitsScript}~{tx},\\
      \fun{hashScript}~\var{validator} = \var{hs} \wedge
      \fun{validateScript}~\var{validator}~\var{tx}\\~\\
      \fun{scriptsNeeded}~\var{utxo}~\var{tx} = \dom (\fun{txwitsScript}~{tx})
      \\~\\
      \var{txbodyHash}\leteq\fun{hash}~(\txbody{tx}) \\
      \forall \var{vk} \mapsto \sigma \in \txwitsVKey{tx},
      \mathcal{V}_{\var{vk}}{\serialised{txbodyHash}}_{\sigma} \\
      \fun{witsVKeyNeeded}~{utxo}~{tx}~{genDelegs} \subseteq witsKeyHashes
      \\~\\
      genSig \leteq
      \left\{
        \var{genDelegate}~\vert~(\var{genDelegate},~\wcard) \in\range{genDelegs}
      \right\}
      \cap
      \var{witsKeyHashes}
      \\
      \left\{
        c\in\txcerts{tx}~\cap\DCertMir
      \right\} \neq\emptyset \implies \vert genSig\vert \geq \Quorum
      \\~\\
      \var{mdh}\leteq\fun{txMDhash}~\var{tx}
      &
      \var{md}\leteq\fun{txMD}~\var{tx}
      \\
      (\var{mdh}=\Nothing \land \var{md}=\Nothing)
      \lor
      (\var{mdh}=\fun{hashMD}~\var{md})
      \\~\\
      {
        \begin{array}{r}
          \var{slot}\\
          \var{pp}\\
          \var{poolParams}\\
          \var{genDelegs}\\
        \end{array}
      }
      \vdash \var{utxoSt} \trans{\hyperref[fig:rules:utxo-shelley]{utxo}}{tx}
      \var{utxoSt'}\\
    }
    {
      \begin{array}{r}
        \var{slot}\\
        \var{pp}\\
        \var{poolParams}\\
        \var{genDelegs}\\
      \end{array}
      \vdash \var{utxoSt} \trans{utxow}{tx} \varUpdate{\var{utxoSt'}}
    }
  \end{equation}
  \caption{UTxO with witnesses inference rules}
  \label{fig:rules:utxow-shelley}
\end{figure}

The UTXOW rule has eight predicate failures:
\begin{itemize}
\item In the case of an incorrect witness,
  there is a \emph{InvalidWitnesses} failure.
\item In the case of a missing witness,
  there is a \emph{MissingVKeyWitnesses} failure.
\item In the case of missing scripts,
  there is a \emph{MissingScriptWitnesses} failure.
\item In the case of an invalid script,
  there is a \emph{ScriptWitnessNotValidating} failure.
\item In the case of a lack of quorum on an instantaneous reward certificate,
  there is a \emph{MIRInsufficientGenesisSigs} failure.
\item In the case the transaction contains metadata,
  but the transaction body does not contain a metadata hash,
  there is a \emph{MissingTxBodyMetaDataHash} failure.
\item In the case that the transaction body contains a metadata hash,
  but there is no metadata outside the body,
  there is a \emph{MissingTxMetaData} failure.
\item In the case that the metadata hash in the transaction body
  is not equal to the hash of the metadata outside the body,
  there is a \emph{ConflictingMetaDataHash} failure.
\end{itemize}

\clearpage


\documentclass[nofootinbib,amssymb,amsmath]{revtex4}
\usepackage{mathtools}
\usepackage{amsthm}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{lmodern}
\usepackage{graphicx}
\usepackage{color}

%Put an averaged random variable between brackets
\newcommand{\ave}[1]{\left\langle #1 \right\rangle}
\newcommand{\HC}{\texttt{HaplotypeCaller}}
\newcommand{\Mutect}{\texttt{Mutect}}
\newcommand{\code}[1]{\texttt{#1}}
\newcommand{\mc}[1]{\mathcal{#1}}

\newtheorem{lemma}{Lemma}
\newtheorem{corollary}{Corollary}

\def\SL#1{{\color [rgb]{0,0,0.8} [SL: #1]}}
\def\DB#1{{\color [rgb]{0,0.8,0} [DB: #1]}}

\begin{document}

\title{Inferring variants from assembled haplotypes in HaplotypeCaller and Mutect}
\author{David Benjamin\footnote{The author took no part in development of the methods described below -- credit belongs to several others on the GATK team. }}
\email{davidben@broadinstitute.org}
\affiliation{Broad Institute, 75 Ames Street, Cambridge, MA 02142}

\date{\today}

\begin{abstract}
Despite its name, \HC does not actually call haplotypes.  Rather, it generates haplotypes as an intermediate step to discover variants at individual loci.  Here we describe how the GATK engine determines which alt alleles exist in locally assembled haplotypes.
\end{abstract}

\maketitle

\section{Forward pass}
The first step is to align each assembled haplotype to the reference haplotype using the Smith-Waterman algorithm.  Although the GATK's implementation is not complicated, it is also not a completely direct translation of the method into code.  As it incurs a non-trivial computational cost, we describe it in detail here.

Our implementation has four score parameters, $w_{\rm match}$ and $w_{\rm mismatch}$ for equal and unequal reference and alternate bases, $w_{\rm open}$ for opening a gap (that is, starting an indel), and $w_{\rm extend}$ for extending a gap.  Note the absence of a scoring matrix treating each possible type of match and mismatch differently.  Although the idea of a score seems like a heuristic, the algorithm is equivalent to finding the maximum likelihood path in a hidden Markov model in which the scores are log transition and emission probabilities.

As in our pair-HMM probabilistic alignment, we fill a matrix $M$, the rows and columns of which correspond to bases of the reference and alternate haplotypes, from top to bottom and left to right.  $M_{ij}$ is the best score of alignments ending at the $i$th reference base and $j$th alternate base that do not end in a gap\footnote{That is, it includes alignments that have gaps somewhere earlier, just not at this position.}.  We also keep track of two arrays pertaining to the last row of our traversal.  $D_j$ is the best score of alignments ending at the previous reference base (ie the $(i-1)$th base when we are at the $i$th base in traversal) and the $j$th alternate base that end in a ``downward'' gap, i.e. a deletion with respect to the reference.  $S^d_j$ is the size of the gap in this best-scoring alignment.  We also fill a backtrack matrix $B$, where $B_{ij}$ is an instruction (see below) for reconstructing the best path after we fill $M$.

First we initialize the zeroth row and column as $M_{0,0}$ = 0, $M_{0,1} = M_{1,0} = w_{\rm open}$, $M_{0,2} = M_{2,0} = w_{\rm open} + w_{\rm extend}$ etc.  The zeroth row and column correspond to one base before the reference and alternate starts and this initialization penalizes leading indels.  That is, this is a global alignment\footnote{The GATK assembly graph merges all alternate paths into the reference, hence alternate haplotypes start and end coincident with the reference and global alignment is appropriate.}.

Then, for each row $1 \le i \le {\rm length(reference)}$ we loop over all columns $1 \le j \le {\rm length(alternate)}$ and do the following:

\begin{itemize}
\item Update deletion scores:  The score for opening a downward gap is $M_{i-1,j} + w_{\rm open}$.  The score for extending an existing deletion is $D_j + w_{\rm extend}$, where $D_j$ at this point still pertains to the $previous$ row $i - 1$.  We set $D_j$ (modifying it in-place) to the greater of these values. If the gap-opening score is greater than the gap-extending score, we set $S^d_j = 1$, otherwise we increment $S^d_j$ by 1.
\item Update insertion scores:  When we begin traversing row $i$ we initialize the current best score for alignments ending in a ``rightward'' gap (i.e. an insertion) as $R = -\infty$ and we initialize the length of the terminal gap in this best-scoring alignment as $S^r = 0$.  Note that these values are local to the inner loop over $j$ and thus do not need to be arrays\footnote{In the code, they \textit{are} arrays, but in the $i$th iteration of the loop over rows, only their $i$th elements are used.  Thus the elements are effectively scalars whose scope is the loop over $j$.}.  At each stage in the loop over $j$ the score for opening a rightward gap is $M_{i,j - 1} + w_{\rm open}$ and score for extending one is $R + w_{\rm extend}$, where $R$ still pertains to the previous column $j - 1$.  We set $R$ (modifying it in-place)  to the greater of these values.  If the gap-opening score is greater than the gap-extending score, we set $S^r_j = 1$, otherwise we increment $S^r_j$ by 1.
\item Record backtrack: The score for no gap, i.e. a match \textit{alignment} (as opposed to matching bases), is $M_{i-1,j-1}$ plus $w_{\rm match}$ if the $i$th reference base and $j$th alternate base agree or $w_{\rm mismatch}$ if they do not.  We now compare this score to the indel scores $D_j$ and $R$.  We set $B_{ij}$ to 0 if the match score is greatest, $S^d_j$ if the downward gap score is greatest, and $-S^r$ if the rightward gap score is greatest.  This convention essentially uses the sign as an enum in order to encode whether the optimal path has an insertion, deletion or match.
\end{itemize}

\section{Backward pass}
After the forward pass the backtrack matrix $B$ is full.  We begin backtracking from the $(i,j)$ that is the maximum among the bottom row and rightmost column of $M$.  If this maximum is in the rightmost column that means all alternate haplotype bases are used and nothing special must be done.  If, however the maximum is on the bottom row it means that the more alternate bases remain.  In this case, we record a soft clip (S) CIGAR element with length equal to the number of remaining bases at the end of the alignment\footnote{The code behaves differently for different values of the \code{OverhangStrategy} enum, but this strategy is hard-coded to \code{SOFTCLIP}, which results in the behavior we describe here.}.  Then, from this $(i,j)$ we iterate the following procedure until reaching $i = 0$ or $j = 0$ (recall that these correspond to immediately \textit{before} the start of the corresponding haplotypes): If $B_{ij} = 0$ add a match (M) CIGAR element to the left end of the alignment and move to $i,j \rightarrow i - 1, j - 1$.  If $B_{ij} = k > 0$ add a length-$k$ deletion (D) CIGAR element to the left of the alignment and move to $i,j \rightarrow i - k, j$.  If $B_{ij} = -k < 0$ add a length-$k$ insertion (I) CIGAR element to the left of the alignment and move to $i,j \rightarrow i, j - k$.

Similar to the initial step, if we end at $j = 0$ nothing more needs to be done because all alternate bases are accounted for.  Otherwise, add a length-$j$ leading soft clip.

\section{Making variants}
For each haplotype from assembly it is easy to create variant alleles from the alignments found above.  Starting from the beginning of the haplotype and its starting reference position, traverse every element in the CIGAR string, advance $k$ bases in the reference for every length-$k$ element.  When we encounter a deletion element or insertion element we record a corresponding allele from the reference and alternate bases.  When we encounter a match element we compare the matched reference and alternate sub-haplotypes base-by-base and record a SNV allele whenever they disagree.

Taking all the unique start positions and variant alleles from all haplotypes give an initial set of variants to genotype but we are not quite done.  If multiple haplotypes have different variant alleles at the same position we may need to reconcile the representations.  For example, we may have a single deletion $AA \rightarrow A$ and a double deletion $AAA \rightarrow A$, which need to be merged as $AAA \rightarrow A, AA$.  Fortunately, this is essentially the canonical example and there are no edge cases to deal with.  That is, all we need to do is find the allele with the longest reference, and pad the other alleles with whatever of these extra reference bases they were missing.

\end{document}


\chapter{Neighbor list}
\label{sec:neighborlist}

A list of neighboring conjugated segments, or neighbor list\index{neighbor list}, contains all pairs of conjugated segments for which \slink{sec:transfer_integrals}{coupling elements}, \slink{sec:reorganization}{reorganization energies}, \slink{sec:site_energies}{site energy differences}, and \slink{sec:rates}{rates} are evaluated.

Two segments are added to this list if the distance between centers of mass of any of their rigid fragments is below a certain cutoff. This allows neighbors to be selected on a criterion of minimum distance of approach rather than center of mass distance, which is useful for molecules with anisotropic shapes.

The neighbor list can be generated from the atomistic trajectory by using the \calc{neighborlist} \calculator. This calculator requires a cutoff, which can be specified in the \xmloptions file. The list is saved to the \sqlstate file:
\votcacommand{Generating a neighbor list}{\cmdnbl}

Sometimes it is convenient to filter the rigid fragments, for example if a conjugated segment has alkyl side chains and we do not want to add a pair of molecules which have conjugated cores far apart but side chains close to each other. In this case one can provide a list of ``active'' fragments. The search algorithm will only look at the pairs of segments, for which these fragments are located closer than the specified cutoff.

It can also happen that a completely custom neighbor list is required. An external list can be imported from a file which has the format pairID, segment1ID, segment2ID, segment1Type, segment2Type, e.g.,
\lstset{language=XML}
\begin{lstlisting}
1  1 2 DCV DCV
2  1 3 DCV DCV
3  2 3 DCV DCV
\end{lstlisting}


\documentclass[12pt]{article}

\usepackage[margin=1.1in]{geometry}
\input{../../syllabi/preamble}

%%edit these!!
\newcommand{\coursedept}{Math}
\newcommand{\coursenumber}{241}
\newcommand{\sixhundredsection}{}
\newcommand{\coursenumbercrosslisted}{}
\newcommand{\semester}{Fall}
\newcommand{\numcredits}{3}
\newcommand{\lectimeandloc}{Tues and Thurs 1:40 -- 2:55PM / on zoom}
\newcommand{\requiredlabtimeandloc}{}
\newcommand{\tataofficehourtimeandloc}{} %TA / TA Office Hours / Loc & Kennly Weerasinghe / see \coursewebpagelink
\newcommand{\numtheoryhws}{6--9}
\newcommand{\lastdatetimetohandinhomeworks}{Dec 15 at noon}
\newcommand{\midtermonedateandlocation}{Thursday, October 7 on zoom during class time}
\newcommand{\midtermtwodateandlocation}{Thursday, November 11 on zoom during class time}
\newcommand{\finaldateandlocation}{TBD but on zoom}

%%%unchanged
\newcommand{\coursewebpageurl}{https://github.com/kapelner/QC_\coursedept_\coursenumber_\semester_\the\year}
\newcommand{\coursewebpagelink}{\href{\coursewebpageurl}{course homepage}}
\newcommand{\slackurl}{https://QC\coursedept\coursenumber\semester\the\year.slack.com/}
\newcommand{\slacklink}{\href{\slackurl}{slack}}

\input{../../syllabi/_header}

\section*{Course Overview}

MATH 241 is an introduction to the basic concepts and techniques of probability and statistics with an emphasis on applications. Topics to be covered are below (not in order of coverage):

\begin{itemize}
\itemsep -0.0em
\item Basic Set Theory
\item Counting Methods --- permutations and combinations
\item Basic Probability Theory --- axioms, conditional probability, in/dependence
\item Modeling with Discrete Random Variables: Bernoulli, Hypergeometric, Binomial, Poisson, Geometric, Negative Binomial, Uniform Discrete, Rademacher and others
\item Expectation, Variance, Covariance, Moments
\item Modeling with Continuous Random Variables: Exponential, Uniform and Normal
\item Moment Generating Functions
\item Law of Large Numbers and the Central Limit Theorem
\item Confidence Intervals and Hypothesis Testing for one-sample proportions (possibly $p$-values and statistical significance as well)
\end{itemize}

Students taking this course may not receive credit for MATH 114, except by permission of the chair. Pre/corequisites include MATH 132 or 143 or 152. \textbf{This is not your typical mathematics course.} This course develops ideas and concepts for helping to make decisions based on randomness and we will do lots of modeling of real-world situations. The course does not dwell on theory nor details of computation but will make use of computation especially using the \texttt{R} statistical language.

\section*{Course Materials}

\paragraph{Textbook:} A First Course in Probability by Sheldon Ross. I prefer the 7th edition which you can buy this used on \href{http://www.amazon.com/First-Course-Probability-7th-Edition/dp/0131856626}{Amazon} for cheap. You can buy \textit{any edition} though if you find it cheaper. There is no excuse not to have this book. It is \textit{required}. However, I will not be teaching \qu{from the book} and most of the material in the class comes from the lecture notes. The textbook is a way to get ``another take'' on the material.

\paragraph{Computer Software:} We will also be using \texttt{R} which is a free, open source statistical programming language and console. You can download it from: \url{http://cran.mirrors.hoobly.com/}. I do not expect you to do \textit{any} programming. I will be giving you \texttt{R} code to run and expect you to interpret the results based on concepts explained during the course.

\paragraph{Calculator:} You can use a TI-84, 85, 89 or any calculator which you wish. I strongly suggest you use \href{http://www.wolframalpha.com/}{Wolfram Alpha} and its smartphone app.

%\input{../../syllabi/_the650section}

\input{../../syllabi/_announcements_on_slack}

\input{../../syllabi/_use_of_slack}

\input{../../syllabi/_standard_class_meetings}

\input{../../syllabi/_jewish_holiday_reschedule}

\input{../../syllabi/_zoom_policies}

%\input{../../syllabi/_lecture_upload}

\section*{Homework}

\input{../../syllabi/_theory_hws_text}

\input{../../syllabi/_theory_hws_submission_text}
\input{../../syllabi/_philosophy_hws}

\input{../../syllabi/_time_spent_hws}

\input{../../syllabi/_late_hw_policy}

\input{../../syllabi/_latex_hw_bonus_policy}

\input{../../syllabi/_hw_ec_policy}

\section*{Examinations}

\input{../../syllabi/_examination_text}

\input{../../syllabi/_standard_exam_schedule}

\subsection*{Exam Policies and Materials}

\input{../../syllabi/_examination_policies}

\input{../../syllabi/_zoom_examination_policies}

\input{../../syllabi/_standard_cheat_sheet_policy}

\input{../../syllabi/_cheating_on_exams_and_missing_exams}
\input{../../syllabi/_special_services}

\input{../../syllabi/_class_participation}

%\input{../../syllabi/_zoom_attendance}

\input{../../syllabi/_standard_grading_and_grading_policy}

\input{../../syllabi/_241_grade_distribution}
%\input{../../syllabi/_advanced_course_grade_distribution}

\input{../../syllabi/_241_cs_c_minus_notice}

\input{../../syllabi/_grade_checking_on_gradesly}

\input{../../syllabi/_auditing_policy}

\pagebreak
\input{../../syllabi/_241_quiz}

\end{document}


\SecDef{intro}{Introduction}

With the advent of the Internet of Things (IoT), a myriad of devices are being connected one to another in order to exchange information. This information has to be secured. Symmetric cryptography can ensure that the data those devices share remains confidential, that it is properly authenticated and that it has not been tampered with.

As such objects have little computing power---and even less so that is dedicated to information security---the cost of the algorithms ensuring these properties has to be as low as possible. To answer this need, the NIST has called for the design of authenticated ciphers and hash functions providing a sufficient security level at as small an implementation cost as possible.

We present a suite of algorithms that answer this call. All our algorithms are built using the same core, namely the \aCipher{} family of permutations. The authenticated ciphers, \aead{}, provide confidentiality of the plaintext as well as both integrity and authentication for the plaintext and for additional public associated data. The hash functions, \hash{}, are (second) preimage and collision resistant. Our aim for our algorithms is to use as little CPU cycles as possible to perform their task while retaining strong security guarantees and a small implementation size. This speed will allow devices to use much fewer CPU cycles than what is currently needed to ensure the protection of their data. To give one of many very concrete applications of this gain, the energy demanded by cryptography for a battery-powered micro-controller will be decreased.

The parameters of instances of \hash{} and \aead{} are summarized in \TabRef{instances}.

\begin{table}
    \resizebox{\linewidth}{!}{
    \begin{tabular}{cccccc}
       \toprule
        \multirow{2}{*}{Type} & \multirow{2}{*}{Name} & Internal state size & Data block size & Security level  \\
        & & (bytes) & (bytes) & (bits) \\
        \midrule
        \multirow{2}{*}{Hash function}
        & \hash{256} & 48 & 16 & 128 \\
        & \hash{384} & 64 & 16 & 192 \\
        \midrule
        % \multirow{4}{*}{AEAD}
        & \aeadversion{128}{128} & 32 & 16 & 120 \\
        Authenticated & \aeadversion{256}{128} & 48 & 32 & 120 \\
        encryption & \aeadversion{192}{192} & 48 & 24 & 184 \\
        & \aeadversion{256}{256} & 64 & 32 & 248 \\
        \bottomrule
    \end{tabular}
    \TabDef{instances}{Algorithms in the lightweight cryptographic suite.}
    }
\end{table}

\subsection*{Permutation \aCipher{}}

\aCipher{} is a family of cryptographic permutations built on the ARX paradigm. Its name comes from the block cipher \sparx{}~\cite{OurSPARX}, which \aCipher{} is closely related to, hence its name:
\begin{center}
  \textbf{SPAR}x, but \textbf{K}ey \textbf{LE}ss.
\end{center}

We provide three versions corresponding to three block sizes, namely \aCipher{256}, \aCipher{384}, and \aCipher{512}. The number of steps used varies with the use case.

\subsection*{Hash Function \hash{}}

A \emph{hash function} takes a message of arbitrary length and outputs a digest with a fixed length. It should provide the cryptographic security notions of \emph{preimage resistance}, \emph{second preimage resistance} and \emph{collision resistance}.
The main instance of \hash{} is \hash{256} which produces a 256-bit digest, offering a security level of 128 bits with regard to the above mentioned security goals. It is based on the permutation family \aCipher{384}.  We also provide the member \hash{384} based on the permutation family \aCipher{512}, which produces a 384-bit digest and offers a security level of 192 bits.

The name \hash{} stands for
\begin{center}
  \textbf{E}fficient, \textbf{S}ponge-based, and \textbf{C}heap \textbf{H}ashing.
\end{center}

\subsection*{Authenticated Cipher \aead{}}

A scheme for \emph{authenticated encryption with associated data (AEAD)} takes a key and a nonce of fixed length, as well as a message and associated data of arbitrary size. The encryption procedure outputs a ciphertext of the message as well as a fixed-size authentication tag. The decryption procedure takes the key, nonce, associated data and the ciphertext and tag as input and outputs the decrypted message if the tag is valid, otherwise a symbolic error $\bot$.
An AEAD scheme should fulfill the security notions of \emph{confidentiality} and \emph{integrity}. Users are expected to \emph{not} reuse nonces for processing messages in a fixed-key instance.

The main instance of \aead{} is \aeadversion{256}{128} which takes a 256-bit nonce, a 128-bit key and outputs a 128-bit authentication tag. It achieves a security level of 120 bits with regard to confidentiality and integrity.
We further provide three other instances, i.e., \aeadversion{128}{128}, \aeadversion{192}{192}, and \aeadversion{256}{256} which differ in the length of key, nonce and tag and in the achieved security level.

The name \aead{} stands for
\begin{center}
  \textbf{S}ponge-based \textbf{C}ipher for
  \\
  \textbf{H}ardened but \textbf{W}eightless \textbf{A}uthenticated \textbf{E}ncryption \\
  on \textbf{M}any~\textbf{M}icrocontrollers
\end{center}

\subsection*{Outline}
This chapter is structured as follows. First, in \SecRef{spec}, I briefly describe the specification of \aCipher{},\hash{} and \aead{} families. In the following sections, I describe the security analysis that I performed on this suite.

In \SecRef{lineariz}, I describe attempts to linearize the S-boxes used in \aCipher{}, which we call ARX-boxes, by finding all inputs that inflict no carries during the ARX computations. The problem requires a solution of a system of quadratic equations. I describe a simple heuristics for a guess-and-determine algorithm that allows to solve the problems in a reasonable time. The results suggest that ARX-boxes are resistant against such linearization.

In \SecRef{truncated}, I describe a truncated differential analysis of \aCipher{}. I propose a generic method for truncated trail analysis based on the binary matrix representation of the linear layer. The results show that \aCipher{} has a strong resistance against structural truncated differential trails.

In \SecRef{division}, I use the division property technique to find integral characteristics of \aCipher{}. First, the best characteristics of maximum dimension are found using MILP-aided bit-based division property. Then, I optimize them and prove by a pen-and-paper argument and the classical division property.

Finally, in \SecRef{cryptanalysis}, I describe several attacks on reduced-round variants of \aead{}. They are based on a technique that I call birthday-differential attacks. It is a variant of known-plaintext attack where particular differences can be found from a relatively small pool of data by the birthday paradox.


\section{General}

\subsection{Conventions}
3D graphics \ac{API}s usually all have their own conventions regarding coordinate systems, vector and matrix forms, and so on. It is therefore imperative for a 3D framework to have well defined conventions that are used throughout. This section outlines the conventions used in the PixelLight renderer \ac{API}.

The renderer component uses the right hand system - matrices are column major order like \ac{OpenGL} does, but unlike Direct3D, x points to right, y upward and z points to the eye (see figure below). The vertices of front faces are specified in counter clockwise winding. Therefore, backface culling will usually be set to cull clockwise-oriented faces.

\begin{figure}
  \centering
  \includegraphics[scale=0.7]{pics/CoordinateSystem.eps}
  \caption{Coordinate system}
  \label{fig:Coordinate system}
\end{figure}

In the PixelLight framework itself scene nodes use the z axe as direction vector, x as left and y as up-vector. Cameras and spot lights invert the \emph{natural} z-axis internally so they can be used in the same way as all other scene nodes without the need to rotate them around 180 degrees along the y-axis - but this components are unknown at this level so we don't go into details in here.

\subsection{Renderer Component High-Level Class Diagram}
The following class diagram shows a high-level view of the most important class hierarchies in the render system, in particular the hierarchy of render resource, render surface and state classes. The subsequent sections describe the most important classes in more detail. The diagram is split up into three parts. The renderer itself, the \ac{API} backends the user will not use directly and and examples how the framework is using some renderer classes in general.

\begin{figure}
  \centering
  \includegraphics[scale=0.7]{pics/PLRendererClassDiagram.eps}
  \caption{Renderer component}
  \label{fig:Renderer component high-level class diagram}
\end{figure}

\subsection{Renderer Instance}
With the interface of the created renderer instance, an application can begin and end a scene, swap back with front buffers, create render targets (different types of windows and textures) and render resources, draw primitives, set render states etc. There's also a function to set the world, view and projection matrices and a whole range of other methods that usually directly map to the underlying rendering \ac{API}. One of the most frequently called methods of the renderer system are the two draw methods used to render primitives to the currently active render target. Supported primitives are point lists, line lists, line strips, triangle lists, triangle strips, and triangle fans. The vertices that compose primitives are stored in a \ac{VBO}. One of the draw methods takes a \ac{VBO} as input parameter and renders the geometry contained in the \ac{VBO}s defined by the mapping to the current render target. Another draw method additionally takes an \ac{IBO} as input parameter and renders indexed primitives to the current render target. It is advisable to use indexed primitives whenever possible, because it reduces the amount of memory sent to the graphics hardware. With indexed primitives, multiple primitives can use the same vertex by reusing its index in the \ac{IBO} without having to duplicate all the vertex attributes of that vertex in the \ac{VBO}.

The renderer classes are actually abstract base classes which classes are derived in the renderer backends that implement most of the functionality using the underlying rendering \ac{API}. Most of the time, the deriving classes have to map the routines to the corresponding routines of the rendering \ac{API} and not actually implement much. Therefore, implementing a renderer for a new 3D graphics \ac{API} is usually a task that is tedious, because of the high number of routines to implement, uncomplicated, since it mostly only involves mapping functions to functions - but headache producing when it comes to hardware dependent irregularities that are often not obvious by just looking at the code and that are not reproducible on each hardware and driver version. But it's also a bit tricky because in some cases the different \ac{API}s are quite different. Direct3D has some special features \ac{OpenGL} has not, \ac{OpenGL} has some special features Direct3D has not - therefore, in the most cases, the PixelLight renderer only supplies things both \ac{API}s have. No \emph{glVertex}... calls, quad primitives etc. because Direct3D does not have these things.

\subsection{Additional Tool Functions}
Within the PixelLight renderer, there are a few special tool functions which are normally not within the native renderer \ac{API}s like \ac{OpenGL} - but they will make your life more comfortable. There are some 2D functions you can for instance use for an ingame \ac{GUI} - or for 2D games.

Using \emph{Begin2DMode()} you begin the 2D mode and a orthogonal projection matrix is set automatically. Now you can draw your images using \emph{DrawImage()}, draw some points, lines, texts etc. By calling \emph{End2DMode()} you return into the previous \emph{3D mode}.

There are also some additional draw functions like \emph{DrawBox()} you can use to visualize some debugging stuff. This functions don't offer that many settings - they are for debugging only. To draw more complex stuff, fill the vertex and \ac{IBO}s by self or use the PixelLight mesh library which can also create a lot of geometric meshes automatically.

\subsubsection{Renderer Resources and States}
Nearly most of the renderer interfaces are dealing with resources and states. In the section \emph{Renderer component high-level class diagram} you can see the different types of resources and states. By using functions like \emph{CreateTextureBuffer2D()} you create such resources, by using state functions like \emph{SetRenderState()} you can set, get etc. states. If you are familiar with Direct3D you shouldn't have any problems by using the PixelLight renderer. For \ac{OpenGL} users this may be unaccustomed to set for instance all the different render states using one function \emph{SetRenderState()} - but this way is more handy, especial if you want to backup/restore all states at once.\footnote{\ac{OpenGL} has push/pop state functions, but they are not that comfortable}

\subsubsection{\ac{VBO} and \ac{IBO}}
All geometry data submitted to the renderer component must be stored in vertex and \ac{IBO}s which are derived of a basis buffer. A \ac{VBO} stores vertices where a vertex has a number of vertex attributes, such as position, normal, color, and so on. An \ac{IBO} stores indices into a \ac{VBO} pointing to certain vertices. To actually draw the geometry data, one of the draw methods of the renderer must be called. The draw methods take the primitive type, such as line list or triangle strip, and the number of primitives that are to be rendered as input parameters. One draw method just requires a \ac{VBO} as parameter containing all the vertices that define the to-be-drawn primitives. The vertices in the buffer are used one after the other. If, for example, the given primitive type is triangle list, every three vertices of the \ac{VBO}s will be used to render a triangle. Another draw method also takes a \ac{VBO} as input parameter and additionally an \ac{IBO} which contains indices into the \ac{VBO} adding a level of indirection. If the specified primitive type is again triangle list, every three indices in the \ac{IBO} will be used to retrieve three vertices from the \ac{VBO} that are then used to render a triangle. Since both vertex and \ac{IBO} are render resources, they can potentially be stored in video memory. When allocating a buffer you have to define it's usage, if it is static, dynamic etc. this definition will help the renderer to store the data in a optimal memory section. Static buffers normally directly stored on the
memory of the \ac{GPU} which is quite performant.

By defining \emph{vertex attributes} you specify the layout of a single vertex... that apply to all vertices within the \ac{VBO}. There are multiple \emph{semantics}, \emph{channels} and \emph{types} an attribute can choose from. The semantic defines the \emph{usual} usage of the attribute. Within a \emph{Position} attribute \emph{normally} a vertex position is stored - but when using shaders you can store everything within every semantic, but we recommend to respect some guidelines. Some semantics can have multiple channels, the \emph{TexCoord} semantic for example for multitexturing or providing additional data for shaders that do not fit into any other semantic.

Here's an excerpt from possible semantics:
\begin{itemize}
\item{Position}
\item{Normal}
\item{Color[n]}
\item{TexCoord[n]}
\end{itemize}

\emph{Color[n]} specifies n sets of colors, were the first is normally the primary color and the second the secondary color. \emph{TexCoord[n]} specifies n sets of texture coordinates, where n is a zero-based texture sampler stage index and cannot be greater than the number of available stages.

The attribute \emph{type} specifies the structure of the concrete attribute data - \emph{Float3} is for example quite common to store the x, y and z components of a vertex position. For the fixed-function pipeline the meaning and type of the vertex attributes is fixed and cannot be modified. But today you only need to take this into consideration when developing for legacy or minimal hardware. Vertex attributes are added using the \ac{VBO} method \emph{AddVertexAttribute()}. Normally you first setup all vertex attributes before allocating the buffer itself, but its possible to also add vertex attributes after a buffer was allocated, but normally this isn't recommended for performance reasons.

The vertex attribute sets do not have to be ordered neither for fixed-function nor for programmable pipeline. So this would be a perfectly legal binding for the pipeline:\\
Normal - TexCoord[2] - Position - TexCoord[0] - TexCoord[1]\\
... but none we personally would use.

Before manipulating or reading of a vertex or \ac{IBO} you have to call \emph{Lock()} were you should define what you want to to with the buffer - reading, writing etc. After your operations on the buffer are done you have to call the \emph{Unlock()} function. Try to avoid manipulating the buffers often per frame. Use the \emph{GetData()} function of the buffer to get the desired data. When working on \ac{VBO}s you have to define what you want to edit. When accessing colors you should use the special color functions. You should never see the \ac{VBO} as a normal well defined linear buffer were you know how many bytes each vertex attributes will have etc. because for instance different \ac{API}s will manage the colors in a different way. Using this provided \ac{VBO} function you don't have to care about this \ac{API} dependent stuff yourself and you are always on the safe side.

\subsubsection{Occlusion Query}
This defines a mechanism whereby an application can query the number of pixels, or more precisely samples, drawn by a primitive or group of primitives.

The primary purpose of such a query, hereafter referred to as an \emph{occlusion query}, is to determine the visibility of an object. Typically, the application will render the major occluders in the scene, then perform an occlusion query for the bounding box of each detail object in the scene. Only if said bounding box is visible, i.e., if at least one sample is drawn, should the corresponding object be drawn.

Here some examples were it's useful to know how many fragments of an object are visible:
\begin{itemize}
\item{Objects that are visible but cover only a very small number of pixels can be skipped at a minimal reduction of image quality.}
\item{Knowing exactly how many pixels an object might cover may help the application decide which level-of-detail model should be used. If only a few pixels are visible, a low-detail model may be acceptable.}
\item{Occlusion queries can replace read functions of the depth buffer to determine whether, for example, a light source is visible for the purposes of a lens flare effect or a halo to simulate glare. Pixel counts allow you to compute the percentage of the light source that is visible, and the brightness of these effects can be modulated accordingly.}
\end{itemize}

\subsection{Rectangle Textures}
\hypertarget{Rectangle textures}{}
Lookout, this section is only relevant for legacy \ac{GPU}s/drivers. Nowadays, the \ac{POT} dimension restriction for textures is gone. If you don't care about support for legacy hardware without universal \ac{NPOT}, just skip this section.

In past days, the hardware texturing was limited to images with \ac{POT} dimensions and an optional 1-texel border. But there was also a special kind of textures called \emph{rectangle textures} without requiring \ac{POT} dimensions.

\ac{NPOT} dimensioned textures are useful for storing video images that do not have \ac{POT} dimensions - for instance the most ingame \ac{GUI} elements. Re-sampling artefacts are avoided and less texture memory may be required by using \ac{NPOT} dimensioned textures. \ac{NPOT} dimensioned textures are also useful for shadow maps and window-space texturing.

However, \ac{NPOT} dimensioned textures have limitations that do not apply to \ac{POT} dimensioned textures. \ac{NPOT} textures may not use mipmap filtering; \ac{POT} textures support both mipmapped and non-mipmapped filtering. \ac{NPOT} textures support only the CLAMP, CLAMP TO EDGE, and CLAMP TO BORDER wrap modes; \ac{POT} textures support CLAMP TO EDGE, REPEAT, CLAMP, MIRRORED REPEAT, and CLAMP TO BORDER. \ac{NPOT} textures do not support an optional 1-texel border;
\ac{POT} textures do support an optional 1-texel border.

Whether a texture is a rectangle texture or not is defined in the textures \emph{plt}-file:

\begin{lstlisting}[caption=\emph{plt}-file rectangle texture setup]
<?xml version="1.0"?>
<Texture Version="1">
	<General Rectangle="1" />
</Texture>
\end{lstlisting}

Were the default value is 0 meaning that by default all textures must have a \ac{POT} dimension! If no rectangle textures are supported by the renderer, this setting will have no effect and \ac{POT} are used.

\subsection{Texture Compression}
Try to use texture compression whenever possible - this will shorten the loading times, enhance the performance and of course, will safe memory on the \ac{GPU}. The \emph{dds} image file format is best suited for this task because it supports many texture compression formats a \ac{GPU}s can directly work with. Other traditional image file formats like \emph{jpg} should not be used as texture - \ac{GPU}s don't understand the used compression techniques, and therefore after loading such an image, the data needs to be decompressed on the \ac{CPU} in order to be able to upload it to the \ac{GPU}. This is a slow process resulting in images with compression artefacts, but without a usable compression - it's even worse if this uncompressed data is compressed for instance to \emph{DXT1}\footnote{Called \emph{BC1} in \emph{DirectX 10}} before uploading to the \ac{CPU}. (slow and additional compression artefacts)

During production, one can use uncompressed textures so no compression artefacts are introduces during production when manipulating a texture multiple times. When publishing something, we highly recommend that the textures are compressed in a proper way. The additional work is really worth the effort, belief us.

\paragraph{Luminance Map Compression}
When using luminance maps, meaning maps with only one channel, \emph{LATC1}\footnote{Called \emph{BC4} in \emph{DirectX 10}, formerly \emph{ATI1N} on ATI \ac{GPU}s only} is a good choice for texture compression with high quality and the same memory footprint as \emph{DXT1}.

\paragraph{Color Map Compression}
When using color maps like photos, meaning maps with three channels, \emph{DXT1} is a good choice for texture compression.

\paragraph{Color Map with Alpha Channel Compression}
When using color maps which contain a alpha channel, meaning maps with four channels, \emph{DXT5}\footnote{Called \emph{BC3} in \emph{DirectX 10}} is a good choice for texture compression. For alpha channels with low coherency, \emph{DXT3}\footnote{Called \emph{BC2} in \emph{DirectX 10}} may reduce the memory footprint while still having a acceptable quality.

\paragraph{Normal Map Compression}
In general, do never ever compress normal maps with standard compression formats like \emph{DXT1}, \emph{DXT3} or \emph{DXT5}, this will lead to ugly visible normal artefacts because this formats are optimized for color based data, but in this case, we have vector data stored in the texture! \emph{DXT5 xGxR} can be used for best backward compatibility (not the best quality) and \emph{LATC2}\footnote{Called \emph{BC5} in \emph{DirectX 10}, formerly \emph{3DC}/\emph{ATI2N} on ATI \ac{GPU}s only} is the recommended format for normal map compression with reasonable quality and the same memory footprint as \emph{DXT5}. PixelLight also supports \emph{Alternate XY Swizzle LATC2} which enables shader code reuse of \emph{DXT5 xGxR}.

\emph{DXT5 xGxR} and \emph{Alternate XY Swizzle LATC2} are no standard extensions of the \emph{dds} image file format. Tools like \emph{The Compressonator} usually store their own additional information within the file to mark the special format, but other tools can't read in and understand this information. Instead of manipulating the \emph{dds} image file format, we decided to add an optional \emph{plt}-file per texture to add additional information about a texture.

To mark a texture as \emph{DXT5 xGxR}, add a \emph{plt}-file with the same name as the texture and with the following content:
\begin{lstlisting}[caption=\emph{plt}-file \emph{DXT5 xGxR} normal map compression]
<?xml version="1.0"?>
<Texture Version="1">
	<General Compression="DXT5_xGxR" />
</Texture>
\end{lstlisting}

To mark a texture as \emph{Alternate XY Swizzle LATC2}, add a \emph{plt}-file with the same name as the texture and with the following content:
\begin{lstlisting}[caption=\emph{plt}-file \emph{Alternate XY Swizzle LATC2} normal map compression]
<?xml version="1.0"?>
<Texture Version="1">
	<General Compression="LATC2_XYSwizzle" />
</Texture>
\end{lstlisting}

\paragraph{More Information and Tools}
For additional information about texture compression, especially about normal map compression, have a look at
\url{http://developer.nvidia.com/object/real-time-normal-map-dxt-compression.html} and \url{http://www.ozone3d.net/tutorials/bump_map_compression.php}.
There are many tools dealing with textures, \emph{The Compressonator}\footnote{\url{http://developer.amd.com/gpu/compressonator/Pages/default.aspx}} from ATI is one of the better ones and can directly deal with for instance \emph{DXT5 xGxR} or \emph{Alternate XY Swizzle LATC2}.