text:
package main

// Libraries that are included to vendor utilities used during build.
// These libraries will not be included in a normal compilation.

import (
	// for lint
	_ "github.com/mgechev/dots"
	_ "github.com/mgechev/revive/formatter"
	_ "github.com/mgechev/revive/lint"
	_ "github.com/mgechev/revive/rule"
	_ "github.com/mitchellh/go-homedir"
	_ "github.com/pelletier/go-toml"

	// for embed
	_ "github.com/shurcooL/vfsgen"

	// for cover merge
	_ "golang.org/x/tools/cover"

	// for vet
	_ "code.gitea.io/gitea-vet"

	// for swagger
	_ "github.com/go-swagger/go-swagger/cmd/swagger"
)


text:
# Clustering Guide

## Overview

This guide covers fundamental topics related to RabbitMQ clustering:

 * How RabbitMQ nodes are identified: [node names](#node-names)
 * [Requirements](#cluster-formation-requirements) for clustering
 * What data is and isn't [replicated between cluster nodes](#cluster-membership)
 * What clustering [means for clients](#clustering-and-clients)
 * [How clusters are formed](#cluster-formation)
 * How nodes [authenticate to each other](#erlang-cookie) (and with CLI tools)
 * Why it's important to [use an odd number of nodes](#node-count) and **two cluster nodes are highly recommended against**
 * [Node restarts](#restarting) and how nodes rejoin their cluster
 * [Node readiness probes](#restarting-readiness-probes) and how they can affect rolling cluster restarts
 * How to [remove a cluster node](#removing-nodes)
 * How to [reset a cluster node](#resetting-nodes) to a pristine (blank) state

and more. [Cluster Formation and Peer Discovery](./cluster-formation.html) is a closely related guide
that focuses on peer discovery and cluster formation automation-related topics. For queue contents
(message) replication, see the [Quorum Queues](./quorum-queues.html) guide.

[VMware RabbitMQ](https://docs.vmware.com/en/VMware-Tanzu-RabbitMQ-for-Kubernetes/index.html) provides an [inter-node traffic compression](clustering-compression.html) feature.

A RabbitMQ cluster is a logical grouping of one or
several nodes, each  sharing users, virtual hosts,
queues, exchanges, bindings, runtime parameters and other distributed state.

## Cluster Formation
### Ways of Forming a Cluster

A RabbitMQ cluster can be formed in a number of ways:

 * Declaratively by listing cluster nodes in [config file](configure.html)
 * Declaratively using DNS-based discovery
 * Declaratively using [AWS (EC2) instance discovery](https://github.com/rabbitmq/rabbitmq-peer-discovery-aws) (via a plugin)
 * Declaratively using [Kubernetes discovery](https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s) (via a plugin)
 * Declaratively using [Consul-based discovery](https://github.com/rabbitmq/rabbitmq-peer-discovery-consul) (via a plugin)
 * Declaratively using [etcd-based discovery](https://github.com/rabbitmq/rabbitmq-peer-discovery-etcd) (via a plugin)
 * Manually with `rabbitmqctl`

Please refer to the [Cluster Formation guide](./cluster-formation.html) for details.

The composition of a cluster can be altered dynamically.
All RabbitMQ brokers start out as running on a single
node. These nodes can be joined into clusters, and
subsequently turned back into individual brokers again.

### Node Names (Identifiers)

RabbitMQ nodes are identified by node names. A node name consists of two parts,
a prefix (usually `rabbit`) and hostname. For example, `rabbit@node1.messaging.svc.local`
is a node name with the prefix of `rabbit` and hostname of `node1.messaging.svc.local`.

Node names in a cluster must be unique. If more than one node is running on a given host
(this is usually the case in development and QA environments), they must use
different prefixes, e.g. `rabbit1@hostname` and `rabbit2@hostname`.

In a cluster, nodes identify and contact each other using node names. This means
that the hostname part of every node name [must resolve](#hostname-resolution-requirement).
[CLI tools](./cli.html) also identify and address nodes using node names.

When a node starts up, it checks whether it has been assigned a node name. This is done
via the `RABBITMQ_NODENAME` [environment variable](./configure.html#supported-environment-variables).
If no value was explicitly configured,
the node resolves its hostname and prepends `rabbit` to it to compute its node name.

If a system uses fully qualified domain names (FQDNs) for hostnames, RabbitMQ nodes
and CLI tools must be configured to use so called long node names.
For server nodes this is done by setting the `RABBITMQ_USE_LONGNAME` [environment variable](./configure.html#supported-environment-variables)
to `true`.

For CLI tools, either `RABBITMQ_USE_LONGNAME` must be set or the `--longnames` option
must be specified.

## Cluster Formation Requirements
### Hostname Resolution

RabbitMQ nodes address each other using a **node name**, a combination
of a prefix and domain name, either short or fully-qualified (FQDNs).

Therefore every cluster member **must be able to resolve hostnames
of every other cluster member**, its own hostname, as well
as machines on which command line tools such as `rabbitmqctl`
might be used.

Nodes will perform hostname resolution early on node boot.
In container-based environments it is important that hostname
resolution is ready before the container is started.
For Kubernetes users, this means the [DNS cache interval for CoreDNS](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#stable-network-id)
to a value in the 5-10 second range.

Hostname resolution can use any of the standard OS-provided
methods:

 * DNS records
 * Local host files (e.g. `/etc/hosts`)

In more restrictive environments, where DNS record or
hosts file modification is restricted, impossible or
undesired, Erlang
VM can be configured to use alternative hostname
resolution methods, such as an alternative DNS server,
a local file, a non-standard hosts file location, or a mix
of methods.  Those methods can work in concert with the
standard OS hostname resolution methods.

To use FQDNs, see `RABBITMQ_USE_LONGNAME` in the [Configuration guide](./configure.html#supported-environment-variables).
See [Node Names](#node-names) above.

## Port Access

RabbitMQ nodes [bind to ports](networking.html#ports) (open server TCP sockets) in order to accept client and CLI tool connections.
Other processes and tools such as SELinux may prevent RabbitMQ from binding to a port. When that happens,
the node will fail to start.

CLI tools, client libraries and RabbitMQ nodes also open connections (client TCP sockets).
Firewalls can prevent nodes and CLI tools from communicating with each other.
The following ports are most relevant to inter-node communication in a cluster:

 * 4369: [epmd](networking.html#epmd), a helper discovery daemon used by RabbitMQ nodes and CLI tools
 * 6000 through 6500: used by [RabbitMQ Stream](stream.html) replication
 * 25672: used for inter-node and CLI tools communication (Erlang distribution server port)
   and is allocated from a dynamic range (limited to a single port by default,
   computed as AMQP port + 20000). Unless external connections on these ports are really necessary (e.g.
   the cluster uses [federation](federation.html) or CLI tools are used on machines outside the subnet),
   these ports should not be publicly exposed. See [networking guide](networking.html) for details.
 * 35672-35682: used by CLI tools (Erlang distribution client ports) for communication with nodes
   and is allocated from a dynamic range (computed as server distribution port + 10000 through
   server distribution port + 10010).

It is possible to [configure RabbitMQ](configure.html)
to use different ports and specific network interfaces.
See [RabbitMQ Networking guide](networking.html) to learn more.

## Nodes in a Cluster

### What is Replicated?

All data/state required for the operation of a RabbitMQ
broker is replicated across all nodes. An exception to this
are message queues, which by default reside on one node,
though they are visible and reachable from all nodes. To
replicate queues across nodes in a cluster, use a queue type
that supports replication. This topic is covered in
the [Quorum Queues](./quorum-queues.html) guide.

### Nodes are Equal Peers

Some distributed systems
have leader and follower nodes. This is generally not true for RabbitMQ.
All nodes in a RabbitMQ cluster are equal peers: there are no special nodes in RabbitMQ core.
This topic becomes more nuanced when [quorum queues](quorum-queues.html) and plugins
are taken into consideration but for most intents and purposes,
all cluster nodes should be considered equal.

Many [CLI tool](./cli.html) operations can be executed against any node.
An [HTTP API](./management.html) client can target any cluster node.

Individual plugins can designate (elect)
certain nodes to be "special" for a period of time. For example, [federation links](federation.html)
are colocated on a particular cluster node. Should that node fail, the links will
be restarted on a different node.

In older (long maintained) versions, [RabbitMQ management plugin](./management.html) used
a dedicated node for stats collection and aggregation.

### How CLI Tools Authenticate to Nodes (and Nodes to Each Other): the Erlang Cookie

RabbitMQ nodes and CLI tools (e.g. `rabbitmqctl`) use a
cookie to determine whether they are allowed to communicate with
each other. For two nodes to be able to communicate they must have
the same shared secret called the Erlang cookie. The cookie is
just a string of alphanumeric characters up to 255 characters in size.
It is usually stored in a local file. The file must be only
accessible to the owner (e.g. have UNIX permissions of `600` or similar).
Every cluster node must have the same cookie.

If the file does not exist, Erlang VM will try to create
one with a randomly generated value when the RabbitMQ server
starts up. Using such generated cookie files are appropriate in development
environments only. Since each node will generate its own value independently,
this strategy is not really viable in a [clustered environment](./clustering.html).

Erlang cookie generation should be done at cluster deployment stage, ideally using automation
and orchestration tools.

In distributed deployment

### Cookie File Locations

#### Linux, MacOS, *BSD

On UNIX systems, the cookie will be typically
located in `/var/lib/rabbitmq/.erlang.cookie` (used by the server)
and `$HOME/.erlang.cookie` (used by CLI tools). Note that since the value
of `$HOME` varies from user to user, it's necessary to place a copy of
the cookie file for each user that will be using the CLI tools.
This applies to both non-privileged users and `root`.

RabbitMQ nodes will log its effective user's home directory location early on boot.

#### Community Docker Image and Kubernetes

[Docker community RabbitMQ image](https://github.com/docker-library/rabbitmq/) uses `RABBITMQ_ERLANG_COOKIE` environment variable value
to populate the cookie file.

Configuration management and container orchestration tools that use this image
must make sure that every RabbitMQ node container in a cluster uses the same value.

In the context of Kubernetes, the value must be specified in the pod template specification of
the [stateful set](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/).
For instance, this can be seen in the [RabbitMQ on Kubernetes examples repository](https://github.com/rabbitmq/diy-kubernetes-examples).

#### Windows

On Windows, the cookie location depends on a few factors:

 * Whether the `HOMEDRIVE` and `HOMEPATH` environment variables are both set
 * Erlang version: prior to 20.2 (these are no longer supported by any [maintained release series of RabbitMQ](versions.html)) or 20.2 and later

##### Erlang 20.2 or later

With Erlang versions starting with 20.2, the cookie file locations are:

 * `%HOMEDRIVE%%HOMEPATH%\.erlang.cookie` (usually `C:\Users\%USERNAME%\.erlang.cookie` for user `%USERNAME%`) if both the `HOMEDRIVE` and `HOMEPATH` environment variables are set
 * `%USERPROFILE%\.erlang.cookie` (usually `C:\Users\%USERNAME%\.erlang.cookie`) if `HOMEDRIVE` and `HOMEPATH` are not both set
 * For the RabbitMQ Windows service - `%USERPROFILE%\.erlang.cookie` (usually `C:\WINDOWS\system32\config\systemprofile`)

If the Windows service is used, the cookie should be copied from
`C:\Windows\system32\config\systemprofile\.erlang.cookie` to the expected
location for users running commands like `rabbitmqctl.bat`.

### Overriding Using CLI and Runtime Command Line Arguments

As an alternative, the option "`-setcookie <value>`" can be added
to `RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS` environment variable value
to override the cookie value used by a RabbitMQ node:

RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-setcookie cookie-value"

CLI tools can take a cookie value using a command line flag:

rabbitmq-diagnostics status --erlang-cookie "cookie-value"

Both are **the least secure options** and generally **not recommended**.

### Troubleshooting

When a node starts, it will [log](./logging.html) the home directory location of its effective user:

node           : rabbit@cdbf4de5f22d
home dir       : /var/lib/rabbitmq

Unless any [server directories](./relocate.html) were overridden, that's the directory where
the cookie file will be looked for, and created by the node on first boot if it does not already exist.

In the example above, the cookie file location will be `/var/lib/rabbitmq/.erlang.cookie`.

### Authentication Failures

When the cookie is misconfigured (for example, not identical), RabbitMQ nodes will log errors
such as "Connection attempt from disallowed node", "", "Could not auto-cluster".

For example, when a CLI tool connects and tries to authenticate using a mismatching secret value:

2020-06-15 13:03:33 [error] <0.1187.0> ** Connection attempt from node 'rabbitmqcli-99391-rabbit@warp10' rejected. Invalid challenge reply. **

When a CLI tool such as `rabbitmqctl` fails to authenticate with RabbitMQ,
the message usually says

* epmd reports node 'rabbit' running on port 25672
* TCP connection succeeded but Erlang distribution failed
* suggestion: hostname mismatch?
* suggestion: is the cookie set correctly?
* suggestion: is the Erlang distribution using TLS?

An incorrectly placed cookie file or cookie value mismatch are most common scenarios for such failures.

When a recent Erlang/OTP version is used, authentication failures contain
more information and cookie mismatches can be identified better:

* connected to epmd (port 4369) on warp10
* epmd reports node 'rabbit' running on port 25672
* TCP connection succeeded but Erlang distribution failed

* Authentication failed (rejected by the remote node), please check the Erlang cookie

See the [CLI Tools guide](./cli.html) for more information.

### Troubleshooting Cookie-based Authentication

#### Server Nodes

When a node starts, it will [log](./logging.html) the home directory location of its effective user:

node           : rabbit@cdbf4de5f22d
home dir       : /var/lib/rabbitmq

Unless any [server directories](./relocate.html) were overridden, that's the directory where
the cookie file will be looked for, and created by the node on first boot if it does not already exist.

In the example above, the cookie file location will be `/var/lib/rabbitmq/.erlang.cookie`.

#### Hostname Resolution

Since hostname resolution is a [prerequisite for successful inter-node communication](#hostname-resolution-requirement),
starting with [RabbitMQ `3.8.6`](./changelog.html), CLI tools provide two commands that help verify
that hostname resolution on a node works as expected. The commands are not meant to replace
[`dig`](https://en.wikipedia.org/wiki/Dig_(command)) and other specialised DNS tools but rather
provide a way to perform most basic checks while taking [Erlang runtime hostname resolver features](https://erlang.org/doc/apps/erts/inet_cfg.html)
into account.

The commands are covered in the [Networking guide](./networking.html#dns-verify-resolution).

#### CLI Tools

Starting with [version `3.8.6`](./changelog.html), `rabbitmq-diagnostics` includes a command
that provides relevant information on the Erlang cookie file used by CLI tools:

rabbitmq-diagnostics erlang_cookie_sources

The command will report on the effective user, user home directory and the expected location
of the cookie file:

Cookie File

Effective user: antares
Effective home directory: /home/cli-user
Cookie file path: /home/cli-user/.erlang.cookie
Cookie file exists? true
Cookie file type: regular
Cookie file access: read
Cookie file size: 20

Cookie CLI Switch

--erlang-cookie value set? false
--erlang-cookie value length: 0

Env variable  (Deprecated)

RABBITMQ_ERLANG_COOKIE value set? false
RABBITMQ_ERLANG_COOKIE value length: 0

## Node Counts and Quorum

Because several features (e.g. [quorum queues](./quorum-queues.html), [client tracking in MQTT](./mqtt.html))
require a consensus between cluster members, odd numbers of cluster nodes are highly recommended:
1, 3, 5, 7 and so on.

Two node clusters are **highly recommended against** since it's impossible for cluster nodes to identify
a majority and form a consensus in case of connectivity loss. For example, when the two nodes lose connectivity
MQTT client connections won't be accepted, quorum queues would lose their availability, and so on.

From the consensus point of view, four or six node clusters would have the same availability
characteristics as three and five node clusters.

The [Quorum Queues guide](./quorum-queues.html) covers this topic in more detail.

## Clustering and Clients

Assuming all cluster members
are available, a client can connect to any node and
perform any operation. Nodes will route operations to the
[quorum queue leader](./quorum-queues.html) or [queue leader replica](ha.html#leader-migration-data-locality)
transparently to clients.

With all supported messaging protocols a client is only connected to one node
at a time.

In case of a node failure, clients should be able to reconnect
to a different node, recover their topology and continue operation. For
this reason, most client libraries accept a list of endpoints (hostnames or IP addresses)
as a connection option. The list of hosts will be used during initial connection
as well as connection recovery, if the client supports it. See documentation guides
for individual clients to learn more.

With [quorum queues](./quorum-queues.html) and [streams](./streams.html), clients will only be able to perform
operations on queues that have a quorum of replicas online.

With classic mirrored queues, there are scenarios where it may not be possible for a client to transparently continue
operations after connecting to a different node. They usually involve
[non-mirrored queues hosted on a failed node](./ha.html#non-mirrored-queue-behavior-on-node-failure).

## Clustering and Observability

Client connections, channels and queues will be distributed across cluster nodes.
Operators need to be able to inspect and [monitor](./monitoring.html) such resources
across all cluster nodes.

RabbitMQ [CLI tools](./cli.html) such as `rabbitmq-diagnostics` and `rabbitmqctl`
provide commands that inspect resources and cluster-wide state. Some commands focus on the state of a single node
(e.g. `rabbitmq-diagnostics environment` and `rabbitmq-diagnostics status`), others
inspect cluster-wide state. Some examples of the latter include `rabbitmqctl list_connections`,
`rabbitmqctl list_mqtt_connections`, `rabbitmqctl list_stomp_connections`, `rabbitmqctl list_users`,
`rabbitmqctl list_vhosts` and so on.

Such "cluster-wide" commands will often contact one node
first, discover cluster members and contact them all to
retrieve and combine their respective state. For example,
`rabbitmqctl list_connections` will contact all
nodes, retrieve their AMQP 0-9-1 and AMQP 1.0 connections,
and display them all to the user.  The user doesn't have
to manually contact all nodes. Assuming a non-changing
state of the cluster (e.g. no connections are closed or
opened), two CLI commands executed against two different
nodes one after another will produce identical or
semantically identical results. "Node-local" commands, however, will not produce
identical results since two nodes rarely have identical state: at the very least their
node names will be different!

[Management UI](./management.html) works similarly: a node that has to respond to an HTTP API request
will fan out to other cluster members and aggregate their responses. In a cluster with multiple nodes that have management plugin
enabled, the operator can use any node to access management UI. The same goes for monitoring tools that use
the HTTP API to collect data about the state of the cluster. There is no need to issue a request to every cluster node in turn.

### Node Failure Handling

RabbitMQ brokers tolerate the failure of individual
nodes. Nodes can be started and stopped at will,
as long as they can contact a cluster member node
known at the time of shutdown.

[Quorum queue](quorum-queues.html) allows queue contents to be replicated
across multiple cluster nodes with parallel replication and a predictable [leader election](#quorum-queues.html#leader-election)
and [data safety](quorum-queues.html#data-safety) behavior as long as a majority of replicas are online.

Non-replicated classic queues can also be used in clusters. Non-mirrored queue [behaviour in case of node failure](./ha.html#non-mirrored-queue-behavior-on-node-failure)
depends on [queue durability](queues.html#durability).

RabbitMQ clustering has several modes of dealing with [network partitions](partitions.html),
primarily consistency oriented. Clustering is meant to be used across LAN. It is
not recommended to run clusters that span WAN.
The [Shovel](shovel.html) or
[Federation](federation.html)
plugins are better solutions for connecting brokers across a
WAN. Note that [Shovel and Federation are not equivalent to clustering](distributed.html).

### Metrics and Statistics

Every node stores and aggregates its own metrics and stats, and provides an API for
other nodes to access it. Some stats are cluster-wide, others are specific to individual nodes.
Node that responds to an [HTTP API](./management.html) request contacts its peers
to retrieve their data and then produces an aggregated result.

In older (long unmaintained) versions [RabbitMQ management plugin](./management.html) used
a dedicated node for stats collection and aggregation.

## Clustering Transcript with `rabbitmqctl`

The following several sections provide a transcript of manually setting up and manipulating
a RabbitMQ cluster across three machines: `rabbit1`, `rabbit2`,
`rabbit3`. It is recommended that the example is studied before
[more automation-friendly](./cluster-formation.html) cluster formation
options are used.

We assume that the user is logged into all three machines,
that RabbitMQ has been installed on the machines, and that
the rabbitmq-server and rabbitmqctl scripts are in the
user's PATH.

This transcript can be modified to run on a single host, as
explained more details below.

## Starting Independent Nodes

Clusters are set up by re-configuring existing RabbitMQ
nodes into a cluster configuration. Hence the first step
is to start RabbitMQ on all nodes in the normal way:

# on rabbit1
rabbitmq-server -detached
# on rabbit2
rabbitmq-server -detached
# on rabbit3
rabbitmq-server -detached

This creates three independent RabbitMQ brokers,
one on each node, as confirmed by the cluster_status
command:

# on rabbit1
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit1 ...
# => [{nodes,[{disc,[rabbit@rabbit1]}]},{running_nodes,[rabbit@rabbit1]}]
# => ...done.

# on rabbit2
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit2 ...
# => [{nodes,[{disc,[rabbit@rabbit2]}]},{running_nodes,[rabbit@rabbit2]}]
# => ...done.

# on rabbit3
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit3 ...
# => [{nodes,[{disc,[rabbit@rabbit3]}]},{running_nodes,[rabbit@rabbit3]}]
# => ...done.

The node name of a RabbitMQ broker started from the
`rabbitmq-server` shell script is
rabbit@shorthostname, where the short
node name is lower-case (as in `rabbit@rabbit1`,
above). On Windows, if `rabbitmq-server.bat`
batch file is used, the short node name is upper-case (as
in `rabbit@RABBIT1`). When you type node names,
case matters, and these strings must match exactly.

## Creating a Cluster

In order to link up our three nodes in a cluster, we tell
two of the nodes, say `rabbit@rabbit2` and
`rabbit@rabbit3`, to join the cluster of the
third, say `rabbit@rabbit1`. Prior to that both
newly joining members must be [reset](./rabbitmqctl.8.html#reset).

We first join `rabbit@rabbit2` in a cluster
with `rabbit@rabbit1`. To do that, on
`rabbit@rabbit2` we stop the RabbitMQ
application and join the `rabbit@rabbit1`
cluster, then restart the RabbitMQ application. Note that
a node must be [reset](./rabbitmqctl.8.html#reset) before it can join an existing cluster.
Resetting the node removes all resources and data that were previously
present on that node. This means that a node cannot be made a member
of a cluster and keep its existing data at the same time. When that's desired,
using the [Blue/Green deployment strategy](./blue-green-upgrade.html) or [backup and restore](./backup.html)
are the available options.

# on rabbit2
rabbitmqctl stop_app
# => Stopping node rabbit@rabbit2 ...done.

rabbitmqctl reset
# => Resetting node rabbit@rabbit2 ...

rabbitmqctl join_cluster rabbit@rabbit1
# => Clustering node rabbit@rabbit2 with [rabbit@rabbit1] ...done.

rabbitmqctl start_app
# => Starting node rabbit@rabbit2 ...done.

We can see that the two nodes are joined in a cluster by
running the cluster_status command on either of the nodes:

# on rabbit1
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit1 ...
# => [{nodes,[{disc,[rabbit@rabbit1,rabbit@rabbit2]}]},
# =>  {running_nodes,[rabbit@rabbit2,rabbit@rabbit1]}]
# => ...done.

# on rabbit2
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit2 ...
# => [{nodes,[{disc,[rabbit@rabbit1,rabbit@rabbit2]}]},
# =>  {running_nodes,[rabbit@rabbit1,rabbit@rabbit2]}]
# => ...done.

Now we join `rabbit@rabbit3` to the same
cluster. The steps are identical to the ones above, except
this time we'll cluster to `rabbit2` to
demonstrate that the node chosen to cluster to does not
matter - it is enough to provide one online node and the
node will be clustered to the cluster that the specified
node belongs to.

# on rabbit3
rabbitmqctl stop_app
# => Stopping node rabbit@rabbit3 ...done.

# on rabbit3
rabbitmqctl reset
# => Resetting node rabbit@rabbit3 ...

rabbitmqctl join_cluster rabbit@rabbit2
# => Clustering node rabbit@rabbit3 with rabbit@rabbit2 ...done.

rabbitmqctl start_app
# => Starting node rabbit@rabbit3 ...done.

We can see that the three nodes are joined in a cluster by
running the `cluster_status` command on any of the nodes:

# on rabbit1
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit1 ...
# => [{nodes,[{disc,[rabbit@rabbit1,rabbit@rabbit2,rabbit@rabbit3]}]},
# =>  {running_nodes,[rabbit@rabbit3,rabbit@rabbit2,rabbit@rabbit1]}]
# => ...done.

# on rabbit2
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit2 ...
# => [{nodes,[{disc,[rabbit@rabbit1,rabbit@rabbit2,rabbit@rabbit3]}]},
# =>  {running_nodes,[rabbit@rabbit3,rabbit@rabbit1,rabbit@rabbit2]}]
# => ...done.

# on rabbit3
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit3 ...
# => [{nodes,[{disc,[rabbit@rabbit3,rabbit@rabbit2,rabbit@rabbit1]}]},
# =>  {running_nodes,[rabbit@rabbit2,rabbit@rabbit1,rabbit@rabbit3]}]
# => ...done.

By following the above steps we can add new nodes to the
cluster at any time, while the cluster is running.

## Restarting Cluster Nodes

Nodes that have been joined to a cluster can be stopped at
any time. They can also fail or be terminated by the OS.

In general, if the majority of nodes is still online after a node
is stopped, this does not affect the rest of the cluster, although
client connection distribution, queue replica placement, and load distribution
of the cluster will change.

### Schema Syncing from Online Peers

A restarted node will sync the schema
and other information from its peers on boot. Before this process
completes, the node **won't be fully started and functional**.

It is therefore important to understand the process node go through when
they are stopped and restarted.

A stopping node picks an online cluster member (only disc
nodes will be considered) to sync with after restart. Upon
restart the node will try to contact that peer 10 times by
default, with 30 second response timeouts.

In case the peer becomes available in that time interval, the node
successfully starts, syncs what it needs from the peer and
keeps going.

If the peer does not become available, the restarted
node will **give up and voluntarily stop**. Such condition can be
identified by the timeout (`timeout_waiting_for_tables`) warning messages in the logs
that eventually lead to node startup failure:

2020-07-27 21:10:51.361 [warning] <0.269.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@node2,rabbit@node1],[rabbit_durable_queue]}
2020-07-27 21:10:51.361 [info] <0.269.0> Waiting for Mnesia tables for 30000 ms, 1 retries left
2020-07-27 21:11:21.362 [warning] <0.269.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@node2,rabbit@node1],[rabbit_durable_queue]}
2020-07-27 21:11:21.362 [info] <0.269.0> Waiting for Mnesia tables for 30000 ms, 0 retries left

2020-07-27 21:15:51.380 [info] <0.269.0> Waiting for Mnesia tables for 30000 ms, 1 retries left
2020-07-27 21:16:21.381 [warning] <0.269.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@node2,rabbit@node1],[rabbit_user,rabbit_user_permission, …]}
2020-07-27 21:16:21.381 [info] <0.269.0> Waiting for Mnesia tables for 30000 ms, 0 retries left
2020-07-27 21:16:51.393 [info] <0.44.0> Application mnesia exited with reason: stopped

2020-07-27 21:16:51.397 [error] <0.269.0> BOOT FAILED
2020-07-27 21:16:51.397 [error] <0.269.0> ===========
2020-07-27 21:16:51.397 [error] <0.269.0> Timeout contacting cluster nodes: [rabbit@node1].

When a node has no online peers during shutdown, it will start without
attempts to sync with any known peers. It does not start as a standalone
node, however, and peers will be able to rejoin it.

When the entire cluster is brought down therefore, the last node to go down
is the only one that didn't have any running peers at the time of shutdown.
That node can start without contacting any peers first.
Since nodes will try to contact a known peer for up to 5 minutes (by default), nodes
can be restarted in any order in that period of time. In this case
they will rejoin each other one by one successfully. This window of time
can be adjusted using two configuration settings:

# wait for 60 seconds instead of 30
mnesia_table_loading_retry_timeout = 60000

# retry 15 times instead of 10
mnesia_table_loading_retry_limit = 15

By adjusting these settings and tweaking the time window in which
known peer has to come back it is possible to account for cluster-wide
redeployment scenarios that can be longer than 5 minutes to complete.

During [upgrades](./upgrade.html), sometimes the last node to stop
must be the first node to be started after the upgrade. That node will be designated to perform
a cluster-wide schema migration that other nodes can sync from and apply when they
rejoin.

### Restarts and Health Checks (Readiness Probes)

In some environments, node restarts are controlled with a designated [health check](./monitoring.html#health-checks).
The checks verify that one node has started and the deployment process can proceed to the next one.
If the check does not pass, the deployment of the node is considered to be incomplete and the deployment process
will typically wait and retry for a period of time. One popular example of such environment is Kubernetes
where an operator-defined [readiness probe](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-readiness-gate)
can prevent a deployment from proceeding when the [`OrderedReady` pod management policy](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#deployment-and-scaling-guarantees) is used. Deployments that use the `Parallel` pod management policy
will not be affected but must worry about the [natural race condition during initial cluster formation](./cluster-formation.html#initial-formation-race-condition).

Given the [peer syncing behavior described above](#restarting-schema-sync), such a health check can prevent a cluster-wide restart
from completing in time. Checks that explicitly or implicitly assume a fully booted node that's rejoined
its cluster peers will fail and block further node deployments.

[Most health check](./monitoring.html#health-checks), even relatively basic ones, implicitly assume that the node has
finished booting. They are not suitable for nodes that are [awaiting schema table sync](#restarting-schema-sync) from a peer.

One very common example of such check is

# will exit with an error for the nodes that are currently waiting for
# a peer to sync schema tables from
rabbitmq-diagnostics check_running

One health check that does not expect a node to be fully booted and have schema tables synced is

# a very basic check that will succeed for the nodes that are currently waiting for
# a peer to sync schema from
rabbitmq-diagnostics ping

This basic check would allow the deployment to proceed and the nodes to eventually rejoin each other,
assuming they are [compatible](./upgrade.html).

### Hostname Changes Between Restarts

A node rejoining after a node name or host name change can start as [a blank node](./cluster-formation.html#peer-discovery-how-does-it-work)
if its data directory path changes as a result. Such nodes will fail to rejoin the cluster.
While the node is offline, its peers can be reset or started with a blank data directory.
In that case the recovering node will fail to rejoin its peer as well since internal data store cluster
identity would no longer match.

Consider the following scenario:

1. A cluster of 3 nodes, A, B and C is formed
2. Node A is shut down
3. Node B is reset
4. Node A is started
5. Node A tries to rejoin B but B's cluster identity has changed
6. Node B doesn't recognise A as a known cluster member because it's been reset

in this case node B will reject the clustering attempt from A with an appropriate error
message in the log:

Node 'rabbit@node1.local' thinks it's clustered with node 'rabbit@node2.local', but 'rabbit@node2.local' disagrees

In this case B can be reset again and then will be able to join A, or A
can be reset and will successfully join B.

### Cluster Node Restart Example

The below example uses CLI tools to shut down the nodes `rabbit@rabbit1` and
`rabbit@rabbit3` and check on the cluster
status at each step:

# on rabbit1
rabbitmqctl stop
# => Stopping and halting node rabbit@rabbit1 ...done.

# on rabbit2
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit2 ...
# => [{nodes,[{disc,[rabbit@rabbit1,rabbit@rabbit2,rabbit@rabbit3]}]},
# =>  {running_nodes,[rabbit@rabbit3,rabbit@rabbit2]}]
# => ...done.

# on rabbit3
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit3 ...
# => [{nodes,[{disc,[rabbit@rabbit1,rabbit@rabbit2,rabbit@rabbit3]}]},
# =>  {running_nodes,[rabbit@rabbit2,rabbit@rabbit3]}]
# => ...done.

# on rabbit3
rabbitmqctl stop
# => Stopping and halting node rabbit@rabbit3 ...done.

# on rabbit2
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit2 ...
# => [{nodes,[{disc,[rabbit@rabbit1,rabbit@rabbit2,rabbit@rabbit3]}]},
# =>  {running_nodes,[rabbit@rabbit2]}]
# => ...done.

In the below example, the nodes are started back, checking on the cluster
status as we go along:

# on rabbit1
rabbitmq-server -detached
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit1 ...
# => [{nodes,[{disc,[rabbit@rabbit1,rabbit@rabbit2,rabbit@rabbit3]}]},
# =>  {running_nodes,[rabbit@rabbit2,rabbit@rabbit1]}]
# => ...done.

# on rabbit2
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit2 ...
# => [{nodes,[{disc,[rabbit@rabbit1,rabbit@rabbit2,rabbit@rabbit3]}]},
# =>  {running_nodes,[rabbit@rabbit1,rabbit@rabbit2]}]
# => ...done.

# on rabbit3
rabbitmq-server -detached

# on rabbit1
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit1 ...
# => [{nodes,[{disc,[rabbit@rabbit1,rabbit@rabbit2,rabbit@rabbit3]}]},
# =>  {running_nodes,[rabbit@rabbit2,rabbit@rabbit1,rabbit@rabbit3]}]
# => ...done.

# on rabbit2
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit2 ...
# => [{nodes,[{disc,[rabbit@rabbit1,rabbit@rabbit2,rabbit@rabbit3]}]},
# =>  {running_nodes,[rabbit@rabbit1,rabbit@rabbit2,rabbit@rabbit3]}]
# => ...done.

# on rabbit3
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit3 ...
# => [{nodes,[{disc,[rabbit@rabbit1,rabbit@rabbit2,rabbit@rabbit3]}]},
# =>  {running_nodes,[rabbit@rabbit2,rabbit@rabbit1,rabbit@rabbit3]}]
# => ...done.

## Forcing Node Boot in Case of Unavailable Peers

In some cases the last node to go
offline cannot be brought back up. It can be removed from the
cluster using the `forget_cluster_node` [rabbitmqctl](./cli.html) command.

Alternatively `force_boot` [rabbitmqctl](./cli.html) command can be used
on a node to make it boot without trying to sync with any
peers (as if they were last to shut down). This is
usually only necessary if the last node to shut down or a
set of nodes will never be brought back online.

## Breaking Up a Cluster

Sometimes it is necessary to remove a node from a
cluster. The operator has to do this explicitly using a
`rabbitmqctl` command.

Some [peer discovery mechanisms](./cluster-formation.html)
support node health checks and forced
removal of nodes not known to the discovery backend. That feature is
opt-in (deactivated by default).

We first remove `rabbit@rabbit3` from the cluster, returning it to
independent operation. To do that, on `rabbit@rabbit3` we
stop the RabbitMQ application, reset the node, and restart the
RabbitMQ application.

# on rabbit3
rabbitmqctl stop_app
# => Stopping node rabbit@rabbit3 ...done.

rabbitmqctl reset
# => Resetting node rabbit@rabbit3 ...done.
rabbitmqctl start_app
# => Starting node rabbit@rabbit3 ...done.

Note that it would have been equally valid to list
`rabbit@rabbit3` as a node.

Running the cluster_status command on the nodes confirms
that `rabbit@rabbit3` now is no longer part of
the cluster and operates independently:

# on rabbit1
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit1 ...
# => [{nodes,[{disc,[rabbit@rabbit1,rabbit@rabbit2]}]},
# => {running_nodes,[rabbit@rabbit2,rabbit@rabbit1]}]
# => ...done.

# on rabbit2
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit2 ...
# => [{nodes,[{disc,[rabbit@rabbit1,rabbit@rabbit2]}]},
# =>  {running_nodes,[rabbit@rabbit1,rabbit@rabbit2]}]
# => ...done.

# on rabbit3
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit3 ...
# => [{nodes,[{disc,[rabbit@rabbit3]}]},{running_nodes,[rabbit@rabbit3]}]
# => ...done.

We can also remove nodes remotely. This is useful, for example, when
having to deal with an unresponsive node. We can for example remove
`rabbit@rabbit1` from `rabbit@rabbit2`.

# on rabbit1
rabbitmqctl stop_app
# => Stopping node rabbit@rabbit1 ...done.

# on rabbit2
rabbitmqctl forget_cluster_node rabbit@rabbit1
# => Removing node rabbit@rabbit1 from cluster ...
# => ...done.

Note that `rabbit1` still thinks it's clustered with
`rabbit2`, and trying to start it will result in an
error. We will need to reset it to be able to start it again.

# on rabbit1
rabbitmqctl start_app
# => Starting node rabbit@rabbit1 ...
# => Error: inconsistent_cluster: Node rabbit@rabbit1 thinks it's clustered with node rabbit@rabbit2, but rabbit@rabbit2 disagrees

rabbitmqctl reset
# => Resetting node rabbit@rabbit1 ...done.

rabbitmqctl start_app
# => Starting node rabbit@rabbit1 ...
# => ...done.

The `cluster_status` command now shows all three nodes
operating as independent RabbitMQ brokers:

# on rabbit1
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit1 ...
# => [{nodes,[{disc,[rabbit@rabbit1]}]},{running_nodes,[rabbit@rabbit1]}]
# => ...done.

# on rabbit2
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit2 ...
# => [{nodes,[{disc,[rabbit@rabbit2]}]},{running_nodes,[rabbit@rabbit2]}]
# => ...done.

# on rabbit3
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit3 ...
# => [{nodes,[{disc,[rabbit@rabbit3]}]},{running_nodes,[rabbit@rabbit3]}]
# => ...done.

Note that `rabbit@rabbit2` retains the residual
state of the cluster, whereas `rabbit@rabbit1`
and `rabbit@rabbit3` are freshly initialised
RabbitMQ brokers. If we want to re-initialise
`rabbit@rabbit2` we follow the same steps as
for the other nodes:

# on rabbit2
rabbitmqctl stop_app
# => Stopping node rabbit@rabbit2 ...done.
rabbitmqctl reset
# => Resetting node rabbit@rabbit2 ...done.
rabbitmqctl start_app
# => Starting node rabbit@rabbit2 ...done.

Besides `rabbitmqctl forget_cluster_node` and the automatic cleanup of unknown nodes
by some [peer discovery](./cluster-formation.html) plugins, there are no scenarios
in which a RabbitMQ node will permanently remove its peer node from a cluster.

### How to Reset a Node

Sometimes it may be necessary to reset a node (wipe all of its data) and later make it rejoin the cluster.
Generally speaking, there are two possible scenarios: when the node is running, and when the node cannot start
or won't respond to CLI tool commands e.g. due to an issue such as [ERL-430](https://bugs.erlang.org/browse/ERL-430).

Resetting a node will delete all of its data, cluster membership information, configured [runtime parameters](./parameters.html),
users, virtual hosts and any other node data. It will also permanently remove the node from its cluster.

To reset a running and responsive node, first stop RabbitMQ on it using `rabbitmqctl stop_app`
and then reset it using `rabbitmqctl reset`:

# on rabbit1
rabbitmqctl stop_app
# => Stopping node rabbit@rabbit1 ...done.
rabbitmqctl reset
# => Resetting node rabbit@rabbit1 ...done.

In case of a non-responsive node, it must be stopped first using any means necessary.
For nodes that fail to start this is already the case. Then [override](./relocate.html)
the node's data directory location or [re]move the existing data store. This will make the node
start as a blank one. It will have to be instructed to [rejoin its original cluster](#cluster-formation), if any.

A node that's been reset and rejoined its original cluster will sync all virtual hosts, users, permissions
and topology (queues, exchanges, bindings), runtime parameters and policies. [Quorum queue](quorum-queues.html)
contents will be replicated if the node will be selected to host a replica.
Non-replicated queue contents on a reset node will be lost.

## Upgrading clusters
You can find instructions for upgrading a cluster in
[the upgrade guide](./upgrade.html#rabbitmq-cluster-configuration).

## A Cluster on a Single Machine

Under some circumstances it can be useful to run a cluster
of RabbitMQ nodes on a single machine. This would
typically be useful for experimenting with clustering on a
desktop or laptop without the overhead of starting several
virtual machines for the cluster.

In order to run multiple RabbitMQ nodes on a single
machine, it is necessary to make sure the nodes have
distinct node names, data store locations, log file
locations, and bind to different ports, including those
used by plugins. See `RABBITMQ_NODENAME`,
`RABBITMQ_NODE_PORT`, and
`RABBITMQ_DIST_PORT` in the [Configuration
guide](./configure.html#supported-environment-variables), as well as `RABBITMQ_MNESIA_DIR`,
`RABBITMQ_CONFIG_FILE`, and
`RABBITMQ_LOG_BASE` in the [File and Directory Locations guide](./relocate.html).

You can start multiple nodes on the same host manually by
repeated invocation of `rabbitmq-server` (
`rabbitmq-server.bat` on Windows). For example:

RABBITMQ_NODE_PORT=5672 RABBITMQ_NODENAME=rabbit rabbitmq-server -detached
RABBITMQ_NODE_PORT=5673 RABBITMQ_NODENAME=hare rabbitmq-server -detached
rabbitmqctl -n hare stop_app
rabbitmqctl -n hare join_cluster rabbit@`hostname -s`
rabbitmqctl -n hare start_app

will set up a two node cluster, both nodes as disc nodes.
Note that if the node [listens on any ports](networking.html) other
than AMQP 0-9-1 and AMQP 1.0 ones, those must be configured to avoid a collision as
well. This can be done via command line:

RABBITMQ_NODE_PORT=5672 RABBITMQ_SERVER_START_ARGS="-rabbitmq_management listener [{port,15672}]" RABBITMQ_NODENAME=rabbit rabbitmq-server -detached
RABBITMQ_NODE_PORT=5673 RABBITMQ_SERVER_START_ARGS="-rabbitmq_management listener [{port,15673}]" RABBITMQ_NODENAME=hare rabbitmq-server -detached

will start two nodes (which can then be clustered) when
the management plugin is installed.

## Hostname Changes

RabbitMQ nodes use hostnames to communicate with each other. Therefore,
all node names must be able to resolve names of all cluster peers. This is
also true for tools such as `rabbitmqctl`.

In addition to that, by default RabbitMQ names the database directory using the
current hostname of the system. If the hostname
changes, a new empty database is created. To avoid data loss it's
crucial to set up a fixed and resolvable hostname.

Whenever the hostname changes RabbitMQ node must be restarted.

A similar effect can be achieved by using `rabbit@localhost`
as the broker nodename.
The impact of this solution is that clustering will not work because
the chosen hostname does not resolve to a routable address from the remote
hosts. The `rabbitmqctl` command fails when
invoked from a remote host. A better solution is to use DNS, for example,
[Amazon Route 53](http://aws.amazon.com/route53/) if running
on EC2. If you want to use the full hostname for your nodename (RabbitMQ
defaults to the short name), and that full hostname is resolvable using DNS,
you may want to investigate setting the environment variable
`RABBITMQ_USE_LONGNAME=true`.

See the section on [hostname resolution](./clustering.html#hostname-resolution-requirement) for more information.

## Firewalled Nodes

Nodes can have a firewall enabled on them. In such case, traffic on certain ports must be
allowed by the firewall in both directions, or nodes won't be able to join each other and
perform all the operations they expect to be available on cluster peers.

Learn more in the [section on ports](#ports) above and dedicated [RabbitMQ Networking guide](networking.html).

## Erlang Versions Across the Cluster

All nodes in a cluster are *highly recommended* to run the same major [version of Erlang](./which-erlang.html): `22.2.0`
and `22.2.8` can be mixed but `21.3.6` and `22.2.6` can potentially introduce breaking changes in
inter-node communication protocols. While such breaking changes are relatively rare, they are possible.

Incompatibilities between patch releases of Erlang/OTP versions
are very rare.

## Connecting to Clusters from Clients

A client can connect as normal to any node within a
cluster. If that node should fail, and the rest of the
cluster survives, then the client should notice the closed
connection, and should be able to reconnect to some
surviving member of the cluster.

Many clients support lists of hostnames that will be tried in order
at connection time.

Generally it is not recommended to hardcode IP addresses into
client applications: this introduces inflexibility and will
require client applications to be edited, recompiled and
redeployed should the configuration of the cluster change or
the number of nodes in the cluster change.

Instead, consider a more abstracted approach: this could be a
dynamic DNS service which has a very short TTL
configuration, or a plain TCP load balancer, or a combination of them.

In general, this aspect of managing the
connection to nodes within a cluster is beyond the scope of
this guide, and we recommend the use of other
technologies designed specifically to address these problems.

## Disk and RAM Nodes

A node can be a disk node or a RAM node.
(Note: disk and disc are used
interchangeably). RAM nodes store internal database tables
in RAM only. This does not include messages, message store
indices, queue indices and other node state.

In the vast majority of cases you want all your nodes to be
disk nodes; RAM nodes are a special case that can be used
to improve the performance clusters with high queue,
exchange, or binding churn. RAM nodes do not provide
higher message rates. When in doubt, use
disk nodes only.

Since RAM nodes store internal database tables in RAM only, they must sync
them from a peer node on startup. This means that a cluster must contain
at least one disk node. It is therefore not possible to manually remove
the last remaining disk node in a cluster.

## Clusters with RAM nodes

RAM nodes keep their metadata only in memory. As RAM nodes
don't have to write to disc as much as disc nodes, they can
perform better. However, note that since persistent queue
data is always stored on disc, the performance improvements
will affect only resource management (e.g. adding/removing
queues, exchanges, or vhosts), but not publishing or
consuming speed.

RAM nodes are an advanced use case; when setting up your
first cluster you should simply not use them. You should
have enough disc nodes to handle your redundancy
requirements, then if necessary add additional RAM nodes for
scale.

A cluster containing only RAM nodes would be too volatile; if the
cluster stops you will not be able to start it again and
**will lose all data**. RabbitMQ will prevent the creation of a
RAM-node-only cluster in many situations, but it can't
absolutely prevent it.

The examples here show a cluster with one disc and one RAM
node for simplicity only; such a cluster is a poor design
choice.

### Creating RAM nodes

We can declare a node as a RAM node when it first joins
the cluster. We do this with
`rabbitmqctl join_cluster` as before, but passing the
`--ram` flag:

# on rabbit2
rabbitmqctl stop_app
# => Stopping node rabbit@rabbit2 ...done.

rabbitmqctl join_cluster --ram rabbit@rabbit1
# => Clustering node rabbit@rabbit2 with [rabbit@rabbit1] ...done.

rabbitmqctl start_app
# => Starting node rabbit@rabbit2 ...done.

RAM nodes are shown as such in the cluster status:

# on rabbit1
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit1 ...
# => [{nodes,[{disc,[rabbit@rabbit1]},{ram,[rabbit@rabbit2]}]},
# =>  {running_nodes,[rabbit@rabbit2,rabbit@rabbit1]}]
# => ...done.

# on rabbit2
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit2 ...
# => [{nodes,[{disc,[rabbit@rabbit1]},{ram,[rabbit@rabbit2]}]},
# =>  {running_nodes,[rabbit@rabbit1,rabbit@rabbit2]}]
# => ...done.

### Changing node types

We can change the type of a node from ram to disc and vice
versa. Say we wanted to reverse the types of
`rabbit@rabbit2` and `rabbit@rabbit1`, turning
the former from a ram node into a disc node and the latter from a
disc node into a ram node. To do that we can use the
`change_cluster_node_type` command. The node must be
stopped first.

# on rabbit2
rabbitmqctl stop_app
# => Stopping node rabbit@rabbit2 ...done.

rabbitmqctl change_cluster_node_type disc
# => Turning rabbit@rabbit2 into a disc node ...done.

rabbitmqctl start_app
# => Starting node rabbit@rabbit2 ...done.

# on rabbit1
rabbitmqctl stop_app
# => Stopping node rabbit@rabbit1 ...done.

rabbitmqctl change_cluster_node_type ram
# => Turning rabbit@rabbit1 into a ram node ...done.

rabbitmqctl start_app
# => Starting node rabbit@rabbit1 ...done.

redpajama_set_name: RedPajamaGithub


text:
package org.opentosca.bus.management.deployment.plugin.remote;

import java.net.URI;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;

import javax.inject.Inject;

import org.apache.camel.Exchange;
import org.apache.camel.Message;
import org.opentosca.bus.management.deployment.plugin.IManagementBusDeploymentPluginService;
import org.opentosca.bus.management.header.MBHeader;
import org.opentosca.bus.management.service.impl.Constants;
import org.opentosca.bus.management.service.impl.collaboration.RequestSender;
import org.opentosca.bus.management.service.impl.collaboration.model.CollaborationMessage;
import org.opentosca.bus.management.service.impl.collaboration.model.KeyValueMap;
import org.opentosca.bus.management.service.impl.collaboration.model.RemoteOperations;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;

@Component
public class ManagementBusDeploymentPluginRemote implements IManagementBusDeploymentPluginService {

    static final private Logger LOG = LoggerFactory.getLogger(ManagementBusDeploymentPluginRemote.class);

    private final RequestSender requestSender;

    @Inject
    public ManagementBusDeploymentPluginRemote(RequestSender requestSender) {
        this.requestSender = requestSender;
    }

    @Override
    public Exchange invokeImplementationArtifactDeployment(final Exchange exchange) {
        LOG.debug("Trying to deploy IA on remote OpenTOSCA Container.");
        final Message message = exchange.getIn();

        // create empty request message (only headers needed)
        final CollaborationMessage requestBody = new CollaborationMessage(new KeyValueMap(), null);
        // perform remote deployment
        final Exchange response = requestSender.sendRequestToRemoteContainer(message, RemoteOperations.INVOKE_IA_DEPLOYMENT, requestBody, 0);

        // extract the endpoint URI from the response
        final URI endpointURI = response.getIn().getHeader(MBHeader.ENDPOINT_URI.toString(), URI.class);
        LOG.debug("Result of remote deployment: Endpoint URI: {}", endpointURI);

        // add the header to the incoming exchange and return result
        message.setHeader(MBHeader.ENDPOINT_URI.toString(), endpointURI);
        return exchange;
    }

    @Override
    public Exchange invokeImplementationArtifactUndeployment(final Exchange exchange) {
        LOG.debug("Trying to undeploy IA on remote OpenTOSCA Container.");
        final Message message = exchange.getIn();

        // create empty request message (only headers needed)
        final CollaborationMessage requestBody = new CollaborationMessage(new KeyValueMap(), null);
        // perform remote undeployment
        final Exchange response = requestSender.sendRequestToRemoteContainer(message, RemoteOperations.INVOKE_IA_UNDEPLOYMENT, requestBody,
            0);

        // extract the undeployment state from the response
        final boolean state = response.getIn().getHeader(MBHeader.OPERATIONSTATE_BOOLEAN.toString(), boolean.class);
        LOG.debug("Result of remote undeployment: Success: {}", state);

        // add the header to the incoming exchange and return result
        message.setHeader(MBHeader.OPERATIONSTATE_BOOLEAN.toString(), state);
        return exchange;
    }

    @Override
    /**
     * {@inheritDoc}
     */
    public List<String> getSupportedTypes() {
        // This plug-in supports only the special type 'remote' which is used to forward deployment
        // requests to other OpenTOSCA Containers.
        return Collections.singletonList(Constants.REMOTE_TYPE);
    }

    @Override
    /**
     * {@inheritDoc}
     */
    public List<String> getCapabilties() {
        // This plug-in is intended to move deployment requests from one OpenTOSCA Container to
        // another one. At the destination OpenTOSCA Container the deployment is done by one of the
        // other available deployment plug-ins. Therefore, it has to be checked if this other
        // deployment plug-in provides all needed capabilities before moving the request to the
        // other Container. So, as this plug-in is only a redirection it does not provide any
        // capabilities.
        return new ArrayList<>();
    }
}


text:
package com.helospark.FakeSsh.cipher;

import java.security.NoSuchAlgorithmException;
import java.security.spec.AlgorithmParameterSpec;

import javax.crypto.Cipher;
import javax.crypto.SecretKey;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.SecretKeySpec;

public abstract class AbstractJavaInterfaceBasedCipherFactory implements CipherFactory {

	protected SshCipher createJavaInterfaceBasedCipher(String cipherAlgorithm, String algorithm, byte[] key, byte[] iv) {
		try {
			assertKeySizeIsLessThanMaximum(cipherAlgorithm, key.length);
			Cipher encryptCipher = createCipher(cipherAlgorithm, algorithm, Cipher.ENCRYPT_MODE, key, iv);
			Cipher decryptCipher = createCipher(cipherAlgorithm, algorithm, Cipher.DECRYPT_MODE, key, iv);
			return new JavaInterfaceBasedCipher(encryptCipher, decryptCipher);
		} catch (Exception e) {
			throw new RuntimeException("Unable to create cipher", e);
		}
	}

	private void assertKeySizeIsLessThanMaximum(String cipherAlgorithm, int length) throws NoSuchAlgorithmException {
		if (length > Cipher.getMaxAllowedKeyLength(cipherAlgorithm)) {
			throw new RuntimeException("Key length is too large, probably because of jurisdiction policy file limits");
		}
	}

	public Cipher createCipher(String name, String algorithm, int mode, byte[] key, byte[] iv) throws Exception {
		Cipher encryptCipher = Cipher.getInstance(name);
		SecretKey keyValue = new SecretKeySpec(key, algorithm);
		AlgorithmParameterSpec IVspec = new IvParameterSpec(iv);
		encryptCipher.init(mode, keyValue, IVspec);
		return encryptCipher;
	}

}


text:
module Nimbu
  module Endpoints
    class Themes::Snippets < Endpoint
      def create(*args)
        arguments(args, required: [:theme_id])

        forced = arguments.params.delete("force")
        query_params = {}
        query_params = { force: forced } unless forced.nil?

        post_request("/themes/#{theme_id}/snippets", arguments.params, params: query_params)
      end

      def get(*args)
        arguments(args, required: [:theme_id, :snippet_id])

        get_request("/themes/#{theme_id}/snippets/#{snippet_id}", arguments.params)
      end

      def delete(*args)
        arguments(args, required: [:theme_id, :snippet_id])

        delete_request("/themes/#{theme_id}/snippets/#{snippet_id}", arguments.params)
      end
      alias_method :remove, :delete
    end # Themes::Snippets
  end # Endpoints
end # Nimbu


text:
package org.killbill.billing.account.api.user;

import java.util.UUID;

import org.joda.time.DateTimeZone;
import org.killbill.billing.account.AccountTestSuiteNoDB;
import org.killbill.billing.account.api.AccountData;
import org.killbill.billing.account.api.AccountEmail;
import org.killbill.billing.account.api.DefaultAccount;
import org.killbill.billing.account.api.DefaultAccountEmail;
import org.killbill.billing.account.dao.AccountDao;
import org.killbill.billing.account.dao.AccountModelDao;
import org.killbill.billing.account.dao.MockAccountDao;
import org.killbill.billing.callcontext.InternalTenantContext;
import org.killbill.billing.catalog.api.Currency;
import org.killbill.billing.util.callcontext.CallContext;
import org.killbill.billing.util.callcontext.CallContextFactory;
import org.killbill.billing.util.callcontext.InternalCallContextFactory;
import org.killbill.bus.api.PersistentBus;
import org.mockito.Mockito;
import org.testng.Assert;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;

public class TestDefaultAccountUserApiWithMocks extends AccountTestSuiteNoDB {

    private final CallContextFactory factory = Mockito.mock(CallContextFactory.class);
    private final InternalCallContextFactory internalFactory = Mockito.mock(InternalCallContextFactory.class);
    private final CallContext callContext = Mockito.mock(CallContext.class);
    private final InternalTenantContext tenantContext = Mockito.mock(InternalTenantContext.class);

    private AccountDao accountDao;
    private DefaultAccountUserApi accountUserApi;

    @BeforeMethod(groups = "fast")
    public void setUp() throws Exception {
        accountDao = new MockAccountDao(Mockito.mock(PersistentBus.class));
        accountUserApi = new DefaultAccountUserApi(internalFactory, accountDao);
    }

    @Test(groups = "fast", description = "Test Account create API")
    public void testCreateAccount() throws Exception {
        final UUID id = UUID.randomUUID();
        final String externalKey = UUID.randomUUID().toString();
        final String email = UUID.randomUUID().toString();
        final String name = UUID.randomUUID().toString();
        final Integer firstNameLength = Integer.MAX_VALUE;
        final Currency currency = Currency.BRL;
        final Integer billCycleDay = Integer.MAX_VALUE;
        final UUID paymentMethodId = UUID.randomUUID();
        final DateTimeZone timeZone = DateTimeZone.UTC;
        final String locale = UUID.randomUUID().toString();
        final String address1 = UUID.randomUUID().toString();
        final String address2 = UUID.randomUUID().toString();
        final String companyName = UUID.randomUUID().toString();
        final String city = UUID.randomUUID().toString();
        final String stateOrProvince = UUID.randomUUID().toString();
        final String country = UUID.randomUUID().toString();
        final String postalCode = UUID.randomUUID().toString();
        final String phone = UUID.randomUUID().toString();
        final Boolean isMigrated = true;
        final Boolean isNotifiedForInvoices = false;
        final AccountData data = new DefaultAccount(id, externalKey, email, name, firstNameLength, currency, billCycleDay,
                                                    paymentMethodId, timeZone, locale, address1, address2, companyName,
                                                    city, stateOrProvince, country, postalCode, phone, isMigrated, isNotifiedForInvoices);

        accountUserApi.createAccount(data, callContext);

        final AccountModelDao account = accountDao.getAccountByKey(externalKey, tenantContext);
        Assert.assertEquals(account.getExternalKey(), externalKey);
        Assert.assertEquals(account.getEmail(), email);
        Assert.assertEquals(account.getName(), name);
        Assert.assertEquals(account.getFirstNameLength(), firstNameLength);
        Assert.assertEquals(account.getCurrency(), currency);
        Assert.assertEquals(account.getBillingCycleDayLocal(), (Integer) billCycleDay);
        Assert.assertEquals(account.getPaymentMethodId(), paymentMethodId);
        Assert.assertEquals(account.getTimeZone(), timeZone);
        Assert.assertEquals(account.getLocale(), locale);
        Assert.assertEquals(account.getAddress1(), address1);
        Assert.assertEquals(account.getAddress2(), address2);
        Assert.assertEquals(account.getCompanyName(), companyName);
        Assert.assertEquals(account.getCity(), city);
        Assert.assertEquals(account.getStateOrProvince(), stateOrProvince);
        Assert.assertEquals(account.getCountry(), country);
        Assert.assertEquals(account.getPostalCode(), postalCode);
        Assert.assertEquals(account.getPhone(), phone);
        Assert.assertEquals(account.getMigrated(), isMigrated);
        Assert.assertEquals(account.getIsNotifiedForInvoices(), isNotifiedForInvoices);
    }

    @Test(groups = "fast", description = "Test ability to add email to Account")
    public void testAddEmail() throws Exception {
        final UUID accountId = UUID.randomUUID();

        // Verify the initial state
        Assert.assertEquals(accountDao.getEmailsByAccountId(accountId, tenantContext).size(), 0);

        // Add the first email
        final String emailAddress1 = UUID.randomUUID().toString();
        final AccountEmail email1 = new DefaultAccountEmail(accountId, emailAddress1);
        accountUserApi.addEmail(accountId, email1, callContext);
        Assert.assertEquals(accountDao.getEmailsByAccountId(accountId, tenantContext).size(), 1);

        // Add a second one
        final String emailAddress2 = UUID.randomUUID().toString();
        final AccountEmail email2 = new DefaultAccountEmail(accountId, emailAddress2);
        accountUserApi.addEmail(accountId, email2, callContext);
        Assert.assertEquals(accountDao.getEmailsByAccountId(accountId, tenantContext).size(), 2);

        // Remove the first second one
        accountUserApi.removeEmail(accountId, email1, callContext);
        Assert.assertEquals(accountDao.getEmailsByAccountId(accountId, tenantContext).size(), 1);
    }
}


text:
(function() {
  'use strict';

  var express = require('express'),
      app     = express(),

      POST_CustomerRegistration = require('../apImpl/admin/customerRegistration/postIndex.js'),
      GET_CustomerRegistration  = require('../apImpl/admin/customerRegistration/getIndex.js');

    app.route('/customerRegistration')
      .get(GET_CustomerRegistration.getOne)
      .post(POST_CustomerRegistration.post);

    module.exports = app;
}());


text:
define([
    'backbone',
    'marionette',

    'hbs!templates/header'
],function(
    Backbone,
    Marionette,

    template
){

    return Backbone.Marionette.ItemView.extend({
        template:template,
    });

});


text:
LdcEventListenerConfig
=============

---
[![Latest Stable Version](https://poser.pugx.org/adamlundrigan/ldc-event-listener-config/v/stable.svg)](https://packagist.org/packages/adamlundrigan/ldc-event-listener-config) [![Total Downloads](https://poser.pugx.org/adamlundrigan/ldc-event-listener-config/downloads.svg)](https://packagist.org/packages/adamlundrigan/ldc-event-listener-config) [![Latest Unstable Version](https://poser.pugx.org/adamlundrigan/ldc-event-listener-config/v/unstable.svg)](https://packagist.org/packages/adamlundrigan/ldc-event-listener-config) [![License](https://poser.pugx.org/adamlundrigan/ldc-event-listener-config/license.svg)](https://packagist.org/packages/adamlundrigan/ldc-event-listener-config)
[![Build Status](https://travis-ci.org/adamlundrigan/LdcEventListenerConfig.svg?branch=master)](https://travis-ci.org/adamlundrigan/LdcEventListenerConfig)
[![Code Coverage](https://scrutinizer-ci.com/g/adamlundrigan/LdcEventListenerConfig/badges/coverage.png?b=master)](https://scrutinizer-ci.com/g/adamlundrigan/LdcEventListenerConfig/?branch=master)
[![Scrutinizer Code Quality](https://scrutinizer-ci.com/g/adamlundrigan/LdcEventListenerConfig/badges/quality-score.png?b=master)](https://scrutinizer-ci.com/g/adamlundrigan/LdcEventListenerConfig/?branch=master)

---

## What?

LdcEventListenerConfig provides an easy mechanism for attaching listeners to the event manager via configuration

## How?

1. Install the [Composer](https://getcomposer.org/) package:

    ```
    composer require adamlundrigan/ldc-event-listener-config:1.*@stable
    ```

2. Enable the module (`LdcEventListenerConfig`) in your ZF2 application.

3. Configure your event listeners ([example](demo/ExampleModule/Module.php#L49))

4. Profit!

## Show me!

If you're fortunate enough to be on a *nix system with PHP >=5.4, pop into the `demo` folder and run the setup script (`run.sh`).  This will build the demo application, install the example modules, and start a webserver.  Once that's all done just open your browser and navigate to `http://localhost:8080/` to see the result!


text:
var Some = require('./option').Some;
var None = require('./option').None;

function Iter(values) {
	this.values = values !== undefined ? values : [];
	this._count = this.values && this.values.length !== undefined ? this.values.length : 0;
	this._curr = 0;
}

Iter.prototype.next = function() {
	var val = this.values[_curr++];
	return val !== undefined ? Some(val) : None;
}

Iter.prototype.count = function() {
	return _count;
}

Iter.prototype.last = function() {
	var val = this.values[_count-1];
	return val !== undefined ? Some(val) : None;
}

Iter.prototype.nth = function(n) {
	var val = this.values[n];
	return val !== undefined ? Some(val) : None;
}

Iter.prototype.chain = function(other) {
	return new Iter(this.values.concat(other.values));
}

exports.Iter = Iter;


text:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">

<title>org.buzzrobotics.autonomous Class Hierarchy</title>
<meta name="date" content="2012-02-11">
<link rel="stylesheet" type="text/css" href="../../../stylesheet.css" title="Style">

JavaScript is disabled on your browser.

Overview

Package

Class

Tree

Deprecated

Index

Help

Prev

Next

Frames

No Frames

All Classes

Hierarchy For Package org.buzzrobotics.autonomous

Package Hierarchies:

All Packages

Class Hierarchy

java.lang.Object

org.buzzrobotics.autonomous.AutonConfig

edu.wpi.first.wpilibj.command.Command (implements edu.wpi.first.wpilibj.smartdashboard.SmartDashboardNamedData)

edu.wpi.first.wpilibj.command.CommandGroup

org.buzzrobotics.autonomous.AutoMode1

org.buzzrobotics.autonomous.AutoMode2

org.buzzrobotics.autonomous.AutoMode3

org.buzzrobotics.autonomous.AutoMode4

org.buzzrobotics.autonomous.AutoMode5

Overview

Package

Class

Tree

Deprecated

Index

Help

Prev

Next

Frames

No Frames

All Classes


text:
module.exports = {
    after : function(browser) {
        process.exit();
    },

    'Fill Timesheet' : function(browser) {
        var testComment = 'Test comment bla';
        var subComment = 'Test sub task';
        browser
            .init()
            .waitForElementVisible('.logo', 2000)
            // Check only one day
            .waitForElementVisible('.table tbody tr:nth-child(2) td:nth-child(2)', 3000)
            .clearValue('.table tbody tr:nth-child(1) td:nth-child(3) input')
            .setValue('.table tbody tr:nth-child(1) td:nth-child(3) input', '8')
            .clearValue('.table tbody tr:nth-child(1) td:nth-child(4) input')
            .setValue('.table tbody tr:nth-child(1) td:nth-child(4) input', testComment)
            .pause(2500)
            //go to report
            .click('.tabs-left li:nth-child(5)')
            .pause(1000)
            .assert.containsText('.ui-grid-canvas .ui-grid-row:nth-child(1)', testComment)
            .assert.containsText('.ui-grid-canvas .ui-grid-row:nth-child(1)', '8')
            //add sub task
            .click('.tabs-left li:nth-child(6)')
            .waitForElementVisible('.table tbody tr:nth-child(1) .add-timesheet', 1000)
            .click('.table tbody tr:nth-child(1) .add-timesheet')
            .clearValue('.table tbody tr:nth-child(2) td:nth-child(3) input')
            .setValue('.table tbody tr:nth-child(2) td:nth-child(3) input', '3')
            .clearValue('.table tbody tr:nth-child(2) td:nth-child(4) input')
            .setValue('.table tbody tr:nth-child(2) td:nth-child(4) input', subComment)
            .pause(2500)
            .click('.tabs-left li:nth-child(5)')
            .waitForElementVisible('.date-range-picker input', 1000)
            .pause(1000)
            .assert.containsText('.ui-grid-canvas .ui-grid-row:nth-child(2) .ui-grid-cell:nth-child(5)', subComment)
            .assert.containsText('.ui-grid-canvas .ui-grid-row:nth-child(2) .ui-grid-cell:nth-child(4)', '3')
            .end();
    },

    'Project creation': function(browser) {
        browser
            .init()
            .waitForElementVisible('.logo', 2000)
            .pause(1000)
            .click('.tabs-left li:nth-child(2)')
            .waitForElementVisible('.add-project', 2000)
            .click('.add-project')
            .pause(500)
            .click('#step1 tbody tr:nth-child(1) #simple-dropdown')
            .click('#step1 tbody tr:nth-child(1) td:nth-child(1) ul li a')
            .setValue('#step1 tbody tr:nth-child(1) td:nth-child(2) input', '5')
            //check new assignment on timesheet page
            .click('.tabs-left li:nth-child(6)')
            .waitForElementVisible('.main-container', 1000)
            .pause(1000)
            .assert.containsText('.main-container', 'New Project')
            .end();
    },

    'Check employee page': function(browser) {
        var email = 'And@red.net';
        browser
            .init()
            .waitForElementVisible('.logo', 2000)
            .pause(1000)
            //invite new person
            .click('.tabs-left li:nth-child(1)')
            .waitForElementVisible('#step1', 1000)
            .setValue('#step2', email)
            .click('#step3')
            .pause(500)
            .assert.containsText('#step4', email)
            //assign new user to project
            .click('.tabs-left li:nth-child(2)')
            .waitForElementVisible('.add-project', 1000)
            .click('.add-project')
            .pause(500)
            .click('#step1 tbody tr:nth-child(1) #simple-dropdown')
            .click('#step1 tbody tr:nth-child(1) td:nth-child(1) ul li a')
            .setValue('#step1 tbody tr:nth-child(1) td:nth-child(2) input', '6')
            //check employee page
            .click('.tabs-left li:nth-child(3)')
            .waitForElementVisible('.main-container form:first-of-type', 1000)
            .setValue('.main-container form p:first-of-type input', email)
            .click('.main-container form button.primary-button')
            .assert.containsText('.main-container section:first-of-type', email)
            .assert.containsText('.main-container section:first-of-type', 'New Project')
            .assert.containsText('.main-container section:first-of-type .employee-assignment .employee-workload', '6')
            .end();
    },
    'Check company page': function(browser) {
        // New user invitation is checked in previous  test
        var newCompanyName = 'Tralala';
        browser
            .init()
            .waitForElementVisible('.logo', 2000)
            .pause(1000)
            .click('.tabs-left li:nth-child(1)')
            .waitForElementVisible('#step1', 1000)
            .clearValue('#step1')
            .setValue('#step1', newCompanyName)
            .pause(900)
            .assert.containsText('.logo', newCompanyName)
            .end();
    },

    'Check report page': function(browser) {
        browser
            .init()
            .waitForElementVisible('.logo', 2000)
            .pause(1000)
            .click('.tabs-left li:nth-child(5)')
            .pause(1000)
            //Check Project report
            .click('.aggregation-options li:nth-child(2) a')
            .pause(500)
            .assert.containsText('.ui-grid-render-container', 'Mifort-Test')
            .assert.containsText('.ui-grid-render-container', '11')
            //Check employee report
            .click('.aggregation-options li:nth-child(3) a')
            .pause(500)
            .assert.containsText('.ui-grid-render-container', 'Test User')
            .assert.containsText('.ui-grid-render-container', '11')
            //Check Employee + Project Page
            .click('.aggregation-options li:nth-child(4) a')
            .pause(500)
            .assert.containsText('.ui-grid-render-container', 'Test User')
            .assert.containsText('.ui-grid-render-container', 'Mifort-Test')
            .assert.containsText('.ui-grid-render-container', '11')
            .end();
    }
};


text:
import json
import logging
from unittest import TestCase

from parameterized import parameterized, param

from hvac.exceptions import ParamValidationError
from tests.utils import vault_version_lt
from tests.utils.hvac_integration_test_case import HvacIntegrationTestCase

class TestAws(HvacIntegrationTestCase, TestCase):
    TEST_MOUNT_POINT = "aws-test"
    TEST_ROLE_NAME = "hvac-test-role"
    TEST_POLICY_DOCUMENT = {
        "Statement": [
            {"Action": "ec2:Describe*", "Effect": "Allow", "Resource": "*"},
        ],
        "Version": "2012-10-17",
    }

    def setUp(self):
        super(TestAws, self).setUp()
        if "%s/" % self.TEST_MOUNT_POINT not in self.client.sys.list_auth_methods():
            self.client.sys.enable_secrets_engine(
                backend_type="aws",
                path=self.TEST_MOUNT_POINT,
            )

    def tearDown(self):
        self.client.sys.disable_secrets_engine(
            path=self.TEST_MOUNT_POINT,
        )
        super(TestAws, self).tearDown()

    @parameterized.expand(
        [
            param(
                "success",
            ),
        ]
    )
    def test_configure_root_iam_credentials(
        self, label, credentials="", raises=None, exception_message=""
    ):
        access_key = "butts"
        secret_key = "secret-butts"
        if raises:
            with self.assertRaises(raises) as cm:
                self.client.secrets.aws.configure_root_iam_credentials(
                    access_key=access_key,
                    secret_key=secret_key,
                    mount_point=self.TEST_MOUNT_POINT,
                )
            self.assertIn(
                member=exception_message,
                container=str(cm.exception),
            )
        else:
            configure_response = self.client.secrets.aws.configure_root_iam_credentials(
                access_key=access_key,
                secret_key=secret_key,
                iam_endpoint="localhost",
                sts_endpoint="localhost",
                mount_point=self.TEST_MOUNT_POINT,
            )
            logging.debug("configure_response: %s" % configure_response)
            self.assertEqual(
                first=bool(configure_response),
                second=True,
            )

    @parameterized.expand(
        [
            param(
                "success",
            ),
        ]
    )
    def test_configure_lease(
        self, label, lease="60s", lease_max="120s", raises=None, exception_message=""
    ):
        if raises:
            with self.assertRaises(raises) as cm:
                self.client.secrets.aws.configure_lease(
                    lease=lease,
                    lease_max=lease_max,
                    mount_point=self.TEST_MOUNT_POINT,
                )
            self.assertIn(
                member=exception_message,
                container=str(cm.exception),
            )
        else:
            configure_response = self.client.secrets.aws.configure_lease(
                lease=lease,
                lease_max=lease_max,
                mount_point=self.TEST_MOUNT_POINT,
            )
            logging.debug("configure_response: %s" % configure_response)
            self.assertEqual(
                first=bool(configure_response),
                second=True,
            )

    @parameterized.expand(
        [
            param(
                "success",
            ),
        ]
    )
    def test_read_lease(
        self,
        label,
        lease="60s",
        lease_max="120s",
        configure_first=True,
        raises=None,
        exception_message="",
    ):
        if configure_first:
            configure_response = self.client.secrets.aws.configure_lease(
                lease=lease,
                lease_max=lease_max,
                mount_point=self.TEST_MOUNT_POINT,
            )
            logging.debug("configure_response: %s" % configure_response)

        if raises:
            with self.assertRaises(raises) as cm:
                self.client.secrets.aws.read_lease_config(
                    mount_point=self.TEST_MOUNT_POINT,
                )
            self.assertIn(
                member=exception_message,
                container=str(cm.exception),
            )
        else:
            read_response = self.client.secrets.aws.read_lease_config(
                mount_point=self.TEST_MOUNT_POINT,
            )
            logging.debug("read_response: %s" % read_response)
            self.assertEqual(
                first=int(lease_max.replace("s", "")),
                second=self.convert_python_ttl_value_to_expected_vault_response(
                    ttl_value=read_response["data"]["lease_max"],
                ),
            )

    @parameterized.expand(
        [
            param(
                "success",
                policy_document={
                    "Statement": [
                        {"Action": "ec2:Describe*", "Effect": "Allow", "Resource": "*"},
                    ],
                    "Version": "2012-10-17",
                },
            ),
            param(
                "with policy_arns",
                policy_arns=["arn:aws:iam::aws:policy/AmazonVPCReadOnlyAccess"],
            ),
            param(
                "assumed_role with policy document",
                policy_document={
                    "Statement": [
                        {"Action": "ec2:Describe*", "Effect": "Allow", "Resource": "*"},
                    ],
                    "Version": "2012-10-17",
                },
                credential_type="assumed_role",
            ),
            param(
                "invalid credential type",
                credential_type="cat",
                raises=ParamValidationError,
                exception_message="invalid credential_type argument provided",
            ),
        ]
    )
    def test_create_or_update_role(
        self,
        label,
        credential_type="iam_user",
        policy_document=None,
        default_sts_ttl=None,
        max_sts_ttl=None,
        role_arns=None,
        policy_arns=None,
        raises=None,
        exception_message="",
    ):
        if raises:
            with self.assertRaises(raises) as cm:
                self.client.secrets.aws.create_or_update_role(
                    name=self.TEST_ROLE_NAME,
                    credential_type=credential_type,
                    policy_document=policy_document,
                    default_sts_ttl=default_sts_ttl,
                    max_sts_ttl=max_sts_ttl,
                    role_arns=role_arns,
                    policy_arns=policy_arns,
                    legacy_params=vault_version_lt("0.11.0"),
                    mount_point=self.TEST_MOUNT_POINT,
                )
            self.assertIn(
                member=exception_message,
                container=str(cm.exception),
            )
        else:
            role_response = self.client.secrets.aws.create_or_update_role(
                name=self.TEST_ROLE_NAME,
                credential_type=credential_type,
                policy_document=policy_document,
                default_sts_ttl=default_sts_ttl,
                max_sts_ttl=max_sts_ttl,
                role_arns=role_arns,
                policy_arns=policy_arns,
                legacy_params=vault_version_lt("0.11.0"),
                mount_point=self.TEST_MOUNT_POINT,
            )
            logging.debug("role_response: %s" % role_response)

            self.assertEqual(
                first=bool(role_response),
                second=True,
            )

    @parameterized.expand(
        [
            param(
                "success",
            ),
        ]
    )
    def test_read_role(
        self, label, configure_first=True, raises=None, exception_message=""
    ):
        if configure_first:
            self.client.secrets.aws.create_or_update_role(
                name=self.TEST_ROLE_NAME,
                credential_type="iam_user",
                policy_document=self.TEST_POLICY_DOCUMENT,
                legacy_params=vault_version_lt("0.11.0"),
                mount_point=self.TEST_MOUNT_POINT,
            )
        if raises:
            with self.assertRaises(raises) as cm:
                self.client.secrets.aws.read_role(
                    name=self.TEST_ROLE_NAME,
                    mount_point=self.TEST_MOUNT_POINT,
                )
            self.assertIn(
                member=exception_message,
                container=str(cm.exception),
            )
        else:
            read_role_response = self.client.secrets.aws.read_role(
                name=self.TEST_ROLE_NAME,
                mount_point=self.TEST_MOUNT_POINT,
            )
            logging.debug("read_role_response: %s" % read_role_response)
            if vault_version_lt("0.11.0"):
                self.assertDictEqual(
                    d1=json.loads(read_role_response["data"]["policy"]),
                    d2=self.TEST_POLICY_DOCUMENT,
                )
            # https://github.com/hashicorp/vault/commit/2dcd0aed2a242f53dae03318b4d68693f7d92b81
            elif vault_version_lt("1.0.2"):
                self.assertEqual(
                    first=read_role_response["data"]["credential_types"],
                    second=["iam_user"],
                )
            else:
                self.assertEqual(
                    first=read_role_response["data"]["credential_type"],
                    second="iam_user",
                )

    @parameterized.expand(
        [
            param(
                "success",
            ),
        ]
    )
    def test_list_roles(
        self, label, configure_first=True, raises=None, exception_message=""
    ):
        if configure_first:
            self.client.secrets.aws.create_or_update_role(
                name=self.TEST_ROLE_NAME,
                credential_type="iam_user",
                policy_document=self.TEST_POLICY_DOCUMENT,
                legacy_params=vault_version_lt("0.11.0"),
                mount_point=self.TEST_MOUNT_POINT,
            )
        if raises:
            with self.assertRaises(raises) as cm:
                self.client.secrets.aws.list_roles(
                    mount_point=self.TEST_MOUNT_POINT,
                )
            self.assertIn(
                member=exception_message,
                container=str(cm.exception),
            )
        else:
            list_roles_response = self.client.secrets.aws.list_roles(
                mount_point=self.TEST_MOUNT_POINT,
            )
            logging.debug("list_roles_response: %s" % list_roles_response)
            self.assertEqual(
                first=list_roles_response["data"]["keys"],
                second=[self.TEST_ROLE_NAME],
            )

    @parameterized.expand(
        [
            param(
                "success",
            ),
        ]
    )
    def test_delete_role(
        self, label, configure_first=True, raises=None, exception_message=""
    ):
        if configure_first:
            self.client.secrets.aws.create_or_update_role(
                name=self.TEST_ROLE_NAME,
                credential_type="iam_user",
                policy_document=self.TEST_POLICY_DOCUMENT,
                legacy_params=vault_version_lt("0.11.0"),
                mount_point=self.TEST_MOUNT_POINT,
            )
        if raises:
            with self.assertRaises(raises) as cm:
                self.client.secrets.aws.delete_role(
                    name=self.TEST_ROLE_NAME,
                    mount_point=self.TEST_MOUNT_POINT,
                )
            self.assertIn(
                member=exception_message,
                container=str(cm.exception),
            )
        else:
            delete_role_response = self.client.secrets.aws.delete_role(
                name=self.TEST_ROLE_NAME,
                mount_point=self.TEST_MOUNT_POINT,
            )
            logging.debug("delete_role_response: %s" % delete_role_response)
            self.assertEqual(
                first=bool(delete_role_response),
                second=True,
            )


text:
package com.mygdx.entity;

import com.badlogic.gdx.graphics.g2d.Animation;
import com.badlogic.gdx.graphics.g2d.TextureRegion;
import com.badlogic.gdx.math.GridPoint2;

import java.util.Hashtable;
import java.util.ArrayList;

public class AnimatedGo extends GameObject {
    private Hashtable<GoState, Animation<TextureRegion>> animTable;
    private float stateTime;

    public AnimatedGo(GridPoint2 origin, Hashtable<GoState, Animation<TextureRegion>> animTable,
                      ArrayList<GoState> allowedStates, GoState startingState) {
        super((TextureRegion)null, origin, allowedStates, startingState);
        this.animTable = animTable;
        this.stateTime = 0;

        /* @TODO: This may screw us if some state does not need an animation */
        for (GoState gs : allowedStates) {
            if (!this.animTable.containsKey(gs))
                throw new RuntimeException("Unmapped state");
        }

        this.tr = this.animTable.get(currentState).getKeyFrame(this.stateTime);
    }

    @Override public void update(float dt) {
        this.stateTime += dt;
        this.tr = this.animTable.get(currentState).getKeyFrame(this.stateTime);
    }
}


text:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.Serialization;
using System.ServiceModel;
using System.ServiceModel.Web;
using System.Text;
using System.Threading;
using System.Windows.Forms;

namespace WcfServiceApp.Implementations
{
    public class Service1 : IService1, IService2, IService3, IService4
    {
        public string GetData1(string value)
        {
            string UserName = Thread.CurrentPrincipal.Identity.Name;
            return string.Format("You entered: {0} in GetData1 of BasicHttpBinding", value);
        }

        public string GetData2(string value)
        {
            string UserName = Thread.CurrentPrincipal.Identity.Name;
            return string.Format("You entered: {0} in GetData2 of BasicHttpsBinding {1}", value, UserName);
        }

        public string GetData3(string value)
        {
            string UserName = Thread.CurrentPrincipal.Identity.Name;
            return string.Format("You entered: {0} in GetData3 of TCPBinding {1}", value, UserName);
        }

        public string GetData4(string value)
        {
            return string.Format("You entered: {0} in GetData4 of WSHttpBinding", value);
        }
    }
}


text:
use std::collections::HashMap;
use std::rc::Rc;
use std::cell::RefCell;
use std::ops::Deref;
use std::fmt;

#[derive (Eq, PartialEq, Clone, Copy, Default, Hash, Debug)]
pub struct InternedStr(usize);

#[derive (Debug, Default)]
struct Interner {
    indices: HashMap<String, usize>,
    strings: Vec<String>,
}

impl Interner {
    fn new() -> Interner {
        Interner {
            indices: HashMap::new(),
            strings: vec![],
        }
    }

    fn intern(&mut self, s: &str) -> InternedStr {
        match self.indices.get(s).cloned() {
            Some(index) => InternedStr(index),
            None => {
                let index = self.strings.len();
                self.indices.insert(s.to_string(), index);
                self.strings.push(s.to_string());
                InternedStr(index)
            }
        }
    }

    fn get_str(&self, InternedStr(i): InternedStr) -> &str {
        if i < self.strings.len() {
            &*self.strings[i]
        } else {
            "--[ error in internedstr state ]---"
        }
    }
}

fn get_local_interner() -> Rc<RefCell<Interner>> {
    thread_local!( static INTERNER: Rc<RefCell<Interner>> = Rc::new(RefCell::new(Interner::new())));
    INTERNER.with(|interner| interner.clone())
}

pub fn intern(s: &str) -> InternedStr {
    let interner = get_local_interner();
    let mut i = interner.borrow_mut();
    i.intern(s)
}

impl Deref for InternedStr {
    type Target = str;
    fn deref(&self) -> &str {
        self.as_ref()
    }
}

impl AsRef<str> for InternedStr {
    fn as_ref(&self) -> &str {
        let interner = get_local_interner();
        let x = (*interner).borrow_mut();
        let r: &str = x.get_str(*self);
        unsafe { ::std::mem::transmute(r) }
    }
}

impl fmt::Display for InternedStr {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        write!(f, "{}", self.as_ref())
    }
}

#[cfg(test)]
mod test {
    use super::*;

    #[test]
    fn simple_intern() {
        let is = intern("foo");

        assert_eq!("foo", is.as_ref());

        let is2 = intern("foo");

        assert_eq!(is, is2);
    }
}


text:
namespace {

struct PacketRecord {
  XFA_PacketType packet_type;
  uint32_t hash;
  uint32_t flags;
  const wchar_t* name;
  const wchar_t* uri;
};

const PacketRecord g_PacketTable[] = {
#undef PCKT____
#define PCKT____(a, b, c, d, e, f)                                          \
  {XFA_PacketType::c, a, XFA_XDPPACKET_FLAGS_##e | XFA_XDPPACKET_FLAGS_##f, \
   L##b, d},
#include "xfa/fxfa/parser/packets.inc"
#undef PCKT____
};

struct ElementRecord {
  uint32_t hash;  // Hashed as wide string.
  XFA_Element element;
  XFA_Element parent;
  const char* name;
};

const ElementRecord g_ElementTable[] = {
#undef ELEM____
#define ELEM____(a, b, c, d) {a, XFA_Element::c, XFA_Element::d, b},
#include "xfa/fxfa/parser/elements.inc"
#undef ELEM____
};

struct AttributeRecord {
  uint32_t hash;  // Hashed as wide string.
  XFA_Attribute attribute;
  XFA_ScriptType script_type;
  const char* name;
};

const AttributeRecord g_AttributeTable[] = {
#undef ATTR____
#define ATTR____(a, b, c, d) {a, XFA_Attribute::c, XFA_ScriptType::d, b},
#include "xfa/fxfa/parser/attributes.inc"
#undef ATTR____
};

struct AttributeValueRecord {
  uint32_t uHash;  // |pName| hashed as WideString.
  XFA_AttributeValue eName;
  const char* pName;
};

const AttributeValueRecord g_AttributeValueTable[] = {
#undef VALUE____
#define VALUE____(a, b, c) {a, XFA_AttributeValue::c, b},
#include "xfa/fxfa/parser/attribute_values.inc"
#undef VALUE____
};

struct ElementAttributeRecord {
  XFA_Element element;
  XFA_Attribute attribute;
  XFA_ATTRIBUTE_CALLBACK callback;
};

const ElementAttributeRecord g_ElementAttributeTable[] = {
#undef ELEM_ATTR____
#define ELEM_ATTR____(a, b, c) {XFA_Element::a, XFA_Attribute::b, c##_static},
#include "xfa/fxfa/parser/element_attributes.inc"
#undef ELEM_ATTR____
};

}  // namespace

XFA_PACKETINFO XFA_GetPacketByIndex(XFA_PacketType ePacket) {
  const PacketRecord* pRecord = &g_PacketTable[static_cast<uint8_t>(ePacket)];
  return {pRecord->name, pRecord->packet_type, pRecord->uri, pRecord->flags};
}

Optional<XFA_PACKETINFO> XFA_GetPacketByName(WideStringView wsName) {
  uint32_t hash = FX_HashCode_GetW(wsName, false);
  auto* elem = std::lower_bound(
      std::begin(g_PacketTable), std::end(g_PacketTable), hash,
      [](const PacketRecord& a, uint32_t hash) { return a.hash < hash; });
  if (elem != std::end(g_PacketTable) && elem->name == wsName)
    return XFA_GetPacketByIndex(elem->packet_type);
  return {};
}

ByteStringView XFA_ElementToName(XFA_Element elem) {
  return g_ElementTable[static_cast<size_t>(elem)].name;
}

XFA_Element XFA_GetElementByName(WideStringView name) {
  uint32_t hash = FX_HashCode_GetW(name, false);
  auto* elem = std::lower_bound(
      std::begin(g_ElementTable), std::end(g_ElementTable), hash,
      [](const ElementRecord& a, uint32_t hash) { return a.hash < hash; });
  if (elem != std::end(g_ElementTable) && name.EqualsASCII(elem->name))
    return elem->element;
  return XFA_Element::Unknown;
}

ByteStringView XFA_AttributeToName(XFA_Attribute attr) {
  return g_AttributeTable[static_cast<size_t>(attr)].name;
}

Optional<XFA_ATTRIBUTEINFO> XFA_GetAttributeByName(WideStringView name) {
  uint32_t hash = FX_HashCode_GetW(name, false);
  auto* elem = std::lower_bound(
      std::begin(g_AttributeTable), std::end(g_AttributeTable), hash,
      [](const AttributeRecord& a, uint32_t hash) { return a.hash < hash; });
  if (elem != std::end(g_AttributeTable) && name.EqualsASCII(elem->name)) {
    XFA_ATTRIBUTEINFO result;
    result.attribute = elem->attribute;
    result.eValueType = elem->script_type;
    return result;
  }
  return {};
}

ByteStringView XFA_AttributeValueToName(XFA_AttributeValue item) {
  return g_AttributeValueTable[static_cast<int32_t>(item)].pName;
}

Optional<XFA_AttributeValue> XFA_GetAttributeValueByName(WideStringView name) {
  auto* it = std::lower_bound(std::begin(g_AttributeValueTable),
                              std::end(g_AttributeValueTable),
                              FX_HashCode_GetW(name, false),
                              [](const AttributeValueRecord& arg,
                                 uint32_t hash) { return arg.uHash < hash; });
  if (it != std::end(g_AttributeValueTable) && name.EqualsASCII(it->pName))
    return it->eName;

  return {};
}

Optional<XFA_SCRIPTATTRIBUTEINFO> XFA_GetScriptAttributeByName(
    XFA_Element element,
    WideStringView attribute_name) {
  Optional<XFA_ATTRIBUTEINFO> attr = XFA_GetAttributeByName(attribute_name);
  if (!attr.has_value())
    return {};

  while (element != XFA_Element::Unknown) {
    auto compound_key = std::make_pair(element, attr.value().attribute);
    auto* it = std::lower_bound(
        std::begin(g_ElementAttributeTable), std::end(g_ElementAttributeTable),
        compound_key,
        [](const ElementAttributeRecord& arg,
           const std::pair<XFA_Element, XFA_Attribute>& key) {
          return std::make_pair(arg.element, arg.attribute) < key;
        });
    if (it != std::end(g_ElementAttributeTable) &&
        compound_key == std::make_pair(it->element, it->attribute)) {
      XFA_SCRIPTATTRIBUTEINFO result;
      result.attribute = attr.value().attribute;
      result.eValueType = attr.value().eValueType;
      result.callback = it->callback;
      return result;
    }
    element = g_ElementTable[static_cast<size_t>(element)].parent;
  }
  return {};
}