Find out how to configure a multi-regional deployment with MongoDB.
A single API Gateway deployment can span multiple regions around the globe to improve regional performance as well as provide a more seamless disaster recovery. To achieve better performance, all messages from a specific region should be brokered by a Network Director in the same region. For disaster recovery, it must be possible to access both the Network Director and Policy Manager in a different region if the local region has suffered a disaster.
A multi-master deployment for the Network Director processes.
A master/slave deployment for the Policy Manager processes.
In each region, of course, the Network Director and Policy Manager processes can be deployed in clusters for scaling.
In the sample scenario in this document, the regional locations to which the API Gateway components are deployed are data centers. The diagram below illustrates the mixture of multi-master Network Directors with master/slave Policy Managers.
Each region has its own data center, with master Network Directors that regional clients interact with. These Network Directors communicate with Policy Manager containers in the same region. Only one data center (Data Center A) has the master Policy Manager. All other data centers have slaves.
The key to a multi-regional software deployment is the speed of access to, and integrity of, persisted data. The API Gateway has traditionally persisted its data in a relational database. Although some relational database vendors claim multi-master capabilities, these solutions have not proven to be 100% reliable. Instead, a more traditional master/slave deployment has proven to be the best option. For this reason, the multi-regional architecture includes a single master Policy Manager.
One data center will host the master RDBMS that the cohosted Policy Manager processes will read and write to.
The RDBMS will utilize replication to propagate changes made through the master Policy Manager to the slave databases in the other data centers.
The Policy Manager processes in data centers with slave databases are slaves themselves, only supporting read activities, no writes.
In case of a disaster that renders the data center hosting the master Policy Manager or Policy Managers inoperable, the fail-over capabilities of the RDBMS are used to promote a different data center's database to be the new master. The Policy Managers in this data center will now become the masters as well.
Using the master/slave RDBMS architecture for writing data would require the Network Directors in all data centers to communicate directly with the single master Policy Manager/RDBMS. This would lead to obvious problems with performance, compounded by the volume of data needing to be written during heavy client traffic periods.
To avoid this limitation the multi-regional architecture introduces a second persistence system, a NoSQL database, for the storage of the data that the Network Directors write. A NoSQL database provides more flexibility than an RDBMS in that it can support sharding.
Sharding allows rows of the same table, or collection, to be stored in different servers in different data centers.
The rows are segmented by a shard key.
Each shard server is the master of rows written with a certain set of keys.
In the Akana architecture, the shard key is the Network Director key.
Each shard can be the master for multiple keys (to support Network Director clusters) but no more than one shard can be the master for a single key.
In the diagram above, each data center has its own shard of a NoSQL data store. For metrics and usage data the Network Director can write directly to the shard or use a RESTful service hosted by the Policy Manager which would write to the NoSQL shard. For the other data, which the Network Director writes, there are services on the Policy Manager that the Network Director calls to insert the data into the shard. Note that even though the Policy Manager processes in data center B are slaves, they can still write to their local shard. This architecture tries to ensure good performance by keeping all database writes local to the same data center.
The use of the NoSQL database is very well suited for the data the Network Director needs to write because the data tends to be historical data that does not have relationships to other data in the Policy Manager data model that must be kept in sync. The data does reference the Policy Manager data model to reference containers, services, and operations, but there aren't relational queries that Policy Manager needs to perform that would join with the historical data that the Network Director is writing.
There is still a need for a relational stateful model for Policy Manager, which is why the RDBMS is still an important part of the architecture. The Network Director configuration information, such as listener and virtual service configuration, is persisted in the RDBMS. Therefore, the Network Directors still query their configuration indirectly through the Policy Manager to the RDBMS. When changes need to be made to the Network Director configuration, they must be made using another Policy Manager. The changes are then replicated to the slave database servers so that Network Directors needing that information in different data centers can read it from their local slave database server.
Although each ND is the master for the records it writes to its local master NoSQL shard, it is necessary for Policy Managers in different data centers to at least gain read access to that data. There are two options for how this can be achieved.
Query Router: Having a query router in each data center ensures that any query made by a local Policy Manager for ND-written information is directed to the shards in the other data centers. The data is queried, possibly from multiple data centers, on the fly. This option requires no duplication of data through replication. The data is always assured to be current. However, read performance may suffer due to having to route to different data centers. Furthermore, if a data center is down the data will not be accessible at all.
Replicated Content: Each master shard replicates its content to other data centers. The other data centers have slave shards. Having a query router in each data center ensures that any query made by a local Policy Manager for ND-written information is directed to the shards in the other data centers. This requires the duplication of data through replication. This provides the best performance for reads; however, if the replication is configured to be infrequent, queries may not get the most current information. This is the recommended option for the multi-regional architecture.
Both a relational database and a NoSQL database must be installed and configured. The relational database is configured with a single master and replication to the slave data centers. Because the relational database vendor is a customer choice, the setup of the relational database master and slaves is out of scope for this guide. Please refer to the documentation from your vendor of choice.
The only NoSQL database vendor currently supported by the API Gateway is MongoDB. This section outlines the steps for setting up MongoDB instances in different data centers using sharding and replication. To download and install the latest supported MongoDB version, please visit https://www.mongodb.com/akana. We highly recommend that you use MongoDB Enterprise Advanced, especially if you're running the API Gateway to support production or mission-critical applications. MongoDB Enterprise Advanced offers enterprise-grade security, operational automation, proactive support, and disaster recovery to run the API Gateway in a secure and reliable way.
For system requirements, see System Requirements for Akana Platform 2018.0.0.
A cluster of sharded MongoDB instances requires a set of three config servers. In order to make sharding configuration changes, all three config servers must be available.
The MongoDB query routers will make use of these configuration servers for query routing.
At least one config server will need to be available to a query router at all times.
To ensure they are available for query routers in case of a disaster, it is recommended that the config servers be installed in the different data centers, along with the query routers in each data center.
Remember that the three config servers need to be available for configuring the sharded cluster.
The following outlines the steps for configuring a config server.
Step 1 - Create a directory where the config server will store its data.
Step 2 - Create a configuration file that the config server will read on startup. This file will include the path to the data directory created in Step 1, the role of this process in the cluster, and the network address of the server. An example file is shown below.
On line 04, the cluster role is configsvr, which tells MongoDB to run as a config server and not as a database server.
Lines 06 and 07 identify the host and port that the config server is listening on. The default port for all MongoDB processes is 27017.
Repeat these steps for each of the config servers in the different data centers.
Step 1 - Create a configuration file that the query router will read on startup. The configuration file will include the addresses of the config servers and the network address of the server. An example file is shown below.
On Line 02, <replica set name> is the name of the replica set passed into the mongod --replSet parameter when starting the configuration database servers.
Lines 04 and 05 identify the host and port that the query router is listening on.
The default port for all MongoDB processes is 27017.
Step 2: Ensure all MongoDB databases that should be part of this deployment are stopped.
Step 3: Start the query router process, referencing the config file. An example is shown below.
queryrouter.cfg is the name of the file created in Step 1.
Repeat these steps for each of the query routers in the different data centers.
The server instances that hold the application data are called database servers. A set of multiple database servers that are reachable from a single query router make up a cluster. Database servers in a cluster can act as shards, or partitions of the application data, that can be used for horizontal scaling. Database servers in a cluster can also act as backups, or replicas, to either provide fault tolerance or read performance improvements.
For fault tolerance, for each master, the database should replicate its data to a slave database in the same data center. This slave database will become the master should the original master server go down.
To enhance read performance in all data centers, the goal is to ensure that from a single data center no read or write is performed against a server in a different data center. All interactions with the data store are local to a single data center. At the same time, the API Gateway in a single data center should have read access to all data written by all data centers.
To achieve this, each master database will replicate its data to slave databases in all the other data centers. When reads are performed, the query router of the local data center will direct the reads to the local master database and the local slave databases. The only data traffic between data centers is replication traffic.
There is one replica set per master database. The following are the steps for creating a database replica set.
Step 1 - Create a directory where the database server will store its data. Do this on the host for each database server.
Step 2 - Create a configuration file the database server will read on startup. The configuration file will include the path to the data directory created in Step 1, the role of this process in the cluster, and the network address of the server. An example file is shown below.
Lines 03–04 are the sharding configuration. That is discussed in the next section, Shards.
Lines 05–07 are the replication configuration.
On Line 6, the oplogSizeMB is the amount of space available for performing replication, and should be roughly 5% of the total space used by the database.
Line 07 gives a name to the replica set that this database is part of. There is one master database per data center and one replica set per master, so it's a good idea to name the replica set after the master database's data center.
Lines 09 and 10 identify the host and port that the shard server is listening on. The default port for all MongoDB processes is 27017.
Step 3: Start the database server process, referencing the config file, as shown in the example below.
Step 4: In each of the other data centers create a slave database. For fault tolerance, you can create more than one in each data center, including the same data center as the master database. However, if you create a slave in the same data center as the master, it is possible that the secondary is used when performing reads which may have stale data. On each of the hosts for the slave servers, create new data directories for the replicated data.
The replica set they are part of.
The network address of the server.
An example file is shown below.
Note: On line 07, the replica set name must be the same as the one named for the master database.
Step 6: Start the slave databases.
Step 7: Connect to the admin shell of the master database, as shown in the example below.
Step 9: From the shell, add the slave databases to the replica set with the rs.add command, as shown below.
In the above, the argument is the host and port of the slave database.
Do this for each database in the replica set.
Step 11: If you choose to create slave replicas in the same data center as the master, you can make sure that the local replicas have priority over replicas in other data centers for taking over the master role if the master goes down. To ensure that the remote replicas do not become the master, change the priority property of each of the remote members to 0, as in the example shown below.
In the above example, 2 is the _id of the remote slave database.
Execute this command for each of the remote slave databases in the configuration.
If there is an even number of replica servers in a replica set, you must create an arbiter. An arbiter is another mongod process, but it does not hold data. It must be run on a machine that is not already running a replica. To add an arbiter, perform the following steps.
Step 1 - Create a directory where the arbiter server will store its data.
The path to the data directory created in Step 1.
The role of this process in the cluster.
Lines 03–04 are specific to arbiters.
Lines 05–06 are the sharding configuration. That is discussed in the next section, Shards.
Lines 07–08 are the replication configuration.
Line 08 gives a name to the replica set that the arbiter is part of.
Lines 10 and 11 identify the host and port the shard server is listening on. The default port for all MongoDB processes is 27017.
Step 3: Start the arbiter server process, referencing the config file, as shown in the example below.
In the above, the only parameter is the host and port the arbiter was deployed on.
Each data center will have at least one master MongoDB database server instance.
Each master server is a shard.
The collection of shards provides the complete set of application data.
So far, this document has described a scenario that includes is a single master database, or shard, in each data center. However, it is possible to have multiple shards in a single data center.
The query routers will direct inserts and queries to the server instances based on shard keys. In this architecture, the shard key is the key of the container generating the record to be stored.
Line 02 identifies this database server as a shard server and not a standalone database.
Step 2: Connect to the admin shell of the local query router, as shown in the example below.
The parameters are the shard address and shard name. The shard address is in the format of: <replica name>/<hostname>:<port>.
Step 5: From the shell, enable sharding for each of the API Gateway databases, as shown in the example below.
In the above, PM, AUDIT_LOG, METRIC_RAW_DATA, and METRIC_ROLLUP_DATA are the databases that the API Gateway uses.
Step 6: From the shell, enable sharding for each of the collections the API Gateway uses, as shown in the example below.
The first argument of each command is the collection name prefixed by the database name.
The second argument is a structure identifying the name of the property to use as the key and the type of property it is.
The second argument is the lower inclusive bound of the range.
The third argument is the exclusive upper bound of the range.
In this example, we are directing all records from containers with keys of 100–199 to be part of this shard. The final argument is the tag of the shard created in Step 6.
Note: It is important to make sure the keys of all containers in the system (including Policy Manager containers) are covered by a tag range. If not, the query routers will choose the server to store the records without guidance, and might store them on servers in a different data center entirely.
Execute the addTagRange commands for each tag, or shard, with a different range. Do not create ranges that overlap with other shard tags.
Perform the shard configuration steps for each replica set. Each replica set represents one shard cluster.
When a read is performed through a query router, the default settings would have the query router route the query to each of the master shards in the cluster. This would not be ideal, because it would cause the queries to span data centers. To avoid this, you can configure the database driver with a read preference.
The read preference is configured as a query parameter of the URI of the MongoDB connection used by a database client. Creating the URI is part of the configuration steps in the next section; see Installing Network Director. The database URI and the possible settings are described in detail in the MongoDB documentation: https://docs.mongodb.org/manual/reference/connection-string/.
The readPreference setting supports a variety of options. The option we are using is the nearest option. This tells the query router to direct all reads to the closest member of a replica set, regardless of whether the member is a master or slave shard. For more information on the nearest read preference, see https://docs.mongodb.org/manual/reference/read-preference/#nearest.
Note: If you choose to have a slave replica in the same data center as the primary, and then, based on this read preference, the database driver determines that the slave replica is closer than the master, the reads is performed against the slave.
An example of a MongoDB URI with this setting is shown in the example below.
To be tolerant of query router outages, you can deploy multiple query routers in a data center, creating a cluster of query routers. If the primary goes down, the database driver will fail over to a different query router in the cluster. To make use of this functionality, the driver must be configured with the addresses of each of the query routers by providing a comma-separated list of host:port values in the MongoDB URI. An example is shown below.
There are several other options for the database driver that can be set using the MongoDB URI. For a full list of options, review the MongoDB reference manual at https://docs.mongodb.org/manual/reference/connection-string/. For convenience, some of the more commonly used settings are listed below.
connectTimeoutMS—The time, in milliseconds, to attempt a connection before timing out. The default is never to timeout, though different drivers might vary.
socketTimeoutMS—The time, in milliseconds, to attempt a send or receive on a socket before the attempt times out. The default is never to timeout, though different drivers might vary.
maxPoolSize—The maximum number of connections in the connection pool. Default: 100.
minPoolSize—The minimum number of connections in the connection pool. Default: 0.
waitQueueMultiple—A number that the driver multiples the maxPoolSize value to, to provide the maximum number of threads allowed to wait for a connection to become available from the pool.
waitQueueTimeoutMS—The maximum time, in milliseconds, that a thread can wait for a connection to become available.
An example of using these options is shown in the example below.
Step 1: Connect to the admin shell of the local query router, as shown in the example below.
Step 2: From the shell, change the current database to the config database, as shown in the example below.
Step 3: Print a formatted report of the sharding configuration and the information regarding existing chunks in a sharded cluster, as shown in the example below.
If the total number of chunks is greater than or equal to 20, the default behavior suppresses the detailed chunk information. For more information about this command and its output, see https://docs.mongodb.org/v3.0/reference/method/sh.status/.
Step 4: Verify that the shards have the correct hosts and ports.
Step 5: Verify that the shards are associated with the correct replica sets.
Step 6: Verify that the database is sharded.
Step 7: Verify the shard key is correct.
Step 8: Verify that the tags are split amongst the shards correctly.
Step 2: From the shell, change the current database to the database that you want to verify, as shown in the example below.
Step 3: Print the data distribution statistics for the sharded collection to verify, as shown in the example below.
For more information about this command and its output, see https://docs.mongodb.com/v3.0/reference/method/db.collection.getShardDistribution/.
Step 4: Verify that the statistics for each of the shards match with your expectations based on the data being stored.
By default, each MongoDB process (query router, configuration server, database server, and arbiter) writes diagnostic and error information to the shell it was started from. Each process can be configured to write this information to a log file instead.
To instruct a process to write to a log, add a systemLog section in its configuration file, as shown in the example below.
Line 01 starts the systemLog section.
Line 02 provides a location to which log files will be written. There are several other options for logging, described in detail at https://docs.mongodb.org/manual/reference/configuration-options/.
If there is a concern that a query router is using stale information because its cache is out of date, you can flush the cache and force the query router to re-read the config database. To flush a query router's cache, perform the following steps.
Step 1: Connect to the admin shell of the query router, as shown in the example below.
Step 2: From the shell, flush the cache, as shown in the example below.
Did not find local voted for document at startup.
If you encounter this error, connect to the configuration database and do an rs.initiate(). An example is shown below.
In the above, <replica set name> is the name of the replica set passed into the mongod --replSet parameter when starting the configuration database servers.
For more information, refer to the MongoDB documentation: https://docs.mongodb.com/manual/tutorial/deploy-shard-cluster/#initiate-the-replica-set.
A walk through of the steps for installing and configuring the Policy Manager and Network Director processes in two different data centers.
Designating one data center as the master data center and the other as the slave to denote the location of the master relational database.
First, as in a typical installation, we will install a container with the Policy Manager Console and Subsystems features. Because we are going to use MongoDB for reading and writing data generated by the Network Director, we must perform an additional step of configuring this container to use MongoDB.
To configure any Akana container to communicate with MongoDB, you must install the Akana MongoDB Support plugin. You can find the plugin on the Available Features page of the Akana Administration Console using the Plug-in filter.
This plugin has only one configuration setting; the URI of the MongoDB database. This URI includes the host and port of the database server as well as the database name in the path. By default, the name of the database installed for all Akana products is AKANA.
In this deployment, we will use the URI to the MongoDB query router installed in the local data center as outlined in Database Driver Settings. At the conclusion of this configuration, after restarting, we will have a Policy Manager process that can write to the master relational database and read/write from the local MongoDB server farm through the query router.
Ultimately we will have Network Directors in different data centers that communicate only with locally installed Policy Manager containers. As part of setting this up, we need to create Policy Manager clusters in each data center so that the Policy Manager containers can be separated from the perspective of the Network Directors.
From the Policy Manager Management Console, navigate to the Akana Policy Manager organization.
Click Add Container, as shown below. The Add Container wizard starts.
On the first page of the wizard, select Container Cluster and then click Next.
On the Specify Container Details page, provide the name of the PM cluster for this data center, and an optional key and description, and then click Next.
At the Select Cluster Nodes page, do not select a container to add to the cluster at this time. Just click Next.
The Select Cluster Capabilities page, the last page of the wizard, prompts you to select a container that has the capabilities you want the cluster to have. Select the Policy Manager container that you just installed and configured, and click Finish.
If only one Policy Manager container is installed in the data center, you can just copy the listener information for the Policy Manager container.
If more than one Policy Manager container is used, a load balancer should be used between the Network Directors and the Policy Managers. Then, you can model the load balancer's interface for the cluster's listener.
To create a listener, click the Add Container Listener button in the Inbound Listeners portlet, as shown below. This starts the Add Container Listener wizard.
For the cluster listener, make sure you use the same name as the listener of the Policy Manager container that is in the cluster. If there are multiple Policy Manager containers in the cluster, they MUST all share the same listener names.
The next step is to host all the Policy Manager services on the cluster.
Select the original Policy Manager container in the tree. Then, on the right, click Clone Hosted Services. The Clone Hosted Services wizard starts.
On the first page, select the cluster as the target container and then click Next.
At the confirmation page, click Finish.
At this point, you should be able to select the cluster in the tree, and the Hosted Services tab, and see all the Policy Manager services listed.
The last step in creating the cluster is to place the original Policy Manager container in the cluster.
Select the original container and, on the right, click Add Container to Cluster, as shown below.
At the Add Container to Cluster page, select the cluster and click Apply.
Now, you have a fully configured Policy Manager cluster with the first Policy Manager container as a cluster node, as shown below.
Follow the previous steps for additional Policy Manager containers to provide increased scale or fault tolerance.
The next step is to install and configure a Network Director.
Follow all the typical steps for installing and configuring a Network Director process. On the page for providing the WS-MetadataExchange Options, enter the URL of the Policy Manager cluster set up in the previous section, as shown in the example below.
Turn off direct database access: set the usage.local.writer.enabled option to false.
Turn on remote Policy Manager access: set the usage.remote.writer.enabled option to true.
If you want the data to be written directly to the local master MongoDB shard instead, leave that option set to false. However, for this to work, you must also install the same Akana MongoDB Support plugin on the Network Director container. The URI of the query router should also be used for the Network Director to shield the Network Director from possible MongoDB configuration changes down the line.
Follow the previous steps for additional Network Director containers to provide increased scale or fault tolerance.
The connection is to the local slave RDBMS and local MongoDB query router.
There is a difference in how the services of the slave Policy Manager are provisioned, outlined below.
When configuring the Policy Manager Services feature in the Akana Administration Console, do not configure an administrator name and password or select the option to Provision Services. Both of these actions require writing to the relational database, and since this container is configured to use a slave database the actions will not work. The administrator name and password have already been set up when configuring the master data center. Service provisioning is done a different way, as described below. When the configuration wizard is complete, restart the container.
Register the slave Policy Manager container with the system as we would for a Network Director.
Log in to the master data center's Policy Manager Management Console and navigate to the Akana Policy Manager organization.
Click Add Container, as shown below. The Add Container wizard opens.
At the Select Container Type page, select Container (the default) and click Next.
On the Specify Metadata Import Options page, enter the metadata address of the slave Policy Manager container (<scheme>://<host>:<port>/metadata), as shown below.
Complete the wizard, adding any trusted certificates if prompted.
At this point, we now have the new Policy Manager container in the organization tree.
Next we will create a cluster for the slave database following procedures similar to those used in the master data center. When choosing a container whose capabilities we want to model in the cluster, choose the slave PM container.
At this point, we want to clone the services to be hosted by cluster.
When doing this with the master data center we choose the master PM container to clone services from.
In the slave case, the slave PM container does not have any services to clone yet. Instead, we will clone services from the Data Center A PM Cluster to the Data Center B PM Cluster.
Remember that the listener names between the two clusters must match in order for the cloning to work properly.
Now that we have a slave PM cluster with hosted services, we can now move the slave PM container into the slave PM cluster, as shown below. Once the container is moved into the cluster, the services from the cluster is provisioned for the PM container.
This performs the same step as the provisioning configuration task in the Akana Administration Console. It records the services the container is hosting in the database.
At this point, restart the slave PM container so that it will start listening for messages for those services.
The slave PM container is now fully functional in a read-only manner, providing all necessary services for the local Network Directors to use.
Follow the previous steps for additional slave Policy Manager containers to provide increased scale or fault tolerance.
The sample file below is the sample sharding script for all the collections that are sharded, in a working installation with two data centers. In this example, sharding is based on the container keys. The container keys are assigned as numbers instead of UUID and are then used as tag ranges.