text:
Category Archives: C10M
In our previous world-record-setting C10M benchmark, we showed that MigratoryData Server can achieve extreme high-scalability by delivering real-time messaging to 10 million concurrent users on a single 1U machine. Here, we show that MigratoryData Server can achieve simultaneously both extreme high-scalability and consistent low latency.
Written in Java, MigratoryData Server runs on a Java Virtual Machine (JVM). In the previous C10M benchmark, some JVM tuning adjustments were necessary such as using CMS garbage collector, enabling huge pages, and using compressed pointers.
In this post, we show that by simply replacing the JVM with Zing JVM out-of-the-box (without any tuning), and preserving the same C10M benchmark scenario and setup, we can reduce the average latency from 61 milliseconds to under 15 milliseconds. Moreover, and more importantly, the latency spikes can be significantly reduced from 585 milliseconds to 25 milliseconds for the 99th percentile latency and from 1700 milliseconds to 126 milliseconds for the maximum latency. Therefore, every single message can be delivered, even in the worst case, with almost no delay.
And so, the relatively high latency spikes we saw in the previous C10M benchmark were due to JVM's Garbage Collection (GC). In the new benchmark, not only Zing JVM didn't introduce high latency spikes, but based on analyzing the logs, it appears that GC effects no longer dominate latency behavior. The dramatically improved 126 ms max latency is not caused by GC but by other condition of the benchmark setup. Anyway, this max latency was so small for a web architecture that we did not spend time to determine at which level exactly it occurred.
In summary, this new C10M benchmark demonstrates that MigratoryData Server running on a single 1U machine can handle 10 million concurrent clients each receiving a 512-byte message per minute (at a total bandwidth of 0.8 Gbps) with a consistent end-to-end latency of under 15 milliseconds.
Benchmark Setup
We used the same benchmark setup and config from the previous C10M benchmark (see sections "Benchmark Setup" and "Configuration Tuning"). The only changes were to replace the JVM with Zing JVM and not use huge pages.
MigratoryData Server is essentially a publish/subscribe system for web messaging. Subscribers are clients which connect to the MigratoryData server – using persistent WebSocket or HTTP connections – and subscribe to subjects (also known as topics). Publishers are clients which connect to the MigratoryData server and publish messages. A message consists mainly of a subject and some data. Upon receiving a message from a publisher, the MigratoryData server delivers that message to all clients which subscribed to the subject of that message.
To briefly describe the benchmark setup (full details, including the specifications of the machines, can be found in our previous C10M benchmark), nine machines were utilized as follows:
One machine was utilized to run one instance of the MigratoryData server.
In order to connect 10 million clients, four machines were utilized to run four instances of Benchsub, each opening 2.5 million HTTP persistent connections to the MigratoryData server. Also, Benchsub subscribed each client to a distinct subject.
In order to send one message per minute to each client, four machines were utilized to run eight instances of Benchpub (two instances per machine) for publishing 168,000 messages/second to the MigratoryData server, each instance publishing 21,000 messages/second.
Finally, a fifth Benchsub instance was used to connect 100 additional clients, representing samples of the population of 10 million concurrent clients. This Benchsub instance was used to compute supplemental latency statistics – in addition to the latency statistics computed by the other four Benchsub instances.
Latency is the end-to-end time needed for a message to propagate from the publisher to the subscriber, via the MigratoryData server. Thus, the latency of a message is the difference between the time at which the message is sent by Benchpub to the MigratoryData server and the time at which the message is received by Benchsub from the MigratoryData server.
Summary of Results
Number of concurrent client connections 10,000,100
Number of concurrent subscribed subjects 10,000,000
Number of messages per client 1 message per minute
Payload of each message 512 random bytes
Total messages throughput 168,000 messages per second
Median latency 13.8 milliseconds
Average latency 13.2 milliseconds
Standard deviation for latency 4.7 milliseconds
95th percentile latency 21.2 milliseconds
Maximum latency 126 milliseconds
Hardware One Dell PowerEdge R610 with 2 x Intel Xeon X5650 CPU at 2.66 GHz and 96 GB RAM
Operating system CentOS / RHEL 7.1 with the default kernel 3.10.0-229 (without kernel recompilation)
CPU utilization under 50%
Network utilization 0.8 Gigabit per second
MigratoryData Server provides advanced monitoring via JMX and other protocols. We used the jconsole tool (part of the Java Development Kit) for JMX monitoring. In the results presented below we show screenshots obtained during JMX monitoring.
Connections and Messages
As mentioned in the Benchmark Setup section above, the 10 million concurrent connections were opened by four instances of Benchsub that simulated 2.5 million concurrent users each. In addition, a fifth instance of Benchsub opened another 100 concurrent connections. Indeed, the indicator ConnectedSessions of the JMX screenshot below shows around 10,000,100. The same number of concurrent socket connections is confirmed by the tools netstat and slabtop (see the screenshot in the Network Utilization subsection below).
As mentioned in the Benchmark Setup section above, eight Benchpub instances published 168,000 messages/second in order to produce one message per minute for each of the 10 million clients. The JMX indicator OutPublishMessagesPerSecond shows that the outgoing message throughput is indeed around 168,000 messages/second.
Memory and CPU Utilization
In the screenshot below, you can see the Memory and CPU usage during the benchmark test. The CPU usage is similar to that of the previous C10M benchmark excepting the CPU spikes. The spikes up to 100% produced by the Full GCs in the previous C10M benchmark are now reduced at under 55%.
Network Utilization
As you can see in the screenshot below, sending a 512-byte message to each of 10 million clients every minute produced a total outgoing bandwidth from server to subscribers of 103 MB/s or 0.8 Gbps. This total bandwidth includes the 512-byte payload of each of the 168,000 messages sent every second to the subscribers, the overhead added by the TCP and MigratoryData protocol for each message, as well as certain traffic produced by the ssh sessions, JMX monitoring, and the TCP acknowledgements sent by the MigratoryData server to publishers for the received messages. Therefore, the overhead introduced by the TCP and MigratoryData protocol was under 131 bytes per message.
This screenshot also shows the top and slabtop information, and the number of sockets as reported by the netstat tool.
As defined in the Benchmark Setup section above, latency is the time needed for a message to propagate from the publisher to the subscriber, via the MigratoryData server. When Benchpub creates a message it includes the creation time as part of it. In this way, Benchsub can compute the latency as the difference between the creation and reception times of the messages. Because the machines were synchronized with ntp, which did not run long enough for perfect time synchronization, we can observe 1-2 milliseconds time differences between publisher machines and subscribers machines (see the negative values of "Latency Min" in the screenshot).
In addition to computing the latency for all messages received, Benchsub also calculates the average, standard deviation, and maximum. These latency statistics are computed incrementally for each new message received. In this way, the statistics are obtained for all messages received, and not just for a sample size.
In the screenshot below, the "Total messages" information shows that each of the four Benchsub instances received around 450 millions messages during more than 3 hours of benchmark test running. Hence, the following latency statistics are accurate, being computed on the entire population of about 1.8 billion messages (up to the ntp time difference of 1-2 milliseconds):
Latency Mean is 13.24 milliseconds
Latency Standard Deviation is 4.7 milliseconds
Latency Maximum is 126 milliseconds
More Latency Statistics
In the previous section we explained that the latency statistics – mean, standard deviation, and maximum – were computed for all messages received by all 10 million clients. However, other stats that would be interesting to look at for a real-time service, including the median, the 95th percentile and the 99th percentile, cannot be computed incrementally. We need all latency values in order to be able to compute such extra statistics. Recording 1.8 billion latency values is not practical during a performance test, so we used sampling to estimate these additional statistics.
As mentioned in the Benchmark Setup section, we used a fifth Benchsub instance to collect samples for 100 concurrent users from the entire population of 10 million. Each of the 100 users subscribed to a randomly selected subject from the 10 million subjects available.
We recorded all latency values for each of the 100 users for more than 3 hours during the benchmark test. Since each user received an average of one message per minute, we computed and recorded approximately 200 latency values for each user. Subsequently, we computed the median, average, 95th percentile, and 99th percentile for each of the 100 users (detailed results are available here).
Finally, we calculated a 99% confidence interval and we can estimate that, should we repeat the test, there is a 99% probability that the average value would be as follows:
Median Latency: 13.80 ms ± 0.14 ms
Mean Latency: 13.33 ms ± 0.12 ms
95th Percentile Latency: 21.27 ms ± 0.20 ms
We showed here that MigratoryData Server can achieve simultaneously both extreme high-scalability and consistent low latency – delivering real-time data to 10 million concurrent users with under 15 ms average latency, under 25 ms 99th percentile latency, and 126 ms maximum latency for the worst case (computed from a total of 1.8 billion latency values!) – provided that a properly tuned JVM for short garbage collection pauses is used, such as Zing JVM.
This entry was posted in Benchmarks, C10M, High Scalability on January 20, 2016 by Mihai Rotaru.
How MigratoryData solved the C10M problem: 10 Million Concurrent Connections on a Single Commodity Server
The C10M problem relates to the classic C10K Internet scalability problem, which originally occurred in the context of Web servers. It consists of successfully supporting 10,000 concurrent connections on a single machine, and while C10K is currently solved by certain Web servers, C10M remains a challenging problem not only for Web servers, but for any Internet server in general.
MigratoryData Server is a real-time Web server using the WebSocket protocol, as well as the HTTP protocol, to communicate with its clients. Unlike traditional Web servers, MigratoryData Server does not use the request-response interaction model (employing short-living connections). Rather it uses the publish/subscribe model along persistent connections. With clients permanently connected, MigratoryData Server makes data delivery to its clients possible in a timely manner, with low latency.
In addition to its role as real-time Web server, MigratoryData Server implements features traditionally provided by Enterprise Messaging Systems such as publish/subscribe interaction, active/active clustering, guaranteed message delivery, entitlement, as well as API libraries for the most popular environments.
While achieving high scalability is a difficult task for a Web server in general, achieving high scalability for a real-time Web server, without sacrificing enterprise messaging features, is even harder.
In this article, we demonstrate that MigratoryData Server is able to handle 10 million concurrent connections on a single commodity machine. Moreover, it is able to push almost 1 Gbps live data to these 10 million users (each user receiving a 512-byte message per minute) with an average latency of under 100 milliseconds.
MigratoryData's Publish/Subscribe Interaction
Subscribing clients connect to the MigratoryData server using persistent WebSocket or HTTP connections and subscribe to one or more subjects (also known as topics) by using MigratoryData's protocol. Publishing clients communicate with the MigratoryData server in the same way as subscribing clients, but they publish messages. A message contains mainly a subject and some data. When a message is received by the MigratoryData server, it distributes that message to all clients that subscribed to the subject of that message.
We used several publishing clients to push messages into a MigratoryData server, which pushed the messages out to several subscribing clients through HTTP persistent connections.
The publishing tool used in the benchmark test – MigratoryData Benchpub – is capable of publishing messages of a configurable size at a configurable frequency. The subscribing tool – MigratoryData Benchsub – is capable of opening a configurable number of concurrent connections, subscribing to a configurable number of subjects, and computing the latency of the messages received for the subscribed subjects. Both Benchpub and Benchsub use MigratoryData's protocol, so they simulate real applications.
The following diagram shows the architecture of the benchmark test.
Nine machines were utilized in the benchmark test, as follows:
One Dell R610 machine was utilized to run one instance of the MigratoryData server. The specifications of this 1U machine are as follows:
2 x Intel Xeon X5650 CPU at 2.66 GHz (6 cores per CPU, totaling 12 cores)
96 GB RAM (DDR3 at 1333 MHz)
Network adapter Intel X520-DA2 (10 Gbps)
CentOS / RHEL 7.1 with the default kernel 3.10.0-229 (without kernel recompilation)
Four Dell R610 machines were utilized to run four instances of Benchsub. Each Benchsub instance simulated 2.5 million concurrent users each, totaling 10 million concurrent users. Each user subscribed to a distinct subject. Therefore, the total number of concurrent subjects was also 10 million.
Four Dell SC1435 machines were utilized to run eight instances of Benchpub (four pairs of instances). Each of the eight Benchpub instances published at a frequency of 21,000 messages per second, with each message containing a sequence of 512 random bytes. The total message throughput was therefore 168,000 messages per second. In this way, each user received one message per minute.
Finally, an additional Benchsub instance was used to simulate 100 concurrent users, representing samples of the population of 10 million concurrent users. This Benchsub instance was used to compute supplemental latency statistics – in addition to the latency statistics computed by the other four Benchsub instances. These other four Benchsub instances used the latencies of all messages received by all 10 million users to compute their statistics, instead of sampling the data.
Latency, depicted in the diagram above, is defined from the time Benchpub creates a message until the Bechsub receives the message from the MigratoryData server.
MigratoryData Server provides advanced monitoring via JMX and other protocols. We used the jconsole tool (included in the Java Development Kit) to monitor the MigratoryData server via JMX. In the results presented below we show screenshots obtained during JMX monitoring.
As depicted in the Benchmark Setup section above, the 10 million concurrent connections were opened by four instances of Benchsub that simulated 2.5 million concurrent users each. Each of the 10 million users subscribed to a distinct subject, hence there were 10 million concurrent subscribed subjects as well. In addition, a fifth instance of Benchsub opened another 100 concurrent connections.
As can be seen from the JMX screenshot below, MigratoryData Server handled 10,000,108 concurrent connections (see the JMX indicator ConnectedSessions). The same number of concurrent socket connections is confirmed by the tools netstat and slabtop (see the screenshot in the Network Utilization subsection below).
As described in the Benchmark Setup section, eight Benchpub instances ran on four machines and sent messages to the MigratoryData server.
In order for each of the 10 million users to receive one message per minute, each of the eight Benchpub instances published 21,000 messages per second. The payload of each message consisted of a sequence of 512 random bytes. Therefore, the eight Benchpub instances sent 168,000 messages per second to the MigratoryData server which were then pushed out to the subscribing clients at the same message frequency.
The screenshot of the Connections subsection above shows that the outgoing messages throughput is around 168,000 messages per second (see the JMX indicator OutPublishMessagesPerSecond).
CPU Utilization
In the screenshot below, it can be seen that the CPU usage of the machine which hosted the MigratoryData server was under 50%, with spikes from time to time when a major JVM Garbage Collection occurred. In fact, from our observations, the CPU percent strictly utilized by the MigratoryData server is under 40%. The variations you can see in the screenshot from under 40% to about 50% occur when minor JVM Garbage Collections happen.
Memory Utilization
In the screenshot below you can see that the memory usage is predictable and the pattern does not change after 3 hours of test running. More importantly, during the 3 hours of test running there were both minor and major JVM Garbage Collections. Thus, the test simulates a real life situation when both types of Garbage Collections might occur.
As you can see in the screenshot below, the outgoing traffic for pushing 168,000 messages per second to 10 million concurrent clients was 103 Megabytes per second, representing 0.8 Gbps.
The payload of each message is 512 bytes and the throughput is 168,000 messages per second, totaling 82 Megabytes per second. The difference of 21 MB/sec, up to the actual bandwidth utilization of 103 MB/sec, was introduced by the overhead added by the MigratoryData protocol as well as by the TCP/IP protocol, resulting in an extra 131 bytes per message.
In fact, the overhead introduced by the MigratoryData protocol and the TCP/IP protocol is even less than 131 bytes per message. When we calculated the bandwidth – using the accurate traffic reported by the kernel into /proc/net/dev – we included all outgoing traffic of the network interface. This traffic is almost entirely produced by messages being pushed to clients. However it also includes some additional traffic produced by several ssh sessions, the JMX monitoring console, as well as the acknowledgements sent to publishers for receiving the messages.
As defined in the Benchmark Setup section, latency is the time needed for a message to propagate from the publisher to the subscriber, via the MigratoryData server. When Benchpub creates a message it includes the creation time as part of it. In this way, Benchsub can compute the latency as the difference between the creation and reception times of the messages.
In addition to computing the latency for all messages received, Benchsub also calculates the average, standard deviation, and maximum. These latency statistics are computed incrementally for each new message received. In this way, statistics are obtained for all messages received, and not just for a sample size.
In the screenshot below, the "Total messages" information shows that each of the four Benchsub instances received around 400 millions messages during the 3 hours of benchmark test running. Therefore, the following latency statistics are very accurate, being computed on the entire population of more than 1.5 billion messages:
Latency Mean is 61 milliseconds
Latency Standard Deviation is 140 milliseconds
Latency Maximum is 1.7 second
Note – Time was synchronized with ntp which did not run long enough for perfect time synchronization, which is the reason for the observed negative minimum latencies. Because minimum latency is normally 0, the negative and positive minimum latencies represent the difference introduced by the imperfect time synchronization among machines.
In the previous section we explained that the latency statistics – mean, standard deviation, and maximum – were computed for all messages received by all 10 million clients. However, other stats that would be interesting to look at for a real-time service, including the median, 95th percentile and 99th percentile, cannot be computed incrementally. We need all latencies in order to be able to compute such extra statistics. Recording 1.5 billion latencies is not practical during a performance test, so we used sampling to estimate these additional statistics.
As outlined in the Benchmark Setup section, we used a fifth Benchsub instance to collect samples for 100 concurrent users from the entire population of 10 million. Each of the 100 users subscribed to a randomly selected subject from the 10 million available.
We recorded all latencies for each of the 100 users for 166 minutes during the benchmark test. Since each user received an average of one message per minute, we computed and recorded approximately 166 latencies for each user. Subsequently, we computed the median, average, 95th percentile, and 99th percentile for each of the 100 users (results are available as CSV file here, which also includes random subject each user subscribed to, as well as the precise number of messages received by each user).
Finally, we calculated a 99% confidence interval and we can estimate that, should we repeat the test, there is a 99% probability that the average value – for all users – would be as follows:
95th Percentile Latency: 374.90 ms ± 21.51 ms
Note – We can see that the mean – 61 milliseconds – computed as detailed in the previous subsection for all users (and for more than 1.5 billion latencies) belongs to the calculated confidence interval for mean: [55.69 ms, 61.35 ms] (i.e. 58.52 – 2.83 ms and 58.52 + 2.83 ms).
Number of concurrent client connections
Number of concurrent subscribed subjects
Number of messages per client
1 message per minute
Payload of each message
512 random bytes
Total messages throughput
168,000 messages per second
Median latency
18 milliseconds
Average latency
Standard deviation for latency
140 milliseconds
95th percentile latency
Maximum latency
One Dell PowerEdge R610 with 2 x Intel Xeon X5650 CPU at 2.66 GHz and 96 GB RAM
0.8 Gigabit per second
From our observations, in the absence of minor and major JVM Garbage Collections, all of the latency statistics above should be around the median values at 18 milliseconds. For example, the maximum latency above was introduced by a major JVM Garbage Collection. However, as major Garbage Collections happen rarely, in real life, such a high latency will occur only a few times per day.
Note – We have customers with large deployments (millions of end users) where the Java Virtual Machine is configured such that no major JVM Garbage Collection occurs. However, in these cases MigratoryData server is restarted on a daily basis.
Configuration Tuning
The benchmark test used standard configurations of Linux Kernel, Java Virtual Machine, and MigratoryData Server with only a few changes, which I am going to detail below.
MigratoryData Server ran on a machine with CentOS / RHEL 7.1 out of the box. In order to demonstrate that MigratoryData Server is able solve the C10M problem on commodity hardware and operating systems typically found in data centers, we did not recompile the kernel, but used the default 3.10.0-229 kernel.
The only system configurations we made are as follows:
Increased the number of socket descriptors, in order to allow the system to handle 10 million sockets:
used the sysctl configuration fs.file-max=12000500
echo 20000500 > /proc/sys/fs/nr_open
ulimit -n 20000000
Increased the maximum number of memory pages for TCP using the sysctl configuration:
net.ipv4.tcp_mem=10000000 10000000 10000000
Adjusted the buffers of TCP connections with sysctl for better memory usage, as follows:
net.ipv4.tcp_rmem=1024 4096 16384
net.ipv4.tcp_wmem=1024 4096 16384
net.core.rmem_max=16384
net.core.wmem_max=16384
Statically balanced the hardware interrupts of the network adapter across the logical CPUs using smp_affinity. The Intel X520-DA2 network adapter has 24 tx/rx queues, each having a hardware interrupt (in the /proc/interrupts there are 24 entries for the p1p1 network interface). Coincidentally, the server also has 24 logical processors corresponding to its two six-core CPUs. We used smp_affinity to statically map each interrupt of the 24 tx/rx queues of the network adapter to each of the 24 logical processors.
Better use of Translation-Lookaside Buffer (TLB) caches by the processor. These caches contain virtual-to-physical address translations and have a small number of entries with the most-recently used pages. Using huge pages of 2 MB instead of the normal 4 KB pages, a TLB entry can handle much more memory, thus making the CPU caching more efficient. Because we allocated 54 GB to the Java Virtual Machine which ran the MigratoryData server, we reserved 60 GB huge pages (30720 huge pages x 2 MB / huge page) using the sysctl configuration:
vm.nr_hugepages=30720
Java Virtual Machine
We used Oracle Java 1.8 update 45. Here are the main Java Virtual Machine (JVM) parameters we used (all JVM parameters can be seen in the screenshot below):
Allocated 54 GB for the JVM
Used Concurrent Mark Sweep (CMS) Garbage Collector
Enabled huge pages as discussed in the Linux Kernel subsection above by using:
+UseLargePages
Used compressed pointers, extended beyond the usual 32 GB limit, in order to optimize the memory footprint (with about 10 GB) by using:
-XX:ObjectAlignmentInBytes=16 -XX:+UseCompressedOops
MigratoryData Server
We used MigratoryData Server 5.0.14. To its default configuration, we made the following changes:
Enabled the JMX monitoring through the port 3000 (without authentication, and via an unencrypted connection):
Monitor = JMX
MonitorJMX.Authentication = false
MonitorJMX.Listen = 192.168.3.115:3000
Configured parallelism. In order to better scale on multiprocessor servers incoming users are separated in workgroups based on their IP address. Workgroups run in parallel, using almost independent threads. Thus, we used the following parameters related to parallelism:
Worgroups = 10
IoThreads = 20
Distributed users across workgroups. Because all 10 million users came from only four IP addresses, originating from the four Benchsub instances, we used a parameter called BenchmarkMode in order to distribute users across workgroups as would happen in real life when they would all come from different IP addresses. To achieve this, we used the configuration:
BenchmarkMode = true
Reduced the default initial size of the buffers. When handling a message, a buffer of 8192 bytes is created. If the message is higher than 8192 bytes, the buffer automatically expands in order to hold the entire message. On the other hand, if the messages is smaller than 8192 some memory space remains unused. Because we know the payload of our messages is 512 bytes, we reduced the default initial size of the buffers from 8192 to 768 using the following parameters (note that the name of the parameters is quite misleading; it's not a hard coded limit but a default initial size):
BufferLimit.Send = 768
BufferLimit.Receive = 768
Reduced memory footprint and allowed better performance by using a native C implementation with JNI for socket handling:
X.Native.Io = true
In a talk cited by HighScalability.com, Robert Graham discussed the C10M problem. He explains why the kernel could be more of a problem than a solution for achieving high scalability and suggested a number of principles for building scalable systems.
Looking at slabtop in the screenshot of the Network Utilization subsection above, we observe that the Linux kernel used around 32 GB to maintain open the 10 million concurrent socket connections. For usual systems, 3.2 KB per socket connection could seem quite reasonable. However with the explosion of Internet devices (mobiles and Internet of Things) we see systems requiring millions of concurrent connections more and more frequently. We therefore echo Robert's concern on the kernel and think that Linux might, for example, provide better memory usage to handle socket connections.
Moreover, many of the principles discussed by Robert can be found in our approach presented above: do as much as possible outside the kernel, use an efficient thread model to scale across all processors, use huge pages to optimize CPU caches, and distribute interrupts across all processors.
That said, in this post we demonstrated that solving the C10M problem is feasible with MigratoryData Server using a commodity server and an off-the-shelf Linux distribution. Also, given the millions of end users that our customers have, who receive real-time data daily with MigratoryData Server running on Linux, the time for easily building highly scalable real-time Internet services is now, with existing ingredients: Linux operating system, MigratoryData's real-time Web server, and MigratoryData's API with libraries for almost any Internet technology (Web, Mobile, Desktop, Server, Internet of Things).
To learn more about MigratoryData Server and how it can help your business achieve effective high scalability, please visit migratorydata.com.
This entry was posted in Benchmarks, C10M, High Scalability, WebSockets and tagged Benchmarks, C10M, High Scalability, WebSockets on May 20, 2015 by Mihai Rotaru.
Scaling to 12 Million Concurrent Connections: How MigratoryData Did It
Massive scalability is the biggest challenge we undertake at MigratoryData, a provider of an enterprise publish-subscribe messaging system for building very scalable real-time web and mobile applications. We recently published a blog post demonstrating 12 million concurrent connections with MigratoryData WebSocket Server running on a single 1U server. I am going to share some lessons learned while pushing the boundaries of scalability with MigratoryData WebSocket Server.
The goal of our benchmark is to demonstrate the vertical scalability of the MigratoryData server using commodity hardware usually found in data centers. The server used for the benchmark was a Dell PowerEdge R610 (see the specs in the initial blog post) running CentOS 6.4 out of the box — with no Linux kernel recompilation.
Opening 12 Million Sockets
Let's discuss the server side and the client side separately.
Server-Side Considerations
Server Port Numbers
A common misunderstanding is that a server cannot accept more than 65,536 (216) TCP sockets because TCP ports are 16-bit integer numbers.
First, the number of ports is limited to 65,536, but this limitation applies only to a single IP address. Supposing that we are limited by the number of ports to have more than 65,536 clients, then adding more IP addresses to the server machine (either by adding new network cards, or simply by using IP aliasing for the existing network card) would solve the problem (even if, for opening 12 million client would need 184 network cards or IP aliases on the server machine).
In fact, the misunderstanding comes from the fact that the server does not use its listening IP address and a different ephemeral port for each new socket to distinguish among the sockets, but it uses the same listening IP address and the same listening port for all sockets and it distinguishes among sockets by using the IP address and the ephemeral port of each client. Therefore, MigratoryData Server uses a single port to accept any number of clients and optionally it uses another few ports for JMX monitoring, HTTP monitoring, etc
Server Socket Descriptors
While the MigratoryData server uses a single port to accept any number of clients, it uses a different socket descriptor for each client. So, to open 12 million sockets, the process of the MigratoryData server should be able to use 12 million socket descriptors. Increasing the maximum number of socket descriptors per process is possible using the command ulimit. Consequently, we increased this limit to about 20 million socket descriptors as follows:
Because one cannot increase the maximum number of socket descriptors per process to a value larger than the current kernel maximum (fs.nr_open) and because the kernel maximum defaults to 1048576 (10242), prior to running the ulimit command, we increased the kernel maximum accordingly as follows:
Client-Side Considerations
We developed a tool named MigratoryData Client Benchmark able to open a configurable number of connections to the MigratoryData server. MigratoryData Client Benchmark is also able to subscribe to a configurable number of subjects for each client connection (where each subject is randomly selected from a configurable set of subjects) and compute various statistics for the received messages.
We used ten servers Dell PowerEdge SC1435 having 16 GB RAM and 2 dual-core AMD Opteron CPU @2.0 GHz to run ten instances of MigratoryData Client Benchmark. Hence, from each client machine we opened 1.2 million sockets.
Client Port Numbers
Each socket connection uses a new ephemeral port number on the client machine. Therefore, we extended the range of the ephemeral ports from 500 to the maximum theoretical limit 65,536 – keeping only the ports from 1 to 500 reserved for the operating system:
sysctl -w net.ipv4.ip_local_port_range="500   65535"
Using this extension of the ephemeral port range, one can open up to 65,035 sockets from a client machine per each IP address of the client machine.
In order to be able to open 1.2 million sockets from a client machine, we had to create 19 IP aliases for the network interface of the client machine. In this way, about 65,000 sockets are opened to the MigratoryData server from each of the 19 IP addresses.
Note — For each socket, MigratoryData Client Benchmark uses precisely one of the 19 IP addresses and a successive port starting with the port 500 and ending with the port 65,535, thus avoiding the random allocation of the ephemeral port numbers.
Client Socket Descriptors
Each socket connection uses a socket descriptor on the client machine. Thus, to open 1.2 million sockets from a client machine, MigratoryData Client Benchmark should be able to use 1.2 million socket descriptors. As discussed in the section "Server Socket Descriptors" above, on each client machine we had to increase the maximum number of socket descriptors per process as follows:
echo 3000000 > /proc/sys/fs/nr_open
ulimit -n 2000000
Other Client Tuning Tips
To avoid delays between successive benchmark test rounds, we configured the kernel to reuse the sockets in state TIME_WAIT as follows:
echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
Another problem difficult to debug was when the kernel maximum number of memory pages allocated to the TCP was reached. When, this happened the MigratoryData Client Benchmark remained up, but the client machine did not accept any new TCP connection. We figured out that we have to increase the maximum number of memory pages allocated to the TCP from 767586 (about 3 GB) to 2303190 (about 8.7 GB) as follows:
sysctl -w net.ipv4.tcp_mem="383865   511820   2303190"
Linux Kernel Tuning
A number of kernel tuning suggestions, mainly in terms of ports and socket descriptors, were already provided above. Now, we focus on kernel tuning suggestions for memory and performance optimization.
Kernel Memory Tuning
We used the Linux kernel version 3.9.4-1.el6.elrepo.x86_64 which consumed about 36 GB of kernel memory for the 12 million open sockets as shown by slabtop in one of the screenshots below.
Note — We started the benchmark tests using the default kernel 2.6.x of Centos 6.4. In version 2.6.x and other older kernel versions (until kernel version 3.7?), the kernel uses, besides the 36 GB memory for 12 million open sockets, another memory page (i.e. 4096 bytes) for each open socket. Even worse, it looks like there is a bug in the Linux kernel because this memory page used per open socket is not reported in /proc/meminfo which usually is not a big deal, but having lots of sockets one can observe a lot of memory vanishing in an inexplicable way. Therefore, a recent Linux kernel such as 3.9 optimizes the kernel memory with about 46 GB for 12 million sockets.
We used the following tuning related to the socket buffer sizes for our benchmark scenario:
sysctl -w net.ipv4.tcp_rmem="1024   4096   16384"
sysctl -w net.ipv4.tcp_wmem="1024   4096   16384"
sysctl -w net.ipv4.tcp_moderate_rcvbuf="0"
Balancing Hardware Interrupts
The server machine used a 10 Gbps network card Intel X520-DA2 having 24 tx/rx queues. We assigned each tx/rx queue to a different CPU core using smp_affinity. First, we identified the interrupts of our network interface named p1p1 as follows:
[root@t15 migratorydata]# grep p1p1 /proc/interrupts
77: 1684 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge  p1p1-TxRx-0
87: 1105 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge  p1p1-TxRx-10
100: 1072 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge  p1p1-TxRx-23
101:    2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge  p1p1
By default, the interrupts are assigned to the first CPU core. To balance the hardware interrupts of each tx/rx queue to a different CPU core, we used:
[root@t15 ~]# cat smp_affinity.sh
echo "1" >/proc/irq/77/smp_affinity
echo "10" >/proc/irq/81/smp_affinity
echo "100" >/proc/irq/85/smp_affinity
echo "1000" >/proc/irq/89/smp_affinity
echo "10000" >/proc/irq/93/smp_affinity
echo "100000" >/proc/irq/97/smp_affinity
echo "800000" >/proc/irq/100/smp_affinity
After running this smp_affinity script, one can see that the interrupts are now balanced on all 24 CPU cores:
[root@t15 ~]# grep p1p1 /proc/interrupts
77:  3002 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge p1p1-TxRx-0
78:  1453 123 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge p1p1-TxRx-1
79:  1475 0 437 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge p1p1-TxRx-2
80:  1453 0 0 123 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge p1p1-TxRx-3
81:  1462 0 0 0 127 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge p1p1-TxRx-4
82:  1457 0 0 0 0 123 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge p1p1-TxRx-5
83:  1456 0 0 0 0 0 125 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge p1p1-TxRx-6
84:  1453 0 0 0 0 0 0 123 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge p1p1-TxRx-7
85:  1453 0 0 0 0 0 0 0 123 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge p1p1-TxRx-8
86:  1453 0 0 0 0 0 0 0 0 123 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge p1p1-TxRx-9
87:  1497 0 0 0 0 0 0 0 0 0 130 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge p1p1-TxRx-10
88:  1453 0 0 0 0 0 0 0 0 0 0 123 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge p1p1-TxRx-11
89:  1454 0 0 0 0 0 0 0 0 0 0 0 123 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge p1p1-TxRx-12
90:  1453 0 0 0 0 0 0 0 0 0 0 0 0 123 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge p1p1-TxRx-13
91:  1454 0 0 0 0 0 0 0 0 0 0 0 0 0 123 0 0 0 0 0 0 0 0 0 PCI-MSI-edge p1p1-TxRx-14
92:  1454 0 0 0 0 0 0 0 0 0 0 0 0 0 0 123 0 0 0 0 0 0 0 0 PCI-MSI-edge p1p1-TxRx-15
93:  1456 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 123 0 0 0 0 0 0 0 PCI-MSI-edge p1p1-TxRx-16
94:  1453 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 123 0 0 0 0 0 0 PCI-MSI-edge p1p1-TxRx-17
95:  1453 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 123 0 0 0 0 0 PCI-MSI-edge p1p1-TxRx-18
96:  1453 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 123 0 0 0 0 PCI-MSI-edge p1p1-TxRx-19
97:  1453 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 123 0 0 0 PCI-MSI-edge p1p1-TxRx-20
98:  1453 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 123 0 0 PCI-MSI-edge p1p1-TxRx-21
99:  1453 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 123 0 PCI-MSI-edge p1p1-TxRx-22
100: 1453 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 123 PCI-MSI-edge p1p1-TxRx-23
101:    2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge p1p1
Benchmark Configuration
Please refer to the sections "Hardware and Setup" and "The Benchmark Scenario" in the initial blog post for precise details about the benchmark scenario and setup used by the involved components: MigratoryData Server, MigratoryData Client Benchmark, and MigratoryData Publisher Benchmark. Here we detail their configurations.
We used the following configuration of the MigratoryData server:
Memory = 55296 MB                       # Allocate 54 GB to the JVM
Listen = *:8800                         # Accept clients on port 8800
Monitor = JMX                           # Enabled JMX monitoring
BufferLimit.Send = 800                  # Set the initial buffer sizes to 800 bytes given
BufferLimit.Receive = 800               # that we publish messages of 512 bytes
PubSubTightlyCoupled = false            # Disable subscribe notifications to publishers
CacheSize = 0                           # Disable the cache
BenchmarkMode = true                    # See details below
The parameter BenchmarkMode is set on true to handle the clients in a realistic way. In fact, in order to take advantage of multiprocessor servers, the incoming users are separated in parallel internal groups. Each client is assigned to a group based on its IP address. Because, in our benchmark a lot of clients come from the same IP address, we use the parameter BenchmarkMode to distribute them to different internal groups of the MigratoryData server, just as it would happen in production.
JVM Tuning
Tuning the JVM and especially the Garbage Collector related parameters is outside the scope of this blog post. We recommend the book Java Performance on JVM tuning. In a screenshot below we provide all the JVM parameters used in the benchmark.
The JVM parameter UseCompressedOops compresses the 64-bit pointers and offers non-negligible memory optimization. The recent versions of JVM enable this parameter by default for the JVM heap sizes smaller than about 30 GB. But UseCompressedOops can be used only for a JVM heap size of maximum 32 GB. The JVM parameter ObjectAlignmentInBytes extends the compression to the JVM heap sizes larger than 32 GB. Using the parameters ObjectAlignmentInBytes and UseCompressedOops allowed us to benefit from pointer compression for the entire memory allocated to the MigratoryData server without the need to run two instances of the MigratoryData server each one using less than 32 GB. Therefore, we included the following JVM parameters:
MigratoryData Client Benchmark
We used 10 instances of MigratoryData Client Benchmark running on 10 client machines Dell SC1435 as described above. Each instance of MigratoryData Client Benchmark opened 1.2 million connections to the MigratoryData server. Along each connection MigratoryData Client Benchmark subscribed to a different subject. Hence, we had 12 million concurrent clients subscribing to 12 million concurrent subjects.
For example, the 1st of the 10 instances of MigratoryData Client Benchmark used the following configuration:
Memory = 9000                         # Allocate 8.7 GB to the JVM
ServerAddresses = 192.168.3.115:8800  # The address of the MigratoryData server
Clients = 1200000                     # Open 1.2 million subjects
Subjects = /p/s{1..1200000}/-         # Subscribe to /p/s1/-, /p/s2/-, ..., /p/s1200000/-
SubjectsPerClient = 1                 # Each client subscribes to one subject
LocalAddresses = 192.168.{10..30}.101 # Use these IP addresses for client sockets
BufferSize = 1024                     # Set the initial buffer size to 1024 bytes
OneToOne = true                       # client 1 subscribes to /p/s1,
# client 2 subscribes to /p/s2,
# client 1200000 subscribes to /p/s1200000
In the same way:
The 2nd instance of the MigratoryData Client Benchmark has been configured to open another 1.2 million concurrent connections and subscribed to the subjects from /p/s1200001/- to /p/s2400000/-
The 3rd instance of the MigratoryData Client Benchmark has been configured to open another 1.2 million concurrent connections and subscribed to the subjects from /p/s2400001/- to /p/s4800000/-
The 10th instance of the MigratoryData Client Benchmark has been configured to open another 1.2 million concurrent connections and subscribed to the subjects from /p/s10800001/- to /p/s12000000/-
Each instance of MigratoryData Client Benchmark displays every 5 seconds various information such as: the number of seconds since the start time, the minimum, maximum, mean, standard deviation for all messages received, the total number of messages received, and the frequency of the messages received during the last 5 seconds, as in the example below:
[2013-06-18, 15:03:11] [INFO] Uptime: 1,607 s
[2013-06-18, 15:03:11] [INFO] Latency min: 0 ms
[2013-06-18, 15:03:11] [INFO] Latency max: 2,024 ms
[2013-06-18, 15:03:11] [INFO] Latency mean: 269.68 ms
[2013-06-18, 15:03:11] [INFO] Latency standard deviation: 420.36 ms
[2013-06-18, 15:03:11] [INFO] 90% CI for the mean: [269.42 - 269.942] ms
[2013-06-18, 15:03:11] [INFO] Total messages: 7,089,699
[2013-06-18, 15:03:11] [INFO] Frequency: 20,905.80 messages/sec
The latency statistics are detailed in the section "Latency" below.
MigratoryData Publisher Benchmark
In our benchmark scenario, there are 200,000 messages per second published to the MigratoryData server. The payload of each message is a 512-byte string of random characters. We have developed a tool named MigratoryData Publisher Benchmark which is able to publish messages of a configurable size at a configurable frequency, for subjects selected randomly from a configurable set of subjects.
We ran 8 instances of MigratoryData Publisher Benchmark on 4 machines Dell SC1435 having the same specs as the client machines, two instances per machine. To achieve 200,000 messages per second, each instance published 25,000 messages per second.
For example, the 1st of the 8 instances of MigratoryData Publisher Benchmark used the following configuration:
PublisherType = p                     # The first prefix of the subjects
Subjects = /s{1..1500000}/-           # Publish messages for /p/s1/-, /p/s2/-, ..., p/s1500000
Frequency = 25000                     # Number of messages per second
MessageSize = 512                     # The number of bytes per message
The 2nd instance of the MigratoryData Publisher Benchmark has been configured to publish 25,000 messages per second where the subject of each message is randomly selected from the subjects between /p/s1500001/- to /p/s3000000/-
The 3rd instance of the MigratoryData Publisher Benchmark has been configured to publish 25,000 messages per second where the subject of each message is randomly selected from the subjects between /p/s3000001/- to /p/s4500000/-
The 8th instance of the MigratoryData Publisher Benchmark has been configured to publish 25,000 messages per second where the subject of each message is randomly selected from the subjects between /p/s10500000/- to /p/s12000000/-
Latency is defined here as the time needed for a message to propagate from the publisher to the client, via the MigratoryData server. In other words, the latency of a message is the difference between the time at which the message is sent by MigratoryData Publisher Benchmark to the MigratoryData server and the time at which the message is received by MigratoryData Client Benchmark from the MigratoryData server. All the client machines and publisher machines have the clocks synchronized using ntp (a few milliseconds difference might exist between the clocks, so you can observe a minimum latency of -1 millisecond due to such small clock differences).
We calculate the following latency statistics: maximum, mean, and standard deviation.
In order to calculate the latency statistics, one can use all latency values or use a reasonable sample size. The sampling approach provides statistically accurate results but it should be typically used when it is not possible to obtain all the values (e.g. election polling). Another disadvantage is that sampling cannot be used to compute the absolute maximum latency.
Because for certain systems, having a predictable latency – under a certain maximum – is more valuable than having low latency values in average, we chosen to calculate the latency statistics for all values, and so also obtaining the true maximum latency. While it might appear that computing the latency statistics for all values would add a certain performance overhead, in fact, one needs not to store the latency values, because the statistics can be calculated on-the-fly for all messages as follows:
numberOfMessages++;
maxLatency = maximum{maxLatency, newLatency}; /* max  */
deltaLatency = newLatency - meanLatency;
meanLatency = meanLatency + (deltaLatency / numberOfMessages); /* mean */
sumOfSquaredDelta += deltaLatency * (newLatency - meanLatency);
stdevLatency = sqrt(sumOfSquaredDelta / numberOfMessages); /* stdev */
Note — Note that we have ten instances of MigratoryData Client Benchmark and each instance computes the latency statistics only for the messages it processes. However, the latency values of each instance of MigratoryData Client Benchmark represent an entire fraction of the population (10%), not only a typical sample. Also, as shown in the screenshot below, all statistics are practically identical for all ten instances of MigratoryData Client Benchmark.
The Linux tools available for bandwidth calculation usually attempt to compute the bandwidth by capturing TCP packets. This approach does not scale for high data throughput and the bandwidth values produced are usually not accurate.
An accurate method for bandwidth calculation is to use the Receive Bytes / Packets and Transmit Bytes / Packets written by the Linux kernel to /proc/net/dev for the network interface used by the MigratoryData server. We compute the bandwidth by reading /proc/net/dev every 10 seconds. You can see the bandwidth transmitted by the MigratoryData server in one of the screenshots below.
These are the screenshots with the results obtained which have been detailed above.
In this post we presented the obstacles faced when scaling to 12 million concurrent connections in terms of kernel tuning, memory tuning, tools, and benchmarking methodology. More insights into how MigratoryData WebSocket Server itself is architectured to achieve this massive vertical scalability might be the subject of a new post.
This entry was posted in Benchmarks, C10M, High Scalability, migratorydata, Push Notifications, WebSockets on October 10, 2013 by Mihai Rotaru.
12 Million Concurrent Connections with MigratoryData WebSocket Server
We have recently completed a new performance benchmark which demonstrates that MigratoryData WebSocket Server is able to handle 12 million concurrent users from a single server Dell PowerEdge R610 while pushing a substantial amount of live data (1.015 gigabit per second). This benchmark scenario shows that MigratoryData WebSocket Server is ideal for infrastructures delivering real-time data to a huge number of users, especially for mobile push notifications infrastructures that are typically demanded by telecom customers with tens of millions users.
In this benchmark scenario, MigratoryData scales up to 12 million concurrent users from a single Dell PowerEdge R610 server while pushing up to 1.015 Gbps live data (each user receives a 512-byte message every minute). The CPU utilization diagram below shows that MigratoryData WebSocket Server scales linearly with the hardware.
According to the chart above, MigratoryData uses only 57% CPU to handle 12 million users. The remaining 43% CPU could be used to scale even more. However, we are limited by the RAM available on this machine (we use Centos 6.4 with standard Linux kernel and only the Linux kernel memory footprint for 12 million sockets is about 36 GB).
Also, the linearity of CPU utilization above indicates that MigratoryData WebSocket Server should normally scale vertically beyond 12 million concurrent users if additional RAM and CPU are available.
Detailed Results of MigratoryData WebSocket Server Running on a Single Dell R610 Server
In the table below, it is important to note that we've obtained the results using the default configuration of MigratoryData WebSocket Server, a fresh installation of Linux Centos 6.4 (without any kernel source code modification or other special tuning), and the standard network configuration (employing the default MTU 1500, etc).
Number of concurrent client connections 3,000,000 6,000,000 9,000,000 12,000,000
Number of messages per minute to each client 1 1 1 1
Total Messages Throughput 50,000 100,000 150,000 200,000
Average Latency (milliseconds) 5 35 92 268
Standard Deviation for Latency (milliseconds) 36 123 263 424
Maximum Latency (milliseconds) 640 951 1292 2024
Network Utilization 0.254 Gbps 0.507 Gbps 0.767 Gbps 1.015 Gbps
CPU Utilization (average) 14% 24% 39% 57%
RAM Memory Allocated to the Java JVM 54 GB 54 GB 54 GB 54 GB
Latency is defined here as the time needed for a message to propagate from the publisher to the client, via the MigratoryData server. In other words, the latency of a message is the difference between the time at which the message is sent by the benchmark publisher to the MigratoryData server and the time at which the message is received by the benchmark client from the MigratoryData server.
Network Intel X520-DA2
(10 Gbps)
Operating System Centos 6.4
Java Version Oracle (Sun) JRE 1.7
The benchmark clients and benchmark publishers ran on 14 identical Dell PowerEdge SC1435 servers as follows:
Four servers Dell SC1435 were used to run up to four instances of the benchmark publisher. For example, to publish 100,000 messages per second, we used four instances of the benchmark publisher on the four servers Dell SC1435, each publisher sending 25,000 messages per second.
Ten servers Dell SC1435 were used to run up to ten instances of the benchmark client. For example, to open 12,000,000 concurrent connections, we used ten instances of the benchmark client on the ten servers Dell SC1435, each client opening 1,200,000 concurrent connections.
The server Dell PowerEdge R610 (used to run a single instance of MigratoryData Server) and the 14 servers Dell PowerEdge SC1435 (used to run benchmark clients and benchmark publishers) were connected via a gigabit switch Dell PowerConnect 6224 enhanced with a 2-port 10 Gbps module.
Each client subscribes to a single different subject; for example, to achieve 12 million concurrent users, we used 12 million subjects.
Each client receives a message every minute; for example, to push a message per minute to 12 million concurrent users, the publisher sent 200,000 messages per second (the subject of each message was chosen randomly from the total of 12 million subjects)
We performed 4 benchmark tests corresponding to the 4 results summarized above, in order to simulate 3,000,000 / 6,000,000 / 9,000,000 / 12,000,000 concurrent users from a single instance of MigratoryData WebSocket Server.
The clock of the Dell R610 server (used to run MigratoryData Server) and the clocks of the 14 servers Dell SC1435 (used to run benchmark clients and benchmark publishers) were synchronized via ntpd. The latency was measured for all messages, not only for a sample. We've measured mean latency, maximum latency and standard deviation for the latency during 10 minutes and the results are reported above. We've also ran the most demanding scenario with 12 million concurrent connections during 6 hours and observed that MigratoryData WebSocket Server remains perfectly stable.
MigratoryData WebSocket Server and its APIs offer the possibility to build a high-availability cluster.
Each instance of MigratoryData WebSocket Server in the cluster runs independently from the other cluster members. It exchanges only negligible coordination information or, depending on the clustering type you configure, does not exchange any information at all with the other cluster members. Therefore, MigratoryData WebSocket Server offers linear horizontal scalability.
One can deploy a high-availability cluster of MigratoryData servers to achieve any number of concurrent users. For example, using the linear horizontal scalability of MigratoryData WebSocket Server and the 12 million vertical scalability demonstrated here, one could achieve say 60 million connections using a cluster with 5 instances of MigratoryData WebSocket Server running on 5 Dell PowerEdge R610 servers.
The implication of this is that, for the example above, in a production deployment, it is recommended to have at least 7-8 servers to achieve 60 million concurrent users such that, if a failure were to occur, each server will have enough reserve to accept part of the users of the cluster member which fails.
In 2010, we've achieved 1 million concurrent connections on a single 1U server. While handling 1 million concurrent connections on a small server still remains a challenge for the WebSocket servers' industry, we prove here that MigratoryData's WebSocket Server scales an order of magnitude higher and achieves 12 million concurrent connections on a single 1U server.
This entry was posted in Benchmarks, C10M, High Scalability, Push Notifications, WebSockets on June 20, 2013 by Mihai Rotaru.


Women Nobel Prize WinnersHow much do you know about female winners of the Nobel Prize?
For Kids: Famous Women World RulersThis quiz for kids will test their knowledge of women rulers throughout history.
Women Adventurers QuizHow much do you know about the bravest female explorers throughout history?
Women in HistoryHow much do you know about famous women in history?
Groundbreaking Women QuizSee how much you know about pioneers and trailblazers for women's rights.
Female Writers and Performers QuizTest your knowledge of famous female writers and performers.
Suffrage Movement QuizDo you know how women earned the right to vote in America? Take the quiz!
First Ladies QuizTake this quiz to see how much you know about the presidential first ladies of the United States.
Test Your Women's History IQFind out how much you know about these fabulous females by taking this quiz from the National PTA&reg.
Joan of Arc QuizJoan of Arc is the Patron Saint of France. See what else you know about this brave heroine.
Women PioneersWho was the first woman in space? Take this quiz and learn about women pioneers.
Women in PoliticsWhich state in the U.S. was the first to have women hold all five top elected positions? Find out!


text:
BenQ Named as the Official Gaming Monitor of the Ultimate Summer Showdown Beginning April 23, 2021 and Concluding with the Finale on July 8, 2021
MIAMI, FL — April 22, 2021 — Motorsport Games Inc. (NASDAQ: MSGM) ("Motorsport Games"), a leading racing game developer, publisher and esports ecosystem provider of official motorsport racing series throughout the world, today announced the Ultimate Summer Showdown. The four event esports series for NASCAR Heat 5 starts on April 30, 2021 and will culminate with a showpiece final on July 22, 2021. BenQ, a leading display technology company, has been named as the "Official Gaming Monitor Provider" for the Ultimate Summer Showdown.
Open qualification for the Ultimate Summer Showdown begins online within NASCAR Heat 5 on April 30, 2021, with players on the PlayStation®4 computer entertainment system and the Xbox One family of devices including the Xbox One X, to take part. The top 24 drivers from each platform will then be invited to take part in the live streamed event finals on Twitch, YouTube and Facebook. The first race of the event will see players take to Darlington Raceway.
"NASCAR Heat 5 has a strong, active community full of great racers and we cannot wait to give them another opportunity to showcase their talent on one of the biggest stages. The Ultimate Summer Showdown promises to build on the momentum of the recent Winter Heat Series," commented Jay Pennell, General Manager, eNASCAR Heat Pro League, Motorsport Games. "We are delighted to welcome BenQ as a partner for the series and look forward to working closely with the team to promote the races."
BenQ is a global leader in display technology, known for extraordinary color performance and groundbreaking monitor audio advancement. The company debuted the MOBIUZ gaming monitor series in 2020 to rave reviews and the newest model, EX3415R, has been engineered for high-end gaming audio-visual performance.
With the MOBIUZ EX3415R curved gaming monitor, BenQ answers the question of how far the limits of sim-racing immersion can be pushed. This monitor acts as a virtual windshield that teleports you to the races of your dreams with performance that blows your mind. The massive 34-inch IPS panel with 1900R curvature wraps your field of view, perfectly simulating the driver's first-person vantage point. Image optimizers, including the newly introduced Racing Mode, put you in the driver's seat and let you control the way your display looks for a truly customized performance. A 144Hz High Refresh Rate and 1ms Response Time take every turn with ease. Lastly, because it's just not the same without cheering fans and revving engines, the treVolo audio system surrounds you with a wall of powerful sound. These innovations elevate the world of sim racing to a more life-like experience.
"BenQ is excited to partner with Motorsport Games to connect with racing fans and deliver the immersive driving thrills and experiences they crave," said Jason Lee, BenQ America LCD Business Head. "Just as pro drivers demand the best in their equipment, we strive to provide sim-racing enthusiasts with an ultrawide curved gaming monitor that delivers all the intensity and immersion of being on the track."
Open qualifiers
Darlington Raceway: April 30, 2021 (12:01 a.m. ET) – May 6, 2021 (11:59 p.m. ET)
Texas Motor Speedway: May 21, 2021 (12:01 a.m. ET) – May 27, 2021 (11:59 p.m. ET)
Pocono Raceway: June 11, 2021 (12:01 a.m. ET) – June 17, 2021 (11:59 p.m. ET)
Atlanta Motor Speedway: July 2, 2021 (12:01 a.m. ET) – July 8, 2021 (11:59 p.m. ET)
Live Streamed Event Finals
Race 1: Thursday, May 20 2021 at Darlington Raceway
Race 2: Thursday, June 10, 2021 at Texas Motor Speedway
Race 3: Thursday, July 1, 2021 at Pocono Raceway
Race 4 (finale): Thursday, July 22, 2021 at Atlanta Motor Speedway
To take part in the Ultimate Summer Showdown, players will need a copy of NASCAR Heat 5, which is available now on the PS4, Xbox One and on PC via Steam. The game delivers the ultimate NASCAR video game experience, with 34 authentic tracks across the country in single player, two-player split screen, and online multiplayer for up to 40 racers.
To stay up to date with the latest news about NASCAR Heat follow the game on Twitter, Facebook, YouTube and Instagram.
SUNSETTING NASCAR HEAT ...

redpajama_set_name: RedPajamaCommonCrawl


text:
Home | India | Top Headlines | Coronavirus impact: the world's richest lose $ 444 billion after a hellish week for markets | International Business News
India Top Headlines
Coronavirus impact: the world's richest lose $ 444 billion after a hellish week for markets | International Business News
| Mar 1, 2020
NEW YORK: Last week was expensive for most investors, including billionaires.
The combined fortunes of the 500 richest people in the world fell by $ 444 billion as the coronavirus continued to spread, and spread fear, shaking stock markets around the world. The Dow Jones Industrial Average fell more than 12%, the biggest five-day drop since the depths of the 2008 financial crisis, in a defeat that vaporized more than $ 6 billion of global stocks.
The erased more than erased the $ 78 billion earnings that the 500 richest people had accumulated from the beginning of the year until last week, according to the Bloomberg billionaire index.
The three richest people in the world, Jeff Bezos of Amazon.com Inc, Microsoft Corp co-founder Bill Gates and LVMH president Bernard Arnault, suffered the greatest losses, with a combined wealth of approximately $ 30 billion.
Elon Musk, the 25th richest person in the world, recorded the fourth largest weekly loss, $ 9 billion, as the shares of his Tesla Inc slipped after a sharp rise to start the year. It has still increased $ 8.8 billion in 2020 and has a net worth of $ 36.3 billion.
Health officials are struggling to contain the virus, which can cause a disease similar to life-threatening pneumonia in a minority of patients and spread from other people who look healthy. The World Health Organization (WHO) has so far refrained from declaring it a pandemic.
About 80% of billionaires in the Bloomberg wealth ranking are now in red numbers this year, including those whose businesses have been swept away by the global drama. Carnival Corp president Micky Arison lost a billion dollars this week, as the world's largest cruise line operator held tourists aboard one of his ships in Japan, where at least five passengers died.
NCP rejected the election of Shiv Sena, relied on an ally to elect Param Bir Singh as police commissioner of Mumbai | Mumbai News
Why climate migration does not have to be a crisis | Opinion - analysis
Telangana ministry yet to function
China's economic growth slumps,2012 will see even sharper slowdown
Kollywood news (Mar 2) : Kamal Hassan and Vijay steal show today!
U.S to support Taliban office in Qatar,agrees to release prisoners
Shootout at Wadala (SAW) : Sunny Leone and Sophie Choudry dazzle at promotion


Horsehair hat. Satin and sequin band around the crown. Topped with metallic trimmed mesh ribbons and sequin trimmed mesh leaf-forms. Rhinestone brooch accents.
Size: Crown size 22 1/2" / Brim width 6.25" / Crown height 4" / Brim to brim 20"


SPONSOR: Get $20 off your first Simple Contacts order by going to https://www.simplecontacts.... or entering VECTOR20 at checkout.
But, I've been able to use the extra time to collect extra information, and from people beyond my own personal use cases and, frankly, the typical tech-sphere considerations.
I've also got to use the latest watchOS update, which adds ECG to Hong Kong and 19 countries across Europe, including France, Germany, Italy, Spain and the United Kingdom. Also, and more applicable to me, some new watch face variations for Nike and Hermes, and AirPods 2 support, which makes a great pairing even better. Like, I don't know dark chocolate and crunchy peanut butter?
There are also new watch bands for the spring. This year's collection includes new sport loops, new sport bands, new Hermès straps, new almost everything.
Let's take a look at all of that!
iPhone XS Review — 7 Months Later - Duration: 14 minutes.
iOS 13: The Breakdown - Duration: 10 minutes.
Has the Samsung Galaxy Fold Failed? - Duration: 11 minutes.
Apple vs. Qualcomm: Settlement & Fallout - Duration: 12 minutes.
Apple Watch Series 4 Review: 6 Months Later - Duration: 16 minutes.
iPad vs. Air vs. mini vs. Pro: Which should you buy? - Duration: 10 minutes.
CBC vs. Apple: iPhone Data Recovery Fact Check - Duration: 14 minutes.
Where is FaceTime for Android? - Duration: 12 minutes.
Has the iPhone Camera Failed? - Duration: 16 minutes.
AirPods 2 (2019) Review: 10 Days Later! - Duration: 12 minutes.
Powerbeats Pro vs. AirPods 2: Which Should You Buy? - Duration: 7 minutes, 7 seconds.
Huawei P30 vs. iPhone XS — From the Android Expert - Duration: 10 minutes.
OMG the iPhone XR Display is...?! - Duration: 12 minutes.
Why Apple is DOOMED. Finally* - Duration: 12 minutes.
iPad Pro (2018) Review: Two weeks later! - Duration: 13 minutes.
NEW iPad Pro (2018) Hands-On - Duration: 5 minutes, 12 seconds.
iPhone XS and Max Review - Duration: 25 minutes.
iPhone XR Review - Duration: 26 minutes.
Apple Watch 4 Review: Two weeks later! - Duration: 8 minutes, 9 seconds.
iPhone XR Review: One month later - Duration: 12 minutes.
The MacBook ARM - Duration: 19 minutes.
Where is iMessage for Android? - Duration: 8 minutes, 46 seconds.
Apple Watch Series 4 Hands On - Duration: 4 minutes, 53 seconds.
The iOS MacBook — or iBook - Duration: 6 minutes, 56 seconds.


text:
Issuance of industrial unit establishment permits rises 25% in 2 months
TEHRAN- Issuance of permits for setting up industrial units in Iran increased 25 percent during the first two months of the current Iranian calendar year (March 21-May 21) compared to the same period of time in the past year, IRNA reported on Sunday citing the data released by the Ministry of Industry, Mining and Trade.
As reported, 3,599 permits have been issued during the two-month period of this year, rising from 2,879 permits issued in the first two months of the previous year.
The units that have received permits are estimated to create jobs for 79,913 people.
As previously announced by the Ministry of Industry, Mining and Trade, issuance of permits for setting up industrial units in Iran increased 16 percent in the past Iranian calendar year (ended on March 20, 2019).
As reported, most of the issued permits were related to the food industries; and Semnan, Qom and Khorasan Razavi were the provinces receiving the highest number of the permits.
Industry, Mining and Trade Minister Reza Rahmani has announced that some 2,200 industrial units will come back to the production cycle by the end of the current Iranian year.
The minister said that last year, 1,900 inactive units were planned to restart activity, but the planning came true for 1,700 units and led to creation of 28,000 jobs.
He further said, "We have some plans for removing the problems of the units which are facing serious challenges."
There are some units that enjoy high potential and capacities, but due to some problems such as inadequate working capital they cannot work with full capacity, the minister noted, adding, "Our priority in the current year is to facilitate the condition for such units."
As previously announced by Deputy Industry Minister Mohsen Salehinia, during the current year the government plans to provide facilities under the framework of subsidies for projects with more than 60 percent of physical development, to supply working capital of firms, to renovate production units and etc.
He said that 360 trillion rials (about $857.1 million) of facilities will be granted to the industrial units in this year in the framework of production flourishing plan.
MA/MA
Industrial-permits
industrial units
Reza Rahmani


text:
Perspectives Volume 40, Number 1 - Page 9
soil of nutrients. Our goal was to help smallholders increase use efficiency, thereby reducing losses and earning more money. We began by researching the effects of coating urea and changing its physical form. This research has achieved tremendous success over the years, leading to climate-smart technologies such as urea deep placement (UDP). Our nitrogen program is still making great strides. Phosphorus fertilizer also presented a challenge because the tropical soils' composition in South America and subSaharan Africa fixed available phosphorus from water-soluble phosphorus fertilizers, thus drastically reducing its availability to plants. When solving a problem, we try to benefit the farmer directly. Our scientists looked for ways for farmers to directly apply locally available phosphate rock to their soils. This method would make phosphorus accessible and affordable for smallholder farmers. Our research shows that this is a sustainable possibility, and we are continuing to learn more about making this approach economical for farmers. The early structure of IFDC was like an "assembly line" for developing these technologies and taking them to farmers. Three divisions worked together to get new fertilizer innovations out of the lab and into the field. Most projects began in the Agro-Economic Division, which identified farmer needs and envisioned efficiency-increasing solutions. The Fertilizer Technology Division made these ideas into reality. Part of this division's work was solving technical and economic issues related to new production technologies. Finally, the Outreach Division rolled out the knowledge and products, taking them to farmers, engineers and other agriculture professionals. "The early structure of IFDC was like an 'assembly line' for developing these technologies and taking them to farmers." This structure gave way to great success in IFDC's first decade. But by the mid-1980s, political environments and agricultural technology were changing and advancing rapidly. The time came for IFDC to make its first major realignment, a move that would define the organization's impact for the next 30 years. IFDC Magazine 9


New Hairstyles 2018 In Africa – This New Hairstyles 2018 In Africa gallery was upload on August, 6 2018 by admin. Here latest New Hairstyles 2018 In Africa gallery collection. Download other wallpapers about New Hairstyles 2018 In Africa in our other reviews. Click on images to tap New Hairstyles 2018 In Africa in high resolution.
You can print this New Hairstyles 2018 In Africa photos for your collection. Don't forget to rate and comment if you interest with this New Hairstyles 2018 In Africa photos. Please contact us if it contains a copyright.
Related Post "New Hairstyles 2018 In Africa"


Is your party going to be frozen over? There's no business like snow business at Party Source in Perth. Whether it's snowflakes or Frosty, we can help you create a whimsical winter wonderland for your next event.


text:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">

<TITLE>
Uses of Package com.netmera.mobile
</TITLE>

<META NAME="date" CONTENT="2013-10-05">

<LINK REL ="stylesheet" TYPE="text/css" HREF="../../../stylesheet.css" TITLE="Style">

      Overview
      Package
      Class
    Use
      Tree
      Deprecated
      Help

 PREV
 NEXT

  FRAMES
 NO FRAMES

  All Classes

Uses of Package
com.netmera.mobile

Packages that use com.netmera.mobile

com.netmera.mobile

com.netmera.mobile.model

Classes in com.netmera.mobile used by com.netmera.mobile

BaseContent

BasePush

          Deprecated. use NetmeraPushObject to create push notifications. After
             preparing, use NetmeraPushSender to send this
             notification. BasePush class is an abstract class to implement
             different channel notifications like NetmeraIOSPush,
             NetmeraAndroidPush and NetmeraPush By the help of these classes
             push notifications can be send different channels.

NetmeraActivity

          This class must be extended from push notification activity of your
 application.

NetmeraCache.CacheType

          Types of the Cache to use while searching.

NetmeraCallback

          NetmeraCallback is used to run code with response data coming from Netmera
 after the background thread finishes running.

NetmeraContent

          The NetmeraContent object is used to run CRUD operations over the data.

NetmeraDeviceDetail

          The NetmeraDeviceDetail class is used to when registering and unregistering
 devices to specified groups or getting the detailed information of a
 registered device.

NetmeraException

NetmeraException.ErrorCode

NetmeraGeofence

          Used to control Netmera Geofence Settings.

NetmeraGeoLocation

          NetmeraGeoLocation is used to create location with the given latitude and
 longitude values.

NetmeraJavascriptInterface

          This class is used to inject the given Java object into rich push WebView.

NetmeraLocalSession

          This class is used to ease the use of SharedPreferences.

NetmeraMedia

          NetmeraMedia object is used to create media content.

NetmeraMedia.PhotoSize

          Type of the photo size.

NetmeraPopup

          This is the class to manage, see and show pop-ups.

NetmeraPushObject

          NetmeraPushObject is the main Push object in Netmera.

NetmeraRichPushObject

          This is the class to manage and create incoming/created rich push
 notifications that is inherited from NetmeraPushObject.

NetmeraService

          NetmeraService is used to get NetmeraContent object by its search()
 and get() methods.

NetmeraService.SortOrder

          Sort order definitions

NetmeraUser

          NetmeraUser object is for managing users of the application.

NetworkConnectivityListener.State

Classes in com.netmera.mobile used by com.netmera.mobile.model

NetmeraService.SortOrder

          Sort order definitions

      Overview
      Package
      Class
    Use
      Tree
      Deprecated
      Help

 PREV
 NEXT

  FRAMES
 NO FRAMES

  All Classes

redpajama_set_name: RedPajamaGithub


We have just posted two new prophecies for Wales on the Words about Wales page. Just click on the link to see them, and the whole collection.
If you would like to receive an email notification each time a new word is listed, you can use sign-up section at the top right on the page.
Jumping for joy in newfound freedom...!
After 5 months of being kept inside sheds, the cows were turned out this beautiful morning at ParcyMarl, our friend Moira Williams' farm in Pembrokeshire. This video clip shows their exuberance as they race onto the fresh green field and frolic in the sun, jumping for joy and kicking up their heels in sheer delight. It is such a simple yet happy event, and makes a wonderful picture of our freedom in Christ.
After five months of enclosure, eating silage and processed feed, to be able to feast on fresh green grass in the full sunshine is worth getting excited about! Imagine a sweet ripe peach you've just plucked from the tree yourself as compared to a can o f split pea soup or spaghetti! It's like taking out the middle man or process, and going directly to the source.
As I dwelt on this the Lord began to impress on my spirit some things about our need for direct relationship with him through the word and prayer. I love good preaching and teaching, and we are all indebted to many wonderful saints over the years that helped to establish us in our faith, especially in the early days. Receiving sound doctrine and teaching "processed" by others is vitally important.
However this is no substitute for personal time alone with God - in prayer, study and meditation in His word, and times of worship. It is in these times, when we purposefully make ourselves available to Him, that God can speak to us clearly.
These experiences of direct revelation are what really ground us into the bedrock of our faith. This should be our daily bread. It is this private, one-on-one relationship with the Holy Spirit that enables us to thrive and prosper in our soul.
Jesus said in John 14:21, "He who has My commandments and keeps them, it is he who loves Me. And he who loves Me will be loved by My Father, and I will love him and manifest Myself to him." And then in verse He adds, "If anyone loves Me, he will keep My word; and My Father will love him, and We will come to him and make Our home with him."
Just as these cows are delighted with fresh green grass, so can and should we be delighted in God's presence in us. He loves to reveal Himself to us, and nurture us from day to day with His truth. This is the essence of our life in Him. He dwells in and empowers us, and shows us Himself and His ways. He is the source of our joy and our strength.
Psalm 1:2-3 says, "But his delight is in the law of the Lord, and in His law he meditates day and night. He shall be like a tree planted by the rivers of water, that brings forth its fruit in its season, whose leaf also shall not wither; and whatever he does shall prosper."
May each of us know the joy of His closeness each and every day.
I love it when the Lord interrupts me in my readings and highlights things. This phrase in Job 42:7 just stopped me in my tracks and made my heart churn! "After the Lord had said these things to Job, he said to Eliphaz the Temanite, 'I am angry with you and your two friends, because you have not spoken the truth about me, as my servant Job has.'"
And then in the very next verse, God says again, "...My servant Job will pray for you, and I will accept him... You have not spoken the truth about me, as my servant Job has."
How easy it is for us to talk about the things of the Lord, and that is a good and right thing to do. But I realized that once we get going sometimes, how easily our words just seem to pour out. And more than once I've heard myself saying things and wondered, "Where did that come from? Is that really true?"
As I reflected on this in my own life, and considered how this may have been happening in God's people since Job's times, I really had something substantial to pray about. Could this be one of the reasons that the church seems to have such minimal effect in our modern society? So we can and should boldly pray, "Lord, teach us to speak your truth."
The Lord in essence says to Job's friends that He will accept Job's prayer for them because he has spoken the truth about Him. So it seems that the more accurately we learn to speak truth about Him, the more effective our lives and our prayers will become for His glory.
As so just how do we get more truth in us to strengthen our prayers? When Jesus prayed for the disciples in John 17:17, He said, "Sanctify them by Your truth; Your word is truth." God's word is our best ammunition!
Proverbs chapter 2 has some wise advice for us.
5 Then you will understand the fear of the Lord, and find the knowledge of God."
On Monday we drove to Swansea to meet with some people at the Bible College of Wales. Before they arrived, we had time to meet some of the new students and get to know a little about them. What a positive delight to find these young folks from many nations with such hunger to learn God's truth! What a gift and encouragement to sense their dedication and determination to be trained and equipped to serve God in their home countries and wherever else He choose to send them from there.
This is now the fourth group of students starting their 12-week course. And what joy to see such vibrant fruit of the prayers of many saints over the years that have been praying for this hallowed place.
This property was originally acquired and consecrated through the faith and prayers of Rees Howells and his co-workers nearly one hundred years ago. And when it was put on the market several years ago, it was once again through the faith and prayers of many people on the ground here in Wales, and especially our friends from Cornerstone Community Church of Singapore who took the bold step of faith, that made this new chapter in its history and re-dedication to the Lord's service possible.
Let's pray that each one of the students to go through this precious BCW training be infused with truth about the living God in the person of Jesus Christ and the power of the Holy Spirit. And may we all be continually prepared day by day to speak His truth in love to one another and the world around us.
Fresh wind in our sails. New local history and the plot thickens!
You can imagine our surprise to have just discovered that the site of our Cilfowyr Chapel has been a place dedicated to prayer and worship since at least 1547, and probably earlier than that. I just found out about this from our friend David Pike who visited us on Monday.
He directed me to the Dyfed Archaeological Trust website which carried out a geophysical survey in late 2015. There is a lot of information in their detailed and illustrated report which you can read at this link. But what caught my attention were the few details about a "chapel-of-ease."
Their report says, "Site of former chapel-of-ease to Manordeifi, mentioned in source from 1547, when it was a free chapel of Manordeifi parish. It was a donative free chapel, established by the patron of the parish. ... we do know the building was noted in 1721 as being disused and had become the home of '…fowles and jackdaws' (parish magazine extract)."
It happens that Tuesday morning, just by "chance", I snapped this picture of a red-legged partridge which disappeared through the hedge into the highlighted area, before I'd had time to read through this report!
According to Wikipedia, "A chapel of ease (or chapel-of-ease) is a church building other than the parish church, built within the bounds of a parish... Chapels of ease are sometimes associated with large manor houses, where they provide a convenient place of worship for the family of the manor, and for the domestic and rural staff of the house and the estate.
What I find exciting is how close this is to where we have been praying all these years. You can see Cilfowyr across the field less than 150 yards away. And people were spending time in prayer and worship here around 500 years ago! I suspect there wouldn't have been many formal services here, but it was a "free chapel", a place open for folks to come and go as they pleased and spend time with God. Back in those days when life was not as easy as we know it today, people may well have frequented this place a lot more than we might expect.
Early on in our time here while praying out in the cemetery one day, when it suddenly dawned on me that many of those buried there surely prayed many times for their own children, and also for their grandchildren, and perhaps even their grandchildren's children. And the Lord impressed on me how I could just say "Amen" to the prayers of those gone before us. I think every prayer ever prayed is still ringing around in heaven where God never forgets, resonating all throughout eternity.
Hebrews 12:1 says, "Therefore we also, since we are surrounded by so great a cloud of witnesses, let us lay aside every weight, and the sin which so easily ensnares us, and let us run with endurance the race that is set before us." And at times we have clearly felt our prayer joining the prayers of those gone before us in an unending concert before the throne of God. And now we know that people have been gathering in these fields since at least the early 1500's, not in the 1680's as we have always thought. And consequently we now realize that our prayers are strengthened by another 200 years worth of prayers!
When King Solomon prayed at the dedication of the temple in 1 Kings 8:59 he said, "And may these words of mine, with which I have made supplication before the Lord, be near the Lord our God day and night, that He may maintain the cause of His servant and the cause of His people Israel, as each day may require." What a glorious joy and privilege to join in with the songs and cries of millions upon millions of saints throughout the ages.
And so may we all echo the words of Solomon's heart in verse 52, "that Your eyes may be open to the supplication of Your servant and the supplication of Your people Israel, to listen to them whenever they call to You."
And also of course, let's add in the greatest revival prayer of all time in the words of Jesus, "Your kingdom come. Your will be done On earth as it is in heaven."
"And he arose and came to his father. But when he was still a great way off, his father saw him and had compassion, and ran and fell on his neck and kissed him." When Jesus tells the story of the prodigal son in Luke 15, this detail in verse 20 always stands out in a powerful way.
It seems obvious that the father has been constantly missing his son, and has always kept his eyes open for the slightest sign of his return. When he saw a movement still "a great way off" he must have watched intently until he recognized his son, and then took off running to receive him home again at long last.
This is a beautiful painting of such a precious scene. I can so clearly see myself in that most amazing of moments years ago. How about you? What a glorious and loving Heavenly Father was watching out for us!
Last week I mentioned how Exodus 4:31 says that had visited his His people and looked on their affliction. It seems like it was more of a general observation. But in contrast this New Testament story shows a more direct and personal watching out for homeward bound individuals. Earlier in this chapter, Jesus asks in verse 4, "What man of you, having a hundred sheep, if he loses one of them, does not leave the ninety-nine in the wilderness, and go after the one which is lost until he finds it?"
In the context of our ongoing prayers for revival, I found myself considering the aspect of focus. Of course we want to continually keep the "big picture" before the Lord in our intercession. But there is also a need for some specific targeting of certain individuals. Perhaps from time to time the Lord might bring to mind one person in particular, or maybe it changes as do people's circumstances.
In Rees Howells, Intercessor, there is an excellent example of this when Rees is directed by the Holy Spirit from general prayers to pray for a sick young drunkard, Will Battery. You can read the story if you just follow this link and scroll down to chapter 6, "Loving An Outcast".
We wanted to encourage you, along with ourselves, to ask the Lord to give you specific assignments to pray for individuals, and include that in your times of prayer. And may we too find the Holy Spirit travailing in us for the lost and needy.


text:
Joe Smith is an inductee (Honor Swimmer) of the International Marathon Swimming Hall of Fame
Joe Smith is an Honor Swimmer in the International Marathon Swimming Hall of Fame
Joe Smith (born 1933) of Deal, Kent is a British marathon swimmer, originally from Rochdale, who was inducted as an Honor Swimmer in 2005 into the International Marathon Swimming Hall of Fame.
English Channel Career
After two failed English Channel attempts in 1961 and 1962, Smith became the oldest Briton to conquer the English Channel in 1999 at the age of 65. 37 years after his first attempt, he joined the Half Century Club with his successful crossing from Shakespeare Cliffs in England to Sangatte, west of Calais in France in 14 hours 9 minutes, escorted by Duncan Taylor.
The former lifeguard and swimming pool manager won most of the British Long Distance Swimming Association races during the late 1950's and early 1960 including races in Windermere.
He served as a director of the Channel Swimming Association.
He was British Long Distance Swimming Association's Veteran Swimmer of the Year and was awarded an ARP British Gold Hero's Medal.
He swims daily and carried the Olympic Torch through Deal, Kent in the run-up to the 2012 London Olympic Games.
Oldest Individuals to Cross the English Channel
1. Otto Thaning (South Africa), 73 years, E/F, 12 hours 52 minutes (2014)
2. Cyril Baldock (Australia), 70 years 9 months, E/F, 12 hours 45 minutes (2014)
3. Roger Allsopp (England), 70 years 4 months, E/F, in 17 hours 51 minutes (2011)
4. George Brunstad (USA), 70 years 3 days, E/F, in 15 hours 59 minutes (2004)
5. Chris Shapland (Australia), 69 years, E/F, in 16 hours 28 minutes (2016)
6. Donald Riddington (Australia), 68, E/F, in 19 hours 45 minutes (2013)
7. Clifford Batt (Australia), 67, F/E, in 18 hours 37 minutes (1987)
8. Ashby Harper (USA), 65, E/F, in 13 hours 52 minutes (1982)
9. Joe Smith (England), 65, E/F, in 14 hours 9 minutes (1999)
10. Roger Allsopp (England), 65, E/F, in 15 hours 30 minutes (2006)
11. Sue Oldham (Australia), 65, E/F, in 17 hours 11 minutes (2010)
Doc Counsilman Helped Usher In An Increasingly Graying Trend In Channel Swimming
How Far Is The English Channel?
International Marathon Swimming Hall of Fame
Standing Up To The Tests Of Time
How Old Can They Go? Very Old
Chris Shapland En Route To Proving Swimming Is Lifelong
Retrieved from "https://openwaterpedia.com/index.php?title=Joe_Smith&oldid=556081"