Question: I'm looking for a tool/framework (preferably in Java if it involves coding) that allows me to load test a server. I need to be able to launch multiple EC2 instances, hit the server hard for a period of time, log the response times to something that can be graphed (i.e. CloudWatch?), and then shutdown the instances. Automation for the tests is also needed.

I haven't had a lot of experience with load testing - I've tried out using the Jmeter GUI for a very simple test to hit a URL, and I've been researching how Grinder works. Could either of these allow creation/shutdown of EC2 instances in their tests? Is there a better option out there to do this?

Thanks.

Answer: 
Before anything, I would recommend you to think about an important point: 

What is the main purpose behind your test? 

Try to define your SLO/SLIs first (KPIs).  

For example if the KPI is "throughput", then you need to think about what should be your scaling policy and allow Autoscaling to do the Adding/Removing instance job for you. 

Obviously you can Add and Terminate the instances using different tools but at the end, most of them will use AWS client to do it.  

So, if you want everything to be automated, my suggestion would be a Scala test which uses: 

[Gatling](https://gatling.io/) + [AWScala](https://github.com/seratch/AWScala) to prepare and shutdown the stack and instances. 

By this, you can create your stacks, monitor the response time both on Gatling and also for example at ELB levels on CloudWatch. 

Last but not the least, Load Tests are usually expensive and is not happening very often, so maybe you don't need everything automated. Just create the stacks manually as you wish, define the scaling policy and then do the Load Testing and monitoring.
There are several ways you can kick off and terminate an AWS instance, to wit:

* [CreateInstance](http://docs.aws.amazon.com/opsworks/latest/APIReference/API_CreateInstance.html) and [TerminateInstances](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_TerminateInstances.html) methods of the [Amazon EC2 REST API](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/Welcome.html), you can use "normal" JMeter's [HTTP Request](http://jmeter.apache.org/usermanual/component_reference.html#HTTP_Request) sampler to perform these pre-requisite steps
* There is [`amazon-cli`](https://aws.amazon.com/cli/) utility which you can use directly from the JMeter's script via i.e. [OS Process Sampler](https://www.blazemeter.com/blog/how-run-external-commands-and-programs-locally-and-remotely-jmeter)
* There is [AWS SDK for Java](https://aws.amazon.com/sdk-for-java/), given you have the relevant libraries in [JMeter Classpath](http://jmeter.apache.org/usermanual/get-started.html#classpath) you should be able to write Groovy (or Java) code in the [JSR223 Sampler](http://jmeter.apache.org/usermanual/component_reference.html#JSR223_Sampler)

---

If you're looking for a code-based load testing tool you can also check out [Gatling](https://gatling.io/) which provides [Scala-based DSL](https://gatling.io/docs/current/cheat-sheet/) for creating the load.