How to launch a EC2 instance and run a task on-demand
You can use a Configuration set in a Cloud formation template to define steps where you can perform copies or run scripts or commands upon start up of your EC2 resource. You can even run it transiently so that it shut itself down after all operations are completed.
You can then use a script or Lambda function to execute the Cloudformation Stack
We are developing an deep learning application on AWS. The automation of model training is quite challenging. The first step is to prepare the data for training which involves generating a huge 3D  numpy array (> 100GB) from hundreds of thousands of small 2D numpy arrays. The small numpy arrays are saved in S3 bucket as small files. After the conversion, the big numpy array will saved to another S3 bucket. The training script will pick up the big 3D numpy array before the training. Since lambda function has a memory limit so we will have to launch an EC2 instance manually. 
I wonder what is the best practice to launch a EC2 instance from external and run the python script on the instance to do the data loading and transformation? 
The whole workflow will probably be automated using AWS step function.