How to launch a EC2 instance and run a task on-demand
did you consider using Amazon SageMaker? It's easy to configure repeatable training jobs. I'd love to hear your feedback and answer any questions.
We are developing an deep learning application on AWS. The automation of model training is quite challenging. The first step is to prepare the data for training which involves generating a huge 3D  numpy array (> 100GB) from hundreds of thousands of small 2D numpy arrays. The small numpy arrays are saved in S3 bucket as small files. After the conversion, the big numpy array will saved to another S3 bucket. The training script will pick up the big 3D numpy array before the training. Since lambda function has a memory limit so we will have to launch an EC2 instance manually. 
I wonder what is the best practice to launch a EC2 instance from external and run the python script on the instance to do the data loading and transformation? 
The whole workflow will probably be automated using AWS step function.