I have a Python algorithm that can be parallelized fairly easily.
I don't have the resources locally to run the whole thing in an acceptable time frame.
For each work unit, I would like to be able to:

Launch an AWS instance (EC2?)
Send input data to the instance
Run the Python code with the data as input
Return the result and aggregate it when all instances are done

What is the best way to do this?
Is AWS Lambda used for this purpose? Can this be done only with Boto3?
I am completely lost here.
Thank you
A common architecture for running tasks in parallel is:

Put inputs into an Amazon SQS queue
Run workers on multiple Amazon EC2 instances that:


Retrieve a message from the SQS queue
Process the data
Write results to Amazon S3
Delete the message from the SQS queue (to signify that the job is complete)


You can then retrieve all the results from Amazon S3. Depending on their format, you could even use Amazon Athena to run SQL queries against all the output files simultaneously.
You could even run multiple workers on the same instance if each worker is single-threaded and there is spare CPU available.
Run parallel Python code on multiple AWS instances