I am trying to spawn a mapreduce job using the mrjob library from AWS Lambda. The job takes longer than the 5 minute Lambda time limit, so I want to execute a remote job. Using the paramiko package, I ssh'd onto the server and ran a nohup command to spawn a background job, but this still waits until the end of job. Is there anyway to do this with Lambda?
Instead of using SSH I've solved this before by pushing a message onto a SQS queue that a process on the server monitors. This is much simpler to implement and avoids needing to keep credentials in the lambda function or place it in a VPC.
How to execute a job on a server on Lambda without waiting for the response?