I have a large manifest file containing about 460,000 entries (all S3 files) that I wish to load to Redshift. Due to issues beyond my control a few (maybe a dozen or more) of these entries contain bad JSON that will cause a COPY command to fail if I pass in the entire manifest at once. Using COPY with a key prefix will also fail in the same way.
To get around this I have written a Python script that will go through the manifest file one URL at a time and issue a COPY command for each one using psycopg2. The script will additionally catch and log any errors to ensure that the script runs even when it comes across a bad file, and allows us to locate and fix the bad files.
The script has been running for a little more than a week now on a spare EC2 instance and is only around 75% complete. I'd like to lower the run time, because this script will be used again. 
My understanding of Redshift is that COPY commands are executed in parallel, and with that I had an idea - will splitting the manifest file into smaller chunks and then running the script each of those chunks reduce the time it takes to load all the files?
COPY command can load multiple files in parallel very fast and efficiently. So when you run one COPY command for each file in your python file, that's going to take a lot of time since you are not taking advantage of parallel loading.
So maybe you can write a script to find bad JSONs in your manifest and kick them out and run a single COPY with the new clean manifest? 
Or like you suggested, I would recommend splitting manifest file into small chunks so that COPY can run for multiple files at a time. (NOT a single COPY command for each file)
Using multiple manifest files to load to Redshift from S3?