python/video - removing frames without re-encoding
Since modern video coding standards uses inter frame prediction, general solution of removing frame without re-encoding is not exist. (Removing reference picture breaks inter prediction, so only non-reference picture can be removed.)
I understand this might be a trivial question, but so far I had no luck with the various solutions I tried and I'm sure there must be a convenient way to achieve this.  
How would I proceed removing frames/milliseconds from a video file without slicing, merging and re-encoding?
All solutions I found involved exporting various times to various formats, and I'm hopeful there will be no need to do so.  
With ffmpeg/avconv it's necessary to convert the temporary streams to .ts, then concatenate, and finally re-encode in the original format.  
Python library MoviePy seemed to do quite exactly what I needed but:  

The cutout function returns a file which can not be exported, as the write_videofile function tries and fails to fetch the removed frames
If I instead slice the original file into various clips and then merge them with concatenate_videoclips, the export doesn't fail but takes twice the length of the video. The resulting video has then a faster frame-rate, with only the cue points for the concatenated videos being timely placed, and audio playing at normal speed. It's also worth noting that the output file, despite being 5-7% shorter in duration, is about 15% bigger.

Is there any other library I'm not aware of I might look into?
What I'm trying to imagine is an abstraction layer providing easy access to the most common video formats, giving me the ability to pop the unwanted frames and update the file headers without delving into the specifics of each and every format.