Basically I want to know the libraries which can do it, preferably in C or Python.
I can think of OpenCV, but it may be overkill, depending on what kind of "manipulation" you need. It is c/c++ lib, but also provides python and java wrappers.
How to grab a frame from a video and then manipulate it and then reassemble the video with it?