I'm looking for a Python 2D graphics library that can basically do the following and not necessarily anything more:

Create a window of specified width and height
Set the RGB of pixel X, Y on the back buffer.
Swap buffers

...and that's it basically. I can't find anything that doesn't come with a massive amount of complex baggage.
I recommend PyQt for this - it's a GUI library/framework but it has very good drawing capabilities. If you look at the examples coming with PyQt, focusing on the graphics & drawing samples, it's quite amazing what you can do with very few lines of code.
Oh, and it does the double-buffering you mention automatically so you don't have to worry about it.

Alternatively, you can use PyGame - a library wrapping SDL, used for game development. Naturally it has very strong 2D graphics capabilities.
Basic Python 2D graphics?