Question: I'd like to write a program able to "use" other programs by taking control of the mouse/keyboard and being able to "see" what's on the screen.

I used [AutoIt](http://www.autoitscript.com/autoit3/) to do something similar, but I had to cheat sometimes because the language is not that powerful, or maybe it's just that I suck and I'm not able to do that much with it :P 

So... I need to:

* Take screenshots, then I will compare them to make the program "understand", but it needs to "see"
* Use the mouse: move, click and release, it's simple, isn't it?
* Using the keyboard: pressing some keys, or key combinations, including special keys like `Alt`,`Ctrl` etc...

How can I do that in python?  

Does it works in both linux and windows? (this could be really really cool, but it is not necessary)

Answer: 
You can use [WATSUP](http://www.tizmoi.net/watsup/intro.html) under Windows.
I've used the Windows (only) [Input API](http://msdn.microsoft.com/en-us/library/ms646310(VS.85).aspx) to write a VNC-like remote-control application in the past. It lets you fake keyboard and mouse input nicely at a system level (ie not just posting events to a single application).

If you're trying to do any sort of automated testing of whole systems at the GUI level, [this excellent USENIX paper](http://www.usenix.org/event/usenix05/tech/freenix/full_papers/zeldovich/zeldovich_html/index.html) describing automated responsiveness testing is a must-read.