Question: I'd like to write a program able to "use" other programs by taking control of the mouse/keyboard and being able to "see" what's on the screen.

I used [AutoIt](http://www.autoitscript.com/autoit3/) to do something similar, but I had to cheat sometimes because the language is not that powerful, or maybe it's just that I suck and I'm not able to do that much with it :P 

So... I need to:

* Take screenshots, then I will compare them to make the program "understand", but it needs to "see"
* Use the mouse: move, click and release, it's simple, isn't it?
* Using the keyboard: pressing some keys, or key combinations, including special keys like `Alt`,`Ctrl` etc...

How can I do that in python?  

Does it works in both linux and windows? (this could be really really cool, but it is not necessary)

Answer: 
You can use [WATSUP](http://www.tizmoi.net/watsup/intro.html) under Windows.
If you are comfortable with pascal, a really powerful keyboard/mouse/screen-reading program is SCAR: <http://freddy1990.com/index.php?page=product&name=scar> It can do OCR, bitmap finding, color finding, etc. It's often used for automating online games, but it can be used for any situation where you want to simulate a human reading the screen and giving input.