Question: I'd like to write a program able to "use" other programs by taking control of the mouse/keyboard and being able to "see" what's on the screen.

I used [AutoIt](http://www.autoitscript.com/autoit3/) to do something similar, but I had to cheat sometimes because the language is not that powerful, or maybe it's just that I suck and I'm not able to do that much with it :P 

So... I need to:

* Take screenshots, then I will compare them to make the program "understand", but it needs to "see"
* Use the mouse: move, click and release, it's simple, isn't it?
* Using the keyboard: pressing some keys, or key combinations, including special keys like `Alt`,`Ctrl` etc...

How can I do that in python?  

Does it works in both linux and windows? (this could be really really cool, but it is not necessary)

Answer: 
I've had some luck with similar tasks using [PyWinAuto](http://pywinauto.openqa.org/).

> 
> pywinauto is a set of python modules
>  to automate the Microsoft Windows GUI.
>  At it's simplest it allows you to send
>  mouse and keyboard actions to windows
>  dialogs and controls.
> 
> 
> 

It also has some support for capturing images of dialogs and such using the Python Imaging Library [PIL](http://www.pythonware.com/products/pil/).
AutoIt is completely capable of doing everything you mentioned. When I'm wanting to do some automation but use the features of Python, I find it easiest to use [AutoItX](http://www.google.com/url?sa=t&source=web&ct=res&cd=1&url=http%3A%2F%2Fwww.autoitscript.com%2Fautoit3%2Fdownloads.shtml&ei=Yk9TSsOsDYnWNfza6fQI&usg=AFQjCNEBCnVe6pRuxIzX_UE4U_3O8SVYOg&sig2=t4kpjNbgspeikHz2SBkIGg) which is a DLL/COM control.

Taken from [this answer](https://stackoverflow.com/questions/151846/get-other-running-processes-window-sizes-in-python/155587#155587) of mine:

```
import win32com.client
oAutoItX = win32com.client.Dispatch( "AutoItX3.Control" )

oAutoItX.Opt("WinTitleMatchMode", 2) #Match text anywhere in a window title

width = oAutoItX.WinGetClientSizeWidth("Firefox")
height = oAutoItX.WinGetClientSizeHeight("Firefox")

print width, height

```