Question: I want to create a desktop recorder that require very little HD space.

It should capture the current display into a buffer, compare it to the previous state, and save only the rectangles that differ to the previous state.

What API, function or library I have to use ?

Answer: 
Well if you want to save the differences from each frame to frame only you could simply use a substraction-method. Simply substract the color values at image(t+1) from image(t)... All parts that stay equal haven't changed... only the parts that are different will result in something non-zero. You can then extract the rectangles around it and save them. But of course be aware since there might be more than one part changing of course and you probably wanna save each one instead of the big rectangle that contains all changes...

You could use OpenCV for this... it has all basic functions for image substraction, rectangle fitting, cropping, ...

Hope that helps...
Consider using Windows Media Screen Capture encoder for the task. You will feed your captured frames to it, and it will do the rest and create highly efficient wmv file for you.