Question: I have one C code app. which i was building using MS-VS2005. I had one output data buffer which was being allocated dynamically using malloc.

For some test cases, the memory size which was being malloc'd was falling short than the the actual output size in bytes which was generated. That larger sized output was written into the smaller sized buffer causing buffer overflow. As a result of which the test-run was crashing with MSVS-2005 showing up a window "Heap corruption ...."

I knew it had to do with some dynamic memory allocation, but i took long time to actually find the root cause, as i did not doubt the memory allocation because i was allocating large enough size necessary for the output. But one particular test case was generating more output than what i had calculated, hence the resulting crash.

My question is:

1.) What tools i can use to detect such dynamic memory buffer over-flow conditions. Can they also help detect any buffer overflow conditions(irrespective of whether the buffer/array is on heap, stack, global memory area)?

2.) Will memory leak tools(like say Purify) or code analysis tools like lint, klocworks would have helped in particular case? I believe they have to be run time analysis tools.

Thank you.

-AD.

Answer: 
Something you can try is allocate enough pages + 1 using VirtualAlloc, use VirtualProtect with PAGE\_READONLY | PAGE\_GUARD flags on the last page, then align the suspected allocation so the end of the object is near the beginning of the protected page. If all goes well you should get an access violation when the guard page is accessed. It helps if you know approximately which allocation is overwritten. Otherwise it requires overriding all allocations which may require a lot of extra memory (at least 2 pages per allocation). A variation on this technique that I'm hereby christening as "statistical page guard" is to only randomly allocate memory for a relatively small percentage of allocations in that manner to avoid large bloat for small objects. Over a large number of execution runs you should be able to hit the error. The random number generator would have to be seeded off something like time in this case. Similarly you can allocate the guard page in front of the object if you suspect an overwrite at a lower address (can't do both at the same time but possible to randomly mix up as well).

An update: it turns out the gflags.exe (used to be pageheap.exe) microsoft utility already supports "statistical page guard" so i reinvented the wheel :) All you need to do is run gflags.exe /p /enable [/random 0-100] YourApplication.exe and run your app. If you are using a custom heap or custom guards on your heap allocations then you can simply switch to using HeapAlloc at least for catching bugs and then switch back. Gflags.exe is a part of Support Tools package and can be downloaded from the microsoft download center, just do a search there.
PC-Lint can catch some forms of malloc/new size problems, but I'm not sure if it would have found yours.

VS2005 has good buffer overflow checking for stack objects in debug mode (runs at end of function). And it does periodic checking of the heap.

As for it helping to track down where the problems occurred, this is where I tend to start using macro's to dump all allocations to match against the corrupted memory later (when it's detected). 

Painful process, so I'm keen to learn better ways also.