Question: I wish to write software which could essentially profile the CPU cache (L2,L3, possibly L1) and the memory, to analyze performance.

Am I right in thinking this is un-doable because there is no access for the software to the cache content? 

Another way of wording my Q: is there any way to know, from the OS/Application level, what data has been loaded into cache/memory?

EDIT: Operating System Windows or Linux and CPU Intel Desktop/Xeon

Answer: 
You might want to look at Intel's PMU i.e. Performance Monitoring Unit. Some processors have one. It is a bunch of special purpose registers (Intel calls them Model Specific Registers, or MSRs) which you can program to count events, like cache misses, using the `RDMSR` and `WRMSR` instructions.

Here is a document about [Performance Analysis on i7 and Xeon 5500](http://software.intel.com/sites/products/collateral/hpc/vtune/performance_analysis_guide.pdf).

You might want to check out Intel's **[Performance Counter Monitor](http://software.intel.com/en-us/articles/intel-performance-counter-monitor/)**, which is basically some routines that abstract the PMU, which you can use in a C++ application to measure several performance metrics live, including cache misses. It also has some GUI/Commandline tools for standalone use.

Apparently, [the Linux kernel has a facility for manipulating MSRs](http://www.kernel.org/doc/man-pages/online/pages/man4/msr.4.html).

There are other utilities/APIs that also use the PMU: [perf](https://perf.wiki.kernel.org/), [PAPI](http://icl.cs.utk.edu/papi/index.html).
[This](https://stackoverflow.com/questions/10082517/simplest-tool-to-measure-c-program-cache-hit-miss-and-cpu-time-in-linux) has some examples.
TAU, an open-source profiler which works using PAPI can also be used. 

If however, you want to write a code to measure the cache statistics you can write a program using PAPI. PAPI allows the user to access the hardware counters without any need to know system architecture.
PMU uses Model Specific Registers, hence you must have the knwoledge of the registers to be used.

Perf allows for measurement of L1 and LLC (which is L2), Cachegrind, on the other hand allows the user to measure L1 and LLC (which can be L2 or L3, whichever the highest level cache is). Use Cachegrind only if you have no need of faster results because Cachegrind runs the program about 10X slower.