Question: What makes the web application perform and scale better is always a big topic. And finding the performance problems and tuning them is another...

Here is some my thoughts of how to "finding" performance problems:

For a "new" api/application or other 

* Analzying the detail api and then preparing the Jmeter/Grinder testing scripts for it.
* Using different load to identify the threshold for the api
* Adding profiling codes find the slownes
* Restart from point one again..

For a "old" api/application or other 

* Analyzing the user pattern from the access detail log
* Simulate the real user load to find the slowness
* Adding profiling codes find the slownes
* Restart from point one again..

So,how can you identify the performance problems?

Answer: 
I prefer not to have performance monitoring in my production code. I try to have pretty good unit test coverage (every significant method has at least one unit test). Visual Studio automatically times how long it takes to perform each unit test. For a general overview, I just run all tests to get a general idea of where bottlenecks might be.
As far as I know, in enterprise applications, there usualy are NFRs present stating the general effectiveness of system under test. By my project, it works like this:

-the customer specifies what he understands as "effective", "responsive", etc.

-we prepare example testdata (not "let's hope for the best" dataset, but "let's ride this app like there's no tommorrow" dataset), and we prepare usage scenarios to be run.

-then, we let it run and burn some CPU cycles, for like 48/72 hours, and gather general system statistics (that way we don't have to place specific and unrealistic monitoring code in the app).

-in the end it is possible to say whetever there is a specific problem (eg. the system should do 10k transactions per hour, but does only 2k - that's performance bottleneck somewhere), and we should be able to tell which usage scenario/data combination caused unwanted behavior. From now on, it is possible to mimic specific examples in specific benchmarks, and work on the solution.

If you need to test specifically a webapp (eg. interface responsiveness under stress), you may consider using distributed selenium (I believe, there are some companies offering such services, using their own datacenters)