Question: What makes the web application perform and scale better is always a big topic. And finding the performance problems and tuning them is another...

Here is some my thoughts of how to "finding" performance problems:

For a "new" api/application or other 

* Analzying the detail api and then preparing the Jmeter/Grinder testing scripts for it.
* Using different load to identify the threshold for the api
* Adding profiling codes find the slownes
* Restart from point one again..

For a "old" api/application or other 

* Analyzing the user pattern from the access detail log
* Simulate the real user load to find the slowness
* Adding profiling codes find the slownes
* Restart from point one again..

So,how can you identify the performance problems?

Answer: 
I've also implemented production performance monitoring systems to help identify performance problems; the key, of course, is making sure your performance monitoring doesn't hinder performance itself!

But really, a simple performance monitor that (in a web application example) just logs how long it takes to complete a request and how long it takes to render a page is enough to focus your efforts on the requests or request types that are the slowest, or present the greatest load (avg. execution time \* execution count), and therefor are good targets for optimization.. Then, using a test platform and the basic steps you outlined in your question, you can narrow down the cause of the performance problems and begin your optimizations.

Depending on your situation, it's also good to monitor any dependent processes individually: profile database performance, disk performance, and so on, to make sure that all of the moving parts of the whole system are working as smoothly as possible. A perfectly optimized application can be slowed to a crawl by a poor-performing database.
As far as I know, in enterprise applications, there usualy are NFRs present stating the general effectiveness of system under test. By my project, it works like this:

-the customer specifies what he understands as "effective", "responsive", etc.

-we prepare example testdata (not "let's hope for the best" dataset, but "let's ride this app like there's no tommorrow" dataset), and we prepare usage scenarios to be run.

-then, we let it run and burn some CPU cycles, for like 48/72 hours, and gather general system statistics (that way we don't have to place specific and unrealistic monitoring code in the app).

-in the end it is possible to say whetever there is a specific problem (eg. the system should do 10k transactions per hour, but does only 2k - that's performance bottleneck somewhere), and we should be able to tell which usage scenario/data combination caused unwanted behavior. From now on, it is possible to mimic specific examples in specific benchmarks, and work on the solution.

If you need to test specifically a webapp (eg. interface responsiveness under stress), you may consider using distributed selenium (I believe, there are some companies offering such services, using their own datacenters)