Question: What makes the web application perform and scale better is always a big topic. And finding the performance problems and tuning them is another...

Here is some my thoughts of how to "finding" performance problems:

For a "new" api/application or other 

* Analzying the detail api and then preparing the Jmeter/Grinder testing scripts for it.
* Using different load to identify the threshold for the api
* Adding profiling codes find the slownes
* Restart from point one again..

For a "old" api/application or other 

* Analyzing the user pattern from the access detail log
* Simulate the real user load to find the slowness
* Adding profiling codes find the slownes
* Restart from point one again..

So,how can you identify the performance problems?

Answer: 
I prefer not to have performance monitoring in my production code. I try to have pretty good unit test coverage (every significant method has at least one unit test). Visual Studio automatically times how long it takes to perform each unit test. For a general overview, I just run all tests to get a general idea of where bottlenecks might be.
For a web application, timing individual requests is a good start. [Firebug](http://getfirebug.com/) works well for that. If you combine that with Ironcode's suggestion to rely on unit tests, then you should be able to locate most of the bottlenecks.

Beyond that... it depends on your app. There are various tools that you could use for profiling various aspects of the code. For example, if you're using Hibernate, it lets you collect statistics on how long all the database queries take to complete.