Question: I've asked a lot of questions about this lately, and I think I'll just quit beating around the bush...

Let's say I've got a storage system that is being used for a variety of applications with different I/O patterns. I've been collecting performance statistics on the system:

* transfers/sec
* sec/trasnfer
* bytes/transfer
* bytes/sec
* %idle

and I've calculated 95th percentile, average, and median for each. I also know what my average read / write ratio is. 

I understand how to calculate potential average IOPS and throughput for a new system given disk, array, and average workload parameters.

I'm struggling to put this all together. Currently observed statistics are limited by the existing system which is struggling to keep up. Thus I might know that I require X IOPS, but this value may be low because of the current disk bottleneck, etc. (I know it's overworked because I'm seeing constant high disk usage and many multi-second periods of very high transfer times)

To be frank, I'm not doing anything hardcore and I can pretty much just buy some faster disks and configure my arrays better and it'll work out. But I'd like to understand how I might take a more formal approach to justifying an expense and to not over-buy.

Answer: 
I don't think you have enough data... what you also need to know is the utilisation of the other parts of the system that might become the bottleneck if the IO speeds up, so you can estimate how far you have to go on the IO before it becomes CPU, bus or network limited.

Some definition of 'fast enough' would help too. But it sounds like you want the long periods of waiting for IO to go away. Depending on exactly what you're doing, you may just not have enough memory to cache it properly.
Be sure to do what I call theorectical drive math to get your max IOPS and reduce by 20% or more based on real world usage. This varies by manufacturer, disk speed, etc. Then compare to your new drive configuration and make sure your peak loads fit well under this limit.