How can I find where an app's files are stored? Besides the app's package, is there a way to find where it stores its other files from during installation?  Are they only allowed to install into the "Library" folders, or are there other places?  I want to wipe my drive and do a fresh install when I upgrade to Mountain Lion.
Deleting the files from an individual app and doing a clean install of Mountain Lion are two different things. As per mckeed's answer, you can use AppZapper to delete an individual app, other alternatives being AppDelete or Clean my mac, although the latter could be a bit overkill for what you want.

Fresh install for ML is a different story though, I did a fresh install the other day and I'm super glad I did - I'm one of those people who just collects apps and files they don't need and everything gets a bit clogged after a while. Simple instructions to do so are:


Start off by backing up whatever you need (I know this ones a bit
obvious), save anything you want to keep to an external harddrive.
You'll need to get your hands on another external harddrive or usb stick at least 8gb, and it has to be empty as we're going to use this to make a mountain lion boot disk. If it's not empty, don't worry we'll erase it in step 7.
Download mountain lion from the app store, but don't install it otherwise the app disappears and you'll have to re download it.
Secondary click (right click, whatever you want to call it) the install icon and choose "show package contents" from the menu.
When the contents of the install file are displayed you'll see a folder called "SharedSupport", open this up and you'll find a file called "installESD.dmg". Copy this file somewhere so you know where to find it.
Open "utilities" in your applications folder and open "Disk Utility"
Plug in your blank USB drive from step 2. You'll now see it in the left pane of the disk utility window, click on it to select it. 
Click on the "Erase" tab in the main disk utility pane. Change the "format" dropdown to MAC OS Extended (Journaled) and then name your disk something that makes sense to you like "Mountain Lion Install". Click erase.
Click on the "Restore" tab in the main disk utility pane. In the "source" section, you'll need to navigate to and select the "InstallESD.dmg" file we copied earlier. In the destination tab you'll need to select your now clean drive. Click Restore.


The drive and your mac will now make whirry and buzzy noises for a while. When it's done cooking it will tell you. You now have a Mountain Lion boot disk.


Restart your computer with the new Mountain Lion drive plugged in still and immediately hold down the option key when it starts back up. If you've done this right, instead of booting OSX you'll see a picture of your drive to select. Click on the disk you created earlier to boot from it.
Now the scary part. You don't want to install just yet, somewhere on the screen there is an option for utilities, I think from memory it's in the top bar. Click utilities and select "disk utility" which looks alot like (actually exactly like) the disk utility app you used to make the disk earlier. This time, select your macs harddrive in the left pane and click the erase tab, make sure the format is set to MAC OS Extended again and click erase.
You've now erased your harddrive. Close out of the utilities window and continue the installation as normal installing Mountain Lion onto your newly clean Harddrive!


Hope all this helps, best of luck!