Question: My PC's ATX power supply stopped working. I opened it and noticed that it makes a high pitch sound when turned on. I can't figure out which component in the power supply makes that sound, so that I can replace it.

Answer: 
The sound could come from a capacitor or a transformer, though that is not necessarily the part that is defective.

Personally I wouldn't mess with a switched power supply (it is very dangerous and a repair is unlikely to succeed) and buy a new one.
How exactly has it stopped working?  

Have you tested the output for voltage? Is the fan working? 

It is quite likely a coil/transformer that is whining due to a e.g. shorted output transistor, bad capacitors or faulty load, causing excess current/pulse skipping which produces audible frequencies. Make sure you have removed all the loads to confirm they are not the problem.  

As mentioned already, I would really just replace it, as they are nasty things to work with even with the proper tools and one problem can often be caused by another less obvious one. The capacitors can hold a lot of charge after power has been removed which can be very dangerous unless discharged properly.  

There are troubleshooting guides out there but often they contain bad information and still need a thorough knowledge of SMPS and the right equipment to be sure of decent results.