Q:

Who scolded whom in Aに怒られる?

I can see that someone got mad or someone made someone else mad. The things that make it difficult are に and the られる form of 怒る. What does this mean? How do you figure out who did what in these types of sentences?

A:

Maybe the fact that に is used in both active and passive sentences with 怒る is making it difficult for you. に can be used either as a dative case marker or as an element leading the agentive-phrase in passive. In English, you can see that the dative is to, whereas the agentive-phrase is headed by by, so an active sentence with dative case and its passive counterpart look pretty much different:

A gave a present to B          (to as dative case)
B was given a present by A    (by heading an agentive phrase in passive)

But in Japanese, they are both expressed as に. So an active sentence with dative case and its passive counterpart may look confusing:

AがBに怒った     (に as dative case)
BがAに怒られた   (に heading an agentive phrase in passive)

The two examples above describe (roughly) the same event; in both sentences, A is who scolded, and B is who was scolded. But in the first sentence, it is B that appears with に, and in the second sentence, it is A. I think that was where you were confused. But you should be able to distinguish the two sentences by looking at whether the verb is in active or passive form. If you see that it is in active form 怒る, then the phrase that comes with に is the one who is being scolded. If it is in passive form 怒られる, then the phrase that comes with に is the one who is scolding.

Advanced part
Language hacker raised an interesting point in the comment, so I will add something. You can also see the difference between the two usages of に when you topicalize the relevant phrase. The dative に becomes optional in front of は, but the agentive に is obligatory:

Aは、Bには怒ったが、Cには怒らなかった     (dative case に preceding は)
Aは、Bは怒ったが、Cは怒らなかった        (dative case に omitted before は)
Bは、Aには怒られたが、Dには怒られなかった   (agentive に preceding は)
* Bは、Aは怒られたが、Dは怒られなかった   (Ungrammatical to omit agentive に preceding は)
(* indicates ungrammatical sentence.)

A:

It may first bear mentioning that 怒られる is usually used specifically to mean "got scolded by", rather than "to became the focus of someone's anger".
This may make it easier to understand.

Aは怒った : A got angry
AはBに怒った : A got angry at B (in this usage, B is usually not a person, but rather an event or state of affairs).
AはBを怒った : A scolded B (B is a person)
BはAに怒られた : B got scolded by A.

A:

This is one of those cases where the Japanese られる doesn't quite line up nicely with the English passive:

Aが怒った。 A got mad.
（私が）Aに怒られた。 (I) was gotten mad at by A. (More naturally in the active: A got mad at me.)
雨が降った。 The rain fell.
（私が）雨に降られた。 (I) was fallen on by the rain. (More naturally: I got rained on.)

With the passive られる, the subject of the sentence (usually 私 if it's omitted) is the "target" (on the receiving end) of the action. In other words, something is done to the subject rather than by the subject. (The by part is marked by に, as sawa mentioned.)
To address your comment about the たい construction, then:

彼女に怒られたい I want to be gotten mad at by her. (More naturally in the active: I want her to get mad at me.)

Also, when に is used not with 怒られる, but with 怒る, it marks the cause of anger:

環境破壊に怒る become angry over destruction of the environment
入場できないことに怒る become angry over not being able to enter

