What happens when a compressed video is played back? I've been reading up on video compression. I understand that 'raw' media file sizes are to big for practical usage so they have to be compressed, but I'm wondering what happens when the video is being played back.

I mean is the whole video decoded (decompressed) and stored somewhere in memory (this doesn't seem possible) or are the individual frames decoded one at the time and then forgotten, so that when you want to replay a portion of the video you need to decode it again?

It's probably a very noob question and it's because I am one, so would appreciate any info.
In most common compression schemes, there isn't a 1::1 correspondence between "video" frames and "file" (or stream) frames **. Any particular video frame may require the contents of several file frames to become a complete viewable image. So a player will read and buffer as many file frames as needed to reconstruct the video frame in question. The player may buffer much more, but it's not required to.

For example, if some video frame ends up as a P frame when encoded in the file, the player will need to look back to find the previous I frame, decode that, then apply the differences encoded in the intervening P frames to get the complete result.

** EDIT: There is of course a file frame for each video frame, but the point is that you can't usually read one frame from the file and get a complete video frame: that's only true for I frames.