Does raising the framerate while not changing the bitrate result in quality loss? AFAIK, the bitrate is the number of bits per second. So let's say I have two videos, both using the same codec, bitrate settings and resolution. If the first one has a framerate of 25fps, but the second one runs at 50fps, does that mean that the second one will have a drastically lower quality since the available bits for each second of the video have to be divided amongst twice as much frames?

Or is this effect diminished by motion compensation (e.g. by having more P-frames and less I-frames) or something like that?
If the result of increasing the frame rate is that more pixels are displayed per second, then yes, keeping the bit rate the same will almost certainly mean a loss in overall quality. Not all such losses are objectionable or even necessarily noticeable. For example, if the bit rate is 30 Mb/s and you reduce it to 15 Mb/s, probably not many people would notice.