What is the deal with serrated/non-smooth rim sides on a bike wheel? I recently decided to replace the rear wheel on my road bike since I couldn't disassemble the old Helicomatic freewheel

I now have a new rim, which oddly has slightly serrated sides (I'll add a picture tomorrow), while the original wheel was smooth. As a consequence, the rear brake doesn't create a lot of friction which I think is dangerous.

However, the seller insisted that the wheel was meant for rim brakes. Am I being lied to or what do I need to do to get proper brake grip on the rim?
These are just machining marks from the tool that cut the braking surface. Rim manufacturers don't bother to get the braking surface completely smooth.

The tiny grooves don't negatively affect brake performance and will eventually wear away. You'll need to look for other causes of brake performance problems. Do you have new pads. New pads on a new rim make need 'breaking in', i.e. get a little wear before they achieve proper friction.