Can I replace my bicycle's dynamo with batteries? I have a working lightning system on my bike with wired front and rear lights, powered by a dynamo. The dynamo's plastic has broken and I have repaired it several times now, but now it is FUBAR.

Thinking about a replacement, I wondered if it would be possible to just connect some batteries to the dynamo cables. AFAIK the usual voltage for dynamo powered lights is 6 volts, so if I were to add 4 normal batteries (4 x 1.5 = 6) in series, and connect this to the power cables, would it work? Or are there other things to consider?

Both front and rear light are LED lights; the rear light has a capacitor to maintain illumination even when you stop. 

I would use a battery holder like the one pictured below, and rechargeable AA batteries, since I have a lot of them already.

If you have any other suggestions on how to do this, I look forward to hearing them!

I want to do this because I have had issues with rim dynamos in the past, and just want a reliable power source; and I am too lazy to install a hub dynamo. It is also a cost issue; I already have the rechargeable batteries, but would have to buy a new dynamo. This is my city bike; it could be stolen at any time, so I do not want to invest a lot of money in components.
Yes, a battery holder and switch from radio shack would accomplish this. Most dynamos are 6vac and can be replaced with 4xAAs or the 5vdc from a USB pack. You can ignore the issue about polarity as the dyno lights that I've worked are either incandescent and don't care or are LED but have built in blocking diodes and don't care.

Whether you would want to do this is another issue. I've gone the opposite direction and put a dynohub in my commuting bike.

Note: if you are handy with a soldering iron and multimeter, you can remove the diodes/rectifier (and likely super cap) from the LED lights and have them run in straight DC. You'd gain brightness especially when your batteries are running low because you would no longer have the voltage drop of the diodes or rectifiers, but you would need to check polarity and make sure you aren't drawing too much current.