Reencoding H.264 to H.264 made it 40% larger I've got some livestreamed footage (encoded with constant bitrate), and I've been trying to reduce the file size for long-term storage.  I figured that since encoding speed is no longer a constraint for me, I could let the encoder take more time and make a smaller file.

The original footage was encoded at 12 Mbps, but when I tried reencoding it with FFmpeg (libx264, CRF 18, and preset veryslow), the result was at 17 Mbps.

My question is, how can using these settings make a file so dramatically larger than the original?  As I understand it, the point of the veryslow preset is to take a long time and make the most compressed H.264 video possible (ignoring the placebo preset), but in this case I'd be far better off doing nothing!  I'd understand it being slightly bigger, because it has to encode some compression artifacts, but not a 40% increase.
You are using a quality based variable bitrate compression factor on something that already has artifacts.  That means it is going to attempt to preserve the video exactly as it is, artifacts included and artifacts tend to not encode well.  This means you have a video input which is likely harder to encode than the original and you chose a quality based compression that is also likely higher quality than your original 12 Mbps could manage.  This means an overall higher data rate must be used to try to avoid additional loss from the second generation of encoding, even with the extra optimizations that a slower compression cycle can handle, you aren't able to overcome the added complexity of the first encoding and the quality setting being used.

If this was any kind of HD stream that's not really that surprising as 12Mbps is a very heavily compressed end consumer level of compression for any HD formats.  It's not really a big surprise that it doesn't re-encode well.