I cannot see any obvious errors, but since your code snippet is not executable, I also cannot debug it.
Based on the error message I guess you might flatten the predictions somehow and/or the batch size calculation is wrong.
A VGG with inputs of [49132, 3, 1024, 512] would need a lot of memory so you are either using the CPU where this might fit into the system RAM or the batch_size calculation is wrong.