I ended up having to do the tuple output to get things working. Unfortunately this messes up my original implementation of the network since my custom layer was being used in a sequential. Had to rewrite some code to fix that.