After more studying ,it seems like implementing my own Layer would be the easiest way to do what Im trying to do, although im still not sure how I would implement this.  I decided to use a  simpler loss function for now as what I wanted to do would take too long to compute anyway.