But, if I stick to this code above. What do you think is the correct way to calculate cont_loss_weight by Idea-1 or Idea-2?

I honestly don’t know. You will need to derive the mathematical formula for what the gradient should be with pen and paper first. Then implement the final formula you get in there.

I tried this as well without nightly version, but GPU went out of memory. 

Does it run out at the first iteration? Or after a while?
Can you try reducing the batch size to reduce memory pressure?