Hi Ptrblck,
I have made a couple of changes to the code before rerunning. I have changed the logpx_z function to:
logpx_z=-torch.sum(BCE,[0,1,2],keepdim=False)
The model has now completed epoch 40:
40 6912 tensor(10543.6299, device='cuda:0', grad_fn=<NegBackward>) tensor(0.2380, device='cuda:0') tensor(3.9410e-06, device='cuda:0') tensor(1., device='cuda:0') tensor(0.2542, device='cuda:0', grad_fn=<MeanBackward0>) tensor(0.0054, device='cuda:0', grad_fn=<MinBackward1>) tensor(1., device='cuda:0', grad_fn=<MaxBackward1>)

the Cuda Launch Blocking yields:
line 334, in <module>
loss.backward()

\torch\tensor.py", line 195, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)

\torch\autograd\__init__.py", line 99, in backward
allow_unreachable=True)  # allow_unreachable flag

I also have the total output of BCE for epoch 37,38,39 & 40 dumped into a pickle file.
chaslie