I’m training an object detection model in pytorch on Ubuntu 16.04 with a TitanX Pascal gpu. In the middle of training (after several thousand training iterations e.g. 70k), the training crashes with the message:
[libprotobuf FATAL google/protobuf/wire_format.cc:830] CHECK failed: (output->ByteCount()) == (expected_endpoint): : Protocol message serialized to a size different from what was originally expected.  Perhaps it was modified by another thread during serialization?
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  CHECK failed: (output->ByteCount()) == (expected_endpoint): : Protocol message serialized to a size different from what was originally expected.  Perhaps it was modified by another thread during serialization?
Command terminated by signal 6

I’ve searched, but couldn’t find anyone experiencing similar errors. I’m using tensorboard-pytorch to visualize the training artifacts. Any ideas on how to resolve this issue?