Your code is currently a bit hard to read so could you please format it by wrapping it into three backticks ```? 
Is the local_rank always set to 0 and are thus all processes using the same device?
I’m not deeply familiar with Horovod, but I assume you’ve checked this doc 2 already?