Coming back with an update:
so we tried to wipe our ubuntu installation and reisntall driver and allennlp(which has pytorch as a depencdency), didnt work. that is when we thought that there could be something off about our installation of pytorch. turns out the cuda version installed by AllenNLP  is 9.0 while if we uninstalled pytorch, reinstalled with 10.0, it worked.
this is something i plan to report to allennlp so they may be able to take a look, but that is how we resolved our issues.