For the next person hitting this issue :–)
the environment was a little odd, it was Ubuntu14, and even though
$ gcc --version
gcc (Ubuntu 5.4.1-2ubuntu1~14.04) 5.4.1 2016090

I had
$ x86_64-linux-gnu-gcc --version
x86_64-linux-gnu-gcc (Ubuntu 4.8.4-2ubuntu1~14.04.4) 4.8.4

It took some time to see, not matter what I set to $CC or $CXX always x86_64-linux-gnu-gcc was used for building the extension, and that pointed to the old gcc. update-altenatives also did not list the multiple gccs.
Anyways, this was the fix:
sudo update-alternatives --install /usr/bin/x86_64-linux-gnu-gcc x86_64-linux-gnu-gcc /usr/bin/x86_64-linux-gnu-gcc-5 40 --slave /usr/bin/x86_64-linux-gnu-g++ x86_64-linux-gnu-g++ /usr/bin/x86_64-linux-gnu-g++-5
sudo update-alternatives --config x86_64-linux-gnu-gcc