 Elder was born at Kirkcaldy, Scotland, the fourth son of George Elder, merchant, and his wife Joanna Haddow, "née" Lang.Thomas' second eldest brother, Alexander Lang Elder (1815–1885), went to South Australia in 1839 and founded the firm of Elder and Company in Adelaide.He was joined by his brothers William (1813–1882) and George (1816–1897).