Bardhi was born in Kallmet or Nënshat in the northern Albanian Zadrima region near Lezhë. He came from a family consisting of many figures high in the hierarchy of the Catholic Church and state officials or military commanders of the Republic of Venice. His uncle was Bishop of Sapa and Sarda.