According to legend, over two hundred years ago local fishermen found a bamboo raft bearing on top a small shrine to Chang, with the words "His Excellency General Chang." The fishermen promptly built a small temple nearby.
安西府原來只是在台西外海海豐島上的小廟，據說二百多年前，台西魚民在附近海域發現一個竹筏，上面有「張千歲」的香火，漁民便在海邊臨時搭了個小廟。