Both countries had refused to hold summit meetings with Japan because of Mr. Koizumi 's annual pilgrimages to the Yasukuni Shrine , the Shinto memorial for Japan 's war dead and war criminals , as well as a potent symbol of Japanese militarism in Asia .