Passage: Elephant Island was remote, uninhabited, and rarely visited by whalers or any other ships. If the party was to return to civilization it would be necessary to summon help. The only realistic way this could be done was to adapt one of the lifeboats for an 800-mile (1,300 km) voyage across the Southern Ocean, to South Georgia. Shackleton had abandoned thoughts of taking the party on the less dangerous journey to Deception Island, because of the poor physical condition of many of his party. Port Stanley in the Falkland Islands was closer than South Georgia, but could not be reached, as this would require sailing against the strong prevailing winds.Shackleton selected the boat party: himself, Worsley as navigator, Crean, McNish, Vincent and McCarthy. On instructions from Shackleton, McNish immediately set about adapting the James Caird, improvising tools and materials. Frank Wild was to be left in charge of the Elephant Island party, with instructions to make for Deception Island the following spring, should Shackleton not return. Shackleton took supplies for only four weeks, knowing that if land had not been reached within that time the boat would be lost.The 22.5-foot (6.9 m) James Caird was launched on 24 April 1916. The success of the voyage depended on the pin-point accuracy of Worsley's navigation, using observations that would have to be made in the most unfavourable of conditions. The prevailing wind was helpfully north-west, but the heavy sea conditions quickly soaked everything in icy water. Soon ice settled thickly on the boat, making her ride sluggishly. On 5 May, a north-westerly gale almost caused the boat's destruction as it faced what Shackleton described as the largest waves he had seen in 26 years at sea. On 8 May, South Georgia was sighted, after a 14-day battle with the elements that had driven the boat party to their physical limits. Two days later, after a prolonged struggle with heavy seas and hurricane-force winds to the south of the island, the party struggled ashore at King Haakon Bay.

Question: Who frequently passed up their opportunity to visit Elephant Island?
whalers