Pierre and his family moved to the Upper Manor House at Peekskill and in late 1777 for a farm in Rhinebeck he leased from kinsman Henry Brockholst Livingston, son of William Livingston. Cornelia and her husband remained at the Upper Manor House. A curious old journal tells of their exodus, and the catalogue of the flocks and herds, man-servants and maid-servants that were sent to Rhinebeck, reads like a biblical story of the journeyings of the Patriarchs. Not only did George Washington sleep at the Van Cortlandt Upper Manor House, he stayed for extended periods commanding the battles of Long Island, New Amsterdam, and Westchester County. He also stayed there during his numerous visits to West Point. Washington, when in Peekskill, had his official headquarters in the village, but would spend evenings in the house which he used as a sort of "safe social" headquarters, doing his dining and entertaining in the house. While staying at the house he slept in the northwest bedroom on the second floor. The list of the famous who either stayed or were entertained in the house is long. Benjamin Franklin stopped here on his way back from the mission to win Canada to the side of the colonies in 1776; the Methodist Bishop Francis Asbury stopped here; Lafayette; von Steuben; General Alexander MacDougall; General Israel Putnam; General Philip Schuyler; Alexander Hamilton and Aaron Burr all visited the house. It was here that Washington entertained the Count de Rochambeau and his French officers. The house was also used as headquarters by General William Heath. Pierre Van Cortlandt always retained a most devoted attachment to George Washington, and was constantly a guest at the dinners given by his Excellency, when the latter was in New York. In August 1776, at the foot of Anthony's Nose, Fort Independence was built on his father's land, on the north bank of the Annsville Creek as it empties into the Hudson River. It combined with Forts Montgomery and Clinton to defend the Hudson River Valley. Forts Montgomery and Clinton were started in June. Fort Hill Park, the site of Camp Peekskill, contained five barracks and two redoubts. On the opposite side of Annsville Creek [north of Peekskill] at the point known as Roa or Roay and formerly Roya Hook, stood the old Revolutionary Fort Independence. In 1846 and for about three years subsequently some of the larger boats used to stop at this point.