In 1876, Potter again elected to the House, and represented New York's 12th District in the 45th Congress, serving from March 4, 1877 to March 3, 1879. During this term Potter was appointed Chairman of the Committee on Pacific Railroads.