His ancestry and surname made him the object of xenophobia in his childhood during World War I, which carried on into the early part of his military career and beyond; as late as 1941, an anonymous letter writer from RAAF Station Wagga would tell Prime Minister Robert Menzies that his "blood ran cold" at the notion of someone called "Scherger" commanding trainee Australian pilots. Scherger nevertheless entered the Royal Military College, Duntroon, in 1921 and graduated as a lieutenant in 1924, winning the King's Medal. Two days prior to graduation, he volunteered for an Air Force secondment, which was later made permanent. On 21 January 1925, he received a permanent commission in the RAAF as a pilot officer (temporary flying officer), and commenced his flight training at RAAF Point Cook, Victoria. He was promoted to flying officer with seniority from 21 January 1926. Scherger quickly took to the art of flying open-cockpit biplanes and gained a reputation as a skilful if occasionally reckless pilot, being berated early in his career by his flight commander for "inverted and very low flying". He was one of the Air Force's first volunteers for parachute instruction, under the tutelage of Flying Officer Ellis Wackett at RAAF Station Richmond, New South Wales, and made the first public freefall descent in Australia, at Essendon, Victoria on 21 August 1926. In February 1927, he was asked by the commanding officer of No. 1 Flying Training School (No. 1 FTS), Wing Commander Adrian "King" Cole, to drop a message to a woman at Port Melbourne before she departed on a steamer. After doing so, Scherger illegally flew his S.E.5 fighter between ship and wharf before heading back to Point Cook, only to be hauled into Cole's office the next morning to find the CO brandishing a photograph taken by a member of the public, catching the young pilot in the act. Sent for a dressing down to the Air Member for Personnel, Group Captain Jimmy Goble, Scherger was forced to admit it was not the first time he had engaged in such stunts. Goble responded, "Good, I'm glad to see we've still got a few in the Air Force with spirit." By the 1930s, as a flight instructor and test pilot, Scherger was, according to historian Alan Stephens, "perhaps the RAAF's outstanding aviator". He married Thelma Harrick on 1 June 1929, and they had a daughter named Jill.