He started work as a junior clerk in the ledger department of a City firm, but was made redundant after pressing his fellow clerks to join a trade union. From 1932, he worked as a fur salesman for the John Lewis Partnership, dropping his Cockney accent to appeal to society customers. Brown earned a great deal on commission. During this time, Brown continued his education through London County Council evening schools and the Workers' Educational Association. The poverty of his upbringing led Brown in later life to resent those who had a more privileged background and a university education. Shortly after his marriage on 27 April 1937 to Sophia, daughter of Solomon Levene, a bookbinder, Brown was employed as a ledger clerk with the Transport and General Workers Union, and appointed District Organiser for Watford the next year. By now Brown was active within the Labour Party and the Labour League of Youth. He ran as a moderate candidate for the Chairmanship but at the Labour Party conference in 1937 he was defeated by a left-wing candidate, Ted Willis, later a writer for television. At the 1939 Labour Party conference, Brown made his mark by a strong speech demanding the expulsion of Stafford Cripps for his advocacy of a Popular Front. For the rest of his life, Cripps refused to speak to Brown. After the outbreak of the Second World War in September 1939, Brown volunteered for the Royal Air Force but Ernest Bevin, the Minister of Labour, kept Brown and other trade union officials in their civilian jobs. Bevin was one of the Labour leaders brought into the wartime coalition government. Brown himself served as a temporary civil servant in the Ministry of Agriculture from 1940 onwards. As a TGWU official, Brown was an attractive candidate to Labour constituencies seeking a candidate, as the TGWU would sponsor him and pay election expenses. He was selected for Belper, a mixed constituency near Derby which was one of Labour's principal target seats. In the 1945 general election, Brown gained the seat from the Conservatives with a majority of 8,881. He was invited as one of a dozen 'Young Victors' to a private dinner given by Hugh Dalton on 30 July 1945 who was talent-spotting and networking. Brown was immediately chosen to be a Parliamentary Private Secretary (PPS) by George Isaacs, who had followed the promoted Bevin as Minister of Labour, but his time with Isaacs was brief. Brown was both adept at understanding political issues and how to communicate them, and convivial and generally popular within the Parliamentary Labour Party (save among the left-wing faction, whom he attacked as 'long-haired intellectuals').