pastor began playing saxophone when he was sixteen. he played tenor sax with john cavallaro (1927), irving aaronson (1928–30), and austin wylie (1930), then opened his own night club in hartford, connecticut and led the band there for three years. after that, he played with smith ballew (1934), joe venuti, paul fredricks, vincent lopez, and artie shaw's first (1936–37) and second (1937–39) orchestras. in november 1939 when shaw walked off the bandstand in the cafe rouge located inside the hotel pennsylvania (essentially quitting his own band), pastor was soon coaxed into leading his own big band, which he did from 1939 to 1959.