Thank you for pointing me to this code! Yes, looking at the code it looks like it is based on frequency since the words are sorted by frequency (from most frequent to least frequent) per the first few lines below.
   # sort by frequency, then alphabetically
    words_and_frequencies = sorted(counter.items(), key=lambda tup: tup[0])
    words_and_frequencies.sort(key=lambda tup: tup[1], reverse=True)

    for word, freq in words_and_frequencies:
        if freq < min_freq or len(self.itos) == max_size:
            break
        self.itos.append(word)

Edit: I think my original understanding was slightly off. If you set max_size=50000, then your vocab size should be 50002 for your 50000 most frequently occurring unique terms plus <unk> + <pad>.
From vocab.py,
if specials_first:
    self.itos = list(specials)
    # only extend max size if specials are prepended
    max_size = None if max_size is None else max_size + len(specials)