Hi Team,
Whilst using the inference API to produce summaries of call using a private model I sometimes get different outputs to when I load the model and tokeniser locally even though I’m using the exact same parameters.
To generate the summary locally I run:
device = 'cuda' if cuda.is_available() else 'cpu'
inputs = tokenizer(txt, return_tensors='pt')
summary_ids = model.generate(inputs['input_ids'].to(device), no_repeat_ngram_size = 2, max_length=75, top_k = 50, top_p=0.95, early_stopping = True)
summary_3 = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in summary_ids]

to generate the summary using pipeline / inference api I run:
output = query({
    "inputs": txt,
    "parameters" : {"max_length": 75,
        "no_repeat_ngram_size": 2,
         "early_stopping": True,
        "top_k": 50,
        "top_p": 0.95},
})

output[0]['summary_text']

or
summarizer = pipeline("summarization",  model="kaizan/production-bart-large-cnn-samsum", use_auth_token=TOKEN)
output = summarizer(txt, max_length=75, no_repeat_ngram_size = 2, top_k = 50, top_p = 0.95, early_stopping = True)
output[0]['summary_text']

In the pipeline / inference API case I get exactly the same output but when I run it manually I get a different output which makes me think there is a variable / seed value that might be set in the pipeline somewhere which I’m not dealing with when running this set up manually. Does anyone know what the variable causing this difference could be?
Thanks,
Karim