Solved by sgugger in post #2 
              
              
                compute_metrics should be a function that takes a namedtuple (of type EvalPredictions) and returns a dictionary metric nane/metric value. 
Look at the text classificatione example or the course section on the Trainer.