hey @ad26kr, thanks for the suggestion!
one complication i see is that some metrics like glue are paired a subtask:
metric = load_metric('glue', sub_task)

so it’s not clear what should happen if someone passes something like load_metric(["bleu", "glue"]) in your proposal. nevertheless, i invite you to open a feature request on the datasets library where it can be discussed in more detail: Issues · huggingface/datasets · GitHub 11