For human annotation projects, what are some commonly used metrics to assess grader reliability? Lots of machine learning datasets are now created by having human raters annotate and provide labels to questions. Usually, a gold set is the most robust way of seeing if the raters are doing a good job. I am wondering what are some metrics commonly used for understanding how well a rater is doing?

In the absence of a gold standard or criterion variable, you can assess inter-rater reliability by computing the intraclass correlation (ICC). The ICC decomposes variance in the ratings that is due to differences between raters and differences between ratees (the questions, in this case).
To follow-up on the comment of Arya McCarthy above (you can’t get something for nothing): This approach would, more or less, take the average rating per question as the 'gold standard'.
If each question was rated by the same set of raters, you can estimate ICCs using function icc from R package irr. If (some) questions were rated by different sets (and numbers) of raters, it is probably more convenient to use function lmer from R package lme4 to compute ICCs.
