How can I learn how to create a new statistic? There are an infinitude of tutorials on how to use this and that statistic, but I can’t find anything at all about how to invent a statistic, and validate it. I can guess, from the papers: You make up a scoring algorithm, and then use simulation to characterize it, and unsure that it meets my, and some set of required desiderata. I can a guess what those sims entail, but usually the code for those isn’t provided. Is there a textbook, or even a clear explanation of exactly how to go about this? It looks like the folks who do it know something that I can find nowhere clearly explained. Is there an advanced course called “Inventing new statistics”, or does everyone who does it just glean how it’s done by reading papers of others who have done it? Nb., I’m talking about statistics here, not new ML methods. I know how to run a cross-validation. What I don’t know, and can’t find clearly explained, is how to figure out how to characterize and validate a statistic.
To be more specific, given that there are an infinitude of functions, and probably a somewhat smaller infinitude that might do what I want, how do I decide between different members of the smaller group, and then prove that it performs the basic desiderata of a general statistic. Let me be more specific: Suppose that we didn’t have a t-test, but wanted to invent the t statistic, a function that generates a score that can be interpreted as a p-value - i mean, today, armed with … Python or R (but, obviously no built in t-testing packages) … not how it was done historically. What steps would one take to do this?

A statistic is just a function of data. To invent a new statistic, all you need to do is define the function. The procedure for understanding your statistic will depend on what you want to understand. It could involve a theoretical analysis (proving mathematically that it is a member of some class of functions, for example), analyzing the behavior of your statistic on simulated data, and/or analyzing the behavior of your statistic on real data.
