Standard convention for test statistic for F-test? According to my textbook, Mathematical Statistics with Applications 7th. Ed, they write the test statistic in a F-test, that compares variances of two different populations, so that the larger variance is in the numerator and the smaller variance is in the denominator. But, using a base function in R, var.test(), it can return a test statistic less 1 than if the smaller variance is in the numerator. Which is the more formal and accepted convention?

I dispute what your textbook says.
It depends on how your using the F test.
A typical reason to use an F test is for ANOVA. In such a situation, you are cleverly using a test of variances to conclude something about means. Briefly, we want to see if a regression model using the group labels as predictors gives us less variance in our predictions than if we always predict the same value. Consequently, we only care to test in one direction: if the model considering the group labels has less variance than the model than doesn’t consider the group labels. Further, while an $F<1$ is possible in an ANOVA, by the way the regression is fit, the model using the group means cannot have a higher variance than the model with just an intercept.
Remember, though, that this is a test about the equality of means that cleverly uses variances, not a direct test about the variances of the groups under consideration.
However, the F test can be used to do a comparison of variances. Consider the weights of McDonald’s Big Macs and Burger King Whoppers. I want to know if the burgers have the same variability, so I weigh a bunch of each and run an F test on my measurements via var.test. In that case, we do not know which group has the higher variance, and we would be interested in knowing which burger is more variable.
It is in this latter case that the var.test function can return a variance ratio that is less than one, which indicates the burger with more variability is the second input into the function.
