How to learn and apply the scientific method in machine learning?
Rigor Theory. I wish to learn the scientific method and how to apply it in machine learning. Specifically, How to verify that a model captured the pattern in data; How to rigorously reach conclusions based on well-justified empirical evidence.
Verification in Practice. My colleagues in both academia and industry tell me measuring accuracy of the model on testing data is sufficient, but I don't feel confident such criteria is sufficient.
Data Science Books. I have picked-up multiple data science books, like Skiena's manual, Dell EMC's book, and Waikato's data mining. Eventhough there had been a section for diagnosing the model and measuring results, My instinct worries these are heuristics but not rigour basis.
Scientific Method Books. 57
Searching for scientific method I found, Statistics and Scientific Method: An Introduction for Students and Researchers and Principles of Scientific Methods , Which seem to answer the crux of my question. I am planning to study both of them.
My Questions. Here are couple of questions I hope to gain guidance on, From your wonderful community. - Is it feasible to rigorously apply the scientific method in machine learning applications like recommendation engines or social sciences, Or is it the case that so far our scientific/technological advancement didn't reach that degree of maturity, and that the best we can hope for is heuristics-based approximations. - Is it feasible to do machine learning in practical industry, by applying the scientific method, or it's the case that industry leaders prefer cheap heuristics in order to minimize project's costs? - Are the scientific method books I mentioned above useful for enhancing my own skills in machine learning? Are they worthwhile the effort and time? - Are you aware of better alternative resources for learning the scientific method? Are there more helpful courses or recorded lectures? - Do you have any recommendations or advise, While studying the scientific method, For someone who is mainly motivated by machine learning in industry like recommendation engines applications and logistical optimization?
You don't need to answer all questions, Just select the points you believe you can contribute the best. Feel free to give me whatever general feedback or advise.
Best, 57