I've already used my entire dataset in a regression, should I not use that as a prediction model? At the hospital I work at we were writing a paper on what variables about a patient predict whether they'll return for a follow-up visit.  We included variables such as age, gender, distance from their home to the hospital, mechanism of injury and other things like that. We had about 600 patients to examine and so we ran a multiple logistic regression with yes/no return as the outcome, and we did this with everyone in our dataset (everyone with that condition at our hospital).
Well we wrote the paper and then someone decided we should try to create an online prediction tool. You could put in variables about a patient, and it would return a guess about whether or not the patient would return, based on our previous regression model. To help me create an online prediction tool I've used this tutorial using R and Shiny and I noticed the author split his data into training and testing sets
Problem is: I never did that. Reading comments such as this I think I understand why someone would split their data, but my question now is:
What can/should I do about it?

*

*I've already used all my data.  Would it be best to delete everything I've done, go back, split the data and start over?  (We didn't publish the paper or anything)

*Should I just proceed? Can an argument be made for NOT splitting the dataset?


With something like a dozen variables to start with and then several tries on which model works best but only 600 data points on a binary output you have a severe risk of overfitting. That is your model works very well on the data you have but maybe its predictive power for new patients is not very good.
What you can do with splitting the data is getting a feel for how much of an issue that is for your specific data. I would not throw away what you have but if you have programmed this in R it should be relatively easy to split the data and check whether you have overfitting.
So split the data randomly into say 500 patients in training and 100 in testing and then look at the following:

*

*is the best model on this set of 500 patients the same as on the whole set of 600?

*how much worse does it perform on the test cases than on the training cases?

*how much better is your complicated model relative to a model that only uses the single variable that is the best predictor?

Repeat this with different random choices of the split into training and testing. The goal is to gain an understanding whether your model is only a good fit on your existing data or whether it is actually a good tool to predict the behavior of future patients.
