I've already used my entire dataset in a regression, should I not use that as a prediction model? At the hospital I work at we were writing a paper on what variables about a patient predict whether they'll return for a follow-up visit.  We included variables such as age, gender, distance from their home to the hospital, mechanism of injury and other things like that. We had about 600 patients to examine and so we ran a multiple logistic regression with yes/no return as the outcome, and we did this with everyone in our dataset (everyone with that condition at our hospital).
Well we wrote the paper and then someone decided we should try to create an online prediction tool. You could put in variables about a patient, and it would return a guess about whether or not the patient would return, based on our previous regression model. To help me create an online prediction tool I've used this tutorial using R and Shiny and I noticed the author split his data into training and testing sets
Problem is: I never did that. Reading comments such as this I think I understand why someone would split their data, but my question now is:
What can/should I do about it?

*

*I've already used all my data.  Would it be best to delete everything I've done, go back, split the data and start over?  (We didn't publish the paper or anything)

*Should I just proceed? Can an argument be made for NOT splitting the dataset?


I disagree with the consens that this is fine. I think it's not, because I can construct a better model than you did: a hashtable that remembers the entries.
Performance estimation
Having a model created from data is fine, it's the first step. But this itself is worthless: we need to assess the performance of the model (otherwise, a random model may be better).
To assess the performance of the model (and distinguish it from random guessing), we need some data.
Overfitting
Now you could just use the same data as you already used to assess the performance. But this can lead to a bias: your model could have overfitted to the data and, in an extreme case, "remember" (hash table) the data. (remark: this is indeed less of an issue with less powerful models such as logistic regression)
Low statistics: cross validation
As mentioned, ways out are to use resampling methods that do not really reduce the sample size used: This can be done by either using bootstrap methods or cross validation. They have their own advantages and disadvantages, however they tend to perform similar in most real-world cases.
*I would suggest you to do Cross-validation to get an estimate, but then, any technique is fine.
Why you need this
For a paper, where you claim to have developed a model, it seems crucial to provide an unbiased estimate of the performance of the model (for completeness: or a very strong theoretical motivation).
I understand that it means you would need to redo some stuff, but that should not be a lot actually. And it is also worth to maybe contact someone who understands more of it: if you say you just tried a bit around, there are many many more pitfalls (and possible improvements) that you can have. Data scientists are a thing these days ;)
