Your concerns are certainly legitimate, and I'm sure there are some situations where they apply. However, in many physical sciences (I would venture to say the vast majority), simulations are not chaotic. They are based on stochastic models that have converging behavior. So, in general, it really is fair for most people to rely on the stability of the simulations.<p>As for your other points, the issue of simulation accuracy is already taken very seriously in the areas where I've seen it used.<p>Let me give an example from my field (high energy physics). The pre-eminent event generator of choice is called PYTHIA. It incorporates as much known physics as possible, and is continually updated. There are groups that regularly convene to compare the distributions it produces to those seen in real experiments and adjust the parameters to improve the results. The code is open source, so you can readily make improvements and submit them to the maintainers for inclusion in official releases.<p>Now the primary uses of these simulations are to either tune your analysis to separate signal from background or calculate corrections for different kinds of detector acceptance issues. No reviewer would ever accept a publication that used simulation if the paper did not include clear evidence that the simulation was valid, usually in the form of a data/simulation comparison.<p>Regarding reproducabilility, I think you're a little off the mark. The idea is not that you should simply repeat my analysis to get the same answer (this important verification step should be, and usually is, present in all scientific groups). Ideally, you should collect your own data, make your own simulations, and do your own analysis. Then we should see if we got the same answer.<p>Edit: grammar