Could we estimate replicability of empirical research with conformal predictions? A review article Threats of a Replication Crisis in Empirical Computer Science reviews reproducibility issues. The authors present distinctions among repeatability, replicability, and reproducibility. They were a bit pessimistic. Now, the question is if there is a statistically "robust" technique or framework that actually provides confidence on repeatability? For example, how can we report a probability or an index that the result of randomised control trials are repeatable: given a significance level and the effect-size? Informally, we would report, $(\mathbb{1}_{s},\epsilon, \rho)$, where $\mathbb{1}_{s}$ is the significance level, effect size $\epsilon$ and $\rho$ is the confidence on both statistical significance and the effect size. Maybe this can be achieved with conformal predictions.
Note: The question is actually about if we can use conformal predictions in getting estimate for repeatability.

Yes, we can absolutely construct predictive p-values and prediction intervals aimed at covering the observed result of a repeated experiment.  However, I do not think this will solve what many call the "replication/reproducibility crisis."  If an experimental result is indeed an exceedingly rare event, not only might we incorrectly reject a hypothesis about a population-level parameter, we might also incorrectly reject a hypothesis about a future experimental result.
If we are always vigilant to remember that no hypothesis is proven false with a single small p-value, nor is it proven true with a large one, then there is no replicability crisis.  We need to temper our expectations when it comes to interpreting statistical results.
Under a Neyman-Pearson framework we have inductive behavior based on controlling a long-run error rate.  Under a Fisherian framework we have inductive reasoning based on an evidential p-value providing the weight of the evidence.  In either approach if we "rule out" a hypothesis only one of two things can be true: either the hypothesis is true and we have witnessed an exceedingly rare event, or the hypothesis is false.  These statements are falsifiable through repeated sampling.
The only irrefutable evidence is to actually repeat an experiment many times to see if the results are replicable/reproducible.  More emphasis should be placed on confidence intervals and prediction intervals - the set of all hypotheses that do well at explaining the observed data.  Additionally, we should examine both meta-analytic (conformal) approaches as well as multiplicity-adjustments.
The role of journal publications is to highlight the work of researchers that is worthy of attempted replication. Nothing more.  A single failed attempt at replicating a small p-value is not irrefutable evidence that the conclusions of the original experiment represent a false positive (there are many hypotheses in an observed confidence interval that do well at explaining the observed result).
