tldr;
Confidence intervals (ci) compute:

the probability that the true mean is in the given interval (usually written mu_n +- ci


Assumptions:

traditional confidence intervals statements only hold for statements about the value (parameter, random quantitiy, etc) we want to estimate being the mean

you have enough samples so that the analysis holds (e.g. the mean $mu_n = 1/n sum_i x_i$, where n>=30 is recommended)

If those assumptions hold (**i.e. your esitmating the true mean via the sample mean with a ± value **) then use the code bellow that I provided called torch_compute_confidence_interval for regression, classification, anything you want.

First, asfaik confidence intervals (ci) is an open research problem in deep learning (DL) - so more sophisticated answers probably exist. But I will provide a practical answer that I plan to use (and see others using when reporting results in DL).
To compute confidence intervals we have to understand a little bit of ci first. What they are is a probabilistic statement over the random surveys/samples of data sets that the mean you are trying to report is withing the reported interval. So when people say:
mean_error +- CI for p=95%

it means if you sampled 95 data sets you’d expect the true mean to lie in that interval 95 of the time (but you wouldn’t know which ones, so you can’t say for any specific interval you compute that the mean will be there).
This means you can only use it for reporting means. This is because the maths that goes behind it (which isn’t very hard) approximates the computation of the probability that the bound holds (or the confidence interval holds) by taking advantage that we can compute probabilities analytically for sample means because the approximate a normal according to the central limit theorem CLT. So the specific CI that is computed assumes the quanity you want to compute is a sample mean and computes your ± numbers using this normal approximation. Thus, usually it’s recomended to have n>=30 data points for the specific data set you are using but things can still work out nicely since ci can be computed with a t distribution instead of a normal (denoted z in stats software).
Given those assumptions you can simply do the following:
def torch_compute_confidence_interval(data: Tensor,
                                           confidence: float = 0.95
                                           ) -> Tensor:
    """
    Computes the confidence interval for a given survey of a data set.
    """
    n = len(data)
    mean: Tensor = data.mean()
    # se: Tensor = scipy.stats.sem(data)  # compute standard error
    # se, mean: Tensor = torch.std_mean(data, unbiased=True)  # compute standard error
    se: Tensor = data.std(unbiased=True) / (n**0.5)
    t_p: float = float(scipy.stats.t.ppf((1 + confidence) / 2., n - 1))
    ci = t_p * se
    return mean, ci

I’ve tested it and compared it to things specialized for classification and they agree in values up to 1e-2 so the code works. Output:
Connected to pydev debugger (build 213.5744.248)
x_bernoulli.std()=tensor(0.5040)
ci_95=0.1881992999915952
ci_95_cls=tensor(0.1850)
ci_95_anything=tensor(0.1882)
x_bernoulli.std()=tensor(0.5085, grad_fn=<StdBackward>)
ci_95_torch=tensor(0.1867, grad_fn=<MulBackward0>)
x.std()=tensor(0.9263)
ci_95=0.3458867459004733
ci_95_torch=tensor(0.3459)
x.std()=tensor(1.0181, grad_fn=<StdBackward>)
ci_95_torch=tensor(0.3802, grad_fn=<MulBackward0>)

For more details see my ultimate-utils library where I comment on the maths in the docs: ultimate-utils/confidence_intervals.py at e81a8c3c4425b33e00b3ade172705f20b626b2b1 · brando90/ultimate-utils · GitHub

Comments on DL
If you are reporting the error of a specific model e.g. neural net, like this you are more or less reporting that the true mean error for that very specific neural net and weights lies in those bounds. But as I said this is an open research area so fancier things must be available e.g. consider some layers are actually random, etc.