Evaluating a survival model when focus in on "real time" Assume a pure prediction problem.
Say I want to evaluate a prediction-focused survival model in the context of actual dates instead of at a specific survival time or integration thereof.
What would be the correct way to approach this/which metrics would be the correct measures of calibration/discrimination in this case?
An Example:
Assume a survival model based on only right-censored data.
Time-since-entry is measured in months. I obtain $\hat{S_i}(t_i|X_i)$. Calculating this survival probability for each $i$ with respect to a specific date (e.g. january year X), makes it necessary to use different $t_i$ for each $i$ (for the population still at risk at this specific date). Of course, a little rounding bias gets introduced due to calculating survival times and then translating them back to real time.
1. How to calculate the (time-dependent) AUC and the Brier/Graf Score in this setting?
2. Specifically, does one even need weights of the censoring distribution for the brier score/AUC? (see e.g. here)

It makes the most sense to evaluate the model with respect its original time origin, with time = 0 as study entry date. If the model's performance differs depending on the calendar date of time = 0 for an individual, you probably haven't incorporated calendar date of entry appropriately into your model. Check that first, and fix if necessary instead.
The need for inverse probability of censoring weights depends on the nature of the data and the use of the model. Weighting can help with causal inference, as explained by Hernán and Robins in Causal Inference: What If. Weighting won't necessarily work as hoped, however, if assumptions aren't met. See for example Howe et al, Limitation of Inverse Probability-of-Censoring Weights in Estimating Survival in the Presence of Strong Selection Bias, Am J Epidemiol 173: 569-77, 2011.
