Hi! This is a simple question but for some reason I’m having a hard time finding the answer. It says here (https://pytorch.org/ignite/metrics.html 14) that metrics doesn’t store the memory of the entire output history of the model.
However, the evaluation loop predicts y values (a regressive MLP), and the MSE is computed with the ground truth y values per batch. Therefore, the “mse” returned by the evaluation metrics thing gets overriden every batch. I want to have the “mse” of the entire validation set - this seems not compatible with the current setup.
def create_evaluation_loop(
    model: nn.Module, cfg: DictConfig, name: str, device="cpu",
) -> Engine:

    # Loss
    if cfg.sum:
        loss_fn = torch.nn.MSELoss(reduction="sum")
    else:
        loss_fn = torch.nn.MSELoss()

    def _inference(engine, batch):
        model.eval()
        with torch.no_grad():
            x, y = batch
            x, y = x.to(device), y.to(device)
            y_pred = model(x)

            if cfg.house:
                factor = 6.0
            else:
                factor = 1777.0

            # pdb.set_trace()

            y = y * factor
            y_pred = y_pred * factor

            if cfg.envelope:
                y_hat_env = torch.tensor(np.abs(hilbert(y.cpu().detach().numpy())))
                y_pred_hat_env = torch.tensor(
                    np.abs(hilbert(y_pred.cpu().detach().numpy()))
                )
                mse_val = loss_fn(y_hat_env, y_pred_hat_env).item()
            else:
                mse_val = loss_fn(y, y_pred).item()

        # Anything you want to log must be returned in this dictionary
        # pdb.set_trace()
        infer_dict = {
            "loss": mse_val,
            "y_pred": y_pred,
            "y": y,
            "ypred_first": [y[0], y_pred[0]],  # * mean + stdv,  # * mean + stdv,
        }

        return infer_dict

    engine = Engine(_inference)

    engine.logger = setup_logger(name=name)

    metrics = {
        "mse": Loss(
            loss_fn,
            output_transform=lambda infer_dict: (infer_dict["y_pred"], infer_dict["y"]),
        ),
    }

    for name, metric in metrics.items():
        metric.attach(engine, name)

    return engine


Also,  what is the  difference between the trainer which returns an “output” and the evaluator which returns the infer_dict? The difference comes when you do “metrics = {
“mse”: Loss(
loss_fn,
output_transform=lambda infer_dict: (infer_dict[“y_pred”], infer_dict[“y”]),
),
}” and then “attach” the engine and name together… but im not really quite clear on what that does. I know for fact that with the trainer, the output dictionary is very online in that it stores no history. So  I’d assume the “attach” stores the history, but then the link above specifically says that it doesn’t?
Thanks for your time!