Hello
I developed a standard Conv1D model in Pytorch to predict time series with classification (4 classes).
I gathered a train set (5000 data) and a test set (1000 data). The model predicts daily data by batches and is quite efficient.
As the results were satisfactory, I then moved to the next step :

I trained my model
I saved the model
I used the trained model on daily new data (bringing data for prediction day-by-day, instead of by batch size).
The results were very deceptive (if not catastrophic).

Therefore, I checked what happen and tried to save my trained model, to clean the GPU cache, to re-load the model and to apply it to the test-set.
Herewith the original code :
    train_dl, test_dl = get_data_loaderRN(X_train, y_train, X_tests, y_tests, batch_size) 
    model = Conv_1D(input_shape, nb_classes, num_features, seq_length, batch_size, iter_model, iter_pre, dropout)
    model = model.double()
    model.cuda()
    criterion = nn.CrossEntropyLoss()  
    optimizer = torch.optim.Adam(model.parameters(), lr= learning)        
    y_predtrain, running_loss = train_Conv_1D(model, criterion, optimizer, epochs, learning, verbose, train_dl)
    acc_train = accuracy_score(y_train, y_predtrain)
    y_predsubt, y_pred_proba, running_loss = eval_model_a(model, nb_classes, criterion, test_dl)
    acc_test = accuracy_score(y_tests, y_predsubt)

And the code with save and re-load model :
    train_dl, test_dl = get_data_loaderRN(X_train, y_train, X_tests, y_tests, batch_size) 
    model = Conv_1D(input_shape, nb_classes, num_features, seq_length, batch_size, iter_model, iter_pre, dropout)
    model = model.double()
    model.cuda()
    criterion = nn.CrossEntropyLoss()  
    optimizer = torch.optim.Adam(model.parameters(), lr= learning)        
    y_predtrain, running_loss = train_Conv_1D(model, criterion, optimizer, epochs, learning, verbose, train_dl)
    acc_train = accuracy_score(y_train, y_predtrain)
    ## Save model and clean GPU cache + re-seed
    torch.save(model, model_path)
    torch.cuda.empty_cache()
    torch.manual_seed(my_seed)
    torch.cuda.manual_seed(my_seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    ## Reload model and use it to predict
    model = torch.load(model_path)
    model.to(device)
    y_predsubt, y_pred_proba, running_loss = eval_model_a(model, nb_classes, criterion, test_dl)
    acc_test = accuracy_score(y_tests, y_predsubt)

The Save / Clean / Load steps are there to simulate what happened with the model used for daily prediction… and the results are far from those with the original model.
How could I address this problem ?
Thanks in advance for your support.
Best,
NB1. The output is double “y_predsubt” recors the predicted class and “y_pred_proba” records the likelihood associated by the model to each of the 4 classes.
NB2. I did the same with a LSTM model and get the same issue