I want to do active (online) learning of a statistical model.

This means that I have an initial train dataset (x-y pairs) which is known at compile time.

However, by the active nature (online), more data comes at runtime, from a 3rd party program (a cpp simulation program).

I am doing this inside python using `GPytorch`, and I am calling the 3rd party program via the `subprocess` python module.

**My problem is of programming type and not GPytorch or statistical type, thus my question here.**

The workflow is:
python dictates at which input parameters to run the .cpp, create a new folder named based on the params, enter the folder, run the .cpp, collect data which appeared in the folder, update the statistical model, python dictates at which input params to run the .cpp, create a new folder named based on the params, enter that folder, run the .cpp, collect data which appeared in that folder, update the statistical model ...
(for, say, 100 times).

Inside a WSL1 terminal, I usually run the .cpp code using: `$ mpirun -n 1 smilei namelist.py` , where this command is run inside a folder which contains both the executable `smilei` but also the .py called `namelist.py`

The python workflow returns exit code 0 (and the necessary data) at the first iteration of my active learning loop, but fails and returns exit code 1 at the second iteration. It basically does its job for the first iteration, but fails for the 2nd one.

I tried with `subprocess.run()` and with `os.system()` (see the code below with all my trials preceeded by comments) where inside the parantheses I type the command I usually run inside a BASH WindowsSubsytemForLinux1 terminal to run the 3rd part cpp program.

I cannot debug why it fails for the second time.

I tried to print out the `stdout` and the `stderr` of the subprocess, they both return empty lines when queried, there are no such things it appears (no stdout and no stderror), for the second iteration of the active learning loop.

I know the code below might look complex, but it's not. It just follows the workflow I presented above.

```
def SMILEI(I):
    os.chdir(top_folder_path)
# create a new folder called a0_942.782348987103 (example value)
    a0 = "%.13f" % a0_from_IntensityWcm2(I)
    dirname = "a0_%.13f" % a0_from_IntensityWcm2(I)
    os.mkdir(dirname)
# enter the created folder
    os.chdir(top_folder_path + "/" + dirname)
    print("We change the directory and entered the newly created one!")
# copy general namelist into this newly created folder
    shutil.copy(top_folder_path + "/" + general_namelist_name, ".")
    print("We copied the general namelist!")
# add the a0 value to the general namelist, i.e. add a line a0 = 942.782348987103 , at row 8 (empty row) in the general namelist.
    with open(general_namelist_name, 'r+') as fd:
        contents = fd.readlines()
        contents.insert(8, "a0 = {}".format(a0))  # new_string should end in a newline
        fd.seek(0)  # readlines consumes the iterator, so we need to start over
        fd.writelines(contents)  # No need to truncate as we are increasing filesize
    print("We modified the general namelist to contain the line a0 = ..., at line 8")
# rename the modified namelist
    os.rename(general_namelist_name, particular_namelist_name)
    print("We renamed the general namelist to namelist_Xe_GPtrial_noOAM_a0included.py")
# run the simulation
    print("We'll be now running the SMILEI command inside the folder: ")
    print(os.getcwd())
    print("The smilei executable's absolute path as dictated by os is: ")
    print(os.path.abspath("../smilei"))
    cp = subprocess.run(["mpirun", "-n", "1", os.path.abspath("../smilei"), particular_namelist_name], 
                                # stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
                            #stdout=subprocess.PIPE, stderr=subprocess.PIPE)
                            #capture_output=True)
                        )
    print("The return code is: ")
    print(cp.returncode)
    #os.system("mpirun -n 1 ../smilei {}".format(particular_namelist_name))         
    #subprocess.run("mpirun -n 1 ../smilei {}".format(particular_namelist_name), shell=True)        
    #print(cp.stdout) # Y
    #print(cp.stderr)
    #print(cp.returncode) 
# get the results of the simulation
    # os.chdir(top_folder_path + "/" + dirname)
    # print("We changed the directory again and entered again the newly created one!")
    S = happi.Open(".")
    pbb = S.ParticleBinning(0).get()
    results_dict = dict()
    for z in range(len(pbb['data'][-1])):
        results_dict['c_%d' % z] = pbb['data'][-1][z]
    return np.asarray(list(results_dict.values()))

if __name__ == '__main__':
    # Initial Train Dataset:
    x_train = torch.from_numpy(np.array([0.1, 0.3, 0.5, 0.6, 0.8]))
    y_train = torch.from_numpy(np.array([0.1, 0.2, 0.3, 0.4, 0.5]))

    # initialize likelihood and model
    likelihood = gpytorch.likelihoods.GaussianLikelihood()
    model = ExactGPModel(x_train, y_train, likelihood)

    model.train()
    likelihood.train()

    # "Loss" for GPs - the marginal log likelihood
    mll = gpytorch.mlls.ExactMarginalLogLikelihood(likelihood, model)
    optimizer = torch.optim.Adam(model.parameters(), lr=0.1)  

    training_iters = 10
    for i in range(training_iters):
        optimizer.zero_grad()
        output = model(x_train)
        loss   = - mll(output, y_train)
        loss.backward()
        print('Iter %d/%d' % (i+1, training_iters))
        optimizer.step()

    Xn = x_train
    Yn = y_train
    ######################################################################################
    # The Active-Learning (AL) loop:
    budget_value = 100
    for i in range(budget_value):
        OldValues = lhs(1, samples=100)
        Xref = range_transform(OldValues, 10.0**20, 10.0**25)
        x_nplus1 = xnp1search(model, Xn, Xref) # x_nplus1 is Intensity in W/cm2 at which to run SMILEI next for Active-Learning the GP fit
        y_nplus1 = SMILEI(x_nplus1.detach().numpy())[53] # SMILEI(x_nplus1.detach().numpy()) returns an ndarray of shape (55,)
        Xn = torch.cat(   ( Xn, torch.reshape(x_nplus1, (1,)) )   )
        Yn = torch.cat(   ( Yn, torch.reshape(torch.from_numpy(np.reshape(y_nplus1, (1,))), (1,)) )   )
        model.set_train_data(Xn, Yn, strict=False)
        for j in range(training_iters):
            optimizer.zero_grad()
            output = model(Xn)
            loss = -mll(output, Yn)
            loss.backward()
            print('Iter %d/%d' % (j+1, training_iters) + 'inside AL step number %d/%d' % (i+1, budget_value))
            optimizer.step()

```

Why does it fail for the 2nd time?

I simply don't see it. And I cannot debug it, I don't get any error message or anything, it just doesn't run the simulation inside the 2nd created folder, that folder at the end of the python script only contains the `namelist_Xe_GPtrial_noOAM_a0included.py`, with the a0 value included (as it should).

Thank you!