Hello everyone,

I don't really know how to put this, but I'm recently trying to improve myself in coding and making tools. I was mostly doing a lot of bash things because it was the easiest way. But I know I should review my habits to be more portable.

This was my first year working on a cluster environment (SLURM), and I think I still don't get all the concepts behind it.
I was fine with just doing simple "sbatch", "srun" or "sarray" command, but I would like to understand more about how it works.

A concrete example : 
I want to create a python script that call a tool (minimap2) installed on our cluster. This tool can only be used after a "module load" of the appropriate module (in that case bioinfo/minimap2-2.5).

So I was giving a shot with `subprocess.call` and `subprocess.run` (from which I understood they were sort of the same but not from the same python version ?) to perfom that module load. 

But it kept failing, and I could'nt find anything about it on the net. I feel like I'm doing something abberant...

I'm not even sure if what I am trying to do is relevant (slurm module load from within python ?).
Can anyone correct me and maybe explain what I'm doing wrong ? Maybe if you have any website or book to recommand to understand better this topic. I feel very nooby, but I still want to improve.

Thanks for your help, sorry if not a proper Biostar question,

Roxane

The reason you are having trouble loading a module on the HPC with `subprocess.call`, etc.,  is because each subprocess is executed in a new, independent process. So even if you did a `subprocess.call` for something like "module load xyz", the loaded module will not persist into your following calls to use that program. If you really wanted to do that, you would have to do something like `subprocess.call('module load xyz; run_my_program')`, probably with the `shell=True` option enabled I believe.

Honestly your best bet is to use a workflow manager like Snakemake or [Nextflow](https://www.nextflow.io/). You should not couple your scripts directly to the HPC environment you are using, because they will then become unusable on other environments. For example on Nextflow, you can easily create a pipeline to run your script on your samples, and then configure Nextflow to submit the task as a batch job on [SLURM](https://www.nextflow.io/docs/latest/executor.html), and also configure it to first load the correct [modules](https://www.nextflow.io/docs/latest/process.html#module). Something like this would be the most portable, since it will allow you to update the execution configuration for a new system without having to modify any aspects of your pipeline's tasks. 