I have been using an HPC cluster for a few years now and regularly need to submit jobs that process large amounts (often over 100) for large files like BAM files etc. 

Despite some experience, I feel I am lacking some of the understanding of the basic concepts such as:

 - How to estimate how much RAM and runtime a job will need - I know, it's mostly based on experience and no one can ever answer that for me
 - The relationship between how much RAM you give a job and how much runtime? Are these 2 parameters independent? Will one affect how long you are in the queue more?

**So my question is:** 
 
Does anyone know of a nice book/online resource that explains these basic concepts and ideas? I find myself struggling to answer these simple questions and the documentation out there is very often geared towards explaining complicated details about how supercomputers work. I am interested in all that but I would like to start with a dummed-down version first that focusses on how to submit jobs properly. 

Any ideas? I should say, the cluster I use works with the Sun Grid Engine[SGE] system.


1. Nope, you just guess, then optimize.
2. Nope, not usually. I have a few very lazy scripts that use a ton of RAM (I have it, why not?) they're fast, some things use very little ram and can take a relatively long time.

So, if your goal was to come up with some metric to allocate your job wall time limits on the RAM for the job, don't, it won't work.

> I find myself struggling to answer these simple questions and the documentation out there is very often geared towards explaining complicated details about how supercomputers work.

I think you sort of answered it yourself, you really only need to dig into that documentation if you're doing things that really rely on the intricate details of a cluster. Just like with coding, don't worry about optimizing till you need to.

Here are some tips:

 - Don't worry about over allocation of resources. As long as you're not
   allocating 50 128 core/1TB RAM nodes to run 1 instance of gzip each,
   you'll be fine. If you take a guess that a job needs 64GB and you
   only use 47, its not the end of the world. If you ask for 8 cores and
   only manage to keep 6 loaded, oh well, next time use less. 
   
 - Considering most systems kill jobs that exceed their allocated   
   resources, it doesn't hurt to round up. If you undershoot and the job
   gets killed, you've still wasted wall time. "Wasting" some ram and   
   some cores is better than restarting an 8 hour job because you needed
   64 and only asked for 48.

   

 - Over-allocating wall time is even less of an issue, the resources are
   free once the job is done, if it takes 1 hour or 1 month. All that   
   happens is your jobs may have less priority over shorter ones.

   

 - Typically programs that rely on a database need that much memory to  
   load the database. So, unless you know the DB stays on the disk, a   
   good idea is adding the size of the query and database and   
   multiplying by 1.25. Tweak accordingly.
 - Small, short jobs take priority over long, large jobs.
 - Don't constantly undershoot your wall time, you'll either piss off   
   the admins with constant requests for increases or constantly waste  
   time having to resubmit.
 - Make good use of the scratch space on nodes and be careful about   
   bogging down the file system by having a ton of jobs trying to I/O   
   the same files or directories. If jobs need to constantly access   
   something (say an index for HiSat2), copy it to the scratch, and have
   the job read it from there.
 - Interactive sessions are great. Use them to debug and benchmark jobs.
   This is a great tool to find out little things that get you in   
   trouble, like a program starting more threads than the CPUs you   
   allocated for it.
 - Different programs behave differently as you add cores. Some don't
   have substantial memory increases, some do. Experiment to find out.
 - When in doubt, ask the admins. They know the system and may have
   helped another user do something similar. 


Try stuff, break stuff, bother the admins. Your goal is to get work done, the cluster lets you do if faster, you're not there to optimize HPC software. A cluster is a tool, use it like a hammer. 