I am doing a Preranked GSEA Analysis on the output of a differential expression analysis (genes ranked by sign(shrunken lfc) * -log10(pvalue) ).

I notice that when I feed the exact same files into GSEA, it can give me fairly different FDRs.

For example, when I ran the analysis the first time, it said it detected "32 gene sets are significantly enriched at FDR < 25%".

Ten minutes later, I ran the same files again, and it detected "4 gene sets are significantly enriched at FDR < 25%".

I know that there is some element of randomness in the GSEA algorithm because it sets a seed based on timepoint, but I'm surprised to see such a large difference in the number of pathways passing threshold from run to run.

Is this to be expected?

Thanks for your help.

**UPDATE** I also notice that the four gene sets called enriched in Round 2 are completely non-overlapping with the thirty-two gene sets enriched in Round 1. This is surprising and a little concerning to me.
It seems the problem is either that I needed to run more permutations or just that I had a very weak phenotype.

Pablo Tamayo at UCSD wrote the following in response to my question on the gsea-help listserv: " If you explicitly set the seed and get the same results then that is the expected behavior. The variability across different seeds (runs) is also expected to be higher on the FRDs than other statistics. The lack of robustness you are observing can be related with not running enough permutations and can be exacerbated because the results are rather weak to begin with. Have you tried running a much larger number of permutations?"

In response to his prompt, I reran the analysis with a higher number of permutations (initial # of permutations was 1000). The pathways called significantly enriched by FDR were still very variable, but the pathways called significantly enriched by nominal p-value much less so.

I decided eventually that this likely meant I was trying to chase down a very tiny-or-nonexistent phenotype, as I have since analyzed other databases with much more obvious and stable outcomes.