Q:

Stack overflow with Fibonacci's recursive call

How exactly is the Java stack set up?
For university, I shall determine what the biggest possible Fibonacci number that is calculated by a recursive method and can be handled by the stack.
The interesting thing is: Tests showed that it doesn't matter how much -Xmx and -Xms the JVM has. I am able to run up to Fib(4438). But the results aren't consistent. Somtimes it goes down to 4436.
Is there formular for the stack?
Any increase of the stack via -Xss 4096m doesn't make a difference.

A:

-Xmx and -Xms sets memory accessible for JVM heap, you need to increase stack size, you do this with the help of -Xss option.

A:

You've misinterpreted the assignment. Stack size matters very little. The problem is exponential. And you cannot possibly have gotten to Fib(4438) with the naive recursive program. Using the following code, you'll be lucky if you make it to Fib(50):
public static BigInteger f(int n) {
    if (n == 0) 
        return BigInteger.ZERO; 
    if (n == 1) 
        return BigInteger.ONE; 
    return f(n-1).add(f(n-2));
}

