(I like parts of several other answers and comments, but they all have their issues so here goes...)
"tl;dr"?* Just the bold* for a concise version.
Note that some answers or comments are about "why not have each pointer position directly correspond to a symbol--not a code at all." The novel quotation (below, and as in <PERSON>'s answer) explains that, but the question was simply "why not use numerals from 0 to 9", i.e. decimal instead of hexadecimal. So I'll start by taking ASCII as a given. In an endnote I'll say a little about more efficient schemes for "indicating the number corresponding to the alphabet", to quote the asker, since I guess he meant a code but possibly not ASCII. (And maybe it isn't as clear in the movie, as it is in the novel, that that's the code being used. I don't know, I was a little tipsy when I watched it.)
Answer to the second part first: simply "yes," sending codes in digits 0 to 9 is totally sufficient. In the novel, the quote (thanks to <PERSON>) that explains why to use a limited number of symbols is:
"Twenty-six letters plus my question card would be twenty-seven cards around the lander. Each one would only get 13 degrees of arc. Even if JPL points the camera perfectly, there’s a good chance I won’t know which letter they meant."
This makes it obvious: 0 to 9 means fewer cards, so also fine.
To those not mathematically inclined, it might seem to be implied that there is something special about hexadecimal (in which the digits A - F represent 10 - 15 and a digit's position indicates the power (exponent) of 16 instead of of 10) in relation to ASCII, such as in this other quote:
"So I’ll have to use ASCII. That’s how computers manage characters. Each character has a numerical code between 0 and 255*. 682
Values between 0 and 255 can be expressed as 2 hexadecimal digits."
But no: ASCII is just a table that "maps" a number to a symbol, so that number can as easily be the more familiar decimal with digits 0 to 9.
The first part--"why hexadecimal?"--isn't answered, at least in the film. Probably the best answer is an out-of-universe one: it seems like nerdy science stuff, and this movie loves its nerdy scientists. Hey, it worked for <PERSON> and <PERSON>.
In principle it is more efficient--fewer moves of the pointer--to have 16 "digits" (pointer positions) than 10. Indeed, the more the better until it becomes too many our favorite Martian to distinguish. Other stackexchangers have described, in varying detail and correctness, other encoding schemes which could take much better advantage of this, especially for our astro-botanist hero's circumstances. But <PERSON> and <PERSON> are the most on point by noting that, by using ASCII as in The Martian, the possible efficiency of having 16 vs 10 digits per pointer move is mostly wasted. To be more explicit about how little difference it makes in the actual case: In ASCII, the numbers that represent each of "space", 0 - 9, A - Z (but not a-z), and symbols except {|}~, are less than 99, so if you don't mind BEING A SHOUTY DICK TO YOUR MARTIAN BUDDY BY TEXTING IN ALLCAPS and can resist "hugging" <PERSON>}} (Can you? No, really--CAN YOU?? 'Cuz he's adorable!) you need no more pointer moves (2) per character with decimal than with hexadecimal*.
Endnotes:
* Hey <PERSON>, no you don't have to use ascii. Apparently you can communicate to NASA any method you make up that you want them to use. For instance, (still using hexadecimal if thinking numerically) one where the 15 most common letters are each represented by only one "digit" (one move of the pointer), 0 to F skipping 1, and the other 11 by the values 10 to 1A. The value '1' is skipped so that when it is sent it is clear that it is the first digit of a two digit code value.
Like ASCII it is a simple symbol encoding scheme, but the large majority of letters sent will take half as long. Sending numbers is given less priority (it wasn't mentioned in the novel quotes, or the movie I think). Even in ascii, the code point for each digit is a two digit number. 855