Question: I have a few interests, and among them belong fiction, math and computer science, and so naturally I like to imagine/think about the possibility of a piece of fiction living in the intersection of these fields. I wonder, what does it mean to "code a novel"? Is there already such a thing as algorithmic fiction?

Answer: 
I used to be a coder, too (PHP, Ruby, Javascript, etc). But my main thing was design. And I used to have the same wish as you: I wanted to think of fiction writing as "designing" a story. But later on, I understood that, even though they share some similarities, they are different kind of thought processes. It was only when I accepted that that I figured out how to be a better writer.

So I don't think you can code or design a novel. You can only *write* a novel.
What about interactive novels - e.g. [Visual Novels](http://en.wikipedia.org/wiki/Visual_novel), a genre rather popular in Japan?

You can certainly *code* these. Western world knew these as "text adventure games" but they usually featured a map of locations between which you could go. Visual Novels, OTOH, read like a book, with points where you make a decision. They are more similar to "Make your own adventure" books, but frequently surpass these in means of complexity by strides, most complex of them often containing thousands of decision nodes, many of them implicit - not obvious to the player, but resulting from prior choices.