Question: I have a few interests, and among them belong fiction, math and computer science, and so naturally I like to imagine/think about the possibility of a piece of fiction living in the intersection of these fields. I wonder, what does it mean to "code a novel"? Is there already such a thing as algorithmic fiction?

Answer: 
Not quite algorithmic, but there are quite a few books on story structure that you may find interesting. If you're like me, understanding how and why stories work can provide you some sort of framework out of which you can build a story.

It's about movies, and half of the people here will hate me forever for daring mention it, but I found **Save the Cat** very useful ([http://www.amazon.com/Save-Cat-Blake-Snyder-ebook/dp/B00340ESIS](http://rads.stackoverflow.com/amzn/click/B00340ESIS))
What about interactive novels - e.g. [Visual Novels](http://en.wikipedia.org/wiki/Visual_novel), a genre rather popular in Japan?

You can certainly *code* these. Western world knew these as "text adventure games" but they usually featured a map of locations between which you could go. Visual Novels, OTOH, read like a book, with points where you make a decision. They are more similar to "Make your own adventure" books, but frequently surpass these in means of complexity by strides, most complex of them often containing thousands of decision nodes, many of them implicit - not obvious to the player, but resulting from prior choices.