I tried to edit my answer to respond to some comments, but for some reason the edits didn't go through, so here's my response.
I think what I'm trying to say is that:
1. Books, or stories in general, represent perspectives. Perspectives can mean a view on an issue, or a perspective can mean how that perspective is communicated. To give a hypothetical example, if two authors write the same book, but only one author knows how to spell, the perspective being communicated in those two books is very different.
2. There is always stuff to talk about when you're talking about a perspective. Academics wrote an entire book on The Girl with a Dragon Tattoo, even though that's the poster child for "popular fiction". Even if you aren't an academic or don't like academic thinking/writing, there is always things to talk about. 460
If you think a book is bad, then you can talk about why you think it's bad!
3. The question is really what perspectives are useful for you to have more of. Academia (the primary source of what books count as "literary", at least in the USA: if a book is added to enough syllabuses in colleges or is taught in enough public schools, etc.) has certain criteria for what perspectives they find useful.
Do you think that someone who just finished <PERSON>'s Ulysses has gained something more that someone who completed <PERSON> code?
Well, I would argue that an alien from outer space, who has had no contact with humans, would probably learn more from a book that has resonated with a lot of people and that represents the "popular" culture than a book that represents "elite" culture. An alien reading Da Vinci Code might not learn very many historical facts, but they would learn something about the value of historical facts to a large portion of the public.
Another example: Great Gatsby is a great book if you want to teach high schoolers about symbolism, which is a goal of academia. If you're looking for characters with depth, then <PERSON> is probably not the book for you, unless you're interested in reading something narrated by a detached, boring man with no personality, who is describing a different man with no personality other than the fact that that he is rich and obsessed with a rich woman. But again, let me emphasize, the fact that the characters are boring does not prevent you from learning things from the fact that they are boring. The fact that people ignore the boringness of the characters because they are rich certainly says something.
What you get out of a book depends on you and what you are looking for, not the book. 460