Database implementation details - per row header overhead
The problem that prompts this question is related to creating ginormous inverted indices similar to those used when building IR systems. The general mantra from the IR community is that relational database are not suitable for building IR systems. Anyhow when looking at posgres the row overhead is 23-bytes (See "How much database disk space is required to store data from a typical text file?" in the [Postgres FAQ][1]). This is prohibitively large (non-scaling) for my work. 

Incidentally my data set is 17 gigs of text requiring 4-5 tables depending on how the problem is sliced. I remember trying out a schema in sqlite and the db file breaking 100 gigs

I'm quite interested in knowing what the per row overheads are for Sql Server / MySql / sqlite / berkeley db (all of its access methods) / kyoto , tokyo db and firebird. Any one person won't be able to answer the question I guess, unless someone has been as curious as me to look into the matter.

  [1]: http://wiki.postgresql.org/wiki/FAQ
Database implementation details - per row header overhead
===

The problem that prompts this question is related to creating ginormous inverted indices similar to those used when building IR systems. The general mantra from the IR community is that relational database are not suitable for building IR systems. Anyhow when looking at posgres the row overhead is 23-bytes (See "How much database disk space is required to store data from a typical text file?" in the [Postgres FAQ][1]). This is prohibitively large (non-scaling) for my work. 

Incidentally my data set is 17 gigs of text requiring 4-5 tables depending on how the problem is sliced. I remember trying out a schema in sqlite and the db file breaking 100 gigs

I'm quite interested in knowing what the per row overheads are for Sql Server / MySql / sqlite / berkeley db (all of its access methods) / kyoto , tokyo db and firebird. Any one person won't be able to answer the question I guess, unless someone has been as curious as me to look into the matter.

  [1]: http://wiki.postgresql.org/wiki/FAQ