"Shared strings"-style table

From: Seamus Abshere <seamus(at)abshere(dot)net>
To: pgsql-general(at)postgresql(dot)org
Subject: "Shared strings"-style table
Date: 2017-10-13 15:49:21
Message-ID: 1507909761.179283.1137828392.30EBE584@webmail.messagingengine.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

hey,

In the spreadsheet world, there is this concept of "shared strings," a
simple way of compressing spreadsheets when the data is duplicated in
many cells.

In my database, I have a table with >200 million rows and >300 columns
(all the households in the United States). For clarity of development
and debugging, I have not made any effort to normalize its contents, so
millions of rows have, for example, "SINGLE FAMILY RESIDENCE /
TOWNHOUSE" (yes, that whole string!) instead of some code representing
it.

Theoretically / blue sky, could there be a table or column type that
transparently handles "shared strings" like this, reducing size on disk
at the cost of lookup overhead for all queries?

(I guess maybe it's like TOAST, but content-hashed and de-duped and not
only for large objects?)

Thanks,
Seamus

--
Seamus Abshere, SCEA
https://www.faraday.io
https://github.com/seamusabshere
https://linkedin.com/in/seamusabshere

Responses

Browse pgsql-general by date

  From Date Subject
Next Message Rob Sargent 2017-10-13 16:09:05 Re: "Shared strings"-style table
Previous Message Daniel Verite 2017-10-13 13:36:10 Re: Restore LargeObjects on different server