postgres 8.4.9: running out of memory (malloc fails) when running a transaction that runs a LOT of selects

We yesterday encountered a program that in a degenerate case issued in
a single transaction a huge number of selects (in a single transaction
but each select in a separate call to PGExec) (huge = ~ 400,000).

That transaction would continue to eat memory up until a point where
calls to malloc (in aset.c) would fail and log for example:

,"out of memory","Failed on request of size 11."

Both from that transaction and random others. In additional observation
of interest to us was that while both VIRT and RES was growing VIRT was
always roughly double of RES. Which seems to indicate that whatever
allocations were done not all of the memory allocated was actually
touched (server is a Centos6 box).

So I have two questions:

- Is that expected expected behaviour? The transaction was
in READ_COMMITED mode, and my best guess is that this implies
that some snapshot is taken before each subselect and all
of them are only freed once the transaction is finished

- Any recommendations on the use of overcommit? We had it
disabled on the assumption that with overcommit the OOM
killer might kill a random process and that it is better
to instead have a process that is actually allocating fail
(and on the additional assumption that any memory allocated
by postgres would actually be touched).

This is not a huge production issue for us as we can fix the
program to no longer issue huge numbers of selects. But we
would like to understand.

Thank you very much,

Bene

PS:
The machine has huge amounts of memory and during normal operations
it looks like this:

-bash-4.1$ cat /proc/meminfo
MemTotal: 49413544 kB
MemFree: 1547604 kB
Buffers: 5808 kB
Cached: 43777988 kB
SwapCached: 0 kB
Active: 18794732 kB
Inactive: 27309980 kB
Active(anon): 13786796 kB
Inactive(anon): 1411296 kB
Active(file): 5007936 kB
Inactive(file): 25898684 kB
Unevictable: 71928 kB
Mlocked: 55576 kB
SwapTotal: 2047992 kB
SwapFree: 2047992 kB
Dirty: 12100 kB
Writeback: 79684 kB
AnonPages: 2393372 kB
Mapped: 12887392 kB
Shmem: 12871676 kB
Slab: 1050468 kB
SReclaimable: 190068 kB
SUnreclaim: 860400 kB
KernelStack: 4832 kB
PageTables: 450584 kB
NFS_Unstable: 23312 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 26754764 kB
Committed_AS: 17394312 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 120472 kB
VmallocChunk: 34333956900 kB
HardwareCorrupted: 0 kB
AnonHugePages: 1599488 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 5604 kB
DirectMap2M: 2078720 kB
DirectMap1G: 48234496 kB