core dump on select

AIX pg version 7.4

Select * from document2 core dump. Did a few more experiments with select *
from document2 limit...

I limit to 500000 it works, 600000 it exits but says "calloc: There is not
enough memory available now."

I did a 573600 and it worked, then I did a 573700 and it core dumped. I
went back in and did the search again with 573700 and it then worked.

AIX:

core file size (blocks) 1048575

data seg size (kbytes) unlimited

file size (blocks) unlimited

max memory size (kbytes) 32768

open files 2000

pipe size (512 bytes) 64

stack size (kbytes) 2097151

cpu time (seconds) unlimited

max user processes 1000

virtual memory (kbytes) unlimited

Any thoughts on this? And how to generate the trace?

Thanks.

john