Could you open a heapdump and figure out what is consuming the memory?
It is kind of hard to guess without having a reproducer.
Vladimir