We ran into similar problems integrating a third party library into a perl
application with SWIG. There appeared to be some confusion between perl and
SWIG over references/pointers and allocating memory for the referred/pointed to
object. We worked around the problem by making sure the memory allocation
happened in perl (by initializing the reference with a suitably large object).