Re: [HACKERS] [QUESTIONS] lo_write cannot > 640Kb? memory leaks?

> Hi all,
>
> Can anybody pinpoint where is wrong on lo_write(and possibly
> lo_read)? I tried to lo_import(and lo_write) few MegaBytes of data
> with large_object but it was painfully SLOW! So I simplified the
> problem, just test "testlo.c" under test/examples directory
>
> with few MegaBytes of file X.dat
>
> and printout debug messages like this with lo_import ()->
> importFile() in main()
>
> IT CANNOT LO_WRITE > 640KB !!! WHY?
>
> testlo.c-------------------
> ...
> in importFile()
> {
> ...
> // with BUFSIZE -> 1024 or 8*1024 or 64*1024 i.e. 1K, 8K, 64K, ...
> /*
> * read in from the Unix file and write to the inversion file
> */
> while ((nbytes = read(fd, buf, BUFSIZE)) > 0)
> {
> printf("COUNT [%d] %d Kb\n", ++count, count*BUFSIZE/1024);
> tmp = lo_write(conn, lobj_fd, buf, nbytes);
> ...
> if (tmp < nbytes)
> {
> fprintf(stderr, "error while reading \"%s\"", filename);
> }
> printf("READ %d , WRITE %d\n", nbytes, tmp);
> _________________________
>
> ./testlo template1 X.dat out
>
> ....
> importing file "in" ...
> COUNT [1] 0 Kb
> READ 65536 , WRITE 65536
> COUNT [2] 64 Kb
> ...
> READ 65536 , WRITE 65536
> COUNT [9] 512 Kb
> error while reading "in"READ 65536 , WRITE -1
> COUNT [10] 576 Kb
> error while reading "in"READ 65536 , WRITE -1
> COUNT [11] 640 Kb
> ________________________________________________
>
> and "strace ./testlo template1 in out" shows
>
> ...
> write(3, "F \0", 3) = 3
> write(3, "\0\0\3\273\0\0\0\2\0\0\0\4\0\0\0"..., 1024) = 1024
> write(3, "\331\5\200\t\317\t\331]\334\331E"..., 64512) = 64512
> write(3, "\213\r\30Lv\10I\211\312\1\322\1\312"..., 20) = 20
> read(4, "EERROR: no empty local buffer.\n"..., 1024) = 33
> write(2, "error while reading \"in\"", 24error while reading "in") = 24
> read(5, "\0\0\0\0\1\310\272\300\276\345\t"..., 65536) = 65536
> write(1, "COUNT [10] 576 Kb\n", 20COUNT [10] 576 Kb
> ) = 20
> write(3, "F \0", 3) = 3
> write(3, "\0\0\3\273\0\0\0\2\0\0\0\4\0\0\0"..., 1024) = 1024
> write(3, "\'j\4j\4h\4\273j\10\241\30Lv\10H"..., 64512) = 64512
> write(3, "\331\311\331]\370\331E\370\336\311"..., 20) = 20
> read(4, "EERROR: cannot read block 62 of"..., 1024) = 45
> write(2, "error while reading \"in\"", 24error while reading "in") = 24
> read(5, "u5hp\366v\10\350pk\374\377\203\304"..., 65536) = 65536
> write(1, "COUNT [11] 640 Kb\n", 20COUNT [11] 640 Kb
> ...
> ___________________________________________________________
>
> So I lo_open/lo_lseek(SEEK_END)/lo_write/lo_close() INSIDE of the loop,
> it solved the problem, which means MEM LEAKS or buffer size problem?
> And just
> lobj_fd = lo_open(conn, lobjId, INV_WRITE| INV_READ);
> ...
> /*
> * read in from the Unix file and write to the inversion file
> */
> while ((nbytes = read(fd, buf, BUFSIZE)) > 0)
> {
> printf("COUNT [%d] %d Kb\n", ++count, count*BUFSIZE/1024);
> o_lseek(conn, lobj_fd, 0, SEEK_END);
> tmp = lo_write(conn, lobj_fd, buf, nbytes);
> ...
>
> Also solved the problem... But I want reliable(lo_read/lo_write) large
> object with BIG data.
>
> Please Help me out.
>
> C.S.Park

Have you tried increasing the number of postgres buffer cache buffers? I am
speculating that the lo_write() is using these buffers but perhaps not
flushing them. As the default configuration has way too few buffers anyway
this might be the problem. Could you try increasing the buffers to say 1024 or
so and rerun your test and post the results?

Thanks,
-dg

David Gould dg(at)illustra(dot)com 510.628.3783 or 510.305.9468
Informix Software (No, really) 300 Lakeside Drive Oakland, CA 94612
"Of course, someone who knows more about this will correct me if I'm wrong,
and someone who knows less will correct me if I'm right."
--David Palmer (palmer(at)tybalt(dot)caltech(dot)edu)