[QUESTIONS] lo_write cannot > 640Kb? memory leaks?

From: "Park, Chul-Su" <pcs(at)mhlx01(dot)kek(dot)jp>
To: pgsql-hackers(at)postgresql(dot)org
Subject: [QUESTIONS] lo_write cannot > 640Kb? memory leaks?
Date: 1998-05-21 09:09:54
Message-ID: 3563EF62.3D5C14DF@mhlx01.kek.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi all,

Can anybody pinpoint where is wrong on lo_write(and possibly
lo_read)? I tried to lo_import(and lo_write) few MegaBytes of data
with large_object but it was painfully SLOW! So I simplified the
problem, just test "testlo.c" under test/examples directory

with few MegaBytes of file X.dat

and printout debug messages like this with lo_import ()->
importFile() in main()

IT CANNOT LO_WRITE > 640KB !!! WHY?

testlo.c-------------------
...
in importFile()
{
...
// with BUFSIZE -> 1024 or 8*1024 or 64*1024 i.e. 1K, 8K, 64K, ...
/*
* read in from the Unix file and write to the inversion file
*/
while ((nbytes = read(fd, buf, BUFSIZE)) > 0)
{
printf("COUNT [%d] %d Kb\n", ++count, count*BUFSIZE/1024);
tmp = lo_write(conn, lobj_fd, buf, nbytes);
...
if (tmp < nbytes)
{
fprintf(stderr, "error while reading \"%s\"", filename);
}
printf("READ %d , WRITE %d\n", nbytes, tmp);
_________________________

./testlo template1 X.dat out

....
importing file "in" ...
COUNT [1] 0 Kb
READ 65536 , WRITE 65536
COUNT [2] 64 Kb
...
READ 65536 , WRITE 65536
COUNT [9] 512 Kb
error while reading "in"READ 65536 , WRITE -1
COUNT [10] 576 Kb
error while reading "in"READ 65536 , WRITE -1
COUNT [11] 640 Kb
________________________________________________

and "strace ./testlo template1 in out" shows

...
write(3, "F \0", 3) = 3
write(3, "\0\0\3\273\0\0\0\2\0\0\0\4\0\0\0"..., 1024) = 1024
write(3, "\331\5\200\t\317\t\331]\334\331E"..., 64512) = 64512
write(3, "\213\r\30Lv\10I\211\312\1\322\1\312"..., 20) = 20
read(4, "EERROR: no empty local buffer.\n"..., 1024) = 33
write(2, "error while reading \"in\"", 24error while reading "in") = 24
read(5, "\0\0\0\0\1\310\272\300\276\345\t"..., 65536) = 65536
write(1, "COUNT [10] 576 Kb\n", 20COUNT [10] 576 Kb
) = 20
write(3, "F \0", 3) = 3
write(3, "\0\0\3\273\0\0\0\2\0\0\0\4\0\0\0"..., 1024) = 1024
write(3, "\'j\4j\4h\4\273j\10\241\30Lv\10H"..., 64512) = 64512
write(3, "\331\311\331]\370\331E\370\336\311"..., 20) = 20
read(4, "EERROR: cannot read block 62 of"..., 1024) = 45
write(2, "error while reading \"in\"", 24error while reading "in") = 24
read(5, "u5hp\366v\10\350pk\374\377\203\304"..., 65536) = 65536
write(1, "COUNT [11] 640 Kb\n", 20COUNT [11] 640 Kb
...
___________________________________________________________

So I lo_open/lo_lseek(SEEK_END)/lo_write/lo_close() INSIDE of the loop,
it solved the problem, which means MEM LEAKS or buffer size problem?
And just
lobj_fd = lo_open(conn, lobjId, INV_WRITE| INV_READ);
...
/*
* read in from the Unix file and write to the inversion file
*/
while ((nbytes = read(fd, buf, BUFSIZE)) > 0)
{
printf("COUNT [%d] %d Kb\n", ++count, count*BUFSIZE/1024);
o_lseek(conn, lobj_fd, 0, SEEK_END);
tmp = lo_write(conn, lobj_fd, buf, nbytes);
...

Also solved the problem... But I want reliable(lo_read/lo_write) large
object with BIG data.

Please Help me out.

C.S.Park

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message The Hermit Hacker 1998-05-21 11:47:13 Re: [HACKERS] sorting big tables :(
Previous Message Peter Mount 1998-05-21 07:07:46 RE: [HACKERS] Kerberos 5 breakage.