Re: Sphinx indexing problem

From: Mladen Gogala <mladen(dot)gogala(at)vmsinfo(dot)com>
To: Joshua Tolley <eggyknap(at)gmail(dot)com>
Cc: "pgsql-novice(at)postgresql(dot)org" <pgsql-novice(at)postgresql(dot)org>
Subject: Re: Sphinx indexing problem
Date: 2010-05-24 12:02:10
Message-ID: 4BFA6AC2.9020605@vmsinfo.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-novice

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
Joshua Tolley wrote:<br>
<blockquote
cite="mid:AANLkTinR6Fo2M3gAuduV78sqYqPEOw0RX_5-BdbeMdZS(at)mail(dot)gmail(dot)com"
type="cite">
<blockquote type="cite">
<pre wrap="">Is there anything I can do to prevent the API from attempting to put the
entire query result in memory?
</pre>
</blockquote>
<pre wrap=""><!---->
Use a cursor, and fetch chunks of the result set one at a time.

<a class="moz-txt-link-freetext" href="http://www.postgresql.org/docs/current/interactive/sql-declare.html">http://www.postgresql.org/docs/current/interactive/sql-declare.html</a>

--
Joshua Tolley / eggyknap
End Point Corporation

</pre>
</blockquote>
<br>
I would have done so,&nbsp; had I written the application. Unfortunately,
the application was written by somebody else. Putting the entire result
set in memory is a bad idea and Postgres client should be changed,
probably by adding some configuration options, like maximum memory that
the client is allowed to consume and a "swap file". These options
should be configurable per user, not system-wide. As I have said in my
post, I do have a solution for my immediate problem but this slows
things down:<br>
<br>
<blockquote><tt><font color="#3333ff">[root(at)medo etc]# ../bin/indexer
--all</font></tt><br>
<tt><font color="#3333ff">Sphinx 0.9.9-release (r2117)</font></tt><br>
<tt><font color="#3333ff">Copyright (c) 2001-2009, Andrew Aksyonoff</font></tt><br>
<br>
<tt><font color="#3333ff">using config file
'/usr/local/etc/sphinx.conf'...</font></tt><br>
<tt><font color="#3333ff">indexing index 'test1'...</font></tt><br>
<tt><font color="#3333ff">collected 12757000 docs, 21313.6 MB</font></tt><br>
</blockquote>
<br>
It's still running, from slightly before I sent my post.<br>
<pre class="moz-signature" cols="72">--
Mladen Gogala
Sr. Oracle DBA
1500 Broadway
New York, NY 10036
(212) 329-5251
<a class="moz-txt-link-abbreviated" href="http://www.vmsinfo.com">www.vmsinfo.com</a>
</pre>
</body>
</html>

Attachment Content-Type Size
unknown_filename text/html 2.1 KB

In response to

Responses

Browse pgsql-novice by date

  From Date Subject
Next Message Marco Bomben 2010-05-24 12:09:39 timestamp of a row
Previous Message Jayadevan M 2010-05-24 03:40:12 Re: Best starter book