Skip site navigation (1) Skip section navigation (2)

Re: COPY-able csv log outputs

From: Greg Smith <gsmith(at)gregsmith(dot)com>
To: pgsql-patches(at)postgresql(dot)org
Subject: Re: COPY-able csv log outputs
Date: 2007-05-28 19:01:10
Message-ID: Pine.GSO.4.64.0705281426270.6944@westnet.com (view raw or flat)
Thread:
Lists: pgsql-patches
The attached patch fixes all the issues I found in the original version of 
this code and completes the review I wanted to do.  Someone else will need 
to take this from here.  As I already mentioned, I can't comment on the 
quality of the piping implementation used to add this feature other than 
to say it worked for me.

Here is a sample line generated by the current code that illustrates most 
of what I adjusted:

2007-05-28 13:45:14.028 EDT,"postgres","pgbench",465b1519.5f3d,laptop:32783,24381,17,idle,2007-05-28 
13:44:57 EDT,60337,LOG,,"statement: select * from branches;"

In addition to cleanup and bug fixes, notable changes made from the 
original version include:

-Changed connection information to standard host:port syntax

-Added a per process line number to allow a unique index.  The code that 
supports that in the current logs is very local to log_line_prefix. 
Rather than try to share that information like is done for the timestamp, 
it seemed easier to just duplicate those few lines of code for the CSV 
output path.  This could cause the CVS logs to have different values for 
the line number of a statement than the text format ones.  I didn't feel 
that was a problem serious enough to justify the code refactoring that 
would be required to assure the line numbers were identical.

-Added a new documentation section to the logging chapter devoted just to 
the csvlog feature.  It gives a sample table and import syntax.  I also 
gave recommendations on how to configure some related log file parameters 
that can interact badly with this feature.  For example, I noticed that if 
log_rotation_size was set to a value, it could split the CSV lines in two; 
the result was two CVS files you couldn't import because of the partial 
lines in each.  Since the rotation size feature causes other issues anyway 
that make importing more complicated, documenting the issue seemed 
sufficient.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD

Attachment: csvlog-2.patch
Description: text/plain (58.3 KB)

In response to

Responses

pgsql-patches by date

Next:From: Tom LaneDate: 2007-05-28 19:05:57
Subject: Re: Interval input: usec, msec
Previous:From: Tom LaneDate: 2007-05-28 18:57:59
Subject: Re: Seq scans status update

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group