Re: Create a Foreign Table for PostgreSQL CSV Logs

From: Николай Чадаев <nick-ch58(at)yandex(dot)ru>
To: Олег Самойлов <splarv(at)ya(dot)ru>, Bruce Momjian <bruce(at)momjian(dot)us>
Cc: David G(dot) Johnston <david(dot)g(dot)johnston(at)gmail(dot)com>, Pg Docs <pgsql-docs(at)lists(dot)postgresql(dot)org>
Subject: Re: Create a Foreign Table for PostgreSQL CSV Logs
Date: 2020-08-26 06:24:07
Message-ID: 7945231598336283@mail.yandex.ru
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-docs

<div>Thank You Oleg.</div><div> </div><div>A simple example is extremely important and necessary.</div><div>I will wait for this example.</div><div> </div><div>-- </div><div>Nick Chadaev<div> </div><div>      +7(916)175-3061</div><div>      +7(958)820-7975</div><div> </div><div> </div><div> </div><div> </div><div>24.08.2020, 18:38, "Олег Самойлов" &lt;<a href="mailto:splarv(at)ya(dot)ru">splarv(at)ya(dot)ru</a>&gt;:</div><blockquote><p><br /> </p><blockquote> 24 авг. 2020 г., в 18:07, Bruce Momjian &lt;<a href="mailto:bruce(at)momjian(dot)us">bruce(at)momjian(dot)us</a>&gt; написал(а):<br /> <br /> On Mon, Aug 24, 2020 at 07:40:49AM -0700, David G. Johnston wrote:<blockquote> On Mon, Aug 24, 2020 at 6:17 AM Олег Самойлов &lt;<a href="mailto:splarv(at)ya(dot)ru">splarv(at)ya(dot)ru</a>&gt; wrote:<br /> <br />    There must not be constraints at all. Constraints are needed to check<br />    incoming data to the table. But here table is read-only for database!<br /> <br /> <br /> Please don't top-post.<br /> <br /> <br /> <br />    So all your constraints is totally useless.<blockquote><blockquote> could become:<br /> <br /> connection_from text check(connection_from ~ '^[^:]+:[0-9]+$) -- the</blockquote></blockquote>    host and<blockquote><blockquote> port of the client, colon-separated</blockquote></blockquote> <br /> <br /> I'll agree that the benefit for adding the constraints to a foreign table are<br /> less than for a normal table but it is still not zero. Constraints are also a<br /> form of documentation. And also can be used (at least non-null ones) during<br /> optimization.</blockquote> <br /> I feel constraints are going to lose focus of what we are trying to<br /> show. Do the constraints actually do anything on a foreign table?</blockquote><p><br />I'll add: "on foreign table based on read only text file without indexes, etc". :) But in this case I indeed added some CHECK() constraints for old PostgreSQL and change them for the partition syntax sugar on new PostgreSQL to point on different files according to PostgreSQL default log config, where logs of each day of a week is kept in the different files. For example:<br /><br />ALTER SYSTEM SET log_destination=csvlog;<br />SELECT pg_reload_conf();<br />CREATE EXTENSION file_fdw;<br />CREATE SERVER file_fdw FOREIGN DATA WRAPPER file_fdw;<br />BEGIN;<br />CREATE SCHEMA pglog;<br />CREATE TABLE pglog.pglog (<br />        log_time timestamp(3),<br />        user_name text,<br />        database_name text,<br />        process_id integer,<br />        connection_from text,<br />        session_id text,<br />        session_line_num bigint,<br />        command_tag text,<br />        session_start_time timestamp with time zone,<br />        virtual_transaction_id text,<br />        transaction_id bigint,<br />        error_severity text,<br />        sql_state_code text,<br />        message text,<br />        detail text,<br />        hint text,<br />        internal_query text,<br />        internal_query_pos integer,<br />        context text,<br />        query text,<br />        query_pos integer,<br />        location text,<br />        application_name text<br />) PARTITION BY LIST (date_part('isodow', log_time));<br />CREATE FOREIGN TABLE pglog.Mon<br />        PARTITION OF pglog.pglog FOR VALUES IN (1)<br />        SERVER file_fdw<br />        OPTIONS ( filename '/var/lib/pgsql/12/data/log/postgresql-Mon.csv', format 'csv' );<br />CREATE FOREIGN TABLE pglog.Tue<br />        PARTITION OF pglog.pglog FOR VALUES IN (2)<br />        SERVER file_fdw<br />        OPTIONS ( filename '/var/lib/pgsql/12/data/log/postgresql-Tue.csv', format 'csv' );<br />CREATE FOREIGN TABLE pglog.Wed<br />        PARTITION OF pglog.pglog FOR VALUES IN (3)<br />        SERVER file_fdw<br />        OPTIONS ( filename '/var/lib/pgsql/12/data/log/postgresql-Wed.csv', format 'csv' );<br />CREATE FOREIGN TABLE pglog.Thu<br />        PARTITION OF pglog.pglog FOR VALUES IN (4)<br />        SERVER file_fdw<br />        OPTIONS ( filename '/var/lib/pgsql/12/data/log/postgresql-Thu.csv', format 'csv' );<br />CREATE FOREIGN TABLE pglog.Fri<br />        PARTITION OF pglog.pglog FOR VALUES IN (5)<br />        SERVER file_fdw<br />        OPTIONS ( filename '/var/lib/pgsql/12/data/log/postgresql-Fri.csv', format 'csv' );<br />CREATE FOREIGN TABLE pglog.Sat<br />        PARTITION OF pglog.pglog FOR VALUES IN (6)<br />        SERVER file_fdw<br />        OPTIONS ( filename '/var/lib/pgsql/12/data/log/postgresql-Sat.csv', format 'csv' );<br />CREATE FOREIGN TABLE pglog.Sun<br />        PARTITION OF pglog.pglog FOR VALUES IN (7)<br />        SERVER file_fdw<br />        OPTIONS ( filename '/var/lib/pgsql/12/data/log/postgresql-Sun.csv', format 'csv' );<br />COMMIT;<br /><br />But there is another point of view. The same table schema used now not only for file_fdw access to the log, but also in example where this is an ordinary table. In this case may be some other optimisation: indexes, etc.<br /><br />But do we really need in the simple example such detailed and specific code? May be better give as simple as possible example as example, which everyone will can adapt for his own needs.</p></blockquote></div>

Attachment Content-Type Size
unknown_filename text/html 5.5 KB

In response to

Responses

Browse pgsql-docs by date

  From Date Subject
Next Message Michael Paquier 2020-08-26 06:56:18 Re: Document "59.2. Built-in Operator Classes" have a clerical error?
Previous Message Daniel Gustafsson 2020-08-25 22:22:25 Re: ALTER SYSTEM between upgrades