== PostgreSQL Weekly News - January 16 2011 ==

== PostgreSQL Weekly News - January 16 2011 ==

The final commitfest of 9.1 has begun. Start reviewing those patches!

In version 9.1, PostgreSQL will have substantial improvements to
SERIALIZABLE transaction mode. Do you currently use SERIALIZABLE?
The PostgreSQL development team needs feedback:
http://www.postgresql.org/community/

== PostgreSQL Product News ==

pgbouncer 1.4 a lightweight connection pooler for PostgreSQL, released.
http://pgfoundry.org/projects/pgbouncer/

repmgr 1.0.0, a Hot Standby and Streaming Replication management
system, released.
http://projects.2ndquadrant.com/repmgr

== PostgreSQL 9.0 Feature of the Week ==

== PostgreSQL Tip of the Week ==

== PostgreSQL Jobs for January ==

http://archives.postgresql.org/pgsql-jobs/2011-01/threads.php

== PostgreSQL Local ==

Selena Deckelmann will be speaking on PostgreSQL Community and
Development February 7, 2010, 4pm at Oregon State University in
Corvallis, OR.

The Call for Papers for the fourth annual Prague PostgreSQL
Developers' Day is open. It will be held February 10, 2011 at Charles
University in Prague.
http://archives.postgresql.org/pgsql-announce/2010-12/msg00009.php

The PostgreSQLFr Call for Projects is open. Projects must target
PostgreSQL and the French-speaking community. Mail appel-projets-2010
AT postgresql DOT fr.
http://www.postgresql.fr/appel_a_projets_2010:call_for_projects

PGDay is scheduled at this year's Southern California Linux Exposition
(SCALE) held in the LAX Hilton Hotel in the city of Los Angeles,
California, on Friday February 25th, 2011. Post your talk submission
to pgday-submissions AT googlegroups DOT com.

PostgreSQL Conference East 2011: NYC March 22nd - 25th
http://www.postgresqlconference.org

Open Database Camp will be on May 7-9, 2011 in Sardinia, Italy
http://datacharmer.blogspot.com/2011/01/announcing-open-database-camp-sardinia.html

PGCon will be May 19-20, 2011 at the University of Ottawa, preceded by
two days of tutorials on May 17-18. The Call for Papers is open!
http://www.pgcon.org/2011/

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm Pacific time.
Please send English language ones to david(at)fetter(dot)org, German language
to pwn(at)pgug(dot)de, Italian language to pwn(at)itpug(dot)org(dot) Spanish language
to pwn(at)arpug(dot)com(dot)ar(dot)

== Reviews ==

Steve Singer reviewed Gurjeet Singh's patch to add a primary key using
an existing index.

Noah Misch reviewed the snapshot synchronization patch.

Hitoshi Harada reviewed the patch to add a PL/Python validator
function.

Jan Urbanski reveiwed the FDW API patch.

== Applied Patches ==

Magnus Hagander pushed:

- Backend support for streaming base backups. Add BASE_BACKUP command
to walsender, allowing it to stream a base backup to the client (in
tar format). The syntax is still far from ideal, that will be fixed
in the switch to use a proper grammar for walsender. No client
included yet, will come as a separate commit. Magnus Hagander and
Heikki Linnakangas.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=0eb59c4591ecf4f1c69d89e9f043a18e7dce9e47

- Set process title to indicate base backup is running.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=2e36343f82377fbb50834bba6557f8f243fecb34

- Reset walsender ps title in the main loop. When in streaming mode
we can never get out, so it will never be required, but after a base
backup (or other operations) we can get back to the loop, so the
title needs to be cleared.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=b7ebda9d8c6f78b3bb31247531d0ef0e64b32a16

- Typo fix. Josh Kupershmidt
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=1c400d330934eb6d70982af522f2bc0458eef48d

- Revert installation of gram.h in 8.3. To make the buildfarm green
again, since there is no file to copy on msvc, and also given
discussion about the necessity of the file at all...
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=c1bcb1fb618fbec07b04f16042bcf9ffbf294fec

- Add missing function prototype, for consistency.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=47a5f3e9dab68f47ebadc759afb97b900c437c54

- Track walsender state in shared memory and expose in
pg_stat_replication.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=4c8e20f815cbdf043d6d27906fd85ae50c9e4870

- Make sure walsender state is only read while holding the spinlock.
Noted by Robert Haas.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=9eacd427e811a97337de1fdd61a3cb90604981ad

- Exit from base backups when shutdown is requested. When the exit
waits until the whole backup completes, it may take a very long
time. In passing, add back an error check in the main loop so we
detect clients that disconnect much earlier if the backup is large.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=688423d004f4092aed73c73a3281c281d476436d

- Use a lexer and grammar for parsing walsender commands Makes it
easier to parse mainly the BASE_BACKUP command with it's options,
and avoids having to manually deal with quoted identifiers in the
label (previously broken), and makes it easier to add new commands
and options in the future. In passing, refactor the case statement
in the walsender to put each command in it's own function.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=fcd810c69adf11b6ec1cff35359be0dd27662eff

- Enumerate available tablespaces after starting the backup. This
closes a race condition where if a tablespace was created after the
enumeration happened but before the do_pg_start_backup() was called,
the backup would be incomplete. Now that it's done while we are in
backup mode, WAL replay will recreate it during restore. Noted by
Heikki Linnakangas.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=3866ff6149a3b072561e65b3f71f63498e77b6b2

Bruce Momjian pushed:

- A toast relid field are no longer needed in pg_upgrade's rel arrays,
so remove them. Also other renaming.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=0a5f11993195d74f23b63cc5c2d7024c6d27d7e2

- Apply libpq documentation patches submitted by Leslie S Satenstein
and reviewed by Robert Haas.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=a0423ec02df3e311d6d5888170cb25a8c14bc6bf

- More libpq documentation adjustments from Leslie S Satenstein,
reviewed by Robert Haas.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=712dd95370fc6c3a8d20f71b8e195a7af3c50f42

- Apply patch for test_fsync to add tests for O_DIRECT. Adjusted
patch by Josh Berkus.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=677b06ca462ec6fd98da9369a2eae6085c9d7fed

- Improve output display of test_fsync.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=3ab80cfe031b616638eb6956010dcc9cb6426631

- Restructure test_fync to use modular C so there is less duplicate
code and it can be enhanced easier.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=169516ad9395e91d206cbf5bf32c5d2fa34d4111

- Have test_fsync output details that fdatasync is the default
wal_sync_method on Linux.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=001d3664e32c0d156215bbfeccea3272aaf17722

- In test_fsync, warn about options without o_direct that are not used
by Postgres, and cases where o_direct does not work with certain
file systems.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=431605f666cfb223cd615ec8c63cbdea07295550

- Reverse number of stars used for test_fsync details.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=3eebb33dddcfe4ac0719b697c1ebd3694038054e

- Use O_DIRECT in O_SYNC test of different size. Restructure O_DIRECT
error reporting to be more consistent.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=6dc15e3befaa6a3ff72633a2084ad1e1466edcde

- In test_fsync, use #define for printf format of ops/sec.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=e0c274679cb50064a92472c94c7ef5849a156536

Heikki Linnakangas pushed:

- Leave temporary files out of streaming base backups.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=dc1305ce5ffef157410b6e0171d71fa16da4cc9e

- Fix the logic in libpqrcv_receive() to determine if there's any
incoming data that can be read without blocking. It used to conclude
that there isn't, even though there was data in the socket receive
buffer. That lead walreceiver to flush the WAL after every received
chunk, potentially causing big performance issues. Backpatch to
9.0, because the performance impact can be very significant.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=a5a02a744555789ab8390dbf57271e9d07127602

- Treat a WAL sender process that hasn't started streaming yet as a
regular backend, as far as the postmaster shutdown logic is
concerned. That means, fast shutdown will wait for WAL sender
processes to exit before signaling bgwriter to finish. This avoids
race conditions between a base backup stopping or starting, and
bgwriter writing the shutdown checkpoint WAL record. We don't want
e.g the end-of-backup WAL record to be written after the shutdown
checkpoint.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=8f5d65e916796aaee1bf7dd66daf45ca56cd13be

Tom Lane pushed:

- Tweak create_index_paths()'s test for whether to consider a bitmap
scan. Per my note of a couple days ago, create_index_paths would
refuse to consider any path at all for GIN indexes if the
selectivity estimate came out as 1.0; not even if you tried to force
it with enable_seqscan. While this isn't really a bad outcome in
practice, it could be annoying for testing purposes. Adjust the
test for "is this path only useful for sorting" so that it doesn't
fire on paths with nil pathkeys, which will include all GIN paths.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=9d1ac2f5fa4043529dbaff5ebdc73405fa73207b

- Adjust basebackup.c to suppress compiler warnings. Some versions of
gcc complain about "variable `tablespaces' might be clobbered by
`longjmp' or `vfork'" with the original coding. Fix by moving the
PG_TRY block into a separate subroutine.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=e6dce4e439e1d271dad9a95bc4b94147be2fc39a

- Fix PlanRowMark/ExecRowMark structures to handle inheritance
correctly. In an inherited UPDATE/DELETE, each target table has its
own subplan, because it might have a column set different from other
targets. This means that the resjunk columns we add to support
EvalPlanQual might be at different physical column numbers in each
subplan. The EvalPlanQual rewrite I did for 9.0 failed to account
for this, resulting in possible misbehavior or even crashes during
concurrent updates to the same row, as seen in a recent report from
Gordon Shannon. Revise the data structure so that we track resjunk
column numbers separately for each subplan. I also chose to move
responsibility for identifying the physical column numbers back to
executor startup, instead of assuming that numbers derived during
preprocess_targetlist would stay valid throughout subsequent
massaging of the plan. That's a bit slower, so we might want to
consider undoing it someday; but it would complicate the patch
considerably and didn't seem justifiable in a bug fix that has to be
back-patched to 9.0.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=d487afbb813b7ca8803e20974b9e45530a1f4ef1

- Revert incorrect memory-conservation hack in inheritance_planner().
This reverts commit d1001a78ce612a16ea622b558f5fc2b68c45ab4c of
2010-12-05, which was broken as reported by Jeff Davis. The problem
is that the individual planning steps may have side-effects on
substructures of PlannerGlobal, not only the current PlannerInfo
root. Arranging to keep all such side effects in the main planning
context is probably possible, but it would change this from a quick
local hack into a wide-ranging and rather fragile endeavor. Which
it's not worth.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=f0f36045b2e3d037bb7647d84373404fa4ba9588

- Code review for postmaster.pid contents changes. Fix broken test
for pre-existing postmaster, caused by wrong code for appending
lines to the lockfile; don't write a failed listen_address setting
into the lockfile; don't arbitrarily change the location of the data
directory in the lockfile compared to previous releases; provide
more consistent and useful definitions of the socket path and
listen_address entries; avoid assuming that pg_ctl has the same
DEFAULT_PGSOCKET_DIR as the postmaster; assorted code style
improvements.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=52948169bcddf443b76d6ff1806259b153a2ac04

- Add .gitignore to silence git complaints about parser/scanner output
files.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=36750dcef58550c652cfff861f9aad057a391fb9

- Move a couple of declarations to reflect where the routines really
are.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=6ca452ba7fca14dad16425a56ffa1c8a93496b5f

Peter Eisentraut pushed:

- Add some subsection headings.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=b95ea9dd628a93f564e460b8870228755b520220

- Re-add recursive coverage target in src/backend/. This was lost
during the recent recursive make change.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=e3094fd3a8052bb600b287c5dd844b3b0ac2fe11

- Don't run regression tests in SQL_ASCII encoding by default.
Instead, run them in the encoding that the locale selects, which is
more representative of real use. Also document how locale and
encoding for regression test runs can be selected.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=35eb0958be476d58dcc8ba462d57384e74a62d88

- Workaround for recursive make breakage. Changing a file two
directory levels deep under src/backend/ would not cause the
postgres binary to be rebuilt. This change fixes it, but no one
knows why. Branch ------ master Details -------
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=c667cc24e888dc4efe4c2412ad8dd13a190295e3

Andrew Dunstan pushed:

- Unbreak regression tests, apparently broken by commit 4c8e20f
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=b7a0b42641e764a1e4abc39cc4311b5c779f5955

Robert Haas pushed:

- Add support for logging the current role. Stephen Frost, with some
editorialization by me.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=a8a8867912c46a68c9ac14903b3dba2fab8f7097

- Revert patch adding support for logging the current role. This
reverts commit a8a8867912c46a68c9ac14903b3dba2fab8f7097, committed
by me earlier today (2011-01-12). This isn't safe inside an aborted
transaction. Noted by Tom Lane.
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=7a32ff97321408afa0ddfcae1a4a060062956d24

== Rejected Patches (for now) ==

No one was disappointed this week :-)

== Pending Patches ==

Jan Urbanski sent in two more revisions of the patch to add SPI-based
exceptions to PL/PythonU.

Kevin Grittner sent in five more revisions of the SSI patch.

Kevin Grittner sent in another revision of the patch for READ ONLY.

Magnus Hagander sent in two more revisions of the patch to use a
parser for walsender commands.

Jan Urbanski sent in a patch which adds PL/PythonU functions for
quoting strings.

Shigeru HANADA sent in a patch for file_fdw that adds a ResetCopyFrom
function, which is intended to improve performance.

Cedric Villemain sent in a patch atop the walsender patch which fixes
and infelicity for absolute paths.

Jeff Davis sent in another WIP patch for range types.

Heikki Linnakangas sent in a two revisions of a patch to allow
multiple concurrent base backups.

Euler Taveira de Oliveira sent in another revision of the patch to
expand pgbench's maximum run size.

ITAGAKI Takahiro sent in another revision of the MULTISET patch.

Shigeru HANADA sent in a patch to unbreak regression tests, apparently
broken by commit 4c8e20f.

Noah Misch sent in four more revisions of the patch to optimize ALTER
TYPE.

Stephen Frost sent in five revisions of a patch to allow logging the
current role.

Alexey Klyukin sent in a patch to allow conversion between PostgreSQL
arrays and Perl arrays for PL/Perl(U).

Robert Haas sent in some code to make the background writer compact
the request queue before fsyncing.

Shigeru HANADA sent in another flock of patches to implement foreign
data wrappers, part of the SQL/MED system.

Fujii Masao sent in a two revisions of a patch to use latches to
implement failover in pg_ctl.

Fujii Masao sent in a patch to change pg_last_xlog_receive_location
not to move backwards.

Hitoshi Harada sent in a patch to check psql better for an encoding
mismatch.

Jan Urbanski sent in two more revisions of the patch that
auto-generates error codes from header files.

Andreas Karlsson and Tom Lane traded patches to fix the bug in
amproctypes in pg_describe_object().

Shigeru HANADA and ITAGAKI Takahiro traded patches for file data
wrappers, a part of SQL/MED.

Tatsuo Ishii sent in a patch to ensure error codes for "terminating
connection due to conflict with recovery" are sensible.

Greg Smith sent in a patch to spread out checkpoint sync.

Marko (johto) Tiikkaja sent in another revision of the writeable CTE
patch.

Alex Hunsaker sent in two more revisions of a patch to optimize PL/Perl
function argument passing.

Simon Riggs sent in a patch to add foreign keys which are presumed to
hold but not checked against existing data.

Marko (johto) Tiikkaja sent in another revision of the patch to add
transaction-scope advisory locks.

Simon Riggs sent in a WIP patch to add ALTER TABLE ... REPLACE WITH.

Peter Eisentraut sent in a patch to add a client_hostname field to
pg_stat_activity.

Greg Smith sent in a patch to help with logging aborted autovacuums.

Magnus Hagander sent in a patch to help streaming base backups by
ordering.

Jeff Davis sent in another revision of the patch to add range types.

Fujii Masao sent in a patch to ensure that all WAL received is flushed
to disk before walreceiver exits.

Florian Pflug sent in a patch to make backends die sooner after the
postmaster does.

Dimitri Fontaine sent in another revision of the extensions patch.

Alvaro Herrera sent in a patch to make foreign key checks less
intrusive.

Greg Smith sent in two revisions of a patch to auto-size wal_buffers.

Robert Haas sent in a patch to limit hint bit I/O.

Marti Raudsepp sent in a patch to add a tag command "REPLACE X" for
CREATE OR REPLACE statements.

Simon Riggs sent in two revisions of a patch to add recovery control
functions.

Hitoshi Harada sent in a patch to allow specifying ENCODING in COPY.

Peter Eisentraut sent in another revision of the patch to infer
client_encoding from client locale.

Jaime Casanova sent in a patch to add named restore points.

Peter Eisentraut sent in another revision of the per-column collation
patch.

Fujii Masao sent in a patch to help reduce data loss on the standby.

Andrew Dunstan sent in two revisions of a patch to add a textarray
option for file FDWs.

Per review by Noah Misch, Pavel Stehule sent in another revision of
the patch to optimize varlena compression in PL/pgsql.

Magnus Hagander sent in two revisions of a patch to add pg_basebackup
for streaming base backups.

Marko (johto) Tiikkaja sent in another revision of the patch to fix
snapshot taking inconsistencies.

Marko (johto) Tiikkaja sent in another revision of the patch to sho in
EXPLAIN ANALYZE the number of rows a plan qual filtered from a node's
input.

Magnus Hagander sent in a patch to include WAL in the base backup.

Simon Riggs sent in another revision of the patch to add synchronous
replication.

Andreas Karlsson sent in another revision of the patch to add \dL
(languages) to psql.