Last updated: Fri Oct 15 12:26:50 EDT 2004
+Last updated: Wed Dec 1 16:11:11 EST 2004
Current maintainer: Bruce Momjian (pgman@candle.pha.pa.us)
+ "mailto:pgman@candle.pha.pa.us">pgman@candle.pha.pa.us)
The most recent version of this document can be viewed at http://www.PostgreSQL.org/docs/faqs/FAQ_DEV.html.
This was written by Lamar Owen:
+Download the code and have a look around. See 1.7. -
2001-06-22
+Subscribe to and read the + pgsql-hackers + mailing list (often termed 'hackers'). This is where the major + contributors and core members of the project discuss development.
- What open source development process is used by the PostgreSQL - team? +Read HACKERS for six months (or a full release cycle, whichever - is longer). Really. HACKERS _is_the process. The process is not - well documented (AFAIK -- it may be somewhere that I am not aware - of) -- and it changes continually.
+PostgreSQL is developed mostly in the C programming language. It also + makes use of Yacc and Lex.
+ +The source code is targeted at most of the popular Unix platforms and + the Windows environment (XP, Windows 2000, and up).
+ +Most developers make use of the open source development tool chain. + If you have contributed to open source software before, you will probably + be familiar with these tools. They include: GCC + (http://gcc.gnu.org, GDB + (www.gnu.org/software/gdb/gdb.html), + autoconf (www.gnu.org/software/autoconf/) AND + GNU make (www.gnu.org/software/make/make.html.
- What development environment (OS, system, compilers, etc) is - required to develop code? +Developers using this tool chain on Windows make use of MingW (see + http://www.mingw.org/).
-Developers Corner on the - website has links to this information. The distribution tarball - itself includes all the extra tools and documents that go beyond a - good Unix-like development environment. In general, a modern unix - with a modern gcc, GNU make or equivalent, autoconf (of a - particular version), and good working knowledge of those tools are - required.
+Some developers use compilers from other software vendors with + mixed results.
- What areas need support? +Developers who are regularly rebuilding the source often pass the + --enable-depend flag to configure. The result is that when you + make a modification to a C header file, all files depend upon that file + are also rebuilt.
-The TODO list.
+You've made the first step, by finding and subscribing to - HACKERS. Once you find an area to look at in the TODO, and have - read the documentation on the internals, etc, then you check out a - current CVS,write what you are going to write (keeping your CVS - checkout up to date in the process), and make up a patch (as a - context diff only) and send to the PATCHES list, prefereably.
+ Outstanding features are detailed in the TODO list. This is located in + doc/TODO in the source distribution or at + http://developer.postgresql.org/todo.php. + +You can learn more about these features by consulting the archives, the + SQL standards and the recommend texts (see 1.10).
+ +Send an email to pgsql-hackers with a proposal for what you want to do + (assuming your contribution is not trivial). Working in isolation is not + advisable: others may be working on the same TODO item; you may have + misunderstood the TODO item; your approach may benefit from the review of + others.
+ +Other than documentation in the source tree itself, you can find some + papers/presentations discussing the code at + http://developers.postgresql.org.
-Discussion on the patch typically happens here. If the patch - adds a major feature, it would be a good idea to talk about it - first on the HACKERS list, in order to increase the chances of it - being accepted, as well as toavoid duplication of effort. Note that - experienced developers with a proven track record usually get the - big jobs -- for more than one reason. Also note that PostgreSQL is - highly portable -- nonportable code will likely be dismissed out of - hand.
+Generate the patch in contextual diff format. If you are unfamiliar with + this, you may find the script src/tools/makediff/difforig useful.
+ +Ensure that your patch is generated against the most recent version + of the code. If it is a patch adding new functionality, the most recent + version is cvs HEAD; if it is a bug fix, this will be the most recently + version of the branch which suffers from the bug (for more on branches in + PostgreSQL, see 1.15).
-Once your contributions get accepted, things move from there. - Typically, you would be added as a developer on the list on the - website when one of the other developers recommends it. Membership - on the steering committee is by invitation only, by the other - steering committee members, from what I have gathered watching - froma distance.
+Finally, submit the patch to pgsql-patches@postgresql.org. It will be + reviewed by other contributors to the project and may be either accepted + or sent back for further work.
-I make these statements from having watched the process for over - two years.
- -To see a good example of how one goes about this, search the - archives for the name 'Tom Lane' and see what his first post - consisted of, and where he took things. In particular, note that - this hasn't been _that_ long ago -- and his bugfixing and general - deep knowledge with this codebase is legendary. Take a few days to - read after him. And pay special attention to both the sheer - quantity as well as the painstaking quality of his work. Both are - in high demand.
- -The source code is over 350,000 lines. Many fixes/features - are isolated to one specific area of the code. Others require - knowledge of much of the source. If you are confused about where to - start, ask the hackers list, and they will be glad to assess the - complexity and give pointers on where to start.
- -Another thing to keep in mind is that many fixes and features - can be added with surprisingly little code. I often start by adding - code, then looking at other areas in the code where similar things - are done, and by the time I am finished, the patch is quite small - and compact.
- -When adding code, keep in mind that it should use the existing - facilities in the source, for performance reasons and for - simplicity. Often a review of existing code doing similar things is - helpful.
- -The usual process for source additions is: -
There are several ways to obtain the source tree. Occasional developers can just get the most recent source tree snapshot from - ftp.postgresql.org. For regular developers, you can use CVS. CVS - allows you to download the source tree, then occasionally update - your copy of the source tree with any new changes. Using CVS, you - don't have to download the entire source each time, only the - changed files. Anonymous CVS does not allows developers to update - the remote source tree, though privileged developers can do this. - There is a CVS section (http://developer.postgresql.org/docs/postgres/cvs.html) - in our documentation that describes how to use remote CVS. You can - also use CVSup, which has similarly functionality, and is available - from ftp.postgresql.org.
+ ftp://ftp.postgresql.org. + +Regular developers may want to take advantage of anonymous access to + our source code management system. The source tree is currently hosted in + CVS. For details of how to obtain the source from CVS see + http://developer.postgresql.org/docs/postgres/cvs.html.
+ +Basic system testing
+ +The easiest way to test your code is to ensure that it builds against + the latest verion of the code and that it does not generate compiler + warnings.
-To update the source tree, there are two ways. You can generate - a patch against your current source tree, perhaps using the - make_diff tools mentioned above, and send them to the patches list. - They will be reviewed, and applied in a timely manner. If the patch - is major, and we are in beta testing, the developers may wait for - the final release before applying your patches.
+It is worth advised that you pass --enable-cassert to configure. + This will turn on assertions with in the source which will often show + us bugs because they cause data corruption of segmentation violations. This + generally makes debugging much easier.
+ +Then, perform run time testing via psql.
+ +Regression test suite
+ +The next step is to test your changes against the existing regression + test suite. To do this, issue "make check" in the root directory of the + source tree. If any tests failure, investigate.
+ +If you've deliberately changed existing behaviour, this change may + cause a regression test failure but not any actual regression. If so, + you should also patch the regression test suite.
+ +Other run time testing
+ +Some developers make use of tools such as valgrind + (http://valgrind.kde.org) for + memory testing, gprof (which comes with the GNU binutils suite) and + oprofile (http://oprofile.sourceforge.net/) + for profiling and other related tools.
+ +What about unit testing, static analysis, model checking...?
+ +There have been a number of discussions about other testing frameworks + and some developers are exploring these ideas.
+ +For hard-core developers, Marc(scrappy@postgresql.org) will give - you a Unix shell account on postgresql.org, so you can use CVS to - update the main source tree, or you can ftp your files into your - account, patch, and cvs install the changes directly into the - source tree.
+First, all the files in the src/tools directory are designed for + developers.
-First, use psql to make sure it is working as you expect. - Then run src/test/regress and get the output of - src/test/regress/checkresults with and without your changes, - to see that your patch does not change the regression test in - unexpected ways. This practice has saved me many times. The - regression tests test the code in ways I would never do, and has - caught many bugs in my patches. By finding the problems now, you - save yourself a lot of debugging later when things are broken, and - you can't figure out when it happened.
- -Aside from the User documentation mentioned in the regular FAQ, - there are several development tools available. First, all the files - in the /tools directory are designed for developers.
RELEASE_CHANGES changes we have to make for each release
- SQL_keywords standard SQL'92 keywords
backend description/flowchart of the backend directories
ccsym find standard defines made by your compiler
+ copyright fixes copyright notices
+
entab converts tabs to spaces, used by pgindent
find_static finds functions that could be made static
find_typedef finds typedefs in the source code
find_badmacros finds macros that use braces incorrectly
+ fsync a script to provide information about the cost of cache
+ syncing system calls
make_ctags make vi 'tags' file in each directory
make_diff make *.orig and diffs of source
make_etags make emacs 'etags' files
make_keywords make comparison of our keywords and SQL'92
make_mkid make mkid ID files
- mkldexport create AIX exports file
- pgindent indents C source files
- pgjindent indents Java source files
+ pgcvslog used to generate a list of changes for each release
pginclude scripts for adding/removing include files
- unused_oids in pgsql/src/include/catalog
-
- Let me note some of these. If you point your browser at the
- file:/usr/local/src/pgsql/src/tools/backend/index.html
- directory, you will see few paragraphs describing the data flow,
- the backend components in a flow chart, and a description of the
- shared memory area. You can click on any flowchart box to see a
- description. If you then click on the directory name, you will be
- taken to the source directory, to browse the actual source code
+ pgindent indents source files
+ pgtest a semi-automated build system
+ thread a thread testing script
+
+
+ In src/include/catalog:
+ ++ + unused_oids a script which generates unused OIDs for use in system + catalogs + duplicate_oids finds duplicate OIDs in system catalog definitions ++ + If you point your browser at the tools/backend/index.html file, + you will see few paragraphs describing the + data flow, the backend components in a flow chart, and a description of the + shared memory area. You can click on any flowchart box to see a description. + If you then click on the directory name, you will be taken to the source + directory, to browse the actual source code behind it. We also have several README files in some source directories to describe the function of the module. The browser will display these when you enter the directory also. The @@ -250,26 +253,22 @@ and then back out twice to return to the original function. Most editors support this via tags or etags files. -
Third, you need to get id-utils from:
-- ftp://alpha.gnu.org/gnu/id-utils-3.2d.tar.gz - ftp://tug.org/gnu/id-utils-3.2d.tar.gz - ftp://ftp.enst.fr/pub/gnu/gnits/id-utils-3.2d.tar.gz -- By running tools/make_mkid, an archive of source symbols can - be created that can be rapidly queried like grep or edited. - Others prefer glimpse. +
Third, you need to get id-utils from ftp://ftp.gnu.org/gnu/id-utils/
+By running tools/make_mkid, an archive of source symbols can + be created that can be rapidly queried.
-make_diff has tools to create patch diff files that can +
Some developers make use of cscope, which can be found at + http://cscope.sf.net/. Others use + glimpse, which can be found at + http://webglimpse.net/. + +
tools/make_diff has tools to create patch diff files that can be applied to the distribution. This produces context diffs, which is our preferred format.
Our standard format is to indent each code level with one tab,
where each tab is four spaces. You will need to set your editor to
- display tabs as four spaces:
+ display tabs as four spaces:
vi in ~/.exrc:
@@ -313,11 +309,11 @@
* End:
*/
- pgindent is run on all source files just before each beta test period. It auto-formats all source files to make them @@ -334,7 +334,7 @@ There is also a script called unused_oids in pgsql/src/include/catalog that shows the unused oids.
-I have four good books, An Introduction to Database Systems, by C.J. Date, Addison, Wesley, A Guide to the SQL @@ -344,10 +344,9 @@ Kaufmann
There is also a database performance site, with a handbook - on-line written by Jim Gray at http://www.benchmarkresources.com.
+ on-line written by Jim Gray at http://www.benchmarkresources.com.. -The files configure and configure.in are part of the GNU autoconf package. Configure allows us to test for @@ -371,7 +370,7 @@ all files derived by configure are removed, so you see only the file contained in the source distribution.
-There are a variety of places that need to be modified to add a new port. First, start in the src/template directory. Add an @@ -390,7 +390,7 @@ handling. There is a backend/port directory if you need special files for your OS.
-This was written by Lamar Owen:
@@ -530,7 +530,7 @@Of course, there are many projects that DO include all the files necessary to build RPMs from their Official Tarball (TM).
-This was written by Tom Lane:
@@ -600,7 +600,7 @@ dot-release or two, so that we won't have to double-patch the first wave of fixes. -There are three versions of the SQL standard: SQL-92, SQL:1999, @@ -633,7 +633,21 @@ href="http://dbs.uni-leipzig.de/en/lokal/standards.pdf"> http://dbs.uni-leipzig.de/en/lokal/standards.pdf (paper) - + +
Many technical questions held by those new to the code have been + answered on the pgsql-hackers mailing list - the archives of which can be + found at http://archives.postgresql.org/pgsql-hackers/.
+ +If you cannot find discussion or your particular question, feel free to + put it to the list.
+ +Major contributors also answer technical questions, including questions + about development of new features, on IRC at irc.freenode.net in the + #postgresql channel.
+ +-
- lfirst(i)
+- lfirst(i), lfirst_int(i), lfirst_oid(i)
-- return the data at list element i.
+- return the data (a point, inteter and OID respectively) at list + element i.
- lnext(i)
@@ -764,7 +765,9 @@ snippet that loops through a List containing Var *'s and processes each one:-List *i, *list; ++ List *list; + ListCell *i; foreach(i, list) { @@ -829,10 +831,11 @@The structures passing around from the parser, rewrite, optimizer, and executor require quite a bit of support. Most structures have support routines in src/backend/nodes used - to create, copy, read, and output those structures. Make sure you + to create, copy, read, and output those structures (in particular, + the files copyfuncs.c and equalfuncs.c. Make sure you add support for your new field to these files. Find any other places the structure may need code for your new field. mkid - is helpful with this (see above).
+ is helpful with this (see 1.9).2.5) Why do we use palloc() and pfree() to allocate memory?