Re: [HACKERS] Tablespaces

jearl(at)bullysports(dot)com wrote:

><tswan(at)idigx(dot)com> writes:
>
>
>
>>>tswan(at)idigx(dot)com wrote:
>>>
>>>
>>>>>"Zeugswetter Andreas SB SD" <ZeugswetterA(at)spardat(dot)at> writes:
>>>>>
>>>>>
>>>>>>>>My feeling is that we need not support tablespaces on OS's without
>>>>>>>>symlinks.
>>>>>>>>
>>>>>>>>
>>>>>>To create symlinked directories on Win2k NTFS see:
>>>>>> http://www.sysinternals.com/ntw2k/source/misc.shtml#junction
>>>>>>I think Win2000 or XP would be a reasonable restriction for Win32 PG
>>>>>>installations that want tablespaces.
>>>>>>
>>>>>>
>>>>>Oh, good --- symlinks for directories are all that we need for this
>>>>>design. I think that settles it then.
>>>>>
>>>>>
>>>>>
>>>>What archival tools are there that would restore this to this back to
>>>>the
>>>>filesystem: tar? zip? What would happen if a symlink were removed or
>>>>pointed to an invalid location while the postmaste was running?
>>>>
>>>>
>>>Well, for backup, just run tar or find on /data with a flag to
>>>follow symlinks, and you are done. Can't get much easier than
>>>that.
>>>
>>>
>>I'm ruferring to NTFS and the win32 platforms. How does tar handle
>>these symlinks on the NTFS filesystem? What about if someone finds
>>that FAT32 is significantly better for the database?
>>
>>
>
>tar doesn't know anything about PostgreSQL system catalogs. If we use
>symlinks for tablespaces then it would be possible to backup downed
>databases with a simple tar command on every platform *I* care about
>(and probably Windows too). Using system catalogs for this stuff
>would simply guarantee that I would have to read the system catalogs
>and then back up each tablespace manually. In short, your idea would
>trade off (maybe) having to backup tablespaces manually on a few
>platforms for the certainty of having to backup tablespaces manually
>on all platforms.
>
>How is that a win?
>
>
>
Apparently, I have failed tremendously in addressing a concern. The
question is does PostgreSQL need to rely on symlinks and will that
dependency introduce problems?

There is an active win32 port underway (see this mailing list). One
proposal was to try to use an OS specific filesystem feature to perform
a symlink on NTFS. Can the special symlink that NTFS allegedly supports
be archived the same way symlinks are archived on Unix? If so, is there
a utility that can do this (zip, tar, etc). The backup operator would
still need to know what directories needed to be archived in addtion to
the pgdata directory. Is this symlink structure a normal/special file
that can be archived by normal means (tar,zip, etc)?

Example:

PGDATA is C:\pgdata
I have a tablespace in Z:\1\ and Z:\2\
There exists an alleged symlink in
C:\pgdata\data\base\tablespaces\schmoo -> Z:\1

Can I archive [ C:\pgdata, Z:\1, Z:\2 ], restore them, and have
postgresql working just as before?

>>It seems a little insane to introduce an OS/filesystem dependency at
>>the onset of a porting effort especially if you hope to be OS
>>agnostic for feature sets. I think someone would be crying foul if
>>a new feature only worked on Linux and not on FreeBSD.
>>
>>
>
>First of all, symlinks are a pretty popular "feature." Even Windows
>supports what would be needed. Second of all, PostgreSQL will still
>run on OSes without symlinks, tablespaces won't be available, but
>PostgreSQL will still run. Since we are all using PostgreSQL without
>tablespaces now, it can hardly be argued that tablespaces are a
>critical feature.
>
>We aren't talking about a "feature that work[s] on Linux on not on
>FreeBSD." We are talking about a feature that works on every OS that
>suports symlinks (which includes even operating systems like Windows
>that PostgreSQL doesn't currently support).
>
>
>

Hello? What was this response from Tom Lane? "My feeling is that we
need not support tablespaces on OS's without symlinks." That seems to
be indicative of a feature set restriction base on platform.

>>Additionally, another developer noted the advantage of a text file
>>is that it would be easy for someone to develop tools to help if it
>>became difficult to edit or parse. Additionally, there could be a
>>change away from a flat file format to an XML format to configure
>>the tablespace area.
>>
>>
>
>The advantage of symlinks is that no tools would have to be written
>and 'ls -l' would show everything you would need to know about where
>your tablespaces actually were.
>
>
>
Where is 'ls -l' on a win32 box? If you will follow the discussion of
symlinks under MinGW you will see that they don't work as commanded.
And, postgresql is supposed to be compiled under MinGW, but not require
it to run.

From Windows 2000, 'ls' is not recognized as an internal or external
command,
operable program or batch file.

>XML files are relatively easy to parse, but they certainly aren't as
>easy as simply letting PostgreSQL follow a symlink. Why reinvent the
>wheel with what would essentially be PostgreSQL's own implementation
>of a symlink?
>
>

Is opening a file recreating a symlink? If you are opening file
descriptors why rely on symlinks. If you know the location either from
the system catalog, a or configuration file, is it any terribly more
complicated? Basically, if a tablespace needed to be renamed, or
moved, or changed, your going to have to do file management anyway.
The symlink saves you just a lookup as to what files go where? If you
kept this small hash in memory, it's not a continuous lookup because you
have the redirection internally. And, it's more portable.

>To go back to your *tar* example, are we going to rewrite tar so that
>it reads PostgreSQL's XML file and *does the right thing*?
>
>

I'm not talking about integrating tar with PostgreSQL or uniting the
universal string theory with how to cook apple pie. Why would think we
would have to rewrite tar?

>
>
>>Another argument against the symlink approach is how they may change
>>in the future. A file location is universal, symlink behavoir may
>>not be. The symlink behavior on different ports may change. To
>>rely on symlinks introduces an unnecessary OS dependency. All that
>>is needed is the file location. This can be derived from a flat
>>file, an XML file, a sysetm catalog.
>>
>>
>
>Yes, and someone can whip out vi and edit a file or fire up psql and
>change the system catalog. Heck, someone could issue a 'mv' or 'rm'
>command on the actual files, for that matter. PostgreSQL can't
>protect itself from careless sysadmins, and symlinks are no more
>inherently dangerous than normal files. They don't just *disappear*.
>
>
>
>>Also, to support some features on some platforms is a real poor
>>prospect. ( This application requires the Linux port of PostgreSQL
>>7.5.1 ) seems like a poor choice for advocating PostgreSQL. The
>>extra effort insures that all ports, current and future, can get the
>>same set of features and nhancements. I like Unix and Linux as
>>much as the next guy, but I have to be real and make the presumption
>>that there are and will be other operating systems and it would be
>>wise to plan a little for that.
>>
>>
>
>The fact of the matter is that PostgreSQL runs better on some
>platforms than others, and it probably always will. Heck, as of
>today, PostgreSQL is officially supported on the Gamecube. Does that
>mean that the PostgreSQL developers should limit themselves to the
>features offered by the Gamecube? What, precisely, is the point of
>developing for the lowest common denominator?
>
>

The other option proposed was to give win32 a subset of features that
would be available to other platforms. In this case, that would be that
the win32 port could support tablespaces. This is strikingly different
than a performance issue. It would be one thing for tablespaces to
perform poorly, it's another for them to fail or not exist altogether.

>Perhaps if you could give us an example of an actual case where some
>actual PostgreSQL users (or potential users) might be affected?
>
>

See the comment from Tom Lane on limiting features. Look at the
potential Win32 market which outnumbers the unix market in number of
computers and developers by a large margin.