Re: Parsing config files in a directory

On Wed, Oct 28, 2009 at 3:28 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Josh Berkus <josh(at)agliodbs(dot)com> writes:
>> Kevin,
>>> Perhaps the ease of writing something like that with sed or perl has
>>> caused me to underestimate the effort required in C.  I am curious
>>> whether you actually mean that, or said it for rhetorical effect.
>
>> I actually mean that.  It *looks* easy in perl, and in fact *is* easy
>> for *your* postgresql.conf which you control.  But writing a parser for
>> every postgresql.conf which exists in the world, no matter how someone
>> has hacked it up creatively?  No matter how they've handled upgrades?
>
> The issue isn't even with writing a parser.  The conf file is certainly
> machine-parsable; guc-file.l is an existence proof, and the relatively
> slow rate of change of that file indicates that it's been a reasonably
> stable format over time.

Right.

> The issue is that if you want to modify the
> file while preserving comments, commenting out superseded entries,
> putting new entries where the user would expect to find them, etc etc,
> you have a hard AI problem in front of you.

Right. In other words, it's not possible. You can drive yourself
crazy trying to approximate it, but it is a hopeless waste of time.

> This is why Robert keeps
> harping on the default commentary being a problem --- if you removed all
> comments (and didn't care about ordering etc), it would be easier.

Yes - and we even had some consensus that this was a good idea, at one
point. Maybe not complete, precise consensus on every detail, but
certainly enough to have a conversation about it.

> But short of telling people who prefer $EDITOR to get lost, that's not
> going to work.

This is where I get off the train. Preferring $EDITOR is not the same
thing as feeling that we need 500 lines of comments in the default
file. There may be some people who hold both opinions, of course.

> I think the point of the discussions here is that we want to establish
> an alternate config file (or set of config files) in which the
> expectations are different: no promise to preserve any comments, no
> intention to be human-friendly for editing, etc.  In one sense this
> would be the same machine-readable format, since the backend is still
> going to parse it with guc-file.l; but changing the human expectations
> can make the editing problem much simpler.

I don't think this idea is without merit, but I don't think it's a
silver bullet, either. If you can change the human expectations for
some file that gets processed along with postgresql.conf, you can
change the expectations for postgresql.conf itself. In fact, you'll
have to: adding more files is BY DEFINITION going to change the
interpretation of postgresql.conf. It will either be the magic bullet
file that overrides the other file, or visca versa - rather than, as
is the case in a default install today, being THE configuration file.

One of the issues that we need to face is: how many new files? There
seems to be some sentiment that we can just "drop in" new files and
expect things to work. I think that's likely to lead to chaos.
Having TWO files - one for $EDITOR and one for tools - may be
manageable. There will be some user confusion as to which one is in
charge, but there are only two choices, so maybe it's not too bad.
But having a whole directory full of files emitted by different tools
sounds like a disaster, and therefore it seems to me that there is no
getting around the need to have a tool which can merge new settings
into an existing configuration file.

It would be completely logical to break up the configuration file into
subfiles by TOPIC. That would complicate things for tool-writers
because they would need to get each setting into the proper file, and
we currently don't have any infrastructure for that. But that's not
why people want this feature anyway. What they want is to be able to
deposit new settings and have them take effect without parsing a
config file. But they can do that today. Just open postgresql.conf
for append, write a newline in case the file didn't already end with
one, write your settings, and close the file. Your settings win
because they are last. The problem is - now you've left a mess for
someone else to clean up. Overtime duplicates will accumulate. The
order of settings won't be preserved. Comments won't be properly
updated. But writing to another file doesn't actually fix any of
that. Merging settings (either in postgresql.conf or in a separate
persistent.conf) does, at least in part.

...Robert