Re: Yet another infrastructure problem

Stefan Kaltenbrunner wrote:
>> * One way around problems like this is to mirror the services.
>> That may involve load balancing, DNS tricks, database replication,
>> and other assorted goodies. It may be difficult, but it's something
>> I'd like to at least start us talking about.
>
> the low hanging fruit in that regard has already been taken (have you
> seen the static part of website being down in the last few years?) -
> most of the other services are much much harder to operate in a
> loadbalanced (or master-master) setup or doing it seems simply overkill.
> Furthermore I don't think that just making services more complex (as in
> redundant) will necessarily result in better availability. Howver I
> aknowledge that we can improve in some areas (like wiki authentication).

I think the most important thing to get a workaround for is our mirror
management. Because now if wwwmaster goes down, nobody can download our
stuff from the website - even if both the webside and 100 ftp servers
are up.

Now, getting that one done shouldn't be too hard. Since the data is
really only one-way (I don't mind if we loose click-thru stats). So we
can either:

1) replicate the mirror database to a secondary jail somewhere, running
the wwwmaster code. Link the downloads to a separate DNS name that maps
to both these machines, and does checking similar to our static machines
to remove them from DNS if they go down.

2) reimplement the mirror management stuff in client-side javascript
somehow, and serve it off the static mirrors. Not entirely sure how to
do this cleanly, or how to fallback if $user has javascript disabled,
but it would have the advantage of not needing another jail.

My vote would be for #1 here, as I think is clear.

The login system would also be good to have distributed, but it's used
by orders of magnitude less number of people. But if we replicate the
database off to the other machine, it should be possible to point the
logins on both machines as well - it's a simple pl/pgsql function that
needs to be called. We'll just need to deal with the "last logged in"
part that won't work then.

//Magnus