Hi,

I want to load balance a postgres server on 4 physical machines, say 127.0.0.11-14.  I can set up a pgbouncer on 127.0.0.10 and connection pooling to my four boxes.  However, the traffic from/to clients will go through an extra hop.  Another way to do this, is to send the client an "redirect" message.  When client connect to 127.0.0.10, instead of accepting the connection, it can reply to client telling it to reconnect to one of the server on 127.0.0.11-14.  

I am planning to write/submit a patch to do that.  I wonder if there is similar effort in extending libpq protocol, or, if you have better ideas on how to achieve this.

Thank you,
Feng

 		 	   		  
_________________________________________________________________
The New Busy think 9 to 5 is a cute idea. Combine multiple calendars with Hotmail. 
http://www.windowslive.com/campaign/thenewbusy?tile=multicalendar&ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_5

feng tian wrote:
> Hi,
>
> I want to load balance a postgres server on 4 physical machines, say 
> 127.0.0.11-14.  I can set up a pgbouncer on 127.0.0.10 and connection 
> pooling to my four boxes.  However, the traffic from/to clients will 
> go through an extra hop.  Another way to do this, is to send the 
> client an "redirect" message.  When client connect to 127.0.0.10, 
> instead of accepting the connection, it can reply to client telling it 
> to reconnect to one of the server on 127.0.0.11-14. 
>
> I am planning to write/submit a patch to do that.  I wonder if there 
> is similar effort in extending libpq protocol, or, if you have better 
> ideas on how to achieve this.

how do you plan on maintaining consistency, transactional integrity and 
atomicity of updates across these 4 machines?

On Apr 20, 2010, at 10:03 PM, feng tian wrote:
> Another way to do this, is to send the client an "redirect" message.  When client connect to 127.0.0.10, instead of accepting the connection, it can reply to client telling it to reconnect to one of the server on 127.0.0.11-14.  

ISTM that this would be better handled at a higher-level. That is, given a server (127.0.0.10) that holds 127.0.0.11-14. Connect to that server and query for the correct target host.

feng tian wrote:
> Hi,
> 
> I want to load balance a postgres server on 4 physical machines, say 127.0.0.11-14.  I can set up a pgbouncer on 127.0.0.10 and connection pooling to my four boxes.  However, the traffic from/to clients will go through an extra hop.  Another way to do this, is to send the client an "redirect" message.  When client connect to 127.0.0.10, instead of accepting the connection, it can reply to client telling it to reconnect to one of the server on 127.0.0.11-14.  

One common way to do that is to set up one DNS entry for those 4 IP
addresses.

-- 
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

While these can be handled at higher level, for example, by setting 
up LDAP or as Hekki suggested, tricking DNS, the problem is that 
I don't have control of how the user connect to the server.  They 
may not use LDAP.  Solution like pgbouncer has advantages.  User
just get one ip/port and everything else happens automatically.

Thanks,


> Subject: Re: [HACKERS] libpq connectoin redirect
> From: lists(at)jwp(dot)name
> Date: Wed, 21 Apr 2010 15:52:39 -0700
> CC: pgsql-hackers(at)postgresql(dot)org
> To: ft_uw(at)hotmail(dot)com
> 
> On Apr 20, 2010, at 10:03 PM, feng tian wrote:
> > Another way to do this, is to send the client an "redirect" message.  When client connect to 127.0.0.10, instead of accepting the connection, it can reply to client telling it to reconnect to one of the server on 127.0.0.11-14.  
> 
> ISTM that this would be better handled at a higher-level. That is, given a server (127.0.0.10) that holds 127.0.0.11-14. Connect to that server and query for the correct target host.
 		 	   		  
_________________________________________________________________
Hotmail is redefining busy with tools for the New Busy. Get more from your inbox.
http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_2

Hi, John,

The change will be on the libpq client side.  I am not saying this is a general
solution for the distributed transaction/scale out.  However, in many cases, it
is very useful.  For example, in my case, I have about 100 departments each has 
it own database.  The balance machine can just redirect to the right box according
to database/user.  The 4 boxes I have may not even get domain name or static IP.
Another scenario, if I have some kind of replication set up, I can send transaction
processing to master and analytic reporting query to slaves. 

Thanks,
Feng

feng tian wrote:
Hi,

I want to load balance a postgres server on 4 physical 
machines, say 
127.0.0.11-14.  I can set up a pgbouncer on 127.0.0.10 and 
connection 
pooling to my four boxes.  However, the traffic from/to clients
 will 
go through an extra hop.  Another way to do this, is to send 
the 
client an "redirect" message.  When client connect to 
127.0.0.10, 
instead of accepting the connection, it can reply to client 
telling it 
to reconnect to one of the server on 127.0.0.11-14. 

I am planning to write/submit a patch to do that.  I wonder if 
there 
is similar effort in extending libpq protocol, or, if you have 
better 
ideas on how to achieve this.

how do you plan on maintaining consistency, transactional 
integrity and 
atomicity of updates across these 4 machines?
 		 	   		  
_________________________________________________________________
The New Busy is not the too busy. Combine all your e-mail accounts with Hotmail.
http://www.windowslive.com/campaign/thenewbusy?tile=multiaccount&ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_4