Re: How to achieve sustained disk performance of 1.25 GB write for 5 mins

On 11/17/10 22:11, Eric Comeau wrote:

>> *) what kind of data do you expect to be writing out at this speed?
>>
> Large Video files ... our s/w is used to displace FTP.
>>
>> *) how many transactions per second will you expect to have?
>>
> Ideally 1 large file, but it may have to be multiple. We find that if we
> send multiple files it just causes the disk to thrash more so we get
> better throughput by sending one large file.
>
>> *) what is the architecture of the client? how many connections will
>> be open to postgres writing?
>>
> Our s/w can do multiple streams, but I believe we get better performance
> with 1 stream handling one large file, you could have 4 streams with 4
> files in flight, but the disk thrashes more... postgres is not be
> writing the file data, our agent reports back to postgres stats on the
> transfer rate being achieved ... postgres transactions is not the issue.
> The client and server are written in C and use UDP (with our own error
> correction) to achieve high network throughput as opposed to TCP.

I hope you know what you are doing, there is a large list of tricks used
by modern high performance FTP and web servers to get maximum
performance from hardware and the operating system while minimizing CPU
usage - and most of them don't work with UDP.

Before you test with real hardware, try simply sending dummy data or
/dev/null data (i.e. not from disks, not from file systems) and see how
it goes.