Re: float output precision questions

Tom Lane wrote:

> My recollection is that other people (perhaps Peter?) were the ones
> objecting before. However I'd be somewhat unhappy with the proposal
> as given:
>
>>>Option 'SHORT' would be default and produce the standard sprintf(ascii,...
>>>Option 'LONG' would produce sprintf(ascii, "%25.18g", num).
>
> since this seems to me to hardwire inappropriate assumptions about the
> number of significant digits in a double. (Yes, I know practically
> everyone uses IEEE floats these days. But it's inappropriate for PG
> to assume that.)

I understand this. Unfortunately I only have IEEE compliant stuff.

> AFAICT the real issue here is that binary float representations will
> have a fractional decimal digit of precision beyond what DBL_DIG claims.

In fact, for some numbers I have been testing with, the double
representation can distinguish up to DBL_BIG+2.

> I think I could support adding an option that switches between the
> current output format:
> sprintf(ascii, "%.*g", DBL_DIG, num);
> and:
> sprintf(ascii, "%.*g", DBL_DIG+1, num);

Easy to find numbers with double representation which would need DBL_BIG+2.

> and similarly for float4. Given carefully written float I/O routines,
> reading the latter output should reproduce the originally stored value.

For some numbers it does not. Not true as I said above.

> (And if the I/O routines are not carefully written, you probably lose
> anyway.) I don't see a need for allowing more flexibility than that.

Tests like a==b will fail for some numbers with DBL_BIG+1.
Its like I said before, the guys from matlab (in x86 IEEE float) go to
DBL_BIG+3 to have 'maximum precision'.

>
> Comments?

Yes. I think there are several options.
I checked the sprintf(ascii, "%A", num) output format and all the
numbers that would fail because of DBL_DIG=15 are ok. After insertion on
a table and conversion to double after a query, comparison a==b holds.
AFAICT "%A" is system independent.

I would (if I may) propose the following:

Have two parameters, say DOUBLE_OUTPUT and EXTRA_DIGITS. DOUBLE_OUTPUT
would select from decimal output or normalized output. EXTRA_DIGITS
would add the required extra digits, from 0 (default) to 3, when output
is decimal.

EXTRA_DIGITS:
in the range [0:3]. 0 as defualt.

DOUBLE_OUTPUT:

'DECIMAL': sprintf(ascii, "%.*g", DBL_DIG+EXTRA_DIGITS, num); (default)
'NORMALIZED': sprintf(ascii, "%A", num);

The same could be done for floats (float4).

This way PG does not assume anything (DOUBLE_OUTPUT as 'NORMALIZED'), it
does not hardwire 'inappropriate' assumptions about the number of
significant digits in a double (default EXTRA_DIGITS=0), and it gives
flexibility (EXTRA_DIGITS!=0) if needed.
I think this is functional and reasonable.

Regards,
Pedro M. Ferreira

>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
> (send "unregister YourEmailAddressHere" to majordomo(at)postgresql(dot)org)
>
>

--
----------------------------------------------------------------------
Pedro Miguel Frazao Fernandes Ferreira
Universidade do Algarve
Faculdade de Ciencias e Tecnologia
Campus de Gambelas
8000-117 Faro
Portugal
Tel./Fax: (+351) 289 800950 / 289 819403
http://w3.ualg.pt/~pfrazao