Re: Invalid field size

Moreno Andreo wrote:

> As you can see I have 2 bytea fields, blob and thumbnail (the one it
> seems it's giving the error), but AFAIK the former is never used, so it
> should be always null.
> Googling around did not help.

In COPY BINARY, NULL is represented as -1 (all bits set)
in the 32-bit length word for the corresponding field.

So if any bit from this word except the bit sign would get flipped
by a hardware error, you'd get the error you mentioned because the
resulting length would come out as negative. From the source code:

if (!CopyGetInt32(cstate, &fld_size))
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("unexpected EOF in COPY data")));
if (fld_size == -1)
{
*isnull = true;
return ReceiveFunctionCall(flinfo, NULL, typioparam, typmod);
}
if (fld_size < 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid field size")));

Despite the auto-correction mechanisms in place in modern drives [1],
the probability of a non-correctable error is not negligible,
so it's plausible that it's what you're experiencing.

If that's the case and only byte is wrong in the whole file, you could
theorically fix it by finding the offset of the offending length and patch
the wrong byte with a 0xff value.

[1]
https://en.wikipedia.org/wiki/Hard_disk_drive#Error_rates_and_handling

Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite