BUG #14251: COPY CSV: missing data for column that shouldn't be

From: mickael(dot)kerjean(at)gmail(dot)com
To: pgsql-bugs(at)postgresql(dot)org
Subject: BUG #14251: COPY CSV: missing data for column that shouldn't be
Date: 2016-07-15 02:03:34
Message-ID: 20160715020334.1425.78931@wrigleys.postgresql.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-bugs

The following bug has been logged on the website:

Bug reference: 14251
Logged by: Mickael Kerjean
Email address: mickael(dot)kerjean(at)gmail(dot)com
PostgreSQL version: 9.5.0
Operating system: Ubuntu 16.04
Description:

Hello,

First of all, thank you for making such a cool database.

I'm having some bad time inserting data in a csv format coming from:
https://www.maxmind.com/en/free-world-cities-database

The data itself contains cities from all around the world.

- My Schema is:
CREATE TABLE cities(
country_code VARCHAR(2),
city VARCHAR(256),
city_accent VARCHAR(256),
region VARCHAR(128),
population FLOAT DEFAULT NULL,
lat FLOAT,
lng FLOAT
)
- Regarding the encoding at the file:
mickael(at)ubuntu:~$ file -i worldcitiespop.csv
worldcitiespop.csv: text/plain; charset=iso-8859-1
- the query is rather simple:
COPY cities FROM '/home/mickael/worldcitiespop.csv' DELIMITER ',' CSV HEADER
ENCODING 'iso-8859-1';
- The error I'm getting:
ERROR: missing data for column "lng"
CONTEXT: COPY cities, line 19970:
"af,dekh"iykh'ya,Dekh"iykh'ya,13,,34.60345,69.2405"
=> The csv is valid but postgres doesn't really get it right

I tried a few things:
1) convert the file to utf8 using the following command:
iconv -f ISO-8859-1 -t UTF-8 worldcitiespop.csv > worldcitiespop_utf8.csv
2) get rid of the header
3) try to reimport the data:
COPY cities FROM '/home/mickael/worldcitiespop_utf8.csv' DELIMITER ','
CSV;

Same results.
Looking at the error, it might not be impossible postgres is thinking the "
characters are part of 1 string -> ->
https://github.com/postgres/postgres/blob/562e44972490196884452e632a0a6d0db81b2335/src/bin/psql/tab-complete.c#L3621

I tried to import this file using excel and everything was working as
expected

Responses

Browse pgsql-bugs by date

  From Date Subject
Next Message John R Pierce 2016-07-15 06:12:26 Re: BUG #14251: COPY CSV: missing data for column that shouldn't be
Previous Message Alvaro Herrera 2016-07-15 00:55:25 Re: BUG #14250: Error in subquery fails silently and parent query continues to execute