Re: Tools to migrate data from Json files to PostgreSQL DB.

From: github kran <githubkran(at)gmail(dot)com>
To: Tony Shelver <tshelver(at)gmail(dot)com>
Cc: pgsql-general <pgsql-general(at)postgresql(dot)org>
Subject: Re: Tools to migrate data from Json files to PostgreSQL DB.
Date: 2019-03-12 01:21:29
Message-ID: CACaZr5S9Run-Ve4LmELcxs_OBrC5bGMNqr32HSC=WiKsftfodA@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

Thank you all for your tips. I will try to see if I can make use of these
useful tips using my java application.

On Fri, Mar 8, 2019 at 1:33 AM Tony Shelver <tshelver(at)gmail(dot)com> wrote:

> You may want to assess how you want to store and access the data in
> Postgres before deciding on an import strategy.
>
> I have a system with a mix of relational and JSON data. The data was
> originally sourced in flat file format.
> I wrote a few Python programs to take the data, then format to JSON, which
> I then submitted to pg functions.
> To make life easier, I submitted it as 1 JSON field to be decomposed by
> Postgres into relational data, and another to be loaded straight into pg
> JSON columns.
>
> The functions then either strip out the data from JSON using the PG JSON
> functions and store it relationally, or plug it straight into a JSON data
> element.
>
> Not terribly difficult to do, especially if you are not navigating complex
> JSON structures in pl/pgsql to strip out the data.
>
> Plenty of python JSON examples out there. Less so for PG :)
>
> On Thu, 7 Mar 2019 at 23:21, github kran <githubkran(at)gmail(dot)com> wrote:
>
>> Hello PostgreSQL Team,
>>
>> Are there are any tools to migrate data present in the json files ? to
>> the postgreSQL database.
>> We have data in flat files about 2 billion records across multiple
>> files.
>>
>> 1) What is the easiest way I can transfer this data to relational
>> database ?.
>> 2) Any tools I can use ?. and any better ways do it ?
>>
>> Appreciate your help.
>>
>>
>> Thanks
>> Kran,
>>
>

In response to

Browse pgsql-general by date

  From Date Subject
Next Message Julie Nishimura 2019-03-12 05:25:44 Re: master-> 2 hot standbys
Previous Message Michael Nolan 2019-03-12 00:48:01 Re: POSTGRES/MYSQL