I m trying to extract some data from a postgres database wit Meltano #plugins-general

I'm trying to extract some data from a postgres da...

tom_elliff

10/07/2020, 5:03 PM

I'm trying to extract some data from a postgres database with the

pipelinewise-tap-postgresql

but when I send it to my target (in my case

target-jsonl

) I get errors when handling

numeric

columns because the defined schema for the column looks like:

Copy code

"duration": {
            "type": [
              "null",
              "number"
            ],
            "exclusiveMaximum": true,
            "maximum": 100000000000000000000000000000000000000000000000000000000000000,
            "multipleOf": 1e-38,
            "exclusiveMinimum": true,
            "minimum": -100000000000000000000000000000000000000000000000000000000000000
          },

which then causes the json schema validation to throw a precision error in Python's decimal class when it tries to divide the column by 1E-38. Has anyone else encountered this? If so how did they handle it?

douwe_maan

10/07/2020, 5:06 PM

@tom_elliff That was actually fixed recently in

target-jsonl

: https://github.com/andyh1203/target-jsonl/commit/95e9fdbf9afcd92fd0ca1d01f92872f4300700e6 Do you have version 0.1.2? You can force it by setting your

pip_url

target-jsonl==0.1.2

and running

meltano install loader target-jsonl

tom_elliff

10/07/2020, 5:08 PM

Copy code

$ .meltano/loaders/target-jsonl/venv/bin/pip freeze
jsonschema==2.6.0
pendulum==1.2.0
python-dateutil==2.8.1
pytz==2020.1
pytzdata==2020.1
simplejson==3.11.1
singer-python==2.1.4
six==1.15.0
target-jsonl==0.1.2
tzlocal==2.1

douwe_maan

10/07/2020, 5:14 PM

@tom_elliff Looks like that fix wasn't sufficient then! @julian_knight also had to make some additional changes in

target-postgres

to address it fully: https://github.com/meltano/target-postgres/pull/6 https://github.com/meltano/target-postgres/pull/7 I suggest filing an issue in https://github.com/andyh1203/target-jsonl, and I'm sure @andy_huynh would gladly take a contribution with a fix as well.

douwe_maan

10/07/2020, 5:15 PM

@tom_elliff It's also likely that a different (more production-grade) target won't have this issue, in case you're just using target-jsonl for testing purposes

tom_elliff

10/08/2020, 8:55 AM

thanks, I'll take a look at it. We use Athena to query all of our data so I need it in a format that Athena can read (not looked into the preview on federated queries yet) and the simplest here is jsonl so that's our default.

tom_elliff

10/08/2020, 1:54 PM

raised https://github.com/andyh1203/target-jsonl/issues/2. I'll see if I get some time for a proper fix there but for now I have a local patch that just sets the precision to 50 for now

Open in Slack

Previous Next