I m trying to do a simple load of a csv to postgres but runn Meltano #getting-started

I'm trying to do a simple load of a csv to postgre...

greg_vaslo

04/12/2022, 7:08 PM

I'm trying to do a simple load of a csv to postgres but running into issues. Getting an error it can't load, but the error isn't super helpful. First thing, maybe not the source of the error, but it is saying a schema and table does not exist but I can see it exists in postgres. Is there an issue with the name in YML vs what it is in the database? I ask because of both the quote and quote marks for account. See the error line in the reply:

greg_vaslo

04/12/2022, 7:08 PM

Copy code

2022-04-12T18:54:02.824139Z [info     ] INFO Sync completed            cmd_type=extractor job_id=2022-04-12T185400--tap-csv--target-postgres name=tap-csv run_id=48734332-7456-40eb-8227-9530d26319ee stdio=stderr

2022-04-12T18:54:03.272414Z [info     ] time=2022-04-12 14:54:03 name=target_postgres level=INFO message=Schema 'tap_csv' does not exist. Creating... CREATE SCHEMA IF NOT EXISTS tap_csv cmd_type=loader job_id=2022-04-12T185400--tap-csv--target-postgres name=target-postgres run_id=48734332-7456-40eb-8227-9530d26319ee stdio=stderr

2022-04-12T18:54:03.607205Z [info     ] time=2022-04-12 14:54:03 name=target_postgres level=INFO message=Table '"accounts"' does not exist. Creating... CREATE TABLE IF NOT EXISTS tap_csv."accounts" ("acct_category" character varying, "acct_description" character varying, "acct_formal_name" character varying, "epm_acct_level_1" character varying, "epm_acct_level_2" character varying, "epm_acct_level_3" character varying, "epm_acct_level_4" character varying, "epm_acct_level_5" character varying, "epm_acct_level_6" character varying, "notes" character varying, "report_line" character varying, "sw_pl_line" character varying, "valspar_pl_line" character varying, "acct_number" character varying, PRIMARY KEY ("acct_number")) cmd_type=loader job_id=2022-04-12T185400--tap-csv--target-postgres name=target-postgres run_id=48734332-7456-40eb-8227-9530d26319ee stdio=stderr

greg_vaslo

04/12/2022, 7:09 PM

message has been deleted

taylor

04/12/2022, 7:18 PM

The variant you’re using is deprecated/archived - can you try using this one? https://github.com/MeltanoLabs/tap-csv the format should mostly be the same

greg_vaslo

04/12/2022, 7:19 PM

I will - I tried and had issues and I only found one tutorial on how to do this but it used that older version

thomas_briggs

04/12/2022, 7:28 PM

I've had issues with the PG loader thinking schemas and tables don't exist when they really do because of differences in the case of the stream names. It's generally benign though because it uses CREATE IF NOT EXISTS so it just keeps trying over and over but doesn't actually do anything. I don't see anything in your output that's mixed case but it feels like the same issue...

thomas_briggs

04/12/2022, 7:28 PM

https://github.com/transferwise/pipelinewise-target-postgres/issues/70

greg_vaslo

04/12/2022, 8:51 PM

getting this same error the other day with the tap-csv which is why I switched to the one that worked in the tutorial

greg_vaslo

04/12/2022, 8:51 PM

Copy code

2022-04-12T20:45:04.927844Z [info     ] ELT could not be completed: Cannot start extractor: Catalog discovery failed: command ['/mnt/c/Users/gxv383/Documents/postgresql_tables/ledger/.meltano/extractors/tap-csv/venv/bin/tap-csv', '--config', '/mnt/c/Users/gxv383/Documents/postgresql_tables/ledger/.meltano/run/elt/2022-04-12T204502--tap-csv--target-postgres/7a905866-4309-4cd3-8cea-55e41c8afaeb/tap.8303e06f-fdcb-4555-b0f4-76ad3fa1233b.config.json', '--discover'] returned 1. cmd_type=elt job_id=2022-04-12T204502--tap-csv--target-postgres name=meltano run_id=7a905866-4309-4cd3-8cea-55e41c8afaeb stdio=stderr
2022-04-12T20:45:04.929185Z [info     ] For more detailed log messages re-run the command using 'meltano --log-level=debug ...' CLI flag. cmd_type=elt job_id=2022-04-12T204502--tap-csv--target-postgres name=meltano run_id=7a905866-4309-4cd3-8cea-55e41c8afaeb stdio=stderr
2022-04-12T20:45:04.929899Z [info     ] Note that you can also check the generated log file at '/mnt/c/Users/gxv383/Documents/postgresql_tables/ledger/.meltano/logs/elt/2022-04-12T204502--tap-csv--target-postgres/7a905866-4309-4cd3-8cea-55e41c8afaeb/elt.log'. cmd_type=elt job_id=2022-04-12T204502--tap-csv--target-postgres name=meltano run_id=7a905866-4309-4cd3-8cea-55e41c8afaeb stdio=stderr
2022-04-12T20:45:04.930557Z [info     ] For more information on debugging and logging: <https://docs.meltano.com/reference/command-line-interface#debugging> cmd_type=elt job_id=2022-04-12T204502--tap-csv--target-postgres name=meltano run_id=7a905866-4309-4cd3-8cea-55e41c8afaeb stdio=stderr
2022-04-12T20:45:04.964747Z [debug    ] ELT could not be completed: Cannot start extractor: Catalog discovery failed: command ['/mnt/c/Users/gxv383/Documents/postgresql_tables/ledger/.meltano/extractors/tap-csv/venv/bin/tap-csv', '--config', '/mnt/c/Users/gxv383/Documents/postgresql_tables/ledger/.meltano/run/elt/2022-04-12T204502--tap-csv--target-postgres/7a905866-4309-4cd3-8cea-55e41c8afaeb/tap.8303e06f-fdcb-4555-b0f4-76ad3fa1233b.config.json', '--discover'] returned 1.

taylor

04/12/2022, 8:52 PM

can you share your meltano.yml?

greg_vaslo

04/12/2022, 8:53 PM

Copy code

version: 1
send_anonymous_usage_stats: true
project_id: 774e6b3c-feb6-48f8-b6f4-dcea9ccf3fc7
plugins:
  extractors:
  - name: tap-csv
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-csv.git>
    config:
      files:
      - entity: accounts
        file: extract/accounts.csv
        keys: [acct_number]
  loaders:
  - name: target-postgres
    variant: transferwise
    pip_url: pipelinewise-target-postgres
    config:
      host: 127.0.0.1
      port: 5432
      user: postgres
      dbname: ledger
      schema: public
environments:
- name: dev
- name: test
- name: prod

greg_vaslo

04/12/2022, 8:54 PM

sorry

taylor

04/12/2022, 8:57 PM

hmm, so the first thing I would check is if you can do

meltano invoke tap-csv

and see what it says there. possibly with the

--log-level=debug

option. The other thing would be to quote the

file

value but I’m not 100% sure if that’s necessary

greg_vaslo

04/12/2022, 9:01 PM

similar error

greg_vaslo

04/12/2022, 9:01 PM

Catalog discovery failed: command ['/mnt/c/Users/gxv383/Documents/postgresql_tables/ledger/.meltano/extractors/tap-csv/venv/bin/tap-csv', '--config', '/mnt/c/Users/gxv383/Documents/postgresql_tables/ledger/.meltano/run/tap-csv/tap.1fc3263b-ccba-4d64-80ff-2c50edff45e6.config.json', '--discover'] returned 1

greg_vaslo

04/12/2022, 9:02 PM

let me try the log error

taylor

04/12/2022, 9:05 PM

@pat_nadolny would appreciate your eyes on this as well 🙂

greg_vaslo

04/12/2022, 9:07 PM

lots of cascading python error gibberish, kind of leads to this

greg_vaslo

04/12/2022, 9:07 PM

Copy code

Traceback (most recent call last):
  File "/mnt/c/Users/gxv383/Documents/postgresql_tables/.venv/lib/python3.8/site-packages/meltano/cli/__init__.py", line 58, in main
    raise CliError(str(err)) from err
meltano.cli.utils.CliError: Catalog discovery failed: command ['/mnt/c/Users/gxv383/Documents/postgresql_tables/ledger/.meltano/extractors/tap-csv/venv/bin/tap-csv', '--config', '/mnt/c/Users/gxv383/Documents/postgresql_tables/ledger/.meltano/run/tap-csv/tap.fb013cea-38c5-4b95-b8fd-a72338d21874.config.json', '--discover'] returned 1
Catalog discovery failed: command ['/mnt/c/Users/gxv383/Documents/postgresql_tables/ledger/.meltano/extractors/tap-csv/venv/bin/tap-csv', '--config', '/mnt/c/Users/gxv383/Documents/postgresql_tables/ledger/.meltano/run/tap-csv/tap.fb013cea-38c5-4b95-b8fd-a72338d21874.config.json', '--discover'] returned 1

greg_vaslo

04/12/2022, 9:07 PM

sorry i dont know how to format better

greg_vaslo

04/12/2022, 9:09 PM

to be clear i removed the old tap and then added this one using just

meltano add extractor tap-csv

taylor

04/12/2022, 9:10 PM

you may need to run meltano install again in that case

taylor

04/12/2022, 9:10 PM

i’m going to spin up a local version of this real quick. can you confirm your python and meltano versions?

greg_vaslo

04/12/2022, 9:11 PM

meltano 1.99

greg_vaslo

04/12/2022, 9:11 PM

python 3.8.10

pat_nadolny

04/12/2022, 9:21 PM

@greg_vaslo can you run

meltano invoke tap-csv --version

taylor

04/12/2022, 9:21 PM

I’m getting the same error testing locally

pat_nadolny

04/12/2022, 9:21 PM

I also have this sample project that uses tap-csv and target-postgres if you want to take a look at that https://github.com/pnadolny13/meltano_example_implementations/tree/main/meltano_projects/dbt_orchestration

taylor

04/12/2022, 9:21 PM

@pat_nadolny I get tap-csv v0.0.3 with that

taylor

04/12/2022, 9:22 PM

but I see a v0.0.4 in MeltanoLabs 👀

pat_nadolny

04/12/2022, 9:22 PM

What csv are you using?

pat_nadolny

04/12/2022, 9:22 PM

I'm also on v0.0.3 in my local

taylor

04/12/2022, 9:22 PM

https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/bundle/discovery.yml#L805

taylor

04/12/2022, 9:22 PM

Copy code

acct_number, value
1, 200
2, 300

taylor

04/12/2022, 9:22 PM

called accounts.csv

pat_nadolny

04/12/2022, 9:24 PM

what does

--log-level=debug

say?

taylor

04/12/2022, 9:24 PM

zoom? 🙂

greg_vaslo

04/12/2022, 9:25 PM

I am also using 0.03

taylor

04/12/2022, 9:25 PM

https://meltano.slack.com/archives/CMN8HELB0/p1649798711367829

taylor

04/12/2022, 9:27 PM

@greg_vaslo Pat figured it out. change

file

path

in your meltano.yml

greg_vaslo

04/12/2022, 9:31 PM

oh wow now the invoke works - im really sorry, was using the yml from the deprecated version. Thanks for looking into it!!

taylor

04/12/2022, 9:34 PM

happy to help! I think a couple of issues will actually come from this, so thanks for asking 🙂

taylor

04/12/2022, 9:58 PM

I made https://gitlab.com/meltano/meltano/-/issues/3394 to capture the INFO messages being swallowed

4 Views

Open in Slack

Previous Next