Hi < eric simmerman> I m just getting started with Meltano a Meltano #plugins-general

Hi <@U06CBNR7GM9> I'm just getting started with Me...

ricky_renner

03/04/2021, 4:10 PM

Hi @eric_simmerman I'm just getting started with Meltano and was informed that you maintain the tap-spreadsheets-anywhere tap. I'm currently having an issue where records are not being loaded from a CSV to a Postgres DB. Any chance you could help out? Here is my log:

spreadsheets-anywhere-to-postgres-job-log.txt

ricky_renner

03/04/2021, 4:12 PM

Specifically, the fact that it "wrote 0 records" is interesting because it still creates the tables, so it's loading in the CSV, but then ignoring the actual content. I'm sure it's some setting on my end that I need to change

douwe_maan

03/04/2021, 4:12 PM

@ricky_renner Can you share your

tap-spreadsheets-anywhere

configuration from

meltano.yml

ricky_renner

03/04/2021, 4:14 PM

Copy code

- name: tap-spreadsheets-anywhere
    pip_url: git+<https://github.com/ets/tap-spreadsheets-anywhere.git>
    config:
      tables:
      - path: <gs://brook-and-whittle>
        name: invoice
        pattern: data-warehouse/data-extraction/connecticut/invoice.csv
        start_date: '2017-05-01T00:00:00Z'
        delimiter: '|'
        key_properties:
        - number
        format: csv
      - path: <gs://brook-and-whittle>
        name: glaccounts
        pattern: data-warehouse/data-extraction/connecticut/chart-of-accounts.csv
        start_date: '2017-05-01T00:00:00Z'
        delimiter: '|'
        key_properties:
        - id
        format: csv
      - path: <gs://brook-and-whittle>
        name: customer
        pattern: data-warehouse/data-extraction/connecticut/customer.csv
        start_date: '2017-05-01T00:00:00Z'
        delimiter: '|'
        key_properties:
        - number
        format: csv
      - path: <gs://brook-and-whittle>
        name: detailactivity
        pattern: data-warehouse/data-extraction/connecticut/detail-activity.csv
        start_date: '2017-05-01T00:00:00Z'
        delimiter: '|'
        key_properties:
        - id
        format: csv
    schema:
      customer:
        territory:
          type:
          - string
          - 'null'

eric_simmerman

03/04/2021, 4:16 PM

Your targeted files are being filtered.

eric_simmerman

03/04/2021, 4:17 PM

That could happen when start_date is enforced - or more likely it’s the pattern defined for each table

eric_simmerman

03/04/2021, 4:18 PM

I’m guessing based on your config that the patterns you have defined are prefixes for the files in each path?

eric_simmerman

03/04/2021, 4:18 PM

So try throwing a ‘.*’ on the end

eric_simmerman

03/04/2021, 4:18 PM

Copy code

pattern: data-warehouse/data-.*

eric_simmerman

03/04/2021, 4:25 PM

Also - if you can turn on debug - the filtering will be logged which would tell us exactly why the files are found but no records are read from them.

ricky_renner

03/04/2021, 4:27 PM

Thank you sir! yes I was just using the specific prefixes for each file for the patterns, but I'll add a wildcard at the end and see how that does

eric_simmerman

03/04/2021, 4:29 PM

So the patterns are read as regular expressions - that’s likely causing some issues here. Unclear why the patterns that fully specify a path aren’t matching though - debug will tell us more.

ricky_renner

03/04/2021, 4:49 PM

Yep that looked to be the problem - thank you! I have a new issue, but I'm going to spin my wheels a bit before bugging you (I'll also post a separate message in that case)

Open in Slack

Previous Next