I m trying to connect a locally hosted CSV file `testdata cs Meltano #getting-started

I'm trying to connect a locally hosted CSV file `t...

taeef_najib

09/03/2023, 7:07 PM

I'm trying to connect a locally hosted CSV file

testdata.csv

to a

PostgreSQL

table. This is my melanto.yml code:

Copy code

version: 1
default_environment: dev
project_id: 05b00b22-24df-4588-a90e-a6b8ecb1e2e8
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-csv
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-csv.git>
    config:
      add_metadata_columns: false
      files:
      - entity: test
        path: data/testdata.csv
        keys: [ID]
  loaders:
  - name: target-postgres
    variant: meltanolabs
    pip_url: meltanolabs-target-postgres~=0.0.7
    config:
      user: meltano
      database: postgres
      add_record_metadata: true
      host: localhost
When I run meltano run tap-csv target-postgres I get an error! Any help would be much appreciated.

taeef_najib

09/03/2023, 7:08 PM

This is the error: ```2023-09-03T190356.773901Z [info ] Environment 'dev' is active 2023-09-03T190357.336123Z [info ] 2023-09-04 010357,335 | INFO | tap-csv | Beginning full_table sync of 'test'... cmd_type=elb consumer=False name=tap-csv producer=True stdio=stderr string_id=tap-csv 2023-09-03T190357.336437Z [info ] 2023-09-04 010357,336 | INFO | tap-csv | Tap has custom mapper. Using 1 provided map(s). cmd_type=elb consumer=False name=tap-csv producer=True stdio=stderr string_id=tap-csv 2023-09-03T190357.337109Z [info ] 2023-09-04 010357,337 | INFO | singer_sdk.metrics | METRIC: {"type": "timer", "metric": "sync_duration", "value": 0.0008733272552490234, "tags": {"stream": "test", "context": {}, "status": "succeeded"}} cmd_type=elb consumer=False name=tap-csv producer=True stdio=stderr string_id=tap-csv 2023-09-03T190357.337223Z [info ] 2023-09-04 010357,337 | INFO | singer_sdk.metrics | METRIC: {"type": "counter", "metric": "record_count", "value": 6, "tags": {"stream": "test", "context": {}}} cmd_type=elb consumer=False name=tap-csv producer=True stdio=stderr string_id=tap-csv 2023-09-03T190357.503002Z [info ] 2023-09-04 010357,502 | INFO | target-postgres | Target 'target-postgres' is listening for input from tap. cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres 2023-09-03T190357.503237Z [info ] 2023-09-04 010357,502 | INFO | target-postgres | Initializing 'target-postgres' target sink... cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres 2023-09-03T190357.503327Z [info ] 2023-09-04 010357,503 | INFO | target-postgres | Initializing target sink for stream 'test'... cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres 2023-09-03T190357.556389Z [info ] Traceback (most recent call last): cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres 2023-09-03T190357.556588Z [info ] File "/media/taeef/Projects2/Sidetrek/DataEngineering/meltano/test2/.meltano/loaders/target-postgres/venv/bin/target-postgres", line 8, in <module> cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres 2023-09-03T190357.556702Z [info ] sys.exit(TargetPostgres.cli()) cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres 2023-09-03T190357.556785Z [info ] File "/media/taeef/Projects2/Sidetrek/DataEngineering/meltano/test2/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/click/core.py", line 1157, in call cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres 2023-09-03T190357.556889Z [info ] return self.main(*args, **kwargs) cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres 2023-09-03T190357.557421Z [info ] File "/media/taeef/Projects2/Sidetrek/DataEngineering/meltano/test2/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/click/core.py", line 1078, in main cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres 2023-09-03T190357.557505Z [info ] rv = self.invoke(ctx) cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres 2023-09-03T190357.557581Z [info ] File "/media/taeef/Projects2/Sidetrek/DataEngineering/meltano/test2/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/click/core.py", line 1434, in invoke cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres 2023-09-03T190357.557653Z [info ] return ctx.invoke(self.callback, **ctx.params) cmd_type=elb consumer=…

Andy Carter

09/03/2023, 8:01 PM

Looks like some kind of encoding issue, your first column is appearing as

'\ufeffID'

Andy Carter

09/03/2023, 8:02 PM

Can you try setting the

encoding: utf8

in your file config?

Andy Carter

09/03/2023, 8:03 PM

ufeff

is the byte order mark (BOM) I think

taeef_najib

09/03/2023, 8:43 PM

@Andy Carter Did you mean this? ~

Copy code

extractors:
  - name: tap-csv
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-csv.git>
    config:
      add_metadata_columns: false
      files:
      - entity: test
        path: data/testdata.csv
        keys: [ID]
        encoding: UTF-8
  loaders:
  - name: target-postgres
    variant: meltanolabs
    pip_url: meltanolabs-target-postgres~=0.0.7
    config:
      user: meltano
      database: postgres
      add_record_metadata: true
      host: localhost

I tried. Same error!

taeef_najib

09/03/2023, 9:04 PM

@Andy Carter thanks! I tried

utf-8-sig

and it worked!

Open in Slack

Previous Next