Hi I m using tap csv and target bigquery to try and get fami Meltano #getting-started

Hi. I'm using tap-csv and target-bigquery to try a...

ben_thompson

02/01/2022, 7:46 PM

Hi. I'm using tap-csv and target-bigquery to try and get familiar with Meltano & Docker. I'm using the gitflix csv's provided in the docs. I have the id columns defined in arrays. The BigQuery replication method is set to 'append'. Data is successfully written to BigQuery. But currently all rows are appended every time I run the job. Shouldn't the ids be providing some kind of marker from which to start from?

ben_thompson

02/01/2022, 7:47 PM

Copy code

plugins:
  extractors:
  - name: tap-csv
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-csv.git>
    config:
      csv_files_definition: extract/csv_files.json

ben_thompson

02/01/2022, 7:47 PM

Copy code

[
  {
    "entity": "users",
    "path": "extract/GitFlixUsers.csv",
    "keys": ["id"]
  },
  {
    "entity": "episodes",
    "path": "extract/GitFlixEpisodes.csv",
    "keys": ["id"]
  },
  {
    "entity": "streams",
    "path": "extract/GitFlixStreams.csv",
    "keys": ["id"]
  }
]

ben_thompson

02/01/2022, 7:52 PM

Thanks in advance, not getting the brain in gear! Required behaviour is to have no rows appended if files haven't changed, only new rows if they have.

pat_nadolny

02/01/2022, 8:02 PM

@ben_thompson All of your configs look good! Usually if you execute your pipeline with

elt

and

job_id

meltano elt tap-x target-y --job_id=123

then Meltano will manage storing and retrieving your bookmarks between executions but I think the problem is that this tap doesnt support incremental state right now.

pat_nadolny

02/01/2022, 8:03 PM

I actually was the one who wrote the new SDK based MeltanoLabs tap variant that youre using during the last tap-a-thon and I think we had a discussion about challenges with a reliable ways to detect modified files cross platform but we never implemented anything

pat_nadolny

02/01/2022, 8:05 PM

also if you run something like

meltano elt --dump=state tap-csv target-jsonl --job_id=123

with the

--dump=state

argument you can see your pipeline bookmarks

ben_thompson

02/02/2022, 3:17 AM

Hi @pat_nadolny thanks for the replies and for the tap. Good that I'm not missing anything obvious, I used this tap assuming easiest use case for a POC. Long term the files would be on Cloud Storage anyway.

pat_nadolny

02/02/2022, 10:40 PM

Hey @ben_thompson just to follow up on this - I created an issue in that repo and we just merged a change to remove the state capability until we implement it. I know you were just using it for testing but let me know if you have any thoughts!

3 Views

Open in Slack

Previous Next