Hi I m new here I am interested in using Meltano to load dat Meltano #announcements

Hi, I'm new here. I am interested in using Meltano...

aaronsteers

10/22/2019, 10:29 PM

Hi, I'm new here. I am interested in using Meltano to load data from a variety of sources into an S3 or Azure data lake. The only relevant Singer target I see is csv, but nothing about loading that CSV back to the cloud. Is this a supported load pattern? Has anyone tackled something like this?

gtsiolis

10/23/2019, 7:55 AM

Hi @aaronsteers! Haven’t seen any plans for this. Maybe it’s worth opening an issue for this in https://gitlab.com/meltano/target-csv. @yannis_roussos do you think this is a common case and could be added to the

target-csv

options? /cc @micael_bergeron @aaronsteers in the meantime, you can use a Postgres database to load that CSV data, see https://meltano.com/tutorials/csv-with-postgres.html.

micael_bergeron

10/23/2019, 12:32 PM

@aaronsteers take the following with a grain of salt: it is possible to use other extractor/loaders than the one we curated (it is simply a mater of testing them). For instance I think the https://github.com/transferwise/pipelinewise-target-s3-csv could actually be a good target for your use-case

micael_bergeron

10/23/2019, 12:33 PM

You may find useful information about how to wire a custom plugin with Meltano here: https://meltano.com/tutorials/postgres-with-postgres.html

micael_bergeron

10/23/2019, 12:35 PM

Please report back your findings 😄

aaronsteers

11/13/2019, 8:42 PM

Hey, folks. Sorry for the delay. I've been exploring other aspects and working on other projects in the meanwhile. The recommendation of using Pipelinewise

target-s3-csv

looks fairly promising, although I see there's really just been a few commits on that repo so it might (or may not) require a tiny bit of debugging to get running. I'd also be interested in seeing if a flag could be added to

target-csv

but probably will first explore the pipelinewise connector, since that is in theory the path of least resistance. ( https://github.com/transferwise/pipelinewise-target-s3-csv) A third option I've been considering would be to expand support on the

dbt-spark

DW backend, since spark can in theory leverage an S3-backed data store OR could be programmed to dump/load all it's data to S3 during destruction/creation of the DW.

aaronsteers

11/18/2019, 11:15 PM

Reporting back on the above thread: I successfully got the

target-s3-csv

loader up and running. Here is my config if anyone is interested in doing the same.

Copy code

...
  loaders:
  - capabilities:
    - catalog
    - state
    executable: target-s3-csv
    name: target-s3-csv
    namespace: target-s3-csv
    pip_url: git+<https://github.com/transferwise/pipelinewise-target-s3-csv.git>
    settings:
      - name: aws_access_key_id
        kind: password
        env: TARGET_S3_CSV_AWS_ACCESS_KEY_ID
      - name: aws_secret_access_key
        kind: password
        env: TARGET_S3_CSV_AWS_SECRET_ACCESS_KEY
      - name: s3_bucket
        env: TARGET_S3_CSV_S3_BUCKET

To use: (1) paste the above into

meltano.yml

, (2) add env variables in .env, (3) run

meltano install

, (4) test with something like

meltano elt tap-carbon-intensity target-s3-csv

4 Views

Open in Slack

Previous Next