Meltano

Has anyone tried using Meltano code or part of SDK on databricks environment?

Let me know if anyone knows a way/concept to do it.

Hi, <@U06C5FV6162> - I've used Meltano+Singer a bit with Spark/Databricks. Are you more interested in data loading, or transformation, or other features? If for extract/load, there are maybe three challenges you'll run into:

1. How do you want to load the data? Specifically:
    a. Parquet vs CSV vs Avro/other
    b. Merge-Upsert vs Append-Only
    c. "Native" Spark vs DeltaLake
2. Access permissions on the durable storage layer
    a. Any access issues to the permanent/durable data stores?
3. Metadata/metastore management
    a. How do you want to manage schema changes in the metastore?
These are logistical challenges but not insurmountable. The short answer (predictably) is "it depends".

Hello AJ,

I want to run taps and load data into cloud storage in jsonl/csv format. Then onwards I can use databricks for transformations and DLT's

Hi, <@U06C5FV6162>! Which cloud storage option are you looking for? We recently added batch file support, and we have pending support for Parquet and S3.

My main goal is to run meltano/singer within databricks env. and create files on my mounted cloud location, which is accessible using dbfs API

I'm not aware of the GCS landscape but a quick search for "gcs" on <http://hub.meltano.com|hub.meltano.com> did surface this loader: <https://hub.meltano.com/loaders/target-gcs>

I see <@U06C8NEFG2Z> and <@U06CBLX2X26> are both contributors on that target so I'm cc'ing them here.

In recent development we have moved to S3, how can I clone the tap and use it in databricks notebook. Pip install tap doesnt work btw