Hey everyone, I’m looking to build a product on t...
# getting-startedj
jorge_machado
10/03/2022, 9:48 AMHey everyone, I’m looking to build a product on top of meltano. But miss some basic things like:
• Ingest gzip file that has jsonlines on it.
• transfer files from s3 to s3 or s3 to gcp
• actually showing a video with the ui
a
aaronsteers
10/03/2022, 5:24 PMHi, @jorge_machado - Can you say more about the third bullet? We don't have video features or support arbitrary rendering in Meltano UI as of now. Do you mean on the hub?
aaronsteers
10/03/2022, 5:27 PMRegarding the first two, it certainly is possible to do file-based operations with Meltano, but most users pull from a source system (SaaS or database) and load to a destination system (most often a database or data warehouse).
Do you have anything upstream from the jsonl source that can be queried directly? And for the file-copy operations (s3-to-s3 or s3-to-gcp), do you only need the files copied or do you also want rich metadata that you'd get by running as an extract-load operation?
j
jorge_machado
10/04/2022, 7:44 AMHi @aaronsteers, thanks for the reply. Would be nice to have a youtube video where someone just show the UI and what is possible. I think meltano is missing a little of marketing in comparation to other tools.
About my transfer usecase. I would like to be able to transfer parquet, Zip, csv and jsonlines file from S3 to AWS Datalake (so S3 to S3 and parse the columns) while this is happening I would like to use dbt as part of the transformations. So something like :
Ingest s3 -> dbt via aws glue -> write to datalake (some S3 bucket) -> Trigger a separated DBT job from a client that uses dbt-glue. At the end the client should be able to read its tables with aws Athena.
This way everything is serveless.