Hello everyone, before studying in deep how Meltano works, I would like to be sure that Is possible to do what I need with Meltano, I want update my snowflake's tables with the last data, so I need to load (periodically and automatically without having to touch nothing) into snowflake, open data that comes in the form of a csv web link, like this: https://cnecovid.isciii.es/covid19/resources/casos_hosp_uci_def_sexo_edad_provres.csv, has Meltano the functionality to do this? Thanks.

based on your post, I'd say Meltano would be a reasonable choice.. you need to find a tap which can download the CSV (MeltanoHub lists all taps and targets so I would start there) and a target for Snowflake (I have not used Snowflake but imagine many many are.. via the Target). Automation would be provided under the hood by the default orchestration apache airflow

Thanks for the help, I've today reading the meltano doc and list of extractors and single tap, I've found one for csv but I think only works for local csv, so I'm not sure if I can use Meltano to load periodically from a csv that comes like a web link

You have a lot of options: 1. If you bootstrap a

meltano

project and add the airflow orchestrator, you have a full-blown airflow instance and can use python

requests

to get the CSV before

meltano run

2. Alternatively on a posix OS, you can add a cron job that does

wget <https://cnecovid.isciii.es/covid19/resources/casos_hosp_uci_def_sexo_edad_provres.csv> -P /path/to/folder/ && meltano run tap-csv target-snowflake

3. Create a utility python script and add it to meltano as a utility with an executable exposed via poetry/setuptools (even in a subdir of the project), then run

meltano run get-covid-csv tap-csv target-snowflake

The list goes on and on