Hi. I was exploring different ETL tools for my work. We have a need to extract data from a few sources and drop them into S3. One of the APIs is time sensitive. Does meltano have built in logging if we want to make sure we have not missed any windows?
t
tom_morris
01/12/2022, 4:02 PM
Meltano does have built-in logging, but generally the best way is to have Airflow trigger the task runs, and then Airflow is handling the logging.
v
visch
01/12/2022, 4:56 PM
another way is to use something else to monitor your data, like
dbt test
in this case it'd be a freshness test to be sure the data is updated in time