Hi everyone :wave: I'm trying out meltano and I h...
# getting-started
a
Hi everyone 👋 I'm trying out meltano and I have a simple use case: I have a Google Cloud Storage bucket with csv files and I want to load them to BigQuery. To do so, I use the https://hub.meltano.com/extractors/tap-spreadsheets-anywhere/ extractor and https://hub.meltano.com/loaders/target-bigquery/ loader with a job https://docs.meltano.com/reference/command-line-interface#job I want meltano to run every hour, check if there is a new file to extract (or updated one) and load it to BigQuery. I stubbled upon an issue where my state get reinitialized when it shouldn't. So it re-process files every 2 runs. From searching, it might come from target BigQuery 🤔 (see related message https://meltano.slack.com/archives/C01TCRBBJD7/p1689942408136719) Is this the right approach ? Should I tone down the schedule of the job ? Should I change my extractor or target config ? How should we configure replication ?
Here's a screenshot of my stata database: