Join Slack
Powered by
Hi everyone :wave: I'm trying out meltano and I h...
# getting-started
a
amine_birani
07/21/2023, 1:17 PM
Hi everyone 👋 I'm trying out meltano and I have a simple use case: I have a Google Cloud Storage bucket with csv files and I want to load them to BigQuery. To do so, I use the
https://hub.meltano.com/extractors/tap-spreadsheets-anywhere/
extractor and
https://hub.meltano.com/loaders/target-bigquery/
loader with a job
https://docs.meltano.com/reference/command-line-interface#job
I want meltano to run every hour, check if there is a new file to extract (or updated one) and load it to BigQuery. I stubbled upon an issue where my state get reinitialized when it shouldn't. So it re-process files every 2 runs. From searching, it might come from target BigQuery 🤔 (see related message
https://meltano.slack.com/archives/C01TCRBBJD7/p1689942408136719
) Is this the right approach ? Should I tone down the schedule of the job ? Should I change my extractor or target config ? How should we configure replication ?
amine_birani
07/21/2023, 1:18 PM
Here's a screenshot of my stata database:
Open in Slack
Previous
Next