Hi all, is there anyway we can schedule the ingestion of data using tap-bigquery and loading into snowflake incrementally? The startdate and enddate in the tap-bigquery configuration should be changed on daily basis in order to extract and load the data incrementally. Is there any way around to do this? Thank you. #C01TCRBBJD7 #C01UW1W4D5Y #tapbigquery

Hey @sam_daniel, could you explain your use case more? By default you could simply use incremental replication and define your date column as datetime_key. This way the tap would automatically do that. The start & endtime attributes are more for full syncs (and to provide a starting point for the incremental first one).

Thanks for replying. I need to extract daily sales from a table in BQ and append it into a table in snowflake. So I need to only extract the sales that happened a day before from the table at BQ. Could you share any documentation on how we can create an incremental replication? I couldn't find any. Thank you very much sir

Sure, here is a specific example for bigquery: https://meltano.slack.com/archives/C01TCRBBJD7/p1660826987685849?thread_ts=1660790738.739249&cid=C01TCRBBJD7 and in general, https://docs.meltano.com/getting-started/#choose-how-to-replicate-each-entity and https://docs.meltano.com/guide/integration#replication-methods should be your points of reference. Does that help?

Hey Sam - typically we recommend folks only post in one channel as putting the same message in multiple channels can lead to confusion. This one in #C01TCRBBJD7 is the one that would likely get most traction so I”ll delete the others.

@Sven Balnojan Thank you very much. It working now!!!