Hello How can i do full refresh with target postgres I need Meltano #troubleshooting

Hello. How can i do full refresh with target-post...

felipe_souto_campelo

07/07/2023, 4:02 AM

Hello. How can i do full refresh with target-postgres? I need to truncate the table before inserting data. Has it a parameter to do that? If not, is there another way to truncate the table? Thanks.

visch

07/07/2023, 12:22 PM

If your Tap supports activate version then run the tap in Full table mode and you're done (with the Meltanolabs version I know this is supported) Otherwise I point folks to dbt, there's some examples out here somewhere I'll have to search slack a bit

visch

07/07/2023, 12:22 PM

https://meltano.slack.com/archives/D042RPN1ZA6/p1674133635891879

felipe_souto_campelo

07/07/2023, 3:01 PM

@visch I already tried with meltanolabs version and it did not works for me. The full table mode only updates the existing data but the other records that are no longer in the source and should be deleted from Postgres remain.

visch

07/07/2023, 7:48 PM

What tap are you using?

felipe_souto_campelo

07/09/2023, 5:12 PM

@visch tap-spreadsheets-anywhere

Andy Carter

07/12/2023, 8:17 AM

@felipe_souto_campelo I have a forked variant of spreadsheets anywhere, and found this old PR: https://github.com/quickbi/tap-spreadsheets-anywhere/pull/2/files I'll try to adapt

Andy Carter

07/12/2023, 8:48 AM

I can already see an issue if you are using multiple csv files to one table/pattern matching via regex, the

modified_since

part will not work correctly, it will assume only files modified recently will get synced, and the new version number applied.

Andy Carter

07/12/2023, 8:49 AM

Basically with full_table mode the 'state' functionality becomes redundant.

Andy Carter

07/12/2023, 9:29 AM

@felipe_souto_campelo try this branch: https://github.com/acarter24/tap-spreadsheets-anywhere/tree/activate_version In my table config I'm setting

full_table_replace: true

and then using

hard_delete:true

for meltanolabs variant of

target-postgres

Seems to be working nicely, but need to do a bit more testing on my end.

felipe_souto_campelo

07/12/2023, 9:57 PM

@Andy Carter Awesome! It worked for me. But, unfortunately, now i need to make this run for target-snowflake. The destination database changed in my project. I tried with target-snowflake but it did not works. Is it possible to make a general modification in tap-spreadsheets-anywhere to do the correct full refresh with any target? Thanks for your help.

Andy Carter

07/13/2023, 6:31 AM

Hi Felipe, the activate version logic needs to be in both the tap and target to work correctly. With this change, the tap now has all the right logic. There would need to be a change to

target-snowflake

to support the

activate_version

message.

Andy Carter

07/13/2023, 7:46 AM

Could you handle the deduplication a stage downstream in dbt or similar? If you're pulling in the full table now with your tap, and have the

_sdc

columns, then do an append from your raw data, and use those

sdc_batched

values to identify the last complete run.

Open in Slack

Previous Next