Hi, I had to create a fork of

tap-mongod

to do some changes (discussed here) and I think I’m going some where now. I have successfully ‘migrate’ to

target-postgres

! Just q quick, and newbie question, why every time I re-lunch a new elt it creates new tables in the target? I was expecting that only new records would update…

INFO Writing table batch with 109885 rows for `('Order__1616169900637',)`...

I’m not sure if I follow. When you mention it creating new tables, are those temporary/interim load tables by chance? Does the data eventually also land in the “main” set of tables?

@aaronsteers hi, the tables, are created in the pg schema (i.e: order table). Running a second time, I see new tables like ‘order_<timestamp>’.

Can you tell if the rows from the second ‘order_<timestamp>’ table also get replicated into ‘order’. Sounds like there could be a cleanup step missing.

@aaronsteers ok, I’ll try to get that information later. thanks

Some taps - on subsequent imports - do have a behavior to first load a new table with the new dataset, then merge or insert back into the main table - but then afterwards they should clean themselves up by deleting the timestamped table.

in case of an error, they clean the temp tables?

I would have to check the target code to be sure, but in most cases I’ve seen the interim table dropped if all is successful. If the load fails for any reason, that timestamped table might be retained for debugging purposes. Again, depends on the specific target’s approach.

OK, I was doing a quick test, and It seams that it’s reaplying all the records, not sure why , maybe the problem it’s the source tap