https://linen.dev logo
Join Slack
Powered by
Hello everyone, I'm a data engineer currently tes...
# troubleshooting
j
Hello everyone, I'm a data engineer currently testing Meltano as a candidate for an ELT tool, to transfer data from MongoDB to PostgreSQL. I'm using the default tap-mongodb and target-postgres. Everything works just fine when using the FULL_TABLE import mode. However, when switching to LOG_BASED mode, the first full import is not working ... Although meltano logs this : 
2022-10-18T15:36:27.461996Z [warning  ] No state was found, complete import.
The mongo tap doesn't detect any change streams ... 
2022-10-18T15:36:29.364039Z [info     ] time=2022-10-18 15:36:29 name=tap_mongodb level=INFO message=No change streams after 1000, updating bookmark and exiting... cmd_type=elb consumer=False name=tap-mongodb producer=True stdio=stderr string_id=tap-mongodb
Is this use case supported ? Thank you in advance !
a
@jeremy_k - Do I read correctly that the log-based is basically a no-op on first run? The log-based implementation appears to be using it's first run to simply bookmark the current pointer. Over time, upon re-execution, do you see new rows appearing for any records changed after the initial log-based execution?
Not all taps are great at transitioning between log-based replication and other replication modes. A common workaround would be to run a full replication, then immediately switch to log-based. Or, this may be slightly safer if you want to be sure to not lose any records between your full-table and first log-based: • run a log-based replication, • then run a full table replication with a different 
--state-id-suffix
tracker so that you don't accidentally wipe your main log-based replication bookmark, • then continue running log-based replication from that point We're working on making this smoother for SDK-based taps in the future, but for now, what do you think of this general approach?
j
Yes, on the first run the log-based is a no-op if nothing new happens on the database during the time the cursor is opened by the tap ... The problem is that if no data is written to the target, the tap does not save the state and the pointer is not bookmarked : re-executions of the log-based are no-ops too. If data change happens during the time the cursor is opened, then it is bookmarked and re-executions are working fine. About the approach you proposed : it should be fine as long as data changes happen during the first log-based execution. It is not ideal, but I will try it ! Thank you very much for your help !
Open in Slack
PreviousNext
Powered by