https://linen.dev logo
Join Slack
Powered by
What part of meltano determines the logic at the t...
# troubleshooting
d
What part of meltano determines the logic at the target for whether or not to wipe the table? The tap (tap-mssql) configures different replication methods (INCREMENTAL, LOG_BASED, FULL_TABLE). How does that get passed on to the target? Trying to figure out why LOG_BASED is clearing all records before inserting.
v
which target are you using? Normally "wipes" happen when an activate version message is sent, but ultimately the target is where the question is here
d
target-snowflake
default variant on both
@visch, any ideas?
v
Ok to be more specific can you share your meltano.yml file, and if possible give an example of how to replicate. There's a bunch of nuance here as there's 3 different snowflake variants here https://hub.meltano.com/loaders/target-snowflake I"m guessing you mean the meltano variant?
d
Yes, the meltano variant
Let me find the original. I temporarily changed it to incremental due to this issue
Copy code
version: 1
default_environment: dev
project_id: 5eb30a8f-fdfa-4025-9543-597f53c7503e
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-mssql
    variant: wintersrd
    pip_url: tap-mssql
    config:
      cursor_array_size: 10000
      database: ...
      host: ...
      use_date_datatype: true
      use_singer_decimal: true
      user: ...
    select:
    - dbo-test.*
    metadata:
      dbo-test:
        table-key-properties: [Id]
        replication-method: LOG_BASED
  loaders:
    - name: target-snowflake
      variant: meltanolabs
      pip_url: meltanolabs-target-snowflake
      config:
        account: ...
        batch_size_rows: 10000
        database: ...
        default_target_schema: ...
        load_method: upsert
        role: ...
        schema: ...
        user: ...
        validate_records: true
        warehouse: ...
v
hmmm I thought it might be activate_version messages, but your hard_delete setting would be false in target-snowflake. I'm not sure, it sounds like a target-snowflake bug to me but we'd need to replicate it some how, so if you could create an output file and steps to replicate your experiance that would be helpful with an issue to target-snowflake
d
Hmm.. that hard_delete setting sounds a little familiar. It may be something that was revised at some point while we were messing with different pocs. I can try doing another round of testing around that. What is the intended purpose of that setting? The docs don't say much about it
v
they handle active version messages
d
It sounds like that should only affect full table sync mode, but we are using upsert. Does the sync mode even matter outside of INCREMENTAL replication mode?
I think that's where the disconnect is for me. The target has some settings on this, but ultimately the desired behavior is configured on the source (from a logical perspective of someone with no knowledge of the internals)
v
yes, and I don't understand why log based currently sends an activate version message from the source that doesn't make any sense to me
seems like a bug to me but I could be missing something
d
@visch Just had some time to test this and it is indeed the hard_delete option causing the issue. I think the correct behavior with that flag using cdc should be that it deletes records instead of just setting the 
_SDC_LSN_DELETED_AT
timestamp and keeping the record. Thanks for helping identify this
➕ 1
27 Views
Open in Slack
PreviousNext
Powered by