Hi, Is batching not working in Snowflake. I am trying to batch snowflake records but it not working, I have marked the batch size as 150000 but I see the records getting written at 3k records only. My API will throw 3-100 records per call, I am not sure if this is the reason This is my config, I tried with just batch_size_rows but it didn't work.

batch_size_rows: 150000
              batch_config:
                batch_size: 150000
                encoding:
                  format: jsonl
                  compression: gzip
                storage:
                  root: ""

Does your tap have parent-child streams?

I am running it in single stream it’s same result

What variant and version of target-snowflake is this?

Meltanolabs

Hmm that's not a valid version of the meltanolabs variant afaict: https://github.com/MeltanoLabs/target-snowflake/tags

I tried running parallel process with parent child child call being done on 300 parent keys in 20 processes , this is also same

Gotcha

this is how the target is configured

- name: target-snowflake
            config:
              database: ${TARGET_SNOWFLAKE_DATABASE}
              account: ${TARGET_SNOWFLAKE_ACCOUNT}
              add_record_metadata: true
              clean_up_batch_files: ${TARGET_SNOWFLAKE_CLEAN_UP_BATCH_FILES}
              default_target_schema: ${TARGET_SNOWFLAKE_DEFAULT_TARGET_SCHEMA}
              flattening_enabled: ${TARGET_SNOWFLAKE_FLATTENING_ENABLED}
              flattening_max_depth: ${TARGET_SNOWFLAKE_FLATTENING_MAX_DEPTH}
              role: ${TARGET_SNOWFLAKE_ROLE}
              schema: ${TARGET_SNOWFLAKE_DEFAULT_TARGET_SCHEMA}
              user: ${TARGET_SNOWFLAKE_USER}
              warehouse: ${TARGET_SNOWFLAKE_WAREHOUSE}
              batch_size_rows: ${TARGET_SNOWFLAKE_BATCH_SIZE_ROWS}
              batch_config:
                batch_size: ${TARGET_SNOWFLAKE_BATCH_SIZE_ROWS}
                encoding:
                  format: jsonl
                  compression: gzip
                storage:
                  root: ".meltano/files/"

I think that's a red herring 🤔

How can we debug el command

--log-level=debug

is a good start

I have enabled it , I am more of looking at setting breakpoints in runtime

Depends on your IDE I suppose

this jsonl will be singer messages right

correct

perfect I can work with it

The target should support flattening

awesome for some reason I thought I needed to configure it on both ends 😄 , I will take it from here and check why it is no working for me, Thanks for your time have a great day

Np!

Ok I tried debugging and it was working as designed, I went with the clean virtual environment and now I can see it drains at the batch size I defined, I think it's more of something messed up with my old environment