Hi, we are looking for a way to apply table level filter when extracting data from MSSQL server using tap-mssql plugin in meltano. We tried below option, but it is not working . Can you help to provide right method to achieve table level filtering during extraction? " extractors: - name: tap-mssql variant: wintersrd pip_url: tap-mssql config: database: host: port: tds_version: '7.3' use_date_datatype: true user: dev password: ** select: - dbo-emp.* filter: empId: =3 metadata: dbo-emp: replication-method: FULL_TABLE (edited)

Can you give this variant a try instead https://hub.meltano.com/extractors/tap-mssql--buzzcutnorman/?

ok, is this to handle filter condition or to enable bulk extraction of huge data?

i use buzzcut norman for mssql extraction, but for bulk you need to do it on the loader side. e.g. if you're using snowflake

plugins:
  loaders:
  - name: target-snowflake
    variant: meltanolabs
    pip_url: meltanolabs-target-snowflake==0.7.0
    config:
      add_record_metadata: false 
      default_target_schema: Raw # ${MELTANO_EXTRACT__LOAD_SCHEMA}
      hard_delete: false
      batch_size_rows: 1000000
      batch_config:
        batch_size: 1000000
        encoding:
          format: jsonl
          compression: gzip
        storage:
          root: file://

Thanks, we are using bigquery as loader.How will loader configuration for bulk load be different from the above snowflake one ?

i dont use bigquery so i dont know, but if you are using meltano SDKs then it should be able to be used like that as is