I'm using

tap-salesforce

and am looking for the config to pass WHERE clauses into the SOQL that Meltano generates. I need to filter to particular values in a particular column in the Account object to make sure specific records aren't extracted. Chat gpt is leading me astray with configs that don't actually exist...does one exist?

You can use the stream map feature to filter out specific records: https://sdk.meltano.com/en/latest/stream_maps.html#filtering-out-records-from-a-stream-using-filter-operation

Thanks, I found the same when doing research but could not figure out how to leverage it in the .yml file to properly filter... @Edgar Ramírez (Arch.dev) Do you have an example of working code using

stream map

that I could leverage to correctly format the same in mine?

Yeah, let me know if https://docs.meltano.com/guide/mappers/#standalone-mapper-plugins is complete enough

Have spent a few hours on this - I can't figure out where in my meltano.yml file to put the stream maps area. Right now it looks like this:

plugins:
  extractors:
  - name: tap-salesforce
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-salesforce.git@v1.9.0>
    config:
      (various configs)
      stream_maps:
        User:
          filter: "record['Email'] == 'email@email.com'"

    select:
     - User.Id
     - User.Email

But the stream_map documentation shows a section called

mappers

below

plugins

, so I'm struggling to figure out where to tell Meltano to filter. When I run the above, it pulls everything from User but never filters, so I figure it's not in the right place

Sorry I just saw this, but for filter you need to use

__filter__

otherwise it will be creating a new property in the stream named "filter" that will contain the result of that expression