Hey everyone, Recently got my tap-mssql set up and connected to the right host and everything and it's at the point where I can pass a --discover flag. When I ran

meltano invoke tap-mssql --discover

I got this huge json output to my CLI with all the different tables/columns I can access along with whether the object is a view/table. This seems expected, but how do I go from here to getting that output into a structure that I can use to filter out the databases/tables/views that I want to replicate over to my warehouse? It's the backend for an ERP system so there's hundreds if not thousands of tables which could be replicated so I obviously don't need to be pulling all of them over. I just need to pull a few tables from one of the seven or so db's within the mssql instance. Should I like get this output into yaml or something then cut it down from there? How does meltano know what the "filtered streams" are each time I run the

meltano run tap-mssql target-snowflake

command for my pipeline runs? Is it all defined in the meltano.yaml or does/can it look at a json or other yaml file to get the proper list? I've only created a custom tap so haven't messed with a pre-built one yet. I'm using the @BuzzCutNorman variant in case that has any sort of implications.

Hey @Ian OLeary, have you tried using select?

Here is another write up on select. https://docs.meltano.com/guide/integration#selecting-entities-and-attributes-for-extraction

So I would run a

meltano select tap-mssql --list --all

to populate all the streams then filter from there?

That will show you the tables that you have available to pull form

Is the granularity of each outputted row on the CLI a table/view

It is at a level of columns. Say you know you want to select all of the columns in

dbo.Users

to replicate. You would say

meltano select tap-mssql dbo-Users '*'

In your meltano.yml you would see this in the config of tap-mssql:

select:
		- 'dbo-Users.*'

Ahh okay gotcha. Do I manage replication keys under the select as well? Or is there somewhere else in the tap config where I'd declare the "dbo-Users" stream and the replication properties?

It’s on a different property, called metadata: https://docs.meltano.com/guide/integration/#setting-metadata You can set the replication method and replication key for each stream 👍

Awesome, thank you!

Just adding to this, If you're looking for a basic example of using both select + metadata, this is from one of my extractors (using buzzcutnorman's variant)

plugins:
  extractors:
  - name: tap-mssql-bookingCustomerAddress
    inherit_from: tap-mssql
    config:
      stream_maps:
        Shopping-BookingCustomerAddress:
          DescriptionLookupId: __NULL__
    select:
    - Shopping-BookingCustomerAddress.*
    metadata:
      Shopping-BookingCustomerAddress:
        replication-method: INCREMENTAL
        replication-key: LastEditDate