Hi guys, I have a quick question regarding the best way to handle multiple taps in Meltano, each targeting around 1,000 tables. These tables are filtered from a source database that contains over 5,000 tables. Due to the volume, it's not feasible to list all 1,000 tables under the

select

parameter in the

meltano.yml

file. On the other hand, if I use a custom

catalog.json

, Meltano seems to ignore the tap configuration defined in

meltano.yml

. What would be the recommended approach in this scenario? If there are any documented use cases or best practices for managing large-scale tap configurations like this, I’d really appreciate it if you could share them.

On the other hand, if I use a custom

catalog.json

, Meltano seems to ignore the tap configuration defined in

meltano.yml

.

Do you mean the

metadata

? or also things defined under

config

? Is there a pattern to the table names that you could use in a glob expression?

➕ to edgar's comments Closest I have to that in production is an oracle -> mssql integration with roughly 400 tables. We use the select filter extra here https://docs.meltano.com/concepts/plugins/#select_filter-extra and a small wrapper around Meltano to verify the tap is selected our calls look like this

runner -tap_name=tap-powerschool --target_name=target-mssql --select_filter="PS-TABLE1" --dbt_modelfilter="+DataMart__Schema__Table1"'

Not saying you should do it this way but it's what we went with and it's been working for ~4 years

@Edgar Ramírez (Arch.dev) Thanks for the reply! Yes, I meant that when using a custom

catalog.json

, Meltano seems to ignore settings under both

metadata

and

select

that are defined in

meltano.yml

. To work around this, I adopted a new approach: I'm breaking down each tap configuration into separate YAML files and using the base

tap-mysql

config through

include_paths

. So now I can handle all the taps configurations without hitting the size limitations of a single

meltano.yml

file, while still preserving selection logic and metadata. please let me know if it's a good way to handle that or if there is a better way?

that with glob syntax as Edgar mentioned ie

*.*

for your select and you should be cooking