Is there a recommended method for taking many different tables from one or more taps and adding all the data to just one target table? My particular situation requires taking a table (same name in each schema) from hundreds of different mysql schemas (all different names) and adding them all to one target table (presumably the same name as the source tables) in the singer target db (postgres, redshift, or snowflake). Each row in the target table should have an additional column indicating the name of the schema from which the row was replicated.

You can configure the target to load everything into the same schema, instead of determining the schema based on the name of the tap, so that all data ends up in the same table. If you want to add a new column with the name of the source schema/db, that'd be a task for dbt, though

is there documentation that describes how to configure the target as you described? It’s not immediately obvious to me how to do this because all of my tables have a default naming convention of

<schema>_<table>.<field>

@josh_lloyd Which specific tap and target variants are you using?

the default tap-mysql to target-postgres for my initial testing

OK, I suggest switching to the

transferwise

variant of the loader: https://meltano.com/plugins/loaders/postgres--transferwise.html#getting-started which handles the

<schema>-<table>

stream names better and lets you set both a default target schema: https://meltano.com/plugins/loaders/postgres--transferwise.html#default-target-schema and a schema mapping: https://meltano.com/plugins/loaders/postgres--transferwise.html#schema-mapping

I have the config option

postgres_schema

set to a particular schema of my choosing. Is that not the same thing?

It would be, but I don't think the

datamill-co

variant knows the unwrap the

<schema>-<table>

stream names

gotcha

So you'd always get new tables with the source schema in the name