Does anyone have thoughts on best practices for structuring projects with multiple data sources? Currently, I have several taps implemented that all push data into a single schema in redshift. I’ve finally reached the point where two of those data sources (specifically zendesk and jira) have tables with the exact same names, which has started causing a collision. An obvious way to deal with this would be to load each data source into a different redshift schema, but (as far as I can tell) one can only set a single schema in the config for

pipelinewise-target-redshift

. Has anyone else dealt with this in a way they like?

Have you tried using plugin inheritance to configure target-redshift differently in the two pipelines? See http://meltano.com/docs/configuration.html#multiple-plugin-configurations

Another option I’ve considered is using plugin inheritance to set up separate targets for different schemas. I think this would work well, except that I haven’t figured out how to get it to play nice with my custom plugins. For example, this:

- name: pipelinewise-target-redshift-public
    inherit_from: pipelinewise-target-redshift
    config:
      default_target_schema: public

That should work fine with custom plugins! What issue are you seeing?

yields an error:

ELT could not be completed: Cannot start loader: Executable 'target-redshift' could not be found. Loader 'pipelinewise-target-redshift-public' may not have been installed yet using `meltano install loader pipelinewise-target-redshift-public`, or the executable name may be incorrect.

Did you run

meltano install loader pipelinewise-target-redshift-public

?

ah! that’s probably it - do I need to install it again to register the “new name” even though the parent plugin is already installed/running fine?

From Meltano's perspective, each plugin currently needs to be installed separately, despite inheritance: https://gitlab.com/meltano/meltano/-/issues/2490

that’ll do it then - want me to put in an issue to add that in the docs?

Please do! I had hoped that error message would get it across, but evidently not 😄

FWIW, I can totally see how that error message is trying to tell me that now that I know what it’s trying to tell me 🙂

Yeah I get why it's not clear. With inheritance, it should just use the existing installation, it just doesn't at the moment!

yep makes sense

Yeah, separate schemas the way to go. To have target-redshift automatically determine the schema based on the extractor you're running it with, you can set the value of the setting to

$MELTANO_EXTRACT__LOAD_SCHEMA

like here: https://gitlab.com/meltano/meltano/blob/master/src/meltano/core/bundle/discovery.yml#L1454

oh interesting! that’s good to know - I’ll play around with both and see what works best