Is it possible to maintain two different version of the same target-(destination) in one meltano project?

Check out inheriting plugin definitions.

meltano add loader target-postgres--custom --inherit-from  target-postgres

let me be more precise, two different variants..

Hmm, I guess not with the same

name

which sort of makes sense...

Yeah it was easy I just suffixed with -alpha, -beta and put an -e flag with the git url

@k_s just for clarity (and because I’ll probably need it at some point), could you post a small example?

Hi @Henning Holgersen the meltano squared repo has a few examples, but if you specifically want to see inheritance with different variants that could be new territory https://github.com/meltano/squared/blob/main/data/load/loaders.meltano.yml

I do a lot of inheritance (multi-level), but all based on the same origin. But, hypothetically, if I need both the transferwise and the meltanolabs version of the snowflake tap, in the same project, I would have to do some trial-and-error.

Right. I can’t say for sure that will work either. I’m increasingly not a fan of the whole variants idea. It creates more code paths and complexity for something that could be simple: a different name or a different version. As an example, with variants idea you need a default, you need a collection under one name, you need optional cmd line args, you need properties with inheritance considering the variant, you need different hub responses, the list goes on…

@Henning Holgersen I can post the yaml spec on monday when I'm back at the pc.. but it was quite a simple solution.. I have both loaders specified, differentiated by the name(-alpha,-beta) and namespace(source_a, source_b). Then in the pip URL i included -e followed by #egg=package_name.. Eg, -e githuburl/target_snowflake.git#egg=target_snowflake, additionally I had to specify the executable. This solved it for me at least.

Good work @k_s - sounds like you've defined a custom plugin. 👍

Indeed

Sounds like the solution came through. 👍 @aaron_phethean, re:

I’m increasingly not a fan of the whole variants idea. It creates more code paths and complexity for something that could be simple: a different name or a different version. As an example, with variants idea you need a default, you need a collection under one name, you need optional cmd line args, you need properties with inheritance considering the variant, you need different hub responses, the list goes on…

Related: this has been on the office hours "to discuss" backburner for a while: Move from variant+name to plugin_id, plugin_uid comprised of maintainer_id and plugin_name#209 If anyone wants to meet and discuss this week, could make a good discussion topic.

I’ll describe a use case we have for this: We have about 40GB of data ingested with pipelinewise-tap-postgres. We would like to try out the melano labs tap-postgres on some new tables as a trial run before switching our other tables to it.

Sounds good @aaronsteers - I've started to think of the plugin name as a ‘fully qualified name' including variant. I could equally support moving the variant into the version which would give a project more certainty about only one instance of a plugin. I'll add some use cases to the issue 👍