Is there anyway to use the pipelinewise transformations? Maybe I’m missing something in the documentation but I don’t see how this fits into the

meltano.yml

file or another way to add it in. Thanks!

This is coming soon!! We're refactoring a bit now to make pipelines more composable within the Meltano ELT engine, and will then add support for inline transformations. @matt_cooley, can you say a bit more about which pipelinewise transformation(s) you are looking to use? Our first implementation would likely leverage the Stream Maps feature in the SDK, which at least in theory should be a superset of the pipelinewise transform capabilities (because that was our inspiration).

The big one is nulling or hashing sensitive columns. Part of the reason for this is the fact that the pipelinewise MySQL tap doesn’t support explicit column selecting with Log Based replication (wish they would). But they are more active in developing their taps so I would like to move away from the singer tap we’re currently using.

Makes sense. And yes, those use cases are well handled in the stream maps capability. 👍

Rad! I did think of one more: the ability to cast a column. Example:

json (mysql) -> super (Redshift)

. Not sure if that is also in scope. I think that the

schema

extra already does some of this. Thanks for the responses!

The stream maps have some basic type casting capabilities, mostly driven by encapsulation of the expression in a hard-coded list of casting wrappers like

str(...

or

int(...

but you are right this can be done more explicitly in the schema extra.