Hi all, I'm relatively new to Meltano and had a question about Data Transformation of streaming data. I'm looking at using Meltano for both batch and streaming processes. For streaming, I'm working with a custom REST API Tap that loads to a custom Kafka Target but would like to do some data transformation directly in the stream before sending it to Kafka. I guess this would be more ETL than ELT, but wondering if there was any documentation or experience with that scenario? Thanks!

What types of transformations are you looking to do? Theres a stream maps feature for taps or targets using the SDK (and very soon any tap/target) that can do some relatively small inline transformations. Would that help for your use case?

Just looking to add a few fields to the schema to add some uniform keys across different data feeds. I might have missed the stream maps feature, I'll take a look at that. Thank you!

Yeah I think stream maps might work for you then!