Is it possible to add extra fields that may be missing from records? If so, how? For instance:

{"a": 1, "b": 2, "c": 3}
{"a": 4, "b": 5}
{"a": 6}

Say I want to have fields a,b,c for all the different messages above, even if they're missing?

What target are you sending this to? If it’s a database then I would expect it to return null for those fields. This also may be appropriate for the Map layer that @aaronsteers has been working on for the SDK

@lasse_vang_gravesen - I don’t think this is possible today, since most of the time the contract between tap and target “EL” part of the pipeline is to replicate exactly what was in the pipeline. But yes, there are possible ways to extend that capability, including adding defaults and other inline transformations. You found the

pipelinewise-transform-field

but I don’t think that supports the feature you are looking for. To @taylor’s point, we do plan to deliver a new “stream_maps” feature in the SDK this month, which would certainly be usable for this purpose once added here as a Meltano feature as well. Can you say a little more about your use case? Any reason not to replace those defaults in a “T” transform layer after the EL completes? For most cases, we would recommend doing the transform after the EL pipeline, for instance in dbt, so that you know you have exactly replicated the upstream source with fidelity.

@aaronsteers hi, my problem is that I want to read from a tap (kafka) and dump to a CSV file using target-csv, but the kafka topic contains messages of different types like in my example

@lasse_vang_gravesen, I see now. Thanks for the extra explanation. Looks like a column ordering bug in the CSV target. Is your final plan to land in another system or just CSV? For instance, you might consider the jsonl target instead if you are only trying to land to a flat file, or there's also a target-s3-csv option which is more production-ready if you plan to ultimately load to S3.

Ultimately I want it to land in Redshift, and I just discovered that the redshift loader actually puts everything into S3 first

Yeah, that makes total sense. You might consider if other variants of target-Redshift retain the CSV. I wanted to same feature actually (

retain_s3_files

) in my Snowflake target so I forked the target and removed the "delete()" code. :) But someone might have done this already for Redshift. It's worth checking.

that was my next thing anyway, fork the redshift target and get rid of the deletion thing

That's correct. "Not easily." And yes, I can see your point regarding ordering vs when headers are written. Long and short is that the target should be adapting to new schemas. But because the generic target-csv doesn't usually get used in production scenarios, it doesn't deal as well with changing schemas. I expect other production-grade targets should deal with this better. I did also just check the pipelinewise variant for target-Redshift and did not see the retain-csv-files option there. https://github.com/transferwise/pipelinewise-target-redshift

Should've been more obvious what the redshift target was using s3 and putting csv files into it, I wouldn't have spent time on the other targets if I knew that

Sounds like you are on the right track. 👍 And yeah, sorry for the confusion around the internals of the target-redshift.