Hi everyone, im new in working with meltano and currently working on some new Rest Taps. I have already build some taps and they run great, but i am know working on the mapping. I had a look at the singer documentation and it is mentioned that using the meltano.yaml, can change the mapping of the taps. I would prefer to use the mapping inside the stream class. I also noticed that the Stream classs has a stream_maps property of type [StreamMap], but i did not find any documentation on Stream Map. Is there a way to define a mapping schema on the stream Class for mapping the input data to specific schema? Mainly aliasing, unit conversion and some string opertions. Thank you in advance!

Would the

post_process

method help? It applies to each record before being emitted. Or do you want tap users to be able to modify the mapping? https://sdk.meltano.com/en/latest/classes/singer_sdk.Stream.html#singer_sdk.Stream.post_process

I've had the same thoughts before too - here are some related issues: https://github.com/meltano/sdk/issues/1231 https://github.com/meltano/sdk/issues/792

Was just looking at post_process. It sound like a good start. Tap Users should not be able to use the mapping. Thanks for the advice

Up to now we've been using

post_process

, as Andy mentioned: https://github.com/Matatika/tap-auth0/blob/d750fc50bea995a604a8b478ee2b6ec9bbe796f2/tap_auth0/streams.py#L181-L182

Yeah if it's a guaranteed 'everyone who uses this tap will want the changes' then

post_process

is ideal. It's also a nice way to separate logic that might otherwise live in

get_records

to keep the delineation between 'here's what the api returns' (

get_records

) and 'heres the extra changes I think you need (

post_process

)