Hi All, I have a use case of using meltano as pure EL , I see the data gets skimmed based on the schema we provide, I would like to know how can we transform complete data to target without removing fields which are not defined in the schema. Thanks

It's not currently possible, the reason being the

SCHEMA

messages would be missing some of the fields and so the target would still fail to accommodate them.

@Edgar Ramírez (Arch.dev) - Can we extend the batching to a different format and store them in S3 without sending them to target by using "meltano invoke-command" • Create a Custom batching technique by extending the current batching • Store batched files in S3 • Invoke all this process by Meltano Invoke -- does this get state and update on completion.

I don't think I understand how that would address the issue of the schema being skimmed off of missing fields. There's also already support for S3 file batching in the singer sdk, so the best way to confirm would be to try it.