Hi all, was wondering thoughts on the best practic...
# best-practices
e
Hi all, was wondering thoughts on the best practice for adding metadata via a mapper to flow runs? Basically what's going on is we have an extractor -> loader -> mapper -> another extractor -> another loader. An example would be external API -> S3 -> either custom mapper or the one of the two below -> S3 extractor -> Redshift loader. What we're thinking is using either the
pipelinewise-transform-field
or
meltano-map-transform
but a large caveat is that we're being SDK agnostic as we evaluate tools still. We're very much leaning towards Meltano/Singer based on the ecosystem that's already built up. Was wondering thoughts on if this would be the easiest flow? We want to keep the S3 files "clean" hence the extended flow so we can maximize the LT part of ELT