https://linen.dev logo
#singer-tap-development
Title
# singer-tap-development
s

silverbullet1

07/14/2023, 9:03 PM
Hi all, I wanted an opinion on designing taps: Say, I am writing a tap-redshift plugin and I intend to use it with target-s3. In this case, instead of streaming data directly in JSON format, it is much more efficient to do an
UNLOAD
operation in Redshift in one go with the select query, which also supports parallelism and other operations. Similarly, if I want to write data to postgres, I could use
COPY
to be more efficient(Redshift->S3->Postgres). So when designing tap, there are optimisations possible depending on what target I choose with it. My question is, should we think about these concerns when developing a new plugin ?