Hello, does meltano support data streaming, or any...
# getting-starteds
springroll12
06/26/2024, 3:55 PMHello, does meltano support data streaming, or any sort of "push data to meltano" model?
e
Edgar Ramírez (Arch.dev)
06/26/2024, 6:59 PMHi @springroll12. What's your use case?
s
springroll12
06/26/2024, 7:00 PMWe have a realtime stream of data coming from a 3rd party provider that we want to send to an S3 bucket.
e
Edgar Ramírez (Arch.dev)
06/26/2024, 7:27 PMIt's an API?
s
springroll12
06/26/2024, 7:28 PMThe 3rd party is sending data to our API (currently). Our API then sends data to s3. Is there a way to achieve this with meltano?
e
Edgar Ramírez (Arch.dev)
06/26/2024, 8:03 PMSo it's kinda like a webhook?
s
springroll12
06/26/2024, 8:10 PMSort of, but with authentication
springroll12
06/26/2024, 8:10 PMand they submit data directly to the hook and we don't have the ability to pull data if they hit the hook (it's a uni-directional connection)
e
Edgar Ramírez (Arch.dev)
06/26/2024, 8:19 PMThe only way Meltano would support that sort of pattern at the moment would be to act as a middle man between the webhook receiver and loading data into s3. For example, you could send the data to sqs and use tap-amazon-sqs and target-s3.
2 Views