does anyone know of open source tools to stream da...
# randomp
prratek_ramchandani
10/22/2021, 5:17 PMdoes anyone know of open source tools to stream data from third-party webhooks (like Stripe or Zendesk) to something like Kafka or straight to a data warehouse? i know snowplow has a small handful of “webhook trackers” but that list hasn’t grown in a while and there doesn’t seem to be an accessible way to add new integrations.
a
aaronsteers
10/22/2021, 5:19 PMI think you can use API Gateway to plug webhooks to Lambda functions. Is this close to what you would need or something else?
aaronsteers
10/22/2021, 5:20 PMOh, you said open source. No, I don't personally.
p
prratek_ramchandani
10/22/2021, 5:21 PMyeah - my current solution was something similar on GCP: cloud function -> pub/sub -> bigquery. i just don’t know enough about building streaming systems to know if i’m missing an important piece
a
aaronsteers
10/22/2021, 5:44 PMcc @florian.hines in case he's got some ideas 🙂
f
florian.hines
10/22/2021, 5:48 PMhrm, theres the confluent rest gateway for kafka, but i don’t think that accepts arbitrary content. I think it still expects the payloads to be formatted a certain. Might be worth checking out.
florian.hines
10/22/2021, 5:49 PMTheres also a bunch of generic webhook catcher service - but i don’t know if any support dropping the result onto a message bus/queue.
florian.hines
10/22/2021, 6:02 PM@prratek_ramchandani fwiw i’ve done the generic cloud function -> pub/sub -> somedb or pattern a lot - including for things like stripe transactions. I think it works fine.
p
prratek_ramchandani
10/22/2021, 6:03 PMthat’s helpful thanks! i’ll stick with that pattern then