https://linen.dev logo
Join Slack
Powered by
Hello, I have used the Stripe tap (both from Melta...
# announcements
d
Hello, I have used the Stripe tap (both from Meltano and Singer), but it doesn’t save the metadata that we add to Charges and other Stripe resources. Does anyone have experience how to extract the Metadata properties also?
d
@dejan_pejcev If neither https://github.com/singer-io/tap-stripe or https://github.com/meltano/tap-stripe currently supports the metadata API endpoint and stream type, you'll have to implement and contribute that yourself
d
huh, I tried by just running it with default settings and storing to a postgres target, but when I browsed the tables in Postgres, it didn’t save the 
metadata
property of the Stripe objects 😕 weird they didn’t implement such an important feature as this is the way to bind domain data with Stripe
d
I agree, that seems like an odd oversight
Are you comfortable enough with Python to attempt adding it yourself?
d
I have successfully avoided Python for years (did code a couple of times tho), but I knew that the day will come sooner or later. Create a PR for the Meltano variant?
d
Can't evade it if you're going to do data work 🙂
I suggest forking https://github.com/singer-io/tap-stripe since it's seen more recent improvements, and creating a PR there. It may not be accepted since the singer-io team hasn't been the best about maintaining their taps, but at least you'll have the most recent jumping-off point. Since https://github.com/meltano/tap-stripe itself is also forked from https://github.com/singer-io/tap-stripe, you can then also submit your PR to our fork so that we get all the latest singer-io improvements + your metadata additions. You can use your fork directly in Meltano: http://meltano.com/docs/plugin-management.html#using-a-custom-fork-of-a-plugin
Dejan mentioned that there is actually a definition for the 
metadata
property: https://github.com/singer-io/tap-stripe/blob/master/tap_stripe/schemas/charges.json#L7
Looking just at what we see there, it's possible thtt the target you're using isn't able to handle it because it defines an arbitrary object without specifying the nested properties. Not all targets may be prepared to handle that.
d
target-postgres
d
We can verify that the metadata is making it out of the tap and into the target by enabling debug mode: https://meltano.com/docs/command-line-interface.html#debugging
That'll show us 
tap-stripe (out)
lines for each SCHEMA and RECORD message. I'd like to see if 
metadata
shows up as expected there
I assume you're currently using the default 
datamill-co
variant of https://meltano.com/plugins/loaders/postgres.html? 
transferwise
or 
meltano
may support this if the default one doesn't.
But let's first verify the metadata is making it out of the tap!
d
Copy code
"metadata": {"orderSource": "xxxx", "orderNumber": "xxxx"}
yep, the metadata is in the output of the tap (I have redacted the values)
which means, target-postgres cannot handle a object with dynamic properties?
d
I'd expect the target to log some warning if it sees a column it doesn't know how to handle. Are you seeing nothing like that? It's definitely possible that one target-postgres silently ignores 
object
fields like that, while another may create a 
jsonl
column, which is presumably what you want
d
are there more postgres targets?
d
d
I feel like I am asking LMGTFY questions 😄
d
I know the docs by heart so it's NBD 😄
d
lemme try with the transferwise target, hopefully I get better results
UPDATE: Transferwise Target Postgres works wonderfully. It supports dynamic objects (like 
metadata
property on Stripe), and is a lot more customizable
d
Perfect!
7 Views
Open in Slack
PreviousNext
Powered by