https://linen.dev logo
Join Slack
Powered by
Hello all!! My name is Panos and I'm the co-founde...
# singer-taps
p
Hello all!! My name is Panos and I'm the co-founder @ Keragon (think Zapier for healthcare) We have been using Meltano for a while now but came across an issue that I have not managed to resolve from the docs or online material. We are using tap-stripe (airbyte variant) with target-clickhouse. Stripe has a metadata object for most streams. In the Subscriptions stream we store an ID that we need to join across our other data. The tap-stripe (airbyte variant) connector always results in a empty object in the metadata column. I also get this log during execution: 
[info     ] 2024-05-20 23:02:25,068 | WARNING  | tap-airbyte.subscriptions | Properties ('discounts', '*metadata.uniqueId*', 'plan.amount_decimal', 'plan.meter') were present in the 'subscriptions' stream but not found in catalog schema. Ignoring.
metadata.uniqueId
 is the property i'm interested in. I've tried to configure this in meltano.yml like this but with no luck!
Copy code
plugins:
  extractors:
  - name: tap-stripe
    schema:
      subscriptions:
        metadata:
          type: ["object", "null"]
          additionalProperties: true
Any pointers would be greatly appreciated! Thanks in advance!
v
I haven't done much with the airbyte taps, but that message normally means the property isn't selected. Can you share more of your meltano.yml?
p
Sure!
Copy code
version: 1
default_environment: dev
project_id: a71f7def-be9d-4723-905b-a8433298d70e
environments:
- name: prod
- name: staging
- name: dev
plugins:
  extractors:
  - name: tap-stripe
    schema:
      subscriptions:
        metadata:
          type: ["object", "null"]
          additionalProperties: true
    variant: airbyte
    pip_url: git+<https://github.com/MeltanoLabs/tap-airbyte-wrapper.git>
    config:
      stream_maps:
        customers:
          __alias__: stripe_customers
        plans:
          __alias__: stripe_plans
        products:
          __alias__: stripe_products
        subscriptions:
          __alias__: stripe_subscriptions
        promotion_codes:
          __alias__: stripe_promotion_codes
    select:
    - customers.*
    - plans.*
    - products.*
    - promotion_codes.*
    - subscriptions.cancel_at
    - subscriptions.cancel_at_period_end
    - subscriptions.canceled_at
    - subscriptions.collection_method
    - subscriptions.created
    - subscriptions.currency
    - subscriptions.current_period_end
    - subscriptions.current_period_start
    - subscriptions.customer
    - subscriptions.days_until_due
    - subscriptions.discount
    - subscriptions.discount.coupon
    - subscriptions.discount.coupon.amount_off
    - subscriptions.discount.coupon.duration
    - subscriptions.discount.coupon.duration_in_months
    - subscriptions.discount.coupon.id
    - subscriptions.discount.coupon.name
    - subscriptions.discount.coupon.object
    - subscriptions.discount.coupon.percent_off
    - subscriptions.discount.coupon.percent_off_precise
    - subscriptions.discount.coupon.times_redeemed
    - subscriptions.discount.coupon.valid
    - subscriptions.discount.customer
    - subscriptions.discount.end
    - subscriptions.discount.object
    - subscriptions.discount.start
    - subscriptions.discount.subscription
    - subscriptions.ended_at
    - subscriptions.id
    - subscriptions.is_deleted
    - subscriptions.latest_invoice
    - subscriptions.livemode
    - subscriptions.metadata
    - subscriptions.plan
    - subscriptions.plan.active
    - subscriptions.plan.amount
    - subscriptions.plan.billing_scheme
    - subscriptions.plan.created
    - subscriptions.plan.currency
    - subscriptions.plan.id
    - subscriptions.plan.interval
    - subscriptions.plan.interval_count
    - subscriptions.plan.name
    - subscriptions.plan.nickname
    - subscriptions.plan.object
    - subscriptions.plan.product
    - subscriptions.plan.usage_type
    - subscriptions.start_date
    - subscriptions.status
    - subscriptions.trial_end
    - subscriptions.trial_start
    - subscriptions.updated
  - name: tap-clerk
    namespace: tap_clerk
    pip_url: tap-clerk
    executable: tap-clerk
    capabilities:
    - catalog
    - discover
    - state
    settings:
    - name: auth_token
    config:
      stream_maps:
        organizations:
          __alias__: clerk_organizations
        organization_membership:
          __alias__: clerk_organization_membership
        users:
          __alias__: clerk_users
  loaders:
  - name: target-clickhouse
    variant: shaped-ai
    pip_url: git+<https://github.com/shaped-ai/target-clickhouse.git>
    config:
      database: db
      host: click
      username: default
      load_method: overwrite
jobs:
- name: stripe_sync
  tasks:
  - tap-stripe target-clickhouse
- name: clerk_sync
  tasks:
  - tap-clerk target-clickhouse
v
try updating the metadata portion of your select to 
- subscriptions.metadata*
You can also view what is selected or not with 
meltano select tap-stripe --list
which should I believe show your missing / included field(s)
p
Tried this: 
- subscriptions.metadata.uniqueId
It didn't work
v
Try my recommendation
p
Just did
- subscriptions.metadata*
Also no luck
😢 1
also tried 
- subscriptions.metadata.*
before
v
Then I'd dive into the catalog if it doesnt' work so
meltano invoke --dump=catalog tap-stripe
Go peak through there and find your stream, maybe it is being selected and there's something else going on. That's where I'd start at least. I'm not sure if there's something related to the airbyte shim or not
p
Is there a way to "override" the schema defined in the connector ?
Or set additionalProperties=true
for this property?
e
Is there a way to "override" the schema defined in the connector ?
Yeah, you've probably seen https://docs.meltano.com/concepts/plugins/#schema-extra Maybe
Copy code
schema:
  subscriptions:
    metadata:
      type: ["object", "null"]
      properties:
        uniqueId:
          type: [string, "null"]
Or set additionalProperties=true for this property?
The latest (0.37.0) release of 
singer-sdk
should cover this via #2301, so it's hopefully a matter of sending MeltanoLabs/tap-airbyte-wrapper a PR bumping the version to make your original 
schema
override work.
dancingpenguin 1
p
Thank you so much @visch & @Edgar Ramírez (Arch.dev) for the help here! I have opened a PR
e
Np!
âž• 1
2 Views
Open in Slack
PreviousNext
Powered by