Hey I am trying to set up a data pipeline to get objects sta Meltano #troubleshooting

Hey ! I am trying to set up a data pipeline to get...

ananya

04/25/2022, 8:25 AM

Hey ! I am trying to set up a data pipeline to get objects (standard and custom) from SFDC to Redshift but I am getting very few

standard

objects only into the Redshfit. 😞 Details are in the thread. Please help 🙏

ananya

04/25/2022, 8:25 AM

Configuration I have tried so far with tap-salesforce

meltano

and

singer-io

variant both: • With this config I am getting only 6 standard objects.

Copy code

config:
      refresh_token: $SECRET_TAP_SALESFORCE_REFRESH_TOKEN
      client_id: $SECRET_TAP_SALESFORCE_CLIENT_ID
      client_secret: $SECRET_TAP_SALESFORCE_CLIENT_SECRET
      start_date: $ENV_TAP_SALESFORCE_START_DATE
      api_type: $ENV_TAP_SALESFORCE_API_TYPE
      select_fields_by_default: true

• With this config my pipeline is running without any error but not getting

Campaign

object into the redshift.

Copy code

plugins:
  extractors:
  - name: tap-salesforce
    namespace: tap_salesforce
    variant: singer-io
    pip_url: git+<https://github.com/singer-io/tap-salesforce.git>
    config:
      refresh_token: $SECRET_TAP_SALESFORCE_REFRESH_TOKEN
      client_id: $SECRET_TAP_SALESFORCE_CLIENT_ID
      client_secret: $SECRET_TAP_SALESFORCE_CLIENT_SECRET
      start_date: $ENV_TAP_SALESFORCE_START_DATE
      api_type: $ENV_TAP_SALESFORCE_API_TYPE
      select_fields_by_default: false
    select: 
    - 'Campaign.*'

taylor

04/25/2022, 2:25 PM

what do you see if you run `meltano select tap-salesforce --list --all`https://docs.meltano.com/reference/command-line-interface#select

ananya

04/25/2022, 3:54 PM

with

meltano

variant, I am getting this error:

Copy code

Cannot list the selected attributes: Catalog discovery failed: command ['/project/.meltano/extractors/tap-salesforce/venv/bin/tap-salesforce', '--config', '/project/.meltano/run/tap-salesforce/tap.31767107-6c0b-4012-a965-8561ce55762a.config.json', '--discover'] returned 1

taylor

04/25/2022, 4:16 PM

Looking at https://github.com/singer-io/tap-salesforce it seems like this tap variant uses the deprecated

properties

cability. Can you add the follwing key to the extractor:

Copy code

capabilities:
  - properties
  - discover
  - state

taylor

04/25/2022, 4:16 PM

and then re-run?

ananya

04/27/2022, 11:12 AM

hey @taylor 👋 thank you for all the help. it’s working with

meltano

variant. 😊 gratitude thank you 🙏

Open in Slack

Previous Next