I have a `tap salesforce` question meltanolabs variant I did Meltano #troubleshooting

I have a `tap-salesforce` question (meltanolabs va...

Matt Menzenski

01/27/2023, 3:41 PM

I have a

tap-salesforce

question (meltanolabs variant). I did a full load yesterday and only got account, lead, opportunity, contact, opportunityhistory, and user streams. None of my org’s custom object types were output. I know that the credential I’m using has permissions to everything. I see lots of messages in the logs about skipping things that aren’t selected:

INFO PayIt_Solution__Share: Skipping - not selected

— how do I make them selected? I don’t see an obvious configuration setting for this. (Also I know I need to upgrade from v.1.5.0 to latest where that log message is debug level) Do I need to first generate a catalog file and then pass that to the tap? How do I do that?

Matt Menzenski

01/27/2023, 3:59 PM

I did generate a catalog with

meltano invoke tap-salesforce --discover | tee -a extract/salesforce/catalog.json

and then added

catalog: extract/salesforce/catalog.json

to the plugin settings in meltano.yml:

Copy code

- name: tap-salesforce
    variant: meltanolabs
    pip_url: git+<https://github.com/meltanolabs/tap-salesforce.git@v1.5.0>
    config:
      start_date: '2015-01-01T00:00:00Z'
    catalog: extract/salesforce/catalog.json

Following an example I found here. Then I ran

meltano run tap-salesforce target-redshift

again and while it did say that it found the catalog

Copy code

2023-01-27T15:55:47.059304Z [info     ] Found catalog in /Users/matt/dev/paw_elt/src/extract/salesforce/catalog.json

I get the same behavior as far as skipping “not selected” fields 😕

Matt Menzenski

01/27/2023, 4:12 PM

AHA

Matt Menzenski

01/27/2023, 4:12 PM

https://docs.meltano.com/getting-started/#select-entities-and-attributes-to-extract

Matt Menzenski

01/27/2023, 4:12 PM

it’s that isn’t it?

pat_nadolny

01/27/2023, 4:12 PM

@Matt Menzenski do you use any select criteria in your configuration? I think the tap is auto selecting certain fields by default which probably doesnt include your custom fields. Maybe if you put

Copy code

select:
    - *.*

pat_nadolny

01/27/2023, 4:12 PM

haha yeah probably

Matt Menzenski

01/27/2023, 4:14 PM

Like this?

Copy code

- name: tap-salesforce
    variant: meltanolabs
    pip_url: git+<https://github.com/meltanolabs/tap-salesforce.git@v1.5.0>
    config:
      start_date: '2015-01-01T00:00:00Z'
    select:
      - '*.*'

Matt Menzenski

01/27/2023, 4:14 PM

I’m a little scared of the volume this will give, lol -my catalog.json file is 255,000 lines with 749 distinct streams

pat_nadolny

01/27/2023, 4:28 PM

that looks right to me - you can select your specific streams with all fields selected like

- 'PayIt_Solution.*'

pat_nadolny

01/27/2023, 4:29 PM

heres my tap-gitlab select criteria as an example https://github.com/meltano/squared/blob/64b7316626fb856d9e0b8564bea8eea53b59df7e/data/extract/extractors.meltano.yml#L81

Matt Menzenski

01/27/2023, 6:15 PM

ok this was indeed the issue. Whitelisting to 15 objects to start and it’s working now 🦜

Open in Slack

Previous Next