I added some fields to tap salesforce in meltano yml technic Meltano #troubleshooting

I added some fields to tap-salesforce in meltano.y...

will_johnson

01/19/2023, 10:28 PM

I added some fields to tap-salesforce in meltano.yml (technically in another file included in meltano.yml). Some of the fields came through but some didn't. I suspect that the catalog was out of date and the newer Salesforce fields didn't pull. In my attempts to fix, I think I set the source extra to nothing (meltano select tap-salesforce --list doesn't show any included or excluded patterns) but that is overriding my meltano.yml. How do I undo this?

visch

01/19/2023, 10:58 PM

I believe

meltano install --clean

works,

rm -rf .meltano/run/tap-name

also works,

rm-rf .meltano/run

works too but does delete all the run history and things

visch

01/19/2023, 10:59 PM

If it's cache causing this, otherwise it's something in your

meltano.yml

file if we could take a peek?

will_johnson

01/19/2023, 11:11 PM

Would

meltano install --clean

affect my other top?

will_johnson

01/19/2023, 11:12 PM

My meltano.yml did not change. I made changed to salesforce.meltano.yml which is included in the include_paths inside of meltano.yml. Only a handful of columns were added to the salesforce.meltano.yml. Some of the columns that were added came through but other did not.

visch

01/19/2023, 11:13 PM

Yes, re https://docs.meltano.com/reference/command-line-interface#install it reinstalls them all If it's cache it's stored here

.meltano/run/tap-name

I think --clean removes that for you I"m not certain though

visch

01/19/2023, 11:13 PM

Otherwise it's something else causing it

will_johnson

01/19/2023, 11:18 PM

does clean reset state?

will_johnson

01/19/2023, 11:20 PM

I don't mind that on tap-salesforce (I run --full-refresh sometimes on that) but on my tap-postgres it would take a long time for that to start from scratch.

visch

01/19/2023, 11:21 PM

I don't think it does, but you can just run install for one tap, if you're worried about state restarting I"d definitely back it up just in case any ways

will_johnson

01/19/2023, 11:29 PM

meltano install extractor tap-salesforce --clean

did not work

will_johnson

01/19/2023, 11:30 PM

meltano select tap-salesforce --list

still showing empty

will_johnson

01/19/2023, 11:36 PM

State of my other tap didn't reset which is good.

will_johnson

01/19/2023, 11:43 PM

rm -rf .meltano/run/tap-name

also didn't work

will_johnson

01/19/2023, 11:45 PM

aghh...This was added to the bottom of my meltano.yml:

will_johnson

01/20/2023, 2:54 AM

tap-salesforce is working again but still have my original problem.

meltano invoke tap-salesforce --discover

isn't picking up four fields from a custom Salesforce object but is picking up other fields from that object. I looked in Salesforce Workbench and see the fields in there. I looked at it in Stitch and that does see the fields. So it seems to be something with the default tap-salesforce. Just throwing it out there, but all of these field names happen to also be field names on other object. Shouldn't matter but something I noticed.

visch

01/20/2023, 2:20 PM

Ok cool glad we solved the first problem which was https://meltano.slack.com/archives/C01TCRBBJD7/p1674169199348479?thread_ts=1674167317.300099&cid=C01TCRBBJD7 I'd try a different variant from here https://hub.meltano.com/extractors/tap-salesforce and maybe you can rule out the variant

visch

01/20/2023, 2:22 PM

Then if it is the variatn we can get an issue into the meltanolabs salesforce tap potentially. I don't use tap-salesforce so I don't know much about the details. It looks like there's settings that could impact this, it's possible there's a bug in getting custom fields for discovery I'm not sure.

will_johnson

01/20/2023, 6:48 PM

I'm working through authentication on the other variants.

will_johnson

01/21/2023, 1:48 AM

It turns out that our Salesforce admin who created the fields missed adding the field level security for those fields for the profile of our system integration user. That was a long rabbit hole but adding that proper permissions fixed it. Didn't know to check there. I still had to run

meltano install extractor tap-salesforce --clean

after fixing the permissions. Thank you for your help!

visch

01/21/2023, 2:28 AM

awesome!!! Glad you found it

visch

01/21/2023, 2:29 AM

I wonder if we could add something to the readme for the meltanolabs salesforce variant. Just going over what to do if you'r emissing custom fields in a custom object

visch

01/21/2023, 2:29 AM

Specifically the permissions issues

will_johnson

01/24/2023, 5:33 PM

Honestly, a line in a FAQ for that may be good particularly when the person setting up Meltano may be different from the person administering Salesforce. I finally uncovered the issue by an answer here: https://salesforce.stackexchange.com/questions/191187/fields-missing-from-api-request I was looking at the tap log and seeing the endpoint for the object and then seeing the same result of the field missing when I was hitting the endpoint myself. That eliminated any issue with the tap. But that was a step or two from the initial symptom (the tap not pulling all fields on an object).

Open in Slack

Previous Next