<https://github.com/meltano/meltano/issues/2848> <...
# troubleshootingv
visch
03/08/2023, 12:57 PMhttps://github.com/meltano/meltano/issues/2848 @jan_soubusta regarding your recent ask about Catalog caching in SFDC. That thread has gone in a few different directions so I didn't want to clutter it
visch
03/08/2023, 12:58 PMI had the opposite problem from you I think. I wanted the catalog to refresh more often
j
jan_soubusta
03/08/2023, 2:15 PMSubscribed to the issue, thanks for mentioning it!
It strongly depends on the type of tap. If the refresh is very slow, you want to suppress the refresh, and you are ok to be careful when you have to trigger it.
v
visch
03/08/2023, 2:16 PMYeah I forgot to mention, a good idea in there I think was TTL's for caching then you could tweak what your ttl is 🤷
j
jan_soubusta
03/08/2023, 4:57 PMThe discussions here in Slack and in the issues are focused on allowing forcing of refresh of catalog.
My requirement is the opposite - allow forcing not refreshing catalog.
jan_soubusta
03/08/2023, 5:00 PMQuestions:
• if I specify a list of particular fields in the tap select section (no wildcard used), would it be possible to skip the refresh of the catalog always?
• When I run tap in a pipeline, I always lose .meltano/run between runs. Does it mean that the refresh is executed always? Does not seem so, when I look at pipeline jobs runtime. Is the catalog cached also in the state backend(in my case S3)?
v
visch
03/08/2023, 5:07 PMif I specify a list of particular fields in the tap select section (no wildcard used), would it be possible to skip the refresh of the catalog always?should work, it's designed for dev workloads so it doesn't refresh all the time.
When I run tap in a pipeline, I always lose .meltano/run between runs. Does it mean that the refresh is executed always? Does not seem so, when I look at pipeline jobs runtime. Is the catalog cached also in the state backend(in my case S3)?If you lose that every time then yes it has to refresh the catalog everytime. catalog is cached in
.runj
jan_soubusta
03/08/2023, 5:30 PMEh, I was wrong. Last pipeline was running 4min30 out of which 3min took the catalog refresh, Horrible.
jan_soubusta
03/08/2023, 5:30 PMWill focus on caching .run in Gitlab and reuse it in following pipeline runs.
jan_soubusta
03/08/2023, 5:32 PMStrangely, I do not see folders as you described:
jan_soubusta
03/08/2023, 5:33 PMShould I cache .meltano/cache?
v
visch
03/08/2023, 5:33 PMI lied to you I'm sorry
visch
03/08/2023, 5:33 PM.meltano/run
visch
03/08/2023, 5:33 PMis the right dir!
j
jan_soubusta
03/08/2023, 5:34 PMI see some data in this folder only for targets + lock file. Where exactly is the tap-salesforce cache(catalog) stored?
v
visch
03/08/2023, 5:35 PMThe .meltano/run/tap-salesforce folder should have it, not sure why you don't 🤷 have to see how you run it
j
jan_soubusta
03/08/2023, 5:35 PMOn the other hand, the file
.meltano/cache/discovery.ymljan_soubusta
03/08/2023, 5:36 PMI run
meltano --environment dev_local run tap-salesforce target-snowflakev
visch
03/08/2023, 5:39 PMThe run dir should have a
tap-namevisch
03/08/2023, 5:40 PMhas to exist as that's where the
configcatalogj
jan_soubusta
03/08/2023, 5:48 PMok, will bootstrap everything from scratch and run it again, we'll see if my local env gets "fixed" 😉