I have been working on a use case for elt between tap-salesforce and target-postgres. Have encountered below issues during the setup and test: 1. The is_sandbox parameter doesn't seem to be working as expected. Even after setting it to true, it's trying to connect to login.salesforce.com . On some analysis, I could see that the parameter is_sandbox is not passed while calling the method from_credentials in ___init___.py file within salesforce folder. 2. After making the fix locally, the discovery seems to be taking long time initially and generates a 10MB file. Also, the process seems to get stuck without any log from target-postgres. 3. I've tried to provide a local catalog file in the catalog key in meltano.yml so that it doesn't discover catalog for every invocation. However, it doesn't seem to pick up the entities I've specified in the select_filter option. As a result, we have to edit the catalog file to select the entity. 4. After editing the catalog file to select the entity, I could see the records in the elt log, but there is no log from the target-postgres and no data in the table as well. 5. The target-postgres seems to be working well as I've tested that successfully with the tap-csv. 6. I also faced an issue similar to this but could not implement the fix as it doesn't work with a local catalog file. So is there any way to specify the list of entities and attributes along with a local catalog file.

@ishan_b Thanks for sharing, let's figure this out together so that the next user will have a better experience!

The is_sandbox parameter doesn't seem to be working as expected. Even after setting it to true, it's trying to connect to login.salesforce.com . On some analysis, I could see that the parameter is_sandbox is not passed while calling the method from_credentials in ___init___.py file within salesforce folder.

That's not good; can you please create an issue in https://gitlab.com/meltano/tap-salesforce and submit your fix in an MR? The

is_sandbox

setting is not currently documented on https://meltano.com/plugins/extractors/salesforce.html, so we should fix that as well once it actually works.

After making the fix locally, the discovery seems to be taking long time initially and generates a 10MB file.

That's not ideal... Meltano caches the discovered catalog when using

meltano select

to select entities (or list selected entities), so that each subsequent run won't be so slow, but this cache isn't currently reused in

meltano elt

to prevent hard to debug issues because of outdated catalogs. I think the better cache invalidation I implemented in https://gitlab.com/meltano/meltano/-/merge_requests/1879 should make that a non-issue, though. Do you think you could create an issue about caching discovery on

meltano elt

as well?

Also, the process seems to get stuck without any log from target-postgres.

Do you mean that

meltano elt

eventually fails before ever getting to

target-postgres

, or that discovery takes so slow that it appears to never get to

target-postgres

?

target-postgres

is pretty quiet on the logs, so it may have run just fine without ever outputting anything.

I've tried to provide a local catalog file in the catalog key in meltano.yml so that it doesn't discover catalog for every invocation. However, it doesn't seem to pick up the entities I've specified in the select_filter option. As a result, we have to edit the catalog file to select the entity.

select_filter

can only be used to narrow down the set of selected entities from what's already selected in the catalog file once it makes it to

select_filter

, so it only works if your custom catalog already has some entities selected that you'd like to select a subset of. You can't use it to select additional entities, which may be why it didn't seem to work for you. As you know, selecting additional entities is where

select

comes in, which only works if you let Meltano handle catalog discovery and processing. Providing a custom catalog suggests that it's already been carefully modified to your liking, and that Meltano should apply its own selection rules on it.

After editing the catalog file to select the entity, I could see the records in the elt log, but there is no log from the target-postgres and no data in the table as well.

The target-postgres seems to be working well as I've tested that successfully with the tap-csv.

Does this mean that you confirmed with

meltano --log-level=debug elt

that

SCHEMA

and

RECORD

messages are flowing out of the tap and into the target, but that the target doesn't appear to be doing anything with them? Or could it be that the tap still isn't extracting records correctly?

I also faced an issue similar to this but could not implement the fix as it doesn't work with a local catalog file. So is there any way to specify the list of entities and attributes along with a local catalog file.

If you provide a custom catalog file, Meltano will no longer try to apply metadata rules to it. If you'd like to keep using a custom catalog file, you'll need to make any desired changes to it manually. I would recommend going back to having Meltano handle your catalog file if at all possible, especially if we fix the caching issue discussed under point 2.

@douwe_maan

That's not good; can you please create an issue in https://gitlab.com/meltano/tap-salesforce and submit your fix in an MR?

I have switched to singer.io tap-salesforce plugin and it seemed to work well. Will submit the fix for the meltano tap-salesforce plugin.

That's not ideal... Meltano caches the discovered catalog when using

meltano select

to select entities (or list selected entities), so that each subsequent

run won't be so slow, but this cache isn't currently reused in

meltano elt

to prevent hard to debug issues because of outdated catalogs. I think the better cache invalidation I implemented in https://gitlab.com/meltano/meltano/-/merge_requests/1879 should make that a non-issue, though. Do you think you could create an issue about caching discovery on

meltano elt

as well?

Sorry for not specifying earlier that it's generating the catalog for each elt run. I didn't face any issue during the select. Regarding the catalog cache, meltano generates the entire catalog on each elt run. For frequent elt runs, the catalog generation takes up considerable amount of time especially if we want to sync 5-10 entities out of the ~500 entities defined in catalog for each run. The discover phase could end up taking more time than the actual elt phase. I'm not sure if it would help to have a catalog cache with full support of meltano features (select, schema etc.) and also catalog refreshed ondemand/schedule. I'll create an issue on this as well but want to check if this is not considered already.

Do you mean that

meltano elt

eventually fails before ever getting to

target-postgres

, or that discovery takes so slow that it appears to never get to

target-postgres

?

target-postgres

is pretty quiet on the logs, so it may have run just fine without ever outputting anything.

I guess there were no actual messages generated from the tap-salesforce, but the meltano command seemed to get stuck after printing the environment. I'll put more info regarding the error with tap-salesforce after creating a separate project with meltano plugin for tap-salesforce. I was able to proceed ahead with my usecase after using the singer.io plugin.

I have switched to singer.io tap-salesforce plugin and it seemed to work well. Will submit the fix for the meltano tap-salesforce plugin.

Ok, thanks. We already have an issue to make the singer-io variant discoverable and consider making it the default: https://gitlab.com/meltano/meltano/-/issues/2430

Sorry for not specifying earlier that it's generating the catalog for each elt run. I didn't face any issue during the select. Regarding the catalog cache, meltano generates the entire catalog on each elt run. For frequent elt runs, the catalog generation takes up considerable amount of time especially if we want to sync 5-10 entities out of the ~500 entities defined in catalog for each run. The discover phase could end up taking more time than the actual elt phase. I'm not sure if it would help to have a catalog cache with full support of meltano features (select, schema etc.) and also catalog refreshed ondemand/schedule. I'll create an issue on this as well but want to check if this is not considered already.

The catalog cache used by

meltano select

already knows to invalidate itself when select, schema and metadata are modified, or the settings which could affect the result of

--discover

. If we start using the cache in

meltano elt

as well, we should add a way to explicitly invalidate it, or add a timeout of some kind, and we should clearly print in debug logs that the cached version. We can discuss that some more in the new issue 🙂