Hi all I m new to Meltano and am trying to extract some data Meltano #troubleshooting

Hi all. I'm new to Meltano and am trying to extra...

matt_neiman

05/04/2023, 9:46 PM

Hi all. I'm new to Meltano and am trying to extract some data from Genesys Cloud. I'm using the dbt-labs variant of tap-purecloud, and while I'm able to extract data successfully, my select directives are being ignored. Each run does a full extraction of all data streams and seemingly for all history. When I run

meltano select tap-purecloud users

I do see the meltano.yml file update correctly. Running

meltano select tap-purecloud --list --all

likewise shows most entities as excluded. I'm not sure if perhaps my environment is not set up correctly. Any help would be appreciated.

edgar_ramirez_mondragon

05/04/2023, 10:26 PM

Hi Matt! Regarding selection, taking a look at their implementation it unfortunately looks like they don’t use the catalog (i.e.

args.catalog

) at all. It’s probably worth a re-implementation on the Singer SDK. As for state, can you try explicitly adding

capabilities

to your plugin definition? E.g.

Copy code

plugins:
  extractors:
  - name: tap-purecloud
    capabilities:
    - catalog # although it isn't really supported
    - discover
    - state

pat_nadolny

05/05/2023, 6:02 PM

Theres an open issue related to this in https://github.com/meltano/hub/issues/996

pat_nadolny

05/05/2023, 6:05 PM

This fork https://github.com/Pathlight/tap-purecloud seems to have more activity although it doesnt look like they use the catalog either. If you wanted to try it you could update your pip_url in your meltano.yml then run

meltano install extractor tap-purecloud --clean

to reinstall it. I'm curious to hear your outcome so we can help the others find the best variant as well!

edgar_ramirez_mondragon

05/05/2023, 6:06 PM

I also logged https://github.com/MeltanoLabs/Singer-Most-Wanted/issues/79

matt_neiman

05/05/2023, 8:26 PM

Thanks @edgar_ramirez_mondragon and @pat_nadolny for the responses! I haven't had a chance to test with a different variant, but I'll report back once I do. It's perhaps also worth mentioning that the existing taps only expose a fraction of what's available through the Genesys API. https://developer.genesys.cloud/devapps/api-explorer With the upcoming deprecation of canned reports, there could be increased interest in automated extraction of data for use in external tools. https://help.mypurecloud.com/articles/deprecation-canned-reports/ I'll dig in more deeply as time allows. I appreciate the attention to this tap!

matt_neiman

05/16/2023, 4:42 PM

Following up on this - I tried the Pathlight variant, and that one seems to make use of the start_date config setting. I'm able to run successfully to target-jsonl, although it does run for all streams. When I try to run with target-bigquery, the process is failing, and so far I can't figure out why. There's a good chance the failure is with a stream I don't need, but since the catalog is not implemented, I can't exclude the extraneous streams. Is there an easy way to permanently disable streams on an installed extractor plugin?

edgar_ramirez_mondragon

05/16/2023, 5:05 PM

Hi Matt!

Is there an easy way to permanently disable streams on an installed extractor plugin?

Unfortunately not, short of editing the installed

.py

files for the extractor to comment out the code paths (equivalent to installing from your own fork of the tap with those same changes).

When I try to run with target-bigquery, the process is failing, and so far I can’t figure out why.

Can you share the traceback you’re seeing?

matt_neiman

05/16/2023, 5:21 PM

Trace attached. FWIW, I have used the target-bigquery loader successfully w/ a different extractor on the same BQ dataset. Thanks for taking a look, Edgar!

pat_nadolny

05/18/2023, 5:48 PM

@matt_neiman FYI I opened a quick PR to hopefully support stream/property selections https://github.com/Pathlight/tap-purecloud/pull/17. I dont have an account to test it though, let me know if you have a chance to try it out. You should be able to just alter your pip_url in your meltano.yml to be

git+<https://github.com/pnadolny13/tap-purecloud.git@respect_catalog>

and run a

meltano install extractor tap-purecloud --clean

to reinstall

matt_neiman

05/19/2023, 5:45 PM

@pat_nadolny - Success! I'm able to run for selected streams after installing the updates. Thanks!! I was able to successfully extract and load using target-bigquery for the Users and User_State streams. The Conversations stream is generating an error. I'm attaching the debug log.

bq_convo.log

pat_nadolny

05/19/2023, 6:06 PM

@matt_neiman thats awesome! The error you're seeing is was a bug in my branch. If you run

meltano install extractor tap-purecloud --clean

again now it should re-install with my bugfix

matt_neiman

05/19/2023, 6:49 PM

@pat_nadolny - Yes! I was able to load the Conversations stream into BQ. Thanks for all the help!

Open in Slack

Previous Next