https://linen.dev logo
Join Slack
Powered by
I’m trying to configure <https://github.com/isabel...
# plugins-general
t
I’m trying to configure tap-okta and getting some interesting errors in meltano
Copy code
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/timfrazer/Projects/meltano-projects/data-to-bigquery/.venv/lib/python3.8/site-packages/meltano/cli/__init__.py", line 45, in main
    cli(obj={"project": None})
  File "/Users/timfrazer/Projects/meltano-projects/data-to-bigquery/.venv/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/timfrazer/Projects/meltano-projects/data-to-bigquery/.venv/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/timfrazer/Projects/meltano-projects/data-to-bigquery/.venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/timfrazer/Projects/meltano-projects/data-to-bigquery/.venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/timfrazer/Projects/meltano-projects/data-to-bigquery/.venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/timfrazer/Projects/meltano-projects/data-to-bigquery/.venv/lib/python3.8/site-packages/meltano/cli/params.py", line 23, in decorate
    return func(*args, **kwargs)
  File "/Users/timfrazer/Projects/meltano-projects/data-to-bigquery/.venv/lib/python3.8/site-packages/meltano/cli/params.py", line 56, in decorate
    func(project, *args, **kwargs)
  File "/Users/timfrazer/Projects/meltano-projects/data-to-bigquery/.venv/lib/python3.8/site-packages/meltano/core/utils/__init__.py", line 72, in wrapper
    return run_async(func(*args, **kwargs))
  File "/Users/timfrazer/Projects/meltano-projects/data-to-bigquery/.venv/lib/python3.8/site-packages/meltano/core/utils/__init__.py", line 51, in run_async
    loop.run_until_complete(future)
  File "/Users/timfrazer/.pyenv/versions/3.8.11/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/Users/timfrazer/Projects/meltano-projects/data-to-bigquery/.venv/lib/python3.8/site-packages/meltano/cli/elt.py", line 138, in elt
    await _run_job(project, job, session, context_builder, force=force)
  File "/Users/timfrazer/Projects/meltano-projects/data-to-bigquery/.venv/lib/python3.8/site-packages/meltano/cli/elt.py", line 217, in _run_job
    await _run_elt(project, context_builder, output_logger)
  File "/Users/timfrazer/Projects/meltano-projects/data-to-bigquery/.venv/lib/python3.8/site-packages/meltano/cli/elt.py", line 251, in _run_elt
    raise CliError(f"ELT could not be completed: {err}") from err
meltano.cli.utils.CliError: ELT could not be completed: Extractor failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/timfrazer/Projects/meltano-projects/data-to-bigquery/.venv/lib/python3.8/site-packages/meltano/cli/__init__.py", line 53, in main
    raise CliError(str(err)) from err
meltano.cli.utils.CliError: ELT could not be completed: Extractor failed
ELT could not be completed: Extractor failed
when I do this 
meltano config tap-okta test
Copy code
meltano config tap-okta test                                                                                                                                      1 ↵
Plugin configuration is invalid
{"type": "SCHEMA", "stream": "users", "schema": {"type": "object", "properties": {"id": {"type": ["null", "string"]}, "status": {"type": ["null", "string"]}, "created": {"type": ["null", "string"], "format": "date-time"}, "activated": {"type": ["null", "string"]}, "statusChanged": {"type": ["null", "string"], "format": "date-time"}, "lastLogin": {"type": ["null", "string"], "format": "date-time"}, "lastUpdated": {"type": ["null", "string"], "format": "date-time"}, "passwordChanged": {"type": ["null", "string"], "format": "date-time"}, "profile": {"type": ["null", "object"], "properties": {"ssoid": {"type": ["null", "string"]}, "firstName": {"type": ["null", "string"]}, "lastName": {"type": ["null", "string"]}, "mobilePhone": {"type": ["null", "string"]}, "secondEmail": {"type": ["null", "string"]}, "grpList": {"type": ["null", "string"]}, "login": {"type": ["null", "string"]}, "email": {"type": ["null", "string"]}}}}}, "key_properties": ["id"]}
getting 
Plugin configuration is invalid
but not really pointing me towards what is invalid
p
hey @tim_frazer 👋. try 
meltano --log-level=debug config tap-okta test
and see if the logs tell you anything else. You might just be missing a required config key or something
t
yeah I was doing that - but not getting too much out of it - I’ll put some logging into the tap and see if find anything
v
From the stack trace it looks like there's a failure with the configuration of meltano. there's nothing from the related tap/target. If you could share your 
meltano.yml
that would be helpful potentially
t
Copy code
version: 1
send_anonymous_usage_stats: true
project_id: 253fc4ee-65ff-46dd-99d5-fbadaaf4009e
plugins:
  extractors:
  - name: tap-okta
    namespace: tap_okta
    pip_url: git+<https://github.com/isabella232/tap-okta.git>
    executable: tap-okta
    capabilities:
    - catalog
    - discover
    settings:
    - name: api_key
    - name: service_url
  loaders:
  - name: target-csv
    variant: hotgluexyz
    pip_url: git+<https://github.com/hotgluexyz/target-csv.git@0.3.3>
@visch thanks for taking a peek
v
hmm first glace this looks good, I'll run it on my end and see
This is an interesting one Ran 
meltano --log-level=debug elt tap-okta target-csv
with bogus okta creds to get an error, recieved a similar error message as you
p
what configs do you guys have set? I get 
Exception: Config is missing required keys: ['api_key', 'service_url']
with the yaml above but if I add those two required keys where I set 
testing
as my service_url I get a log message like 
requests.exceptions.MissingSchema: Invalid URL 'testingusers': No schema supplied. Perhaps you meant <http://testingusers>?
but it looks like the tap was running. @tim_frazer what else are you seeing in the logs?
t
the tap seems to run, but bails - I also noticed that the schema on the tap is out of date with what okta is doing today also. I started rewriting the hole thing with the meltano sdk last night - just stuck on the 
next_page_token
logic
at this rate, I’ll probably have a fully rewritten tap for okta before I debug the existing one
v
Next debug step is what the heck is going on in the tap so I ran 
meltano invoke tap-okta
, which outputs the schema propery (this is fine, the tap probably needs a catalog provided)
Copy code
meltano invoke tap-okta -d > catalog.json
meltano invoke tap-okta -c catalog.json
Copy code
visch@visch-ubuntu:~/git/sandbox/oktatest$ meltano invoke tap-okta -d > catalog.json
INFO Loading schemas
INFO End Date Time : 2021-11-24 09:14:10.289267
visch@visch-ubuntu:~/git/sandbox/oktatest$ meltano invoke tap-okta -c catalog.json
Traceback (most recent call last):
  File "/home/visch/git/sandbox/oktatest/.meltano/extractors/tap-okta/venv/bin/tap-okta", line 8, in <module>
    sys.exit(main())
  File "/home/visch/git/sandbox/oktatest/.meltano/extractors/tap-okta/venv/lib/python3.8/site-packages/tap_okta/__init__.py", line 158, in main
    args = singer.utils.parse_args(REQUIRED_CONFIG_KEYS)
  File "/home/visch/git/sandbox/oktatest/.meltano/extractors/tap-okta/venv/lib/python3.8/site-packages/singer/utils.py", line 178, in parse_args
    check_config(args.config, required_config_keys)
  File "/home/visch/git/sandbox/oktatest/.meltano/extractors/tap-okta/venv/lib/python3.8/site-packages/singer/utils.py", line 186, in check_config
    raise Exception("Config is missing required keys: {}".format(missing_keys))
Exception: Config is missing required keys: ['api_key', 'service_url']
I'd check on your config keys and run this command. What makes no sense to me is why when we run 
meltano elt tap-okta target-csv
we don't get this error message listed. I almost wonder if tap-okta is sending the error to stdout instead of stderr
Checking now
Nope the error
Copy code
Traceback (most recent call last):
  File "/home/visch/git/sandbox/oktatest/.meltano/extractors/tap-okta/venv/bin/tap-okta", line 8, in <module>
    sys.exit(main())
  File "/home/visch/git/sandbox/oktatest/.meltano/extractors/tap-okta/venv/lib/python3.8/site-packages/tap_okta/__init__.py", line 158, in main
    args = singer.utils.parse_args(REQUIRED_CONFIG_KEYS)
  File "/home/visch/git/sandbox/oktatest/.meltano/extractors/tap-okta/venv/lib/python3.8/site-packages/singer/utils.py", line 178, in parse_args
    check_config(args.config, required_config_keys)
  File "/home/visch/git/sandbox/oktatest/.meltano/extractors/tap-okta/venv/lib/python3.8/site-packages/singer/utils.py", line 186, in check_config
    raise Exception("Config is missing required keys: {}".format(missing_keys))
Exception: Config is missing required keys: ['api_key', 'service_url']
Is sent to stderr, so why meltano isn't showing it is a bit concerning to me
p
@visch I see that error message when I run in 
--log-level=debug
Copy code
, line 186, in check_config
tap-okta         | discovery |     raise Exception("Config is missing required keys: {}".format(missing_keys))
tap-okta         | discovery | Exception: Config is missing required keys: ['api_key', 'service_url']
tap-okta         | discovery |
v
hmm Maybe I"m missing it!
```tap-okta (out) | extractor | {"type": "SCHEMA", "stream": "users", "schema": {"type": "object", "properties": {"id": {"type": ["null", "string"]}, "status": {"type": ["null", "string"]}, "created": {"type": ["null", "string"], "format": "date-time"}, "activated": {"type": ["null", "string"]}, "statusChanged": {"type": ["null", "string"], "format": "date-time"}, "lastLogin": {"type": ["null", "string"], "format": "date-time"}, "lastUpdated": {"type": ["null", "string"], "format": "date-time"}, "passwordChanged": {"type": ["null", "string"], "format": "date-time"}, "profile": {"type": ["null", "object"], "properties": {"ssoid": {"type": ["null", "string"]}, "firstName": {"type": ["null", "string"]}, "lastName": {"type": ["null", "string"]}, "mobilePhone": {"type": ["null", "string"]}, "secondEmail": {"type": ["null", "string"]}, "grpList": {"type": ["null", "string"]}, "login": {"type": ["null", "string"]}, "email": {"type": ["null", "string"]}}}}}, "key_properties": ["id"]} target-csv | loader | INFO Sending version information to singer.io. To disable sending anonymous usage data, set the config parameter "disable_collection" to true meltano | elt | DEBUG Deleted configuration at /home/visch/git/sandbox/oktatest/.meltano/run/elt/2021-11-24T140819--tap-okta--target-csv/17d04601-1b38-427c-8bce-add105327ec2/target.18430268-6bd4-45a9-b6ca-de2ec36da36e.config.json meltano | elt | DEBUG Deleted configuration at /home/visch/git/sandbox/oktatest/.meltano/run/elt/2021-11-24T140819--tap-okta--target-csv/17d04601-1b38-427c-8bce-add105327ec2/tap.1d466423-3f43-48da-95e9-113e539e8c20.config.json meltano | elt | ERROR Extraction failed (1): (see above) meltano | elt | DEBUG ELT could not be completed: Extractor failed Traceback (most recent call last): File "/home/visch/.local/lib/python3.8/site-packages/meltano/cli/elt.py", line 242, in _run_elt await _run_extract_load(elt_context, output_logger) File "/home/visch/.local/lib/python3.8/site-packages/meltano/cli/elt.py", line 284, in _run_extract_load await singer_runner.run( File "/home/visch/.local/lib/python3.8/site-packages/meltano/core/runner/singer.py", line 224, in run await self.invoke( File "/home/visch/.local/lib/python3.8/site-packages/meltano/core/runner/singer.py", line 204, in invoke raise RunnerError("Extractor failed", {PluginType.EXTRACTORS: tap_code}) meltano.core.runner.RunnerError: Extractor failed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/visch/.local/lib/python3.8/site-packages/meltano/cli/elt.py", line 230, in _redirect_output yield File "/home/visch/.local/lib/python3.8/site-packages/meltano/cli/elt.py", line 251, in _run_elt raise CliError(f"ELT could not be completed: {err}") from err meltano.cli.utils.CliError: ELT could not be completed: Extractor failed meltano | elt | ELT could not be completed: Extractor failed [2021-11-24 090822,747] [1800|MainThread|meltano.cli.utils] [DEBUG] ELT could not be completed: Extractor failed Traceback (most recent call last): File "/home/visch/.local/lib/python3.8/site-packages/meltano/cli/elt.py", line 242, in _run_elt await _run_extract_load(elt_context, output_logger) File "/home/visch/.local/lib/python3.8/site-packages/meltano/cli/elt.py", line 284, in _run_extract_load await singer_runner.run( File "/home/visch/.local/lib/python3.8/site-packages/meltano/core/runner/singer.py", line 224, in run await self.invoke( File "/home/visch/.local/lib/python3.8/site-packages/meltano/core/runner/singer.py", line 204, in invoke raise RunnerError("Extractor failed", {PluginType.EXTRACTORS: tap_code}) meltano.core.runner.RunnerError: Extractor failed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/visch/.local/lib/python3.8/site-…
meltano --version meltano, version 1.85.0
p
and youre running 
meltano --log-level=debug elt tap-okta target-csv
?
v
Yes just updated, going to give you a full dump here
p
I reverted to that version and still get a proper log in mine 🤷
I'm not 100% sure how catalog/property caching works but maybe yours is using a cached catalog thats invalid and mine is still trying to generate a catalog
I get 
Exception: Config is missing required keys: ['api_key', 'service_url']
as part of discovery and yours doesnt look like its doing discovery? idk
v
okta meltano oddness.txt
okta meltano oddness.txt
Added select everything to meltano.yml still getting the same thing
p
It looks like the difference is that you have 
TAP_OKTA_SERVICE_URL
and 
TAP_OKTA_API_KEY
set in your ENV
v
Copy code
version: 1
send_anonymous_usage_stats: true
project_id: 253fc4ee-65ff-46dd-99d5-fbadaaf4009e
plugins:
  extractors:
  - name: tap-okta
    namespace: tap_okta
    pip_url: git+<https://github.com/isabella232/tap-okta.git>
    executable: tap-okta
    capabilities:
    - catalog
    - discover
    settings:
    - name: api_key
    - name: service_url
    config:
      api_key: abc
      service_url: <https://abc.com/>
    select:
    - '*.*'
  loaders:
  - name: target-csv
    variant: hotgluexyz
    pip_url: git+<https://github.com/hotgluexyz/target-csv.git@0.3.3>
curious!
Copy code
meltano invoke tap-okta
{"type": "SCHEMA", "stream": "users", "schema": {"type": "object", "properties": {"id": {"type": ["null", "string"]}, "status": {"type": ["null", "string"]}, "created": {"type": ["null", "string"], "format": "date-time"}, "activated": {"type": ["null", "string"]}, "statusChanged": {"type": ["null", "string"], "format": "date-time"}, "lastLogin": {"type": ["null", "string"], "format": "date-time"}, "lastUpdated": {"type": ["null", "string"], "format": "date-time"}, "passwordChanged": {"type": ["null", "string"], "format": "date-time"}, "profile": {"type": ["null", "object"], "properties": {"ssoid": {"type": ["null", "string"]}, "firstName": {"type": ["null", "string"]}, "lastName": {"type": ["null", "string"]}, "mobilePhone": {"type": ["null", "string"]}, "secondEmail": {"type": ["null", "string"]}, "grpList": {"type": ["null", "string"]}, "login": {"type": ["null", "string"]}, "email": {"type": ["null", "string"]}}}}}, "key_properties": ["id"]}
This doesn't fail
Does Meltano run this, not get records back and report an extractor failure?
Yes it all makes sense now
Copy code
meltano invoke tap-okta --catalog catalog.json
This is the proper command
So Okta is running successfully, but not returning any records (obviously from the bogus config information I put in) Okta should really fail when the config data doesn't allow you to connect to okta
Not an issue with Meltano at all
I do think Meltano could be more helpful for this case and say why it's saying the Extractor failed (Maybe okta is giving a non 0 response code)
Copy code
meltano invoke tap-okta --catalog catalog.json
{"type": "SCHEMA", "stream": "users", "schema": {"type": "object", "properties": {"id": {"type": ["null", "string"]}, "status": {"type": ["null", "string"]}, "created": {"type": ["null", "string"], "format": "date-time"}, "activated": {"type": ["null", "string"]}, "statusChanged": {"type": ["null", "string"], "format": "date-time"}, "lastLogin": {"type": ["null", "string"], "format": "date-time"}, "lastUpdated": {"type": ["null", "string"], "format": "date-time"}, "passwordChanged": {"type": ["null", "string"], "format": "date-time"}, "profile": {"type": ["null", "object"], "properties": {"ssoid": {"type": ["null", "string"]}, "firstName": {"type": ["null", "string"]}, "lastName": {"type": ["null", "string"]}, "mobilePhone": {"type": ["null", "string"]}, "secondEmail": {"type": ["null", "string"]}, "grpList": {"type": ["null", "string"]}, "login": {"type": ["null", "string"]}, "email": {"type": ["null", "string"]}}}}}, "key_properties": ["id"]}
(.venv) visch@visch-ubuntu:~/git/sandbox/oktatest$ echo $?
1
Okta does fail, just doesn't give you any helpful message as to why it's failing
p
ahhh that makes sense
the tap just exits if it cant connect
sounds like @tim_frazer is on his way to a new SDK tap though, the original is only 174 lines so it shouldnt be too bad. Let us know if you want it to live in MeltanoLabs when youre done!
v
Dove in deep already thought I"d offer a bit of help with the next page token @tim_frazer https://developer.okta.com/docs/reference/core-okta-api/#pagination To do this with the SDK it'll be a bit tough. I swear something posted an issue related to HATEOAS in the SDK, but I couldn't find it. My brain went to dynamically changing 
path
based on 
context
. The issue is to do this you kind of have to bastardize the path, context "auto fill" that happens. By doing something like
Copy code
@property
def path:
    return "{dynamicurl}"
And set the initial 
context.dynamicurl
equal to the first url, and update that based on the 
rel
somewhere like in 
post_process()
That would do the trick I think
t
so I got the next page token stuff and wrote this
Copy code
def get_next_page_token(
            self,
            response: requests.Response,
            previous_token: Optional[Any]
        ) -> Optional[Any]:
        """Return a token for identifying next page or None if no more pages."""
        response_links = requests.utils.parse_header_links(response.headers['Link'].rstrip('>').replace('>,<', ',<'))
        for link in response_links:
            if link['rel'] == 'next':
                next_page_token = link['url']
            else:
                next_page_token = None
        return next_page_token
gnarly little thing
I got data out of Okta and mostly streaming, just working on debugging the next page part still
Well 3 hours later - still stuck on the pagination - I feel like I’m not quite understanding the pass between
Copy code
"""REST client handling, including oktaStream base class."""

import requests
from pathlib import Path
from typing import Any, Dict, Optional, Union, List, Iterable

from memoization import cached

from singer_sdk.helpers.jsonpath import extract_jsonpath
from singer_sdk.streams import RESTStream
from singer_sdk.authenticators import APIKeyAuthenticator


SCHEMAS_DIR = Path(__file__).parent / Path("./schemas")


class oktaStream(RESTStream):
    """okta stream class."""
    limit: int = 10
    @property
    def url_base(self) -> str:
        """Return the API URL root, configurable via tap settings."""
        return self.config["api_url"]

    @property
    def authenticator(self) -> APIKeyAuthenticator:
        """Return a new authenticator object."""
        return APIKeyAuthenticator.create_for_stream(
            self,
            key="apikey",
            value=self.config.get("api_key"),
            location="header"
        )

    @property
    def http_headers(self) -> dict:
        """Return the http headers needed."""
        headers = {}
        headers["Authorization"] = "SSWS " + self.config.get("api_key")
        return headers

    def get_next_page_token(
            self,
            response: requests.Response,
            previous_token: Optional[Any]
        ) -> Optional[Any]:
        """Return a token for identifying next page or None if no more pages."""

        response_links = requests.utils.parse_header_links(response.headers['Link'].rstrip('>').replace('>,<', ',<'))
        for link in response_links:
            if link['rel'] == 'next':
                next_page_token = link['url']
            else:
                next_page_token = None
        return next_page_token


    def get_url_params(
        self,
        context: Optional[dict],
        next_page_token: Optional[Any]
    ) -> Dict[str, Any]:
        """Return a dictionary of values to be used in URL parameterization."""
        params: dict = {}
        params["limit"] = self.limit
        if next_page_token:
            params["page"] = next_page_token
        if self.replication_key:
            params["sort"] = "asc"
            params["order_by"] = self.replication_key
        return params
@pat_nadolny @visch keep running into 
RuntimeError: Loop detected in pagination. Pagination token
any thoughts would be most welcome - I feel like i’m 99% there just can’t seem to get past the loop. It’s like I’m sending out the next URL but then it gets back the original URL
of the next_token
p
@tim_frazer is it returning data, paginating, then eventually throwing that error? I'm wondering if the last iteration page is still returning a link
t
1st pass gets 
<https://staging-auth.xxx.com/api/v1/users?limit=10>
2nd pass gets 
<https://staging-auth.xxx.com/api/v1/users?after=200u13gd20mpR1CELg1d7&limit=10>
but then next_token seems to get stuck - I think there is gap in how I’m doing handling
p
I feel like maybe you should be overriding 
get_url
instead of 
get_next_page_token
since youre really retrieving the whole link from the header links
t
ah let me dig into that
v
can you push your code up? Happened to have some time at the aiport might be able to dive a bit
t
I was talking to @pat_nadolny he thinks I should or could overide this handler https://gitlab.com/meltano/sdk/-/blob/main/singer_sdk/streams/rest.py#L269
p
it sounds like the okta link is a cursor so the link doesnt actually change between requests which means the next page token loop wont really work. thats why i suggested maybe overriding the 
request_records
and 
get_url
methods to store the link and keep reusing it until its out of records
but until next week - happy thanksgiving 🦃
m
Hello @tim_frazer, Really helpful thread, i also got the same loop pagination error. I was able to get the data but my pagination is not working. Right now in certain limit of records store in csv file. Can you help me that or did you able to achieve pagination.
Thank you in advance.
t
Hey @monika_rajput we forked this repo and made improvements
@sean_glynn can we share?
m
It's great, if it's possible to share it would be great learning for me.
s
Hey @monika_rajput, I've forked the tap-okta repo from our private GH org. You will find the repo here: https://github.com/seanglynn-thrive/tap-okta The pagination logic happens in the request_records function within the oktaStream class There's a while loop that will keep iterating, parsing the next_page_token from the response until there are no more *next_page_token*'s. At that point it will break out of the loop and finish the stream process
m
Thank you @sean_glynn, but getting 404 error. May be it's a private repo.
My github username : https://github.com/monika0123
Hello @sean_glynn @tim_frazer, I was able to get the first batch of data but my pagination seems not working, Currently i am getting this below error.
Copy code
return self.scan_once(s, idx=_w(s, idx).end())
target-csv        | simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
tap-shopify       | Traceback (most recent call last):
tap-shopify       |   File "/Users/monika/Desktop/tap-shopify-users/.meltano/extractors/tap-shopify/venv/bin/tap-shopify", line 8, in <module>
tap-shopify       |     sys.exit(Tapshopify.cli())
tap-shopify       |   File "/Users/monika/Desktop/tap-shopify-users/.meltano/extractors/tap-shopify/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
tap-shopify       |     return self.main(*args, **kwargs)
tap-shopify       |   File "/Users/monika/Desktop/tap-shopify-users/.meltano/extractors/tap-shopify/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
tap-shopify       |     rv = self.invoke(ctx)
tap-shopify       |   File "/Users/monika/Desktop/tap-shopify-users/.meltano/extractors/tap-shopify/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
tap-shopify       |     return ctx.invoke(self.callback, **ctx.params)
tap-shopify       |   File "/Users/monika/Desktop/tap-shopify-users/.meltano/extractors/tap-shopify/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
tap-shopify       |     return __callback(*args, **kwargs)
tap-shopify       |   File "/Users/monika/Desktop/tap-shopify-users/.meltano/extractors/tap-shopify/venv/lib/python3.8/site-packages/singer_sdk/tap_base.py", line 499, in cli
tap-shopify       |     tap.sync_all()
tap-shopify       |   File "/Users/monika/Desktop/tap-shopify-users/.meltano/extractors/tap-shopify/venv/lib/python3.8/site-packages/singer_sdk/tap_base.py", line 379, in sync_all
tap-shopify       |     stream.sync()
tap-shopify       |   File "/Users/monika/Desktop/tap-shopify-users/.meltano/extractors/tap-shopify/venv/lib/python3.8/site-packages/singer_sdk/streams/core.py", line 1020, in sync
tap-shopify       |     self._sync_records(context)
tap-shopify       |   File "/Users/monika/Desktop/tap-shopify-users/.meltano/extractors/tap-shopify/venv/lib/python3.8/site-packages/singer_sdk/streams/core.py", line 946, in _sync_records
tap-shopify       |     for record_result in self.get_records(current_context):
tap-shopify       |   File "/Users/monika/Desktop/tap-shopify-users/.meltano/extractors/tap-shopify/venv/lib/python3.8/site-packages/singer_sdk/streams/rest.py", line 424, in get_records
tap-shopify       |     for record in self.request_records(context):
tap-shopify       |   File "/Users/monika/Desktop/tap-shopify-users/tap_shopify/client.py", line 111, in request_records
tap-shopify       |     resp = decorated_request(prepared_request, context)
tap-shopify       |   File "/Users/monika/Desktop/tap-shopify-users/.meltano/extractors/tap-shopify/venv/lib/python3.8/site-packages/backoff/_sync.py", line 94, in retry
tap-shopify       |     ret = target(*args, **kwargs)
tap-shopify       |   File "/Users/monika/Desktop/tap-shopify-users/.meltano/extractors/tap-shopify/venv/lib/python3.8/site-packages/singer_sdk/streams/rest.py", line 235, in _request
tap-shopify       |     self.validate_response(response)
tap-shopify       |   File "/Users/monika/Desktop/tap-shopify-users/.meltano/extractors/tap-shopify/venv/lib/python3.8/site-packages/singer_sdk/streams/rest.py", line 165, in validate_response
tap-shopify       |     raise FatalAPIError(msg)
tap-shopify       | singer_sdk.exceptions.FatalAPIError: 400 Client Error: Bad Request for path: /users.json
meltano           | DEBUG Deleted configuration at /Users/monika/Desktop/tap-shopify-users/.meltano/run/elt/2022-06-14T183046--tap-shopify--target-csv/8bb65e72-0e2f-4189-8499-982a3abd68af/target.002fca8f-ba7f-42cc-b617-c4f3203e6e45.config.json
t
this looks like tap-shopify?
s
Hey @monika_rajput I've the tap-okta lib I've mentioned above is now available here: https://github.com/seanglynn-thrive/tap-okta/tree/main I hope this helps
Regarding your error above, it seams that your 
users
stream is invalid. You should check that the schema defined in your 
users.json
conforms to the API's data model. I would recommend testing the API with a simple python requests script and parse the json response to ensure you are getting the expected object. Even a curl command or postman
m
@tim_frazer, Yes it is tap-shopify .
@sean_glynn, Thank you. I resolved that above error.
The code you shared me that was so helpful, i did some changes in the code and got the data of pages but when the code reached to the last page, it's not saving the response in the CSV file( as i am using target-csv loader to save the response. Like it's fetching the all records of last page but not saving it and the loader is failed. Above is the log of current error. ```tap-shopify (out) | {"type": "RECORD", "stream": "users", "record": {"id": 71903674423}, "time_extracted": "2022-07-04T075401.998432Z"} tap-shopify (out) | {"type": "RECORD", "stream": "users", "record": {"id": 71903707191}, "time_extracted": "2022-07-04T075401.998535Z"} tap-shopify (out) | {"type": "RECORD", "stream": "users", "record": {"id": 71903739959}, "time_extracted": "2022-07-04T075401.998641Z"} tap-shopify (out) | {"type": "RECORD", "stream": "users", "record": {"id": 71903838263}, "time_extracted": "2022-07-04T075401.998745Z"} tap-shopify (out) | {"type": "RECORD", "stream": "users", "record": {"id": 71906295863}, "time_extracted": "2022-07-04T075401.998848Z"} tap-shopify (out) | type of next_page_token <class 'str'> tap-shopify (out) | previous_token:.......... https://xxx.com/admin/api/2022-01/users.json?limit=250&amp;page_info=xxxxxxxx tap-shopify (out) | type of previous token <class 'str'> tap-shopify (out) | type of next_page_token 2 ......... <class 'NoneType'> tap-shopify (out) | next_page_token:....... None tap-shopify (out) | {"type": "STATE", "value": {"bookmarks": {"users": {}}}} tap-shopify | time=2022-07-04 132401 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'counter', 'metric': 'record_count', 'value': 385, 'tags': {'stream': 'users'}} meltano | DEBUG Deleted configuration at /Users/monika/Desktop/tap-shopify-users/.meltano/run/elt/2022-07-04T075354--tap-shopify--target-csv/7862caf2-9780-490f-8b0f-bfbf60f8682a/target.a3738f87-e712-4bc0-a389-2b9226754276.config.json meltano | DEBUG Deleted configuration at /Users/monika/Desktop/tap-shopify-users/.meltano/run/elt/2022-07-04T075354--tap-shopify--target-csv/7862caf2-9780-490f-8b0f-bfbf60f8682a/tap.11f6dbf7-d8d6-45c7-a7bd-316a03eeffb8.config.json meltano | ERROR Loading failed (1): simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0) meltano | DEBUG ELT could not be completed: Loader failed Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/meltano/cli/elt.py", line 239, in _run_elt await _run_extract_load(elt_context, output_logger) File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/meltano/cli/elt.py", line 277, in _run_extract_load await singer_runner.run( File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/meltano/core/runner/singer.py", line 224, in run await self.invoke( File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/meltano/core/runner/singer.py", line 206, in invoke raise RunnerError("Loader failed", {PluginType.LOADERS: target_code}) meltano.core.runner.RunnerError: Loader failed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/meltano/cli/elt.py", line 227, in _redirect_output yield File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/meltano/cli/elt.py", line 248, in _run_elt raise CliError(f"ELT could not be completed: {err}") from err meltano.cli.utils.CliError: ELT could not be completed: Loader failed meltano | time=2022-07-04 132359 name=tap-shopify level=INFO message=tap-shopify v0.0.1, Meltano SDK v0.4.9) meltano | time=2022-07-04 132359 name=tap-shopify level=INFO message=Skipping parse of env var settings... meltano | time=2022-07-04 132359 n…
4 Views
Open in Slack
PreviousNext
Powered by