Hey I m trying to rewrite `tap stripe` using the Singer SDK Meltano #singer-tap-development

Hey! I’m trying to rewrite `tap-stripe` using the ...

prratek_ramchandani

04/14/2021, 7:10 PM

Hey! I’m trying to rewrite

tap-stripe

using the Singer SDK (because why not) and am running into some issues. I figured I’d choose the “custom” tap and auth types since I wanted to use Stripe’s client library, filled in all the TODOs, but when running

meltano elt tap-stripe target-zendesk

the target throws this error

Copy code

meltano      | Running extract & load...
target-jsonl | Traceback (most recent call last):
target-jsonl |   File "/Users/pramchandani/projects/tap_stripe/.meltano/loaders/target-jsonl/venv/bin/target-jsonl", line 8, in <module>
target-jsonl |     sys.exit(main())
target-jsonl |   File "/Users/pramchandani/projects/tap_stripe/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/target_jsonl.py", line 94, in main
target-jsonl |     state = persist_messages(input_messages, config.get('destination_path', ''), config.get('do_timestamp_file', True))
target-jsonl |   File "/Users/pramchandani/projects/tap_stripe/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/target_jsonl.py", line 47, in persist_messages
target-jsonl |     o = singer.parse_message(message).asdict()
target-jsonl |   File "/Users/pramchandani/projects/tap_stripe/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/singer/messages.py", line 156, in parse_message
target-jsonl |     obj = json.loads(msg)
target-jsonl |   File "/Users/pramchandani/projects/tap_stripe/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/simplejson/__init__.py", line 516, in loads
target-jsonl |     return _default_decoder.decode(s)
target-jsonl |   File "/Users/pramchandani/projects/tap_stripe/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/simplejson/decoder.py", line 370, in decode
target-jsonl |     obj, end = self.raw_decode(s)
target-jsonl |   File "/Users/pramchandani/projects/tap_stripe/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/simplejson/decoder.py", line 400, in raw_decode
target-jsonl |     return self.scan_once(s, idx=_w(s, idx).end())
target-jsonl | simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
meltano      | Loading failed (1): simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Not sure if this is relevant but I’m able to run

poetry run tap-stripe --config path/to/config.json

and see a mix of Singer logs and logs from the Stripe client library printed to stdout. Also,

meltano config tap-stripe

shows me the config I have set in meltano.yml but

meltano invoke tap-stripe

throws

Plugin 'tap-stripe' is not known to Meltano

Any thoughts?

aaronsteers

04/14/2021, 7:12 PM

As you call out, sounds like non-singer data is going to STDOUT. I think you can identify exactly what is in STDOUT (vs STDERR) by piping to a file and then examining the output. something like

poetry run tap-stripe --config path/to/config.json > dryrun.jsonl

douwe_maan

04/14/2021, 7:13 PM

In Meltano, running

meltano --log-level=debug elt <tap> <target>

will also print tap output with a

tap (out)

prefix. Like AJ says, it looks like there's non-Singer stuff in there that the target is stumbling over

aaronsteers

04/14/2021, 7:13 PM

Can you let me know any lines in that output that do not start with

as the first character? (Minus any PII please. 😇 )

prratek_ramchandani

04/14/2021, 7:43 PM

yeah sounds like that’s what is happening. here’s an example

Copy code

time=2021-04-14 15:08:20 name=stripe level=INFO message=message='Request to Stripe api' method=get path=<https://api.stripe.com/v1/events?type=customer.subscription.%2A&created[gte]=1618012800&limit=100&starting_after=evt_1IgD8fI8OBWPiWBSEGmIaBvA>

maybe i can change the log level for for the client library as a workaround?

aaronsteers

04/14/2021, 7:46 PM

Changing the log level may solve your issue. It still doesn’t explain though why the logs are going to STDOUT instead of STDERR. It probably makes sense to go ahead and change the log level, but I’d be interested to know if there’s anything we can recommend to other developers in the case they run into something similar in the future.

prratek_ramchandani

04/14/2021, 7:48 PM

actually when i pipe the output of

poetry run …

to a file i don’t see any of those logs, so maybe they are going to STDERR. everything written to the file looks like well-formatted Singer logs starting with a

aaronsteers

04/14/2021, 7:54 PM

@prratek_ramchandani - Looking more closely at the logs, the exact json parsing error is ‘expecting value’ which makes me wonder if a blank line is being emitted.

aaronsteers

04/14/2021, 7:55 PM

Could you check to see if that output contains any blank lines?

prratek_ramchandani

04/14/2021, 7:57 PM

i let it run for a few seconds in which time it wrote about 800 lines, none of which were blank. is the error message suggesting the first line is blank? because that one is

Copy code

{"type": "SCHEMA", "stream": "subscriptions", "schema": {...}}

aaronsteers

04/14/2021, 7:59 PM

Thanks for confirming. No, I don’t think the error should be read as indicating the first line of the file. The “line 1” reference is within that specific call, wherein each line is getting parsed individually.

aaronsteers

04/14/2021, 8:00 PM

It might be worth catting that file to the target directly. I’m not sure what else to check…

prratek_ramchandani

04/14/2021, 8:15 PM

okay i think it was just a misleading error. did ^that and it looks like the record was missing a property required by the jsonschema

prratek_ramchandani

04/14/2021, 9:08 PM

oof okay @aaronsteers i was wrong. i managed to get to a place where first piping stdout from the tap to a jsonl file and then catting that to the target succeeds but

meltano elt …

throws the same error 🤔

aaronsteers

04/14/2021, 10:36 PM

@prratek_ramchandani - Thanks for the update! Very puzzling indeed! Does anything interesting come from

--log-level=debug

as described here? If you want to open an issue for us with the full debug log output here, that could help us figure out what’s going on.

aaronsteers

04/14/2021, 10:36 PM

@douwe_maan fyi 👆

prratek_ramchandani

04/14/2021, 10:43 PM

this one line stood out - creating the issue with the full debug log soon

Copy code

tap-stripe (out)   | The virtual environment found in /Users/pramchandani/projects/tap_stripe/.meltano/extractors/tap-stripe/venv seems to be broken.

douwe_maan

04/14/2021, 10:43 PM

I think that's it!

douwe_maan

04/14/2021, 10:43 PM

That line does not look like JSON 🙂

douwe_maan

04/14/2021, 10:44 PM

https://stackoverflow.com/questions/61351659/the-virtual-environment-found-seems-to-be-broken-python-poetry

douwe_maan

04/14/2021, 10:45 PM

Can you reinstall the tap?

meltano install extractor tap-stripe

and see if that fixes it?

douwe_maan

04/14/2021, 10:46 PM

Is it possible that you originally installed tap-stripe using a different Python version than the one you're using now to run

meltano elt

prratek_ramchandani

04/14/2021, 10:46 PM

i reinstalled a few times already. btw it’s followed immediately by

Copy code

tap-stripe (out)   | Recreating virtualenv tap-stripe-k2j-jsmB-py3.8 in /Users/pramchandani/Library/Caches/pypoetry/virtualenvs/tap-stripe-k2j-jsmB-py3.8

douwe_maan

04/14/2021, 10:48 PM

That's interesting, it sounds like poetry is messing things up somehow...

douwe_maan

04/14/2021, 10:48 PM

Can you share your

meltano.yml

definition for

tap-stripe

aaronsteers

04/14/2021, 10:50 PM

I was going to say the same. Smells like a poetry / virtualenv issue. Feels like we’re getting closer though. 🙂

prratek_ramchandani

04/14/2021, 10:51 PM

here you go

Untitled

aaronsteers

04/14/2021, 10:52 PM

@prratek_ramchandani - Thanks. @douwe_maan - Looks like this is using my newly added shell script.

douwe_maan

04/14/2021, 10:52 PM

OK, I think

/Users/pramchandani/projects/tap_stripe/tap-stripe.sh

is not quite doing the right thing then. Can you share the contents? Did you edit anything compared to the template?

aaronsteers

04/14/2021, 10:53 PM

cookiecutter/tap-template/{{cookiecutter.library_name}}/{{cookiecutter.tap_id}}.sh · main · meltano / Singer SDK · GitLab

prratek_ramchandani

04/14/2021, 10:53 PM

no i left it untouched

Untitled

aaronsteers

04/14/2021, 10:53 PM

@prratek_ramchandani - Do you also have tap-stripe installed? In that case, there may be a conflict between versions and what is top of path.

douwe_maan

04/14/2021, 10:54 PM

I think we should: • Redirect all

poetry install

stdout output to stderr, so that its output isn't interpreted as Singer tap output • Figure out why it would be saying "The virtual environment found in ... seems to be broken."

prratek_ramchandani

04/14/2021, 10:55 PM

i don’t think i have it installed in this project but the log for

meltano install …

seems to indicate that might be the case?

Untitled

prratek_ramchandani

04/14/2021, 10:57 PM

umm…so i just realized

.meltano/extractors/tap-stripe/venv

is empty! that doesn’t seem like a good thing

douwe_maan

04/14/2021, 10:58 PM

That's expected if you didn't provide a

pip_url

. Since you provided an absolute path to

executable

, Meltano shouldn't need to use the venv.

douwe_maan

04/14/2021, 10:58 PM

If it is anyway, that's surprising

douwe_maan

04/14/2021, 10:59 PM

In the

meltano --log-level=debug ...

output, can you share the

Invoking

line that should have the full path to the executable being invoked?

prratek_ramchandani

04/14/2021, 11:00 PM

Copy code

meltano            | DEBUG Invoking: ['/Users/pramchandani/projects/tap_stripe/tap-stripe.sh', '--config', '/Users/pramchandani/projects/tap_stripe/.meltano/run/elt/2021-04-14T223716--tap-stripe--target-jsonl/059ebbb9-afe0-4706-bdb4-248e9b9d75d3/tap.config.json']

douwe_maan

04/14/2021, 11:00 PM

OK, there's no

.meltano/extractors/tap-stripe/venv

in there. Are you still seeing that path in the

tap-stripe (out)   | The virtual environment found in ...

line?

prratek_ramchandani

04/14/2021, 11:01 PM

yeah the path in that line is

/Users/pramchandani/projects/tap_stripe/.meltano/extractors/tap-stripe/venv

douwe_maan

04/14/2021, 11:07 PM

I think I've got it! I was wondering how that path is even making it into the executable, and I think it's coming from here: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin_invoker.py#L153

douwe_maan

04/14/2021, 11:08 PM

So all we need to do (hopefully) is to unset that

VIRTUAL_ENV

env var before running

poetry install

, so that it won't think it's running in that venv by adding

unset VIRTUAL_ENV

ahead of that line. It's still very possible, though, that that will cause

poetry

to fail to run if it is itself installed inside a venv that it won't be able to locate because VIRTUAL_ENV is now unset... But at least we'll be one step closer

douwe_maan

04/14/2021, 11:08 PM

The real solution will be to not have Meltano set

VIRTUAL_ENV

if it detects that no

pip_url

is set, as I suggested in https://gitlab.com/meltano/meltano/-/issues/2341#note_550791385

douwe_maan

04/14/2021, 11:09 PM

@prratek_ramchandani Can you try that, unsetting the env var from the shell script using

unset VIRTUAL_ENV

prratek_ramchandani

04/14/2021, 11:14 PM

okay that got rid of the broken env error but not the

simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

😕

douwe_maan

04/14/2021, 11:14 PM

That suggests something non-JSON is still making it to tap's stdout. Any more suspect lines with the

tap-stripe (out)

prefix?

aaronsteers

04/14/2021, 11:15 PM

WIP MR happening in sync with our debug session: Draft: Resolve "Virtual environments conflicting when running in combination with Meltano" (!64) · Merge requests · meltano / Singer SDK · GitLab

prratek_ramchandani

04/14/2021, 11:15 PM

no, this is the only line now with a

tap-stripe (out)

prefix

Copy code

tap-stripe (out)   | Installing dependencies from lock file

aaronsteers

04/14/2021, 11:16 PM

@prratek_ramchandani - Can you remove the

poetry install

line or replace with

poetry install 2>&1

douwe_maan

04/14/2021, 11:16 PM

@aaronsteers That's stderr to stdout, you want stdout to stderr:

1>&2

douwe_maan

04/14/2021, 11:17 PM

Stdout is for tap Singer output, Stderr is for logging 🙂

aaronsteers

04/14/2021, 11:17 PM

Thanks! I just always get the integers mixed up in the redirect. 🤦‍♂️

prratek_ramchandani

04/14/2021, 11:19 PM

that did it!

prratek_ramchandani

04/14/2021, 11:19 PM

it appears to be running successfully now

douwe_maan

04/14/2021, 11:19 PM

Amazing!

prratek_ramchandani

04/14/2021, 11:20 PM

@aaronsteers @douwe_maan thanks so much for jumping on to debug

douwe_maan

04/14/2021, 11:20 PM

Thanks for helping us fix our SDK script! 😄

aaronsteers

04/14/2021, 11:20 PM

Fantastic! 🙌 Thanks, @douwe_maan! I’ve got this MR ready to go - thanks very much @prratek_ramchandani!

3 Views

Open in Slack

Previous Next