Hey guys getting this problem in all my meltnao sdk taps thi Meltano #troubleshooting

Hey guys, getting this problem in all my meltnao s...

Stéphane Burwash

03/30/2023, 1:23 PM

Hey guys, getting this problem in all my meltnao sdk taps this morning:

Copy code

cannot import name 'Sink' from 'singer_sdk.target_base'

What could be the cause of this?

visch

03/30/2023, 1:24 PM

I think there was an sdk release last night? probably tied to that, maybe pin your sdk version to one back and see if it works

visch

03/30/2023, 1:27 PM

If you could post the stack trace it'll probably help us track this down to see if it's an sdk issue or a target thing

Stéphane Burwash

03/30/2023, 1:30 PM

Copy code

[2023-03-30, 13:17:12 UTC] {subprocess.py:92} INFO - [2m2023-03-30T13:17:12.373885Z[0m [[32m[1minfo     [0m] [1m  File "/project/.meltano/loaders/target-bigquery/venv/bin/target-bigquery", line 5, in <module>[0m [36mcmd_type[0m=[35melb[0m [36mconsumer[0m=[35mTrue[0m [36mname[0m=[35mtarget-bigquery--wrike[0m [36mproducer[0m=[35mFalse[0m [36mstdio[0m=[35mstderr[0m [36mstring_id[0m=[35mtarget-bigquery--wrike[0m
[2023-03-30, 13:17:12 UTC] {subprocess.py:92} INFO - [2m2023-03-30T13:17:12.374077Z[0m [[32m[1minfo     [0m] [1m    from target_bigquery.target import TargetBigQuery[0m [36mcmd_type[0m=[35melb[0m [36mconsumer[0m=[35mTrue[0m [36mname[0m=[35mtarget-bigquery--wrike[0m [36mproducer[0m=[35mFalse[0m [36mstdio[0m=[35mstderr[0m [36mstring_id[0m=[35mtarget-bigquery--wrike[0m
[2023-03-30, 13:17:12 UTC] {subprocess.py:92} INFO - [2m2023-03-30T13:17:12.374251Z[0m [[32m[1minfo     [0m] [1m  File "/project/.meltano/loaders/target-bigquery/venv/lib/python3.8/site-packages/target_bigquery/target.py", line 18, in <module>[0m [36mcmd_type[0m=[35melb[0m [36mconsumer[0m=[35mTrue[0m [36mname[0m=[35mtarget-bigquery--wrike[0m [36mproducer[0m=[35mFalse[0m [36mstdio[0m=[35mstderr[0m [36mstring_id[0m=[35mtarget-bigquery--wrike[0m
[2023-03-30, 13:17:12 UTC] {subprocess.py:92} INFO - [2m2023-03-30T13:17:12.374421Z[0m [[32m[1minfo     [0m] [1m    from singer_sdk.target_base import Sink, Target[0m [36mcmd_type[0m=[35melb[0m [36mconsumer[0m=[35mTrue[0m [36mname[0m=[35mtarget-bigquery--wrike[0m [36mproducer[0m=[35mFalse[0m [36mstdio[0m=[35mstderr[0m [36mstring_id[0m=[35mtarget-bigquery--wrike[0m
[2023-03-30, 13:17:12 UTC] {subprocess.py:92} INFO - [2m2023-03-30T13:17:12.374593Z[0m [[32m[1minfo     [0m] [1mImportError: cannot import name 'Sink' from 'singer_sdk.target_base' (/project/.meltano/loaders/target-bigquery/venv/lib/python3.8/site-packages/singer_sdk/target_base.py)[0m [36mcmd_type[0m=[35melb[0m [36mconsumer[0m=[35mTrue[0m [36mname[0m=[35mtarget-bigquery--wrike[0m [36mproducer[0m=[35mFalse[0m [36mstdio[0m=[35mstderr[0m [36mstring_id[0m=[35mtarget-bigquery--wrike[0m
[2023-03-30, 13:17:12 UTC] {subprocess.py:92} INFO - [2m2023-03-30T13:17:12.440556Z[0m [[31m[1merror    [0m] [1mLoader failed

Stéphane Burwash

03/30/2023, 1:31 PM

Ouh that's gross, but it seems to be the target @alexander_butler

edgar_ramirez_mondragon

03/30/2023, 2:07 PM

Yeah, the target is importing that from the wrong module 😬. I'll send him a PR.

visch

03/30/2023, 2:08 PM

imo the target should probably be pinned to an sdk version as well!

edgar_ramirez_mondragon

03/30/2023, 2:08 PM

Yup, that too

visch

03/30/2023, 2:09 PM

@Stéphane Burwash reinstalling the targets every run is a bit dangerous due to dependency changes like this, if you had a container host these that would prevent this as well 🤷 Just an fyi!

Stéphane Burwash

03/30/2023, 2:10 PM

@visch we have it setup on a CI/CD pipeline in a docker image, so everytime we push to main we make a fresh

meltano install --clean

. I'll look into putting some checks on that, but I feel that still falls under good practices (if the versions of our taps are pinned)

visch

03/30/2023, 2:12 PM

That's a perfect setup, but it sounded like in production your job failed. If so then presumably the container isn't working? Maybe this just failed on a new MR or something on your end if so that's perfect! if it failed in an MR/PR that's a success story in my book!

Stéphane Burwash

03/30/2023, 2:32 PM

Yes no sadly it merged in prod; I have no tests curretnly before merging to see if a tap / target is functional, only on the dbt side. What kind of tests could I put in place to validate meltano? That would definitely help with my deployments 😄

visch

03/30/2023, 2:34 PM

If costs for tap/target data aren't a concern during your CI you can run

meltano --environment=ci run tap-name target-name

the environment could point you to a special target schema just for that CI environment. If you need to optimize more you can but some iteration of that!

visch

03/30/2023, 2:38 PM

https://meltano.com/data-and-analytics/why-isolated-test-environments-are-valuable/ what timing on this blog post 😄

christian_hollinger

03/30/2023, 2:44 PM

Newbie question: What’s the best way to fix this until the PR is merged (other than a fork)? Is there a setting in the

meltano.yml

to overwrite tap/target dependencies (for an earlier SDK version)?

Will Da Silva (Arch)

03/30/2023, 2:45 PM

You can specify an earlier version of the tap/target/plugin in the

pip_url

using the same syntax as you would for a

pip install

command.

Will Da Silva (Arch)

03/30/2023, 2:45 PM

You can also add additional dependencies to the

pip_url

, for instance to pin an earlier version of the SDK

edgar_ramirez_mondragon

03/30/2023, 2:46 PM

Yup, something like

Copy code

pip_url: git+<https://github.com/z3z1ma/target-bigquery.git> singer-sdk==0.22.0

christian_hollinger

03/30/2023, 2:46 PM

neat, ty!

Stéphane Burwash

03/30/2023, 2:47 PM

Great question @christian_hollinger, was wondering that myself 😅

edgar_ramirez_mondragon

03/30/2023, 4:17 PM

Change has landed in

main

so going back to

Copy code

pip_url: git+<https://github.com/z3z1ma/target-bigquery.git>

should work

alexander_butler

03/30/2023, 4:31 PM

Thanks for the PR. I will be pinning an exact version of the SDK moving forward. I will set up dependabot instead to open PRs where I can trigger, minimally, the unit tests. That should be solid. thankyou

aaronsteers

03/30/2023, 5:49 PM

Thanks, @alexander_butler. We've seen dependabot working well for us so far - and we also have a similar challenge efficiently maintaining SDK version bumps for the MeltanoLabs variants.

aaronsteers

03/30/2023, 5:57 PM

Consider explicit `__all__` declarations to reduce breakages on internal refactors · Issue #1565 · meltano/sdk

christian_hollinger

03/30/2023, 5:58 PM

Seeing this with the quote transformation enabled:

Copy code

Invalid field name "`search_name`". Fields must contain the allowed characters, and be at most 300 characters long.

Wondering if there’s another dependency that broke here. This doesn’t seem like normal behavior for BQ.

alexander_butler

03/30/2023, 6:10 PM

@christian_hollinger That looks unrelated to this thread. It could be worth popping an issue over on GH though? I would also say that the quote transformation is probably unnecessary since bq is case sensitive anyway IIRC. But in either case, the quote thing is very straightforward so not sure whats up. Would need more informtion

Copy code

>>> transform_column_name("search_name", quote=True)
'`search_name`'
>>> transform_column_name("`search_name`", quote=True)
'`search_name`'
>>>

christian_hollinger

03/30/2023, 10:54 PM

I’ll do you one better and give you a PR: https://github.com/z3z1ma/target-bigquery/pull/22 🙂 There are two remaining issues I briefly mentioned in that ticket, I believe I found some more rather obscure corner cases. But I guess that can be discussed on GH.

Open in Slack

Previous Next