https://linen.dev logo
Join Slack
Powered by
Hi Guys, I'm working on a custom extractor and I u...
# troubleshooting
j
Hi Guys, I'm working on a custom extractor and I used the SDK(that is great by the way) But now that I'm trying to use it on my project, the plugin is not installing and I can't find out why:
Copy code
meltano install extractor tap-smart-score-api-json-extractor                                                                
Installing 1 plugins...
Installing extractor 'tap-smart-score-api-json-extractor'...
Skipped installing 1/1 plugins
Is there a way that I can see the log of it?
v
Can you share your meltano.yml? And yes --log-level=debug
j
sure:
Copy code
version: 1                                                                                           
send_anonymous_usage_stats: false                                                                    
project_id: 1f34b85e-63c8-491f-a6c1-0846d508ebc6                                                     
plugins:                                                                                             
  extractors:                                                                                                                                       
  - name: tap-smart-score-api-json-extractor                                                         
    namespace: tap_smart_score_api_json_extractor                                                    
    executable: tap-smart-score-api-json-extractor/tap-smart-score-api-json-extractor.sh             
  - name: tap-csv                                                                                    
    variant: meltano                                                                                 
    pip_url: git+<https://gitlab.com/meltano/tap-csv.git>
Copy code
meltano --log-level=debug install extractor tap-smart-score-api-json-extractor                                              ruby-2.7.3 create-tap-smart-score-api-json-extractor b721e35 ✗
[2021-10-21 16:43:57,132] [1185144|MainThread|root] [DEBUG] Creating engine <meltano.core.project.Project object at 0x7fd6faf20760>@sqlite:////home/joaotorquato/cashu_projects/meltano-poc/first-project/.meltano/meltano.db
[2021-10-21 16:43:57,197] [1185144|MainThread|urllib3.connectionpool] [DEBUG] Starting new HTTPS connection (1): <http://www.meltano.com:443|www.meltano.com:443>
[2021-10-21 16:43:58,123] [1185144|MainThread|urllib3.connectionpool] [DEBUG] <https://www.meltano.com:443> "GET /discovery.yml HTTP/1.1" 200 138761
Installing 1 plugins...
[2021-10-21 16:43:58,963] [1185144|MainThread|asyncio] [DEBUG] Using selector: EpollSelector
Installing extractor 'tap-smart-score-api-json-extractor'...
Skipped installing 1/1 plugins
v
The reason is 
executable: tap-smart-score-api-json-extractor/tap-smart-score-api-json-extractor.sh
There's no install needed
I don't see very much documentation on this, but if you don't have a pip_url nothing will install
meltano invoke tap-smart-score-api-json-extractor
should just work
j
I got it but if i run independently, it works perfect. But when I invoke throught meltano I get: ```meltano elt tap-smart-score-api-json-extractor target-postgres ruby-2.7.3 create-tap-smart-score-api-json-extractor b721e35 ✗ meltano | Running extract & load... tap-smart-score-api-json-extractor | Installing dependencies from lock file tap-smart-score-api-json-extractor | tap-smart-score-api-json-extractor | Package operations: 9 installs, 0 updates, 0 removals tap-smart-score-api-json-extractor | tap-smart-score-api-json-extractor | • Installing cryptography (3.4.7) tap-smart-score-api-json-extractor | tap-smart-score-api-json-extractor | RuntimeError tap-smart-score-api-json-extractor | tap-smart-score-api-json-extractor | Retrieved digest for link cryptography-3.4.7.tar.gz(sha256:3d10de8116d25649631977cb37da6cbdd2d6fa0e0281d014a5b7d337255ca713) not in poetry.lock metadata ['sha256:a00cf305f07b26c351d8d4e1af84ad7501eca8a342dedf24a7acb0e7b7406e14', 'sha256:f44d141b8c4ea5eb4dbc9b3ad992d45580c1d22bf5e24363f2fbf50c2d7ae8a7', 'sha256:0a7dcbcd3f1913f664aca35d47c1331fce738d44ec34b7be8b9d332151b0b01e', 'sha256:34dae04a0dce5730d8eb7894eab617d8a70d0c97da76b905de9efb7128ad7085', 'sha256:1eb7bb0df6f6f583dd8e054689def236255161ebbcf62b226454ab9ec663746b', 'sha256:9965c46c674ba8cc572bc09a03f4c649292ee73e1b683adb1ce81e82e9a6a0fb', 'sha256:3c4129fc3fdc0fa8e40861b5ac0c673315b3c902bbdc05fc176764815b43dd1d', 'sha256:695104a9223a7239d155d7627ad912953b540929ef97ae0c34c7b8bf30857e89', 'sha256:21ca464b3a4b8d8e86ba0ee5045e103a1fcfac3b39319727bc0fc58c09c6aff7', 'sha256:3520667fda779eb788ea00080124875be18f2d8f0848ec00733c0ec3bb8219fc', 'sha256:d2a6e5ef66503da51d2110edf6c403dc6b494cc0082f85db12f54e9c5d4c3ec5', 'sha256:a305600e7a6b7b855cd798e00278161b681ad6e9b7eca94c721d5f588ab212af', 'sha256:3fa3a7ccf96e826affdf1a0a9432be74dc73423125c8f96a909e3835a5ef194a', 'sha256:d9ec0e67a14f9d1d48dd87a2531009a9b251c02ea42851c060b25c782516ff06', 'sha256:5b0fbfae7ff7febdb74b574055c7466da334a5371f253732d7e2e7525d570498', 'sha256:94fff993ee9bc1b2440d3b7243d488c6a3d9724cc2b09cdb297f6a886d040ef7', 'sha256:8695456444f277af73a4877db9fc979849cd3ee74c198d04fc0776ebc3db52b9', 'sha256:cd65b60cfe004790c795cc35f272e41a3df4631e2fb6b35aa7ac6ef2859d554e', 'sha256:94cc5ed4ceaefcbe5bf38c8fba6a21fc1d365bb8fb826ea1688e3370b2e24a1c'] tap-smart-score-api-json-extractor | tap-smart-score-api-json-extractor | at ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/installation/chooser.py:113 in _get_links tap-smart-score-api-json-extractor | 109│ tap-smart-score-api-json-extractor | 110│ selected_links.append(link) tap-smart-score-api-json-extractor | 111│ tap-smart-score-api-json-extractor | 112│ if links and not selected_links: tap-smart-score-api-json-extractor | → 113│ raise RuntimeError( tap-smart-score-api-json-extractor | 114│ "Retrieved digest for link {}({}) not in poetry.lock metadata {}".format( tap-smart-score-api-json-extractor | 115│ link.filename, h, hashes tap-smart-score-api-json-extractor | 116│ ) tap-smart-score-api-json-extractor | 117│ ) tap-smart-score-api-json-extractor | tap-smart-score-api-json-extractor | Traceback (most recent call last): tap-smart-score-api-json-extractor | File "<string>", line 1, in <module> tap-smart-score-api-json-extractor | File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module tap-smart-score-api-json-extractor | return _bootstrap._gcd_import(name[level:], package, level) tap-smart-score-api-json-extractor | File "<frozen importlib._bootstrap>", line 1014, in _gcd_import tap-smart-score-api-json-extractor | File "<frozen importlib._bootstrap>", line 991, in _find_and_load tap-smart-score-api-json-ext…
It looks like it didn't install the dependencies of the custom tap
v
When you say run it independently what do you mean? What's the command that works
j
I run the .sh from the sdk
v
Can you show the output of ./`tap-smart-score-api-json-extractor/tap-smart-score-api-json-extractor.sh`
I really mean ./tap-smart-score-api-json-extractor/tap-smart-score-api-json-extractor.sh is what meltano runs if you run 
meltano invoke tap-smart-score-api-json-extractor
, it'll run the same executable
j
sorry, when I run ./`tap-smart-score-api-json-extractor/tap-smart-score-api-json-extractor.sh` I get the same error. What i did to work was install the plugin with poetry and run it:
Copy code
☻  poetry install                                                                                                                
Installing dependencies from lock file

Package operations: 0 installs, 3 updates, 1 removal

  • Removing atomicwrites (1.4.0)
  • Updating jsonschema (2.6.0 -> 3.2.0)
  • Updating simplejson (3.17.5 -> 3.11.1)
  • Updating click (7.1.2 -> 8.0.3)

Installing the current project: tap-smart-score-api-json-extractor (0.0.1)
(.venv) 
~/cashu_projects/meltano-poc/first-project/tap-smart-score-api-json-extractor
☻  tap-smart-score-api-json-extractor                                                                                            
time=2021-10-21 17:14:04 name=tap-smart-score-api-json-extractor level=INFO message=tap-smart-score-api-json-extractor v0.0.1, Meltano SDK v0.3.11)
time=2021-10-21 17:14:04 name=tap-smart-score-api-json-extractor level=INFO message=Skipping parse of env var settings...
time=2021-10-21 17:14:04 name=tap-smart-score-api-json-extractor level=INFO message=Config validation passed with 0 errors and 0 warnings.
time=2021-10-21 17:14:04 name=root level=INFO message=Operator '{MAPPER_ELSE_OPTION}=None' was not found. Unmapped streams will be included in output.
time=2021-10-21 17:14:04 name=tap-smart-score-api-json-extractor level=INFO message=Beginning full_table sync of 'receivables'...
time=2021-10-21 17:14:04 name=tap-smart-score-api-json-extractor level=INFO message=Tap has custom mapper. Using 1 provided map(s).
{"type": "SCHEMA", "stream": "receivables", "schema": {"properties": {"recebiveis": {"items": {}, "type": "array"}}, "type": "object"}, "key_properties": ["num_documento"]}
{"type": "RECORD", "stream": "receivables", "record": {"recebiveis": [{"cedente": {"nome": "Ana L\u00edvia Bou\u00e7as"}, "sacado": {"cpf_cnpj": "07502097988"}, "nfe": {"num_documento": "36382406982644777831"}, "titulo": {"num_documento": "3788934"}}]}, "time_extracted": "2021-10-21T20:14:04.339518Z"}
time=2021-10-21 17:14:04 name=tap-smart-score-api-json-extractor level=INFO message=INFO METRIC: {'type': 'counter', 'metric': 'record_count', 'value': 1, 'tags': {'stream': 'receivables'}}
{"type": "STATE", "value": {"bookmarks": {"receivables": {}}}}
I think I must have installed poetry with pipx and meltano with pip
From what I got: Since I created a custom plugin in the same folder of meltano project and I am only setting the custom plugin executable path, it isn't installing all the dependencies from the plugin.
it is assuming that all the dependencies are already installed. Right?
v
Yes, I'd take a look at your 
tap-smart-score-api-json-extractor.sh
file itself Normally I'd run a job by running 
poetry run tap-smart-score-api-json-extractor
You could change the executable to 
tap-smart-score-api-json-extractor
but you'd have to run your install everytime
j
I definitely installed meltano and my custom plugin in different venv, when I changed the executable on yml file to 
tap-smart-score-api-json-extractor
it worked!
thanks @visch!
Open in Slack
PreviousNext
Powered by