https://linen.dev logo
Join Slack
Powered by
Weird error: I’ve just re-installed Meltano with a...
# troubleshooting
p
Weird error: I’ve just re-installed Meltano with a custom pip_url for a tap fork. I’m getting a pretty low-level failure running `meltano elt`:
Copy code
{
  "name": "meltano",
  "run_id": "00be09dc-0648-41ff-8550-cb5c464bd945",
  "job_id": "2022-03-01T230534--tap-mysql--incremental--target-bigquery",
  "stdio": "stderr",
  "cmd_type": "elt",
  "event": "ELT could not be completed: Cannot start extractor: Catalog discovery failed: command ['/Users/paul/bi-pipieline/.meltano/extractors/tap-mysql--incremental/venv/bin/tap-mysql', '--config', '/Users/paul/bi-pipieline/.meltano/run/elt/2022-03-01T230534--tap-mysql--incremental--target-bigquery/00be09dc-0648-41ff-8550-cb5c464bd945/tap.f4072651-ea5c-4e31-82d8-f390e9cbf1bc.config.json', '--discover'] returned 1",
  "level": "info",
  "timestamp": "2022-03-01T23:05:52.119644Z"
}
The 
/.meltano/run/elt/.../<uuid>.config.json
file just isn’t present after the run. I can’t tell if it’s just being deleted somewhere… The only thing I have different from my working config is:
Copy code
# pip_url: pipelinewise-tap-mysql
    pip_url: git+<https://github.com/paultiplady/pipelinewise-tap-mysql.git@ae3b94a6c30d0408c192edd79875ed45b87d1e27>
And inspecting the venv for the extractor, it looks like it’s loaded correctly (the tap code is there etc). I can run the tap manually from the extractor venv. Any idea where I should look to get to the bottom of this? I can see the PluginConfigService is creating non-config files. 
meltano elt --dump extractor-config
works (but the config files get deleted from the 
run
directory by something). I’m sure something in my (modified) tap is probably breaking things, but I have no idea how to debug this failure.
p
@paul_tiplady hey 👋 - this might be related to the 
executable
field. Can you share your meltano.yml? I wonder if its getting confused looking for pipelinewise-tap-mysql but the actual executable is called tap-mysql or similar
c
@paul_tiplady Most of the time when I see cryptic error messages relating to the extractor failing to start, it is due to 
MELTANO_ENVIRONMENT
being empty.
p
A bit delayed here, I have been on vacation. Thanks for the replies.
@pat_nadolny - I tried explicitly setting the 
executable
and it’s giving the same error. Based on the original log message I think it was already trying to use the correct executable — 
command ['/Users/paul/bi-pipieline/.meltano/extractors/tap-mysql--incremental/venv/bin/tap-mysql', '--config'
looks correct. My extractor config looks like this:
Copy code
plugins:
  extractors:
  - name: tap-mysql
    variant: transferwise
#    pip_url: pipelinewise-tap-mysql
    # TODO: Upstream this fix. WIP: <https://github.com/transferwise/pipelinewise-tap-mysql/pull/96>
    pip_url: git+<https://github.com/paultiplady/pipelinewise-tap-mysql.git@ae3b94a6c30d0408c192edd79875ed45b87d1e27>
    config:
      database: qwil
      host: $TAP_MYSQL_HOST
      user: $TAP_MYSQL_USER
      password: $TAP_MYSQL_PASSWORD
      session_sqls:
      - SET SESSION wait_timeout=28000
      - SET SESSION net_read_timeout=3600
      - SET SESSION net_write_timeout=3600
      - SET SESSION connect_timeout=3600
      - SET SESSION innodb_lock_wait_timeout=3600
    select: *id001

  - name: tap-mysql--incremental
    inherit_from: tap-mysql
    metadata: *id002

  - name: tap-mysql--full
    inherit_from: tap-mysql
    metadata: *id003
With a big select blob copied in. You can see the only changed bit of config commented out from my working version with the original tap.
@christoph I am not using environments at all — I don’t have the env set or the --environment flag set. I think this should be OK if I don’t have any env config in my meltano.yml file? Is that not the case? (Note that I do have a working config already, and the only difference is I’ve switched the tap git source to my fork).
c
I am not using environments at all
@paul_tiplady I think it's safe to ignore my comment then. 😄
Open in Slack
PreviousNext
Powered by