Hi everyone, We have configured multiple plugins ...
# troubleshooting
z
Hi everyone, We have configured multiple plugins (the same tap) in meltano.yml by inheriting
inherit_from
the same tap and deployed Meltano app in container and successfully ran a pipeline. After removing
inherit_from
and configuring plugin separately (the same tap but with different name) in meltano.yml file, Meltano gives the following error. Reverting meltano.yml file to previous version doesn't help and gives the same error:
Copy code
ELT could not be completed: Cannot start extractor: Executable 'tap-exactonline' could not be found. Extractor 'Client1' may not havebeen installed yet using `meltano install extractor Client1`, or the executable name may be incorrect.
'tap-exactonline' - main tap name. 'Client1' - inherited tap name. Does anyone know or have experienced this problem? Please can anyone help on this issue? How we can fix this? Cc: @conno_kuyt
p
@zafar hey 👋 - can you provide your meltano.yml? I think that will help with debugging the problem
z
Hi @pat_nadolny
Copy code
version: 1
send_anonymous_usage_stats: false
project_id: 60140d5b-32f1-49b1-aa37-e7a48dd2ec43
plugins:
  extractors:
  - name: tap-exactonline
    namespace: tap_exactonline
    pip_url: <git+ssh://git@github.com/ebit-wise/tap-exactonline.git>
    executable: tap-exactonline
    capabilities:
    - catalog
    - discover
    - state
    settings:
    - name: username
      kind: string
    - name: password
      kind: password
    - name: exactonline_url
      kind: string
    - name: redirect_url
      kind: string
    - name: client_id
      kind: password
    - name: client_secret
      kind: password
    - name: verification_secret_key
      kind: password
  - name: Client1
    inherit_from: tap-exactonline
    config:
      username: <mailto:conno@ebitwise.nl|conno@ebitwise.nl>
      password: $TAP_EXACTONLINE_PASSWORD_CLIENT1
      exactonline_url: <http://start.exactonline.nl|start.exactonline.nl>
      redirect_url: <https://www.ebitwise.nl/>
      client_id: $TAP_EXACTONLINE_CLIENT_ID_CLIENT1
      client_secret: $TAP_EXACTONLINE_CLIENT_SECRET_CLIENT1
      verification_secret_key: $TAP_EXACTONLINE_VERIFICATION_SECRET_KEY_CLIENT1
  - name: Client2
    inherit_from: tap-exactonline
    config:
      username: <mailto:conno@ebitwise.nl|conno@ebitwise.nl>
      password: $TAP_EXACTONLINE_USERNAME_CLIENT2
      exactonline_url: <http://start.exactonline.nl|start.exactonline.nl>
      redirect_url: <https://www.ebitwise.nl/>
      client_id: $TAP_EXACTONLINE_CLIENT_ID_CLIENT2
      client_secret: $TAP_EXACTONLINE_CLIENT_SECRET_CLIENT2
      verification_secret_key: $TAP_EXACTONLINE_VERIFICATION_SECRET_KEY_CLIENT2
  loaders:
  - name: target-bigquery
    variant: adswerve
    pip_url: git+<https://github.com/adswerve/target-bigquery.git@0.11.3>
  orchestrators:
  - name: airflow
    pip_url: psycopg2 apache-airflow==1.10.13 --constraint <https://raw.githubusercontent.com/apache/airflow/constraints-1.10.13/constraints-3.8.txt>
  files:
  - name: airflow
    pip_url: git+<https://gitlab.com/meltano/files-airflow.git>
p
@zafar I dont see anything immediately obvious, have you tried reinstalling those two extractors? You can use
meltano install extractors Client2 tap-exactonline --clean
to clear out and reinstall
z
@pat_nadolny I ran
meltano install --clean
command on my local machine and it removed old virtual environment and created new one for extractors/loaders and it worked for me. But when I added the same command in Dockerfile but it didn't help.
Copy code
# Install all plugins into the `.meltano` directory
COPY ./meltano.yml .
RUN meltano install --clean
during docker image build, I see only these logs. Looks like it just installed the plugins (not re-installing). How can I fix it in containerized app? @pat_nadolny
Copy code
Step #0 - "build image": Installing 6 plugins...
Step #0 - "build image": Installing file bundle 'airflow'...
Step #0 - "build image": Installing orchestrator 'airflow'...
Step #0 - "build image": Installed file bundle 'airflow'
Step #0 - "build image": Run `meltano upgrade files` to update your project's 'airflow' files.
Step #0 - "build image": Installing loader 'target-bigquery'...
Step #0 - "build image": Installed loader 'target-bigquery'
Step #0 - "build image": Installing extractor 'Client2'...
Step #0 - "build image": Installed extractor 'Client2'
Step #0 - "build image": Installing extractor 'Client1'...
Step #0 - "build image": Installed extractor 'Client1'
Step #0 - "build image": Installing extractor 'tap-exactonline'...
Step #0 - "build image": Installed orchestrator 'airflow'
Step #0 - "build image": Installed extractor 'tap-exactonline'
Step #0 - "build image": Installed 6/6 plugins
p
@zafar hmm good to know its solved on your local, its just an image building problem then. I wonder if your image is being built from a cached version. Have you tried running with
--no-cache
or deleting the existing image off your local before building?
z
Yes, it was an image building problem. Thank you