I m running a forked version of tap mysql from within a Dags Meltano #troubleshooting

I’m running a forked version of tap-mysql from wit...

josh_lloyd

07/27/2023, 10:30 PM

I’m running a forked version of tap-mysql from within a Dagster job. The tap runs fine locally, but when I run it in my stage environment (ECS container) I get an error depending on how I configure it. The error I get is:

Cannot start plugin tap-mysql--workflows: [Errno 2] No such file or directory: '/opt/dagster/app/.meltano/run/tap-mysql--workflows/tap.properties.json'

josh_lloyd

07/27/2023, 10:37 PM

meltano.yml

config is as follows:

Copy code

- name: tap-mysql--bilbo
    namespace: tap_mysql
    pip_url: git+<https://github.com/Widen/pipelinewise-tap-mysql.git>
    executable: tap-mysql
    capabilities:
    - state
    - catalog
    - discover
    settings:
    - name: host
    - name: port
    - name: user
    - name: password
    - name: export_batch_rows
    - name: session_sqls
    config:
      export_batch_rows: 500000
      session_sqls:
      - SET @@session.max_execution_time=0
      - SET @@session.net_read_timeout=3600
      - SET @@session.net_write_timeout=3600
      - SET @@session.wait_timeout=28800
      - SET @@session.innodb_lock_wait_timeout=3600
    metadata:
      '*':
        replication-method: FULL_TABLE
  - name: tap-mysql--workflows
    inherit_from: tap-mysql--bilbo

josh_lloyd

07/27/2023, 10:43 PM

hmmm.. I’m starting to wonder if this is having to do with the fact that I’m running two taps in parallel on the same machine on different threads though they have different `state-id-suffix`’s

visch

07/28/2023, 12:48 PM

After a bunch of pain I highly recommend against

pymssql

and tell everyone to use

pyodbc

now there's a bunch of small bugs in pymssql that haven't been addressed like https://github.com/pymssql/pymssql/issues/695 To your point! Yes running multiple instances of the same tap can cause issues due to the properties file being located at the same place (they edit and delete the same file). I think a good fix for this would be a

uuid

similar to the

config

files in Meltano. The fix I go with is just run the jobs in seperate containers, or if you really need to run them at the same place use a different venv (inherit from)

visch

07/28/2023, 12:50 PM

I wonder if we have an issue in about the mutliple jobs at the same time thing in with Meltano, I'm not sure hmm

josh_lloyd

08/04/2023, 7:31 PM

yeah, to follow up on this, I did run them serially and it did work. Would like to see the multiple jobs at one time thing fixed at some point though, if indeed it can be.

visch

08/04/2023, 7:32 PM

Can you make an issue 😄

Open in Slack

Previous Next