https://linen.dev logo
Join Slack
Powered by
Hey team, question about meltano 2.6-2.7 (I know, ...
# troubleshooting
s
Hey team, question about meltano 2.6-2.7 (I know, I'm way behind) I remember that there was a change to the database in meltano 2.6 - 2.7 that made it so that when all 3 of my meltano-running containers (ui, airflow-webserver, airflow-scheduler) pinged it at the same time, it would have some lockin and timeout. Would you mind specifying what these changes were, and how I could manually perform them if necessary (example by connecting to our meltano instance and manually running update db) Thanks!
w
Here's the changes between 
v2.6.0
and `v2.7.0`: https://github.com/meltano/meltano/compare/v2.6.0...v2.7.0 The upgrade function from https://github.com/meltano/meltano/blob/1a97f3d2f68a59e1a07c1d04672603e2ae1f0d64/s[…]tions/versions/f4c225a9492f_create_dedicated_job_state_table.py is used to upgrade the DB. @cody_hanson Later on you made changes to that file: https://github.com/meltano/meltano/commits/main/src/meltano/migrations/versions/f4c225a9492f_create_dedicated_job_state_table.py The current version has upgrade and downgrade as a no-op. That's only for 
v2.7.1
, right? Should the 
v2.7.0
version still be used?
s
@Will Da Silva (Arch) does this mean I should directly jump to meltano 
2.7.1
?
w
You should be able to upgrade directly to the latest version, and each change required to the system DB will be applied sequentially during the upgrade process. That said, I'm not sure what the best practice is around this. @cody_hanson or @pat_nadolny likely know better.
s
Yeah I just tried to update to 2.7 from 2.5 and I'm expriencing the same issue I had back then; 3 processes trying to update the db at the same time 😅
@Will Da Silva (Arch) Would there be a reason why 2 services couldnt be running meltano at the same time? I killed all my services except 1, upgraded it to meltano 2.13 (currently running), and then reintroduced our second service (airflow webserver). On the second deployment (with the scheduler and webserver), the integration won't deploy; it waits in limbo with no logs until the task is killed. Could both services be locking each other out? I don't believe this is a db update issue; our first integration seems to be running fine on meltano v2.13
Would it be possible for me to remove for me to simply remove the meltano database_uri to solve this issue?
c
@Will Da Silva (Arch) IIRC that no-op is just to accommodate users who upgraded during a short window between the initial release (which had a migration bug) and the rollback and patch release, so they wouldn't end up with an orphaned revision ID. I'm not aware of any existing unresolved issues with migrations when updating to the latest version.
@Stéphane Burwash I'm not totally sure I follow the context here. Do you have multiple meltano instances (UI, Airflow webserver, and Airflow scheduler) all pointed at the same backend DB? Are all three of those services upgraded to 2.13? If the db is upgraded to 2.13 but any of the services are running an older version, that would likely cause an issue.
s
Hey @cody_hanson thanks for answering! Basically, here is the context: • We had 3 services (tasks) running on ECS fargate (webserver, ui and scheduler), all through meltano • Upgrade from 2.5 to 2.7 would always timeout - services would emit no logs except "environment "prod" is active" • Thought it was a DB issue - To solve this (2 months later), I deleted the webserver and ui tasks from our task definition in order to perform the DB update only with the scheduler task • This worked with only the scheduler, but when I tried reintroducing the 2 other tasks 1 at a time (starting with webserver), the same interaction would come up - timeout after designated amount of time Any ideas?
(Side note - all 3 services would be running on the same image of meltano 2.13)
Update: When running only the scheduler, it deploys, but doesnt output anything; could our airflow configs be failing when above meltano 2.5? This is our current airflow setup
Copy code
plugins:
  orchestrators:
  - name: airflow
    pip_url: psycopg2-binary apache-airflow==2.3.2 --constraint <https://raw.githubusercontent.com/apache/airflow/constraints-2.3.2/constraints-${MELTANO__PYTHON_VERSION}.txt>
    config:
      webserver:
        expose_config: true
        hide_paused_dags_by_default: true

  files:
  - name: files-airflow
    variant: meltano
    pip_url: git+<https://github.com/meltano/files-airflow.git>
Open in Slack
PreviousNext
Powered by