https://linen.dev logo
Join Slack
Powered by
Hi, I am trying to install the latest Meltano vers...
# troubleshooting
r
Hi, I am trying to install the latest Meltano version for Python 3.8 and getting the current error message when running the image locally via Docker:
Copy code
docker run -it new-meltano-version invoke airflow scheduler         
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

This Meltano project is deployed as read-only
Can you point me to the right direction of how to start debugging this? Dockerfile in thread 🧵 ⬇️
Copy code
ARG MELTANO_IMAGE=meltano/meltano:v2.16.1-python3.8
FROM $MELTANO_IMAGE

WORKDIR /projects

# Install any additional requirements
COPY ./requirements.txt .
RUN pip install -r requirements.txt

# To fix the pg_config executable not found error
RUN apt-get update && \
    apt-get install libpq-dev -y && \
    apt-get clean;

# Install all plugins into the `.meltano` directory
COPY ./meltano.yml .
RUN meltano install
RUN meltano upgrade files

# Add Java JDK for
RUN apt-get update && \
    apt-get install -y openjdk-11-jdk && \
    apt-get clean;

# Pin `discovery.yml` manifest by copying cached version to project root
RUN cp -n .meltano/cache/discovery.yml . 2>/dev/null || :

# Don't allow changes to containerized project files
ENV MELTANO_PROJECT_READONLY 1

# Copy over remaining project files
COPY . .

# Expose default port used by `meltano ui`
EXPOSE 5000

ENTRYPOINT [ "meltano" ]
CMD [ "invoke", "airflow", "scheduler" ]
I get the same error when running:
Copy code
docker run -it new-meltano-version install
w
Because 
ENV MELTANO_PROJECT_READONLY 1
is in the Dockerfile, I wouldn't expect the 
install
command to work. What changes have been made to the Dockerfile between now and when it was last working? Was the only change the Meltano version bump?
r
Yes that was the only change! It is OK for it to not run locally, but when deploying it, I get the exact same error from k8s
Now I am just debugging with removing that line from the Dockerfile
w
There may be a bug here. Could you please run with 
MELTANO_PROJECT_READONLY
set to 
1
, with 
--log-level=debug
passed to 
meltano
?
r
Removing the line from the Dockerfile and running locally worked, as expected
Is there a specific Postgres version of the meltano and airflow-db I should choose for this version meltano:v2.16.1-python3.8 of Meltano to work?
Currently the error I get upon deploying it is: ```[error ] Traceback (most recent call last): │ │ File "/projects/.meltano/orchestrators/airflow/venv/bin/airflow", line 25, in <module> │ │ from airflow.configuration import conf │ │ File "/projects/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/__init__.py", line 31, in <module> │ │ from airflow.utils.log.logging_mixin import LoggingMixin │ │ File "/projects/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/utils/__init__.py", line 24, in <module> │ │ from .decorators import apply_defaults as _apply_defaults │ │ File "/projects/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/utils/decorators.py", line 36, in <module> │ │ from airflow import settings │ │ File "/projects/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/settings.py", line 38, in <module> │ │ from airflow.configuration import conf, AIRFLOW_HOME, WEBSERVER_CONFIG # NOQA F401 │ │ File "/projects/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/configuration.py", line 731, in <module> │ │ conf.read(AIRFLOW_CONFIG) │ │ File "/projects/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/configuration.py", line 421, in read │ │ self._validate() │ │ File "/projects/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/configuration.py", line 213, in _validate │ │ self._validate_config_dependencies() │ │ File "/projects/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/configuration.py", line 245, in _validate_config_dependencies │ │ raise AirflowConfigException( │ │ airflow.exceptions.AirflowConfigException: error: cannot use sqlite with the LocalExecutor │ │ │ │ Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to │ │ join our friendly Slack community. │ │ …
I have a CloudSQL instance vPostgreSQL 13 and have created two empty databases there, an airflow-db and a meltano-db
w
@pat_nadolny might know more about that than I
p
@rigerta that 
airflow.exceptions.AirflowConfigException: error: cannot use sqlite with the LocalExecutor
message makes it seem like your configuration for using the postgres database isnt working and its falling back to the default sqlite option. Is it possible to share your meltano.yml? What meltano version are you going from/to?
r
This is my meltano.yml file, we are thinking it might be an issue related to the airflow version we are installing:
Copy code
version: 1
send_anonymous_usage_stats: false
plugins:
  extractors:
  - name: tap-xactly
    namespace: tap_xactly
    pip_url: git+<https://gitlab.com/gitlab-data/meltano_taps.git#subdirectory=tap-xactly>
    executable: tap-xactly
    capabilities:
    - catalog
    - discover
    - state
    settings:
    - name: user
    - name: password
    - name: client_id
    - name: consumer
  loaders:
  - name: target-snowflake--edcast
    inherit_from: target-snowflake
    variant: meltano
    pip_url: git+<https://gitlab.com/gitlab-data/target-snowflake-edcast>
    config:
      account: gitlab
      database: RDEMIRI_PROD
      warehouse: LOADING
      role: MELTANO
      username: $TARGET_EDCAST_SNOWFLAKE_USERNAME
      password: $TARGET_EDCAST_SNOWFLAKE_PASSWORD
  orchestrators:
  - name: airflow
    pip_url: psycopg2 apache-airflow==1.10.13 --constraint <https://raw.githubusercontent.com/apache/airflow/constraints-1.10.13/constraints-3.8.txt>
  files:
  - name: airflow
    pip_url: git+<https://gitlab.com/meltano/files-airflow.git>
schedules:
- name: xactly-to-snowflake
  interval: 00 0,5 * * *
  extractor: tap-xactly
  loader: target-snowflake--edcast
  transform: skip
  start_date: 2021-07-13


project_id: 60140d5b-32f1-49b1-aa37-e7a48dd2ec43
meltano, version 1.104.0 was the prev one, but I am not doing an upgrade, I have started fresh with the meltano:v2.16.1-python3.8 version on a testing instance
so this is a clean installation
getting the same error also when changing the airflow version to:
Copy code
pip_url: apache-airflow==2.3.2 --constraint <https://raw.githubusercontent.com/apache/airflow/constraints-2.3.2/constraints-3.8.txt>
p
Do you know if you were setting the airflow database uri using environment variables? It seems like its not picking those up. This answer explains that you cant use LocalExecutor with sqlite.
It looks like in 2.3.0 they renamed the environment variable used for the database connection string https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#sql-alchemy-conn
r
Yes we have those new env variables set according to the new version in our deployment file:
Copy code
# Kubernetes Secrets :: Airflow
        - name: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
          valueFrom:
            secretKeyRef:
              name: airflow
              key: db_connection
        - name: AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT
          value: "120"
        - name: AIRFLOW__CORE__EXECUTOR
          value: "LocalExecutor"
However, currently I have a new error (debugging, fun 😄) :
Copy code
[error    ] Traceback (most recent call last):                                                                                                                                   │
│   File "/projects/.meltano/orchestrators/airflow/venv/bin/airflow", line 5, in <module>                                                                                                                      │
│     from airflow.__main__ import main                                                                                                                                                                        │
│   File "/projects/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/__init__.py", line 47, in <module>                                                                                 │
│     settings.initialize()                                                                                                                                                                                    │
│   File "/projects/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/settings.py", line 564, in initialize                                                                              │
│     configure_orm()                                                                                                                                                                                          │
│   File "/projects/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/settings.py", line 270, in configure_orm                                                                           │
│     engine = create_engine(SQL_ALCHEMY_CONN, connect_args=connect_args, **engine_args)                                                                                                                       │
│   File "<string>", line 2, in create_engine                                                                                                                                                                  │
│   File "/projects/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/sqlalchemy/util/deprecations.py", line 298, in warned                                                                      │
│     return fn(*args, **kwargs)                                                                                                                                                                               │
│   File "/projects/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/sqlalchemy/engine/create.py", line 548, in create_engine                                                                   │
│     dbapi = dialect_cls.dbapi(**dbapi_args)                                                                                                                                                                  │
│   File "/projects/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 753, in dbapi                                                            │
│     import psycopg2                                                                                                                                                                                          │
│ ModuleNotFoundError: No module named 'psycopg2'
so currently the step that is failing is whenever meltano is trying to install / add airflow as an orchestrator
Solved it, this thread was super useful - https://meltano.slack.com/archives/C01E3MUUB3J/p1635437273021600?thread_ts=1635437074.021500&amp;cid=C01E3MUUB3J Adding psycopg2 to the installation command before Airflow worked; however I am super unclear as to why is it needed there and it does not find psycopg although I had installed it via requirements.txt, in the dockerfile itself. Anyway! Thanks for your support thankyou, I appreciate the time you took to debug with me 🙂
2 Views
Open in Slack
PreviousNext
Powered by