I keep reading in different spots that I need to "*setup different dbs for airflow and meltano"* (ex: https://meltano.slack.com/archives/C01TCRBBJD7/p1625505400493200?thread_ts=1625502310.487400&cid=C01TCRBBJD7) I didn't setup our meltano project; could someone explain to me how I would go about doing that? I'm hoping this will fix a lot of my ghost airflow issues

@Stéphane Burwash Did you override the database_uri for Meltano or sqlalchemy_connection_uri (or something like that) for Airflow, in meltano.yml or through the environment? Or are they both using the default SQLite databases inside the .meltano directory?

Thanks for the answer @douwe_maan In local, we don't seem to do anything (besides run meltano ui, airflow scheduler and airflow webserver in 3 different containers) I prod, we have the same context in ECS (3 task definitions) + these variables are set (I can go check in secret manager what their value is if needs be) -AIRFLOW__CORE__EXECUTOR -AIRFLOW__CORE__SQL_ALCHEMY_CONN -MELTANO_DATABASE_URI Is this what you were talking about?

Yeah the last 2! Can you share their (sanitized) values?

They're different; guess my issue can be found elsewhere

What symptoms/errors are you seeing?

I think it's just a mixture of different airflow issues that could have many different causes. Currently, my main issue is getting the logs in production for the airflow webserver (since nothing from the actual run seems to appear in meltano-ui, airflow-webserver, airflow-scheduler ) My latest fix: expose port 8793 in my task definition for the scheduler, but that does not seem to have fixed it

Do the airflow webserver and scheduler have shared persistent storage?

No, only the webserver

Yeah I'd look into where the scheduler stores its execution logs, and whether webserver has access to that same location or will always find it empty because its not shared with the scheduler

Yeah I think this is as much linked to my lack of knowledge in all things infrastructure Because in my local machine it shows the logs perfectly while running in 3 seperate docker containers

Hmm, do you have a directory mounted into each container locally that you don't have in prod?

Our main dockerfile:

ARG MELTANO_IMAGE=meltano/meltano:latest
FROM $MELTANO_IMAGE

WORKDIR /project

# Install any additional requirements
COPY ./requirements.txt .
RUN pip install -r requirements.txt

# Install all plugins into the `.meltano` directory
COPY ./meltano.yml .
COPY ./extract ./extract
COPY ./load ./load
RUN meltano install --clean

# Pin `discovery.yml` manifest by copying cached version to project root
RUN cp -n .meltano/cache/discovery.yml . 2>/dev/null || :

# Don't allow changes to containerized project files
ENV MELTANO_PROJECT_READONLY 1

# Copy over remaining project files
COPY . .

# # Run command to generate dbt documentation
# RUN meltano invoke dbt docs generate

# Expose default port used by `meltano ui`
EXPOSE 5000

# Create airflow user
# RUN meltano invoke airflow users create \
#     --username potloc \
#     --firstname potloc \
#     --lastname potloc \
#     --role Admin \
#     --email <mailto:spiderman@superhero.org|spiderman@superhero.org> \
#     --password potloc

ENTRYPOINT ["meltano"]

Which we use in prod, with an added docker-compose (only local)

version: '3.8'

x-meltano-image: &meltano-image
  image: meltano-potloc:dev
  build: .
  volumes:
    - .:/project

services:
  meltano-ui:
    <<: *meltano-image
    command: ui
    env_file:
      - .env
    expose:
      - 5000
    ports:
      - 5000:5000
    restart: unless-stopped

  airflow-scheduler:
    <<: *meltano-image
    command: invoke airflow scheduler
    expose:
      - 8793
    ports:
      - 8793:8793
    restart: unless-stopped

  airflow-webserver:
    <<: *meltano-image
    command: invoke airflow webserver
    expose:
      - 8080
    ports:
      - 8080:8080
    restart: unless-stopped

  dbt-docs:
    <<: *meltano-image
    command: invoke dbt docs serve --port 8081
    expose:
      - 8081
    ports:
      - 8081:8081
    restart: unless-stopped

Okay so the Docker-compose file gives all the containers access to the same local project directory, which is where airflow also stores its logs.

We use an AWS code pipeline to automatically deploy them (please tell me if that doesn't answer you question)

That's definitely easiest. I'm not sure how to configure that with AWS though

All 3? Or do we want to keep meltano ui separate from the other 2?

Depends on if you also want to be able to see the run logs from Meltano UI. If you want to make this as similar to your local deployment as possible, you can create persistent storage and mount it into each of the containers at “/project/.meltano”, which is where all the supporting files like logs will be stored

Awesome thanks. When we run

meltano run tap target

and we get a stream of all the records being extracted, which service would be generating those logs?

Not sure what you mean, the logs are generated by “meltano run” but you already know that 😄

Hahaha yes 🤣 I'm just trying to figure out where they're stored in my production architecture in cloud watch. Because currently the answer seems to be "nowhere"

When you say you run

meltano run tap target

, is that managed by some automation or manually in an (ssh) terminal?

I mean using airflow scheduler (the default meltano architecture). We currently have 3 tasks (ui, webserver and scheduler) who all have logs, but none of them are about actual runs (from what I can see) So I'm trying to find where the output of runs would be stored (ex: when my dag fails, where would the "why" be)

Alright, so I could point them to the meltano-ui container or something to pick them up in cloudwatch

ah yeah, by mounting that entire subdirectory, the plug-in installation files that “meltano install” had put into the container are also gone. So mounting all of “.meltano” is not actually going to work. Can you try “.meltano/run”?

We're in buisness !!!!!