https://linen.dev logo
Join Slack
Powered by
I’m running into an interesting issue trying to se...
# troubleshooting
n
I’m running into an interesting issue trying to set up a docker-compose for more production-oriented meltano use. I’d like to be able to run separate jobs with different 
--select
args on the same tap (to allow some tables to get updated more often than others). As far as I can tell, the meltano schedule options don’t support this, so I need to use airflow directly. I was able to set up the custom DAG, start the airflow scheduled and webserver locally, and have everything run just fine. I then followed the instructions here, but when I bring everything up, I get 
[Errno 2] No such file or directory: '/project/.meltano/run/airflow/airflow.cfg'
. Oddly, when I look in the equivalent directory in my working non-containerized example, I also don’t find that file. Any ideas what I might be doing wrong here?
The only apparently similar thing I’ve come across so far is this issue, which describes similar symptoms but I suspect isn’t likely related because it’s older https://gitlab.com/meltano/meltano/-/issues/2113
d
I’d like to be able to run separate jobs with different 
--select
 args on the same tap (to allow some tables to get updated more often than others). As far as I can tell, the meltano schedule options don’t support this, so I need to use airflow directly
@nick_hamlin Have you considered using inheritance to create different plugins with their own 
select:
or 
select_filter:
definition, so that you can reference each inheriting plugin individually by name from 
meltano schedule
and the 
schedules
list in 
meltano.yml
? That's the official recommended solution:
Copy code
plugins:
  extractors:
  - name: tap-foo--my_stream
    inherit_from: tap-foo
    select_filter:
    - my_stream
schedules:
  - name: my_stream
    extractor: tap-foo--my_stream
    # ...
Another option is to continue to use the same plugin, but to set the 
--select
using the 
<EXTRACTOR>__SELECT_FILTER
env var (https://meltano.com/docs/plugins.html#select-filter-extra) in the 
env
dict under 
schedules
(https://meltano.com/docs/integration.html#pipeline-specific-configuration):
Copy code
schedules:
- name: ...
  env:
    TAP_FOO__SELECT_FILTER: '["my_stream"]'
I then followed the instructions here, but when I bring everything up, I get 
[Errno 2] No such file or directory: '/project/.meltano/run/airflow/airflow.cfg'
.
Are you still invoking airflow through 
meltano invoke airflow
? That command creates the 
airflow.cfg
on the fly based on the Airflow config Meltano stores: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin/airflow.py#L74 Can you change the command to 
meltano --log-level=debug invoke airflow ...
, so that we can see if that debug log message shows up as expected, with the correct path?
Oddly, when I look in the equivalent directory in my working non-containerized example, I also don’t find that file. Any ideas what I might be doing wrong here?
It's automatically deleted when 
meltano invoke airflow
finishes, so it's expected that you wouldn't find it there: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin/airflow.py#L116
n
Got it - yes, I ran 
meltano invoke airflow scheduler
and 
meltano invoke airflow webserver
in my local testing, and the docker-compose is doing basically the same thing
d
All right. I'd be curious to see the result of this:
Can you change the command to 
meltano --log-level=debug invoke airflow ...
 , so that we can see if that debug log message shows up as expected, with the correct path?
And more complete output logs that include 
[Errno 2] No such file or directory: '/project/.meltano/run/airflow/airflow.cfg'
. I wonder if that error is coming from Meltano or Airflow
n
cool - let me tweak the commands in docker-compose to alter the log level
ok, confirmed that the error is coming from meltano, here’s the relevant chunk of debug level logs:
Copy code
airflow-scheduler_1  | [2021-05-05 15:23:51,056] [1|MainThread|meltano.cli.utils] [DEBUG] [Errno 2] No such file or directory: '/project/.meltano/run/airflow/airflow.cfg'
airflow-scheduler_1  | Traceback (most recent call last):
airflow-scheduler_1  |   File "/usr/local/lib/python3.6/site-packages/meltano/core/plugin_invoker.py", line 220, in _invoke
airflow-scheduler_1  |     yield (popen_args, popen_options, popen_env)
airflow-scheduler_1  |   File "/usr/local/lib/python3.6/site-packages/meltano/core/plugin_invoker.py", line 228, in invoke
airflow-scheduler_1  |     return subprocess.Popen(popen_args, **popen_options, env=popen_env)
airflow-scheduler_1  |   File "/usr/local/lib/python3.6/subprocess.py", line 729, in __init__
airflow-scheduler_1  |     restore_signals, start_new_session)
airflow-scheduler_1  |   File "/usr/local/lib/python3.6/subprocess.py", line 1364, in _execute_child
airflow-scheduler_1  |     raise child_exception_type(errno_num, err_msg, err_filename)
airflow-scheduler_1  | FileNotFoundError: [Errno 2] No such file or directory: '/project/.meltano/orchestrators/airflow/venv/bin/airflow': '/project/.meltano/orchestrators/airflow/venv/bin/airflow'
airflow-scheduler_1  | 
airflow-scheduler_1  | The above exception was the direct cause of the following exception:
airflow-scheduler_1  | 
airflow-scheduler_1  | Traceback (most recent call last):
airflow-scheduler_1  |   File "/usr/local/lib/python3.6/site-packages/meltano/core/plugin_invoker.py", line 157, in prepared
airflow-scheduler_1  |     self.prepare(session)
airflow-scheduler_1  |   File "/usr/local/lib/python3.6/site-packages/meltano/core/plugin_invoker.py", line 141, in prepare
airflow-scheduler_1  |     with self.plugin.trigger_hooks("configure", self, session):
airflow-scheduler_1  |   File "/usr/local/lib/python3.6/contextlib.py", line 81, in __enter__
airflow-scheduler_1  |     return next(self.gen)
airflow-scheduler_1  |   File "/usr/local/lib/python3.6/site-packages/meltano/core/behavior/hookable.py", line 70, in trigger_hooks
airflow-scheduler_1  |     self.__class__.trigger(self, f"before_{hook_name}", *args, **kwargs)
airflow-scheduler_1  |   File "/usr/local/lib/python3.6/site-packages/meltano/core/behavior/hookable.py", line 97, in trigger
airflow-scheduler_1  |     raise err
airflow-scheduler_1  |   File "/usr/local/lib/python3.6/site-packages/meltano/core/behavior/hookable.py", line 89, in trigger
airflow-scheduler_1  |     hook_func(target, *args, **kwargs)
airflow-scheduler_1  |   File "/usr/local/lib/python3.6/site-packages/meltano/core/plugin/airflow.py", line 52, in before_configure
airflow-scheduler_1  |     stderr=subprocess.DEVNULL,
airflow-scheduler_1  |   File "/usr/local/lib/python3.6/site-packages/meltano/core/plugin_invoker.py", line 228, in invoke
airflow-scheduler_1  |     return subprocess.Popen(popen_args, **popen_options, env=popen_env)
airflow-scheduler_1  |   File "/usr/local/lib/python3.6/contextlib.py", line 99, in __exit__
airflow-scheduler_1  |     self.gen.throw(type, value, traceback)
airflow-scheduler_1  |   File "/usr/local/lib/python3.6/site-packages/meltano/core/plugin_invoker.py", line 224, in _invoke
airflow-scheduler_1  |     ) from err
airflow-scheduler_1  | meltano.core.plugin_invoker.ExecutableNotFoundError: Executable 'airflow' could not be found. Orchestrator 'airflow' may not have been installed yet using `meltano install orchestrator airflow`, or the executable name may be incorrect.
That last line is a new one that I haven’t seen previously - I did a fresh build of the container prior to starting this and confirmed that airflow did install (at least, the build command said that it did and didn’t return any errors)
and if it’s helpful, here’s the entire docker-compose (based on the standard meltano boilerplate for this)
Copy code
version: '3.8'

x-meltano-image: &meltano-image
  image: globalgiving/meltano:latest
  volumes:
    - .:/project

services:
  meltano-ui:
    <<: *meltano-image
    command: ui
    expose:
      - 5000
    ports:
      - 5000:5000
    restart: unless-stopped

  # Uncomment if you are using the Airflow orchestrator, delete otherwise
  airflow-scheduler:
    <<: *meltano-image
    command: --log-level=debug invoke airflow scheduler
    expose:
      - 8793
    restart: unless-stopped
  
  airflow-webserver:
    <<: *meltano-image
    command: --log-level=debug invoke airflow webserver
    expose:
      - 8080
    ports:
      - 8080:8080
    restart: unless-stopped
d
@nick_hamlin All right. Is 
globalgiving/meltano
a containerized Docker image for the Meltano project in question, or just your fork of 
meltano/meltano
? In the former case, you shouldn't need to mount the project volume, since it should already be baked into the container
So I'd try that first, not mounting the project dir.
The other thing you can try is explicitly installing the plugins using 
docker-compose exec meltano-ui meltano install
, since the last error you saw suggests that the executable can't be found
n
ah that might be it - it’s the former (though it’s built off of the usual meltano/meltano container, again using the official boilerplate for the dockerfile
Dockerfile for globalgiving/meltano image:
Copy code
ARG MELTANO_IMAGE=meltano/meltano:latest
FROM $MELTANO_IMAGE

WORKDIR /project

# Install any additional requirements
COPY ./requirements.txt . 
RUN pip install -r requirements.txt

# Install all plugins into the `.meltano` directory
COPY ./meltano.yml . 
RUN meltano install

# Pin `discovery.yml` manifest by copying cached version to project root
RUN cp -n .meltano/cache/discovery.yml . 2>/dev/null || :

# Don't allow changes to containerized project files
ENV MELTANO_PROJECT_READONLY 1

# Copy over remaining project files
COPY . .

# Expose default port used by `meltano ui`
EXPOSE 5000

ENTRYPOINT ["meltano"]
d
All right, that looks good. Let's see if docker-compose behaves better if you're actually using the project dir baked into the container instead of mounting it from outside
n
yep - looks to be! airflow webserver now loads and I can access it in the browser. Looks like it’s having trouble connecting to the scheduler though
d
If your Docker image already contains the project, you'll need a separate DB for Meltano and Airflow that bot containers can connect with, which 
docker-compose.prod.yml
handles for you: https://gitlab.com/meltano/files-docker-compose/-/blob/master/bundle/README.md#production-usage
See also https://meltano.com/docs/containerization.html#docker-compose:
If you'd like to use Docker Compose to experiment with a production-grade setup of your containerized project, you can add the appropriate 
docker-compose.prod.yml
file to your project by adding the `docker-compose` file bundle :
I think you're currently using the non-production copy that assumes you'll use 
meltano/meltano
(not a containerized project) + mounting your project
If that wasn't clear, we should update the docs 🙂
n
nope - that was clear and you’re right! I have the docker-compose.prod.yml as well, but when I started running into issues I figured I’d be better off narrowing the variables to the simpler version while debugging
I’m also working with our devops folks about exactly how we’re going to handle the meltano/airflow DBs in practice
Thanks for the push in the right direction @douwe_maan! Knowing this, I might go back to the other docker-compose and see how that goes since it looks like the scheduler’s still having trouble in the simpler version. Will report back!
d
@nick_hamlin All right, let me know if you need any more help!
And consider joining the #C01QS0RV78D right now if you have any additional topics to discus 🙂
n
If I wasn’t just going into a meeting at this moment, I’d have brought this question up there 🙂
Confirmed that the prod-level approach solves the scheduler problem, so that’s great! Specifically, I’m using this docker-compose with all the airflow stuff uncommented https://gitlab.com/meltano/files-docker-compose/-/blob/master/bundle/docker-compose.prod.yml
d
That's great!
n
The only (likely really dumb) thing I’m still a bit hazy on is how the meltano and airflow postgres DBs are persisted beyond the container in this model. I can see that they’re each mapped to externally (e.g. 
meltano_postgresql_data
), and so I’d expect to find those somewhere on my local machine if I’m understanding correctly but I’m not seeing them anywhere?
I can tell that it’s persisting that data somehow, because I can tear the stack down and bring it back up again with 
docker-compose up/down
and I still have stuff persisting in my airflow UI the way I’d expect
Actually, I may have spoken too soon - looks like it’s not actually able to run jobs because of issues accessing the logs. Here’s what I get in the airflow logs for the failing jobs:
Copy code
Jump to end
Toggle wrap
*** Log file does not exist: /project/.meltano/run/airflow/logs/meltano_test/extract_load/2021-05-05T21:01:16.002963+00:00/1.log
*** Fetching from: <http://562d5dfe87fd:8793/log/meltano_test/extract_load/2021-05-05T21:01:16.002963+00:00/1.log>
*** Failed to fetch log file from worker. HTTPConnectionPool(host='562d5dfe87fd', port=8793): Max retries exceeded with url: /log/meltano_test/extract_load/2021-05-05T21:01:16.002963+00:00/1.log (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f71e6ae64e0>: Failed to establish a new connection: [Errno 111] Connection refused',))
d
@nick_hamlin I'm not sure where that 
meltano_postgresql_data
volume would actually be stored, but these docs should help you figure that out: https://docs.docker.com/compose/compose-file/compose-file-v3/#volume-configuration-reference
looks like it’s not actually able to run jobs because of issues accessing the logs
Are the jobs actually failing to be run by the scheduler? Or is the UI failing to show their logs?
n
i’m not totally sure, but I’m starting to suspect the latter
I made a very simple test job that appears to have completed successfully, even though the logs aren’t accessible.
d
All right. I'm not sure where Airflow stores its own logs or how those can be made available between a scheduler and webserver running in separate containers
n
yep - more debugging/research needed on my end, will share anything useful I figure out here!
Ok, I think I’ve got this all figured out! Root cause of the log weirdness was that the 
docker-compose.prod.yml
template assumes you’re using the meltano UI and not the airflow UI. It exposes the meltano logs between the various containers properly, but the airflow logs are technically separate/stored in a different place (even if they wind up containing essentially the same information). Following the existing pattern for setting up the shared volume for the meltano logs, it was straightforward to add another one to share the airflow logs between the scheduler and the webserver
another interesting discovery that happened in the process of figuring this out: if you delete a DAG in the airflow UI while a job is running, the corresponding entry in the Meltano DB gets stuck in the RUNNING state. This has been taken care of for jobs running via the meltano UI, but it looks a similar gap still exists for direct airflow use.
I was able to workaround this for now by manually deleting the stuck records (I also tweaked the docker-compose to support direct connections to the meltano PG database, which was useful)
Happy to share more info or field questions as needed!
d
@nick_hamlin Glad you figured it out! If you could create issues for these things you discovered that'd be much appreciated
n
Can do! Thanks for the clear direction - I contemplated doing that preemptively, but I wasn’t sure whether or not these would be considered in scope for meltano itself or not
d
Either way it's worth tracking the issue. The resolution may just be a matter of documentation, or our Airflow integration could be improved
r
I was able to workaround this for now by manually deleting the stuck records (I also tweaked the docker-compose to support direct connections to the meltano PG database, which was useful)
@nick_hamlin - can you please share your docker compose changes there?
n
@rodney_greenfield all I did to enable this was to open up port 5432 by adding this to the 
meltano-system-db
section of the docker compose:
Copy code
ports:
      - 5432:5432
Once that’s done, anything that can connect to postgres can be set up to interact with the database in the container on that port
o
@douwe_maan @nick_hamlin was this issue resolved? Because I am experiencing this when running 
meltano run elt
from my airflow task and when airflow is interrupted sometimes that task is stuck on RUNNING STATE in the database. I am using 
v1.77.0
d
@or_barda I suggest posting this as a new message in #C01TCRBBJD7 , with a link to this original thread so it’ll show up on our support radar 🙂
o
Gotcha, will do
2 Views
Open in Slack
PreviousNext
Powered by