https://linen.dev logo
Join Slack
Powered by
Hello. I've successfully switched to docker versio...
# announcements
j
Hello. I've successfully switched to docker version of meltano. Uncommented airflow related lines on docker-compose.prod.yml and meltano ui seems to work well. But airflow webserver is not responding, and and telnet and nginx to that port (8080) returns connection refused. Any idea why this is happening?
d
I assume the 
meltano invoke airflow webserver
output is not revealing anything? Meltano directly invokes the airflow executable with the 
webserver
arg and has no further influence over how it runs, so if it's refusing connections that's more likely to be something to debug in Airflow than in Meltano
j
I've just ran the invoke command under the ui service and it returns 
Airflow metadata database could not be initialized: airflow initdb failed
So that give me a clue
d
Ah interesting, try running with 
--log-level=debug
so that you get the full error output
j
This is weird? 
airflow.exceptions.AirflowConfigException: error: cannot use sqlite with the LocalExecutor
d
Ah that makes sense, SQLite only allows 1 concurrent connection which is incompatible with the concurrent LocalExecutor which will want to make more than 1 connection
Note the second recommendation under https://meltano.com/docs/production.html#airflow-orchestrator 🙂
j
They Docker imagen doesnt use they postres config?
d
j
I did
d
Hmm, then I'm not sure why Airflow still thinks it should use SQLite 😕 Can you run 
meltano config airflow
to verify it gets picked up correctly?
In the output for the 
--log-level=debug invoke airflow scheduler
command that failed, you should also see a line prefixed with 
Updated section [core] with
. Does the 
sql_alchemy_conn
in that line match the Postgres URL?
(I'm about to go offline for the day, I'll continue helping you debug this tomorrow!)
j
'sql_alchemy_conn': 'sqlite:////project/.meltano/orchestrators/airflow/airflow.db'
d
Interesting
That's in 
meltano config airflow
?
j
No, thats the previous log
well, config airflow does show that also
d
All right, then Meltano is not picking up the 
AIRFLOW__CORE__SQL_ALCHEMY_CONN
env var for some reason 😕
j
Ok. I will keep trying, Perhams is some type on the yml
typo*
d
If you run 
meltano config airflow
and explicitly set 
AIRFLOW__CORE__SQL_ALCHEMY_CONN
using 
docker run -e "AIRFLOW__CORE__SQL_ALCHEMY_CONN=foo"
, does "foo" show up?
That works correctly for me, without Docker:
Copy code
$ AIRFLOW__CORE__SQL_ALCHEMY_CONN=foo meltano config airflow
{
  "core": {
    "dags_folder": "/Users/douwemaan/Development/meltano-projects/demo-project/orchestrate/dags",
    "plugins_folder": "/Users/douwemaan/Development/meltano-projects/demo-project/orchestrate/plugins",
    "sql_alchemy_conn": "foo",
    "load_examples": "False",
    "dags_are_paused_at_creation": "False"
  }
}
(Dinner's ready, see you tomorrow!)
j
Thanks for the help.
Hey. The problem was because I didn't added Psycog2 to the pip url if the orchestrator. Now the issue is related to a connection timeout to the DB
Copy code
[2021-01-15 13:06:41,971] [1|MainThread|root] [INFO] DB connection failed. Will retry after 5s. Attempt 1/3
airflow-scheduler_1    | [2021-01-15 13:07:18,579] [1|MainThread|root] [INFO] DB connection failed. Will retry after 5s. Attempt 2/3
airflow-scheduler_1    | [2021-01-15 13:07:55,187] [1|MainThread|root] [INFO] DB connection failed. Will retry after 5s. Attempt 3/3
airflow-scheduler_1    | [2021-01-15 13:08:31,795] [1|MainThread|root] [ERROR] Could not connect to the Database. Max retries exceeded.
airflow-scheduler_1    | (psycopg2.OperationalError) could not connect to server: Connection timed out
airflow-scheduler_1    |        Is the server running on host "meltano-system-db" (172.22.0.2) and accepting
airflow-scheduler_1    |        TCP/IP connections on port 5432?
d
@jean_rumeau If you look at the output for the meltano-system-db container, does it take a while to become ready to accept connections? You may need to increase https://meltano.com/docs/settings.html#database-max-retries, if the current 3 tries * 5s = 15s waiting period isn't enough
j
I don't think its a timeout related issue. I've incremented to 5 retries of 30s
d
Hmm, all right. Are you learning anything from the 
meltano-system-db
container output?
j
Its a AWS lightsail instance with 4gb ram and 2vcpu, could this be an issue?
Copy code
meltano-system-db_1    |
meltano-system-db_1    | PostgreSQL Database directory appears to contain a database; Skipping initialization
meltano-system-db_1    |
meltano-system-db_1    | 2021-01-15 16:43:09.756 UTC [1] LOG:  starting PostgreSQL 13.1 (Debian 13.1-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
meltano-system-db_1    | 2021-01-15 16:43:09.756 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
meltano-system-db_1    | 2021-01-15 16:43:09.756 UTC [1] LOG:  listening on IPv6 address "::", port 5432
meltano-system-db_1    | 2021-01-15 16:43:09.769 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
meltano-system-db_1    | 2021-01-15 16:43:09.776 UTC [26] LOG:  database system was shut down at 2021-01-15 16:42:19 UTC
meltano-system-db_1    | 2021-01-15 16:43:09.781 UTC [1] LOG:  database system is ready to accept connections
d
Its a AWS lightsail instance with 4gb ram and 2vcpu
That's not a lot for 2 Postgres instances, 2 web UIs, a scheduler, and the actual ELT workloads, but unless you're already seeing processes quit because of memory issues, I don't think that's the issue here
j
I did got memory problems for a previous instance so I increased resource tho this one
d
That log output looks OK. I'd start debugging Docker networking in general
If 
airflow-scheduler
can't connect with 
meltano-system-db:5432
even though it's on the same 
meltano
network (https://gitlab.com/meltano/files-docker-compose/-/blob/master/bundle/docker-compose.prod.yml#L63), that doesn't look like a Meltano-specific issue
j
I suspect that. So im afraid is an issue with networking on lightsail instances
As lightsail have a separated service for containers
d
I have no experience with lightsail so I'm afraid I can't help you there
If you run locally with docker-compose, are Meltano and Airflow able to connect with their Postgres databases?
j
message has been deleted
d
😕
I'd continue debugging Docker networking... Please let me know if it turns out we should change https://gitlab.com/meltano/files-docker-compose/-/blob/master/bundle/docker-compose.prod.yml somehow!
j
It seems to work well locally. So it doesnt seems to be a problem with the Docker configuration but with lightsail specifically
d
Good to know
j
Just to ket you know. I've managed to ran everyting on EC2 instances instead of Lighthouse finnally (seems lighthouse has some networking limitation). Thanks for the time and support.
d
@jean_rumeau Great!
8 Views
Open in Slack
PreviousNext
Powered by