https://meltano.com/ logo
Channels
Powered by
#announcements
Title
# announcements
t

thousands-library-17999

01/15/2021, 12:12 AM
Hello. I've successfully switched to docker version of meltano. Uncommented airflow related lines on docker-compose.prod.yml and meltano ui seems to work well. But airflow webserver is not responding, and and telnet and nginx to that port (8080) returns connection refused. Any idea why this is happening?
✅ 1
r

ripe-musician-59933

01/15/2021, 12:14 AM
I assume the 
meltano invoke airflow webserver
output is not revealing anything? Meltano directly invokes the airflow executable with the 
webserver
arg and has no further influence over how it runs, so if it's refusing connections that's more likely to be something to debug in Airflow than in Meltano
t

thousands-library-17999

01/15/2021, 12:18 AM
I've just ran the invoke command under the ui service and it returns 
Airflow metadata database could not be initialized: airflow initdb failed
So that give me a clue
r

ripe-musician-59933

01/15/2021, 12:18 AM
Ah interesting, try running with 
--log-level=debug
so that you get the full error output
t

thousands-library-17999

01/15/2021, 12:20 AM
This is weird? 
airflow.exceptions.AirflowConfigException: error: cannot use sqlite with the LocalExecutor
r

ripe-musician-59933

01/15/2021, 12:22 AM
Ah that makes sense, SQLite only allows 1 concurrent connection which is incompatible with the concurrent LocalExecutor which will want to make more than 1 connection
Note the second recommendation under https://meltano.com/docs/production.html#airflow-orchestrator 🙂
t

thousands-library-17999

01/15/2021, 12:24 AM
They Docker imagen doesnt use they postres config?
r

ripe-musician-59933

01/15/2021, 12:25 AM
t

thousands-library-17999

01/15/2021, 12:27 AM
I did
r

ripe-musician-59933

01/15/2021, 12:29 AM
Hmm, then I'm not sure why Airflow still thinks it should use SQLite 😕 Can you run 
meltano config airflow
to verify it gets picked up correctly?
In the output for the 
--log-level=debug invoke airflow scheduler
command that failed, you should also see a line prefixed with 
Updated section [core] with
. Does the 
sql_alchemy_conn
in that line match the Postgres URL?
(I'm about to go offline for the day, I'll continue helping you debug this tomorrow!)
t

thousands-library-17999

01/15/2021, 12:34 AM
'sql_alchemy_conn': 'sqlite:////project/.meltano/orchestrators/airflow/airflow.db'
r

ripe-musician-59933

01/15/2021, 12:34 AM
Interesting
That's in 
meltano config airflow
?
t

thousands-library-17999

01/15/2021, 12:35 AM
No, thats the previous log
well, config airflow does show that also
r

ripe-musician-59933

01/15/2021, 12:36 AM
All right, then Meltano is not picking up the 
AIRFLOW__CORE__SQL_ALCHEMY_CONN
env var for some reason 😕
t

thousands-library-17999

01/15/2021, 12:36 AM
Ok. I will keep trying, Perhams is some type on the yml
typo*
r

ripe-musician-59933

01/15/2021, 12:38 AM
If you run 
meltano config airflow
and explicitly set 
AIRFLOW__CORE__SQL_ALCHEMY_CONN
using 
docker run -e "AIRFLOW__CORE__SQL_ALCHEMY_CONN=foo"
, does "foo" show up?
That works correctly for me, without Docker:
Copy code
$ AIRFLOW__CORE__SQL_ALCHEMY_CONN=foo meltano config airflow
{
  "core": {
    "dags_folder": "/Users/douwemaan/Development/meltano-projects/demo-project/orchestrate/dags",
    "plugins_folder": "/Users/douwemaan/Development/meltano-projects/demo-project/orchestrate/plugins",
    "sql_alchemy_conn": "foo",
    "load_examples": "False",
    "dags_are_paused_at_creation": "False"
  }
}
(Dinner's ready, see you tomorrow!)
t

thousands-library-17999

01/15/2021, 12:39 AM
Thanks for the help.
Hey. The problem was because I didn't added Psycog2 to the pip url if the orchestrator. Now the issue is related to a connection timeout to the DB
Copy code
[2021-01-15 13:06:41,971] [1|MainThread|root] [INFO] DB connection failed. Will retry after 5s. Attempt 1/3
airflow-scheduler_1    | [2021-01-15 13:07:18,579] [1|MainThread|root] [INFO] DB connection failed. Will retry after 5s. Attempt 2/3
airflow-scheduler_1    | [2021-01-15 13:07:55,187] [1|MainThread|root] [INFO] DB connection failed. Will retry after 5s. Attempt 3/3
airflow-scheduler_1    | [2021-01-15 13:08:31,795] [1|MainThread|root] [ERROR] Could not connect to the Database. Max retries exceeded.
airflow-scheduler_1    | (psycopg2.OperationalError) could not connect to server: Connection timed out
airflow-scheduler_1    |        Is the server running on host "meltano-system-db" (172.22.0.2) and accepting
airflow-scheduler_1    |        TCP/IP connections on port 5432?
r

ripe-musician-59933

01/15/2021, 4:14 PM
@thousands-library-17999 If you look at the output for the meltano-system-db container, does it take a while to become ready to accept connections? You may need to increase https://meltano.com/docs/settings.html#database-max-retries, if the current 3 tries * 5s = 15s waiting period isn't enough
t

thousands-library-17999

01/15/2021, 5:10 PM
I don't think its a timeout related issue. I've incremented to 5 retries of 30s
r

ripe-musician-59933

01/15/2021, 5:11 PM
Hmm, all right. Are you learning anything from the 
meltano-system-db
container output?
t

thousands-library-17999

01/15/2021, 5:11 PM
Its a AWS lightsail instance with 4gb ram and 2vcpu, could this be an issue?
Copy code
meltano-system-db_1    |
meltano-system-db_1    | PostgreSQL Database directory appears to contain a database; Skipping initialization
meltano-system-db_1    |
meltano-system-db_1    | 2021-01-15 16:43:09.756 UTC [1] LOG:  starting PostgreSQL 13.1 (Debian 13.1-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
meltano-system-db_1    | 2021-01-15 16:43:09.756 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
meltano-system-db_1    | 2021-01-15 16:43:09.756 UTC [1] LOG:  listening on IPv6 address "::", port 5432
meltano-system-db_1    | 2021-01-15 16:43:09.769 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
meltano-system-db_1    | 2021-01-15 16:43:09.776 UTC [26] LOG:  database system was shut down at 2021-01-15 16:42:19 UTC
meltano-system-db_1    | 2021-01-15 16:43:09.781 UTC [1] LOG:  database system is ready to accept connections
r

ripe-musician-59933

01/15/2021, 5:13 PM
Its a AWS lightsail instance with 4gb ram and 2vcpu
That's not a lot for 2 Postgres instances, 2 web UIs, a scheduler, and the actual ELT workloads, but unless you're already seeing processes quit because of memory issues, I don't think that's the issue here
t

thousands-library-17999

01/15/2021, 5:13 PM
I did got memory problems for a previous instance so I increased resource tho this one
r

ripe-musician-59933

01/15/2021, 5:14 PM
That log output looks OK. I'd start debugging Docker networking in general
If 
airflow-scheduler
can't connect with 
meltano-system-db:5432
even though it's on the same 
meltano
network (https://gitlab.com/meltano/files-docker-compose/-/blob/master/bundle/docker-compose.prod.yml#L63), that doesn't look like a Meltano-specific issue
t

thousands-library-17999

01/15/2021, 5:15 PM
I suspect that. So im afraid is an issue with networking on lightsail instances
As lightsail have a separated service for containers
r

ripe-musician-59933

01/15/2021, 5:17 PM
I have no experience with lightsail so I'm afraid I can't help you there
If you run locally with docker-compose, are Meltano and Airflow able to connect with their Postgres databases?
t

thousands-library-17999

01/15/2021, 5:19 PM
image.png
r

ripe-musician-59933

01/15/2021, 5:22 PM
😕
I'd continue debugging Docker networking... Please let me know if it turns out we should change https://gitlab.com/meltano/files-docker-compose/-/blob/master/bundle/docker-compose.prod.yml somehow!
t

thousands-library-17999

01/15/2021, 6:02 PM
It seems to work well locally. So it doesnt seems to be a problem with the Docker configuration but with lightsail specifically
r

ripe-musician-59933

01/15/2021, 6:02 PM
Good to know
t

thousands-library-17999

01/15/2021, 11:02 PM
Just to ket you know. I've managed to ran everyting on EC2 instances instead of Lighthouse finnally (seems lighthouse has some networking limitation). Thanks for the time and support.
r

ripe-musician-59933

01/15/2021, 11:19 PM
@thousands-library-17999 Great!
Powered by