I have a question - is it possible to write all the meltano elt logs to stdout?

Meltano currently outputs some things to both stdout and stderr, but we noted here that we should just be putting everything to stdout.

How to set it inside docker container?

I’m not sure I understand your question @xinge_li. What are you trying to set?

So I tried to deploy meltano on aws and something was wrong with airflow scheduler, however, I was not able to see what exactly was wrong but only the task was marked as failed. I would like to ask whether it is possible to see the elt logs.

are you able to run the airflow webserver?

meltano invoke airflow webserver

yes

you should be able to view the airflow logs through that. other logs will be stored in the

.meltano/

folder on the filesystem.

Is it possible to store meltano elt logs to e.g cloudWatch?

from within Meltano itself, I don’t believe so, but if you’re running Meltano with docker you should be able to ship the logs to cloudwatch from docker (since the docker logs should just be the meltano logs too).

I see, I can give it a try, thanks!

It would be very useful to be able to configure Meltano to write the ELT logs to STDOUT. I'm running in Heroku and they don't support mounting volumes in docker containers, so there's simply no way to get to the logs in

.meltano

.

@kat_crichton-seager can you make an issue around this? would be good to document the need

Thanks @taylor. I've raised this ticket; https://gitlab.com/meltano/meltano/-/issues/2948

#thread-necromancy Do we think this could be solved by mounting a Docker volume in the

$pwd/.meltano

directory? So we can run the copies of the base container for Meltano UI, Airflow UI and Airflow Scheduler, and ensure they are all reading/writing from the same volume

I’m not sure - @edgar_ramirez_mondragon any thoughts?

That should work afaict. I wonder what's not working? Perhaps the mount sync is not bidirectional?

I get this kind of an error from within the Airflow webserver

*** Log file does not exist: /project/.meltano/run/airflow/logs/meltano_airtable-remediate/extract_load/2022-03-01T11:50:00+00:00/1.log
*** Fetching from: <http://ip-10-47-109-250.ap-southeast-2.compute.internal:8793/log/meltano_airtable-remediate/extract_load/2022-03-01T11:50:00+00:00/1.log>
*** Failed to fetch log file from worker. HTTPConnectionPool(host='ip-10-47-109-250.ap-southeast-2.compute.internal', port=8793): Max retries exceeded with url: /log/meltano_airtable-remediate/extract_load/2022-03-01T11:50:00+00:00/1.log (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f7010724f40>, 'Connection to ip-10-47-109-250.ap-southeast-2.compute.internal timed out. (connect timeout=5)'))

Hey @david_tout any luck on this front? We're trying to improve our logging as well @monika_rajput @connor_flynn @jo_pearson

Hi @edward_ryan Not yet I'm afraid. I beleive my problem is somewhat related to my container/task definition in ECS

Ah got it How are you currently storing Meltano logs?

We're not storing them unfortunately...They're just on the container. We don't have a strong need to go back historically, but if it was easy to fire them at CloudWatch Logs we would definitely do that

Thanks!!