https://linen.dev logo
Join Slack
Powered by
<@U06D0TMQ2U8> I took you're advice and stripped d...
# meltano-plugin-development
f
@pat_nadolny I took you're advice and stripped down the logging fields by default. Also baked in support so that meltano can automatically to toggle on json formatted output down the road, so that we can parse the log output and selectively display bits of it (eventually).
e.g. no timestamps or level key by default but easily addable if you're debugging. so you can do things like
Copy code
LOG_LEVEL=debug LOG_TIMESTAMPS=true meltano invoke someplugin somecommand
To amp up the output.
p
Nice! So by default the extension is basically just passing up a log message to meltano which then logs it to the console. Then its not a double log situation. Am I understanding that right?
f
well in most cases we at least make it look like its not a log wrapped in a log 😁
If you're extension is invoking something like 
airflow webserver
you still have that problem though.
The extension dev will need to configure that thing they're calling appropriately.
Copy code
(melty-3.8) ➜  extdev meltano invoke airflow:version --nope

usage: airflow [-h] GROUP_OR_COMMAND ...
airflow command error: unrecognized arguments: --nope, see help above.

positional arguments:
  GROUP_OR_COMMAND

    Groups:
      celery         Celery components
      config         View configuration
      connections    Manage connections
      dags           Manage DAGs
      db             Database operations
      jobs           Manage jobs
      kubernetes     Tools to help run the KubernetesExecutor
      pools          Manage pools
      providers      Display providers
      roles          Manage roles
      tasks          Manage tasks
      users          Manage users
      variables      Manage variables

    Commands:
      cheat-sheet    Display cheat sheet
      dag-processor  Start a standalone Dag Processor instance
      info           Show information about current Airflow and environment
      kerberos       Start a kerberos ticket renewer
      plugins        Dump information about loaded plugins
      rotate-fernet-key
                     Rotate encrypted connection credentials and variables
      scheduler      Start a scheduler instance
      standalone     Run an all-in-one copy of Airflow
      sync-perm      Update permissions for existing roles and optionally DAGs
      triggerer      Start a triggerer instance
      version        Show the version
      webserver      Start a Airflow webserver instance

optional arguments:
  -h, --help         show this help message and exit


error invoking airflow version error_message=airflow invocation failed returncode=2
So looks like that for a "regular" command.
But 
meltano run
will still double up in some cases:
Copy code
(melty-3.8) ➜  extdev meltano run airflow:webserver
2022-08-05T16:12:46.084921Z [info     ] [2022-08-05 11:12:46 -0500] [7940] [INFO] Starting gunicorn 20.1.0 cmd_type=command name=airflow stdio=stderr
2022-08-05T16:12:46.294203Z [info     ] [2022-08-05 11:12:46 -0500] [7940] [INFO] Listening at: <http://0.0.0.0:8088> (7940) cmd_type=command name=airflow stdio=stderr
2022-08-05T16:12:46.294378Z [info     ] [2022-08-05 11:12:46 -0500] [7940] [INFO] Using worker: sync cmd_type=command name=airflow stdio=stderr
2022-08-05T16:12:47.815068Z [info     ] [2022-08-05 11:12:47 -0500] [7940] [WARNING] Worker with pid 7942 was terminated due to signal 11 cmd_type=command name=airflow stdio=stderr
2022-08-05T16:12:47.828349Z [info     ] [2022-08-05 11:12:47 -0500] [7940] [WARNING] Worker with pid 7943 was terminated due to signal 11 cmd_type=command name=airflow stdio=stderr
that second set of timestamps and levels comes from the airflow webserver (which we can't make do our bidding directly).
I should mention for airflow specifically, i think we can disable the timestamp/level fields in the airflow.cfg by specifying a custom log format, that we ship by default. But it wasn't clear to me if that was a global option or what, and I haven't gotten a chance to try it.
turning off timestamps in the scheduler/dag output logs would probably not-so-hot.
p
that all makes sense to me, im not sure how we'd be able to do it better than that. Especially run a subprocess in subprocess in subprocess hahah
f
xzibit
lol , just wanna point out i searched giphy for xzibit and got 50.
but the fact that i got disapproving 50 is kinda perfect.
Open in Slack
PreviousNext
Powered by