https://linen.dev logo
Join Slack
Powered by
Random though as I'm hacking on the airflow extens...
# meltano-plugin-development
f
Random though as I'm hacking on the airflow extension. I wonder how frequently plugin devs are gonna end up shipping a templated version of their services config or maybe try to read the meltano yaml or 
meltano config
. Because doing something like the pattern we currently do for Airflow, where you generate a brand new config on the fly and then try to update it from env's....is not super pleasant. (great for end-users though)
@aaronsteers @pat_nadolny how strongly do y'all feel about doing ENV injection only vs also still writing out a config ?
p
I'm not totally clear how it works right now so its hard to say, could you summarize it a little more if you know the details?
f
Today its not terrible, since it happens "within" meltano the config objects are easy to handle.
So basically we call 
airflow --help
to trigger creation of a airflow.cfg, and then we can go through and update/add entries to relevant airflow config sections.
moving this out of meltano , we don't have access to the nicely pre-formatted config (
invoker.plugin_config_processed
) anymore
So you have to start looking in the env for the config option
lol standby fixing to paste an example env, but my laptops not cooperating
So you might end up with a env that looks like:
Copy code
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION=blah
AIRFLOW__WEBSERVER__WEB_SERVER_PORT=8080
AIRFLOW_WEBSERVER_WEB_SERVER_PORT=8080
That you then have to turn back into a structure you can use to actually update the correct airflow config sections and keys.
i guess its not that terrible since we'll end up with a little utility method to at least convert those env vars back to a dict for you.
much more annoying though if you where just gonna create a quick extension by wrapping something in a bash script.
p
ahh ok yeah that makes sense. That was sort of kens point in that we're limited in the SDK if we only get passed env vars, like if the SDK plugin could ask meltano for settings/configs and also ask meltano to do operations like update meltano.yml or whatever, that would be ideal
That is a pain also because I think airflow wants dot separated configs so we'd have to know how to convert the envs to configs based on the plugin.
f
I think airflow wants dot separated configs
Yea thats mostly the annoying bit for airflow
But other things are gonna have similarly annoying behaviors
p
do you want to jump on a zoom real quick?
f
yea, let me top up my coffee real quick
a
I'll join too. Just catching up on the thread
f
👍 dropped a zoom in chan
Open in Slack
PreviousNext
Powered by