https://linen.dev logo
Join Slack
Powered by
Is there a way to navigate to execute a utility co...
# plugins-general
p
Is there a way to navigate to execute a utility command in a subdirectory? I need assess to environments so I have to run my meltano command at the top level but my utility plugin (great expectations) needs commands to be run in the project subdirectory
Copy code
meltano.yml
environments.meltano.yml
utilities/great_expectations/great_expectations.yml
so when I run 
meltano --environment=prod invoke great_expectations checkpoint run x
it needs to be within the 
utilities/great_expectations/
or else it says 
Error: No great_expectations directory was found here!
if I 
cd
into that directory I get 
Environment 'prod' was not found.
from Meltano. But if I remove the environment then it works
e
hmm could it be because meltano considers paths in 
include_paths
as relative to where the command is run?
p
ahhh good call, its sort of a hacky solution but it works if I add 
- ../../*.meltano.yml
to my 
include_paths
thanks @edgar_ramirez_mondragon
e
I'll file an issue
@pat_nadolny can you check if the 
GE_HOME=$MELTANO_PROJECT_ROOT/utilities/great_expectations/
environment variable tells GE to look for the 
great_expectations.yml
file there?
p
ooo nice - let me try that
where should that be set? I think I just logged an issue around this
when I set it in the .env or in the 
env
key it doesnt populate the template - 
'GE_HOME': '$MELTANO_PROJECT_ROOT/utilities/great_expectations/'
from the logs
e
I wonder how dbt configs in discovery.yml do it 🤔
p
I figured something inside the meltano code base was doing it for dbt, and GE is just a custom plugin right now so it doesnt work yet but 🤷
e
I think this works
Copy code
- name: great_expectations
    namespace: great_expectations
    pip_url: great_expectations; sqlalchemy; PyAthenaJDBC>1.0.9; PyAthena>1.2.0
    executable: great_expectations
    settings:
      - name: ge_home
        value: $MELTANO_PROJECT_ROOT/utilities/great_expectations
or close enough, I do see 
'GREAT_EXPECTATIONS_GE_HOME': '/Users/edgarramirez/meltano/squared/data/utilities/great_expectations'
 in the debug output
Copy code
- name: great_expectations
    namespace: great_expectations
    pip_url: great_expectations; sqlalchemy; PyAthenaJDBC>1.0.9; PyAthena>1.2.0
    executable: great_expectations
    settings:
      - name: ge_home
        value: $MELTANO_PROJECT_ROOT/utilities/great_expectations
        env: GE_HOME
@pat_nadolny I think this ☝️ should do it
p
hmm mine still isnt working
debug logs say 
'GE_HOME': '$MELTANO_PROJECT_ROOT/utilities/great_expectations/',
even though 
'MELTANO_PROJECT_ROOT': '/Users/pnadolny/Documents/Git/GitLab/squared/data'
is also available
e
I think this line has to be prepended with 
self.root
and 
GE_HOME
correctly expands 
MELTANO_PROJECT_ROOT
for me 🤔
Copy code
..., 'GE_HOME': '/Users/edgarramirez/meltano/squared/data/utilities/great_expectations', 'GREAT_EXPECTATIONS_GE_HOME': '/Users/edgarramirez/meltano/squared/data/utilities/great_expectations', 'MELTANO_UTILIZE_GE_HOME': '/Users/edgarramirez/meltano/squared/data/utilities/great_expectations', 'VIRTUAL_ENV': '/Users/edgarramirez/meltano/squared/data/.meltano/utilities/great_expectations/venv'}
p
ok so yours is only working after a code change, or was I doing something wrong with mine?
e
it's working after updating 
meltano.yml
with this
Copy code
- name: great_expectations
    namespace: great_expectations
    pip_url: great_expectations; sqlalchemy; PyAthenaJDBC>1.0.9; PyAthena>1.2.0
    executable: great_expectations
    settings:
      - name: ge_home
        value: $MELTANO_PROJECT_ROOT/utilities/great_expectations
        env: GE_HOME
and running from the meltano root (in 
squared/data
)
p
awesome! yes thats working for me now
Open in Slack
PreviousNext
Powered by