has anyone set up pytests for their meltano setup?
# best-practices
d
has anyone set up pytests for their meltano setup?
e
@devon_seitz can you say more about your case? I'm assuming you don't mean running
pytest
to test Meltano itself, but test your own project or plugins?
d
yeah exactly - would love to bake in some testing that meltano “works”. I guess i could probably set up some subprocess things around the invoke command
The way we are currently thinking about this internally is: • 'test' commands are any command that starts with 'test*' in the name • For test and CI use cases, Meltano environments can be customized so that smaller slices of data or pure test data is used to speed up testing. • For specific tap-to-target combos, a mapper can be placed between the tap and target to either filter down the records or to apply custom obfuscation/mapping. • meltano run can sequence a set of commands that collectively prove that the project is sound:
meltano run tap-foo target-snowflake tap-bar target-snowflake dbt-snowflake:run dbt-snowflake:test other-validator:test
• The
run
command itself can be saved as a
job
and invoked in the shorthand:
meltano --environment=localtest run my-build-and-test-job
.
What we don't really have a solid path forward on is whether/how we should formalize this into something like
meltano build ...
- although that's discussed in the issues I linked above.
Does this help at all? Would love your thoughts on this, and I encourage you to leave ideas and use cases there in those threads!