<#6855 Add `meltano test` support for targets> Iss...
# meltano-repo-activity
g
#6855 Add `meltano test` support for targets Issue created by aaronsteers ## Background re: testing extractors A while back we added
meltano test
support for taps - this is now featured on the Hub as the setup procedure for new taps:
Copy code
meltano add extractor EXTRACTOR
meltano config EXTRACTOR set --interactive
meltano test EXTRACTOR
Ex: https://hub.meltano.com/extractors/tap-github/#installation ## Goal: add test functionality for extractors It could be valuable to add the same for targets, although it is basically impossible to do this without any side effects. Notes: • We'll need to caveat to the user that the test will attempt to save data into their target system. • We may also want to prompt for interactive confirmation or use a
-y
flag to skip the request for confirmation. • When all is done, we'll want to report back to the user if there's any cleanup they need to do on their target system. ## Sample Invocation Following the same pattern as for taps, we would just always guide the user to run these in sequence: (1) install, (2) config, (3) test. We'd promote this to be run before taps and targets are paired together in a real pipeline.
Copy code
meltano add loader target-postgres
meltano config target-postgres set --interactive
meltano test target-postgres [--load_schema=zzz_deleteme] [-y]
## Testing Proposal Here is a basic flow for
meltano test target-snowflake
(for example): 1. Loop through sample pre-generated outputs from
tap-smoke-test
, i.e.:
countries-sample.singer.jsonl
,
users-sample.singer.jsonl
. 2. Start with the simplest test (no nested fields, no complex structures) and progress to harder tests. 3. If the first test fails, abort immediately and report a connection failure with a link to the target's log file. 4. If subsequent tests fail, continue testing and report back to the user which tests passed or failed: • nested fields • rare/complex data types • ability to handle batch messages • ability to handle activate_version messages • etc. 5. If
--load_schema
is provided and the target declares a
load_schema
setting, use this to specify a destination schema for sample data. 6. At the end of the operation, print a list of tables created (or attempted to be created) and let the user know that they can safely drop those. ## Incremental improvement For the MVP of this feature, we really could focus purely on ensuring connectivity - and just create a single table with a single row. If that fails, the most likely explanation is a problem with creds or other configuration logic. Letting the user know that right away, without having to be paired with a real and working tap, could add a lot of value. ## Troubleshooting support As and when we add more tests, the test results could become standard info to request from users if they are having compatibility problems with a specific tap. ## Related A similar issue was logged to add loader testing in Meltano UI: • #6855 meltano/meltano