#3439 Capture and store dbt artifacts ("state") following job run Issue created by MeltyBot Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/3524 Originally created by @kgpayne on 2022-05-25 133703 --- As with Singer STATE messages (containing bookmarks in json form), dbt also produces json artefacts that can be used to track model build times and to defer execution of models in another namespace (speeding up development and CI/CD use cases). Meltano can help facilitate these use cases by capturing dbt artefacts and making them available (either via the new

meltano state

command, or during invocations of dbt as with Singer state), handling a common pain point for users self-hosting dbt. dbt generates 4 json artefacts (schemas available), with `manifest.json` and `run_results.json` being the most useful.

manifest.json

is a complete representation of a dbt project at the point of execution, used by the

--defer --state

commands to defer execution of unchanged models to another namespace (typically prod) during development and CI/CD.

run_results.json

is used for tracking operational metrics such as model build times after a run has completed. Examples: # invoke dbt run, deferring to most recent production state files # for this to work, Meltano would have to be able to fetch state from a different Meltano environment # (i.e. a different job-id prefix) to a location on disk before invoking dbt meltano invoke dbt run --select [...] --defer --state ".meltano/transformer/state/prod:$MELTANO_JOB_ID_WITHOUT_ENV/" # alternative syntax, with template rendering by Meltano before invocation meltano invoke dbt run --select [...] --defer --state "{{ get_state_dir('prod') }}" meltano/meltano