Hi there! I've recently started to use Meltano and really enjoy working with it. 🙏 I just finished reading https://meltano.com/blog/5-helpful-extract-load-practices-for-high-quality-raw-data/ and in tip #1 the author recommends including ingestion time, ingestion process and source system as metadata columns when ingesting new rows. Further, the article says...

Don’t just provide the “Airflow-OracleDB-Ingester” as a process, but the “Airflow-OracleDB-Ingester-ID1244” where ID1244 clearly identifies the specific run of ingestion.

What would be the best way to add such metadata columns? I use a mix of different extractors together with

target-postgres

(transferwise) and Airflow for orchestration. I saw that

target-postgres

has its own config setting for metadata columns, but these are a bit different from whats suggested in the article. Does Meltano offer some way to inject additional attributes during loading? 🤔

Glad you’re finding it useful! If the target you’re using was built on the Meltano SDK you can set

add_record_metadata=True

and you’ll get a bunch of useful fields added https://sdk.meltano.com/en/latest/implementation/record_metadata.html Another way to do this would be to use stream maps to add a new column https://sdk.meltano.com/en/latest/stream_maps.html#built-in-functions cc @Sven Balnojan @aaronsteers

The transferwise target-postgres has a setting called

add_metadata_columns

that you can use also

Thanks for the quick answers! 👍 I wasn't aware of the

add_record_metadata

setting for Meltano SDK based targets. Since I currently use the transferwise

target-postgres

I'll experiment with its

add_metadata_columns

setting. In addition to these timestamps, I liked the idea of having the Airflow run ID as metadata, but looks like this is only possible via some type of custom stream mapping.

The transferwise target-postgres has a setting called

add_metadata_columns

that you can use also

the transferwise target-redshift plugin does too, we use it