Hey Team I am little new to meltanto so stuck at one place. I was able to successfully extract data from mongodb and load into postgres along with some basic transformations. But I want to use dbt transformation layer before dumping data into postgres but keep running into some errors can some one help here or point to some docs that I can refer for this? Appreciate any help here thanks

Can you share the errors you are getting? Typically you would do only very light (or no) in-flight transformation from the tap before landing in target postgres in a dedicated

raw

schema or similar. Then build your DBT models from that raw schema as a source.

hmm what I am trying to do right now is basically flatten few nested jsons from mongo before loading into postgres so wanted to use dbt there. Are you suggesting first load everything into postgres as in raw and then use dbt to create views and go forward from there?

Yes, ideally retain as close as possible to your original DB data unless you have good reason not to.

hmm interesting out of curiosity why this approach is preferred? Apologies for asking noob questions here but still new to all this. I was of the opinion since I know none of our downstreams for which we are planning to use will need most of the data points why load complete nested json in postgres why not keep the db light only?

I would say given that most of the time is involved in replication from your sources, so if it turns out you are missing a field you need, you don't need to reingest from your tap every time, you just alter your dbt staging model slightly to grab that new field. Obviously you need to make consideration to the size of the data extracted too, other factors. It's a subjective thing in the end.

Makes sense thanks a lot @Andy Carter