Hi, In meltano, I have tap-mysql target-postgres....
# troubleshooting
a
Hi, In meltano, I have tap-mysql target-postgres. In mysql I have 2 tables User : userId, userName, companyId (Record example : 1, "Anita", 101) Company : companyId, shortName ((Record example : 101, "Sales")) Now I want to create custom transformer/mapper & fit it in between that - read companyId from User table. - based on companyId, need to get shortName from Company table. - In postgres I want tables like User_Sales i.e, Company specific user tables. I want to tweak tap-mysql's json's table name. because I want to create company specific User table in post-gres using target-postgres. I want to read tap-mysql records & by doing small transformation, I want to send tweaked json to target-postgres. How to do it? Can you give me example of meltano.yml config file, custom transformer code code? I tried many ways but failed!
h
Hi Anita, the operations your are looking to do don't feel natural for the meltano mapper. I'm not even sure if that's totally doable. In the ELT design methodology which singer/meltano adheres to, this type of join would be carried out in the destination by a separate tool (e.g. dbt). The mapper is meant to transform individual records within the stream, so for instance if your raw stream contained user emails, but the destination was only allowed to receive email domains, you would be able to transform that using a mapper.
👍 1