I am trying to overwrite tap's schema (override ty...
# meltano-plugin-development
v
I am trying to overwrite tap's schema (override type, maxLength...), so it's reflect on target. Example: • I have source column say
phone_number
in MySQL with type
VARCHAR(20)
It's PII so I applied inline mapping hash, the length increased to 32 => it failed on record validations (max length > 20, which is taken from the source length). • I would like to override the schema so it pass the validation • I can workaround by mapping to a new field:
phone_number_hash
and set null/drop original field
phone_number
. However, there are some other cases too, which override schema would solve them. Therefore I got some questions, it would be great if someone would help me to answer: 1/ Is it possible? Because looks like it's not. I found this issue and it's still open: https://github.com/meltano/meltano/issues/2424 2/ Where in the code that the
schema
extra is applied to catalog discovery? I failed to find it in the sdk code. 3/ How can I apply the same logic of catalog discovery to the stream? Overwrite the
SCHEMA
message, by modifying
tap_base
?
@Edgar Ramírez (Arch.dev) , can you help me with these questions?
e
What have you tried so far, ie what does your
meltano.yml
look like?
1