Hello! I'm trying to replicate a table from a tap...
# troubleshooting
b
Hello! I'm trying to replicate a table from a tap-postgres to a target-postgres. The target table already exists (meltano didn't create it). The target was created using the exact schema of the source table. For some reason, meltano is issuing some alter table rename commands at the start of it's run and creating a bunch of columns with the date in them (such as id -> id_20220824_1840). I'm guessing this is happening because it thinks that the types of the columns in the target table are incorrect? But that isn't the case. I believe that the tap is just not interpreting the source schema correctly, for example it's reading in id as text, but in reality it's uuid. Anyone have any experience with an issue like this? I'd prefer if it just errored out rather than try to alter the table. Or ultimately, just read the schema correctly from the source table and recognize that the target schema is a perfect match. Thanks!
Ah, in digging around it seems as though I can impose schema via the catalog option? https://docs.meltano.com/reference/command-line-interface#parameters-1
t
The schema extraction is "lossy" in the sense that it's converting an RDBMS schema definition into a general JSON schema definition that can be used by things other than an RDBMS. Imagine if your pipeline was PG to MySQL; you wouldn't expect the exact PG datatypes to be used to create the table in MySQL because they wouldn't even be valid.
If you let target-postgres create the table it won't have the same data types as the original for that same reason - tap-postgres is providing generalized types to target-postgres, and target-postgres is using them to create or update the target table as necessary.
b
Okay cool, that makes a ton of sense. Thanks very much!