I posted a question yesterday but didn't get a res...
# troubleshooting
w
I posted a question yesterday but didn't get a response so trying again. Using tap-postgres and target-postgres. Pulling data from a table called "production_salesforce.Account" and putting it into "dl_salesforce.account" using schema mapping. "Id" is the primary key. Getting an error after the data is loaded when it is trying to create an index on a column that doesn't exist: CREATE INDEX IF NOT EXISTS i_account_I on dl_salesforce.account (I); Relevant part of meltano.yml: schema_mapping: production_salesforce: target_schema: dl_salesforce indices: Account: Id
So, it appears that all columns are being pulled into the target schema and not just the ones I have specified under the extractor: select: - production_salesforce-Account.Id - production_salesforce-Account.LastModifiedDate - production_salesforce-Account.Name Additionally, it is changing all column names to lowercase when creating the target table. It is also making the target table name lowercase. Based on that behavior, I changed the following in the schema mapping to a lowercase column name: indices: account: id That seems to have resulted in a successful load. I now need to figure out the extractor grabbing all columns rather than the ones I requested. I don't recall seeing anything in the documentation about forcing lower case so either I am still doing something wrong there or maybe a line should be added about that.
This bug still seems to be an issue with tap-postgres: https://gitlab.com/meltano/meltano/-/issues/603 When I run
meltano select tap-postgres --list
I am seeing just the columns I have selected. When I run the elt, all of the tables columns (not just the ones I selected) are being created and populated with data in the target.
Known issue that was discussed here a month ago: https://meltano.slack.com/archives/C01TCRBBJD7/p1652253018834649 Still without a fix. I don't think I can do the view work around that is mentioned due to a view not having a primary key. A materialized view isn't an option due to size.