It seems like when I run the pipeline, the field n...
# troubleshootinga
Ayush
07/01/2025, 12:11 AMIt seems like when I run the pipeline, the field names in my jsonl file (which follow salesforce mappings) automatically change to lowercase (“Id” to “id”, “Name” to “name”, and “AccountNumber” to “accountnumber”) . I tried adding Meltano Map Transformer which has a “stream-maps” capability, but not sure the config for this. Could this be a possible workaround?
e
Edgar Ramírez (Arch.dev)
07/01/2025, 12:54 AMHave you looked at https://docs.meltano.com/guide/mappers/?
a
Ayush
07/01/2025, 1:22 AMwill take a look thanks Edgar
Ayush
07/01/2025, 7:15 AMhello
Ayush
07/01/2025, 7:16 AMI dont get whats causing this issue, but it seems like whenever salesforce is involed, the field names automatically become lower-case. I tried regular ETL from salesforce to mongo, and it wrote field names in lower case (like “name”, “accountnumber” etc).
Ayush
07/01/2025, 7:18 AMI think it should be upper case (default salesforce field names) because someone else in my organization tried the ETL and got normal field names in Salesforce
a
Andy Carter
07/01/2025, 7:29 AMCan you share your meltano.yml to see exact config of taps and targets & variants? What meltano command are you running?
a
Ayush
07/01/2025, 7:41 AMit seems like the tap-spreadsheets-anywhere extractor changes field names during the ETL pipeline
Ayush
07/01/2025, 7:42 AMit makes everything lower case
a
Andy Carter
07/01/2025, 12:10 PMYes that seems like generic behaviour for that tap regardless of source file type
https://github.com/search?q=repo%3Aets%2Ftap-spreadsheets-anywhere%20lower&type=code
a
Ayush
07/01/2025, 10:25 PMthanks for replying Andy
Ayush
07/01/2025, 10:30 PMLet me send you my meltano.yml file. However, I tried salesforce -> target-singer-jsonl, then tap-singer-jsonl) which does not mess with the upper case letters. Target-salesforce still throws incompatible schema error
Ayush
07/01/2025, 10:31 PMplugins:
extractors:
- name: tap-salesforce
variant: meltanolabs
pip_url: git+https://github.com/MeltanoLabs/tap-salesforce.git@v1.9.0
config:
api_type: BULK2
client_id: 3MVG91oqviqJKoEEieszTYeWJpE0cyVixkTb2ma1pvOpRdGSTJ_zUk.TwBvX25o2RAwTvnQBb9JWXtccXW6ar
start_date: ‘2003-01-01T000000Z’
username: ayushmahale27@gmail.com
select:
- Account.*
- name: tap-spreadsheets-anywhere
variant: ets
pip_url: git+https://github.com/ets/tap-spreadsheets-anywhere.git
config:
tables:
- name: Account
path: file://output
pattern: Account.jsonl
format: jsonl
key_properties: [“Id”]
start_date: “2003-01-01T000000Z”
loaders:
- name: target-jsonl
variant: andyh1203
pip_url: target-jsonl
- name: target-salesforce
variant: dan-ladd
pip_url: git+https://github.com/dan-ladd/target-salesforce.git
config:
action: insert
allow_failures: true
username: ayushmahale27@gmail.com
Ayush
07/01/2025, 11:28 PMHi @Andy Carter how do we disable this auto updating field names by tap-spreadsheets-anywhere
a
Andy Carter
07/02/2025, 7:17 AMYou cannot unless you fork that repo and change the code appropriate for you. That is not a configurable on the tap.
Andy Carter
07/02/2025, 7:17 AMWhat are you trying to achieve? salesforce to jsonl, then jsonl -> salesforce?
Andy Carter
07/02/2025, 7:19 AMIf you want to share the exact meltano commands and then the full error stack trace in this thread?
a
Ayush
07/03/2025, 5:48 AMGot it resolved with singer, thank you!
🙌 2
3 Views