https://linen.dev logo
Join Slack
Powered by
hey y’all, i need some assistance with a workflow ...
# troubleshooting
r
hey y’all, i need some assistance with a workflow i’m attempting to implement. it’s using 
tap-redshift
to query a table in order to enrich salesforce data via 
target-salesforce
. i’m having trouble mapping the redshift records to the expected salesforce object. if you have worked with 
target-salesforce
, especially with 
tap-redshift
as the extractor, let me know and i’d love some help. thanks!
p
Hey @rafa ! I know @dan_ladd built the target so he may have some hints. What issues are you running into so far? I would recommend using a mapper (see mapper tutorial) or the target's built in stream maps config if you need to rename field names to match the target's expectations. Here an example where I convert uppercase field names to lowercase since target-apprise requires lowercase.
d
Hey @rafa, thanks for using target-salesforce! I have some pending updates that have been pretty helpful. I'll try and get those merged in today/tomorrow. I'd be more than happy to jump on a call and help troubleshoot.
r
hey @pat_nadolny, i really appreciate that example! it helps me better understand how to use stream maps, in that there are two parts required in order to change the field name format. i.e. 
TITLE: null
& 
title: TITLE
hey @dan_ladd, thanks for the heads up that there are helpful changes coming! blob intensifies and also that you’re willing to work through it with me on a zoom call! that’s very generous of you. with pat’s hint and the changes coming, let me give it a go and if i’m still running into trouble, i’ll take you up on that offer ☺️
hm. maybe i’m doing something else incorrectly 🤔 
meltano.yaml
Copy code
- name: target-salesforce
        config:
          action: update
          is_sandbox: true
          stream_maps:
            account:
              __alias__: Account
              id: null
              Id: id
              resources: null
              Resources: resources
              group_organization_id: null
Copy code
Traceback (most recent call last):
   File "/project/.meltano/loaders/target-salesforce/venv/lib/python3.9/site-packages/target_salesforce/sinks.py", line 66, in _validate_schema_against_object
     validate_schema_field(
   File "/project/.meltano/loaders/target-salesforce/venv/lib/python3.9/site-packages/target_salesforce/utils/validation.py", line 26, in validate_schema_field
     raise InvalidStreamSchema(f"{field_name} does not exist in SF Object")
 target_salesforce.utils.exceptions.InvalidStreamSchema: group_organization_id does not exist in SF Object
d
@pat_nadolny, does 
group_organization_id: null
remove 
group_organization_id
from the record or just set the value to null?
p
It supposed to remove it (sdk stream maps docs) but I know there was a bug at one point 🤔
You can also try deselecting it on the tap side, so it never even gets pulled into the stream
r
makes sense. i can definitely do that during this research phase. it is a field that i will surface later on but wanted to first verify that i can get a simple workflow going, so that’s why it’s set to 
null
confused dog
Copy code
target_salesforce.utils.exceptions.InvalidStreamSchema: id does not exist in SF Object
it’s definitely user error at this point
d
Yea, so 
Id
, not 
id
. Seems the inline maps isn't removing those fields from the record
The target is very picky compared to database targets. If you want to see it work, you can try executing the target directly to insert a test account into your sandbox
p
If you run a 
meltano --environment=<your_env> config target-salesforce
what does it show (excluding creds)? I wonder if its properly picking up that config
^^ updated with environment
r
ah! great point @dan_ladd, i’ll try isolating the target with a json file. @pat_nadolny, here’s that config:
Copy code
{
  "client_id": "REDACTED",
  "client_secret": "REDACTED",
  "refresh_token": "REDACTED",
  "is_sandbox": true,
  "action": "update",
  "stream_maps": {
    "account": {
      "__alias__": "Account",
      "Id": "id",
      "Resources": "resources"
    }
  }
}
oh! it’s not picking up the 
id: null
?
p
ahh you know what, mine isnt either 😞 . I thought it was but its because that target doesnt care about extra fields being in the stream...testing more
r
It supposed to remove it (sdk stream maps docs) but I know there was a bug at one point 🤔
in this earlier comment, you mentioned that it was a bug at one point. i’m assuming that means it was fixed. could this be something that is fixed by updating to the latest version of meltano?
i’m on 
v2.4.0-python3.9
d
Hmm, if stream maps doesn't remove the field, maybe we could just have the target warn and pop off any fields that don't exist in the SF Object instead of failing.
p
@edgar_ramirez_mondragon any idea if were doing something wrong with stream maps here? It looks like setting fields to null isnt doing anything
^^ calling in support 😅
d
So I am validating the SCHEMA against the SF Account object here before any records are actually processed. So I'm guessing the inline mapping could be removing the field from records, but it's still in the schema
e
Meltano does remove any 
null
values from config, which is a problem for plugins that consider that a special value rather than simply missing. (#6382). A workaround has been implemented on the SDK side by allowing the string 
__NULL__
to be used as a null value, but it sits unreleased atm.
p
@dan_ladd I confirmed locally that if the stream map config is correct and the null removes the field, it also removes it from the schema so you should be good on your implementation
r
sorry, just to make sure i follow, this is assuming #6382 is released, yeah? i follow
p
To circle back after knowing all this - I think using select criteria to not include those fields that we're trying to drop is the move for now. Then once the SDK release is out with https://github.com/meltano/sdk/pull/1262 and updated in target-salesforce you can use the new 
"__NULL__"
syntax to exclude fields properly.
4 Views
Open in Slack
PreviousNext
Powered by