https://linen.dev logo
Join Slack
Powered by
does anyone have experience pulling custom fields ...
# plugins-general
d
does anyone have experience pulling custom fields from the Zendesk API using https://github.com/twilio-labs/twilio-tap-zendesk/?
in particular i'm wondering if there is any way to select fields that are nested within another field this seems to be how custom fields are accessed via the Zendesk API, e.g., user custom fields are found nested within the 
users.user_fields
object (docs)
e
Hi @daniel. What have you tried so far?
d
so far i've tried using 
users.user_fields.{custom_field_name}
but this seems to result in nothing being selected for that field, is there some other way in Meltano to index into nested objects using 
select
?
for compliance reasons i don't believe we'll be able to ingest the entire 
user_fields
object
e
Ok thanks for the info. What do you see when you run 
meltano select tap-zendesk --list --all
? Is the subfield selected?
j
We use it, it works. Example:
Copy code
select:
        - sla_policies.created_at
        - sla_policies.description
        - sla_policies.filter
        - sla_policies.filter.all
        - sla_policies.filter.any
        - ticket_comments.attachments
        - ticket_metrics.agent_wait_time_in_minutes
        - ticket_metrics.agent_wait_time_in_minutes.business
        - ticket_metrics.agent_wait_time_in_minutes.calendar
But not sure if any of them are custom fields. I can ask the owner if you would appreciate it
d
thanks @edgar_ramirez_mondragon and @jan_soubusta somehow i do actually see the specific fields that i want to select as being "selected", but they don't appear in the JSON output 
meltano select tap-zendesk --list --all
Copy code
...
        [selected ] users.user_fields.user_role
...
        [selected ] organizations.organization_fields.back_office_account_uuid
...
select list from Meltano YAML:
Copy code
select:
        - organizations.organization_fields.back_office_account_uuid
        - users.user_fields.user_role
however neither shows in the output when running 
meltano elt tap-zendesk target-jsonl
, for example:
Copy code
{"id": 18431587202459, "updated_at": "2023-09-01T00:01:24.000000Z"}
{"id": 18431665953307, "updated_at": "2023-09-01T00:03:23.000000Z"}
{"id": 18328316385563, "updated_at": "2023-09-01T00:04:53.000000Z"}
...
e
Ok. It might be worth trying using a custom catalog file: 1. Save the current catalog definition with 
meltano invoke --dump=catalog tap-zendesk > catalog.json
2. Edit the file manually to add metadata entries:
Copy code
{
          "breadcrumb": [
            "properties",
            "user_fields"
          ],
          "metadata": {
            "inclusion": "available",
            "selected": true
          }
        },
        {
          "breadcrumb": [
            "properties",
            "user_fields",
            "properties",
            "test_pii_field"
          ],
          "metadata": {
            "inclusion": "available",
            "selected": false
          }
        },
        {
          "breadcrumb": [
            "properties",
            "user_fields",
            "properties",
            "test_custom_field"
          ],
          "metadata": {
            "inclusion": "available",
            "selected": true
          }
        },
3. Use the custom catalog:
Copy code
plugins:
  extractors:
  - name: tap-zendesk
    catalog: catalog.json
I think another option is to add any custom fields to the schema:
Copy code
plugins:
  extractors:
  - name: tap-zendesk
    select:
    - "users"                                    # select the stream
    - "users.user_fields"                        # select the field
    - "users.user_fields.*"                      # select all subfields
    - "!users.user_fields.field_you_dont_want"   # deselect each unwanted fields
    schema:
      users:
        user_fields:
          type: object
          properties:
            field_you_want:
              type: [string, "null"]
            field_you_dont_want:
              type: [string, "null"]
j
My 2c: sometimes some taps can't process fields of rare data types. The custom schema suggested by Edgar should help in this case too. When I hit this issue, I clone the tap repo and add debug directly to the source code to find out what happens the a specific field 😉
d
thanks, @edgar_ramirez_mondragon will this override the entire schema for the 
user_fields
object, so that fields we don't specify will be ignored? if so, it seems we could do that and then select the entire object
e
will this override the entire schema for the 
user_fields
object, so that fields we don't specify will be ignored? if so, it seems we could do that and then select the entire object
Don't know at the moment, but it's worth trying!
d
this works!
we have another issue which is related to the one in the thread up above the situation is we want to ingest specific elements from an array field, and we cannot ingest all of them for compliance reasons, does anyone know if there's an easy way to do this? the field in question is the 
custom_fields
array from the Zendesk Tickets API (we'll have to filter based on the 
id
attribute): • https://developer.zendesk.com/documentation/ticketing/managing-tickets/creating-and-updating-tickets/#setting-custom-field-values • https://developer.zendesk.com/api-reference/ticketing/tickets/tickets/ cc @edgar_ramirez_mondragon @jan_soubusta
j
sorry, don't know, we ingest all. Do you really need to skip some? Personally, I would drop them later using dbt
d
yes, some of the fields contain PII and we aren't allowed to ingest them at all (even if we filter later in dbt)
e
I agree with Jan. Dropping items from an array field is complex. Your best option is to create a custom mapper plugin.
Open in Slack
PreviousNext
Powered by