https://linen.dev logo
Join Slack
Powered by
Hey everyone, I'm trying to use the `tap-jira` wit...
# troubleshooting
j
Hey everyone, I'm trying to use the 
tap-jira
with 
target-bigquery
to give meltano a quick test. It works fine, but it seems some customfields are natively discarded when running the full pipeline :
Copy code
2023-11-20T16:56:38.841386Z [info     ] 2023-11-20 17:56:38,840 | WARNING  | tap-jira             | Properties ('fields.customfield_10190', 'fields.customfield_10191', 'fields.customfield_10192', 'fields.customfield_10193', 'fields.customfield_10194', 'fields.customfield_10195', 'fields.customfield_10196', 'fields.customfield_10197', 'fields.customfield_10110', 'fields.customfield_10198', 'fields.customfield_10111', 'fields.customfield_10199', 'fields.customfield_10114', 'fields.customfield_10104', 'fields.customfield_10105', 'fields.customfield_10106', 'fields.customfield_10107', 'fields.customfield_10108', 'fields.customfield_10109', 'fields.customfield_10180', 'fields.customfield_10181', 'fields.customfield_10182', 'fields.customfield_10183', 'fields.customfield_10184', 'fields.customfield_10185', 'fields.customfield_10065', 'fields.customfield_10186', 'fields.customfield_10187', 'fields.customfield_10220', 'fields.customfield_10188', 'fields.customfield_10101', 'fields.customfield_10189', 'fields.customfield_10102', 'fields.customfield_10223', 'fields.customfield_10103', 'fields.customfield_10219', 'fields.customfield_10170', 'fields.customfield_10171', 'fields.customfield_10172', 'fields.customfield_10173', 'fields.customfield_10174', 'fields.customfield_10175', 'fields.customfield_10176', 'fields.customfield_10177', 'fields.customfield_10210', 'fields.customfield_10178', 'fields.customfield_10211', 'fields.customfield_10212', 'fields.customfield_10179', 'fields.customfield_10213', 'fields.customfield_10203', 'fields.customfield_10204', 'fields.customfield_10207', 'fields.customfield_10209', 'fields.customfield_10160', 'fields.customfield_10162', 'fields.customfield_10163', 'fields.customfield_10164', 'fields.customfield_10166', 'fields.customfield_10167', 'fields.customfield_10200', 'fields.customfield_10201', 'fields.customfield_10168', 'fields.customfield_10202', 'fields.customfield_10169', 'fields.customfield_10159', 'fields.customfield_10154', 'fields.customfield_10155', 'fields.customfield_10156', 'fields.customfield_10157', 'fields.customfield_10158', 'fields.customfield_10260', 'fields.customfield_10141', 'fields.customfield_10143', 'fields.customfield_10144', 'fields.customfield_10266', 'fields.customfield_10267', 'fields.customfield_10137', 'fields.customfield_10258', 'fields.customfield_10259', 'fields.customfield_10138', 'fields.customfield_10092', 'fields.customfield_10251', 'fields.customfield_10132', 'fields.customfield_10011', 'fields.customfield_10133', 'fields.customfield_10012', 'fields.customfield_10134', 'fields.customfield_10256', 'fields.customfield_10257', 'fields.customfield_10247', 'fields.customfield_10126', 'fields.customfield_10127', 'fields.customfield_10240', 'fields.customfield_10086', 'fields.customfield_10241', 'fields.customfield_10120', 'fields.customfield_10242', 'fields.customfield_10243', 'fields.customfield_10089', 'fields.customfield_10245', 'fields.customfield_10124', 'fields.customfield_10246', 'fields.customfield_10118', 'fields.customfield_10119') were present in the 'issues' stream but not found in catalog schema. Ignoring.
So far I tried : • modifying the catalog to include some custom fields I want --> same result • using an 
extra
so it updates the catalog -> same result. • I do have the customfields when using 
meltano invoke tap-jira > out.jsonl
but i never got a working pipeline. • in 
meltano/run/tap-jira/tap-properties.json
I can see that no matter what I do, the custom field does not appear in schema. I'm pretty sure I can get this working but adding a line in https://github.com/MeltanoLabs/tap-jira/blob/1523ad89455bff9efb2158c73544297d6dded6d7/tap_jira/streams.py#L832 but there must be another way ?
I confirm editing the tap to add my custom_field works. Ideally i'd like to do this without editing the tap.
e
Hi @jean-baptiste_delafosse! tap-zendesk has similar custom fields that can be included by using Meltano's schema extra: See this thread for an example: https://www.linen.dev/s/meltano/t/15704334/does-anyone-have-experience-pulling-custom-fields-from-the-z#8b40beeb-c905-4464-8ae0-07de2d40715c
j
Ahh I tried to use it but I can see my yaml was quite different from what is displayed here. Thanks.
Ok it does work to some extent : Adding
Copy code
schema:
  issues:
      fields:
        type: object
        properties:
          customfield_10074:
            type: object
Does make it correctly synchronyzed but also discard all the customfields that are added by default in the native tap.
Adding
Copy code
schema:
  issues:
      fields:
        properties:
          customfield_10074:
            type: object
Does not add this custom field in my final table.
I'll keep using a fork for now.
25 Views
Open in Slack
PreviousNext
Powered by