hey team! I saw this PR for adding jql filtering to the

tap-jira

plugin. I don't see those fields described on the meltano hub page and it hasn't seemed to be working when I tested it locally. has this feature been released yet?

it hasn't seemed to be working when I tested it locally.

How are you configuring the setting?

has this feature been released yet?

The plugin installs the

main

branch, so yeah it's been available since the PR merged.

I don't see those fields described on the meltano hub page

For some reason, we were skipping metadata auto-updates for that plugin, I'm looking into it.

here's my configuration, where

PROJ

is a project that exists in my jira:

extractors:
          - name: tap-jira
            config:
              domain: "..."
              email:"..."
              start_date: "-2m" # past two minutes
              end_date: "2025-03-19"
            select:
              - issues.key
              - issues.created
              - issues.fields.created
              # - issues.*
            stream_options:
              issues:
                jql: "project=PROJ"

I'm relatively new to using Meltano so it could be a configuration issue on my end. this may or may not be related but when I run

meltano invoke tap-jira > raw_output.json

with the above configuration, only

id

and

key

are present in the record field despite what I've passed into the

select

option. when I pass

issues.*

into the

select

on the other hand, I get all the fields but then the extractor fails with a broken pipe error when I try to use the tap with a target (

target-csv

). my jira instance has hundreds of custom fields so a huge warning is emitted to stdout, similar to this issue. is it possible to somehow suppress that warning?

stream_options

should be nested in

config

ah indentation errors facepalm thanks that fixed the jql issue! I'm still running into the select issue I mentioned above though 🤔

do see fields greyed out if you run

meltano select tap-jira --list --all

?

nope, they're all either excluded or selected. none of them seem to be grey or labelled as

unsupported

I see. Is

issues.created

among the excluded fields?

it says it's selected, bit hard to parse from this chunk of text without the colours when pasting into slack haha

[selected   ] issues.fields.created
        [excluded   ] issues.fields.creator
        [excluded   ] issues.fields.creator.accountId
        [excluded   ] issues.fields.creator.accountType
        [excluded   ] issues.fields.creator.active
        [excluded   ] issues.fields.creator.avatarUrls
        [excluded   ] issues.fields.creator.avatarUrls.16x16
        [excluded   ] issues.fields.creator.avatarUrls.24x24
        [excluded   ] issues.fields.creator.avatarUrls.32x32
        [excluded   ] issues.fields.creator.avatarUrls.48x48
        [excluded   ] issues.fields.creator.displayName
        [excluded   ] issues.fields.creator.emailAddress
        [excluded   ] issues.fields.creator.self
        [excluded   ] issues.fields.creator.timeZone
        [excluded   ] issues.fields.description
        [excluded   ] issues.fields.description.content
        [excluded   ] issues.fields.description.text
        [excluded   ] issues.fields.description.type
        [excluded   ] issues.fields.description.version
        [excluded   ] issues.fields.duedate
        [excluded   ] issues.fields.editmeta
        [excluded   ] issues.fields.environment
        [excluded   ] issues.fields.environment.content
        [excluded   ] issues.fields.environment.text
        [excluded   ] issues.fields.environment.type
        [excluded   ] issues.fields.environment.version
        [excluded   ] issues.fields.fixVersions
        [excluded   ] issues.fields.histories
        [excluded   ] issues.fields.id
        [excluded   ] issues.fields.issuelinks
        [excluded   ] issues.fields.issuerestriction
        [excluded   ] issues.fields.issuetype
        [excluded   ] issues.fields.issuetype.avatarId
        [excluded   ] issues.fields.issuetype.description
        [excluded   ] issues.fields.issuetype.entityId
        [excluded   ] issues.fields.issuetype.hierarchyLevel
        [excluded   ] issues.fields.issuetype.iconUrl
        [excluded   ] issues.fields.issuetype.id
        [excluded   ] issues.fields.issuetype.name
        [excluded   ] issues.fields.issuetype.self
        [excluded   ] issues.fields.issuetype.subtask
        [excluded   ] issues.fields.key
        [excluded   ] issues.fields.labels
        [excluded   ] issues.fields.lastViewed
        [excluded   ] issues.fields.parent
        [excluded   ] issues.fields.parent.fields
        [excluded   ] issues.fields.parent.fields.issuetype
        [excluded   ] issues.fields.parent.fields.issuetype.avatarId
        [excluded   ] issues.fields.parent.fields.issuetype.description
        [excluded   ] issues.fields.parent.fields.issuetype.entityId
        [excluded   ] issues.fields.parent.fields.issuetype.hierarchyLevel
        [excluded   ] issues.fields.parent.fields.issuetype.iconUrl
        [excluded   ] issues.fields.parent.fields.issuetype.id
        [excluded   ] issues.fields.parent.fields.issuetype.name
        [excluded   ] issues.fields.parent.fields.issuetype.self
        [excluded   ] issues.fields.parent.fields.issuetype.subtask
        [excluded   ] issues.fields.parent.fields.priority
        [excluded   ] issues.fields.parent.fields.priority.iconUrl
        [excluded   ] issues.fields.parent.fields.priority.id
        [excluded   ] issues.fields.parent.fields.priority.name
        [excluded   ] issues.fields.parent.fields.priority.self
        [excluded   ] issues.fields.parent.fields.status
        [excluded   ] issues.fields.parent.fields.status.description
        [excluded   ] issues.fields.parent.fields.status.iconUrl
        [excluded   ] issues.fields.parent.fields.status.id
        [excluded   ] issues.fields.parent.fields.status.name
        [excluded   ] issues.fields.parent.fields.status.self
        [excluded   ] issues.fields.parent.fields.status.statusCategory
        [excluded   ] issues.fields.parent.fields.status.statusCategory.colorName
        [excluded   ] issues.fields.parent.fields.status.statusCategory.id
        [excluded   ] issues.fields.parent.fields.status.statusCategory.key
        [excluded   ] issues.fields.parent.fields.status.statusCategory.name
        [excluded   ] issues.fields.parent.fields.status.statusCategory.self
        [excluded   ] issues.fields.parent.fields.summary
        [excluded   ] issues.fields.parent.id
        [excluded   ] issues.fields.parent.key
        [excluded   ] issues.fields.parent.self
        [excluded   ] issues.fields.priority
        [excluded   ] issues.fields.priority.iconUrl
        [excluded   ] issues.fields.priority.id
        [excluded   ] issues.fields.priority.name
        [excluded   ] issues.fields.priority.self
        [excluded   ] issues.fields.progress
        [excluded   ] issues.fields.progress.progress
        [excluded   ] issues.fields.progress.total
        [excluded   ] issues.fields.project
        [excluded   ] issues.fields.project.avatarUrls
        [excluded   ] issues.fields.project.avatarUrls.16x16
        [excluded   ] issues.fields.project.avatarUrls.24x24
        [excluded   ] issues.fields.project.avatarUrls.32x32
        [excluded   ] issues.fields.project.avatarUrls.48x48
        [excluded   ] issues.fields.project.id
        [excluded   ] issues.fields.project.key
        [excluded   ] issues.fields.project.name
        [excluded   ] issues.fields.project.projectTypeKey
        [excluded   ] issues.fields.project.self
        [excluded   ] issues.fields.project.simplified
        [excluded   ] issues.fields.reporter
        [excluded   ] issues.fields.reporter.accountId
        [excluded   ] issues.fields.reporter.accountType
        [excluded   ] issues.fields.reporter.active
        [excluded   ] issues.fields.reporter.avatarUrls
        [excluded   ] issues.fields.reporter.avatarUrls.16x16
        [excluded   ] issues.fields.reporter.avatarUrls.24x24
        [excluded   ] issues.fields.reporter.avatarUrls.32x32
        [excluded   ] issues.fields.reporter.avatarUrls.48x48
        [excluded   ] issues.fields.reporter.displayName
        [excluded   ] issues.fields.reporter.emailAddress
        [excluded   ] issues.fields.reporter.self
        [excluded   ] issues.fields.reporter.timeZone
        [excluded   ] issues.fields.resolution
        [excluded   ] issues.fields.resolution.description
        [excluded   ] issues.fields.resolution.id
        [excluded   ] issues.fields.resolution.name
        [excluded   ] issues.fields.resolution.self
        [excluded   ] issues.fields.resolutiondate
        [excluded   ] issues.fields.security
        [excluded   ] issues.fields.status
        [excluded   ] issues.fields.status.description
        [excluded   ] issues.fields.status.iconUrl
        [excluded   ] issues.fields.status.id
        [excluded   ] issues.fields.status.name
        [excluded   ] issues.fields.status.self
        [excluded   ] issues.fields.status.statusCategory
        [excluded   ] issues.fields.status.statusCategory.colorName
        [excluded   ] issues.fields.status.statusCategory.id
        [excluded   ] issues.fields.status.statusCategory.key
        [excluded   ] issues.fields.status.statusCategory.name
        [excluded   ] issues.fields.status.statusCategory.self
        [excluded   ] issues.fields.statuscategorychangedate
        [excluded   ] issues.fields.subtasks
        [excluded   ] issues.fields.summary
        [excluded   ] issues.fields.timeestimate
        [excluded   ] issues.fields.timeoriginalestimate
        [excluded   ] issues.fields.timespent
        [excluded   ] issues.fields.timetracking
        [excluded   ] issues.fields.updated
        [excluded   ] issues.fields.versions
        [excluded   ] issues.fields.votes
        [excluded   ] issues.fields.votes.hasVoted
        [excluded   ] issues.fields.votes.self
        [excluded   ] issues.fields.votes.votes
        [excluded   ] issues.fields.watches
        [excluded   ] issues.fields.watches.isWatching
        [excluded   ] issues.fields.watches.self
        [excluded   ] issues.fields.watches.watchCount
        [excluded   ] issues.fields.worklog
        [excluded   ] issues.fields.workratio
        [automatic  ] issues.id
        [selected   ] issues.key
        [excluded   ] issues.self
        [excluded   ] issues.updated

Ok, so

issues.fields.created

is selected and you're not seeing it in the output?

yup exactly, here's an example of one of the output records from

meltano invoke tap-jira > raw_output.json

with the above schema

{"type":"SCHEMA","stream":"issues","schema":{"properties":{"id":{"type":["string","null"]},"key":{"type":["string","null"]},"created":{"type":["string","null"]}},"type":"object","$schema":"<https://json-schema.org/draft/2020-12/schema>"},"key_properties":["id"],"bookmark_properties":["id"]}
{"type":"RECORD","stream":"issues","record":{"id":"3848297","key":"PROJ-92"},"time_extracted":"2025-03-19T14:35:42.416084+00:00"}

I see, can you try adding

issues.fields.*

to your

select

list?

all the

issues.fields

are selected when I run

meltano select tap-jira --list --all

but I get the same output:

{"type":"SCHEMA","stream":"issues","schema":{"properties":{"id":{"type":["string","null"]},"key":{"type":["string","null"]},"created":{"type":["string","null"]}},"type":"object","$schema":"<https://json-schema.org/draft/2020-12/schema>"},"key_properties":["id"],"bookmark_properties":["id"]}
{"type":"RECORD","stream":"issues","record":{"id":"3848297","key":"PROJ-92"},"time_extracted":"2025-03-19T14:54:34.702888+00:00"}

Ok, so looking at https://developer.atlassian.com/cloud/jira/platform/rest/v3/api-group-issue-search/#api-rest-api-3-search-get, is doesn't seem like

fields.created

is part of the response so it's likely impossible to get it and the tap's implementation is incorrect

hm I queried the endpoint with my jira instance and

created

was present within

fields

along with all my other custom fields. I think the jira docs may only show a subset of what should be present in the API response? since fields like status, assignee, etc. aren't present in the jira docs example either

> hm I queried the endpoint The

/rest/api/3/search

endpoint?

yup, something like this is what I queried:

https://<domain>.<http://atlassian.net/rest/api/3/search?jql=key=PROJ-63|atlassian.net/rest/api/3/search?jql=key=PROJ-63>

Ok, gotcha. Let me actually try it.

are you able to get only

key

and

created

? if you perform a select like the following:

select:
    - issues.key
    - issues.fields.created

issue.fields

returns all the fields present but this causes some issues I mentioned above here when used in conjunction with a target. I'm trying to see if it's possible to only select certain sub properties from

fields

rather than everything all at once

Ok, https://github.com/MeltanoLabs/tap-jira/pull/144 should let do this:

select:
    - issues.key
    - issues.created

hm is it currently not possible to use

select

with

fields

subproperties? I think the ideal solution here would be that you could use

select

with any

issues.fields.X

value (or for any other field with subproperties for that matter).

hm is it currently not possible to use

select

with

fields

subproperties?

It is possible, the issue is that is also selects custom fields and it's not possible to deselect them because they're not in the discovered catalog for the tap. One common workaround for that is to use the schema extra, but that doesn't work here because they're nested, so if you add them with that trick, you end up removing all other

fields

sub-properties, which defeats the purpose of the workaround. If you're curious, that logic lives here: https://github.com/meltano/meltano/blob/7e05ed031c68e97434d1e2c98ada91f33bdd1e04/src/meltano/core/plugin/singer/catalog.py#L550-L564.

gotcha thanks for the resources! so to make sure I'm understanding correctly,

issue.fields

can be used to select all subproperties but atm it is not possible to select specific subproperties like

issue.fields.created

or

issue.fields.assignee

without modifying the tap and exposing those fields?

without modifying the tap and exposing those fields?

Specifically exposing them at the top-level of the record and schema. And this is only because of the particulars of tap-jira: fields that are nested and dynamic.

thanks for the clarification!

Another approach we can try is dynamically discovering the schema using https://developer.atlassian.com/cloud/jira/platform/rest/v3/api-group-issue-fields/#api-rest-api-3-field-get.

I think this could be a good idea, I'm still getting familiar with the tap-jira code and Meltano itself but let me know if there's any way I can help with this

thanks! do you have an estimate on the level of effort that would be involved to implement this?