I’ve run into an issue with properly selecting fields to load via custom plugins that I’m hoping someone might be able to help me unblock. I was having issues getting both

tap-jira

and

tap-typeform

to work, and after digging through their respective sources I was able to confirm that no fields were syncing because nothing was marked as

selected

in the respective catalogs (when I temporarily removed the lines that limited the syncs only to selected fields, everything loaded fine, so I know that this is the issue). Where I’m stuck is in figuring out why those fields aren’t selected. I’ve tried varying approaches to including them in

meltano.yml

, including patterns that have worked fine for other taps. I’ve also tried combinations of including them both the yaml and/or the

meltano elt

command. Has anyone else run into something similar or have ideas for what I should try next?

@nick_hamlin Can you share the output of

meltano select tap-jira --list --all

and

meltano select tap-typeform --list --all

?

Sure can, and I’d checked their as well. Weirdly, those outputs lead me to believe that at least some things are selected. Here’s tap-jira: ``` [selected ] changelogs.author [automatic] changelogs.author.accountId [automatic] changelogs.author.active [automatic] changelogs.author.avatarUrls [automatic] changelogs.author.displayName [automatic] changelogs.author.emailAddress [automatic] changelogs.author.key [automatic] changelogs.author.name [automatic] changelogs.author.self [automatic] changelogs.author.timeZone [selected ] changelogs.created [selected ] changelogs.historyMetadata [automatic] changelogs.historyMetadata.activityDescription [automatic] changelogs.historyMetadata.activityDescriptionKey [automatic] changelogs.historyMetadata.actor [automatic] changelogs.historyMetadata.actor.avatarUrl [automatic] changelogs.historyMetadata.actor.displayName [automatic] changelogs.historyMetadata.actor.displayNameKey [automatic] changelogs.historyMetadata.actor.id [automatic] changelogs.historyMetadata.actor.type [automatic] changelogs.historyMetadata.actor.url [automatic] changelogs.historyMetadata.cause [automatic] changelogs.historyMetadata.cause.avatarUrl [automatic] changelogs.historyMetadata.cause.displayName [automatic] changelogs.historyMetadata.cause.displayNameKey [automatic] changelogs.historyMetadata.cause.id [automatic] changelogs.historyMetadata.cause.type [automatic] changelogs.historyMetadata.cause.url [automatic] changelogs.historyMetadata.description [automatic] changelogs.historyMetadata.descriptionKey [automatic] changelogs.historyMetadata.emailDescription [automatic] changelogs.historyMetadata.emailDescriptionKey [automatic] changelogs.historyMetadata.extraData [automatic] changelogs.historyMetadata.generator [automatic] changelogs.historyMetadata.generator.avatarUrl [automatic] changelogs.historyMetadata.generator.displayName [automatic] changelogs.historyMetadata.generator.displayNameKey [automatic] changelogs.historyMetadata.generator.id [automatic] changelogs.historyMetadata.generator.type [automatic] changelogs.historyMetadata.generator.url [automatic] changelogs.historyMetadata.type [automatic] changelogs.id [selected ] changelogs.issueId [selected ] changelogs.items [selected ] issue_comments.author [automatic] issue_comments.author.accountId [automatic] issue_comments.author.active [automatic] issue_comments.author.avatarUrls [automatic] issue_comments.author.displayName [automatic] issue_comments.author.emailAddress [automatic] issue_comments.author.key [automatic] issue_comments.author.name [automatic] issue_comments.author.self [automatic] issue_comments.author.timeZone [selected ] issue_comments.body [selected ] issue_comments.created [automatic] issue_comments.id [selected ] issue_comments.issueId [selected ] issue_comments.jsdPublic [selected ] issue_comments.properties [selected ] issue_comments.renderedBody [selected ] issue_comments.self [selected ] issue_comments.updateAuthor [automatic] issue_comments.updateAuthor.accountId [automatic] issue_comments.updateAuthor.active [automatic] issue_comments.updateAuthor.avatarUrls [automatic] issue_comments.updateAuthor.displayName [automatic] issue_comments.updateAuthor.emailAddress [automatic] issue_comments.updateAuthor.key [automatic] issue_comments.updateAuthor.name [automatic] issue_comments.updateAuthor.self [automatic] issue_comments.updateAuthor.timeZone [selected…

after digging through their respective sources I was able to confirm that no fields were syncing because nothing was marked as 

selected

 in the respective catalogs

Did you dump the catalog file using

meltano invoke --dump=catalog <tap>

? Do the

selected

keys and values in there match the output of

meltano select --list --all

? Or is nothing selected as you found earlier?

I did that my tweaking the tap code to dump the catalog contents to the logs, but your way is probably far better 🙂 Let me try that approach and I’ll circle back!

@nick_hamlin That's very odd. Can you share the

meltano.yml

custom plugin definitions for both taps? I wonder if the

capabilities

are defined incorrectly

I was wondering that too (based on the answer to a previous question you helped me with). Here’s what I currently have for tap-jira:

- name: tap-jira
    pip_url: tap-jira
    namespace: tap-jira
    variant: singer-io
    capabilities:
      - discover
      - catalog
      - state
      - properties
    config:
      start_date: "2020-01-01"
      username: REDACTED
      password: REDACTED
      base_url: REDACTED
      user_agent: REDACTED

Ah, you're listing both

catalog

and

properties

but these taps seem to only support the latter!

ah! I didn’t have properties in there originally but added it after doing some investigating there

I think it'll work if you drop

catalog

and leave

properties

but I didn’t realize that including both might cause problems! let me give that a shot

as usual, @douwe_maan FTW! That was it

tap-typeform has the same line 🙂

good to know - won’t get tripped up by that one again 🙂

woohoo! I was just trying

tap-jira

last night as well and this seems to have gotten me a step closer!

Glad we figured it out! Perhaps you can file an issue about warning if both catalog and properties are specified in capabilities? And/Or Mentioning in the docs how to determine whether to use catalog or properties based on the source

@andrew_stewart - I’ve got mine running now, happy to help if you get stuck @douwe_maan - you bet, thanks again!

im just trying to figure out what user_agent should be

@nick_hamlin Also consider contributing the tap-jira definition now that you've got it working: https://meltano.com/docs/contributor-guide.html#discoverable-plugins

gotcha. Ok, I was attributing an “Unauthorized for url” error to that, but it’s probably something else in my authentication.

Along those lines, something that’s not documented but I found hidden in a GH issue and have since confirmed is that you don’t need to use oauth for this to work, but you do need to generate an API token. Once you’ve done that, put the token in the

password

field in your config

yep

ugh yeah, that is annoying - I ran into a similar issue with tap-zendesk

Thank you! I'll review them this week