So I'm trying to figure out why I'm not getting data from

tap-github

. I'm running both

elt

and

invoke

in debug mode, and I'm getting a lot of output like this:

DEBUG Visiting CatalogNode.PROPERTY at '.streams[22].schema.properties.commit.properties.committer.properties.date'.

DEBUG Skipping node at '.streams[22].schema.properties.commit.properties.committer.properties.date.format'

In the docs for the tap, it says I need to

--discover > properties.json

and I'm wondering if these outputs are because of not running this command? I've set up the proper values for the config file and have double checked in the tap's config file.

Those output messages are just Meltano-internal debugging information, nothing to be concerned about 🙂 Manually discovering the catalog is not necessary, since Meltano handles entity selection for you: https://meltano.com/docs/integration.html#selecting-entities-and-attributes-for-extraction Since you're not getting any data, can you verify that all entities are selected using

meltano select tap-github --list --all

?

Running that command yields a long list where all of them are either

selected

or

automatic

All right, then we're good on the entity selection/catalog front.

catalog

,

discover

,

properties

, and

state

Ah, that explains it!

properties

is the legacy name of the argument used to provide the catalog file path, while

catalog

is the modern name. Meltano needs to know which one a tap supports to invoke it correctly. If it thinks it supports both,

catalog

is preferred, so in your case

tap-github

is getting called with

--catalog=path/to/catalog.json

. However, according to the readme at https://github.com/singer-io/tap-github, it actually only supports

--properties

, so it's currently not getting a catalog at all and therefore it's not extracting anything!

Specifically this output:

meltano          | Running extract & load...
meltano          | No state was found, complete import.
tap-github       | INFO Starting sync of repository: CoopTang/market2table-be
meltano          | Incremental state has been updated at 2020-08-25 20:40:06.243804.
meltano          | Extract & load complete!
meltano          | Transformation skipped.

Removing

catalog

from

capabilities

should fix it 🙂

Perfect.

🎉