Hi Team,
I am new to meltano and having an issue w...
# troubleshootings
srinivasula_reddy_yennapusa
04/27/2022, 6:41 AMHi Team,
I am new to meltano and having an issue while executing “meltano elt tap-twitter target-jsonl --catalog .meltano/extractors/tap-twitter/venv/catalog.json”
Can anyone please help me with the error.
Appreciate your help.
a
atharh
04/27/2022, 8:18 AMHi @srinivasula_reddy_yennapusa. To remove the warning, you would need to add
catalogcapabilitiesmeltano.yml Copy code
plugins:
extractors:
- name: tap-twitter
capabilities:
- catalog
- discover
- states
srinivasula_reddy_yennapusa
04/27/2022, 9:35 AMThanks for your reply.
I have already added these capabilities.
Along with this warning, I am getting “INFO No properties were selected” because of which data load is not happening.
srinivasula_reddy_yennapusa
04/27/2022, 10:37 AMHi @atharh, can you please guide me in resolving this issue.
a
atharh
04/27/2022, 11:27 AM@srinivasula_reddy_yennapusa You can refer to the Select entities and attributes to extract section in the Getting Started guide to see how you can list the entities supported by
tap-twitters
srinivasula_reddy_yennapusa
04/27/2022, 11:40 AM@atharh, even after the selection I am getting the same issue.
[warning ] A catalog file was found, but it will be ignored as the extractor does not advertise the
catalogpropertiessrinivasula_reddy_yennapusa
04/27/2022, 1:17 PM@atharh, after correctly adding capabilities (as you mentioned) the warning has gone but the error “INFO No properties were selected” is still there when I run “meltano elt tap-twitter target-jsonl”.
p
pat_nadolny
04/27/2022, 1:23 PM@srinivasula_reddy_yennapusa if you run
meltano select tap-twitter --lists
srinivasula_reddy_yennapusa
04/27/2022, 1:25 PM@pat_nadolny, I guess no
p
pat_nadolny
04/27/2022, 1:27 PM@srinivasula_reddy_yennapusa try following the getting started guide link that Athar sent earlier (https://docs.meltano.com/getting-started#select-entities-and-attributes-to-extract) for making selections now that your have your catalog/properties fixed
a
atharh
04/27/2022, 1:31 PMmeltano.ymls
srinivasula_reddy_yennapusa
04/27/2022, 1:40 PM@pat_nadolny, I have selected “blocks” entity , and this is showing only under enabled patters but not under selected attributes.
srinivasula_reddy_yennapusa
04/27/2022, 1:41 PMshould I make any changes to catalog.json aswell?
p
pat_nadolny
04/27/2022, 1:50 PMoh wait thats weird, it should have listed attributes under the
Selected attributess
srinivasula_reddy_yennapusa
04/27/2022, 1:57 PM@pat_nadolny , sharing the .yml file after removing credentials
srinivasula_reddy_yennapusa
04/27/2022, 2:01 PMNot sure of the tap variant , I have installed via :
Copy code
meltano add --custom extractor tap-twitterp
pat_nadolny
04/27/2022, 2:06 PM@srinivasula_reddy_yennapusa thanks for sharing, are you using custom catalogs and state files? Meltano usually handles the creation of those based on the settings in your meltano.yml and state in the system database. I think right now youre overriding the default behavior by setting
catalog: .meltano/extractors/tap-twitter/venv/catalog.jsons
srinivasula_reddy_yennapusa
04/27/2022, 2:14 PM@pat_nadolny, After removing catalog and state from meltano.yml file as you mentioned selected attributes got populated. Thanks.
but when I run “meltano elt tap-twitter target-jsonl” I am getting the same earlier issue.
p
pat_nadolny
04/27/2022, 2:21 PMok progress! What does
meltano select tap-twitter --lists
srinivasula_reddy_yennapusa
04/27/2022, 2:25 PMSelected attributes are getting populated
p
pat_nadolny
04/27/2022, 2:31 PMHmm thats weird - @atharh @edgar_ramirez_mondragon any ideas?
pat_nadolny
04/27/2022, 2:34 PM@srinivasula_reddy_yennapusa can you try removing
propertiess
srinivasula_reddy_yennapusa
04/27/2022, 2:37 PMSame error “INFO No properties were selected” even after removing properties from meltano.yml
a
atharh
04/27/2022, 2:49 PMI am wondering if the
INFOs
srinivasula_reddy_yennapusa
04/27/2022, 2:53 PMyes, all the Selected attributes are getting populated.
But concern is when I run “meltano elt tap-twitter target-jsonl” I am getting this issue and load is not happening to output folder.
e
edgar_ramirez_mondragon
04/27/2022, 2:58 PM@srinivasula_reddy_yennapusa can you remove the
catalogedgar_ramirez_mondragon
04/27/2022, 2:59 PM(it may be preventing meltano from calling the tap with
--propertiess
srinivasula_reddy_yennapusa
04/27/2022, 3:08 PMhi @edgar_ramirez_mondragon, thanks , I have removed catalog and added properties to the capabilities. And this issue is gone “INFO No properties were selected”.
But the data load is not happening, when I run “meltano elt tap-twitter target-jsonl” to the output folder.
Please find below the updated .yml file.
srinivasula_reddy_yennapusa
04/27/2022, 3:09 PMmessage has been deleted
e
edgar_ramirez_mondragon
04/27/2022, 3:20 PMit looks like you also removed your
selectmeltano select tap-twitter "*" "*"s
srinivasula_reddy_yennapusa
04/27/2022, 3:24 PMEven after adding meltano select tap-twitter “*” “*” , data load is not happening, when I run “meltano elt tap-twitter target-jsonl” to the output folder.
e
edgar_ramirez_mondragon
04/27/2022, 3:44 PMwhat's the output of
meltano select tap-twitter --list --alls
srinivasula_reddy_yennapusa
04/27/2022, 3:47 PMwe are getting some automatic Selected attributes.
srinivasula_reddy_yennapusa
04/28/2022, 10:31 AMHi @edgar_ramirez_mondragon, Now if I execute “meltano select tap-twitter --list --all”, we are getting all the Selected attributes as automatic.
srinivasula_reddy_yennapusa
04/28/2022, 10:35 AMAnd if I run “meltano config tap-twitter test” , I am getting “Plugin configuration is invalid
No RECORD message received”
srinivasula_reddy_yennapusa
04/28/2022, 10:35 AMmessage has been deleted
srinivasula_reddy_yennapusa
04/28/2022, 5:19 PMcan you please help me with this.
srinivasula_reddy_yennapusa
05/04/2022, 11:42 AMHi @edgar_ramirez_mondragon and @pat_nadolny, can you please help me on this issue.
e
edgar_ramirez_mondragon
05/04/2022, 3:18 PMhi @srinivasula_reddy_yennapusa. what's the output of
meltano invoke tap-twitters
srinivasula_reddy_yennapusa
05/05/2022, 6:09 AMHi @edgar_ramirez_mondragon, attached is the output for meltano invoke tap-twitter
srinivasula_reddy_yennapusa
05/05/2022, 6:10 AMAlso If I run meltano config tap-twitter test, I am getting “Plugin configuration is invalid
No RECORD message received” as mentioned above.
srinivasula_reddy_yennapusa
05/05/2022, 6:11 AMmessage has been deleted
srinivasula_reddy_yennapusa
05/05/2022, 6:16 AMAlso sharing the meltano.yml file for your reference (removed the credentials)
e
edgar_ramirez_mondragon
05/05/2022, 4:23 PM@srinivasula_reddy_yennapusa
meltano.ymlmeltano --log-level=debug config tests
srinivasula_reddy_yennapusa
05/06/2022, 7:48 AMHi @edgar_ramirez_mondragon, Please find attached error log for the above command.
FYI : ‘MELTANO_EXTRACTOR_VARIANT’: ‘original’
srinivasula_reddy_yennapusa
05/11/2022, 12:14 PMHi @edgar_ramirez_mondragon,
can you please check the above log and let me know in case there is any issue.
3 Views