Hello
Can anyone show me how can I use a tab from ...
# plugins-generalt
truc_nguyen
03/21/2021, 6:32 AMHello
Can anyone show me how can I use a tab from GitHub. I need to extract data from mssql and found this one on GitHub https://github.com/singer-io/tap-mssql. I try to add this to my Meltano docker ussing
Copy code
meltano add --custom extractorv
visch
03/21/2021, 2:24 PMhttps://github.com/wintersrd/pipelinewise-tap-mssql was recommended by @aaronsteers , I'm actually going to dive into this today. I'll get back with you and let you know how it goes!
t
truc_nguyen
03/21/2021, 3:33 PMAmazing! Can't wait to hear your good news, I really need this tap.
a
aaronsteers
03/21/2021, 5:09 PM@truc_nguyen - For background, the singer version of the MSSSQL tap was built on Closure, which is java based and also not directly executable in the normal way, due to intricacies in how lein/clojure are implemented. There are workarounds to this, but those take some effort and are not turnkey at this time. The wintersrd fork is newer and python based, so it does not carry the same difficulties.
v
visch
03/21/2021, 5:11 PMConfirmed it's working, had a little snag with the catalog file as I had to manually add FULL_TABLE sync to the tables I wanted but it works on a simple schema
visch
03/21/2021, 5:12 PM Copy code
INFO Beginning sync
INFO Preparing Catalog
INFO Fetching tables
INFO Tables fetched, fetching columns
INFO Columns Fetched
INFO Catalog ready
INFO selected-by-default: False
INFO database-name: dbo
INFO is-view: False
INFO replication-method: FULL_TABLE
INFO table-key-properties: []
INFO selected: True
INFO Need to sync abc
{"type": "STATE", "value": {"currently_syncing": "dbo-abc"}}
INFO Table abc proposes FULL_TABLE sync
INFO Table abc will use FULL_TABLE sync
INFO syncing abc full table
{"type": "SCHEMA", "stream": "dbo-abc", "schema": {"properties": {"id": {"inclusion": "available", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}}, "type": "object"}, "key_properties": []}
{"type": "ACTIVATE_VERSION", "stream": "dbo-abc", "version": 1616346750991}
{"type": "RECORD", "stream": "dbo-abc", "record": {"id": 1}, "version": 1616346750991, "time_extracted": "2021-03-21T17:12:30.996100Z"}
{"type": "RECORD", "stream": "dbo-abc", "record": {"id": 2}, "version": 1616346750991, "time_extracted": "2021-03-21T17:12:30.996100Z"}
{"type": "RECORD", "stream": "dbo-abc", "record": {"id": 3}, "version": 1616346750991, "time_extracted": "2021-03-21T17:12:30.996100Z"}
{"type": "RECORD", "stream": "dbo-abc", "record": {"id": 4}, "version": 1616346750991, "time_extracted": "2021-03-21T17:12:30.996100Z"}
INFO METRIC: {"type": "counter", "metric": "record_count", "value": 4, "tags": {"database": "dbo", "table": "abc"}}
{"type": "STATE", "value": {"currently_syncing": "dbo-abc"}}
{"type": "ACTIVATE_VERSION", "stream": "dbo-abc", "version": 1616346750991}
{"type": "STATE", "value": {"currently_syncing": "dbo-abc", "bookmarks": {"dbo-abc": {"initial_full_table_complete": true}}}}
INFO METRIC: {"type": "timer", "metric": "job_duration", "value": 0.010980367660522461, "tags": {"job_type": "sync_table", "database": "dbo", "table": "abc", "status": "succeeded"}}
{"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"dbo-abc": {"initial_full_table_complete": true}}}}visch
03/21/2021, 5:18 PMhttps://gitlab.com/vischous/oracle2mssql - (name is oracle2mssql but right now this just pulls from mssql) might be helpful who knows 🙂
visch
03/21/2021, 5:22 PMOh, and to add it properly follow https://meltano.com/docs/plugin-management.html#custom-plugins
t
truc_nguyen
03/22/2021, 2:53 PMI will take a look at it. Thank you @visch
c
chad_petrovay
04/13/2021, 7:49 PMTried installing this from
pip install pipelinewise-tap-mssqlERROR: Could not find a version that satifies the requirement pipelinewise-tap-mssql
ERROR: No matching distribution found for pipelinewise-tap-mssql
v
visch
04/13/2021, 8:22 PMI used Meltano custom plugins https://meltano.com/docs/plugin-management.html#custom-plugins . Not sure if they publish to pip
a
aaronsteers
04/13/2021, 8:24 PMc
chad_petrovay
04/13/2021, 8:25 PMYes. I opened the github page, and followed the installation instructions, and got an error (as posted).
a
aaronsteers
04/13/2021, 8:26 PMAh, yes. That makes sense. And sorry for not re-reading this thread more carefully. I think that readme was created from the fork this was originally copied from. I checked pypi for anything called
tap-mssqlaaronsteers
04/13/2021, 8:27 PMYou should be able to install via URL though…
aaronsteers
04/13/2021, 8:28 PMsomething like:
Copy code
pip install git+<https://github.com/wintersrd/pipelinewise-tap-mssql@master>aaronsteers
04/13/2021, 8:29 PM(If you have pipx installed, that would be preferrable, since it keeps dependencies from conflicting, but syntax is identical.)
aaronsteers
04/13/2021, 8:32 PMMeltano should accept the same/similar syntax since it is all pip-based. As @visch has here: https://gitlab.com/vischous/oracle2mssql/-/blob/master/oracle2mssql/meltano.yml#L8
c
chad_petrovay
04/13/2021, 8:38 PMI think I managed to install from git, and then add the extractor as a custom plugin.
chad_petrovay
04/13/2021, 8:39 PMMy pipeline is failing, so clearly it does not like my config.json file
a
aaronsteers
04/13/2021, 8:40 PMA step in the right direction! 😄 I said it before, but I’m excited that you are testing that python-based fork of tap-mssql. If you are successful with it, would be great to start promoting it within the community.