https://linen.dev logo
Join Slack
Powered by
Hello Everyone, I am using tap-rest-api-msdk and s...
# troubleshooting
s
Hello Everyone, I am using tap-rest-api-msdk and see this error. Can some one help me understand what does this mean?
s
Hey @sudeep_puvadi can you post your meltano.yml for reference?
s
Sure.
locations.txt,meltano.txt
locations.txtmeltano.txt
s
Great, @pat_nadolny @visch I think you have experience with the tap, do you have any inputs here?
v
I haven't used it, I think writing taps with the sdk is easier than configuring them with yaml 🤷
s
message has been deleted
This is the current error I am facing now
p
I havent used this tap yet so I wont be too much help. I found https://github.com/Widen/tap-rest-api-msdk/issues/27#issuecomment-1304385379 which is an example config though if thats helpful at all. (Although the start and end date are flipped in that example)
Oh I think the 
schema: ./locations.json
that you're passing in is invalid. What error did you get in the first screenshot you shared when it inferred the schema? This is an example of a valid schema that was produced as part of testing that ^^ earthquake example I shared above ```{ "streams": [ { "tap_stream_id": "us_earthquakes", "replication_method": "FULL_TABLE", "key_properties": [ "id" ], "schema": { "properties": { "type": { "type": "string" }, "properties_mag": { "type": "number" }, "properties_place": { "type": [ "null", "string" ] }, "properties_time": { "type": "integer" }, "properties_updated": { "type": "integer" }, "properties_tz": { "type": "null" }, "properties_url": { "type": "string" }, "properties_detail": { "type": "string" }, "properties_felt": { "type": [ "integer", "null" ] }, "properties_cdi": { "type": [ "null", "number" ] }, "properties_mmi": { "type": [ "null", "number" ] }, "properties_alert": { "type": [ "null", "string" ] }, "properties_status": { "type": "string" }, "properties_tsunami": { "type": "integer" }, "properties_sig": { "type": "integer" }, "properties_net": { "type": "string" }, "properties_code": { "type": "string" }, "properties_ids": { "type": "string" }, "properties_sources": { "type": "string" }, "properties_types": { "type": "string" }, "properties_nst": { "type": [ "integer", "null" ] }, "properties_dmin": { "type": [ "null", "number" ] }, "properties_rms": { "type": "number" }, "properties_gap": { "type": [ "null", "number" ] }, "properties_magType": { "type": "string" }, "properties_type": { "type": "string" }, "properties_title": { "type": "string" }, "geometry_type": { "type": "string" }, "geometry_coordinates": { "type": "string" }, "id": { "type": "string" } }, "type": "object" }, "stream": "us_earthquakes", "metadata": [ { "breadcrumb": [ "properties", "type" ], "metadata": { "inclusion": "available", "selected": true } }, { "breadcrumb": [ "properties", "properties_mag" ], "metadata": { "inclusion": "available", "selected": true } }, { "breadcrumb": [ "properties", "properties_place" ], "metadata": { "inclusion": "available", "selected": true } }, { "breadcrumb": [ "properties", "properties_time" ], "metadata": { "inclusion": "available", "…
Thats probably what the error is, its having trouble parsing youre locations.json schema.
s
ok @pat_nadolny!
Thank you!
Hello @pat_nadolny, Can I validate tap configuration.
is there a command for it.
message has been deleted
Is receive this error. So checking if there is a way to validate configuration before testing tap connectivity.
p
That would be the way I recommend, 
meltano config tap-rest-api-msdk test
. Do you have any environment configurations that you need to connect? It looks like the default behavior for config is to ignore default environments. If so, add the environment flag like the log suggests
s
Currently I am not using any environment. I am kind of creating a POC in my local environment now.
interesting thing is I am able to access this API from postman using the same URL, Path, username & password
But this is throwing error from this tap
p
Did you run 
meltano config tap-rest-api-msdk
to confirm your config looks as expected? Mine from the demo I linked above looks like this:
Copy code
{
  "api_url": "<https://earthquake.usgs.gov/fdsnws>",
  "streams": [
    {
      "name": "us_earthquakes",
      "params": {
        "format": "geojson",
        "starttime": "2022-12-07",
        "endtime": "2022-12-08",
        "minmagnitude": 1
      },
      "path": "/event/1/query",
      "primary_keys": [
        "id"
      ],
      "records_path": "$.features[*]",
      "num_inference_records": 200
    }
  ]
}
s
Yes
Just a min
message has been deleted
p
Why are you choosing to pass in a custom schema? Not that its wrong but theres more that could go wrong when managing the schema yourself
s
ok
Can you guide me how to do that
p
You would just remove the 
schema
key from your streams config. By default it will try to infer the schema for you based on the API response
s
Great
message has been deleted
p
what command are you running?
s
meltano config tap-rest-api-msdk test
p
what about with 
meltano invoke tap-rest-api-msdk
? Also if sharing your postman request might be helpful (without secrets)
s
ok
```root@PC-5CD2317P11:/mnt/c/wsl/meltano/otm-api/tap-otm-api# meltano invoke tap-rest-api-msdk 2022-12-09T185822.917264Z [info ] Environment 'dev' is active Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to join our friendly Slack community. Catalog discovery failed: command ['/mnt/c/wsl/meltano/otm-api/tap-otm-api/.meltano/extractors/tap-rest-api-msdk/venv/bin/tap-rest-api-msdk', '--config', '/mnt/c/wsl/meltano/otm-api/tap-otm-api/.meltano/run/tap-rest-api-msdk/tap.7851ffae-0fc8-4e6d-9e9d-61666f1c6fce.config.json', '--discover'] returned 1 with stderr: time=2022-12-09 125825 name=tap-rest-api-msdk level=INFO message=tap-rest-api-msdk v1.0.0, Meltano SDK v0.3.18) time=2022-12-09 125825 name=tap-rest-api-msdk level=INFO message=Skipping parse of env var settings... time=2022-12-09 125825 name=tap-rest-api-msdk level=INFO message=Config validation passed with 0 errors and 0 warnings. time=2022-12-09 125825 name=root level=INFO message=Operator '__else__=None' was not found. Unmapped streams will be included in output. time=2022-12-09 125825 name=tap-rest-api-msdk level=INFO message=No schema found. Inferring schema from API call. Traceback (most recent call last): File "/mnt/c/wsl/meltano/otm-api/tap-otm-api/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen httplib_response = self._make_request( File "/mnt/c/wsl/meltano/otm-api/tap-otm-api/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request self._validate_conn(conn) File "/mnt/c/wsl/meltano/otm-api/tap-otm-api/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn conn.connect() File "/mnt/c/wsl/meltano/otm-api/tap-otm-api/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.8/site-packages/urllib3/connection.py", line 414, in connect self.sock = ssl_wrap_socket( File "/mnt/c/wsl/meltano/otm-api/tap-otm-api/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket ssl_sock = _ssl_wrap_socket_impl( File "/mnt/c/wsl/meltano/otm-api/tap-otm-api/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl return ssl_context.wrap_socket(sock, server_hostname=server_hostname) File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket return self.sslsocket_class._create( File "/usr/lib/python3.8/ssl.py", line 1040, in _create self.do_handshake() File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake self._sslobj.do_handshake() ConnectionResetError: [Errno 104] Connection reset by peer During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/mnt/c/wsl/meltano/otm-api/tap-otm-api/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.8/site-packages/requests/adapters.py", line 489, in send resp = conn.urlopen( File "/mnt/c/wsl/meltano/otm-api/tap-otm-api/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen retries = retries.increment( File "/mnt/c/wsl/meltano/otm-api/tap-otm-api/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment raise six.reraise(type(error), error, _stacktrace) File "/mnt/c/wsl/meltano/otm-api/tap-otm-api/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise raise value.with_traceback(tb) File "/mnt/c/wsl/meltano/otm-api/tap-otm-api/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen httplib_response = self._make_request( File "/mnt/c/wsl/meltano/otm-api/tap-otm-a…
message has been deleted
p
Are you sure youre auth header is correct? Maybe try purposefully mispelling them in postman to see the response. @edgar_ramirez_mondragon Any idea what might cause this? I dont know the usual causes of "connection reset by peer" that might be causing this
s
Copy code
{
  "status": 401,
  "title": "Unauthorized",
  "detail": "This request requires HTTP authentication."
}
When I misspelled credentials .. I got this error.
p
hmm thats not it then 🤔
s
On the other side .. I tried to implement my own tap .. I still receive the same error. When I run poetry pytest.
e
I see there are more headers (10) in postman. Perhaps you’re missing one of those?
(mostly just guessing)
s
ok
any idea .. what could I be missing in the headers
e
Dunno, some APIs expect 
Content-Type
, for example
s
let me try taht
that*
No luck. Same Error
See this
message has been deleted
e
Ok, so it’s something in the request that’s different from postman. Can postman export request to curl syntax?
s
Let me try
Yup.. I was able
curl --location --request GET 'https://otm.com/logisticsRestApi/resources-int/v2/corporations' \ --header 'Authorization: Basic RkhSL0YwMS5JTlZlUOkludGVncjhlcg==' \ --header 'Cookie: JSESSIONID=fQ_4ATU3V97idxHWipImxtmUsntVKNQTQ1dckk7bq3JHgPLJGJ7S!1009956005; ROUTEID=.1'
e
Ok. So • 
--location
means redirects are ok, no changes needed in the request in the SDK 👍 • Is the session cookie a hardcoded value in postman?
s
I have not hardcoded any value here. May be username & password saved as cookie here.
e
Is there a way in postman to see the redirect history of a response? What may be happening is that the first hit responds with a 
Set-Cookie
header and so the next request has the right cookie.
I didn’t ask, did the above curl command work?
s
Let me try
Nope. That did not work.
e
Hmm, one sec let me give you a python script to test
Copy code
import requests

YOUR_TOKEN = "..."

session = requests.Session()
session.get(
    "<https://otm.com/logisticsRestApi/resources-int/v2/corporations>",  # or the right URL
    headers={
        "Authorization": f"Bearer: {YOUR_TOKEN}"
    },
)
can you try running that in Python >= 3.6?
s
ok
in a minute
Copy code
sudeeppuvadi@PC-5CD2317P11:~$ python3 ./test.py
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 996, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 366, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 370, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sudeeppuvadi/.local/lib/python3.8/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 996, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 366, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 370, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./test.py", line 6, in <module>
    session.get(
  File "/home/sudeeppuvadi/.local/lib/python3.8/site-packages/requests/sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
  File "/home/sudeeppuvadi/.local/lib/python3.8/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/sudeeppuvadi/.local/lib/python3.8/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/home/sudeeppuvadi/.local/lib/python3.8/site-packages/requests/adapters.py", line 547, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
e
weird…
s
message has been deleted
Here are the headers that are passing to request from a successful postman request
e
I’m not too familiar with postman. Is the proxy url something that postman sets for all requests or is that only for your api?
s
ok. I tried something different and I see a different error now.
message has been deleted
2 Views
Open in Slack
PreviousNext
Powered by