https://linen.dev logo
Join Slack
Powered by
Hey all, Im building a custom rest api tap and I c...
# singer-tap-development
j
Hey all, Im building a custom rest api tap and I can't seem to send the fetched data to a target (jsonl in this case). I've customized 
parse_response
in client.py
Copy code
def parse_response(self, response: requests.Response) -> Iterable[dict]:
        """Parse the response and return an iterator of result records.

        Args:
            response: The HTTP ``requests.Response`` object.

        Yields:
            Each record from the source.
        """
        # TODO: Parse response body and return a set of records.
        # print(response.json())


        keys = [column['name'] for column in response.json()['columnDefinitions']]
        combined_data = []

        for data_row in response.json()['data']:
            if len(keys) != len(data_row):
                raise ValueError('Mismatch between keys and values length')
            combined_data.append(dict(zip(keys, data_row)))
        print('data_data@@@', json.dumps(combined_data))
        json_data = json.dumps(combined_data)
        yield from extract_jsonpath('$[1]', input=json_data)
And im 100% sure the yield is a valid json. Im getting this error from jsonl
Copy code
simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Which is leading me to believe that whatever im yielding is not getting to the target. Am I handling the response in the correct place? Do I need a 
get_records
or something in my streams.py so the proper message gets passed to my target?
w
I believe you need to pass a object to extract_jsonpath not a string
here is how I got my tap to work yield from extract_jsonpath(self.records_jsonpath, input=response.json())
e
@Jared Garvin-Joseph you probably need to get rid of the 
print
call or do 
print(..., file=sys.stderr)
. Singer taps use on stdout to pass data to the target 🙂.
👍 1
j
Ok thank you both, will give that a try
Sorry for the delay here, I was able to update my code but still getting the same error (code below)
Copy code
combined_data = []

        for data_row in response.json()['data']:
            if len(keys) != len(data_row):
                raise ValueError('Mismatch between keys and values length')
            combined_data.append(dict(zip(keys, data_row)))
        print('data_data@@@', json.dumps(combined_data))
        json_data = json.dumps(combined_data)
        yield from extract_jsonpath('$[1]', input=combined_data)
combined_data
is a list of json objects (that are valid)
Does the 
parse_response
yield go directly to the loader?
e
There's a bit of processing, but mostly yes. You can check the output of your extractor with 
meltano invoke <my-tap> > my-tap.singer.jsonl
and inspect the contents of that file to see if there's any offending line. I'm curious, what target are you using?
j
target-jsonl
variant: andyh1203
e
Yeah, so that target-jsonl unfortunately doesn't perform any handling of the exception so there's no way to display the offending line automatically. The best way to find it is with my 
meltano invoke
suggestion above. Watch out for lines that are not valid json objects.
j
Thank you, I did run that suggestion below is the output. It looks ok to me so...Im kinda stuck
output.signer.jsonl
e
Line 3 is not valid JSON
j
Oh that's the source of the logger comment. got it
@Edgar Ramírez (Arch.dev) Thank you so much. I think I know what is happening
np 1
🙌 1
4 Views
Open in Slack
PreviousNext
Powered by