I've noticed that going from the 1.42.0 to 1.43.0 ...
# announcementsa
al_whatmough
08/06/2020, 7:23 AMI've noticed that going from the 1.42.0 to 1.43.0 release, the tap-postgres extractor isn't working (at least for me). When I run the pipeline, I see the following error from Singer:
asyncio.streams.LimitOverrunError: Separator is not found, and chunk exceed the limitd
douwe_maan
08/06/2020, 3:22 PM@al_whatmough Hmm, we may have regressed on https://gitlab.com/meltano/meltano/-/issues/2166 😬
Can you please run the pipeline again in debug mode (https://meltano.com/docs/command-line-interface.html#debugging), and share the backtrace that gets printed so that I can figure out where the issue originates exactly?
douwe_maan
08/06/2020, 3:31 PMI've created an issue: https://gitlab.com/meltano/meltano/-/issues/2219 Please share your backtrace in there!
douwe_maan
08/06/2020, 3:34 PM@al_whatmough https://gitlab.com/meltano/meltano/-/merge_requests/1797 may fix it, but I'm not confident until I see the stacktrace 🙂 If you like, you can apply those changes to your local Meltano installation and see if it makes a difference! You can find out easily where Meltano is installed by running
pip show meltanoa
al_whatmough
08/07/2020, 5:33 AMThanks @douwe_maan, I added the stacktrace to the issue. FYI I'm using Docker locally - I wasn't sure how to modify the Meltano source from outside the container (is that possible?) so I just ran it inside a virtualenv instead
al_whatmough
08/07/2020, 5:33 AMI applied the changes in https://gitlab.com/meltano/meltano/-/merge_requests/1797/diffs but unfortunately this doesn't fix the issue
al_whatmough
08/07/2020, 5:34 AMI have a feeling that since Meltano is outputting the log output of Singer, in my case, the log messages can be very long (since I have some columns which contain long series of latitude and longitude coordinates)
al_whatmough
08/07/2020, 6:10 AMI can confirm that the above is the case now - I excluded these fields in my
meltano.ymlmeltano elt tap-postgres target-postgresal_whatmough
08/07/2020, 6:10 AMPerhaps the solution might be to truncate very long Singer log messages?
d
douwe_maan
08/07/2020, 2:54 PMI'd like to figure out if the limit is being exceeded by a log message, or by an actual Singer RECORD message.
douwe_maan
08/07/2020, 2:55 PMWe've already raised the line length limit to 128KB in that MR, for both Singer messages and log messages, but perhaps we need to go beyond that.
douwe_maan
08/07/2020, 2:55 PMTruncating log messages would be fine, but truncating Singer messages before forwarding them to the target obviously wouldn't be 🙂
douwe_maan
08/07/2020, 3:05 PM2 Views