https://linen.dev logo
Join Slack
Powered by
I configured tap-github and target-postgres. I exe...
# getting-started
j
I configured tap-github and target-postgres. I executed 
meltano run tap-github target-postgres
Error:
Copy code
AttributeError: module 'collections' has no attribute 'MutableMapping'
I googled and it seems that 
MutableMapping
was moved to different library in Python 3.10 (which I have in the underlying virtualenv). This github issue in Medusa seems to be related: https://github.com/pymedusa/Medusa/issues/10253 Is it a known bug/limitation? Is a fix in progress? Any workaround solution?
s
Hey @jan_soubusta, yes there's an issue with the target and the current Python version. downgrading to python 3.9 will solve the issue, as would using a different version of the target (https://github.com/MeltanoLabs/target-postgres, still in development, but made exactly to get rid of these kinds of problem).
@jan_soubusta, btw. this has been fixed inside the target, I'm currently checking why the new target didn't make it through to you (I assume you did ran your meltano install command like today).
j
Today, I executed 
meltano add loader target-postgres
s
Yes, so i figured out the "problem". The github repository already is fixed, but they haven't built a new wheel for pip. So you're still getting the old one. I'm thinking about how to update the tutorial in a good way here. So you could: 1. downgrade your python 2. use the meltanoLabs provided variant 3. Install the target as custom loader, then you can add the git URL and it will install the latest version (that includes the python 3.10) fix. https://docs.meltano.com/guide/plugin-management#custom-plugins
a
@Sven Balnojan - I agree downgrading to Python 3.9 is probably a good idea as a quick fix if feasible. On our side, we can also consider changing the Hub entry for this plugin to point to the github repo instead of the PyPi package. (Although we probably should also ping the maintainer in a GitHub issue to see if they can push a new release.)
j
We investigated the issue with @Sven Balnojan in DM. I downgraded to 3.8, but then it failed while doing INSERT into commits table, because parents column is created as JSONB in Python, but JSONB[] array is expected in the database. Sven is working on the fix. Also, I notified him about couple of confusions I had to workaround in the 
meltano.yml
file: • 
dbname
in DOC but 
database
is required • 
password
must be specified in 
meltano.yml
file(not secure), neither 
.env
file not manually exporting 
TARGET_POSTGRES_PASSWORD
env variable works.
c
We can also change the Hub entry for this plugin to point to the github repo instead of the PyPi package.
I think that sounds like a great idea, considering that it's already taking quite a lot of time to merge critical dependabot security issues into the master branch, I would not hold my breath for any Pypi releases at all.
(Although we probably should also ping the maintainer in a GitHub issue to see if they can push a new release.)
I have done that from time to time, but it just doesn't feel like the most efficient way of keeping the basic house keeping alive (security fixes, python version support and other dependabot merges ...)
I downgraded to 3.8,
I feel like downgrading Python will just get you into more trouble. It is my belief that Python 3.10 is probably the most widely used version right now (considering that that's what is shipped in the latest Ubuntu LTS)
@jan_soubusta: You can follow these steps to switch to using the latest known good code of the 
target-postgres
from github, without needing to wait for a Pypi release by following the steps in this comment: https://meltano.slack.com/archives/C01TCRBBJD7/p1670477332101509?thread_ts=1670449431.189059&cid=C01TCRBBJD7
j
@christoph will do today. The rest of our company is travelling to Christmas party, so no one is distracting me 😉
Works like a charm, thanks! Including incremental crawling from Github (which I struggled to implement it by myself). I had to completely recreate my virtualenv (Python 3.10), something was "corrupted" after playing with multiple variants of loaders. Additional feedback: • when 
meltano.yml
contains errors, the error report is confusing. Huge stacktrace, where I have to find raw key error and realize that the key relates to a configuration property • onboarding DOC is focused on the interactive experience. I don't like it. Instead, I expect a documented list of mandatory config properties in a YAML format, so I can copy&paste them into my config file and update them accordingly. Including properties stored in 
.env
file.
Should I keep the custom pip_url forever or should I wait for an update from your side how to "standardize" it? Now, I am going to extend the project to the full scale. If I hit any issue, will let you know in a separate thread. Again, thanks for you help!
c
Should I keep the custom pip_url forever or should I wait for an update from your side how to "standardize" it?
Well. This is indeed a specific workaround for this variant of target-postgres. Understanding how to resolve installation issues with this variant is not an obvious task and does always require a bit of analysis. One problem with this particular variant is that security fixes simply are not merged in a timely manner. (The security fixes in this variant are mostly the embedded OpenSSL native C library that comes with the psycopg2-binary python wheel.) So, this workaround of pinning to a github PR is not really great, but it's the best I have found so far. The alternative is to use the newly developed MeltanoLabs variant, which won't suffer from these maintenance problems. https://hub.meltano.com/loaders/target-postgres--meltanolabs/ I am planning of migrating to the new MeltanoLabs variant myself as early as I can find time for testing it.
Open in Slack
PreviousNext
Powered by