https://linen.dev logo
Join Slack
Powered by
Does anyone have an example of configuring `stream...
# troubleshooting
a
Does anyone have an example of configuring 
stream_maps
with the 
tap-postgres meltano labs
variant? I’m trying to remove some columns but I’m getting an error message — ultimately, I’m trying to transform some dates, but want to get it working first.
Copy code
- name: tap-postgres--app-portal
      inherit_from: tap-postgres
      config:
        stream_maps:
          public-gp:
            updated_at: null
            inserted_at: null
    - name: tap-postgres--app-portal-gp
      inherit_from: tap-postgres--app-portal
      select:
        - public-gp.*
      metadata:
        "*":
          replication-method: FULL_TABLE
☝️ — this is my current confg
this is the error: 
site-packages/meltano/core/settings_store.py", line 1049, in get_with_metadata
metadata = self.config_with_metadata[name]
KeyError: 'stream_maps.public-gp.updated_at'
v
Smells like https://github.com/meltano/meltano/issues/6382#issuecomment-1233423611 But the error doesn't seem like that at all. I don't have time to dive I"ll look next week early
a
Yeah, I was thinking this: https://github.com/meltano/meltano/issues/6942 or this helped me get squared away — anyway, I sorta got it working by adding an environment variable for the config path with the transform I wanted
Thank you @visch for the assist 🙂
v
np I wanted to be sure it wasn't the 
tap-postgres
😄
a
I am seeing a different issue with the 
tap-postgres
troubleshooting that now -> (psycopg2.ProgrammingError) can’t adapt type ‘Column’
v
Can you get an issue in to the tap? Or post more details here will be fine. I'm off to dinner and probably won't be able to check until ~ Monday 😄
😮 but I'll def look
a
will do - I’ll scan the issues first — have a great weekend!
FYI, this isn’t related to the tap itself, but is a combination of the 
singer_sdk
and 
sqlalchemy
— I’m trying to come up with a solution, but will post additional details back to the main 
troubleshooting
channel/open an issue on the 
singer_sdk
repo
v
Pushed a fix up you should be good to go @andy_crowe
That issue was very helpful
a
Awesome! Thank you @visch 🙏 — looks like we just implemented the 
get_record
logic form the SDK directly in the tap-postegres (?), just curious why we didn’t just update the SDK directly? Won’t other taps encounter the same issue, or maybe this was an isolated event? (I’m not super familiar with SQLAlchemy) I was trying to get the SDK spun-up locally to make the change, but haven’t been able to successfully passing tests locally
v
Definitely want to update to the SDK, and definitely should just create a PR for the SDK. I can make a change immediately to tap-postgres without waiting for the SDK to be updated, and it's a good way to test a change out
a
Perfect — just wanted to make sure I didn’t mis-diagnose the issue 😆
v
You were dead on!
a
I will continue working on getting the SDK up and running locally, make the change(s) and then get a PR going
v
Include the test I wrote as well if you could! It could be improved on as well (Like by checking the data in the server is set correctly ) You'll probably see why I wasn't gunghoe on getting it into the sdk yet 🙂
a
I can already tell — I spent a considerable amount of time just trying to get the SDK tests to pass post-initial install, its clearly something with my local configuration because all the tests are successful in the git workflow
your fix will let me keep moving forward in the meantime 🙂 thankyou
v
Thanks for finding it, I learned some things along the way as well! https://meltano.slack.com/archives/C01PKLU5D1R/p1670985507591619 is still puzzling me
Open in Slack
PreviousNext
Powered by