Hey guys! (First message here), hope everyone is doing well. I am a data analyst attempting to ELT my data into a postgreSQL database using Meltano. I'm trying to use Singer's tap-shopify and I am currently attempting to install the loader for pgSQL. Has anyone got any experience with this? I am getting a pg_config error. Happy to detail if need be

Seeing the full error message would be helpful. My guess is that there is a missing Postgres library on your system.

Very much a shot in the dark here, but psycopg2 (which I’m guessing is what the target uses) needs a os-level library, at least on linux: libpq-dev. See https://www.psycopg.org/install/. I have had to google that one so many times…

Yes you guys were correct thank you @Will Da Silva (Arch) and @Henning Holgersen. I now have an issue with this 😂 :

2022-08-18T18:43:09.487462Z [info     ] ImportError: DLL load failed while importing _psycopg: The specified module could not be found. cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres

2022-08-18T184309.501380Z [error ] Loader failed

I helped someone with shopify a while back to athena, https://github.com/AutoIDM/shopify2athena/blob/main/meltano.yml Might be helpful

I'll have a read through this when I can, thank you so much @visch!

Not as popular as the singer variant yet, but we created an SDK based tap-shopify. At least a couple of people live on it. https://hub.meltano.com/taps/shopify--matatika Here's a fully configured project (with target-postgres) I created minute ago: https://github.com/MatatikaBytes/Shopify-example-qfuvyxc

oh @james_hopgood if you're on linux

apt install libpq-dev

ImportError: DLL load failed

Sounds like a certain operating system from a company in "Redmond"

Hey @aaron_phethean! Yes I saw this variant, I was surprised to see that this was in stable status but the singer default was no longer in stable status. Is there any reason Matatika hasn't been updated since earlier this year? I will give them a try too! Thank you 🙂

Hey @james_hopgood good questions! This tap isn't that old and the only shopify variant built with the meltano SDK. As maintainers of the hub, I think meltano are keen to bump the SDK based taps to the default when the adoption for a variant picks up. I don't know where the hub is getting that last updated date from or why the singer built tap is in ‘Inactive / stale’. We made a release end of last month. I also notice the execution count is across all variants - not that helpful to choose the best one! We test this tap end to end daily, have had a couple of conversations with people using it, but at this stage not many issues or reported issues. Our team is glad to help if you run into any issues. Our short term plan is to build some default dbt models for use with this tap. Ideally for a client, but we’ll get to those fairly soon in any case.

Hey @aaron_phethean Ah I see! That makes much more sense! Great to hear that things have been going so positively for you and your team 🙂 Well default dbt models sound very exciting and I will keep a close eye on this! I had been writing all my scripts manually in Python until I discovered the likes of Meltano and dbt so any growth in that area makes me very excited for the future

Super. I'll keep you posted then! We're working towards making these tap + dbt + analytics all upgradable plugins. Not far off now!

Please do! Thank you!