Hey guys, I'm new to Meltano and I love the tool. ...
# singer-tap-development
j
Hey guys, I'm new to Meltano and I love the tool. I'm working on overhauling our old ELT framework and replacing it with a meltano and dbt driven system. One issue I've run into is that the ERP at our company uses a Progress OpenEdge Database. With the current ETL system, I connect using on Open Edge odbc driver. Does anyone know I could develop a tap to integrate with this sort of source?
v
If you can find a sqlalchemy dialect you'd be in luck. I found https://pypi.org/project/progress_sa/0.4.1/#files quickkly but I don't see a source repo anymore. If there was one then the meltano sdk makes life easy for you. If not then you can still use the base sdk but you'll have to write the methods yourself.
j
Thanks for such a quick reply! Like you said, it doesn't seem like there is a current sqlalchemy dialect for progress open edge. When you say "If not then you can still use the base sdk but you'll have to write the methods yourself." will these methods utilize pyodbc instead of sqlalchemy?
v
sqlalchemy can use things like pyodbc to connect
j
Okay. Do you know of any resources for new developers for developing sql based taps? I've installed the sdk, but don't feel super confident writing my own methods
Okay, well thanks anyway!
p
I'd recommend getting your IDE debugger set up so you can step through the code as you start developing. Probably referencing other taps is the best way to understand what needs to be overridden right now: • https://github.com/MeltanoLabs/tap-athenahttps://github.com/MeltanoLabs/tap-postgreshttps://github.com/MeltanoLabs/tap-snowflakehttps://github.com/MeltanoLabs/tap-duckdbhttps://github.com/meltano/sdk/blob/main/samples/sample_tap_sqlite/__init__.py
j
Hey guys. I wanted to share an update! I searched github for the repo for the progress_sa dialect that @visch pointed out to me. The first result I found was actually a fork of the original repo, found here. It turns out someone made an updated dialect for the newest version of OpenEdge (Progress OpenEdge 11.7), which happened to be exactly what I needed. Using this dialect ended up allowing me to connect to my data source in testing*! Now I have to test this new connection string and dialect in my tap-progress project, and finally bring my tap into my Meltano project and test there. *One small note about installing and setting up this dialect. I had no trouble when following the install instructions in the README of the repo. However, when following the "Usage" instructions I ran into an issue where attempting to run a select script on the database using the newly created engine was returning this error:
Copy code
InterfaceError: (pyodbc.InterfaceError) ('IM002', '[IM002] [unixODBC][Driver Manager]Data source name not found, and no default driver specified (0)
I did some digging and found my resolution here. Super simple fix! Thanks everyone for the suggestions, and I'll update once I've done further testing.