Hi Meltano folks, tap-mssql question! I'm new to M...
# troubleshooting
s
Hi Meltano folks, tap-mssql question! I'm new to Meltano, and what seemed like a straightforward docker-meltano install is bringing up some very strange errors. I have WSL2 installed to Windows Server 2025 so I can orchestrate unattended ELT when the server gets restarted every weekend. I have Docker inside WSL2 and Meltano in that Docker container. I have installed just one connector to begin testing with - tap-mssql, the new buzzcutnorman default. When I run invoke or test on the tap-mssql, after configuring, it says it cannot find the driver. I've installed the driver, and if I run any pyodbc query from within the Meltano Docker container, it runs just fine and finds the driver by name. Here is my meltano.yml file with a few values redacted (but tested with vanilla pyodbc without issue): ___________________________________________________________________________ version: 1 default_environment: dev project_id: c531ef95-bcc4-439f-84fa-e414b36aa2** environments: - name: dev - name: staging - name: prod plugins: extractors: - name: tap-mssql variant: buzzcutnorman pip_url: git+https://github.com/BuzzCutNorman/tap-mssql.git config: host: [my server FQDN] port: 1433 user: [MyUser] password: $TAP_MSSQL_PASSWORD database: [MyDB] dialect: mssql driver_type: pyodbc sqlalchemy_url_query: driver: ODBC Driver 18 for SQL Server TrustServerCertificate: Yes ________________________________________________________ The error message right at the end is: sqlalchemy.exc.DBAPIError: (pyodbc.Error) ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 18 for SQL Server' : file not found (0) (SQLDriverConnect)") Does anything look obviously wrong with these values? If the same container can run the odbc queries successfully with just the driver name, what am I doing wrong? Thank you for looking at this. P.S. This is the command that works every time, with my values redacted. python3 -c "import pyodbc; pyodbc.connect("DRIVER={ODBC Driver 18 for SQL Server};SERVER=***;UID=^^^;PWD=""";DATABASE=$$;TrustServerCertificate=yes")'
e
should this be
Copy code
sqlalchemy_url_query:
        driver: pyodbc
        TrustServerCertificate: Yes
?
s
That's what I initially had! But it also couldn't find the driver. Then I found this post that Mr. Norman commented on, and saw the driver format he was using didn't match up. https://discuss.meltano.com/t/18781692/hallo-i-have-a-question-about-the-correct-use-of-meltano-tap
e
Ah, you're right! Pinging @BuzzCutNorman in case he knows what's going on here 🙏
🙏 1
b
Sabrina, reading the chain I am wondering did you install the driver in the docker container or wsl ?
s
I did it from within Docker, but the path doesn't seem to reflect anything? \\wsl.localhost\Ubuntu\opt\microsoft\msodbcsql18\lib64 <-- install location Directions I followed: https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-[…]tall%2Cdebian8-install%2Credhat7-13-install%2Crhel7-offline
Do you have a link to driver installation instructions for Docker? I would really love to do that to solve this. 🙂
I should add -- I'm not using Docker Desktop at all, just Docker CE.
b
Sabrina, I don't have a link to share with you. To be honest I just run the tap on windows server most of the time. I will see if I can find a good link for you.
I found this link in which some was having the same issue. ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 17 for SQL Server' : file not found (0) (SQLDriverConnect)") · Issue #717 · mkleehammer/pyodbc The person having the issue found that odbcinst.ini was not getting created.
command:
docker run <docker image name> cat /etc/odbcinst.ini
cat: /etc/odbcinst.ini: No such file or directory
They found a work around with the following.
Resolved it -
• Installed MS repository
• Installed MS drivers
• Installed Sql server Tools
• openssl configurations set
Now able to connect to remote sql server from VM -> docker -> sql server
I am still looking so see if I can find more.
I found this nice walk through. Configuring ODBC Driver 18 in a Dockerfile for Azure App Service on Linux | by Jonathan Z | Medium . After reading this I think a task to create a odbc.ini file needs to be part of of the build. I hope this information helps.
s
Thank you so much!! I will try these, I really appreciate your time and research more than I can say. I've been struggling with this for a couple days.
s
Hi Sabrina, I also ran into issues installing an ODBC driver in Docker. It was a different driver and part of a different workflow (not Meltano), but I thought sharing my approach might still be helpful. I had to install a few additional packages: •
openjdk-17-jre-headless
(required by the driver I was using) •
libpq-dev
gcc
python3-dev
unixodbc-dev
The last one was necessary to get
odbc.ini
and
odbcinst.ini
under
/etc
. I saw that your link already includes this step, so you probably have it covered. After installing the packages, I needed to configure the two
.ini
files, as they are empty by default. My approach was to keep preconfigured versions of these files in the repo and use
COPY
to overwrite the ones in the image: •
odbcinst.ini
contains the driver configuration and its location. •
odbc.ini
stores the data source configuration (hostname, port, database, etc.). • Both were required in my case. It might also be worth checking if there are environment variables that control the paths to these
.ini
files. I’m not sure how it works with the Microsoft driver, but in my case,
ODBCINI
and
ODBCINST
were set to point to the respective file paths. I can imagine that this could be a potential reason why the Python script works in your case, but the Meltano workflow fails. I'm not sure, but maybe you need some environment variables in your
.env
file? Here’s what my
odbc.ini
looks like:
Copy code
[ODBC Data Sources]
my_data_src=SQL-92 ODBC Driver for OpenEdge

[my_data_src]
Driver=/usr/dlc/odbc/lib/pgoe27.so
Database=my_db
HostName=my_host
PortNumber=my_port
And `odbcinst.ini`:
Copy code
[ODBC Drivers]
SQL-92 ODBC Driver for OpenEdge=Installed

[SQL-92 ODBC Driver for OpenEdge]
Driver=/usr/dlc/odbc/lib/pgoe27.so
Hope this helps in some way, let me know!