Hi Guys, I was trying the MS SQL airflow extractor. However, after configuring my meltano.yml file shown below.

version: 1
default_environment: dev
project_id: 617bd1be-526a-4e47-82c5-c8fcf5e30d55
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-postgres
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-postgres.git>
    config:
      sqlalchemy_url: <postgresql://postgres>:******@localhost/data
    select:
    - dbo-device.*
    - list.*
  - name: tap-mssql
    variant: airbyte
    pip_url: git+<https://github.com/MeltanoLabs/tap-airbyte-wrapper.git>
    config:
      docker_mounts: [{"source": "/home/dwhadmin/meltano-projects/dockermount/","target": "/local/", "type": "bind"}]
      airbyte_spec:
        image: ' airbyte/source-mssql'
        tag: latest
      airbyte_config:
        host: localhost
        port: 1433
        database: data
        username: user
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl

I get the following error when I run meltano config tap-mssql test

Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Plugin configuration is invalid
Catalog discovery failed: command ['/home/dwhadmin/meltano-projects/my-pg-proj/.meltano/extractors/tap-mssql/venv/bin/tap-airbyte', '--config', '/home/dwhadmin/meltano-projects/my-pg-proj/.meltano/run/tap-mssql/tap.ef7efdd1-e7aa-4105-a7f4-35972b56de11.config.json', '--discover'] returned 1 with stderr:
 Traceback (most recent call last):
  File "/home/dwhadmin/meltano-projects/my-pg-proj/.meltano/extractors/tap-mssql/venv/bin/tap-airbyte", line 8, in <module>
    sys.exit(TapAirbyte.cli())
  File "/home/dwhadmin/meltano-projects/my-pg-proj/.meltano/extractors/tap-mssql/venv/lib/python3.8/site-packages/singer_sdk/helpers/_classproperty.py", line 12, in __get__
    return super().__get__(objtype)
  File "/home/dwhadmin/meltano-projects/my-pg-proj/.meltano/extractors/tap-mssql/venv/lib/python3.8/site-packages/tap_airbyte/tap.py", line 218, in cli
    config: tuple[str, ...] = (),
TypeError: 'type' object is not subscriptable

What Python version? The fix looks like changing the annotation to use typing.Tuple but curious on the version because we test 3.8 3.9 and 3.10 in CI

@alexander_butler it looks like python 3.8

So the funny thing @visch Is that is exactly how it is in Meltano's SDK, see here So how does anyone run an SDK based tap below whatever version they added support for tuple[] as a valid typing annotation Like I said the fix is simple but this question should probably be investigated first because its in their codebase

Probably a futures import

Yup you right

I think I've done it a few times I forget where

Ok thats on me, CI tests on 3.8 but didnt catch it so it must be backported to the latest 3.8 branch

Thanks @alexander_butler, Let me give it a shot again

Looks like its getting mad at this:

[{"source": "/home/dwhadmin/meltano-projects/dockermount/","target": "/local/", "type": "bind"}]

That would translate to this:

--mount source=/home/dwhadmin/meltano-projects/dockermount/,target=/local/,type=bind

which looks valid. Does this work?

docker run --mount source=/home/dwhadmin/meltano-projects/dockermount/,target=/local/,type=bind hello-world

Yup it works. See below

/my-pg-proj$ docker run --mount source=/home/dwhadmin/meltano-projects/dockermount/,target=/local/,type=bind hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 <https://hub.docker.com/>

For more examples and ideas, visit:
 <https://docs.docker.com/get-started/>

Hm what about this

image: ' airbyte/source-mssql'

your image in airbyte spec has a space in it?

Yea my bad the space was the issue. But now I'm getting this error:

Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

'ascii' codec can't decode byte 0xe2 in position 16438: ordinal not in range(128)

Must be close now... Is that the full stack trace?

Yes that's the full error message

Ok, give it one last shot

meltano install tap-mssql --clean

I think the bytes splitlines method in the stdlib was using ascii instead of utf8

Unfortunately still getting the same Ascii error.

Oof I have never seen such an undescript error lol

Yea, I can imagine, this is very vague to debug and it could be coming from anywhere

Is

meltano --log-level=debug invoke tap-mssql --discover

any more elucidating? (dont copy paste logs because this command will dump env vars too)

Ok something strange is happening here all other command list discover, select tap-mssql --list run without error but config tap-mssql test fails

Interesting sounds like its more on the

meltano config

subcommand side 🤔 Either way let me push up the port mapping update for you as I am still not 100% sure docker will be able to talk to your mssql instance. Let me know if you discover otherwise 🙂

Yup that works 🙂

Oh nice, so you see something roughly along these lines?

{
  "type": "CONNECTION_STATUS",
  "connectionStatus": {
    "status": "SUCCEEDED"
  }
}

Nice. I am still tempted to push to port mapping update as I foresee it being a need but do let us know if you get some data to replicate to jsonl as you have it

This is the response from meltano invoke tap-mssql --test

2023-01-15T19:36:18.630195Z [info     ] Environment 'dev' is active
2023-01-15 19:36:22,485 {'level': 'INFO', 'message': 'starting source: class io.airbyte.integrations.source.mssql.MssqlSource'}
2023-01-15 19:36:22,485 {'level': 'INFO', 'message': 'integration args: {check=null, config=/tmp/config.json}'}
2023-01-15 19:36:22,485 {'level': 'INFO', 'message': 'Running integration: io.airbyte.integrations.base.ssh.SshWrappedSource'}
2023-01-15 19:36:22,485 {'level': 'INFO', 'message': 'Command: CHECK'}
2023-01-15 19:36:22,485 {'level': 'INFO', 'message': "Integration config: IntegrationConfig{command=CHECK, configPath='/tmp/config.json', catalogPath='null', statePath='null'}"}
2023-01-15 19:36:22,485 {'level': 'WARN', 'message': 'Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword'}
2023-01-15 19:36:22,485 {'level': 'WARN', 'message': 'Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword'}
2023-01-15 19:36:22,485 {'level': 'INFO', 'message': 'Starting connection with method: NO_TUNNEL'}
2023-01-15 19:36:22,485 {'level': 'INFO', 'message': 'HikariPool-1 - Starting...'}
2023-01-15 19:36:22,485 {'level': 'INFO', 'message': 'HikariPool-1 - Start completed.'}
2023-01-15 19:36:22,485 {'level': 'INFO', 'message': 'Attempting to get metadata from the database to see if we can connect.'}
2023-01-15 19:36:22,485 {'level': 'INFO', 'message': 'HikariPool-1 - Shutdown initiated...'}
2023-01-15 19:36:22,485 {'level': 'INFO', 'message': 'HikariPool-1 - Shutdown completed.'}
2023-01-15 19:36:22,485 Configuration has been verified via the Airbyte check command.

No worries, I love OSS work and its a passion to help others so its all good! Here is an open issue: https://github.com/airbytehq/airbyte/issues/16971 And the link it points to: https://docs.airbyte.com/integrations/sources/mssql/#upgrading-from-0417-and-older-versions-to-0418-and-newer-versions

After changing the tag to 0.4.17 this is what I get for meltano config tap-mssql test

Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Plugin configuration is invalid
... 13 more

And this is for meltano invoke tap-mssql test

2023-01-15 19:58:32,055 Could not parse message: 2023-01-15 19:58:31 INFO i.a.i.s.m.MssqlSource(main):464 - starting source: class io.airbyte.integrations.source.mssql.MssqlSource
2023-01-15 19:58:32,055 Could not parse message: 2023-01-15 19:58:31 INFO i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {check=null, config=/tmp/config.json}
2023-01-15 19:58:32,055 Could not parse message: 2023-01-15 19:58:31 INFO i.a.i.b.IntegrationRunner(runInternal):104 - Running integration: io.airbyte.integrations.base.ssh.SshWrappedSource
2023-01-15 19:58:32,055 Could not parse message: 2023-01-15 19:58:31 INFO i.a.i.b.IntegrationRunner(runInternal):105 - Command: CHECK
2023-01-15 19:58:32,055 Could not parse message: 2023-01-15 19:58:31 INFO i.a.i.b.IntegrationRunner(runInternal):106 - Integration config: IntegrationConfig{command=CHECK, configPath='/tmp/config.json', catalogPath='null', statePath='null'}
2023-01-15 19:58:32,055 Could not parse message: 2023-01-15 19:58:31 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-01-15 19:58:32,056 Could not parse message: 2023-01-15 19:58:31 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-01-15 19:58:32,056 Could not parse message: 2023-01-15 19:58:31 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-01-15 19:58:32,056 Could not parse message: 2023-01-15 19:58:31 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword multiline - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-01-15 19:58:32,056 Could not parse message: 2023-01-15 19:58:31 INFO i.a.i.b.s.SshTunnel(getInstance):172 - Starting connection with method: NO_TUNNEL
2023-01-15 19:58:32,056 Could not parse message: 2023-01-15 19:58:31 INFO c.z.h.HikariDataSource(<init>):80 - HikariPool-1 - Starting...
2023-01-15 19:58:32,056 Could not parse message: 2023-01-15 19:58:31 INFO c.z.h.HikariDataSource(<init>):82 - HikariPool-1 - Start completed.
2023-01-15 19:58:32,056 Could not parse message: 2023-01-15 19:58:31 INFO i.a.i.s.j.AbstractJdbcSource(lambda$getCheckOperations$1):140 - Attempting to get metadata from the database to see if we can connect.
2023-01-15 19:58:32,056 Could not parse message: 2023-01-15 19:58:31 INFO c.z.h.HikariDataSource(close):350 - HikariPool-1 - Shutdown initiated...
2023-01-15 19:58:32,056 Could not parse message: 2023-01-15 19:58:31 INFO c.z.h.HikariDataSource(close):352 - HikariPool-1 - Shutdown completed.

Ok whew lets opt for the second suggestion of adding

"replication_method": {
    "method": "STANDARD"
}

to your airbyte config

Ok this is what I did. Is that what you mean?

airbyte_config:
        host: localhost
        port: 1433
        database: database
        username: user
        replication_method:
          method: "STANDARD"

Yep exactly

This is the response from meltano invoke tap-mssql test

2023-01-15T20:09:10.138798Z [info     ] Environment 'dev' is active
2023-01-15 20:09:14,025 {'level': 'INFO', 'message': 'starting source: class io.airbyte.integrations.source.mssql.MssqlSource'}
2023-01-15 20:09:14,025 {'level': 'INFO', 'message': 'integration args: {check=null, config=/tmp/config.json}'}
2023-01-15 20:09:14,025 {'level': 'INFO', 'message': 'Running integration: io.airbyte.integrations.base.ssh.SshWrappedSource'}
2023-01-15 20:09:14,025 {'level': 'INFO', 'message': 'Command: CHECK'}
2023-01-15 20:09:14,025 {'level': 'INFO', 'message': "Integration config: IntegrationConfig{command=CHECK, configPath='/tmp/config.json', catalogPath='null', statePath='null'}"}
2023-01-15 20:09:14,025 {'level': 'WARN', 'message': 'Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword'}
2023-01-15 20:09:14,025 {'level': 'WARN', 'message': 'Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword'}
2023-01-15 20:09:14,025 {'level': 'INFO', 'message': 'Starting connection with method: NO_TUNNEL'}
2023-01-15 20:09:14,025 {'level': 'INFO', 'message': 'HikariPool-1 - Starting...'}
2023-01-15 20:09:14,026 {'level': 'INFO', 'message': 'HikariPool-1 - Start completed.'}
2023-01-15 20:09:14,026 {'level': 'INFO', 'message': 'Attempting to get metadata from the database to see if we can connect.'}
2023-01-15 20:09:14,026 {'level': 'INFO', 'message': 'HikariPool-1 - Shutdown initiated...'}
2023-01-15 20:09:14,026 {'level': 'INFO', 'message': 'HikariPool-1 - Shutdown completed.'}
2023-01-15 20:09:14,026 Configuration has been verified via the Airbyte check command.

I mean what happens if you just run a pipeline?

Ok here is the error. I think it's coming from airbyte though

consumer=False name=tap-mssql producer=True stdio=stderr string_id=tap-mssql
2023-01-15T20:44:43.736826Z [info     ]     raise AirbyteException(    cmd_type=elb consumer=False name=tap-mssql producer=True stdio=stderr string_id=tap-mssql
2023-01-15T20:44:43.736944Z [info     ] tap_airbyte.tap.AirbyteException: Airbyte process terminated early: cmd_type=elb consumer=False name=tap-mssql producer=True stdio=stderr string_id=tap-mssql
2023-01-15T20:44:43.737060Z [info     ] AirbyteException: java.lang.RuntimeException: com.microsoft.sqlserver.jdbc.SQLServerException: Parameters were not supplied for the function 'dbo.fn_SC_Reports_GetStrsFromStringDelimitedBy'. cmd_type=elb consumer=False name=tap-mssql producer=True stdio=stderr string_id=tap-mssql
2023-01-15T20:44:43.737173Z [info     ]         at io.airbyte.db.jdbc.StreamingJdbcDatabase.unsafeQuery(StreamingJdbcDatabase.java:79) cmd_type=elb consumer=False name=tap-mssql producer=True stdio=stderr string_id=tap-mssql

Yes,

fn_SC_Reports_GetStrsFromStringDelimitedBy

is this a function in your database?

Ok it worked. I didn't select the entities to pull from