https://linen.dev logo
Join Slack
Powered by
Hi all, I apologize in advance but I seem to be ha...
# troubleshooting
j
Hi all, I apologize in advance but I seem to be having trouble with what should be a simple stream-mapping. Still trying to use the sdk supported version of 
tap-mssql.
Details in 🧵.
YAML file:
Copy code
version: 1
default_environment: dev
project_id: dad99788-9e54-4f81-8c1e-4f58c600a14d
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-mssql
    variant: buzzcutnorman
    pip_url: git+<https://github.com/BuzzCutNorman/tap-mssql.git> #We can add a version or hash here later for specific version usage
    config:
      dialect: mssql
      driver_type: pyodbc
      host: 127.0.0.1,4433
      port: 1433
      user: abc
      password: xyz
      database: db
      sqlalchemy_eng_params:
        fast_executemany: 'True'
      sqlalchemy_url_query:
        driver: ODBC Driver 17 for SQL Server
        TrustServerCertificate: yes
  # Full load of admin data - Upon initial and once a release (optional?)
  - name: tap-mssql-admin
    inherit_from: tap-mssql
    select:
    - Admin-*.*
    stream_maps:
      Admin-UserNameLookup:
        Name: "test" # Set To Test
        __else__: __NULL__
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
Pertinent logs around the extraction, noting that the stream name is 
Admin-UserNameLookup
Copy code
2023-06-26T13:26:20.049385Z [info     ] 2023-06-26 09:26:20,046 | INFO     | tap-mssql            | Beginning full_table sync of 'Admin-UserNameLookup'... cmd_type=elb consumer=False name=tap-mssql-admin producer=True stdio=stderr string_id=tap-mssql-admin
2023-06-26T13:26:20.050534Z [info     ] 2023-06-26 09:26:20,046 | INFO     | tap-mssql            | Tap has custom mapper. Using 1 provided map(s). cmd_type=elb consumer=False name=tap-mssql-admin producer=True stdio=stderr string_id=tap-mssql-admin
2023-06-26T13:26:20.124167Z [info     ] 2023-06-26 09:26:20,123 | INFO     | singer_sdk.metrics   | METRIC: {"type": "timer", "metric": "sync_duration", "value": 0.07641863822937012, "tags": {"stream": "Admin-UserNameLookup", "context": {}, "status": "succeeded"}} cmd_type=elb consumer=False name=tap-mssql-admin producer=True stdio=stderr string_id=tap-mssql-admin
2023-06-26T13:26:20.124694Z [info     ] 2023-06-26 09:26:20,123 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "record_count", "value": 244, "tags": {"stream": "Admin-UserNameLookup", "context": {}}} cmd_type=elb consumer=False name=tap-mssql-admin producer=True stdio=stderr string_id=tap-mssql-admin
While the output is still pumping out the full record
Copy code
{"UserNameLookupId": -2147481906, "Name": "Aaron", "Checksum": 531013531}
{"UserNameLookupId": -2147482007, "Name": "Alec", "Checksum": 1249784188}
(etc...)
I feel like I'm doing something silly here
v
What is the command you're running in console?
j
meltano run tap-mssql-admin target-jsonl
v
I was hoping I caught something simple but nope
Oh I do see it
Run this for me 
meltano invoke --dump=config tap-mssql-admin
does it have the stream_map config?
j
No it does not 😞
v
Look at your yaml a bit closer 😉
(config block is missing!)
Need to get to other stuff so
Copy code
- name: tap-mssql-admin
    inherit_from: tap-mssql
    select:
    - Admin-*.*
    config:
      stream_maps:
        Admin-UserNameLookup:
          Name: "test" # Set To Test
          __else__: __NULL__
This is how it should be
j
Thank you Derek, after more fiddling I realized I also needed to still include the PK, so it looks something like this now. Still playing with it but at least the etl is working now
Copy code
config:
      stream_maps:
        Admin-UserNameLookup:
          UserNameLookupId: UserNameLookupId
          ...
          __else__ : __NULL__
Much appreciated
v
Welcome to the community 😄
Open in Slack
PreviousNext
Powered by