Stream maps question. I am working with an singer-sdk based tap and am trying to get a handle on the basics with a test senario. I have the following stream.

{"type": "SCHEMA", "stream": "dbo-ColumnTrouble", "schema": {"properties": {"Id": {"type": ["integer"]}, "Item": {"type": ["string"]}, "index": {"type": ["integer", "null"]}}, "type": "object", "required": ["Id", "Item"]}, "key_properties": ["Id"]}
{"type": "RECORD", "stream": "dbo-ColumnTrouble", "record": {"Id": 1, "Item": "Park Ticket", "index": 100}, "time_extracted": "2024-05-06T16:16:06.284982+00:00"}
{"type": "RECORD", "stream": "dbo-ColumnTrouble", "record": {"Id": 2, "Item": "Churro", "index": 200}, "time_extracted": "2024-05-06T16:16:06.284982+00:00"}
{"type": "RECORD", "stream": "dbo-ColumnTrouble", "record": {"Id": 3, "Item": "Lunch", "index": 300}, "time_extracted": "2024-05-06T16:16:06.284982+00:00"}
{"type": "RECORD", "stream": "dbo-ColumnTrouble", "record": {"Id": 4, "Item": "Character Signature", "index": 400}, "time_extracted": "2024-05-06T16:16:06.284982+00:00"}

The first stream map I am attempting is to make the property

index

disappear. I placed this stream map at the end of my meltano.yml.

stream_maps:
  dbo-ColumnTrouble:
    Item: __NULL__

Then ran

meltano invoke

, and found out I am a poor data magician. The property

index

was still present. Any words of wisdom and guidance are much apricated.

Did you mean something like:

stream_maps:
  dbo-ColumnTrouble:
    index: __NULL__

or maybe

stream_maps:
  dbo-ColumnTrouble:
    Id: Id
    Item: Item
    __else__: __NULL__

I meant to put this one as you pointed out. Thanks 🙏

stream_maps:
  dbo-ColumnTrouble:
    index: __NULL__

Ok. Is the name of the field

index

or perhaps

Index

(capital first letter)?

Just checked it is lower case. This is the create table statement in mssql.

use [testdata]
go
/*********************************
Create the simple test table
in an MSSQL database
*********************************/
DROP TABLE IF EXISTS [ColumnTrouble];
CREATE TABLE ColumnTrouble (
    Id int IDENTITY(1,1) PRIMARY KEY,
    Item varchar(255) NOT NULL,
    [index] int,
);

Oh, you do need to put it in the tap's

config

mapping

Like this maybe ? No, No wait. 😅 Like this. 😀 When put in the proper place it works beautifully. 🎉

version: 1
default_environment: dev
project_id: 5b0fdd46-1796-4e3d-91ef-f4a5ffbbd101
environments:
- name: dev
- name: staging
- name: prod
send_anonymous_usage_stats: false
plugins:
  extractors:
  - name: tap-mssql
    variant: buzzcutnorman
    pip_url: git+<https://github.com/BuzzCutNorman/tap-mssql.git>
    config:
      dialect: mssql
      driver_type: pymssql
      host: checkyourself
      user: user
      database: testdata
      stream_maps:
        dbo-ColumnTrouble:
          index: __NULL__
    select:
    - 'dbo-ColumnTrouble.*'

Rather

config:
      dialect: mssql
      driver_type: pymssql
      host: checkyourself
      user: user
      database: testdata
      stream_maps:
        dbo-ColumnTrouble:
          index: __NULL__

Thanks Edger

Of course!