#1942 bug: Tap fails if stream maps config contains a key for an unselected stream
Issue created by
edgarrmondragon
Singer SDK Version
0.31.1
Is this a regression?
• Yes
Python Version
NA
Bug scope
Mapping (stream maps, flattening, etc.)
Operating System
NA
Description
Consider the countries sample in this repo.
1. Generate the Singer catalog for the tap:
$ poetry run python samples/sample_tap_countries --discover > countries.catalog.json
2. Create a stream maps config for the tap. This one creates slugs for both the continent and country name:
$ jq < countries.stream_maps.json
{
"stream_maps": {
"continents": {
"slug": "str(name.lower().replace(' ', '-'))"
},
"countries": {
"slug": "str(name.lower().replace(' ', '-'))"
}
}
}
3. Run the tap with the generated catalog and stream maps config:
$ poetry run python samples/sample_tap_countries --catalog countries.catalog.json --config countries.stream_maps.json > countries.singer.jsonl
This should work as expected.
4. Now, open
countries.catalog.json
and edit it to deselect the
countries
stream
$ jq '.streams[1].metadata[-1]' < countries.catalog.json
{
"breadcrumb": [],
"metadata": {
"inclusion": "available",
"selected": false, <----- this right here needs to be 'false'
"selected-by-default": true,
"table-key-properties": [
"code"
]
}
}
5. Run the tap again:
$ poetry run python samples/sample_tap_countries --catalog countries.catalog.json --config countries.stream_maps.json > countries.singer.jsonl
You'll end up with
singer_sdk.exceptions.StreamMapConfigError: Invalid key properties for 'countries': [code]. Property 'code' was not detected in schema.
Code
No response
meltano/sdk