https://linen.dev logo
Join Slack
Powered by
I'm trying to set a custom property in the catalog...
# announcements
a
I'm trying to set a custom property in the catalog file. The output I'm trying to produce is the following in the catalog/properties file:
but instead I end up with this in the config file:
message has been deleted
Can I use 
meltano config
to produce the first result?
n
I would think yes, but I will play with it quick and see if I can replicate
@adam_roderick I think I have an answer, but first. Does that metadata key need to be an array of objects?
adding this to the config in meltano.yml worked for me, but note that the key I am putting it under is "test"
Copy code
config:
        test:
          - breadcrumbs: []
            metadata:
              table-key-properties:
                - id
              schema-name: public
              database-name: packet_api
              row_count: 11374
              is-view: false
              selected: false
              replication-method: FULL_TABLE
The reason for that is that the metadata key from what I can tell when populated by Meltano can't be a list. Not sure why that is, but maybe @douwe_maan would have a better idea. If what you needed was to add a key to the metadata object you can 100% do that easily enough, but if the metadata needs to be an array of objects I haven't been able to get that to work
meltano config <plugin> set metadata.replication-method FULL_TABLE
should do the trick if you wanted to go that route and set only the replication method property on the metadata object
a
I don't need an array of objects
This still generates the values in the config file, rather than the properties/catalog file. Any idea why that is? This is a metadata property that applies to only a single stream, so it should be in the catalog, rather than a global setting in the config file.
I'm reading here now https://meltano.com/docs/plugins.html#metadata-extra. Didn't find it earlier
This is definitely the right tree to bark up, but I must be missing something. In meltano.yml:
Copy code
extractors:   
  - name: tap-postgres
    ...
    metadata:
      hardware_reservation_histories:
        replication-method: FULL_TABLE
does not apply the replication-method metadata property in the catalog:
message has been deleted
n
@douwe_maan Is there a command to generate the properties and config files without running elt or invoke? It would be helpful when debugging a tap / target config to just be able spit out those files for review
d
@adam_roderick Your latest YAML snippet is correct, but you appear to be missing a 
tap_stream_id
in your catalog file, which is required per https://github.com/singer-io/getting-started/blob/master/docs/DISCOVERY_MODE.md#the-catalog. That's the value Meltano will match the entity identifier against, so if it's missing it'll never match 😬 Could you add a 
tap_stream_id
? We could also have Meltano fall back on 
stream
if no 
tap_stream_id
is set, but since it's defined as required, I was assuming all taps would have one.
@nevin_morgan Great idea! We have 
meltano config <plugin>
already, which outputs the exact same config that will be dumped in 
config.json
, but I'll create an issue to add a new 
meltano catalog <extractor>
command as well!
a
@douwe_maan tap_stream_id is in there, just not in the screenshot I posted. But it is a different value than the table_name value. If I understand you right, I need to the config to match tap_stream_id, whereas now it matches table_name. I'll give that a try!
d
@adam_roderick That's exactly right! As the doc you linked states: 😉
Stream (entity) metadata 
<key>: <value>
pairs (e.g. 
{"replication-method": "INCREMENTAL"}
) are nested under top-level entity identifiers that correspond to Singer stream 
tap_stream_id
values.
Also:
Entity and attribute names can be discovered using 
meltano select --list --all <plugin>
.
That command's output would've listed 
tap_stream_id
s too, not 
table_name
. I get how this could be easy to miss though, since using table names seems so natural.
a
badabing bada boom! I need those wink hints 😉 on a regular basis!
d
😄
a
The docs are clear. Thanks for the hint
d
8 Views
Open in Slack
PreviousNext
Powered by