Hello everyone, I'm trying to apply INCREMENTAL loading to my tap-jira extractor. To do so, I've added the 'metadata' step on my tap-jira configuration

- name: tap-jira
        config:
          auth:
            flow: password
            username: karel@rauva.com
          domain: rauva.atlassian.net
          stream_maps:
            issues:
              __filter__: key.startswith('DATA')
              updated_test: fields.updated
        select:
          - issues.key
          - issues.fields
          - issues.fields.updated
          - updated_test
        metadata:
          issues:
            replication-method: INCREMENTAL
            # replication-key: updated_test       # no
            # replication-key: fields.updated         # no
            # replication-key: fields__updated        # no
            replication-key: issues.fields.updated    # no

issues.fields.updated - is the field I want to use to set the state of the loading. Initially I've tried with this:

replication-key: issues.fields.updated

2024-04-07T14:59:54.185389Z [info     ] singer_sdk.exceptions.InvalidReplicationKeyException: Field 'issues.fields.updated' is not in schema for stream 'issues' cmd_type=elb consumer=False name=tap-jira producer=True stdio=stderr string_id=tap-jira

Then I thought that the problem was that the column I'm trying to use to the

INCREMENTAL

loading is not flattened. To fix that I've created a new column named

updated_test

(using

stream_maps

). When I select it (without the metadata step), I'm getting exactly what I wanted. A copy of

fields.updated

column But then when I try to use that new column on metadata step, I get the same error as before :

2024-04-07T14:55:32.185389Z [info     ] singer_sdk.exceptions.InvalidReplicationKeyException: Field 'updated_test' is not in schema for stream 'issues' cmd_type=elb consumer=False name=tap-jira producer=True stdio=stderr string_id=tap-jira

What am I doing wrongly here? Is the creation of the column not being correctly done? If not, how should it be done? And, is there a way to set replication-key to a not flattened column? Let me know. Thanks in advance, all the help is welcomed 🙂

A replication key has to be a top-level stream property (i.e. not nested), as far as I am aware. The invalid replication key error is pretty self-explanatory to me: you have created a new property

updated_test

that isn't defined in the tap

issue

stream schema and are trying to set it as the replication key. You probably want to provide a schema override for

updated_test

.

After reading the docu, 'A replication key has to be a top-level stream property (i.e. not nested), as far as I am aware.' now I also get this idea. I'll then follow your advice on overriding the schema, adding the

updated_test

, then with the

stream_map

assign

*updated_test:* fields.updated

, then applying the replication-key. I'll keep you posted. Thanks a lot for the help

Have a look at https://sdk.meltano.com/en/latest/stream_maps.html#automatic-schema-detection and https://sdk.meltano.com/en/latest/stream_maps.html#schema-detection-capabilities-are-limited? Sounds like it could be related.

I'm not having issues on assigning

updated_test: fields.updated

The issue I'm having is that when

metadata:
          issues:
            replication-method: INCREMENTAL
            replication-key: updated_test_1     # no
            updated_test_1:
              is-replication-key: true

tries to get

updated_test_1

as replication-key, it says it does not exist on the catalog. Which does not originally, but I'm creating and overwriting it with:

- name: tap-jira
        schema:
          issues:
            updated_test_1:
              type: ["string", "null"]

😕

Yeah sorry, I didn't really read those docs links before sending.

If a schema is specified for a property that does not yet exist in the discovered stream's schema, the property (and its schema) will be added to the catalog. This allows you to define a full schema for taps such as

tap-dynamodb

that do not themselves have the ability to discover the schema of their streams.

This makes it sound like it should work,..

Hi @Reuben (Matatika) Yes, yes. Just changed it, and the following message does not show up anymore:

2024-04-07T20:12:39.108944Z [warning  ] Stream `updated_test` was not found in the catalog
2024-04-07T20:12:39.109025Z [warning  ] Stream `updated_test_1` was not found in the catalog
2024-04-07T20:12:39.109082Z [warning  ] Stream `updated_test_2` was not found in the catalog

The question here is the order of execution of the commands:

schema

,

select

,

metadata

and

stream_maps

. And how to configure

metadata

to use the newly created columns and not only the tables on the

catalog

. Cause I'm still getting the:

2024-04-07T20:17:11.599170Z [info     ] singer_sdk.exceptions.InvalidReplicationKeyException: Field 'updated_test_1' is not in schema for stream 'issues' cmd_type=extractor name=tap-jira run_id=2d93dcda-09f8-4c0e-8d97-67c35b1fb51c state_id=2024-04-07T201709--tap-jira--target-s3 stdio=stderr

When

metadata:
          issues:
            replication-method: INCREMENTAL
            replication-key: updated_test_1     # no

This is about at my Meltano/Singer knowledge limit, sorry - @Edgar Ramírez (Arch.dev) might have more to say.

Thanks a lot for all the help provided @Reuben (Matatika) 🙌 Let's see if @Edgar Ramírez (Arch.dev) can help us here 😄

Catching up! Is

fields.updated

always present or is it particular to your Jira installation?

Hi Edgar!

(I wouldn't rely on

stream_maps

for setting a replication key, so I'm trying to see if we can change it upstream in the tap itself)

fields.updated

is always present yes

issues.updated

-> this is column is empty (at least on my side)

issues.fields.updated

-> this is the columns I'm talking about, and I want to use as replication-key

Yup, the PR is ensuring the column is not empty

Got it

Correct

Ok perfect

Can you try that PR with

pip_url: git+<https://github.com/MeltanoLabs/tap-jira@refs/pull/71/head>

?

ah ok

Yeah no prob

Thanks a lot @Edgar Ramírez (Arch.dev) for the help. I'll try it out later on, and get you posted. 🙌

Make sure you run

meltano install extractor tap-jira --clean

Hi Edgar, I'll try that later on. Thanks a lot for your help!