https://linen.dev logo
#singer-tap-development
Title
# singer-tap-development
d

david_gauthier-dussureault

01/27/2023, 8:15 PM
I have a custom tap with a stream I want to run incrementally. My stream definition is this and the
replication_key
is set:
Copy code
class GetProjectMentionsStream(BrandMentionsStream):
    """Define custom stream."""
    name = "GetProjectMentions"
    primary_keys = ["project_id", "url"]
    replication_key = "published"
    schema = th.PropertiesList(
        th.Property("project_id", th.IntegerType,),
        th.Property("published", th.DateTimeType,),
        th.Property("url", th.StringType,),
        th.Property("title", th.StringType,),
        th.Property("image_url", th.StringType,),
        th.Property("performance", th.IntegerType,),
        th.Property("language", th.StringType,),
        th.Property("country", th.StringType,),
        th.Property("social_network", th.StringType,),
        th.Property("text", th.StringType,),
        th.Property("sentiment", th.StringType,),
        th.Property("linked", th.IntegerType,),
        th.Property("hashtags", th.ArrayType(th.StringType),),
        th.Property("author", th.ObjectType(),),
        th.Property("social", th.ObjectType(),),
    ).to_dict()
When I run
meltano state get dev:tap-brandmentions-to-target-jsonl
, I see that the state bookmark is saved properly:
Copy code
{
  "singer_state": {
    "bookmarks": {
      "GetProjectMentions": {
        "replication_key": "published",
        "replication_key_value": "2023-01-27 00:04:00"
      }
    }
  }
}
But when I do subsequent runs
meltano run tap-brandmentions target-jsonl
, the full dataset is always sent to my target-jsonl. What am I missing for my stream to only send incremental data to my target?