david_gauthier-dussureault
01/27/2023, 8:15 PMreplication_key
is set:
class GetProjectMentionsStream(BrandMentionsStream):
"""Define custom stream."""
name = "GetProjectMentions"
primary_keys = ["project_id", "url"]
replication_key = "published"
schema = th.PropertiesList(
th.Property("project_id", th.IntegerType,),
th.Property("published", th.DateTimeType,),
th.Property("url", th.StringType,),
th.Property("title", th.StringType,),
th.Property("image_url", th.StringType,),
th.Property("performance", th.IntegerType,),
th.Property("language", th.StringType,),
th.Property("country", th.StringType,),
th.Property("social_network", th.StringType,),
th.Property("text", th.StringType,),
th.Property("sentiment", th.StringType,),
th.Property("linked", th.IntegerType,),
th.Property("hashtags", th.ArrayType(th.StringType),),
th.Property("author", th.ObjectType(),),
th.Property("social", th.ObjectType(),),
).to_dict()
When I run meltano state get dev:tap-brandmentions-to-target-jsonl
, I see that the state bookmark is saved properly:
{
"singer_state": {
"bookmarks": {
"GetProjectMentions": {
"replication_key": "published",
"replication_key_value": "2023-01-27 00:04:00"
}
}
}
}
But when I do subsequent runs meltano run tap-brandmentions target-jsonl
, the full dataset is always sent to my target-jsonl.
What am I missing for my stream to only send incremental data to my target?