Getting a little bit of trouble trying to understand the documentation surrounding cli’s state commands and it’s inputs. Trying to find some answers here on slack too but no luck. i’m trying to create a state file before running an incremental pipeline ( it’s pretty big and i won’t need to load everything) This is what sits underneath a test file:

{
  "singer_state": {
    "bookmarks": {
      "public-chargeable_items": {
        "field": "time",
        "last_record": "2023-04-10T00:00:00Z"
      }
    }
  }
}

meltano state set --force core-aws:tap-postgres-to-target-jsonl-transactional --input-file /path_to_file/testfile.json when i try to run it meltano can’t identify the state and will start to load everything. I don’t know what i’m missing here. Any hints?

Apologies if this seems like a dumb suggestion, but have you checked the formatting of your state file? Those field names don’t seem right to me. For instance, a test state for mine looks like

{
    "singer_state": {
        "bookmarks": {
            "public-table_name": {
                "last_replication_method": "INCREMENTAL",
                "replication_key": "id",
                "version": 1679511215154,
                "replication_key_value": 140
            }
        },
        "currently_syncing": null
    }

Yep I had a sample state from testing the meltanolabs tap-postgres variant and my state looks like:

{
  "singer_state": {
    "bookmarks": {
      "public-newtable": {
        "replication_key": "key",
        "replication_key_value": 3
      }
    }
  }
}

where my meltano.yml looks like:

select:
    - public-newtable.*
    metadata:
    metadata:
      public-newtable:
        replication-method: INCREMENTAL
        replication_key: key
        key_properties:
        - key

so the replication key is

key

and the most recent value found is

3

Ok, thank you both - can’t find any of those bookmarks patterns anywhere in documentation. Apparently meltano still can’t find the state even using the new pattern here

can’t find any of those bookmarks patterns anywhere in documentation.

@guilherme_petris yeah it looks like youre right. Would you mind creating an issue to document your experience? It would probably be helpful to document some examples like we provided and how to get/set them and maybe if theres a pattern for generating your own state file (e.g. I'm thinking maybe tricks like recommending doing a small 1 day sync if theres a

start_date

parameter then manipulating the state from there 🤷).

It should be the sort of that that can be done programmatically, but I don’t know how tap/target specific the formatting has to be. I’d love to see a

meltano state init

command, or something like it, that generates the state for existing target tables. Number of ways to do this (I’ve done it myself in a hardcoded way), but Meltano already knows the targets and how to check if they exist, whether the table is intended to be incremental, the column for checking state, etc, so it seems like all the necessary information is already available, even without actually committing any writes to the target.

Good idea! Yeah that would be really helpful and I can also think of a few reasonable ways it could be done

I have no idea if an issue exists already, but seems like a problem plenty of folks have had. I have a gist of the very tailor-made way I did it here (it’s really pretty inflexible, but the basic idea is — use the yaml to get the table/columns, and the .env to get password info, then format a json blob and write it to STDOUT).