https://linen.dev logo
Join SlackCommunities
Powered by
We're looking to migrate our meltano state backend...
# singer-taps
m
We're looking to migrate our meltano state backend from postgres to s3. One benefit of the postgres backend is that I've got a regular task using tap-postgres to backup meltano state to Snowflake. Before I look into developing one, is there a tap for S3 state files? Alternatively would it be a better idea to update meltano to allow some sort of 
meltano state to-target target-snowflake
or 
meltano run state target-snowflake
functionality that uses inbuilt methods to send state data to a given target, regardless of the state backend in use?
v
Why not swap to the s3 backend and then just use a tap-s3 (spreadsheetsanywhere or universal-file) and target-snowflake?
e
So what's the Postgres table(s) you were backing up to Snowflake?
m
Thank you for the quick reply! The tables are called 
state
and 
runs
. I set up the sync process a while back, but the metadata settings are:
Copy code
{
  "public-runs": {
    "replication-method": "INCREMENTAL",
    "replication-key": "id",
    "table-key-properties": [
      "id"
    ]
  },
  "public-state": {
    "replication-method": "INCREMENTAL",
    "replication-key": "updated_at",
    "table-key-properties": [
      "state_id"
    ]
  }
}
👍 1
@visch - I'll look into the taps you've suggested, - the format of a state.json file in the backend looks like this, which might make things difficult (or I could use a mapper):
Copy code
{
  "completed": {
    "singer_state": {
      "bookmarks": {
        "DSSADM-DWH_TABLE": {
          "last_replication_method": "FULL_TABLE",
          "version": 1699581819348,
          "ORA_ROWSCN": null
        }
      },
      "currently_syncing": null
    }
  },
  "partial": {}
}
After fixing a 
numpy<2
issue, I've managed to get tap-universal file working with the following config:
Copy code
{
        "protocol":"s3",
        "file_path":"meltano-state-dev-2-5-1",
        "file_type":"jsonl",
        "file_regex":".*\\.json",
        "jsonl_type_coercion_strategy":"envelope",
...
}
❤️ 1
v
Can you put in an issue about numpy?
m
yep... on it now 🙂
➕ 1
2 Views
Open in Slack
PreviousNext
Powered by