https://linen.dev logo
Join Slack
Powered by
Do SDK targets that process `BATCH` messages suppo...
# singer-targets
r
Do SDK targets that process 
BATCH
messages support manifests referencing public HTTPS URLs? Or it is still only local filesystem/S3 as per https://sdk.meltano.com/en/v0.44.3/batch.html?
✅ 1
The HTTPS URL I am dealing with looks to be backed by S3, so I wonder if there is a way to convert it to a supported S3 URL...
Copy code
{
  "type": "BATCH",
  "stream": "custom_event",
  "encoding": {
    "format": "jsonl"
  },
  "manifest": [
    "<https://iterable-exports-prd-us-east-1.s3.amazonaws.com/org-.../export-.../.../file-0000000000001.json?X-Amz-Security-Token=...&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20250212T140337Z&X-Amz-SignedHeaders=host&X-Amz-Expires=1799&X-Amz-Credential=...&X-Amz-Signature=>...",
    "<https://iterable-exports-prd-us-east-1.s3.amazonaws.com/org-.../export-.../.../file-0000000000002.json?X-Amz-Security-Token=...&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20250212T140337Z&X-Amz-SignedHeaders=host&X-Amz-Expires=1800&X-Amz-Credential=...&X-Amz-Signature=>...",
    "<https://iterable-exports-prd-us-east-1.s3.amazonaws.com/org-.../export-.../.../file-0000000000003.json?X-Amz-Security-Token=...&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20250212T140337Z&X-Amz-SignedHeaders=host&X-Amz-Expires=1800&X-Amz-Credential=...&X-Amz-Signature=>...",
    "<https://iterable-exports-prd-us-east-1.s3.amazonaws.com/org-.../export-.../.../file-0000000000004.json?X-Amz-Security-Token=...&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20250212T140337Z&X-Amz-SignedHeaders=host&X-Amz-Expires=1799&X-Amz-Credential=...&X-Amz-Signature=>...",
    "<https://iterable-exports-prd-us-east-1.s3.amazonaws.com/org-.../export-.../.../file-0000000000005.json?X-Amz-Security-Token=...&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20250212T140337Z&X-Amz-SignedHeaders=host&X-Amz-Expires=1800&X-Amz-Credential=...&X-Amz-Signature=>..."
  ]
}
Probably not due to all the 
X-Amz-*
query params.
I think the answer is no:
Copy code
sqlalchemy.exc.ProgrammingError: (snowflake.connector.errors.ProgrammingError) 001020 (42601): SQL compilation error:
invalid source URL scheme for PUT command: '<https://iterable-exports-prd-us-east-1.s3.amazonaws.com/org-.../export-.../.../file-0000000000001.json?X-Amz-Security-Token=...&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20250212T145235Z&X-Amz-SignedHeaders=host&X-Amz-Expires=1799&X-Amz-Credential=...&X-Amz-Signature=...>'. Only local file URL is allowed.
[SQL: put %(file_uri)s '@~/target-snowflake/custom_event-46f32aca-fd20-4340-ab06-98f8510d7433']
[parameters: {'file_uri': '<https://iterable-exports-prd-us-east-1.s3.amazonaws.com/org-.../export-.../.../file-0000000000001.json?X-Amz-Security-Token=...&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20250212T145235Z&X-Amz-SignedHeaders=host&X-Amz-Expires=1799&X-Amz-Credential=...&X-Amz-Signature=...'}>]
(Background on this error at: <https://sqlalche.me/e/20/f405>)
e
Yup, not currently supported
I wanna switch the BATCH implementation to 
fsspec
. HTTP locations would be supported out-of-the-box, I think.
➕ 1
r
Big vouch for 
fsspec
from me - we use it in a couple places.
HTTP locations would be supported out-of-the-box, I think
Yep, and a bunch of others too: https://filesystem-spec.readthedocs.io/en/latest/api.html#built-in-implementations
Open in Slack
PreviousNext
Powered by