Has anyone experienced and correct the following error? Additionally, since listing state is broken (https://github.com/meltano/meltano/issues/8425) if there are files in the root directory of the bucket, can I expect Meltano to operate correctly, or does it depend on that operation?

2024-07-21T23:05:08.040258Z [info     ] Writing state to AWS S3       
2024-07-21T23:05:08.051446Z [info     ] Found credentials in environment variables.
2024-07-21T23:05:08.129575Z [info     ] Found endpoint for s3 via: environment_service.
2024-07-21T23:05:08.161200Z [info     ] smart_open.s3.MultipartWriter('mybucket', 'state/dev:tap-mssql-to-target-s3/lock'): uploading part_num: 1, 17 bytes (total 0.000GB)
2024-07-21T23:05:09.171625Z [info     ] smart_open.s3.MultipartWriter('mybucket', 'state/dev:tap-mssql-to-target-s3/lock'): uploading part_num: 1, 17 bytes (total 0.000GB)
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

An error occurred (InvalidObjectName) when calling the UploadPart operation: The specified object name is not valid.

Additionally, since listing state is broken (https://github.com/meltano/meltano/issues/8425) if there are files in the root directory of the bucket, can I expect Meltano to operate correctly, or does it depend on that operation?

The issue is specific to Google Cloud, and only if there happens to be a file at the root of the bucket, but you seem to be using S3. Does the

meltano state list

operation also not work for you? I have not seen this error in particular, and the AWS docs don't seem to suggest

'state/dev:tap-mssql-to-target-s3/lock'

should be problematic. Did you previously create the bucket in S3?

It is a self-hosted S3 (VAST DB) so may not be entirely representative of AWS S3, although I do not know the specific differences. Trying through s3cmd I get the following:

hayden.ness@Machine ~ % s3cmd cp <s3://mybucket/sample.parquet> <s3://mybucket/dev:state/cptest.parquet>
ERROR: Copy failed for: '<s3://mybucket/sample.parquet>' (400 (InvalidObjectName): The specified object name is not valid.)

So it seems I'll have to work around that too, unless I can somehow remove the colon from the state name?

Ah, it makes sense if it's a semi-compatible S3 API. I don't see anything in their docs that suggests a colon would cause issues:

• Object names (keys) must be up to 1024 characters long.

• Object names can contain up to 200 '/' characters.

• Object names can contain slashes. This is useful for 'S3 directory' type usage.

unless I can somehow remove the colon from the state name?

Unfortunately, that's not currently possible and the delimiter is hardcoded: https://github.com/meltano/meltano/blob/72454b4a973e279c383796bd37d5f8a6fd128f13/src/meltano/core/block/extract_load.py#L887-L924 But please do create an issue!

I was able to get past that particular trouble by using "meltano state set custom-name --input-file file". However, due to the other problem, Meltano fails to retrieve the state during the run, and terminates. If the latter issue is fixed it should be fine, I've put in a comment on the issue as a notice to those interested. I'll just implement the same system myself for now.

The crashing issue was fixed by https://github.com/meltano/meltano/pull/8648 and will ship in the next release