Hey guys Meltano uses locking <https docs meltano com concep Meltano #troubleshooting

Hey guys .. Meltano uses locking <https://docs.mel...

bassel

07/24/2023, 10:51 PM

Hey guys .. Meltano uses locking https://docs.meltano.com/concepts/state_backends#locking when specifying state backends for the replication metadata .. I had it working before, but right now, I'm running the pipeline and its not generating any lock file at all .. don't know if this is an expected behaviour or not. it only generate state.json where it has all the replication method metadata

bassel

07/24/2023, 10:57 PM

More details about the logs I'm having when running the pipeline

Copy code

Making request for OperationModel(name=DeleteObjects) with params: {'url_path': '?delete', 'query_string': {}, 'method': 'POST', 'headers': {'User-Agent': 'Boto3/1.26.147 Python/3.9.16 Linux/4.14.318-241.531.amzn2.x86_64 exec-env/AWS_ECS_EC2 Botocore/1.29.147', 'Content-MD5': 'FdYOXo6N5+HngpLwbQKDSg=='}, 'body': b'<Delete xmlns="<http://s3.amazonaws.com/doc/2006-03-01/>"><Object><Key>us-east-1/metadata/mysql-to-s3/lock</Key></Object></Delete>', 'auth_path': '/etl-pipeline?delete', 'url': '<https://etl-pipeline.s3.amazonaws.com/?delete>', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fa605f21e50>, 'has_streaming_input': False, 'auth_type': 'v4', 's3_redirect': {'redirected': False, 'bucket': 'etl-pipeline', 'params': {'Bucket': 'etl-pipeline', 'Delete': {'Objects': [{'Key': 'us-east-1/metadata/mysql-to-s3/lock'}]}}}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'checksum': {'request_algorithm': 'conditional-md5'}}}

bassel

07/24/2023, 10:59 PM

the state-backend uri is not only a bucket .. its a bucket with folders in it:

<s3://etl-pipeline/us-east-1/metadata>

and the etl command end with:

--state-id=mysql-to-s3

edgar_ramirez_mondragon

07/25/2023, 5:57 PM

Hey Basel! • Were you seeing the state lock before but not anymore? • what version of Meltano you using? • did you upgrade recently?

bassel

07/25/2023, 5:59 PM

Hey @edgar_ramirez_mondragon

• Were you seeing the state lock before but not anymore?

Yes, i was seeing it before

• what version of Meltano you using?

Latest - from the docker image

• did you upgrade recently?

bassel

07/25/2023, 5:59 PM

is it necessary to have ?

edgar_ramirez_mondragon

07/25/2023, 6:02 PM

Yeah, it's necessary if you have concurrent pipelines

bassel

07/25/2023, 6:07 PM

i noticed that when I set the uri to bucket + folders the lock file disappeared .. but when it was only a bucket, it was showing

edgar_ramirez_mondragon

07/25/2023, 6:47 PM

Ok, I think this is the same as https://github.com/meltano/meltano/issues/7938. If the S3 URI is

<s3://some_bucket/some/path>

, the backend is trying to acquire the lock in

<s3://some_bucket/some/path/some/path/my-state-id/lock>

cody_hanson

07/25/2023, 10:19 PM

@bassel Thanks for calling this out! If there are no privacy or security implications of doing so, could you share the specific value of your

MELTANO_STATE_BACKEND_URI

setting? Also: are you seeing the same path duplication as described in the bug report that @edgar_ramirez_mondragon shared? And are you running into errors when running the pipeline or are you just not seeing the lock file as you would expect?

bassel

07/26/2023, 1:53 PM

@cody_hanson

Could you share the specific value of your
MELTANO_STATE_BACKEND_URI
setting

Copy code

state_backend:
  uri: $STATE_BACKEND_URI
  s3:
    aws_access_key_id: $STATE_BACKEND_AWS_ACCESS_KEY_ID
    aws_secret_access_key: $STATE_BACKEND_AWS_SECRET_ACCESS_KEY

the uri field was:

<s3://etl-pipeline/us-east-1/provider/metadata>

and the meltano's elt command ended with this flag:

--state-id=mysql-to-s3

So, what I was expecting: to have a folder named

mysql-to-s3

with two files in it:

lock & state.json

(that's what I had before when the

uri

was only a bucket) and now I'm only having the

state.json

I guess its not an issue - since out case will not have two or more pipelines running at the same time from the same source to the same destination .

Also: are you seeing the same path duplication as described in the bug report that @edgar_ramirez_mondragon shared?

Had it before .. it was an issue in the uri itself .. then not anymore, but with only

state.json

lock

And are you running into errors when running the pipeline or are you just not seeing the lock file as you would expect?

Getting an error where it complains about the state and lock files, when running for the first time - if they don't exist, then they will get created.

Open in Slack

Previous Next