hello everyone! I've been trying to use `target-s3...
# troubleshooting
g
hello everyone! I've been trying to use
target-s3
without success. This is the error I'm getting: ```2023-04-19T150241.855769Z [info ] Traceback (most recent call last): cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3 2023-04-19T150241.856082Z [info ] File ".../venv/bin/target-s3", line 8, in <module> cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3 2023-04-19T150241.856522Z [info ] sys.exit(Targets3.cli()) cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3 2023-04-19T150241.856812Z [info ] File ".../venv/lib/python3.9/site-packages/click/core.py", line 1130, in call cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3 2023-04-19T150241.857079Z [info ] return self.main(*args, **kwargs) cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3 2023-04-19T150241.857340Z [info ] File ".../venv/lib/python3.9/site-packages/click/core.py", line 1055, in main cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3 2023-04-19T150241.857606Z [info ] rv = self.invoke(ctx) cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3 2023-04-19T150241.857863Z [info ] File ".../venv/lib/python3.9/site-packages/click/core.py", line 1404, in invoke cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3 2023-04-19T150241.858215Z [info ] return ctx.invoke(self.callback, **ctx.params) cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3 2023-04-19T150241.858504Z [info ] File ".../venv/lib/python3.9/site-packages/click/core.py", line 760, in invoke cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3 2023-04-19T150241.858773Z [info ] return __callback(*args, **kwargs) cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3 2023-04-19T150241.859058Z [info ] File ".../venv/lib/python3.9/site-packages/singer_sdk/target_base.py", line 564, in cli cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3 2023-04-19T150241.859248Z [info ] target.listen(file_input) cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3 2023-04-19T150241.859373Z [info ] File ".../venv/lib/python3.9/site-packages/singer_sdk/io_base.py", line 34, in listen cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3 2023-04-19T150241.859481Z [info ] self._process_lines(file_input) cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3 2023-04-19T150241.859585Z [info ] File ".../venv/lib/python3.9/site-packages/singer_sdk/target_base.py", line 265, in _process_lines cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3 2023-04-19T150241.859689Z [info ] counter = super()._process_lines(file_input) cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3 2023-04-19T150241.859794Z [info ] File ".../venv/lib/python3.9/site-packages/singer_sdk/io_base.py", line 78, in _process_lines cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3 2023-04-19T150241.859896Z [info ] self._process_schema_message(line_dict) cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3 2023-04-19T150241.859997Z [info ] File ".../venv/lib/python3.9/site-packages/singer_sdk/target_base.py", line 364, in _process_schema_message cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3 2023-04-19T150241.860102Z [info ] …
this is my config by the way (keys removed):
Copy code
{
  "aws_access_key": "",
  "aws_secret_access_key": "",
  "aws_region": "us-east-1",
  "bucket": "meltano",
  "prefix": "meltano-test",
  "append_date_to_prefix": true,
  "append_date_to_prefix_grain": "day",
  "append_date_to_filename": false,
  "append_date_to_filename_grain": "day",
  "format_type": "json",
  "flatten_records": false,
  "set_dtype_string": false,
  "flattening_enabled": false
}
p
@guido_trucco there was recently a major refactor to this target https://github.com/crowemi/target-s3/pull/8 with a new direction, comment from the maintainer:
I was thinking this target could become target-data-lake and serve as a multi-cloud storage/multi-format target.
We'll either pin the pre-1.0 release in meltanohub or remove this target in its current state as target-s3. I would suggest trying one of the other s3 file type specific variants like https://hub.meltano.com/loaders/target-s3-csv/ or https://hub.meltano.com/loaders/target-s3-parquet/ or https://hub.meltano.com/loaders/target-s3-avro/
g
cool, I was aiming for a datalake approach. Hope it comes out soon!
p
Update - I opened a PR to refresh the settings metadata for this target https://github.com/meltano/hub/pull/1267 so if you wanted to give it another shot after this merges then you'll likely have better luck configuring it.