Hmm, this seems to no longer be helping. The pkg_r...
# announcements
p
Hmm, this seems to no longer be helping. The pkg_resources package seems to now have been completely removed from the setuptools repository, so even if forcing its use, it can't be downloaded and so the build of the plugin fails at install time (I'm trying target-s3).
Copy code
Using Python 3.13.12 environment at: .meltano/loaders/target-s3/venv                                                                                                                                           │
│ Resolved 60 packages in 548ms                                                                                                                                                                                  │
│    Building pandas==1.5.3                                                                                                                                                                                      │
│    Building pyarrow==10.0.1                                                                                                                                                                                    │
│   × Failed to build `pyarrow==10.0.1`                                                                                                                                                                          │
│   ├─▶ The build backend returned an error                                                                                                                                                                      │
│   ╰─▶ Call to `setuptools.build_meta:__legacy__.build_wheel` failed (exit status: 1)                                                                                                                           │
│                                                                                                                                                                                                                │
│       [stderr]                                                                                                                                                                                                 │
│       Traceback (most recent call last):                                                                                                                                                                       │
│         File "<string>", line 14, in <module>                                                                                                                                                                  │
│           requires = get_requires_for_build({})                                                                                                                                                                │
│         File "/Users/pklavins/.cache/uv/builds-v0/.tmpfD6fP9/lib/python3.13/site-packages/setuptools/build_meta.py", line 333, in get_requires_for_build_wheel                                                 │
│           return self._get_build_requires(config_settings, requirements=[])                                                                                                                                    │
│                  ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                    │
│         File "/Users/pklavins/.cache/uv/builds-v0/.tmpfD6fP9/lib/python3.13/site-packages/setuptools/build_meta.py", line 301, in _get_build_requires                                                          │
│           self.run_setup()                                                                                                                                                                                     │
│           ~~~~~~~~~~~~~~^^                                                                                                                                                                                     │
│         File "/Users/pklavins/.cache/uv/builds-v0/.tmpfD6fP9/lib/python3.13/site-packages/setuptools/build_meta.py", line 520, in run_setup                                                                    │
│           super().run_setup(setup_script=setup_script)                                                                                                                                                         │
│           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                         │
│         File "/Users/pklavins/.cache/uv/builds-v0/.tmpfD6fP9/lib/python3.13/site-packages/setuptools/build_meta.py", line 317, in run_setup                                                                    │
│           exec(code, locals())                                                                                                                                                                                 │
│           ~~~~^^^^^^^^^^^^^^^^                                                                                                                                                                                 │
│         File "<string>", line 36, in <module>                                                                                                                                                                  │
│       ModuleNotFoundError: No module named 'pkg_resources'                                                                                                                                                     │
│                                                                                                                                                                                                                │
│       hint: This error likely indicates that `pyarrow@10.0.1` depends on `pkg_resources`, but doesn't declare it as a build dependency. If `pyarrow` is a first-party package, consider adding `pkg_resources` │
│       to its `build-system.requires`. Otherwise, either add it to your `pyproject.toml` under:                                                                                                                 │
│                                                                                                                                                                                                                │
│       [tool.uv.extra-build-dependencies]                                                                                                                                                                       │
│       pyarrow = ["pkg_resources"]                                                                                                                                                                              │
│                                                                                                                                                                                                                │
│       or `uv pip install pkg_resources` into the environment and re-run with `--no-build-isolation`.                                                                                                           │
│   help: `pyarrow` (v10.0.1) was included because `target-s3` (v1.0.7) depends on `pyarrow`                                                                                                                     │
r
Seems to be an issue with Python 3.12, since
pyarrow==10.0.01
does not have pre-built wheels for these versions: https://pypi.org/project/pyarrow/10.0.1/#files
target-s3
probably needs some love to get it properly working, but you could try dropping down to Python 3.11?
Copy code
- name: target-s3
    python: python3.11
    variant: crowemi
    pip_url: git+<https://github.com/crowemi/target-s3.git> setuptools<82
p
I tried with 3.11 first, that's what we were using, but it even fails there. pkg_resources simply can't be downloaded during the target build process. Try it with an empty Meltano project and target-s3, even with 3.11. Here's some definitive text about pkg_resource removal on 8 Feb 2026 that I found "The pkg_resources package was deprecated in setuptools 67.5.0 (05 Mar 2023) and removed on setuptools 82.0.0 (8 Feb 2026).". https://pypi.org/project/packagedata/
I agree that target-s3 needs some love too. The problem is that our daily ETL's are now failing due to not being able to copy to S3, so the fix needs to be something more definitive and sure.
As far as I can see, the poetry file in target-s3 is referencing old versions of dependencies. I am not familiar at all with poetry and am not the developer of target-s3. What's the best way of boosting the dependencies in target-s3 from the outside?
r
Copy code
reuben@reuben-Inspiron-14-5425:/tmp/p$ cat meltano.yml 
version: 1
default_environment: dev
project_id: 019c571b-25b4-7c0f-aa84-6f85de4bd127
environments:
- name: dev
- name: staging
- name: prod
plugins:
  loaders:
  - name: target-s3
    python: python3.11
    variant: crowemi
    pip_url: git+<https://github.com/crowemi/target-s3.git> setuptools<82
reuben@reuben-Inspiron-14-5425:/tmp/p$ meltano invoke target-s3
2026-02-13T13:31:07.470728Z [info     ] Environment 'dev' is active   
2026-02-13T13:31:07.569355Z [info     ] Installing loader 'target-s3' 
2026-02-13T13:31:08.461526Z [info     ] Installed loader 'target-s3'  
/tmp/p/.meltano/loaders/target-s3/venv/lib/python3.11/site-packages/fs/__init__.py:4: UserWarning: pkg_resources is deprecated as an API. See <https://setuptools.pypa.io/en/latest/pkg_resources.html>. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  __import__("pkg_resources").declare_namespace(__name__)  # type: ignore
Traceback (most recent call last):
  File "/tmp/p/.meltano/loaders/target-s3/venv/bin/target-s3", line 10, in <module>
    sys.exit(Targets3.cli())
             ^^^^^^^^^^^^^^
  File "/tmp/p/.meltano/loaders/target-s3/venv/lib/python3.11/site-packages/click/core.py", line 1485, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/p/.meltano/loaders/target-s3/venv/lib/python3.11/site-packages/click/core.py", line 1406, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/tmp/p/.meltano/loaders/target-s3/venv/lib/python3.11/site-packages/click/core.py", line 1269, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/p/.meltano/loaders/target-s3/venv/lib/python3.11/site-packages/click/core.py", line 824, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/p/.meltano/loaders/target-s3/venv/lib/python3.11/site-packages/singer_sdk/target_base.py", line 549, in invoke
    target = cls(
             ^^^^
  File "/tmp/p/.meltano/loaders/target-s3/venv/lib/python3.11/site-packages/singer_sdk/target_base.py", line 76, in __init__
    super().__init__(
  File "/tmp/p/.meltano/loaders/target-s3/venv/lib/python3.11/site-packages/singer_sdk/plugin_base.py", line 153, in __init__
    self._validate_config(raise_errors=validate_config)
  File "/tmp/p/.meltano/loaders/target-s3/venv/lib/python3.11/site-packages/singer_sdk/plugin_base.py", line 392, in _validate_config
    raise ConfigValidationError(summary)
singer_sdk.exceptions.ConfigValidationError: Config validation failed: 'format_type' is a required property
JSONSchema was: {'type': 'object', 'properties': {'format': {'type': ['object', 'null'], 'properties': {'format_type': {'type': ['string'], 'enum': ['parquet', 'json', 'jsonl']}, 'format_parquet': {'type': ['object', 'null'], 'properties': {'validate': {'type': ['boolean', 'null'], 'default': False}, 'get_schema_from_tap': {'type': ['boolean', 'null'], 'default': False, 'description': "Set true if you want to declare schema of the                                         resulting parquet file based on taps. Doesn't                                          work with 'anyOf' types or when complex data is                                         not defined at element level. Doesn't work with                                          validate option for now."}}}, 'format_json': {'type': ['object', 'null'], 'properties': {}}, 'format_csv': {'type': ['object', 'null'], 'properties': {}}}, 'required': ['format_type']}, 'cloud_provider': {'type': ['object', 'null'], 'properties': {'cloud_provider_type': {'type': ['string'], 'enum': ['aws']}, 'aws': {'type': ['object', 'null'], 'properties': {'aws_access_key_id': {'type': ['string', 'null'], 'secret': True, 'writeOnly': True}, 'aws_secret_access_key': {'type': ['string', 'null'], 'secret': True, 'writeOnly': True}, 'aws_session_token': {'type': ['string', 'null'], 'secret': True, 'writeOnly': True}, 'aws_region': {'type': ['string']}, 'aws_profile_name': {'type': ['string', 'null']}, 'aws_bucket': {'type': ['string']}, 'aws_endpoint_override': {'type': ['string', 'null']}}, 'required': ['aws_region', 'aws_bucket']}}, 'required': ['cloud_provider_type']}, 'prefix': {'type': ['string', 'null'], 'description': 'The prefix for the key.'}, 'stream_name_path_override': {'type': ['string', 'null'], 'description': 'The S3 key stream name override.'}, 'include_process_date': {'type': ['boolean', 'null'], 'default': False, 'description': 'A flag indicating whether to append _process_date to record.'}, 'use_raw_stream_name': {'type': ['boolean', 'null'], 'default': False, 'description': 'A flag to force the filename to be identical to the stream name.'}, 'append_date_to_prefix': {'type': ['boolean', 'null'], 'default': True, 'description': 'A flag to append the date to the key prefix.'}, 'partition_name_enabled': {'type': ['boolean', 'null'], 'default': False, 'description': "A flag (only works if append_date_to_prefix is enabled) to have partitioning name formatted e.g. 'year=2023/month=01/day=01'."}, 'append_date_to_prefix_grain': {'type': ['string', 'null'], 'default': 'day', 'description': 'The grain of the date to append to the prefix.', 'enum': dict_keys(['year', 'month', 'day', 'hour', 'minute', 'second', 'microsecond'])}, 'append_date_to_filename': {'type': ['boolean', 'null'], 'default': True, 'description': 'A flag to append the date to the key filename.'}, 'append_date_to_filename_grain': {'type': ['string', 'null'], 'default': 'day', 'description': 'The grain of the date to append to the filename.', 'enum': dict_keys(['year', 'month', 'day', 'hour', 'minute', 'second', 'microsecond'])}, 'max_batch_age': {'type': ['number', 'null'], 'default': 5.0, 'description': 'Maximum time in minutes between state messages when records are streamed in.'}, 'max_batch_size': {'type': ['integer', 'null'], 'default': 10000, 'description': 'Maximum size of batches when records are streamed in.'}, 'partition_by': {'type': ['array', 'null'], 'items': {'type': ['string']}, 'description': "List of key-value strings (e.g., 'tenant=${TENANT}') to prepend as partitions in the S3 key path after the stream name."}, 'add_record_metadata': {'type': ['boolean', 'null'], 'description': 'Add metadata to records.'}, 'load_method': {'type': ['string', 'null'], 'default': <TargetLoadMethods.APPEND_ONLY: 'append-only'>, 'description': 'The method to use when loading data into the destination. `append-only` will always write all input records whether that records already exists or not. `upsert` will update existing records and insert new records. `overwrite` will delete all existing records and insert all input records.', 'enum': [<TargetLoadMethods.APPEND_ONLY: 'append-only'>, <TargetLoadMethods.UPSERT: 'upsert'>, <TargetLoadMethods.OVERWRITE: 'overwrite'>]}, 'stream_maps': {'type': ['object', 'null'], 'properties': {}, 'description': 'Config object for stream maps capability. For more information check out [Stream Maps](<https://sdk.meltano.com/en/latest/stream_maps.html).>'}, 'stream_map_config': {'type': ['object', 'null'], 'properties': {}, 'description': 'User-defined config values to be used within map expressions.'}, 'flattening_enabled': {'type': ['boolean', 'null'], 'description': "'True' to enable schema flattening and automatically expand nested properties."}, 'flattening_max_depth': {'type': ['integer', 'null'], 'description': 'The max depth to flatten schemas.'}}}
I would expect it to fail earlier with
setuptools
unpinned (I have no config set so the actual error makes sense).
Like so:
Copy code
reuben@reuben-Inspiron-14-5425:/tmp/p$ cat meltano.yml 
version: 1
default_environment: dev
project_id: 019c571b-25b4-7c0f-aa84-6f85de4bd127
environments:
- name: dev
- name: staging
- name: prod
plugins:
  loaders:
  - name: target-s3
    python: python3.11
    variant: crowemi
    pip_url: git+<https://github.com/crowemi/target-s3.git>
reuben@reuben-Inspiron-14-5425:/tmp/p$ meltano invoke target-s3
2026-02-13T13:35:07.136483Z [info     ] Environment 'dev' is active   
2026-02-13T13:35:07.234050Z [info     ] Installing loader 'target-s3' 
2026-02-13T13:35:07.751959Z [info     ] Installed loader 'target-s3'  
Traceback (most recent call last):
  File "/tmp/p/.meltano/loaders/target-s3/venv/bin/target-s3", line 4, in <module>
    from target_s3.target import Targets3
  File "/tmp/p/.meltano/loaders/target-s3/venv/lib/python3.11/site-packages/target_s3/target.py", line 7, in <module>
    from singer_sdk.target_base import Target
  File "/tmp/p/.meltano/loaders/target-s3/venv/lib/python3.11/site-packages/singer_sdk/__init__.py", line 5, in <module>
    from singer_sdk import streams
  File "/tmp/p/.meltano/loaders/target-s3/venv/lib/python3.11/site-packages/singer_sdk/streams/__init__.py", line 5, in <module>
    from singer_sdk.streams.core import Stream
  File "/tmp/p/.meltano/loaders/target-s3/venv/lib/python3.11/site-packages/singer_sdk/streams/core.py", line 26, in <module>
    from singer_sdk.helpers._batch import (
  File "/tmp/p/.meltano/loaders/target-s3/venv/lib/python3.11/site-packages/singer_sdk/helpers/_batch.py", line 12, in <module>
    import fs
  File "/tmp/p/.meltano/loaders/target-s3/venv/lib/python3.11/site-packages/fs/__init__.py", line 4, in <module>
    __import__("pkg_resources").declare_namespace(__name__)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'pkg_resources'
What's the best way of boosting the dependencies in target-s3 from the outside?
Probably forking it and making the dependency changes yourself, unfortunately. 😬
p
If you look in the target-s3 repo, you see sporadic development and multiple new tags, but the poetry definition still requires the original version 1.0.7, and that's what gets reflected in the meltano build. The latest version is 3.3.0. Unless it's trivial to set up using the Meltano SDK etc., we can't really fork the repo. I suppose we have to look for alternative S3 targets.
Never mind. Thanks @Reuben (Matatika), you gave me enough command line help to get this running. It works with "setuptools<81", as per the error message when it fails. I'll go with that for our needs at the moment.
👍 1
r
I don't believe the
poetry.lock
is respected by pip, so really the problem is the explicit pin to
pyarrow==10.0.1
in the
pyproject.toml
in causing the build error.
Oh, I see what you are saying.
Copy code
version = "1.0.7"
in the
pyproject.toml
is purely informational though and likely just hasn't been updated in a while either. The
pip_url
tells you what version of
target-s3
you are installing - in the case of
git+<https://github.com/crowemi/target-s3.git>
, you're getting the latest from the default branch (
main
), rather than a specific version.
e
The maintainer is rather active on GitHub, so it might be worth sending them a PR
p
Cool, it's a relief to know that the 1.0.7 is informational... With setuptools<81, pyarrow doesn't get installed, and so pkg_resources doesn't raise its head. I'm quite out of practice with dependency management, but good to have it working again. I'll maybe try the e-mail approach and enquire about 3.14 support or something like that to the dev, I feel I couldn't come up with a PR too quickly...
I just e-mailed the dev, see what he says. Thanks again.
e
uv overrides also seem to work:
Copy code
# meltano.yml
env:
  UV_OVERRIDE: ./overrides.txt
plugins:
  loaders:
  - name: target-s3
    python: python3.11
    variant: crowemi
    pip_url: git+<https://github.com/crowemi/target-s3.git>
Copy code
# overrides.txt
pyarrow==23.0.0
setuptools==81.0.0
r
Even with Python 3.13?
e
Let me try that
Ah,
pkg_resources
is a build dependency there. One sec
👍 1
This combination seems to work for Python 3.13
Copy code
# overrides.txt
boto3==1.42.42
fsspec==2026.2.0
pandas==2.3.3
pyarrow==23.0.0
s3fs==2026.2.0
setuptools==80.10.2
Or at least the plugin can be installed and
meltano invoke target-s3 --help
doesn't crash
p
I simply can't get target-s3 to install with python 3.13. If I try to use the overrides.txt file as you show, inside a 3.13 venv, meltano nevertheless installs a 3.11 python venv for target-s3, i.e., files under .meltano/loaders/target-s3/venv/lib/python3.11. And then the invoke fails because it can't import pkg_resources. At the moment, the only working solution is a 3.11 venv with pip_url specifying setuptools<81.
972 Views