GitHub
05/06/2025, 5:58 PMPyFilesystem has arguably become stagnant. I'm concerned that the Python ecosystem is moving and PyFilesytem and plugins will be left behind.
A recent example of this was the release of urllib3 2.0, which is incompatible with botocore. The offender is currently the botocore library, but it's actively developed so it's more likely that the maintainers will add support for urllib3 2.0+, but that may require them to bump the major version of their package, which fs-s3fs will not be compatible with, and it's not clear if the fs-s3fs is still being maintained (last activity on 2019).
## Alternatives
### fsspec
#### Advantages
• Automatically read/write from S3 when a s3://... URL is used, similar to the current situation with PyFilesystem.
• Good community support and maintenance. Used by Dask.
#### Disadvantages
• See breaking changes
### smart-open
#### Advantages
• Simple to use
#### Disadvantages
• Requires explicitly defining the transport config
• See breaking changes
## Breaking changes
Changing the library we use to support different file storage systems will break any taps that currently have rely on having fs-s3fs as a dependency.
cc @kgpayne
### Code
No response
meltano/sdkGitHub
05/06/2025, 5:58 PM