<#1679 bug: Consider an alternative filesystem fra...
# meltano-repo-activity
g
#1679 bug: Consider an alternative filesystem framework to PyFilesystem for batch and other operations Issue created by edgarrmondragon ### Singer SDK Version 0.26.0 ### Is this a regression? • Yes ### Python Version NA ### Bug scope Other ### Operating System NA ### Description ## Why Development of
PyFilesystem
has arguably become stagnant. I'm concerned that the Python ecosystem is moving and PyFilesytem and plugins will be left behind. A recent example of this was the release of urllib3 2.0, which is incompatible with botocore. The offender is currently the botocore library, but it's actively developed so it's more likely that the maintainers will add support for urllib3 2.0+, but that may require them to bump the major version of their package, which
fs-s3fs
will not be compatible with, and it's not clear if the fs-s3fs is still being maintained (last activity on 2019). ## Alternatives ###
fsspec
#### Advantages • Automatically read/write from S3 when a
s3://...
URL is used, similar to the current situation with PyFilesystem. • Good community support and maintenance. Used by Dask. #### Disadvantages • See breaking changes ###
smart-open
#### Advantages • Simple to use #### Disadvantages • Requires explicitly defining the transport config • See breaking changes ## Breaking changes Changing the library we use to support different file storage systems will break any taps that currently have rely on having
fs-s3fs
as a dependency. cc @kgpayne ### Code No response meltano/sdk