A collection of self-contained fsspec-based filesystems
|PyPI| |Status| |Python Version| |License|
|Tests| |Codecov| |pre-commit| |Black|
.. |PyPI| image:: https://img.shields.io/pypi/v/morefs.svg :target: https://pypi.org/project/morefs/ :alt: PyPI .. |Status| image:: https://img.shields.io/pypi/status/morefs.svg :target: https://pypi.org/project/morefs/ :alt: Status .. |Python Version| image:: https://img.shields.io/pypi/pyversions/morefs :target: https://pypi.org/project/morefs :alt: Python Version .. |License| image:: https://img.shields.io/pypi/l/morefs :target: https://opensource.org/licenses/Apache-2.0 :alt: License .. |Tests| image:: https://github.com/iterative/morefs/workflows/Tests/badge.svg :target: https://github.com/iterative/morefs/actions?workflow=Tests :alt: Tests .. |Codecov| image:: https://codecov.io/gh/iterative/morefs/branch/main/graph/badge.svg :target: https://app.codecov.io/gh/iterative/morefs :alt: Codecov .. |pre-commit| image:: https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white :target: https://github.com/pre-commit/pre-commit :alt: pre-commit .. |Black| image:: https://img.shields.io/badge/code%20style-black-000000.svg :target: https://github.com/psf/black :alt: Black
morefs provides standalone fsspec-based filesystems like:
AsyncLocalFileSystem
that provides async implementation of LocalFileSystem
.DictFileSystem
built on nested dictionaries and MemFS
built on tries, and are much faster than fsspec's MemoryFileSystem
.OverlayFileSystem
that allows to overlay multiple fsspec-based filesystems.You can install morefs via pip_ from PyPI_:
.. code:: console
$ pip install morefs
You might need to install with extras for some filesystems:
.. code:: console
$ pip install morefs[asynclocal] # for installing aiofile dependency for AsyncLocalFileSystem $ pip install morefs[memfs] # for installing pygtrie dependency for MemFS
AsyncLocalFileSystem
Extended version of ``LocalFileSystem`` that also provides async methods.
.. code:: python
import asyncio
from morefs.asyn_local import AsyncLocalFileSystem
async def main():
fs = AsyncLocalFileSystem(auto_mkdir=False)
async with fs.open_async("foo", mode="w") as f:
await f.write("foobar")
content = await fs._cat("foo")
print(content)
print(fs.cat("foo")) # you can still use sync methods
asyncio.run(main())
DictFS
~~~~~~
DictFS is a nested dictionary-based, in-memory filesystem
and acts more like a real LocalFileSystem.
.. code:: python
from morefs.dict import filesystem
fs = DictFS()
MemFS
~~~~~
MemFS is a trie-based in-memory filesystem, and acts like a bucket storage.
.. code:: python
from morefs.memory import MemFS
fs = MemFS()
OverlayFileSystem
~~~~~~~~~~~~~~~~~
.. code:: python
from morefs.overlay import OverlayFileSystem
# use localfilesystem for write, overlay all filesystems for read
fs = OverlayFileSystem(file={"auto_mkdir": True}, s3={"anon": True})
# or you can pass filesystem instances directly
# as variable positional arguments or with keyword argument `filesystems=[]`
fs = OverlayFileSystem(LocalFileSystem(), s3={"anon": True})
Contributing
------------
Contributions are very welcome.
To learn more, see the `Contributor Guide`_.
License
-------
Distributed under the terms of the `Apache 2.0 license`_,
*morefs* is free and open source software.
Issues
------
If you encounter any problems,
please `file an issue`_ along with a detailed description.
.. _Apache 2.0 license: https://opensource.org/licenses/Apache-2.0
.. _PyPI: https://pypi.org/
.. _file an issue: https://github.com/iterative/morefs/issues
.. _pip: https://pip.pypa.io/
.. github-only
.. _Contributor Guide: CONTRIBUTING.rst