Read GCS, ABS and local paths with the same interface, clone of tensorflow.io.gfile
This is a library that provides a Python-like interface for reading local and remote files (only from blob storage), with an API similar to open()
as well as some of the os.path
and shutil
functions. blobfile
supports local paths, Google Cloud Storage paths (gs://<bucket>
), and Azure Blob Storage paths (az://<account>/<container>
or https://<account>.blob.core.windows.net/<container>/
).
The main function is BlobFile
, which lets you open local and remote files that act more or less like local ones. There are also a few additional functions such as basename
, dirname
, and join
, which mostly do the same thing as their os.path
namesakes, only they also support GCS paths and ABS paths.
This library is inspired by TensorFlow's gfile
but does not have exactly the same interface.
pip install blobfile
# write a file, then read it back
import blobfile as bf
with bf.BlobFile("gs://my-bucket-name/cats", "wb") as f:
f.write(b"meow!")
print("exists:", bf.exists("gs://my-bucket-name/cats"))
with bf.BlobFile("gs://my-bucket-name/cats", "rb") as f:
print("contents:", f.read())
There are also some examples processing many blobs in parallel.
Here are the functions in blobfile
:
BlobFile
- like open()
but works with remote paths too, data can be streamed to/from the remote file. It accepts the following arguments:
streaming
:
streaming
is True
when mode
is in "r", "rb"
and False
when mode
is in "w", "wb", "a", "ab"
.streaming=True
:
flush()
will not cause an early write.streaming=False
:
close()
or during destruction.close()
.buffer_size
: number of bytes to buffer, this can potentially make reading more efficient.cache_dir
: a directory in which to cache files for reading, only valid if streaming=False
and mode
is in "r", "rb"
. You are reponsible for cleaning up the cache directory.file_size
: size of the file being opened, can be specified directly to avoid checking the file size when opening the file. While this will avoid a network request, it also means that you may get an error when first reading a file that does not exist rather than when opening it. Only valid for modes "r" and "rb". This valid will be ignored for local files.Some are inspired by existing os.path
and shutil
functions:
copy
- copy a file from one path to another, this will do a remote copy between two remote paths on the same blob storage serviceexists
- returns True
if the file or directory existsglob
/scanglob
- return files matching a glob-style pattern as a generator. Globs can have surprising performance characteristics when used with blob storage. Character ranges are not supported in patterns.isdir
- returns True
if the path is a directorylistdir
/scandir
- list contents of a directory as a generatormakedirs
- ensure that a directory and all parent directories existremove
- remove a filermdir
- remove an empty directoryrmtree
- remove a directory treestat
- get the size and modification time of a filewalk
- walk a directory tree with a generator that yields (dirpath, dirnames, filenames)
tuplesbasename
- get the final component of a pathdirname
- get the path except for the final componentjoin
- join 2 or more paths together, inserting directory separators between each componentThere are a few bonus functions:
get_url
- returns a url for a path (usable by an HTTP client without any authentication) along with the expiration for that url (or None)md5
- get the md5 hash for a path, for GCS this is often fast, but for other backends this may be slow. On Azure, if the md5 of a file is calculated and is missing from the file, the file will be updated with the calculated md5.set_mtime
- set the modified timestamp for a fileconfigure
- set global configuration options for blobfile
log_callback=default_log_fn
: a log callback function log(msg: string)
to use instead of printing to stdout. If you use parallel=True
, you probably want to use a log callback function that is pickleable.connection_pool_max_size=32
: the max size for each per-host connection poolmax_connection_pool_count=10
: the maximum count of per-host connection poolsazure_write_chunk_size=8 * 2 ** 20
: the size of blocks to write to Azure Storage blobs in bytes, can be set to a maximum of 100MB. This determines both the unit of request retries as well as the maximum file size, which is 50,000 * azure_write_chunk_size
.google_write_chunk_size=8 * 2 ** 20
: the size of blocks to write to Google Cloud Storage blobs in bytes, this only determines the unit of request retries.retry_log_threshold=0
: set a retry count threshold above which to log failures to the log callback functionretry_common_log_threshold=2
: set a retry count threshold above which to log very common failures to the log callback functionconnect_timeout=10
: the maximum amount of time (in seconds) to wait for a connection attempt to a server to succeed, set to None to wait foreverread_timeout=30
: the maximum amount of time (in seconds) to wait between consecutive read operations for a response from the server, set to None to wait foreveroutput_az_paths=True
: output az://
paths instead of using the https://
for azureuse_azure_storage_account_key_fallback=False
: fallback to storage account keys for azure containers, having this enabled requires listing your subscriptions and may run into 429 errors if you hit the low azure quotas for subscription listingget_http_pool=None
: a function that returns a urllib3.PoolManager
to be used for requestsuse_streaming_read=False
: if set to True
, use a single read per file instead of reading a chunk at a time (not recommended for azure)default_buffer_size=io.DEFAULT_BUFFER_SIZE
: the default buffer size to use for reading files (and writing local files)save_access_token_to_disk=True
: if set to True
to save access tokens to disk so that other processes can read the access tokens to avoid the small amount of time it usually takes to get a token (if the token is still valid).multiprocessing_start_method="spawn"
: the start method to use when creating processes for parallel workcreate_context
- (same arguments as configure
), creates a new instance of blobfile
with a custom configuration instead of modifying the global configurationThe following methods will be tried in order:
GOOGLE_APPLICATION_CREDENTIALS
for a path to service account credentials in JSON format.gcloud auth application-default login
.The following methods will be tried in order:
AZURE_USE_IDENTITY=1
is set, use DefaultAzureCredential from the azure-identity
package to acquire tokens. Note: your application must install the azure-identity
package; blobfile
does not specify it as a required dependency.AZURE_STORAGE_KEY
for an azure storage account key (these are per-storage account shared keys described in https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage)AZURE_APPLICATION_CREDENTIALS
which should point to JSON credentials for a service principal output by the command az ad sp create-for-rbac --name <name>
AZURE_CLIENT_ID
, AZURE_CLIENT_SECRET
, AZURE_TENANT_ID
corresponding to a service principal described in the previous step but without the JSON file.AZURE_STORAGE_CONNECTION_STRING
for an Azure Storage connection stringaz
command line tool if they can be found.If access using credentials fails, anonymous access will be tried. blobfile
supports public access for containers marked as public, but not individual blobs.
For Google Cloud Storage and Azure Blobs, directories don't really exist. These storage systems store the files in a single flat list. The "/" separators are just part of the filenames and there is no need to call the equivalent of os.mkdir
on one of these systems.
To make local behavior consistent with the remote storage systems, missing local directories will be created automatically when opening a file in write mode.
These are just normal paths for the current machine, e.g. /root/hello.txt
GCS paths have the format gs://<bucket>/<blob>
, you cannot perform any operations on gs://
itself.
Azure Blobs URLs have the format az://<account>/<container>
or https://<account>.blob.core.windows.net/<container>/<blob>
. The highest you can go up the hierarchy is az://<account>/<container>/
, blobfile
cannot perform any operations on az://<account>/
. The https://
url is the output format by default, but the az://
urls are accepted as inputs and you can set output_az_paths=True
to get az://
urls as output.
Error
- base class for library-specific exceptionsRequestFailure(Error)
- a request has failed permanently, the status code can be found in the property response_status:int
and an error code, if available, is in error:Optional[str]
.RestartableStreamingWriteFailure(RequestFailure)
- a streaming write has failed permanently, which requires restarting from the beginning of the stream.ConcurrentWriteFailure(RequestFailure)
- a write failed because another process was writing to the same file at the same time.VersionMismatch(RequestFailure)
- a write failed because the remote file did not match the version specified by the user.FileNotFoundError
, FileExistsError
, IsADirectoryError
, NotADirectoryError
, OSError
, ValueError
, io.UnsupportedOperation
blobfile
will keep retrying transient errors until they succeed or a permanent error is encountered (which will raise an exception). In order to make diagnosing stalls easier, blobfile
will log when retrying requests.
To route those log lines, use configure(log_callback=<fn>)
to set a callback function which will be called whenever a log line should be printed. The default callback prints to stdout with the prefix blobfile:
.
logging
moduleIf you use the python logging
module, you can have blobfile
log there:
bf.configure(log_callback=logging.getLogger("blobfile").warning)
While blobfile
does not use the python logging
module by default, it does use other libraries which use that module. So if you configure the python logging
module, you may need to change the settings to adjust logging behavior:
urllib3
: logging.getLogger("urllib3").setLevel(logging.ERROR)
filelock
: logging.getLogger("filelock").setLevel(logging.ERROR)
Also, as a tip, make sure to use a format that tells you the name of the logger:
logging.basicConfig(format="%(asctime)s [%(name)s] %(levelname)s: %(message)s", level=logging.WARNING)
This will let you see which package is producing log messages.
The library should be thread safe and fork safe with the following exceptions:
BlobFile
instance is not thread safe (only 1 thread should own a BlobFile
instance at a time)bf.configure()
are not thread-safe and should ideally happen before performing any operationsGoogle Cloud Storage supports multiple writers for the same blob and the last one to finish should win. However, in the event of a large number of simultaneous writers, the service will return 429 or 503 errors and most writers will stall. In this case, write to different blobs instead.
Azure Blobs doesn't support multiple writers for the same blob. With the way BlobFile
is currently configured, the last writer to start writing will win. Other writers will get a ConcurrentWriteFailure
. In addition, all writers could fail if the file size is large and there are enough concurrent writers. In this case, you can write to a temporary blob (with a random filename), copy it to the final location, and then delete the original. The copy will be within a container so it should be fast.
See CHANGES
Create testing buckets for each cloud provider with appropriate credentials.
To make a new release:
rm -rf build dist
python -m build .
twine upload dist/*
This will auto-format the code, check the types, and then run the tests:
python testing/run.py
Run a single test:
python testing/run.py -v -s -k test_windowed_file
Modify testing/run.py
if you only want to do some of these things. The tests are rather slow, ~7 minutes to run (even though large file tests are disabled) and require accounts with every cloud provider.