BentoML: Build Production-Grade AI Applications
BentoML is a framework for building reliable, scalable, and cost-efficient AI applications. It comes with everything you need for model serving, application packaging, and production deployment.
👉 Join our Slack community!pip install bentoml
Save or import models in BentoML local model store:
import bentoml
import transformers
pipe = transformers.pipeline("text-classification")
bentoml.transformers.save_model(
"text-classification-pipe",
pipe,
signatures={
"__call__": {"batchable": True} # Enable dynamic batching for model
}
)
View all models saved locally:
$ bentoml models list
Tag Module Size Creation Time
text-classification-pipe:kn6mr3aubcuf… bentoml.transformers 256.35 MiB 2023-05-17 14:36:25
Define how your model runs in a service.py
file:
import bentoml
model_runner = bentoml.models.get("text-classification-pipe").to_runner()
svc = bentoml.Service("text-classification-service", runners=[model_runner])
@svc.api(input=bentoml.io.Text(), output=bentoml.io.JSON())
async def classify(text: str) -> str:
results = await model_runner.async_run([text])
return results[0]
Now, run the API service locally:
bentoml serve service.py:svc
Sent a prediction request:
$ curl -X POST -H "Content-Type: text/plain" --data "BentoML is awesome" http://localhost:3000/classify
{"label":"POSITIVE","score":0.9129443168640137}%
Define how a Bento can
be built for deployment, with bentofile.yaml
:
service: 'service.py:svc'
name: text-classification-svc
include:
- 'service.py'
python:
packages:
- torch>=2.0
- transformers
Build a Bento and generate a docker image:
$ bentoml build
...
Successfully built Bento(tag="text-classification-svc:mc322vaubkuapuqj").
$ bentoml containerize text-classification-svc
Building OCI-compliant image for text-classification-svc:mc322vaubkuapuqj with docker
...
Successfully built Bento container for "text-classification-svc" with tag(s) "text-classification-svc:mc322vaubkuapuqj"
$ docker run -p 3000:3000 text-classification-svc:mc322vaubkuapuqj
For a more detailed user guide, check out the BentoML Tutorial.
BentoML supports billions of model runs per day and is used by thousands of organizations around the globe.
Join our Community Slack 💬, where thousands of AI application developers contribute to the project and help each other.
To report a bug or suggest a feature request, use GitHub Issues.
There are many ways to contribute to the project:
#bentoml-contributors
channel here.Thanks to all of our amazing contributors!
BentoML collects usage data that helps our team to improve the product. Only
BentoML's internal API calls are being reported. We strip out as much
potentially sensitive information as possible, and we will never collect user
code, model data, model names, or stack traces. Here's the
code for usage
tracking. You can opt-out of usage tracking by the --do-not-track
CLI option:
bentoml [command] --do-not-track
Or by setting environment variable BENTOML_DO_NOT_TRACK=True
:
export BENTOML_DO_NOT_TRACK=True
If you use BentoML in your research, please cite using the following citation:
@software{Yang_BentoML_The_framework,
author = {Yang, Chaoyu and Sheng, Sean and Pham, Aaron and Zhao, Shenyang and Lee, Sauyon and Jiang, Bo and Dong, Fog and Guan, Xipeng and Ming, Frost},
license = {Apache-2.0},
title = {{BentoML: The framework for building reliable, scalable and cost-efficient AI application}},
url = {https://github.com/bentoml/bentoml}
}