Compute distance between the two texts.
TextDistance -- python library for comparing distance between two or more sequences by many algorithms.
Features:
Algorithm | Class | Functions |
---|---|---|
Hamming | Hamming |
hamming |
MLIPNS | Mlipns |
mlipns |
Levenshtein | Levenshtein |
levenshtein |
Damerau-Levenshtein | DamerauLevenshtein |
damerau_levenshtein |
Jaro-Winkler | JaroWinkler |
jaro_winkler , jaro |
Strcmp95 | StrCmp95 |
strcmp95 |
Needleman-Wunsch | NeedlemanWunsch |
needleman_wunsch |
Gotoh | Gotoh |
gotoh |
Smith-Waterman | SmithWaterman |
smith_waterman |
Algorithm | Class | Functions |
---|---|---|
Jaccard index | Jaccard |
jaccard |
Sørensen–Dice coefficient | Sorensen |
sorensen , sorensen_dice , dice |
Tversky index | Tversky |
tversky |
Overlap coefficient | Overlap |
overlap |
Tanimoto distance | Tanimoto |
tanimoto |
Cosine similarity | Cosine |
cosine |
Monge-Elkan | MongeElkan |
monge_elkan |
Bag distance | Bag |
bag |
Algorithm | Class | Functions |
---|---|---|
longest common subsequence similarity | LCSSeq |
lcsseq |
longest common substring similarity | LCSStr |
lcsstr |
Ratcliff-Obershelp similarity | RatcliffObershelp |
ratcliff_obershelp |
Normalized compression distance with different compression algorithms.
Classic compression algorithms:
Algorithm | Class | Function |
---|---|---|
Arithmetic coding | ArithNCD |
arith_ncd |
RLE | RLENCD |
rle_ncd |
BWT RLE | BWTRLENCD |
bwtrle_ncd |
Normal compression algorithms:
Algorithm | Class | Function |
---|---|---|
Square Root | SqrtNCD |
sqrt_ncd |
Entropy | EntropyNCD |
entropy_ncd |
Work in progress algorithms that compare two strings as array of bits:
Algorithm | Class | Function |
---|---|---|
BZ2 | BZ2NCD |
bz2_ncd |
LZMA | LZMANCD |
lzma_ncd |
ZLib | ZLIBNCD |
zlib_ncd |
See blog post for more details about NCD.
Algorithm | Class | Functions |
---|---|---|
MRA | MRA |
mra |
Editex | Editex |
editex |
Algorithm | Class | Functions |
---|---|---|
Prefix similarity | Prefix |
prefix |
Postfix similarity | Postfix |
postfix |
Length distance | Length |
length |
Identity similarity | Identity |
identity |
Matrix similarity | Matrix |
matrix |
Only pure python implementation:
pip install textdistance
With extra libraries for maximum speed:
pip install "textdistance[extras]"
With all libraries (required for benchmarking and testing):
pip install "textdistance[benchmark]"
With algorithm specific extras:
pip install "textdistance[Hamming]"
Algorithms with available extras: DamerauLevenshtein
, Hamming
, Jaro
, JaroWinkler
, Levenshtein
.
Via pip:
pip install -e git+https://github.com/life4/textdistance.git#egg=textdistance
Or clone repo and install with some extras:
git clone https://github.com/life4/textdistance.git
pip install -e ".[benchmark]"
All algorithms have 2 interfaces:
All algorithms have some common methods:
.distance(*sequences)
-- calculate distance between sequences..similarity(*sequences)
-- calculate similarity for sequences..maximum(*sequences)
-- maximum possible value for distance and similarity. For any sequence: distance + similarity == maximum
..normalized_distance(*sequences)
-- normalized distance between sequences. The return value is a float between 0 and 1, where 0 means equal, and 1 totally different..normalized_similarity(*sequences)
-- normalized similarity for sequences. The return value is a float between 0 and 1, where 0 means totally different, and 1 equal.Most common init arguments:
qval
-- q-value for split sequences into q-grams. Possible values:
as_set
-- for token-based algorithms:
t
and ttt
is equal.t
and ttt
is different.For example, Hamming distance:
import textdistance
textdistance.hamming('test', 'text')
# 1
textdistance.hamming.distance('test', 'text')
# 1
textdistance.hamming.similarity('test', 'text')
# 3
textdistance.hamming.normalized_distance('test', 'text')
# 0.25
textdistance.hamming.normalized_similarity('test', 'text')
# 0.75
textdistance.Hamming(qval=2).distance('test', 'text')
# 2
Any other algorithms have same interface.
A few articles with examples how to use textdistance in the real world:
For main algorithms textdistance try to call known external libraries (fastest first) if available (installed in your system) and possible (this implementation can compare this type of sequences). Install textdistance with extras for this feature.
You can disable this by passing external=False
argument on init:
import textdistance
hamming = textdistance.Hamming(external=False)
hamming('text', 'testit')
# 3
Supported libraries:
Algorithms:
Without extras installation:
algorithm | library | time |
---|---|---|
DamerauLevenshtein | rapidfuzz | 0.00312 |
DamerauLevenshtein | jellyfish | 0.00591 |
DamerauLevenshtein | pyxdameraulevenshtein | 0.03335 |
DamerauLevenshtein | textdistance | 0.83524 |
Hamming | Levenshtein | 0.00038 |
Hamming | rapidfuzz | 0.00044 |
Hamming | jellyfish | 0.00091 |
Hamming | distance | 0.00812 |
Hamming | textdistance | 0.03531 |
Jaro | rapidfuzz | 0.00092 |
Jaro | jellyfish | 0.00191 |
Jaro | textdistance | 0.07365 |
JaroWinkler | rapidfuzz | 0.00094 |
JaroWinkler | jellyfish | 0.00195 |
JaroWinkler | textdistance | 0.07501 |
Levenshtein | rapidfuzz | 0.00099 |
Levenshtein | Levenshtein | 0.00122 |
Levenshtein | jellyfish | 0.00254 |
Levenshtein | pylev | 0.15688 |
Levenshtein | distance | 0.28669 |
Levenshtein | textdistance | 0.53902 |
Total: 24 libs.
Yeah, so slow. Use TextDistance on production only with extras.
Textdistance use benchmark's results for algorithm's optimization and try to call fastest external lib first (if possible).
You can run benchmark manually on your system:
pip install textdistance[benchmark]
python3 -m textdistance.benchmark
TextDistance show benchmarks results table for your system and save libraries priorities into libraries.json
file in TextDistance's folder. This file will be used by textdistance for calling fastest algorithm implementation. Default libraries.json already included in package.
All you need is task. See Taskfile.yml for the list of available commands. For example, to run tests including third-party libraries usage, execute task pytest-external:run
.
PRs are welcome!
textdistance
. More users, more contributions, more amazing features.Thank you :heart: