tokenizers BpeBuilder Buffer Overflow via merge token
Default status is the baseline for the product, each version can override it (e.g. patched versions marked unaffected).
| Vendor | Product | Default status | Versions | ||||||
|---|---|---|---|---|---|---|---|---|---|
| huggingface | tokenizers | unaffected |
|
No data.
No data.
| Vendor | Product | Versions | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Huggingface | Tokenizers |
|
No CVSS v3.0
No CVSS v2
This CVE is not in the KEV list.
Exploit prediction score: < 1% (0.467% probability within 30 days)
| Type | Values Removed | Values Added |
|---|---|---|
| Metrics |
ssvc
|
| Type | Values Removed | Values Added |
|---|---|---|
| First Time appeared |
Huggingface
Huggingface tokenizers |
|
| Vendors & Products |
Huggingface
Huggingface tokenizers |
| Type | Values Removed | Values Added |
|---|---|---|
| Description | tokenizers (Hugging Face) is affected by an out-of-bounds buffer access in BpeBuilder::build (tokenizers/src/models/bpe/model.rs). When loading a tokenizer.json via Tokenizer::from_file/from_str, the builder sizes a scratch buffer to the longest vocabulary key, then writes each concatenated merge rule into it. A merge whose concatenated token exceeds the longest vocabulary key overruns the buffer, which Rust turns into a panic that aborts the process in Rust and FFI embeddings. This occurs at load time with no encoding required, so an attacker who supplies a crafted tokenizer.json can cause a denial of service. A secondary defect at the same location can cause a usize underflow (panic in debug, potential memory corruption in release) when continuing_subword_prefix is set and a merge token is shorter than the prefix. Observed in version 0.23.1. | |
| Title | tokenizers BpeBuilder Buffer Overflow via merge token | |
| Weaknesses | CWE-787 | |
| References |
| |
| Metrics |
cvssV3_1
|