• Features
  • Pricing
  • Contact
    Login with:
  • Features
  • Pricing
  • Contact
Login with:
master
tokenizers
Repository badge with B rating
View on GitHub

  • 524 issues
  • 0 pull requests
  • 100 files
  • 1 active branch

CodeFactor Rating B

  • {{format.Name}}
Grade
Name
Complexity
Churn
Issues
A
docs\source\_ext\entities.py
64 1 3
F
bindings\node\lib\bindings\tokenizer.test.ts
54 1 131
A
bindings\python\scripts\spm_parity_check.py
49 2 4
A
bindings\python\stub.py
46 2 1
F
bindings\node\lib\bindings\utils.test.ts
44 1 63
B
bindings\python\py_src\tokenizers\tools\visualizer.py
43 1 4
F
bindings\node\lib\bindings\raw-encoding.test.ts
43 1 105
A
bindings\python\py_src\tokenizers\implementations\base_tokenizer.py
40 4 1
F
bindings\node\lib\bindings\models.test.ts
40 1 38
A
bindings\node\lib\implementations\encoding.ts
38 1 1
C
bindings\python\tests\bindings\test_tokenizer.py
26 1 6
A
bindings\node\lib\implementations\tokenizers\base.tokenizer.ts
26 1 2
F
bindings\python\tests\bindings\test_pre_tokenizers.py
23 5 14
A
bindings\python\tests\utils.py
21 1 -
F
bindings\node\lib\bindings\post-processors.test.ts
20 1 29
B
bindings\python\tests\bindings\test_normalizers.py
19 3 4
A
docs\source\_ext\rust_doc.py
17 2 -
B
bindings\python\tests\bindings\test_models.py
16 4 4
A
bindings\python\py_src\tokenizers\implementations\char_level_bpe.py
14 3 -
D
bindings\node\lib\implementations\encoding.test.ts
14 1 22
  • Next Page
  • Security
  • Terms
  • Privacy
  • Contact
© 2026 CodeFactor

We use cookies in order to offer you the most relevant information. See our Privacy Policy.