baby636/tokenizers

master

tokenizers

Grade	Name	Complexity	Churn	Issues
A	docs\source\_ext\entities.py	64	1	3
F	bindings\node\lib\bindings\tokenizer.test.ts	54	1	131
A	bindings\python\scripts\spm_parity_check.py	49	2	4
A	bindings\python\stub.py	46	2	1
F	bindings\node\lib\bindings\utils.test.ts	44	1	63
B	bindings\python\py_src\tokenizers\tools\visualizer.py	43	1	4
F	bindings\node\lib\bindings\raw-encoding.test.ts	43	1	105
A	bindings\python\py_src\tokenizers\implementations\base_tokenizer.py	40	4	1
F	bindings\node\lib\bindings\models.test.ts	40	1	38
A	bindings\node\lib\implementations\encoding.ts	38	1	1
C	bindings\python\tests\bindings\test_tokenizer.py	26	1	6
A	bindings\node\lib\implementations\tokenizers\base.tokenizer.ts	26	1	2
F	bindings\python\tests\bindings\test_pre_tokenizers.py	23	5	14
A	bindings\python\tests\utils.py	21	1	-
F	bindings\node\lib\bindings\post-processors.test.ts	20	1	29
B	bindings\python\tests\bindings\test_normalizers.py	19	3	4
A	docs\source\_ext\rust_doc.py	17	2	-
B	bindings\python\tests\bindings\test_models.py	16	4	4
A	bindings\python\py_src\tokenizers\implementations\char_level_bpe.py	14	3	-
D	bindings\node\lib\implementations\encoding.test.ts	14	1	22