baby636/tokenizers

master

tokenizers

Grade	Name	Complexity	Churn	Issues
D	bindings\node\lib\implementations\encoding.test.ts	14	1	22
A	bindings\python\py_src\tokenizers\implementations\byte_level_bpe.py	13	3	-
D	bindings\node\lib\bindings\pre-tokenizers.test.ts	13	2	22
A	bindings\python\scripts\sentencepiece_extractor.py	12	1	-
D	bindings\node\lib\bindings\decoders.test.ts	12	2	20
A	bindings\node\build.js	12	1	1
A	bindings\python\tests\bindings\test_trainers.py	11	4	-
C	bindings\node\lib\implementations\tokenizers\base.tokenizer.test.ts	10	1	16
A	bindings\python\tests\bindings\test_encoding.py	10	1	-
A	bindings\python\tests\documentation\test_tutorial_train_from_iterators.py	10	2	-
C	bindings\node\lib\bindings\normalizers.test.ts	9	1	16
A	bindings\node\lib\implementations\tokenizers\bert-wordpiece.tokenizer.ts	9	1	-
A	bindings\node\lib\implementations\tokenizers\bpe.tokenizer.ts	7	1	-
A	docs\source\_ext\toctree_tags.py	6	1	-
A	bindings\python\conftest.py	6	1	-
A	bindings\node\lib\implementations\tokenizers\byte-level-bpe.tokenizer.ts	6	1	-
A	bindings\node\lib\implementations\tokenizers\sentence-piece-bpe.tokenizer.ts	6	1	1
B	bindings\node\lib\implementations\tokenizers\bert-wordpiece.tokenizer.test.ts	6	1	9
A	bindings\python\tests\implementations\test_char_bpe.py	5	2	-
A	bindings\python\tests\implementations\test_byte_level_bpe.py	5	2	-