Commits

Branch selector

 master

User selector

All users

Datepicker

All time

Commit History

Commits on Jun 14, 2024

• Merge pull request #42 from yarcat/master

  sugarme

  committedJun 14, 2024
  2b7834e

• Merge pull request #41 from cwarden/fix-missing-pairEncoding

  sugarme

  committedJun 14, 2024
  dc1def5

Commits on Jun 13, 2024

• pretrained/tokenizer: fix FromFile, add FromReader

  yarcat

  committedJun 13, 2024
  21343b4

Commits on Jun 12, 2024

• Fix TruncateEncodings for No Pair Encoding

  cwarden

  committedJun 12, 2024
  247ae37

Commits on Feb 24, 2024

• Merge pull request #40 from jackielii/safer-input

  sugarme

  committedFeb 24, 2024
  7599e43

Commits on Feb 23, 2024

• use private() for better type safety

  jackielii

  committedFeb 23, 2024
  83ebeea

Commits on Aug 29, 2023

• Merge pull request #33 from kasra73/master

  sugarme

  committedAug 29, 2023
  448e79b

• Fix initialization issue in for loop to retain first token in output string

  kasra73

  committedAug 29, 2023
  4bbf6d7

Commits on Aug 23, 2023

• Merge branch 'master' of ssh://github.com/sugarme/tokenizer

  sugarme

  committedAug 23, 2023
  dcc38d3

• fixed issue #31 assigning to a nil map

  sugarme

  committedAug 23, 2023
  c5f182f

Commits on Jul 5, 2023

• Merge pull request #30 from yujong-lee/fix-pad

  sugarme

  committedJul 5, 2023
  7a17b4e

• Fix padd direction

  yujonglee

  committedJul 5, 2023
  c9c6bfa

Commits on Jul 4, 2023

• Merge pull request #29 from yujong-lee/sbert

  sugarme

  committedJul 4, 2023
  590c8f1

• Add failing test case

  yujonglee

  committedJul 4, 2023
  a261e6a

• Fix padding strategy params

  yujonglee

  committedJul 4, 2023
  793eb36

• Fix typo

  yujonglee

  committedJul 4, 2023
  1310f79

Commits on Jun 25, 2023

• bump new version

  sugarme

  committedJun 25, 2023
  1f25ffe

• Merge pull request #28 from sugarme/decoder

  sugarme

  committedJun 25, 2023
  0e1bd3b

• fixed pretrained/createReplaceDecoder

  sugarme

  committedJun 25, 2023
  584d1ec

Commits on Jun 24, 2023

• bump new version

  sugarme

  committedJun 24, 2023
  9029df5

• Merge pull request #27 from sugarme/pretrained

  sugarme

  committedJun 24, 2023
  e78f652

• fixed bugs at pretrained.FromFile

  sugarme

  committedJun 24, 2023
  6225518

Commits on Jun 21, 2023

• tag new version

  sugarme

  committedJun 21, 2023
  0ab2e3b

• Merge pull request #25 from sugarme/decoder

  sugarme

  committedJun 21, 2023
  782839f

• fixed normalizer.NFC()

  sugarme

  committedJun 21, 2023
  e205bac

• added completed list of pretokenizers

  sugarme

  committedJun 21, 2023
  8f58d07

Commits on Jun 20, 2023

• WIP: pretokenizer

  sugarme

  committedJun 20, 2023
  1585cf0

Commits on Jun 19, 2023

• WIP: pretrained/processor

  sugarme

  committedJun 19, 2023
  deea443

Commits on Jun 14, 2023

• WIP: TemplateProcessing

  sugarme

  committedJun 14, 2023
  c3f5995

Commits on Jun 10, 2023

• WIP: processor.Sequence

  sugarme

  committedJun 10, 2023
  efee0cb

Commits on Jun 9, 2023

• WIP: Create Tokenizer FromFile continue...

  sugarme

  committedJun 9, 2023
  269720b

• WIP: added CreateNormalizer

  sugarme

  committedJun 9, 2023
  6382a37

• WIP: pretrained.FromFile API

  sugarme

  committedJun 9, 2023
  5ee01cc

• WIP: refactor constructing Tokenizer from json file

  sugarme

  committedJun 9, 2023
  cfab7d1

Commits on Jun 8, 2023

• WIP: New tokenizer from tokenizer.json

  sugarme

  committedJun 8, 2023
  3f282ce

Pagination

PreviousNext