Package: morphemepiece 1.2.3

Jonathan Bratt

morphemepiece: Morpheme Tokenization

Tokenize text into morphemes. The morphemepiece algorithm uses a lookup table to determine the morpheme breakdown of words, and falls back on a modified wordpiece tokenization algorithm for words not found in the lookup table.

Authors:Jonathan Bratt [aut, cre], Jon Harmon [aut], Bedford Freeman & Worth Pub Grp LLC DBA Macmillan Learning [cph]

morphemepiece_1.2.3.tar.gz
morphemepiece_1.2.3.zip(r-4.5)morphemepiece_1.2.3.zip(r-4.4)morphemepiece_1.2.3.zip(r-4.3)
morphemepiece_1.2.3.tgz(r-4.4-any)morphemepiece_1.2.3.tgz(r-4.3-any)
morphemepiece_1.2.3.tar.gz(r-4.5-noble)morphemepiece_1.2.3.tar.gz(r-4.4-noble)
morphemepiece_1.2.3.tgz(r-4.4-emscripten)morphemepiece_1.2.3.tgz(r-4.3-emscripten)
morphemepiece.pdf |morphemepiece.html
morphemepiece/json (API)
NEWS

# Install 'morphemepiece' in R:
install.packages('morphemepiece', repos = c('https://macmillancontentscience.r-universe.dev', 'https://cloud.r-project.org'))

Peer review:

Bug tracker:https://github.com/macmillancontentscience/morphemepiece/issues

On CRAN:

5.04 score 11 stars 8 scripts 223 downloads 10 exports 38 dependencies

Last updated 3 years agofrom:bc071b1a03. Checks:OK: 3 NOTE: 4. Indexed: yes.

TargetResultDate
Doc / VignettesOKOct 29 2024
R-4.5-winNOTEOct 29 2024
R-4.5-linuxNOTEOct 29 2024
R-4.4-winNOTEOct 29 2024
R-4.4-macNOTEOct 29 2024
R-4.3-winOKOct 29 2024
R-4.3-macOKOct 29 2024

Exports:load_lookupload_or_retrieve_lookupload_or_retrieve_vocabload_vocabmorphemepiece_cache_dirmorphemepiece_lookupmorphemepiece_tokenizemorphemepiece_vocabprepare_vocabset_morphemepiece_cache_dir

Dependencies:bitbit64cachemclicliprcpp11crayondigestdlrfansifastmapfastmatchfsgluehmslifecyclemagrittrmemoisemorphemepiece.datapiecemakerpillarpkgconfigprettyunitsprogresspurrrR6rappdirsreadrrlangstringistringrtibbletidyselecttzdbutf8vctrsvroomwithr

Generating a Vocabulary and Lookup

Rendered fromgenerating_vocab.Rmdusingknitr::rmarkdownon Oct 29 2024.

Last update: 2021-10-26
Started: 2021-07-29

Testing the fall-through algorithm

Rendered fromalgorithm_test.Rmdusingknitr::rmarkdownon Oct 29 2024.

Last update: 2021-09-06
Started: 2021-07-29