Package: morphemepiece 1.2.3

Jonathan Bratt

morphemepiece: Morpheme Tokenization

Tokenize text into morphemes. The morphemepiece algorithm uses a lookup table to determine the morpheme breakdown of words, and falls back on a modified wordpiece tokenization algorithm for words not found in the lookup table.

Authors:Jonathan Bratt [aut, cre], Jon Harmon [aut], Bedford Freeman & Worth Pub Grp LLC DBA Macmillan Learning [cph]

morphemepiece_1.2.3.tar.gz
morphemepiece_1.2.3.zip(r-4.5)morphemepiece_1.2.3.zip(r-4.4)morphemepiece_1.2.3.zip(r-4.3)
morphemepiece_1.2.3.tgz(r-4.5-any)morphemepiece_1.2.3.tgz(r-4.4-any)morphemepiece_1.2.3.tgz(r-4.3-any)
morphemepiece_1.2.3.tar.gz(r-4.5-noble)morphemepiece_1.2.3.tar.gz(r-4.4-noble)
morphemepiece_1.2.3.tgz(r-4.4-emscripten)morphemepiece_1.2.3.tgz(r-4.3-emscripten)
morphemepiece.pdf |morphemepiece.html✨
morphemepiece/json (API)
NEWS

# Install 'morphemepiece' in R:

install.packages('morphemepiece', repos = c('https://macmillancontentscience.r-universe.dev', 'https://cloud.r-project.org'))

Bug tracker:https://github.com/macmillancontentscience/morphemepiece/issues

On CRAN:

5.04 score 11 stars 8 scripts 311 downloads 10 exports 38 dependencies

Last updated 3 years agofrom:bc071b1a03. Checks:3 OK, 5 NOTE. Indexed: yes.

Target	Result	Latest binary
Doc / Vignettes	OK	Feb 26 2025
R-4.5-win	NOTE	Feb 26 2025
R-4.5-mac	NOTE	Feb 26 2025
R-4.5-linux	NOTE	Feb 26 2025
R-4.4-win	NOTE	Feb 26 2025
R-4.4-mac	NOTE	Feb 26 2025
R-4.3-win	OK	Feb 26 2025
R-4.3-mac	OK	Feb 26 2025

Exports:load_lookup load_or_retrieve_lookup load_or_retrieve_vocab load_vocab morphemepiece_cache_dir morphemepiece_lookup morphemepiece_tokenize morphemepiece_vocab prepare_vocab set_morphemepiece_cache_dir

Dependencies:bit bit64 cachem cli clipr cpp11 crayon digest dlr fansi fastmap fastmatch fs glue hms lifecycle magrittr memoise morphemepiece.data piecemaker pillar pkgconfig prettyunits progress purrr R6 rappdirs readr rlang stringi stringr tibble tidyselect tzdb utf8 vctrs vroom withr

Generating a Vocabulary and Lookup

Rendered fromgenerating_vocab.Rmdusingknitr::rmarkdownon Feb 26 2025.

Last update: 2021-10-26
Started: 2021-07-29

Testing the fall-through algorithm

Rendered fromalgorithm_test.Rmdusingknitr::rmarkdownon Feb 26 2025.

Last update: 2021-09-06
Started: 2021-07-29

Citation

Development and contributors

Readme and manuals

Help Manual

Help page	Topics
morphemepiece: Morpheme Tokenization	morphemepiece-package
Load a morphemepiece lookup file	load_lookup
Load a lookup file, or retrieve from cache	load_or_retrieve_lookup
Load a vocabulary file, or retrieve from cache	load_or_retrieve_vocab
Load a vocabulary file	load_vocab
Retrieve Directory for Morphemepiece Cache	morphemepiece_cache_dir
Tokenize Sequence with Morpheme Pieces	morphemepiece_tokenize
Format a Token List as a Vocabulary	prepare_vocab
Set a Cache Directory for Morphemepiece	set_morphemepiece_cache_dir