Package: morphemepiece 1.2.3

Jonathan Bratt

morphemepiece: Morpheme Tokenization

Tokenize text into morphemes. The morphemepiece algorithm uses a lookup table to determine the morpheme breakdown of words, and falls back on a modified wordpiece tokenization algorithm for words not found in the lookup table.

Authors:Jonathan Bratt [aut, cre], Jon Harmon [aut], Bedford Freeman & Worth Pub Grp LLC DBA Macmillan Learning [cph]

morphemepiece_1.2.3.tar.gz
morphemepiece_1.2.3.zip(r-4.7)morphemepiece_1.2.3.zip(r-4.6)morphemepiece_1.2.3.zip(r-4.5)
morphemepiece_1.2.3.tgz(r-4.6-any)morphemepiece_1.2.3.tgz(r-4.5-any)
morphemepiece_1.2.3.tar.gz(r-4.7-any)morphemepiece_1.2.3.tar.gz(r-4.6-any)
morphemepiece_1.2.3.tgz(r-4.6-emscripten)
manual.pdf |manual.html✨
DESCRIPTION |NEWS
card.svg |card.png
morphemepiece/json (API)

# Install 'morphemepiece' in R:

install.packages('morphemepiece', repos = c('https://macmillancontentscience.r-universe.dev', 'https://cloud.r-project.org'))

Bug tracker:https://github.com/macmillancontentscience/morphemepiece/issues

On CRAN:

5.08 score 11 stars 11 scripts 267 downloads 10 exports 37 dependencies

Last updated from:bc071b1a03. Checks:7 NOTE, 2 OK. Indexed: yes.

Target	Result	Time
linux-devel-x86_64	NOTE	134
source / vignettes	OK	183
linux-release-x86_64	NOTE	137
macos-release-arm64	NOTE	89
macos-oldrel-arm64	NOTE	97
windows-devel	NOTE	76
windows-release	NOTE	84
windows-oldrel	NOTE	82
wasm-release	OK	122

Exports:load_lookup load_or_retrieve_lookup load_or_retrieve_vocab load_vocab morphemepiece_cache_dir morphemepiece_lookup morphemepiece_tokenize morphemepiece_vocab prepare_vocab set_morphemepiece_cache_dir

Dependencies:bit bit64 cachem cli clipr cpp11 crayon digest dlr fastmap fastmatch fs glue hms lifecycle magrittr memoise morphemepiece.data piecemaker pillar pkgconfig prettyunits progress purrr R6 rappdirs readr rlang stringi stringr tibble tidyselect tzdb utf8 vctrs vroom withr

Generating a Vocabulary and Lookup

Check coverage | Other measures of coverage quality

Rendered fromgenerating_vocab.Rmdusingknitr::rmarkdown

Last update: 2021-10-26
Started: 2021-07-29

Testing the fall-through algorithm

Rendered fromalgorithm_test.Rmdusingknitr::rmarkdown

Last update: 2021-09-06
Started: 2021-07-29

Citation

Development and contributors

Readme and manuals

Help Manual

Help page	Topics
morphemepiece: Morpheme Tokenization	morphemepiece-package
Load a morphemepiece lookup file	load_lookup
Load a lookup file, or retrieve from cache	load_or_retrieve_lookup
Load a vocabulary file, or retrieve from cache	load_or_retrieve_vocab
Load a vocabulary file	load_vocab
Retrieve Directory for Morphemepiece Cache	morphemepiece_cache_dir
Tokenize Sequence with Morpheme Pieces	morphemepiece_tokenize
Format a Token List as a Vocabulary	prepare_vocab
Set a Cache Directory for Morphemepiece	set_morphemepiece_cache_dir