Package: wordpiece 2.1.3

Jonathan Bratt

wordpiece: R Implementation of Wordpiece Tokenization

Apply 'Wordpiece' (<arxiv:1609.08144>) tokenization to input text, given an appropriate vocabulary. The 'BERT' (<arxiv:1810.04805>) tokenization conventions are used by default.

Authors:Jonathan Bratt [aut, cre], Jon Harmon [aut], Bedford Freeman & Worth Pub Grp LLC DBA Macmillan Learning [cph]

wordpiece_2.1.3.tar.gz
wordpiece_2.1.3.zip(r-4.7)wordpiece_2.1.3.zip(r-4.6)wordpiece_2.1.3.zip(r-4.5)
wordpiece_2.1.3.tgz(r-4.6-any)wordpiece_2.1.3.tgz(r-4.5-any)
wordpiece_2.1.3.tar.gz(r-4.7-any)wordpiece_2.1.3.tar.gz(r-4.6-any)
wordpiece_2.1.3.tgz(r-4.6-emscripten)
manual.pdf |manual.html✨
DESCRIPTION |NEWS
card.svg |card.png
wordpiece/json (API)

# Install 'wordpiece' in R:

install.packages('wordpiece', repos = c('https://macmillancontentscience.r-universe.dev', 'https://cloud.r-project.org'))

Bug tracker:https://github.com/macmillancontentscience/wordpiece/issues

On CRAN:

4.60 score 8 stars 8 scripts 290 downloads 7 exports 18 dependencies

Last updated from:3eb92c7595. Checks:7 NOTE, 2 OK. Indexed: yes.

Target	Result	Time
linux-devel-x86_64	NOTE	127
source / vignettes	OK	204
linux-release-x86_64	NOTE	127
macos-release-arm64	NOTE	109
macos-oldrel-arm64	NOTE	108
windows-devel	NOTE	87
windows-release	NOTE	87
windows-oldrel	NOTE	81
wasm-release	OK	122

Exports:load_or_retrieve_vocab load_vocab prepare_vocab set_wordpiece_cache_dir wordpiece_cache_dir wordpiece_tokenize wordpiece_vocab

Dependencies:cachem cli digest dlr fastmap fastmatch fs glue lifecycle magrittr memoise piecemaker rappdirs rlang stringi stringr vctrs wordpiece.data

Using wordpiece

Provided Vocabularies | Loading a Vocabulary | Tokenizing Text | Vocabulary Case | Representing "Unknown" Tokens

Rendered frombasic_usage.Rmdusingknitr::rmarkdown

Last update: 2021-09-27
Started: 2021-01-12

Citation

Development and contributors

Readme and manuals

Help Manual

Help page	Topics
Load a vocabulary file, or retrieve from cache	load_or_retrieve_vocab
Load a vocabulary file	load_vocab
Format a Token List as a Vocabulary	prepare_vocab
Set a Cache Directory for wordpiece	set_wordpiece_cache_dir
Retrieve Directory for wordpiece Cache	wordpiece_cache_dir
Tokenize Sequence with Word Pieces	wordpiece_tokenize