python-tokenizers 0.19.1

Implementation of various popular tokenizers

This package provides an implementation of today’s most used tokenizers, with a focus on performance and versatility.

Outputs: out
Website: https://huggingface.co/docs/tokenizers
Licenses: ASL 2.0
Package source: gnu/packages/machine-learning.scm
Builds: See build status
Issues: See known issues

Installation

Install python-tokenizers 0.19.1 as follows:

guix install python-tokenizers@0.19.1

Or install the latest version:

guix install python-tokenizers

You can also install packages in augmented, pure or containerized environments for development or simply to try them out without polluting your user profile. See the guix shell documentation for more information.