python-tokenizers
Implementation of various popular tokenizers
This package provides an implementation of today’s most used tokenizers, with a focus on performance and versatility.
- Versions: 0.19.1
- Website: https://huggingface.co/docs/tokenizers
- Licenses: ASL 2.0
- Package source: gnu/packages/machine-learning.scm
- Builds: See build status
- Issues: See known issues
Installation
Install the latest version of python-tokenizers as follows:
guix install python-tokenizers
Or install a particular version:
guix install python-tokenizers@0.19.1
You can also install packages in augmented, pure or containerized environments for development or simply to try them out without polluting your user profile. See the guix shell documentation for more information.
Badge code
You can use the following badge to inform users of python-tokenizers about the latest version available in Guix.
Example HTML:
<a href='https://packages.guix.gnu.org/packages/python-tokenizers'><img src='https://packages.guix.gnu.org/packages/python-tokenizers/badges/latest-version.svg' alt='Version in GNU Guix'></a>
Example Markdown:
[](https://packages.guix.gnu.org/packages/python-tokenizers)
Example Org:
[[https://packages.guix.gnu.org/packages/python-tokenizers][https://packages.guix.gnu.org/packages/python-tokenizers/badges/latest-version.svg]]