Tokenizers

Concept and definition

Tokenizers

What is Tokenizers?

In natural language processing, a tokeniser is a tool used to break up text into discrete units called tokens. A token can be a word, punctuation, number, symbol or other meaningful unit in the text. The purpose of the tokeniser is to prepare the text for machine learning analysis and modelling.

There are different types of tokenisers, including rule-based and machine learning-based tokenisers. Rule-based tokenisers use predefined patterns to divide text into tokens, while machine learning-based tokenisers use language models to identify patterns and structures in the text and divide it into tokens.

Tokenisers are an important tool in natural language processing, as proper representation of input data is essential for training accurate machine learning models.

« Back to glossary

Do you want to get in touch?

CDRs contain data that a telecommunications company collects about phone calls, such as time and length of call. This data can be used in analytical applications.
Fill the form
Share:
What is Natural Language Processing?

Natural Language Processing or NLP analyzes how machines understand, interpret and process human language.

Read More »
What are the advantages of artificial intelligence?�

The use of Artificial Intelligence in business is becoming more and more common and necessary for the optimization and evolution of processes. In one of our [...]

Read More »
Abbreviated History of Artificial Intelligence

The content of this article synthesizes part of the chapter "Concept and brief history of Artificial Intelligence" of the thesis Generation of Artificial [...]

Read More »
What is Surety Insurance?

You are probably wondering, what is surety insurance and how does it help your company? In today's economic environment, [...]

Read More »
See more entries
© Gamco 2021, All Rights Reserved - Legal notice - Privacy - Cookies