tokenize

English

Alternative forms

• tokenise

Etymology

Etymology tree

English token

Proto-Indo-European *-id-

Proto-Indo-European *-yéti

Proto-Indo-European *-idyéti

Proto-Hellenic *-íd͏̌d͏̌ō

Ancient Greek -ῐ́ζω (-ĭ́zō)bor.

Late Latin -izōder.

Middle French -iserbor.

Middle English -isen

English -ize

English tokenize

From token + -ize.

Pronunciation

• (General American) IPA^(key): /ˈtoʊ.kən.aɪz/

Verb

tokenize (third-person singular simple present tokenizes, present participle tokenizing, simple past and past participle tokenized)

1. (transitive, computing) To reduce to a token or set of tokens by lexical analysis.

   Coordinate terms: codify, encode

2. (transitive, computing) To substitute sensitive data with meaningless placeholders.
3. (transitive) To treat as a token minority.

Derived terms

• detokenize
• mistokenize
• retokenize
• tokenizable
• tokenization
• tokenizer
• untokenized

Related terms

• tokenism

Anagrams

• ketonize

Portuguese

Verb

tokenize

1. inflection of tokenizar:
   1. first/third-person singular present subjunctive
   2. third-person singular imperative