//clika-runtime/io.clika.runtime/Token

Token

[common]
data class Token(val id: Int, val text: String, val begin: Long, val end: Long, val special: Boolean)

One token of Tokenizer.tokenize: its id, the surface text it spells, the half-open byte span [begin, end) it covers in the source text, and whether the tokenizer injected it itself (special: bos, eos, ...) rather than reading it from the input.

Constructors

Token	[common] constructor(id: Int, text: String, begin: Long, end: Long, special: Boolean)

Properties

Name	Summary
begin	[common] val begin: Long
end	[common] val end: Long
id	[common] val id: Int
special	[common] val special: Boolean
text	[common] val text: String