scscm/lexer
nil-value
(nil-value)
list-nil?
(list-nil? [lst : int])
cons
(cons [value : int next : int])
head
(head [lst : int])
tail
(tail [lst : int])
lex-ok
(lex-ok [x : int])
lex-err
(lex-err [e : int])
lex-ok?
(lex-ok? [r ptr<void>])
lex-err?
(lex-err? [r ptr<void>])
lex-ok-val
(lex-ok-val [r ptr<void>])
lex-err-val
(lex-err-val [r ptr<void>])
token-type
(token-type [tok : int])
return the type keyword string for a token handle.
| tok | token handle returned by tokenize |
One of ":atom" ":symbol" ":keyword" ":number" ":string" ":lparen" ":rparen" ":lbracket" ":rbracket" ":lbrace" ":rbrace" ":quote" ":quasiquote" ":unquote" ":unquote-splice" ":hash-paren" ":eof"
(token-type (head (lex-ok-val (tokenize "(+ 1 2)")))) ; => ":lparen"
Since: SC0
token-value
(token-value [tok : int])
return the raw text value of a token.
| tok | token handle returned by tokenize |
The raw scanned text as :cstr (e.g. "foo-bar", ":freq", "440", "hello").
(token-value tok) ; => "foo-bar"
Since: SC0
token-line
(token-line [tok : int])
return the 1-based source line of a token.
| tok | token handle |
Line number as :int (1-based).
(token-line tok) ; => 1
Since: SC0
token-col
(token-col [tok : int])
return the 1-based source column of a token.
| tok | token handle |
Column number as :int (1-based).
(token-col tok) ; => 1
Since: SC0
scscm-list-rev
(scscm-list-rev [lst : int acc : int])
scscm-cstr-as-int
(scscm-cstr-as-int [s : cstr])
scscm-int-as-cstr
(scscm-int-as-cstr [n : int])
scscm-cstr-eq?
(scscm-cstr-eq? [a : cstr b : cstr])
tokenize
(tokenize [source : cstr])
scan scscm source text and return a cons list of token handles.
| source | NUL-terminated scscm source string |
lex-ok(tokens :int) on success, where tokens is a cons list of token handles. lex-err(message :cstr) on scan error.
(let [res (tokenize "(+ 1 2)")]
(if (lex-ok? res) (lex-ok-val res) (lex-err-val res)))
Since: SC0
tokens-free
(tokens-free [ts : int])
free all token STok structs in a cons list.
| ts | cons list of token handles returned by tokenize |
(tokens-free my-tokens)
Since: SC0
Internal definitions
scscm-tokenize-raw-- scan all tokens from source into a heap array.scscm-rtl-error?-- true if the raw token list contains a scan error.scscm-rtl-error-- return the error message from the raw token list.scscm-rtl-count-- return the number of tokens in the raw token list.scscm-rtl-get-- return the i-th token handle from the raw token list.scscm-rtl-free-- free the raw token list struct and all token structs.scscm-rtl-free-shallow-- free the STL wrapper only, not the STok structs.scscm-tokens-buildscscm-token-free-one-- free one STok struct (value string + struct itself).scscm-void-noop