tur-html Guide

tur-html parses HTML with lexbor, which implements the WHATWG HTML parsing algorithm -- the one browsers use. The result is the shared tur-dom-core tree, read with the same node API as an XML document; the XML guide documents that API, and this guide covers what is particular to HTML.

1. Documents never fail
2. Fragments
3. The shape of the tree
4. The depth cap
5. Serializing (and why it is not sanitizing)
6. Tokenizer events
7. Options, files and encodings
8. Editing and token-level transforms

0. Installing the spice

:spices #map{
  "html" #map{:url    "https://github.com/turmeric-lang/turmeric-spices"
              :ref    "main"
              :subdir "spices/html"}
}

tur fetch builds the HTML subset of lexbor 3.0.1. tur-dom-core is pulled in automatically.

1. Documents never fail

XML is strict: a mismatched tag is fatal. HTML is the opposite -- the standard defines a tree for every possible input -- so html-parse returns a Document, not a Result:

(import dom/ownstr :refer [to-cstr release])
(import dom/core   :refer [dom-node dom-free dom-serialize])
(import html/dom   :refer [html-parse])

(defn main [] : int
  (let [d (html-parse "<title>T</title><p>one<p>two <b>bold <i>both</b> italic</i>")
        s (dom-serialize (dom-node d))]
    (println (to-cstr s))
    (release s)
    (dom-free d))
  0)

#lang sweet-exp
import dom/ownstr :refer [to-cstr release]
import dom/core   :refer [dom-node dom-free dom-serialize]
import html/dom   :refer [html-parse]

defn main [] : int
  let [d html-parse("<title>T</title><p>one<p>two <b>bold <i>both</b> italic</i>")
       s dom-serialize(dom-node(d))]
    println $ to-cstr s
    release(s)
    dom-free(d)
  0

prints

<html><head><title>T</title></head><body><p>one</p><p>two <b>bold <i>both</i></b><i> italic</i></p></body></html>

The implied <html>, <head> and <body> appear, the first <p> is closed by the second, and the misnested <b><i> is repaired the way every browser repairs it (the "adoption agency"). Tables get their <tbody>, text that strays into a table is moved before it, a <template> at the start lands in <head>, and so on. The test suite checks this against the html5lib tree-construction tests.

2. Fragments

html-parse-fragment context src parses markup as the contents of a context element -- the algorithm behind innerHTML. The context matters:

The fragment's top-level nodes are the children of the returned document's node -- there is no <html>/<head>/<body> wrapper -- and dom-root is its first element, or none for a fragment that is all text. The context is an HTML tag name; SVG and MathML contexts are not supported.

3. The shape of the tree

• Names. HTML element and attribute names are lower-case. SVG and MathML elements and attributes keep the case the parser adjusts them to (foreignObject, viewBox, definitionURL). Matching in find-all, find-first, node-matches? and node-attr ignores ASCII case in an HTML document, so (find-all n "A") finds <a>.
• Namespaces. node-ns-uri is the HTML, SVG or MathML namespace URI. Attributes are in no namespace except the adjusted xlink:, xml: and xmlns ones; (node-attr a "{http://www.w3.org/1999/xlink}href") finds xlink:href. The HTML parser never gives an element a prefix: <x:y> is an element whose local name is x:y.
• Templates. A <template>'s contents (a separate document fragment in the browser DOM) are the template element's children here.
• No positions. lexbor's tree does not record source positions, so node-pos is none for every HTML node.
• Doctype. Kept as a DoctypeNode with its name and identifiers.

4. The depth cap

Deeply nested input (<div> a million times) is a cheap way to exhaust a recursive tree walker. tur-html caps the depth the way Blink does: an element nested deeper than max-depth (512 by default) is attached beside its parent instead of under it. Nothing is dropped -- every element and all the text are still in the tree -- but no node is deeper than the cap. Every walk inside dom-core is iterative anyway, so a 100 000-deep input parses without trouble; the cap protects your own recursive code.

5. Serializing

dom-serialize on an HTML document follows the HTML fragment serialization algorithm: void elements (<br>, <img>) have no end tag, the contents of <script>, <style> and the other raw-text elements are written verbatim, &, <, > and U+00A0 are escaped in text ( ), and & and " in attribute values (plus < and >, as current browsers do).

Parsing is not sanitizing. A parsed and re-serialized document keeps every <script>, every onclick= and every javascript: URL it came with. The round trip makes markup well-formed, never safe. Do not embed untrusted HTML on the strength of having parsed it; that needs a sanitizer, which is separate (planned) work.

6. Tokenizer events

html/sax reports tokens, in the same SaxEvent vocabulary as tur-xml's SAX (see the XML guide for handlers, Stop, ownership and push parsing). It is the tokenizer alone, with no tree construction behind it:

Names are lower-case, character references are decoded in text and attribute values, NUL becomes U+FFFD, and the contents of <script>, <style>, <title>, <textarea>, <xmp>, <iframe>, <noembed>, <noframes> and <plaintext> are text. That last switch is normally the tree builder's job; here it is decided by tag name, so an SVG <style> is treated the same way. Use the tokenizer for fast scans (every link, every <meta>), and html-parse whenever you need the structure a browser would see.

The push tokenizer (html-sax-parser-new, html-sax-feed!, html-sax-finish!) delivers the same events for any chunking. lexbor 3.0.1 mishandles a chunk boundary inside a few lookaheads (a <![CDATA[, a DOCTYPE's PUBLIC / SYSTEM), so the push path holds back the tail from the first < that no > has closed yet and feeds it with the next chunk.

7. Options, files and encodings

HtmlOpts has two fields: max-depth (512) and max-bytes (64 MiB). The never-failing html-parse and html-parse-fragment apply the depth cap and no byte cap -- the input is already in memory. The -opts variants and html-parse-file apply both and return (Result Document ParseError): kind Limit past max-bytes, Io when a file cannot be read.

Input is UTF-8. lexbor's encoding sniffing (<meta charset>, BOMs) lives in a module this spice does not build, so convert other encodings first.

8. Editing and token-level transforms

A parsed tree is edited with tur-dom-core's dom/edit (see the dom-core README), and html/edit adds the HTML-specific pieces: set-inner-html! parses markup in an element's own context and puts it in place of the element's children, checked like any other edit, so markup that would close a <script> early is refused; node-inner-html and node-outer-html serialize.

html/transform is the streaming counterpart, and it works on tokens, like html/sax: html-sax-transform runs every event through a function that returns the events to write in its place, then serializes them. See the XML guide for the function's ownership rule. The identity transform gives back markup that tokenizes to the same events -- including a doctype's identifiers, so the quirks mode does not change -- which for ordinary pages parses to the same tree too. Two things to know:

• Since the stream has no tree behind it, end tags are written as the tokenizer saw them (missing, stray or misnested), and <br/> comes back as <br/>. A self-closed non-void element, <div/>, comes back as <div></div>: the two tokenize alike, but a parser keeps <div/> open.
• The writer refuses what it cannot write back as given, and the transform stops there with OutputError: a <plaintext> element (BadName), a comment that holds <!--, and <script> text holding <!-- or a </script lookalike (BadContent). The test suite runs every html5lib input through the identity; about 9% of them -- that suite aims at exactly these edges -- are refused, the rest come back token for token.

For input too large to hold, plug dom/writer's sax-transform-handler into html-sax-parser-new, with a writer from sax-writer-to-file, and feed the input in chunks.