Index News Docs Issues Source

# Architecture of Chawan

This document describes some aspects of how Chawan works.

Table of contents

## Module organization

Explanation for the separate directories found in src/:

Additionally, "adapters" of various protocols and file formats can be found in adapter/:

## Process model

Described as a tree:

### Main process

The main process runs code related to the pager. This includes processing user input, printing buffer contents to the screen, and managing buffers in general. The complete list of buffers is only known to the main process.

Mailcap commands are executed by the main process. This depends on knowing the content type of the resource, so the main process also reads in all network headers of navigation responses before launching a buffer process. More on this in Opening buffers.

### Forkserver

For forking the loader process, buffer processes and CGI processes, a fork server process is launched at the very beginning of every 'cha' invocation.

We use a fork server for two reasons:

  1. It helps clean up child processes when the main process crashes. (We open a UNIX domain socket between the main process and the fork server, and kill all child processes from the fork server on EOF.)
  2. It allows us to start new buffer processes without cloning the pager's entire address space. This reduces the impact of memory bugs somewhat, and also our memory usage.

For convenience reasons, the fork server is not used for mailcap processes.

### Loader

The loader process takes requests from the main process and the buffer processes. Then, depending on the scheme, it performs one of the following steps:

The loader process distinguishes between clients (i.e processes) through their control stream (one end of a socketpair created by loader). This control stream is closed when the pager discards the buffer, so discarded buffers are unable to make further requests even if their process is still alive.

### Buffer

Buffer processes parse HTML, optionally query external resources from loader, run styling, JS, and finally render the page to an internal canvas.

Buffers are exposed to the (JS-based) browser shell through BufferInterface objects, which communicate with the buffer process through a Unix domain socket. There is a second layer for the user-facing API as well in the form of Buffer objects, which are implemented entirely in JS. (The latter is described in detail by cha-api(7).)

## Opening buffers

Scenario: the user executes the command cha https://example.org.

  1. The main process (henceforth "pager") forks the fork server, reads the config, and relays relevant config options to the fork server. Then, the fork server forks the loader itself.

  2. Pager loads and evaluates the startup script (aka init.js) from ROM. This sets up the JS API and the browser shell, which itself is mostly implemented in JS.

  3. Pager queries the terminal about its cell size, whether it supports images, etc. This is fully async; in fact, it can happen that the website loads faster than the terminal responds to the queries, and then the pager will send a first approximation of how the website will look like (usually without color) before reflowing the page.

  4. Pager sends a request for "https://example.org" to the loader. Then, it does something else (e.g., process input) until poll() reports activity on the file descriptor.

  5. Loader looks up the browsecap handler for "http", and then asks the fork server to launch the associated "http" program, passing the URL parts as arguments.

  6. The "http" program opens a connection to example.org. When connected, it starts writing received headers (and then, the body) to stdout. Loader parses these and relays them to pager.

  7. Pager receives the headers, and decides what to do based on the Content-Type:

    • If Content-Type is found in mailcap, then the response body is piped into the command in that mailcap entry. If the entry has x-htmloutput, then the command's stdout is taken instead of the response body, and Content-Type is set to text/html. Otherwise, the buffer is discarded.

    • If Content-Type is text/html, then a new "buffer process" is created, which then parses the response body as HTML. If it is any text/* subtype, then the response is simply inserted into a <plaintext> tag.

    • If Content-Type is not a text/* subtype, and no mailcap entry for it is found, then the user is prompted about where they wish to save the file.

## Terminal I/O

Readers of this section may also be interested in cha-terminal(7), which discusses more practical questions associated with terminal handling (compatibility etc.)

As noted above, the terminal module is completely asynchronous. This applies not only to terminal querying (as described in the previous section), but also to the output, for which we use double buffering.

Normally, we write frames to the primary buffer. If the terminal can process this before we start writing the next frame, all is good, no buffering happens. But if the terminal is slower than Chawan, then we must sync our output to ensure the browser remains responsive:

  1. Copy the primary buffer to a secondary buffer. This includes not just the text to be written (which, in fact, is not directly copied, just references to immutable buffers), but also the screen state. E.g., which cells contain which text, where do we have images, scroll state, etc.

  2. Wait for the next frame. If a) the terminal reads the previous frame in the meantime, then we "unbuffer": the secondary buffer is copied to the place of the primary buffer.

    But if b) we receive the next frame before the terminal reads the previous one, then we drop the secondary frame, and start writing the next frame in its place. Repeat until the terminal catches up.

This way, every time the terminal starts receiving a new frame, it is guaranteed that it's at most one frame behind. Note that this only backpressures output, but not input; it's not really a problem right now, but it will be with things like animated GIFs (which would not overwhelm the link itself, but might spin the CPU at 100% despite the terminal not processing most of the frames).

Input, meanwhile, is rather straightforward: we have a state machine (again, async), which parses the sequences a terminal might possibly respond with to our queries. Once it is determined that the input is not such an escape sequence or another kind of input event (mouse, bracketed paste), we backtrack.

Then, we decode the data using the display charset (this ordering is a faithful interpretation of ECMA-48), and finally, look up keybindings this might match.

## Cache

Chawan's caching mechanism is largely inspired by that of w3m, which does not have a network cache. Instead, it simply saves source files to the disk before displaying them, and lets users view/edit the source without another network request.

The only difference in Chawan is that it simultaneously streams files to the cache and buffers:

  1. Client (pager or buffer) initiates request by sending a message to loader.
  2. Loader starts CGI script, reads headers, sends a response, and waits.
  3. Client now may send an "addCacheFile" message, which prompts loader to add a cache file for this request.
  4. Client sends "resume", now loader will stream the response both to the client and the cache.

Cache items may be shared between clients; this is how rewinding on wrong charset guess is implemented. They are also manually reference counted and are unlinked when their reference count drops to zero.

The cache is used in the following ways:

Crucially, the cache does not understand Cache-Control headers, and will never skip a download when requested by a user. Similarly, loading a "cache:" URL (e.g. view source) is guaranteed to never make a network request.

Future directions: for non-JS buffers, we could kill idle processes and reload them on-demand from the cache. This could solve the problem of spawning too many processes that then do nothing.

## Parsing HTML

Note: the HTML parser ("chame") and character encoding modules ("chagashi") are also available as a separate libraries. However, as it stands, only the version in Chawan is actively maintained.

Buffer processes decode and parse HTML documents asynchronously. When bytes from the network are exhausted, the buffer will 1) partially render the current document as-is, 2) return it to the pager so that the user can interact with the document.

For character encoding detection, we simply enumerate the list specified in encoding.document-charset until either no errors are produced by the decoder, or no more charsets exist. In some extremely rare edge cases, the document is re-downloaded from the cache, but this pretty much never happens. (The most common case is that the UTF-8 validator just runs through the entire document without reporting errors.)

The HTML parser then consumes the decoded (or validated) input buffer. In some cases, a script calls document.write and then the parser is called recursively. (Debugging this is not very fun.)

## JavaScript

QuickJS is used by both the pager as a scripting language, and by buffers for running on-page scripts when JavaScript is enabled.

In the past, a standalone variant of Chawan's JS binding generator had existed as well, but that is abandoned now.

### General

To avoid having to type out all the type conversion & error handling code manually, we have JS pragmas to automagically turn Nim procedures into JavaScript functions. (For details on the specific pragmas, see the manual.)

Still, sometimes we have to deal with JSValues manually; in this case, the fromJS and toJS functions are used. fromJS in particular returns a status code after filling a var parameter.

### JS in the pager

Keybindings can be assigned JavaScript functions in the config, and then the pager executes those when the keybindings are pressed.

Also, contents of the start.startup-script option are executed at startup. This is used when cha is called with the -r flag.

There is an API, described at api.md. Web APIs are exposed to the pager too, but you cannot operate on the DOM itself from the pager, unless you create one yourself with DOMParser.parseFromString.

cha-config(5) describes all commands that are used in the default config.

### JS in the buffer

The DOM is implemented through the same wrappers as those in pager, except the pager modules are not exposed to buffer JS.

This is mostly straightforward, except for the implementation of document.write (don't ask) and Node.

As common in browsers with a DOM, Node is a doubly-linked list, with a firstChild pointer allocated for node types that can hold children (Element etc.) To reduce memory use, Chawan also applies these space optimizations:

## CSS

css/ contains CSS parsing, cascading, layout, and rendering.

Note that CSS (at least 2.0 and onward) was designed for pixel-based displays, not for character-based ones, so we have to round a lot. The layout engine includes some heuristics around this so that the result is usually still acceptable.

Also, some (now) commonly used features like CSS grid are not implemented yet, so websites using those look ugly. For an exact list of supported features, see css.md.

### Parsing, cascading

The parser is based on the CSS 3 syntax module. It is optimized for single-pass parsing, so the output does not really resemble the CSSOM at first glance.

Cascading works OK. To speed up selector matching, various properties are hashed to filter out irrelevant CSS rules. In addition, class descendant selectors are optimized using a Bloom filter, and identical style objects are hashed and shared with unrelated elements.

Style calculation is incremental, and results are cached until an element's style is invalidated, so re-styles are quite fast. (The invalidation logic is primitive, but it's good enough in most cases.)

### Layout

Layout runs whenever a page is loaded, or some action (e.g. hover, DOM change by JS, etc.) invalidates the page currently visible to the user.

Our layout engine is a "simple" procedural implementation which consists of two passes:

  1. css/csstree.nim: build a layout tree, possibly reusing the tree from the previous layout. Anonymous block and table boxes are generated here. After this pass, the tree is no longer mutated, only the state and render fields of the respective boxes.

  2. css/layout.nim: position said boxes, always relative to their parent. This pass takes input and compares it with input previously taken; if they differ, it recurses through its children and then stores the box size and other output in state.

    But if input is the same as in the previous pass and keepLayout wasn't unset, it is assumed that layout for the subtree has not changed, and it is simply skipped (for the box itself as well as for all its children.)

In practice, step 2 is often repeated for subsections of the tree to resolve cyclic dependencies in CSS layout (e.g. in table, flex). However, this rarely results in quadratic behavior thanks to the aforementioned caching mechanism.

### Rendering

After layout is finished, the document is rendered onto a text-based canvas, which is represented as a sequence of strings associated with their formatting. (Right now, "formatting" also includes a reference to the respective DOM nodes; in the future, it won't.)

Additionally, boxes are assigned an offset in the render field here, which is used when jumping to anchors.

The entire document is rendered, which is a performance bottleneck in some cases. (Styling and layout are both slower, but those are cached. Rendering isn't, but all it really does is just copying around a bunch of strings and computers are very good at this.)

The positive side of this design is that search is very simple (and fast), since we are just running regexes over a linear sequence of strings.