8. Vanes I: Behn, Dill, Kahn, Lick

A vane is an Arvo kernel module that performs essential system operations. The vanes are:

• Ames, the peer-to-peer networking vane.
• Behn, the timer vane.
• Clay, the filesystem, revision-control and build system vane.
• Dill, the terminal-driver vane.
• Eyre, the HTTP vane.
• Gall, the application vane.
• Iris, the server HTTP vane.
• Jael, the security vane.
• Khan, the control vane.
• Lick, the interprocess communication (IPC) vane.

As described above, we use Arvo proper to route and control the flow of moves. However, Arvo proper is rarely directly responsible for processing the event data that directly causes the desired outcome of a move. This event data is contained within a card. Instead, Arvo proper passes the card off to one of its vanes, which each present an interface to clients for a particular well-defined, stable, and general-purpose piece of functionality.

 Vane Interface

Arvo is a message dispatcher, which doesn't really know about the vanes except via their existence in a van=(map term vane) in Arvo's +$soul.

Formally, a vane must be a “vane-shaped noun”—an interface presenting the arms:

|%
::  +call: handle a +task request
++  call
::  +load: migrate an old state to a new vane version
++  load
::  +scry: view vane state at a particular /path
++  scry
::  +stay: extract state before reload
++  stay
::  +take: handle $response sign
++  take
--

• ++load and ++stay are necessary to update the vane.
• ++call is used to pass a request in (“advance to target”).
• ++scry exposes the read-only scry namespace of the vane.
• ++take receives a response from another vane (“retreat along call stack”).

(Now the formerly-elided distinctions between sign, gift, task, and note start to matter.)

• A note is sent by a vane to the Arvo kernel's ++call arm.
• Arvo dispatches a task to a vane's ++call arm.
• The vane performs the work.
• If a result needs to be passed back, it is emitted as a gift along the duct back to Arvo's ++take arm.
• Arvo dispatches a sign to the original caller's ++take arm.

The actual mechanics of this are that the moves are placed into the appropriate duct, which is a (list wire), simply an ordered collection of moves representing the causal history.

/sys/arvo tracks what little it knows about vanes at a few points, e.g.:

::  van: vanes while we desire it (in larval stage)
van=(map term (trap vase))
::
++  grow
  |=  way=term
  ?+  way  way
    %a  %ames
    %b  %behn
    %c  %clay
    %d  %dill
    %e  %eyre
    %g  %gall
    %i  %iris
    %j  %jael
    %k  %khan
    %l  %lick
  ==
::
+$  vane  [=vase =worm]

• vase is of course a generic vase, but specifically it expects a noun with the correct +$type.
• worm is the worm cache as discussed in ca04.

Arvo interacts with vanes in vase mode; for instance, a scry takes place via a call to the ++scry arm via a ++slap against the %limb named %scry: (~(slap wa sac) rig [%limb %scry]). As usual, working in vase mode permits dynamic updates to the source.

Vanes have as their subject:

• /sys/hoon for language definitions.
• /sys/arvo for message dispatch.
• /sys/lull for a shared interface definition.
• /sys/zuse for various stdlib utilities.

In particular, /sys/lull acts as a header so that vanes can “see” each other's interface.

Updates

As with other parts of the system, vanes are rebuilt if the inner core on which they rely has been updated or if the vane itself has changed.

An update to a vane is triggered by ++mod:what:pith in the ++le event-loop engine. (Recall from ca05 that ++what is involved in a system upgrade.) While there are some unfamiliar types here, note particularly the %= centis clause building each vane.

++  mod
  |=  [del=news all=?]
  ^+  ..pith
  =^  job=oped  fat.mod.sol  (~(adorn adapt fat.mod.sol) del all)
  =?  lul.mod.sol  ?=(^ lul.job)
    (smit:va "lull" pit /sys/lull/hoon u.lul.job)
  =?  zus.mod.sol  ?=(^ zus.job)
    (smit:va "zuse" lul.mod.sol /sys/zuse/hoon u.zus.job)
  %-  %+  need:wyrd   kel.ver.zen
      :~  lull/;;(@ud q:(slap lul.mod.sol limb/%lull))
          zuse/;;(@ud q:(slap zus.mod.sol limb/%zuse))
      ==
  %=    ..pith
      van.mod
    %+  roll  van.job
    |=  [[nam=term txt=cord] van=_van.mod.sol]
    ^+  van
    =/  nex  (create:va our zus.mod.sol nam /sys/vane/[nam]/hoon txt)
    =/  nav  (~(get by van) nam)
    =?  nex  ?=(^ nav)  (update:va vase.u.nav nex)
    (~(put by van) nam (settle:va nex))
  ==

The recompilation against %zuse takes place in ++adorn:adapt:part. (Arvo Pärt↗)

::  kernel modules
::
::    %zuse is the subject of the vanes; force all if we have a new %zuse
::
=.  all  |(all ?=(^ zus))
=|  nav=(map term cord)
=?  nav  all
  %-  ~(gas by nav)
  %+  turn
    ~(tap by dir:(~(dip of fat) /sys/vane))
  |=([name=@ta _fat] [`@tas`name (sole (need fil))])

 Behn

Behn is a timer/wake-up call system. Since it's a simple vane, let's approach it obliquely, by looking at a generator that calls it.

• Open /base/gen/timers/hoon and examine the code.

.^((list [date=@da =duct]) %bx (en-beam [our %$ [%da now]] /debug/timers))

• “Behn Overview”

/sys/lull Definition

The interface to Behn is defined in /sys/lull:

::                                                      ::::
::::                    ++behn                            ::  (1b) timekeeping
  ::                                                    ::::
++  behn  ^?
  |%
  +$  gift                                              ::  out result <-$
    $%  [%doze p=(unit @da)]                            ::  next alarm
        [%wake error=(unit tang)]                       ::  wakeup or failed
        [%meta p=vase]
        [%heck syn=sign-arvo]                           ::  response to %huck
    ==
  +$  task                                              ::  in request ->$
    $~  [%vega ~]                                       ::
    $%  $>(%born vane-task)                             ::  new unix process
        [%rest p=@da]                                   ::  cancel alarm
        [%drip p=vase]                                  ::  give in next event
        [%huck syn=sign-arvo]                           ::  give back
        $>(%trim vane-task)                             ::  trim state
        $>(%vega vane-task)                             ::  report upgrade
        [%wait p=@da]                                   ::  set alarm
        [%wake ~]                                       ::  timer activate
    ==
  --  ::behn

(A %vega task informs the vane that the kernel has been upgraded.)

Structure

+$  behn-state
  $:  %2
      timers=(tree [key=@da val=(qeu duct)])
      unix-duct=duct
      next-wake=(unit @da)
      drips=drip-manager
  ==

• How does Behn think of timers?

/sys/behn presents three primary cores:

1. A type definition core.
2. A helper core.
3. The primary vane interface.

Behn only has two kinds of moves: %wait tasks and %wake gifts. How are these processed and what do they result in? (See the +$timer-map structure too.)

• What is a drip? How is it used?

Say an app (the Target) is subscribed to updates from Clay (the Client). If Clay %gives updates to the app directly and the app crashes, this may cause Clay to crash as well. If instead Clay %passes Behn a %drip task wrapping the update gift, Behn will set a timer for now that, when fired, will cause the update gift to be given. If it causes a crash then it will have been in response to the %drip move, thereby isolating Clay from the crash. Thus %drip acts as a sort of buffer against cascading sequences of crashes.

The Nested Core Pattern

(~rabsef-bicrym calls this the “++abet engine” and while it's not a popular term inside of the core development team, I like the pithiness of it. “Engine” as a term of art is frequently used in the kernel to refer to ++le two-letter doors so this usage is not completely inconsistent, just more specialized in application.)

The basic concept of the nested core pattern is to have an outer core which builds a list of cards and state changes, then produces the queued changes all at once. (I always think about this as being one of those wind-up cars that you crank and then set down to whir away.)

Behn's nested core pattern is pretty simple: it has an alias to this, one ++emit arm to prepend a move to a list of moves, and an ++abet arm to yield the [moves state]. The ++per-event core is used to script the neighboring ++scry and ++call arms for the vane without leaking state invariants. Behn's instantiation of the ++abet pattern centralizes the helper outer core as a centralized state machine.

These are some common ++abet pattern arms. These are not all unique, and many cores will omit all or most of these.

• ++abed—initialize. Set up the state of the inner core.
• ++yoke—initialize. Start from a particular value.
• ++abet—finalize. Exit from an inner core to an outer core, taking changes. Commonly, take a modified valued and overwrite it into the final state with a ++put:by.
• ++abut—finalize. Alternative exit from ++abet for a deletion.
• ++move—send a move. Generalization for ++pass/++give.
• ++pass—request an action. Prepend a %pass move to the current list of moves.
• ++give—return a result. Prepend a standard %give to the current list of moves.
• ++emit—submit a card. Prepend a card to the current list of cards.
• ++emil—submit cards. Prepend a list of cards to the current list of cards.

If some state needs to be maintained, this can be built in a door, but Behn's particular example is even more basic. In files with associated doors or with multiple nested core instances, it is common to prepend a two-letter identifier to disambiguate which outer core is being scripted at any given time, such as ++mo-abet or ++ap-emil.

• ~lagrev-nocfep, “The Nested Core Design Pattern (As Seen Through ++abet)”

Vere I/O Driver: vere/io/behn.c

Arvo acquires its timer updates from Unix via vere/io/behn.c. This file presents its primary interface at u3_behn_io_init() to initialize a timer. This simply retrieves the current Unix time for a starting point and sets up the interface.

Each communicating vane is linked various I/O drivers in vere/auto.c. (These do not correspond one-to-one with vanes.) These are registered into car_u, a global _u3_auto used for I/O driver invocations and callbacks.

For Behn, the timer is set in _behn_ef_doze() using uv_timer_start(). The corresponding wakeup timer is emitted in _behn_time_cb() when the libuv main event loop handler.

Finally, we can examine how the injection comes back into Arvo in _behn_time_cb(). An ovum is produced by u3_ovum_init(), manually injected using u3_auto_plan(), and subscribed to with u3_auto_peer(). /sys/behn then processes this wakeup event as a %wake via its ++call arm.

 Dill

Dill is Urbit's terminal driver.

|pass [%d %text "hello world"]

Dill as a vane is mostly responsible for actually constructing terminal sessions and coordinating input and output. Thus most of the terminal stack actually lives in userspace (instrumented by Gall) rather than in /sys/dill.

What do we mean when we talk about a terminal? Originally, of course, computer were directly programmed by moving wires between vacuum tubes or chips; later, this evolved to the ability to read and output cards. Computer terminals with CRT-based character displays began to be used in the late 1950s and gradually became more common. In fact, the original plasma display screens were used with PLATO in the 1970s–1990s.

When we refer to the terminal today, we typically mean a modern terminal emulator↗, which presents a terminal-like text user interface (TUI) for software to treat as if it were an actual character display. Terminal emulators need to track information like dimensions ($x$, $y$), content layout, active sessions or connexions, and cursor position. They provide affordances like a color space, escape codes, layout libraries, and scrollable sessions.

Dill is responsible for interfacing with keystrokes and with the terminal emulator session. Since Urbit can be run in a daemon mode, it's not necessary for Dill to actually have a terminal session for Urbit to run.

Due to terminal emulator limitations, Dill sessions are only properly supported today in the %webterm app.

• “Developer Call: Urbit’s improved Terminal Stack”↗
• “Dill Overview”

/sys/lull Definition

The /sys/lull interface specification for Dill is more complicated than that of Behn. Unlike Behn, a number of supporting types are necessary to produce the basic pair of gift/task for Dill.

::                                                      ::::
::::                    ++dill                            ::  (1d) console
  ::                                                    ::::
++  dill  ^?
  |%
  +$  gift                                              ::  out result <-$
    $%  [%blit p=(list blit)]                           ::  terminal output
        [%logo ~]                                       ::  logout
        [%meld ~]                                       ::  unify memory
        [%pack ~]                                       ::  compact memory
        [%trim p=@ud]                                   ::  trim kernel state
        [%logs =told]                                   ::  system output
    ==                                                  ::
  +$  task                                              ::  in request ->$
    $~  [%vega ~]                                       ::
    $%  [%boot lit=? p=*]                               ::  weird %dill boot
        [%crop p=@ud]                                   ::  trim kernel state
        [%flog p=flog]                                  ::  wrapped error
        [%heft ~]                                       ::  memory report
        $>(%init vane-task)                             ::  after gall ready
        [%logs p=(unit ~)]                              ::  watch system output
        [%meld ~]                                       ::  unify memory
        [%pack ~]                                       ::  compact memory
        [%seat =desk]                                   ::  install desk
        [%shot ses=@tas task=session-task]              ::  task for session
        $>(%trim vane-task)                             ::  trim state
        $>(%vega vane-task)                             ::  report upgrade
        [%verb ~]                                       ::  verbose mode
        [%knob tag=term level=?(%hush %soft %loud)]     ::  deprecated removeme
        session-task                                    ::  for default session
        told                                            ::  system output
    ==                                                  ::
  ::                                                    ::
  +$  session-task                                      ::  session request
    $%  [%belt p=belt]                                  ::  terminal input
        [%blew p=blew]                                  ::  terminal config
        [%flee ~]                                       ::  unwatch session
        [%hail ~]                                       ::  terminal refresh
        [%open p=dude:gall q=(list gill:gall)]          ::  setup session
        [%shut ~]                                       ::  close session
        [%view ~]                                       ::  watch session blits
    ==                                                  ::
  ::                                                    ::
  +$  told                                              ::  system output
    $%  [%crud p=@tas q=tang]                           ::  error
        [%talk p=(list tank)]                           ::  tanks (in order)
        [%text p=tape]                                  ::  tape
    ==                                                  ::
  ::
  ::::                                                  ::  (1d2)
    ::
  +$  blew  [p=@ud q=@ud]                               ::  columns rows
  +$  belt                                              ::  client input
    $?  bolt                                            ::  simple input
        [%mod mod=?(%ctl %met %hyp) key=bolt]           ::  w/ modifier
        [%txt p=(list @c)]                              ::  utf32 text
    ==                                                  ::
  +$  bolt                                              ::  simple input
    $@  @c                                              ::  simple keystroke
    $%  [%aro p=?(%d %l %r %u)]                         ::  arrow key
        [%bac ~]                                        ::  true backspace
        [%del ~]                                        ::  true delete
        [%hit x=@ud y=@ud]                              ::  mouse click
        [%ret ~]                                        ::  return
    ==                                                  ::
  +$  blit                                              ::  client output
    $%  [%bel ~]                                        ::  make a noise
        [%clr ~]                                        ::  clear the screen
        [%hop p=$@(@ud [x=@ud y=@ud])]                  ::  set cursor col/pos
        [%klr p=stub]                                   ::  put styled
        [%mor p=(list blit)]                            ::  multiple blits
        [%nel ~]                                        ::  newline
        [%put p=(list @c)]                              ::  put text at cursor
        [%sag p=path q=*]                               ::  save to jamfile
        [%sav p=path q=@]                               ::  save to file
        [%url p=@t]                                     ::  activate url
        [%wyp ~]                                        ::  wipe cursor line
    ==                                                  ::
  +$  dill-belt                                         ::  arvo input
    $%  belt                                            ::  client input
        [%cru p=@tas q=(list tank)]                     ::  errmsg (deprecated)
        [%hey ~]                                        ::  refresh
        [%rez p=@ud q=@ud]                              ::  resize, cols, rows
        [%yow p=gill:gall]                              ::  connect to app
    ==                                                  ::
  +$  dill-blit                                         ::  arvo output
    $%  blit                                            ::  client output
        [%qit ~]                                        ::  close console
    ==                                                  ::
  +$  flog                                              ::  sent to %dill
    $%  [%crop p=@ud]                                   ::  trim kernel state
        $>(%crud told)                                  ::
        [%heft ~]                                       ::
        [%meld ~]                                       ::  unify memory
        [%pack ~]                                       ::  compact memory
        $>(%text told)                                  ::
        [%verb ~]                                       ::  verbose mode
    ==                                                  ::
  ::                                                    ::
  +$  poke                                              ::  dill to userspace
    $:  ses=@tas                                        ::  target session
        dill-belt                                       ::  input
    ==                                                  ::
  --  ::dill

The main concepts to keep in mind:

• Dill receives %belt tasks and sends %blit gifts.
• %belt tasks result from keystrokes, terminal resizing,
• %blit gifts result from output events: putting a character, clearing the screen, placing the cursor.

Structure

Dill's primary state is its +$axle with a logging level:

|%                                                      ::  console protocol
+$  axle                                                ::
  $:  %7                                                ::
      hey=(unit duct)                                   ::  default duct
      dug=(map @tas axon)                               ::  conversations
      eye=(jug @tas duct)                               ::  outside observers
      ear=(set duct)                                    ::  syslog listeners
      lit=?                                             ::  boot in lite mode
      egg=_|                                            ::  see +take, removeme
  ==                                                    ::
+$  axon                                                ::  dill session
  $:  ram=term                                          ::  console program
      tem=(unit (list dill-belt))                       ::  pending, reverse
      wid=_80                                           ::  terminal width
  ==                                                    ::
+$  log-level  ?(%hush %soft %loud)                     ::  none, line, full
--

As with /sys/behn, the primary cores of Dill include:

1. Type definitions (two cores).
2. Helper core (++as per-cause engine).
3. The primary vane interface.

A good way to familiarize yourself with Dill operations is to follow the full lifecycle of input and output in the next section.

Dill is the first vane in the boot sequence, and is used to boot Jael. (Compare %aqua, which does not need to start Dill and can initialize Jael directly.)

Vere I/O Driver: vere/io/term.c, ptty.c

The main entrypoint for the terminal is u3_term_io_init(), which simply sets up the interface and callbacks.

As we noted before, each communicating vane is linked various I/O drivers in vere/auto.c. Here the global car_u has term.c connected for invocations and callbacks.

vere/vere.h:

// u3_term_start_spinner(): prepare spinner state. RETAIN.
void u3_term_start_spinner(u3_noun say, c3_o del_o);

// u3_term_stop_spinner(): reset spinner state and restore input line.
void u3_term_stop_spinner(void);

// u3_term_get_blew(): return window size [columns rows].
u3_noun u3_term_get_blew(c3_l tid_l);

// u3_term_ef_winc(): window change.
void u3_term_ef_winc(void);

// u3_term_ef_ctlc(): send ^C.
void u3_term_ef_ctlc(void);

// u3_term_io_init(): initialize terminal I/O.
u3_auto* u3_term_io_init(u3_pier* pir_u);

// u3_term_io_hija(): hijack console for cooked print.
FILE* u3_term_io_hija(void);

// u3_term_it_log(): writes a log message
void u3_term_io_log(c3_c* line);

// u3_term_io_loja(): release console from cooked print.
void u3_term_io_loja(int x, FILE* f);

// u3_term_log_init(): initialize terminal for logging
void u3_term_log_init(void);

// u3_term_log_exit(): clean up terminal.
void u3_term_log_exit(void);

// u3_ptty_init(): initialize platform-specific tty.
u3_utty* u3_ptty_init(uv_loop_t* lup_u, const c3_c** err_c);

For instance, a keystroke is processed in the following way:

• uv_read_start() is the libuv event loop injector.
• _term_read_cb() is the character keystroke callback.
• _term_suck() processes input.
• _term_io_suck_char() decides if it's an xterm terminal emulator issue or something for Arvo to know about.
• _term_io_spit() inputs the buffer and belt.
• _term_io_belt() actually sends a value along the belt (as an ovum).

Now in Arvo, the keystroke is routed to Dill:

• ++call takes the task from Arvo.
• ++call:as receives the input and dispatches on %belt.
• ++send:as sends the action to the proper session.
• ++deal:as signals to pass the keystroke to Gall.
• ++pass:as executes the pass to Gall.

Output can happen in three ways:

1. Some output traverses a path from (say) Gall outbound. These are conventionally known as %slogs.

   • Somehow a noun is marked for output using a %slog hint:
     • ~& sigpam does this directly.
     • ~> siggar can do this with a %slog hint and a priority value.
     • ++slog wraps this as a function.

   Back in the runtime:

   • Once we have a %slog hint for the runtime, it can be emitted from the Nock processor via Nock Eleven. noun/nock.c:_n_bint() dispatches this via SLOG and thence do_slog in the bytecode processor.
   • noun/trace.c:u3t_slog() prints a value directly through the u3C.slog_f print handler, which is _cw_serf_send_slog().
   • vere/main.c::_cw_serf_send_slog() sends the hint output to the serf.
   • vere/main.c:_cw_serf_send() is a plea handler to send pleas to the daemon.
   • vere/newt.c:u3_newt_send() sends a ++jammed noun of the output (u3s_jam_xeno()) as a buffer to a stream. In this case, the value ends up at the libuv buffer using uv_write().

   In this case, output results in the fact of the layout of the terminal handler rather than being explicitly known about by Dill. As a consequence, if you are building a TUI application and you don't want to have misaligned output, you need to build it directly using %blits and suppress %slogs within your app. (See tui-toys for an example in %snek.)

2. Other output goes via Dill because the terminal vane explicitly needs to know about its position. This are %blits.

   • A program (like %snek) can specify to manually output a %blit like %klr (styled text) or %put (plain text) at the cursor.
     • /lib/etui offers some interesting demonstrations in this vein; see the ++zo engine.
   • vere/io/term.c:_term_io_kick() applies effects sent to term.c, including blits. It was registerd as the effect handler when the I/O drivers were registered.
   • vere/io/term.c:_term_ef_blit() switches on the type of blit (notice it can also track a cursor position).
   • vere/io/term.c:_term_it_show_tour() emits UTF-32 to the cursor location.
   • Finally, vere/io/term.c:_term_it_show_line() prints at the actual cursor position.

3. The runtime terminal manager automatically handles aspects of layout such as maintaining the input line at the bottom of the screen. See vere/io/term.c:_term_it_restore_line() for details. (CSI = Control Sequence Introducer sequence) A different terminal handler (like %webterm) may handle these decisions differently.

Other output may follow yet a different path; for instance, u3l_log() directly prints using vsnprintf()to stderr. Some output goes via the libuv event loop, such as u3_ptty_init()..

• Examine u3_term_log_init() in vere/io/term.c.
• Examine _term_it_show_line() in the same file.

Dojo, %hood, %helm, %drum

Dojo is Urbit's primary CLI interface, and while it is too complicated to delve deeply into here, the major parts to consider include:

1. Hoon parser. Real-time parsing of input, which evaluates Hoon code for syntactical correctness. (This is the reason that typing at the Dojo prompt is frequently slower than typing at other CLIs.)
2. Specialized syntax. Besides Hoon input, generators and pokes can be set up and invoked directly from the Dojo prompt. %say and %ask generators are head-tagged pairs with gates following that return sole-results.
3. +generator prefixes cause Dojo to look in /gen for a particular generator file.
4. |generator is a Hood generator, on which more in a moment.
5. +desk!generator invokes a generator on a particular desk.
6. :agent|generator takes the output from a generator (at /gen/agent/generator) and feeds it as a noun to the agent's ++on-poke arm.
7. -thread invokes a thread, with a similar desk-specific prefix as above.
8. Hood. Most of the interesting Urbit instrumentation is provided to Dojo by the Hood/Helm agent pipeline.
9. %hood is the overarching system app, to which Dojo redirects generator invocations prefixed with | bar such as |pass and |install. (Thus, |pass is in fact shorthand for :hood|pass.)
10. %helm provides the interface for kernel and system functionality, such as |verb, |moon, etc. Hood calls into Helm.
11. %drum manages the active CLI apps (|link, Ctrl+x).
12. %kiln instruments filesystem operations using Clay.

The overall call web of these is surprisingly tangled; as an example, let's trace |link, which tells Dojo to register a CLI interface to a Gall agent.

• /gen/hood/link
• /app/hood
• /lib/drum→++poke w/ %drum-link
• /lib/drum→++poke-link
• /lib/drum→++se-link
• eel (what is this?)

Exercises

• Trace the entire lifecycle of |pass [%d %text "Hello Mars!"]. Include a function-by-function annotation and commentary.
• Write a basic vane, /sys/vane/. ~& on receiving a task.