The boundary · L4 and L3

The network

/net is the network, served by netd, a task that runs sysn's stack: Ethernet, ARP, IPv4, ICMP echo, UDP and TCP, as plain sysp, every obligation proven. Beneath it the network card's driver passes frames to and from a virtio network device, on QEMU's user network. cat /net/eth0/arp shows the neighbours the stack has heard from (sysk 052; 053: the card's own directory), and ping 10.0.2.2 talks to one (sysk 054), a program lent only its conversation (sysk 058). tell 10.0.2.2!8080 says a line to a host's TCP port and prints what comes back (sysn 012).

Four tasks in a line

The shell asks netd over the files protocol, as it asks the disk's server and the entropy driver: a request and a reply of 576 bytes each (Files). netd asks the entropy driver too, once, for the stack's key (below). netd and the driver pass frames over a pair of streams of their own, typed channels the deadlock model sees, rather than files: a frame is up to 1,514 bytes, more than a file's payload (docs/directions.md). Pick a moment to follow it.

A card and its files

The card has a directory, eth0, and a second name for it, eth: the card, whichever it is, so a script that wants the network need not know its number. With a second card, eth1 would be the next directory, and eth still the first (sysk 053). The walk follows names one at a time, so both lead to the same node, and a stat or a read says the same through either. Its three files say how the card is. Beside it are the protocols, icmp and tcp, each a directory of conversations, as Plan 9's /net has them (below; under the card until sysn 012, though a conversation is the host's, not the card's). Each file is rendered when it is read, as one s-expression a line, so the shell and a host lay them out as values (syst). Nothing of the card's can be written: changing the address or flushing the neighbours is control, which waits for the operator profile (docs/authority.md). These are a boot test, after a datagram from the host:

/ % ls /net
eth/   eth0/  icmp/  tcp/
/ % ls /net/eth0
addr   arp    stats
/ % cat /net/eth0/addr
(addr (mac 52:54:00:12:34:56) (ip 10.0.2.15) (mask 255.255.255.0) (gateway 10.0.2.2))
/ % cat /net/eth/arp
(neighbour (ip 10.0.2.2) (mac 52:55:0a:00:02:02))
/ % cat /net/eth0/stats
(taken (arp 1) (icmp 0) (udp 1) (tcp 0))
(dropped (short 0) (malformed 0) (not-ours 0) (protocol 0) (bad-sum 0) (fragment 0) (full 0) (window 0) (throttled 0))

The address is QEMU's guest's, 10.0.2.15 on 10.0.2.0/24, with the gateway at 10.0.2.2; the hardware address is the card's own, read from its configuration. At the start netd asks for the gateway's address, so it is a neighbour before anything needs it. The counts are the stack's, by protocol and by the reason a frame was dropped. Without a card, the tree is there and addr says (addr none).

Where /net is, the shell learns from the manifest: its profile mounts netd's tree there, ("/net" netd), and tools/wire.rkt makes the two streams between them, netd-ask-shell and netd-reply-shell (sysk 062; Files). netd takes the requests of every holder of its tree in turn, each with fids of its own, through the kernel's Tree (sysk 063, Several clients, one server). A read that waits for an echo waits inside the request it answers, so another holder's next request waits for it too, at most until its time is up. A holder whose mount names a directory below netd's root is answered no-such-name on attach: its tree has no directory worth mounting alone (sysk 064).

Neither waits to send

netd sends frames to the driver, and the driver sends frames to netd. If each could block sending to the other, both full, the two would wait forever. Neither does: each waits with alt for whichever comes first, a frame from the other, room to send its own, or (the driver) the card. A frame that cannot go yet waits in its task:

The deadlock model proves every run can still finish. Its first verdict was a cycle: at the end, the driver drained frames-out while it still owed frames-in. The fix made the driver plainer: once it stops taking frames from the card, for any reason, it closes frames-in first, then drains frames-out. And netd's loop ends where its holders' requests end, the shell's in the default system, as the model reads a server's (sysk 047, 063), so the shell's view of /net's replies does not carry their end through every step. A third server still costs the shell's configurations a third more, 17,078 from 11,824: every call of ask has one more way to go.

The card

A virtio network device has two queues: 0 receives, 1 sends (virtio 1.2 §5.1). Every frame either way follows a 12-byte header, all zero here: no checksums offloaded, no segments merged. The driver lends the card one buffer of 1,526 bytes at a time to receive into, and lends each frame to send as one part the card reads. Both go through the generic queue of sysk 042, so the kernel still knows no device's requests; what it gives a device stays proven and private.

One thing was new: a buffer the device may never give back. The disk and the entropy device return every request; a receive buffer comes back only when a frame arrives. So the driver could not finish, holding a buffer only the device could return. sysk 052 added one function to the kernel's device file, virtio-reset: it stops the device and waits until it reads as reset, when the device has let go of every queue, and the buffer is the driver's again (Storage). That is the only change in kernel/devices.sysp; the measure sysk 042 set was "no line changed", so it was met but for this.

The stack, from another project

sysn's files are compiled into sysk's program when a file of it calls them: tools/wire.rkt --uses lists ../sysn/net/ as it lists lib/, with the files of other parts it calls: sysc's SipHash (sysn 003), and sysl's bytes (sysn 007) and text, for the frame decoder (sysn 015), which sysk's kernel fields and lib/'s lines are written with too (sysk 085: --uses counts the kernel's own calls). The stack allocates nothing: netd lends it 432 bytes of its heap, 192 for its neighbours, counts, ports and key and 120 for each of two TCP conversations (sysn 002, 003), and gives it each frame as a view of a channel's message. It answers ARP requests for 10.0.2.15 and echo requests, carries TCP conversations, and counts what it takes. It holds no packets: a send whose next hop is not known yet writes an ARP request instead, and says so.

Conversations, and ping

Talking to another host is a conversation, in Plan 9's shape (sysk 054). icmp, beside the card, holds clone and up to four conversations (three from sysn 021, which gave a slot to dns, until sysn 024 widened the fids). Opening clone to write takes one not in use, made new, and the fid is that conversation's ctl from then on; reading it says which. A conversation is in use while a fid names one of its files, the shell's or a job's, or while it is lent (sysk 058); only one in use is listed, or can be named. So it ends when its holders let go: nothing to hang up, nothing left behind, and no stale count to read.

ping is a program of the catalogue (sysk 058; the shell's own command before). Naming the peer is granting it: when a program needs to talk, (needs (talk icmp host)) (sysk 059), the shell reads its word as an address and dials it in the need's protocol, as Plan 9's dial does, opening clone and writing connect, and then lends the job only the conversation's data, to read and write. The job cannot write ctl, so it cannot say where its echoes go (docs/authority.md, Talk). Nor can it if the shell were confused: netd caps the lend by the lease's stamp, the program's need, so a job may write only a conversation's data, and only if its stamp says write and icmp; anything else lent is read-only to it (Within its need). It writes an echo, reads the reply with a read that waits up to a second, and paces the next by its clock, a second after the first. A boot test, on QEMU's user network:

/ % ping 10.0.2.2
(echo (from 10.0.2.2) (seq 0) (ttl 255) (us 2029))
(echo (from 10.0.2.2) (seq 1) (ttl 255) (us 1035))
(echo (from 10.0.2.2) (seq 2) (ttl 255) (us 1581))
(ping (sent 3) (got 3))
job ended
/ % ping 10.0.2.99 --times 1
(lost (seq 0))
(ping (sent 1) (got 0))
job ended
/ % ping 300.1.1.1
not an address: 300.1.1.1
/ % explain (ping 10.0.2.2 (times 2))
(task (program ping) (ask talk-ask) (answer talk-reply)
  (out out) (times 2))
(lends (data (dial 10.0.2.2)) (rights read write))
(verdict passed)

The round trip is netd's to measure: it reads its clock when the echo goes and when the reply comes, so the time is the network's and the stack's, not the job's. A reply is matched by the conversation's identifier and the last echo's sequence number, and kept once. No host answers 10.0.2.99: the stack asks for its hardware address instead of sending the echo, as it holds no packets, and the read's second passes.

netd serves jobs

netd serves the pool's talk service as fsd serves files (sysk 031): when a job that binds it starts, netd opens a lease and serves the job's requests beside the shell's, and frames, until the job's requests end; then the job's fids, and the lend, are gone, and the conversation with them. It never waits on the job but through alt. Which service a job binds is where the thing lent is: the namespace says (/net 7 (jobs 5 6)), so wc /net/eth/addr is a job of netd's too, and wc /notes/x one of fsd's (Lending). Programs are clients of a service of their message size, not of one by name.

A read that waits waits inside netd's answer to it: netd takes the request, then goes on taking frames, and sending them, until the reply comes or its clock says the time is up, and then answers. The other client's next request waits that second at most. Its first version held the read and answered later from its main loop; the deadlock model saw a reply that could come when the shell was waiting on another server, and rejected it. Answered in the request's own branch, every reply follows its request, as the model reads a server.

TCP, and tell

tcp is the second protocol with conversations (sysn 012), in the same shape as icmp: clone, and up to four conversations, each a directory of ctl, data, status, listen (sysn 013) and last (sysn 015). With dns (below) the four share one table of eleven slots, 0 to 3 icmp's, 4 to 7 tcp's, 8 dns's and 9 and 10 udp's (below), each named in its protocol's directory by its place there, so a node's number says which conversation it is and which of its files, with no lookup. Two of the four carry bytes at once, a pair of buffers each; one that only announces a port holds none:

sysn's TCP keeps sequence numbers and timers, and nothing else: no bytes, no clock, no allocation (sysn: TCP). So netd keeps the rest, for each tcp slot that carries bytes: the bytes written and not yet acknowledged, in a queue of 2,048, which it lends the stack from the first unacknowledged byte whenever a segment may go; the bytes that came and are not yet read, in a buffer of 2,048, whose room is the window the stack advertises; and a few fields, the stack's conversation, the host, what it owes the stack, how it ended. A read that empties the buffer lets the stack say the window has opened. Every segment goes out one at a time through the one frame that waits for frames-out (net-pump), and netd wants its clock in alt only while some conversation has a deadline, so an idle netd waits on nothing but its clients and the card.

When nothing names a tcp conversation any more, no fid and no lend, netd closes it: the host still gets every byte queued, then the FIN, and what it sends after is let go as it comes. The slot is free again, and its buffer pair, once the stack says the conversation ended, so a clone never takes a slot whose last conversation is still closing. hangup is the abrupt way, a reset.

tell is a program of the catalogue, as ping is, with (needs (talk tcp host)). The shell reads its word as an address with a port, 10.0.2.2!8080 as Plan 9 writes it, dials it in /net/tcp, and lends the job the conversation's data. The job writes (hello sysk), then reads, each read waiting up to two seconds, and prints what comes a line at a time, until 0 bytes (the host closed) or busy (quiet for two seconds). A job may write a tcp conversation's data only if its stamp says write and tcp, as an icmp one needs icmp. A boot test, with a server on the host that answers two lines and closes:

/ % tell 10.0.2.2!8080
(hello back)
(and more)
(tell (sent 13) (got 24))
job ended
/ % ls /net/tcp
clone

After the job, the conversation is closed and its slot listed no more. netd's heap grew to 32,768 bytes, from 12,288, for the stack's conversations and the buffers. Why two pairs: the stack advertises its window in whole segments (RFC 9293's receiver side of silly window avoidance), so a receive buffer smaller than slirp's segment of 1,460 bytes would advertise a window of 0, and nothing could come in; and each of the two tables, tq and tr, is one allocation, at most the 4,096 bytes the kernel's alloc gives. So two pairs of 2,048 bytes each way, and the stack is lent a conversation for each. A slot takes a pair when it is connected or a call comes to it, and lets it go with the slot. More pairs want a table each (a followup).

Listening

A conversation takes calls as Plan 9's does (sysn 013). A holder writes announce 7777 to its ctl, and sysn's stack answers a SYN to 7777 with a cookie, keeping nothing. When the cookie comes back, the stack opens a conversation. netd sees it open passively, gives it a free slot and buffer pair, and keeps it waiting on the announcer, established, as a socket waits in a listen queue. A read of the announcer's listen says how many calls wait, and a read that waits waits for one. Opening listen to write takes the first: the fid is the call's ctl from then on, as an open of clone is a new conversation's. A call with no free slot or pair is reset. When nothing names the announcer any more, the port is withdrawn and the calls still waiting are reset, as Plan 9 closes them with the announcer.

Plan 9's open of listen sleeps until a call comes. sysk's files protocol has reads that wait and no opens that wait, so the wait is a read and the open only takes. A read of listen waits as a read of data does, inside netd's answer to it: until it ends, netd takes no other holder's request. A server that holds a waiting read aside, one per holder, as fsd does, comes before the network shell (docs/directions.md). Only a holder opens listen: a job never takes a call, whatever it was lent.

At the console, a talk need's word may be *!PORT, Plan 9's way of writing any address on a port. Then the shell answers instead of dialling, as Plan 9's listen1 runs a command for a call, but for one call. It clones, announces, reads listen a second's wait at a time until a call waits (or answers busy after a minute), takes the call, and lends the job its data. Then it lets go of the announcer, so the port is withdrawn. The job holds only the call's data, as a dialled job does. *! means nothing anywhere else, and the protocols that have no ports refuse it. A boot test, with QEMU forwarding a host port to 7777 and a client on the host that connects until a call holds:

/ % tell *!7777
(hello back)
(and more)
(tell (sent 13) (got 24))
job ended
/ % ls /net/tcp
clone
/ % tell *!99999
not an address: *!99999
/ % ping *!7
not an address: *!7

The stack's key

TCP's initial sequence numbers (RFC 6528) and its SYN cookies are SipHash under a 16-byte key, which the stack takes from its caller and keeps secret. netd reads it from the entropy driver, once, before it serves anything: a profile of its own, (profile keyed ("/" rng)), mounts only the rng's tree, so netd can read random bytes and nothing else of the system. It attaches, walks to rng, reads 16 bytes and lets go, ending its requests, so the model sees one client that finishes. If it gets fewer than 16, netd lends the stack no conversations, never a fixed or zero key: every SYN is answered with a reset, and ICMP and UDP carry on.

Names, and host

dns is the third protocol with conversations (sysn 021), in the same shape again: clone, and one conversation, slot 8, a directory of ctl, data, status and last. It runs sysn's resolver (sysn: DNS) against QEMU's, 10.0.2.3. Plan 9's /net/dns is one file, whose query lives with the open file; netd keeps nothing per fid but its node and two flags, so the query lives with a conversation, and a job is lent one conversation's data, as for ping. One conversation for now. Until sysn 024 a fid was one byte, node × 4 and the flags, so nodes stopped at 63 and the slots at eight, and icmp gave one up for dns; a fid is two bytes now, nodes may go to 16,383, and icmp has its fourth again. A second clone while it is held is busy.

A write to data takes a query id and a source port from the stack's key (RFC 5452: both unpredictable off the path) and sends the query. The server's hardware address is not known at first, so the first query's frame is an ARP request; the conversation owes the query, and sends it from take-frame as soon as the reply teaches the address, in fout, which is free then. Only a datagram from port 53 to the query's port, with its id and its question, is taken as the answer; any other is let go and the read waits on, so a forged answer cannot end the wait. The wait is icmp's: net-ready and net-waited, in the same alt as the tree and the clock, no new source. Without the key, no id can be picked, and a write is refused, io.

host is a program of the catalogue with (needs (talk dns name)). A talk need's last word says how the shell reads the program's word: host, an address, as for ping and tell; name, a host's name, letters, digits, hyphens and dots, a letter or digit first. The shell dials it as before, writing connect and the word to ctl, and lends the job the data; host asks once and waits two seconds. A job may write a dns conversation's data only if its stamp says write and dns, the talk service's third right. At the console:

/ % host example.com
(addr (ip 172.66.147.243) (ttl 275))
job ended
/ % ls /net/dns
(entry (name "clone") (type file) (format sexp)
  (rights read write) (length 0))
/ % host a_b
not a name: a_b
/ % host nosuch.invalid
(failed (rcode 3))
job ended

A program that talks to a host, ping or tell, may be given the host's name in place of its address (sysn 022): tell example.com!80, ping example.com. The shell asks netd's connection server first, as Plan 9's dial asks /net/cs (sysn 025; in 022 the shell asked /net/dns itself): it writes the need's protocol and the word, tcp!example.com!80, to /net/cs, reads the answer with a wait of two seconds, (dial /net/tcp/clone 104.20.23.154!80 (from dns)), and dials that address exactly as it dials 10.0.2.2!8080. netd asks with a query of its own, not a dns conversation's, so host and a dial by name never take one slot from each other. (from dns) says the address is DNS's, which nothing authenticates. The system's own names come first (sysn 026): the manifest says what they mean, (names (gateway (ip 10.0.2.2))), netd copies them at start, and cs answers one of them at once, (from table), with no query sent; names compare without case. So tell gateway!7777 reaches QEMU's host with no DNS at all. Then the cache (sysn 027): what DNS answered cs before, kept for the answer's TTL, a day at most, by the time netd hands in (the stack reads no clock), six names of up to 64 bytes (eight until sysn 029's udp slots), so the whole table stays in one heap block of 4,096 bytes. A name in it is answered at once, (from cache), as unauthenticated as DNS, which said it; a name dialed twice asks DNS once. That is the order sysn's note on names sets: table, cache, DNS. Only a holder may write /net/cs, and no job is lent it. The program's need and its stamp are as they were: it is lent a tcp conversation's data and never sees the name. An address is dialed as it is, never looked up; a name the resolver has no address for is said, and nothing is dialed:

/ % tell localhost!7781
(hello back)
(tell (sent 13) (got 13))
job ended
/ % tell nosuch.invalid!80
no address for: nosuch.invalid!80
/ % ping example.com --times 1
(echo (from 172.66.147.243) (seq 0) (ttl 255) (us 6918))
(ping (sent 1) (got 1))
job ended

localhost is the boot test's name: slirp asks the host's resolver, which says 127.0.0.1, and slirp connects to that on the host, so the test needs no network beyond the machine.

Datagrams, and datagram

udp is the fourth protocol with conversations (sysn 029), in Plan 9's shape: clone, and two conversations, slots 9 and 10, each a directory of ctl, data, status and last. connect A.B.C.D!PORT to ctl says the host; a write to data sends one datagram from a port picked under the stack's key (RFC 6056); a read is the next datagram that came from the host's port to ours, once, up to 512 bytes, and a read that waits waits for one, then gives 0 bytes. One datagram waits at a time: another that comes before it is read is dropped and counted in status, as UDP may drop any. UDP keeps nothing to send again, so while the host's hardware address is being asked for, the ARP request goes and the write is busy, to be written again. A conversation's state lives after the tcp slots' in ts, which every function that serves them already holds, so no signature grew; netd's heap grew from 32 to 36 KB for it.

A datagram for a port no conversation holds, and not from a DNS server, is answered with ICMP's port unreachable (sysn 009), at most one each tenth of a second (RFC 1812 4.3.2.8), so a stream of datagrams to closed ports cannot make netd a reflector. datagram is a program of the catalogue with (datagram (quiet 2000) (needs (talk udp host))): the shell dials /net/udp for it as it dials tcp for tell, and lends it the data; it sends (hello sysk), prints each datagram that comes back, and ends after its quiet time with nothing new. A job may write a udp conversation's data only if its stamp says write and udp, the talk service's fourth right. Against a UDP echo on the host:

/ % datagram 10.0.2.2!7790
(echo one)
(echo two)
(datagram (sent 1) (got 2))
job ended
/ % datagram 10.0.2.2!7791 --quiet 1000
(datagram (sent 1) (got 0))
job ended

An ICMP error about something a conversation sent is kept by the conversation (sysn 031): the stack matches it to the datagram or segment it quotes (sysn 009) and netd to the udp or tcp slot, and the slot's status ends with the last, (error (unreachable port)), (unreachable host), (time-exceeded), or the type and code as numbers. A udp read that waits ends at once when one comes, with 0 bytes, so datagram to a host that cannot be reached ends without waiting out its quiet; TCP goes on, as RFC 5927 asks, its error kept for status. slirp answers a datagram it cannot send on, to 0.0.0.1 for one, with an unreachable, and datagram ends within a second rather than after its twenty; the conversation, and its status, go with the job:

/ % datagram 0.0.0.1!9 --quiet 20000
(datagram (sent 1) (got 0))
job ended

A conversation's last frame

Each conversation, icmp's, tcp's and dns's, has a file last (sysn 015): the first 160 bytes of the last frame that came for it, as they came, for sysn/frame to decode (sysn: frames). An echo reply the conversation keeps is its last frame, and so is every segment for a tcp slot, and the answer a dns slot takes, whose DNS message frame says too (sysn 023). 160 bytes hold Ethernet's header, the longest IPv4 header and the longest TCP header, with a few bytes of what they carry. netd keeps them in the conversations' table it already had, grown from 320 bytes to 1,664, so no function gained a parameter, and opening clone forgets the slot's last frame, so a new conversation never shows its predecessor's.

Only frames the conversation already receives: a holder sees its own traffic, nobody else's, so last needs no right beyond holding the conversation, as status does. Every frame of the card is capture, which docs/authority.md keeps for an operator, recorded in /sys/audit, when those are built. A conversation lives while it is held, and the shell runs one job at a time and netd serves one lease, so no job reads last yet: while a ping runs, sysn/frame /net/icmp/0/last is refused as busy, and after it the file is gone with the conversation. It waits for jobs side by side; the shell's ls sees it meanwhile:

/ % ping 10.0.2.2 --times 3 &
/ % ls /net/icmp/0
(entry (name "ctl") (type file) (format sexp)
  (rights read write) (length 0))
(entry (name "data") (type file) (format sexp)
  (rights read write) (length 0))
(entry (name "status") (type file) (format sexp) (rights read)
  (length 0))
(entry (name "last") (type file) (format bytes) (rights read)
  (length 0))

What comes next

Capture of the card's frames, an operator's, once sysk builds operator and /sys/audit. A waiting read held aside, so a holder's wait for a call or for bytes delays no other. Then the network shell, commands from outside, which needs the profiles and declared needs first (docs/authority.md).