Racket Beyond Languages

Chris Krycho, a person I really respect, is learning Racket to build programming languages. Racket is generally slotted as a language to build languages. The popular books focus on Racket innovations related to constructing Domain Specific Languages. This include hygienic macros, the Racket loading and evaluation phases and the module system. While it is uniquely suited for creating languages ¹, Racket is also a research vehicle for a large body of programming language research. I wanted to call out other powerful abstractions that Racket offers, most of which aren’t present in main stream languages. This is a survey post, so I haven’t spent time showing examples ². However, I’ve tried to link to other people that demonstrates the power of these abstractions with some examples. This focuses only on features available in the Racket runtime or standard library, not third-party packages.

Note that while I’ve been programming for a long time, I’ve only messed around with Racket for about a year. I am likely still in the honeymoon phase. However, I still think I’ve seen enough cool things to add another voice shouting from the rooftops.

Delimited Continuations

Continuations are the most brain-rewiring of the things discussed in this post. Out of all of Racket’s (in this case, Scheme’s) cool things, continuations are probably the ones that have seen the most percolation into other languages, but few have implemented them to the extent Scheme has. Those other languages usually ship a restricted version of continuations to implement async/await, exception handling or coroutines. Continuations are a very fundamental programming primitive, and have been around in some form or the other for decades. LISP and Scheme have had unbounded call/cc. Over time, the continuations space has been segmented by power and implementation detail. Delimited continuations were introduced in the late 80s, and since one of Racket’s creators was widely involved in that effort, it is no surprise that Racket has always supported, and been a research vehicle, for the full exploration of continuations. In my opinion, continuations are mind-bending enough that their use should be limited to very specific abstractions that hide away most of the raw power, and present a more reasonable (generally one-shot and delimited) surface. However, since Racket does have the fullest form of them, it is perfect to play around with them and really understand how they work.

Alexis King’s keynote is the most intuitive explanation of continuations as “the rest of the computation”. However, for someone like me, with more of a systems (than PLT) background, I find it easier to visualize them as keeping copies of stack frames, that can replace or compose with the current stack frame. Additional great resources for those not steeped in PLT or Scheme:

• What’s in a continuation
• Continuations, coroutines, fibers, effects
• Revisiting Coroutines

The Racket web framework uses continuations to great effect to allow a very readable coding style in the presence of form submissions from the browser.³ Note that effect handlers, a topic that has had some resurgence recently, is closely related to delimited continuations.

ConcurrentML inspired concurrency

Racket technically has one kind of concurrency and two kinds of parallelism⁴. The concurrency primitives are inspired by ConcurrentML (CML). The closest analogy in mainstream languages is Go-style channels and coroutines with green threads. What CML offers over Go is that channels, and the operations to act on them are values and functions, including sync, which is the select analog. This means events can be composed and manipulated up and down the function call graph, allowing one to build comprehensive trees of concurrency. At the same time, actions that care only about specific events can act on them by using constructs like handle-evt or wrap-evt. Beyond allowing this kind of composability for concurrent operations, CML also introduces first class cancellation with nack-guard-evt.

The prolific Andy Wingo has a series of posts introducing and comparing CML to other concurrency mechanisms:

• An incomplete history of language facilities for concurrency
• Concurrent ML vs Go
• Is Go an acceptable CML?
• A new ConcurrentML

Racket builds on the CML foundation by allowing nearly all of its primitive operations to be treated as events. File and socket ports (for non-blocking I/O), channels and buffered channels, as well as threads themselves can act as CML events. Your own custom structures can also act as CML events. In addition, even C libraries can hook into this using unsafe-poller, unsafe-poll-fd and friends.

The Racket Reference very comprehensively documents the individual CML primitives and how they inter-op with other Racket values, however it does lack a good tutorial that brings all the concepts together. The closest thing is probably Flatt and Findler’s paper Kill-Safe Synchronization Abstractions. It demonstrates the power of CML and custodians (see below) to build several abstractions in userspace that would generally need to be in the kernel or runtime in other languages. Bogdan Popa’s racket-resource-pool is also worth reading. It makes use of nack-guard-evt to remove requests from the queue if the caller goes away.

Resource Control

One of the interesting realizations from reading the Racket meta-literature is how many of these concepts were added to Racket to solve the core team’s aims of making it easier to teach programming to young students. Racket/PLTscheme are nearly 30 years old at this point, and for that entire time, the team has focused on pedagogy a lot. The seminal textbook How to Design Programs uses Racket and builds up from basic languages to more advanced languages. A lot of the resource control mechanisms below evolved because the developers wanted to offer restricted execution environments for teaching. That way, students couldn’t mess up too much when they were just beginning to learn programming, such as accidentally exiting the REPL or similar. However these mechanisms also have applicability in creating secure, production grade programs. Similarly, a lot of Racket’s introspection and reflection mechanisms have arisen to support teaching. DrRacket’s built in debugger, stepper and macro stepper are powerful educational tools, and I recommend using them even if your regular editor is something else ⁵.

Custodians

Custodians are a mechanism to ensure clean shutdown of system resources. Unlike RAII or Python context managers, where every object up the stack should manage its resources, custodians do not require anything but the lowest Racket primitives to register themselves with the runtime. Custodians can also limit the memory used by code running within them.

Custodians are really powerful for isolating parts of the program, and then having confidence those parts don’t leak any resources on exit. The most common pattern is when running any user-provided or user-started code. For example, the Racket More tutorial has a section where they demonstrate handling every web server request in a custodian. Using that and a timer thread allows the server to reliably terminate malicious clients that are holding a connection open for too long.

While new fangled structured concurrency isn’t a part of Racket, I strongly suspect that CML and custodians offer enough tools to build it out.

Security Guards and Sandboxed Evaluation

Security guards allow gating which files and network addresses a thread may access. Sandboxed evaluators allow restricting what arbitrary code can do.

Contract system

Contracts certainly predate Racket and are now present in a few different languages (notably D, Clojure, Kotlin and Ada/SPARK), however Racket is probably the one that has gone all-in on them, with pervasive use throughout the standard library and wider ecosystem. Racket contracts can apply to values beyond just function preconditions and postconditions. They are also independent from the value they are imposed on, in the sense that the implementor can choose whether contracts should always apply, or only apply at the module boundary. Contracts build upon chaperones and impersonators, a powerful proxy mechanism in Racket. This wrapping of values allows contracts to travel with values, such that their invariants are never broken.

Typed Racket

Typed Racket is the next level up from contracts. A statically typed dialect of Racket, Typed Racket is cool in the sense that a lot of its checks can degenerate to runtime contracts when a typed module is used from untyped Racket code. It was also one of the pioneers of occurrence typing. Typed and untyped code can seamlessly interop, even within the same module. The compiler can leverage types to generate optimal code for numeric operations. Typed Racket also has experimental support for dependent types in certain contexts. This sort of incremental yet powerful leveraging of types is pretty rare in the PL world. TypeScript is the only other one I know.

wrap-evting up

Racket has been playing with a bunch of interesting programming abstractions for years. I am not saying everyone should adopt Racket in place of Python, but it wouldn’t hurt to pay it some attention, read the excellent docs, and bring some of these ideas to your favorite language.

The Racket Manifesto is a good watch and read to understand how the Racket team thinks of the world and brings that world into Racket. It also has lots of references to inspiring papers.

Even though Racket is somewhat widely taught in US higher education, its near total absence from industry and industrial technology discussion is indicative of the academia/industry gulf in software engineering. Here is to bridging that a little!