Allthoughts

Shared Memory Consistency From Scratch Part 1: Causality

2026-09-05T00:00:00+01:00

Hardware and programming language memory models are rightly seen as difficult subjects, yet the confusion that surrounds them is a side effect of poor understanding, even amongst experts, and not part of the intrinsic difficulty. While I can't invent the universe to explain shared memory consistency from scratch, I can design a novel computer architecture, together with a set of atomic instruction, to highlight the core problems and how to overcome them. The architecture will provide fewer ordering guarantees than anything that exists today, and serve as a proving ground for various abstract memory models, including the now ubiquitous C++ memory model. Along the way, I'll introduce new semantics to facilitate efficient synchronization and reasoning about correctness, as well as discuss how to fix problems with the C++ model.

Outline
Memory Models
Write Atomicity

3.1 Local Reasoning and Synchronization Asymmetry
Hardware Wish List
Message Passing
Mutual Exclusion
Repairing Write Causality

7.1 External Cumulativity

7.2 Internal Cumulativity

7.3 Error Detection
Asynchronous Cumulativity

8.1 Asynchronous Internal Cumulativity

8.2 Asynchronous External Cumulativity

8.3 Partially-Ordered Interconnect
Commit-Reconcile Semantics

9.1 Store Buffering

9.2 Reconcile Loads

9.3 Delegated Store Buffering
Read Causality
Coherence Causality

11.1 External Coherence Cumulativity

11.2 Internal Coherence Cumulativity
Sequential Consistency

12.1 Total Order

12.2 Causal Transitivity

12.3 Global Commit

12.4 Implementation Optimization

12.5 Shattering the Illusion

12.6 The Fundamental Problem
Special Loads

13.1 Consume Semantics

13.2 Verify Semantics
Prepare Semantics
Same-Location Causality

15.1 Load-Load Bugs
AI Disclosure
Conclusion

1. Outline

This article will cover the basic building blocks of the hardware, and the problems they cause when attempting to synchronize access to memory. I will also introduce the specialized memory operations we'll need to tame the chaos. Part 2 will introduce a novel framework for synchronizing data, then cover read-modify-write operations, local dependencies, fences, compiler mappings, and a summarized description of the machine, alongside its associated informal memory model. The remaining articles in the series will be dedicated to reasoning about correctness and exploring how formal models are synthesized, and where things tend to go wrong, particularly in the context of the C++11 and 20 memory models.

2. Memory Models

A shared memory consistency model defines the rules specifying how updates to shared data may be observed by different participants. In hardware, these rules govern both access to the same physical memory location, as well as management of cached copies (replicas) of different locations. From here on, hardware and software consistency models will be referred to simply as memory models.

Replication improves data locality, reduces read latency, and increases read throughput, but unlike distributed systems, hardware architectures do not, generally, use replication for fault tolerance, so availability is not a concern for memory models.

We could, in principle, provide exclusive access (serialization) to every memory location (a byte for byte-addressable memory) to only one participant - a processor - at a time, but that would ruin the throughput of the system. A simple fix is to split RAM into multiple granules, and provide a mechanism for per-granule exclusive access. We could also improve read throughput by allowing multiple processors to read from the same granule at the same time. If you're familiar with parallel programming terminology, what I'm describing is, conceptually, striped reader-writer locking for RAM. I will challenge this assumption in Part 2, but, for now, it's a useful and practical abstraction.

Once we bring caching into the picture, the reader-writer lock analogy becomes more complicated, because a read operation involves copying the granule into the reader's cache, and servicing future reads from the local copy. Therefore, in addition to acquiring exclusive access to the granule, a write operation needs to also make sure the cached copies are eventually updated. The choice of what "eventually" means will turn out to be the most important part of the hardware's design, and I'll discuss it at length later.

The size of the granule is also critical. If we make it too small, say, a byte, we'll introduce a lot of overhead in both tracking metadata and transaction frequency. If we make it too big, we'll increase latency due to large data transfers, and reduce concurrency by locking regions of memory the processor might not actually need, resulting in an overall loss of throughput. A conventional choice, nowadays, is to make the size of the cache line 64 bytes.

For the remainder of this article, you can assume that a granule spans multiple memory locations. The observable per-location behaviour is called coherence, while the mechanism that coordinates access and update propagation is a cache-coherence protocol. Because of the caching involved, a granule is typically called a coherence unit, or a cache line.

While coherence, often called per-location consistency, governs behaviour with respect to a single location, consistency, as a whole, also includes the observable ordering of memory operations across different locations, which involves a complex interplay between the processors, and the coherence protocol. Throughout this article, you may assume that variables used in examples reside in different cache lines, because that's when things get interesting. Unless stated otherwise, all variables used in the various examples will also be initialized to zero.

It may be tempting to assume that per-location behaviour is the same as the observed behaviour for individual cache lines. That's typically true in modern hardware, but I'd caution against making this assumption by default. I'll cover both deliberate and accidental violations at the end of this article.

If you're new to the subject, I recommend starting with Paul E. McKenney's "Memory Barriers: a Hardware View for Software Hackers". For a detailed overview of cache coherence and basic consistency, the second edition of A Primer on Memory Consistency and Cache Coherence (APMCCC) is both open access and highly regarded. If you're interested in how processors communicate at the physical level, I'd recommend Principles and Practices of Interconnection Networks as a starting point. The remainder of this article will address topics that, in my opinion, are covered poorly or are completely ignored by existing resources.

3. Write Atomicity

Suppose you've implemented some arbitration mechanism that grants access to individual cache lines. At what point is a write operation considered complete, and should you service reads before the write reaches that point (read-others'-writes-early, or simply early reads)? Although these two questions are related to coherence, as you'll soon find out, they also have an enormous impact on consistency as a whole. In fact, when it comes to shared memory synchronization, the very first thing I want to know about a new hardware architecture is how it answers these questions. The answer to the first question, at least in Part 1, will be that a write is considered complete when all processors confirm that they can observe the new data. This matches the reader-write lock abstraction, and will serve as a stepping stone for understanding the framework I'll introduce in Part 2 - it's also the reality in most modern CPUs.

Taking a conservative approach, let's say you choose to wait for all copies to be marked stale, or invalidated, before the write is considered complete, and read or write access can be granted again. Other write propagation mechanisms, such as immediate replication (update), temporal leasing (time-based self-invalidation), and various hybrids come with their own trade-offs, and will not be considered here. This property of a coherence protocol is called write atomicity (WA), store atomicity, or multi-copy atomicity (MCA). For performance reasons, it's often assumed that the processor which performed the write can service its same-location reads early, which is called read-own-write-early multiple-copy atomicity (rMCA) or other-multicopy-atomicity (oMCA). The rMCA term is popular in the academic memory models literature, while oMCA was popularised by ARM's architecture manuals:

In an Other-multi-copy atomic system, a write from an Observer, if observed by a different Observer, must be observed by all other Observers that access the location coherently. However, an Observer can observe its own writes before making them visible to other observers in the system.

Lack of multi-copy atomicity is usually called non-multi-copy atomicity (nMCA), and is becoming increasingly rare on modern architectures. Here's a classification of popular architectures:

oMCA or MCA: x86, x86-64, ARMv8-A (2017 revision), RISC-V, SPARC v9 (all models), IBM z/Architecture (since System/370, though 370 was strict MCA), DEC Alpha (with caveats)
nMCA: IBM Power, ARMv7, Itanium, NVIDIA PTX (GPU)

As you can see, the only modern nMCA architectures are Power and PTX. Memory model tests have identified nMCA behaviour in Intel's Iris Pro 650 GPU, and Intel's oneAPI GPU documentation uses language that, while not conclusive, indicates nMCA:

If the program author/compiler does not make appropriate use of fences, it is not guaranteed that all threads see the result of any given memory operation at the same time, or in any particular order with respect to updates to other memory addresses.

I couldn't find concrete information on AMD's GPUs, though it might be possible to infer the behaviour from a detailed examination of examples used in their manuals. This is a common, and concerning, trend because write atomicity is essential in selecting both correct and efficient synchronization primitives. Experienced developers are often forced to infer what's happening from non-standard (quasi-axiomatic) definitions, vague statements, an incomplete set of examples, or claims from people involved in the design. This has, historically, caused problems even for compiler developers who need to rely on expert interpretation of architecture manuals, even though the experts also struggle to get this right. The highly influential C/C++11 mappings to processors required multiple revisions (as seen in the changelog link at the top) to get things right, and, Itanium, which took an unusual approach to write atomicity, was completely broken for years after the initial mapping was proposed.

More recently, researchers have begun to compile systematic tests to determine what the hardware is actually doing, and while that's helped clarify the vagaries of manuals significantly, it still requires access to new or exotic hardware to run those tests. I'll cover those tests extensively throughout this article, particularly the Power ones, because that's the closest existing architecture to what I'm building.

3.1 Local Reasoning and Synchronization Asymmetry

So why do I claim that write atomicity is the most important property of memory models? Because all externally observable reordering of operations becomes a processor-local phenomenon, and an entire class of synchronization problems is eliminated. One of my primary motivations for working on this series was explaining the intrinsic difficulty of nMCA consistency from the ground up, without getting bogged down in the extrinsic complexity of various formal descriptions and their errors.

When writes aren't atomic, the burden of restoring atomicity, when necessary, falls on the reader, and the mechanisms for achieving that are either inefficient, or non-trivial. This asymmetry also causes a lot of problems for programming language memory models that target both oMCA and nMCA hardware. The C++11 memory model, which doesn't mandate write atomicity for the sake of compatibility and efficient implementation, has seen a wide, though sometimes unofficial, adoption, across a variety of low level programming languages including C, Rust, Odin, and any language that depends on LLVM, like Zig. Go was also inspired by C++11, though, like Java's volatile, it only provides operations that implicitly require write atomicity.

It took 6 years after the release of C++11 for researchers to find issues with its specification, and while C++20 made an attempt to repair the holes, it accidentally made instruction selection on some architectures, notably x86, horrendously inefficient, prompting Hans-J. Boehm, one of the chief architects of the model, to exclaim:

The fact that nobody noticed this for a very long time, and implementers were not bothered by it, suggests that the audience for this part of the standard is nearly empty. We conjecture that implementers actually rely on atomics mappings generated by memory model experts, who are more interested in formal models than standardese. A more formal description is likely to increase the size of the audience, and would definitely ease verification and reduce the probability of mistakes like this.

If you skim through the links I've provided, you'll find only a single paragraph related to Itanium in the "Repairing Sequential Consistency in C/C++11" paper by Lahav et al., and an insightful comment by Boehm:

This lack of "multi-copy atomicity" is also the core distinguishing property of the Power and ARM[v7] memory models.

All 3 links discuss "sequential consistency" (SC), and I suspect most people familiar with the term won't be able to see the connection with write atomicity, because prohibiting early reads should have nothing to do with the concept of a "total order of operations." I'll explain what those terms mean and correct the misconception later in the article, but before I do that, I need to introduce the virtual machine that will serve as the foundation for both understanding and deriving formal rules.

4. Hardware Wish List

Given all the, claimed but yet to be established, benefits of write atomicity, it might surprise you that the purpose of this section, and the rest of the article, is to relax that requirement. The primary reason why is because I want to develop a virtual machine architecture capable of exhibiting almost all subtleties of the C++11 memory model. However, the choice is not entirely arbitrary from an efficiency perspective either.

One of the main benefits of hierarchical caches is locality, and locality is only really relevant if you can exploit it. I'll cover how cache hierarchies interact with the memory model in Part 2.

Prohibiting early reads also means two processors which share a local cache can't exchange data until all readers, potentially very far away, have acknowledged an invalidation. The processor may also leverage SMT (Simultaneous Multithreading) or SIMT (Single Instruction, Multiple Threads) execution, and allow forwarding of yet-to-complete writes from its store buffer to reads performed by local hardware threads (store-to-load forwarding). We want to permit efficient sharing of data between hardware threads on processors that support SMT or SIMT execution, which means the store buffer (the hardware component that handles writing registers to cache lines) will not be statically or dynamically partitioned to isolate hardware threads, and cross-thread read requests can be satisfied directly from it.

Applying the invalidation could also take a long time if the cache is busy servicing other invalidations or processor requests, so it would be convenient if processors only acknowledged that they've received an invalidation, and have placed it in an invalidation buffer (IB). The reason why I don't call it an invalidation queue is because parallel application of invalidations may be desirable for large caches, which would mean that a first-in-first-out (FIFO) processing order could be inefficient to maintain by default.

Another benefit of deferred invalidation is the possibility of caches acting as coherence proxies when bridging two potentially unsynchronized coherence domains, as is the case for GPUs and some ARM devices. The cache itself may appear as a processor on the local network, while actually managing traffic on behalf of an unknown processor elsewhere on the network. In this model, a shared cache, or an I/O controller can be modelled as local processors. While the C++11 model doesn't concern itself with different coherence domains, the need for understanding how algorithms could work in such environments exists for a non-trivial percentage of developers.

In summary, the virtual machine will support early reads, invalidation buffering, and out-of-order invalidation processing. The processor is allowed to execute instructions out of order whenever data dependencies permit that, but you should always consider the difference between out-of-order execution, which is a processor-local optimization, and the reordering effects of the hardware components and network that connect different processors. I'll also make no assumptions about the topology of the fabric (interconnect) used for inter-processor communication until Part 2, not even deterministic point-to-point routing (messages sent in order can arrive out of order). This means that perfectly in-order local execution may result in visibly out-of-order behaviour. Here's a simplified diagram of a single processor:

In addition to what I mentioned already, the Line Fill Buffer (LFB) handles interconnect requests for data when the processor wants to fetch a new cache line, and the Writeback Buffer forwards modified lines to a cache, either because it got evicted, or downgraded to a clean state. The load and store buffers are the processor's interface for reading and writing data (sometimes they're described as one Load-Store unit), they handle loading data into registers, and writing register data to cache lines. The store buffer can forward data directly to the load buffer if a read is issued for a variable whose prior modification hasn't completed yet. You could think of the LFB and Writeback Buffer as the cache's equivalent of the load and store buffers respectively.

Given all these theoretical hardware benefits, you might wonder why so many popular architectures remain oMCA, or have pivoted to it like ARM. One of the reasons why, besides developer comfort, is their use of speculation. They may be just as permissive under the hood, but they'll aggressively cancel speculative computations when they detect that violations of write atomicity can become externally observable. Even though speculation can work around ordering constraints, it's not a silver bullet. Excessive coherence traffic spikes and pipeline flushing can be triggered by mis-speculation, which could result in either overall performance decline (average latency and throughput), or increased tail latency.

It's also possible to create a special set of store instructions that retain their write atomicity, while gaining additional functionality that allows them to assist in coordinating regular access to memory regions. Typically, only a small subset of memory is used for synchronization, so we'd still gain the theoretical advantages of our nMCA architecture. This is the approach Itanium took with its st.rel and read-modify-write (semaphore) instructions, and, to my knowledge, it was the first to do so. Nowadays, a lot of the decisions Itanium engineers made are seen in a negative light, but I think this particular feature of the architecture is actually quite good given the design tradeoffs. I won't be as practical as Itanium in this regard, because I want to limit test the permissiveness of the architecture.

On a side note, if we follow Adve and Gharachorloo's definition, the DEC Alpha is classified as MCA, because a write-atomic protocol "prohibits a read from returning a newly written value until all cached copies have acknowledged receipt of the invalidates or updates generated by the write (that is, until the write becomes visible to all processors)." As I'll demonstrate, a mere receipt of invalidates is not sufficient to preserve the appearance of MCA, unless additional synchronization functionality is provided by the hardware or instruction set architecture (ISA). I'm sure the authors understand this nuance, but I want to make it clear for people who're not familiar with this space of ideas.

5. Message Passing

Removing the write atomicity requirement is only acceptable if we're able to deal with the consequences. One of the backbones of any synchronization protocol is message passing, so we need to provide the tools necessary to accomplish this task.

Plain load and store instructions will be called ld and st respectively, and in pseudo-assembly syntax inspired by the Intel x86 style, the first message passing example will be:

// Processor 0 (P0) (this is a comment)
st [message], 42 // store 42 in the message variable
st [flag], 1

// Processor 1 (P1)
ld r1, [flag] // load the value of the flag variable in the r1 register
ld r2, [message]

Consider whether it's possible for P1 to see r1 = 1 and r2 = 0. Walking back from an undesirable state to a chain of events that caused it is an acquired skill, so let me go over it in detail.

If the receiver (P1) doesn't observe the new message value, then it must've had a stale copy of the cache line containing the variable (remember, all variables are in separate cache lines for these examples), and didn't apply or receive the invalidation.

However, the receiver did read the updated flag value, so it either didn't have it cached, or it processed that particular invalidation before the one for the message.

Both scenarios are interesting, but let's start with the latter. Our invalidation buffer allows out of order processing, so it's entirely possible that both messages were sitting in it, and the flag got processed first. As long as the sender side can guarantee that the message invalidation will arrive before the flag, we can simply snapshot the invalidations in the IB (mark the current head of, say, a ring buffer, and wait for the tail to catch up, or a similar mechanism) at the time of performing the flag load and wait for them to be processed before executing any subsequent loads.

Waiting for invalidations on every load is wasteful, so let's introduce a new operation, ld.rcv (rcv stands for "receive"). The most efficient way to handle the invalidation buffer is to only snapshot and stall when performing a line fill for a receive operation, because the snapshotted invalidations may complete while waiting for the requested data to arrive. A side effect of this implementation is that a regular load of the synchronization variable (flag in this case) that requires a line fill could break the visibility guarantees, because a subsequent receive load that hits the cache can no longer guarantee prior invalidations in the IB are fully processed. C++ actually forbids this outcome, which, in my opinion, is a mistake, and I'll demonstrate how C++20 also introduced the write counterpart to this behaviour in Part 2.

On the sender's side (P0), it's easy to guarantee that the invalidation for the message arrives before the invalidation for the flag if the processor waits for the message acknowledgement before issuing the flag write. However, that's also wasteful, because we might have to process many writes that are part of the message payload, and waiting for each of them to be acknowledged introduces unnecessary round-trip latency, and prevents bulk processing such as combining multiple invalidations into one packet sent across the interconnect (the receiver can implement similar optimizations). We can use the same approach as the receiver and implement a st.snd (snd for "send") operation that waits for all preceding stores to be acknowledged before making its write visible. The correct implementations is therefore:

// P0
st     [message], 42
st.snd [flag], 1

// P1
ld.rcv r1, [flag]
ld     r2, [message]

If the cache line containing the flag was already evicted for capacity or conflict reasons, the same logic still applies. We might fetch a new value directly, but we still need to snapshot the IB and stall until completion. A real implementation could also spin/loop until it sees the flag update, but I've omitted that for the sake of simplicity.

A robust cache coherence protocol must also deal with race conditions between data requests and invalidation messages. If an invalidation arrives after fresh data is supplied, it will result in a spurious invalidation, but the real correctness problem is installing stale data after an invalidation was processed.

We can take a pessimistic approach to solving the lost invalidation problem: announce that we've evicted the cache line and wait for acknowledgement before requesting new data. Such an explicit eviction scheme will guarantee that no invalidations arrive in the first place, but it introduces some unnecessary traffic if the same line is immediately requested soon after.

The optimistic approach is to perform a silent eviction (no announcement), but keep track of any invalidation messages that arrive while waiting for the response. If an invalidation address matches the address of the an outstanding read request, we can mark the register responsible for tracking the request with an "ignore data" flag, and immediately discard and re-issue the data when it arrives. This incurs significantly more unnecessary interconnect traffic compared to explicit eviction, but it only happens when a race is detected. I won't consider versioning the cache lines as a method of detecting staleness, because every bit in the cache is precious, and the metadata overhead relative to the size of each cache line would be too large.

From a protocol standpoint, explicit eviction usually results in simpler implementations, but, for our purposes, it really doesn't matter which approach is taken.

An alternative approach to consider, and I suspect this is what every major modern architecture does under the hood, is to snapshot the IB and stall data requests for every line fill. This will effectively reduce the ld.rcv into a simple read barrier (if processor-local reordering is allowed), hiding almost all reordering issues caused by invalidation buffering and out-of-order processing - speculative racy reads followed by a verification read of a synchronization variable, as is the case with sequence locks (seqlock), could still expose the presence of an invalidation buffer. Needless to say, I won't be taking this approach because my goal is to be as permissive as possible, and explore how complicated things can get on the software side.

On the subject of reordering, once we allow that, there's really no reason why ld.rcv should block subsequent writes, or wait for prior reads and writes to complete. The term used for this kind of instruction is a one-way read barrier. Similarly, st.snd can become a one-way write barrier.

The ld.rcv and st.snd instructions allow us to implement single-producer, single-consumer data structures like ring buffers, which can be used for efficient point-to-point communication between two processors. Such a ring buffer is already sufficient, though potentially inefficient, for design patterns like the Actor model, but, ideally, we should allow more than two processors to access the same data structure.

6. Mutual Exclusion

The second backbone of shared memory synchronization is mutual exclusion, or locking, as it's commonly called. A simple implementation of a lock can be done by synchronizing all accesses against a single variable. In pseudo-code, it looks like this:

lock.acquire()
// Critical section. Shared data accesses are only performed here.
lock.release()

The release operation can be as simple as storing the "unlocked" value in the lock variable:

st.snd [lock], 0 // 0 = unlocked, 1 = locked

I'm using the st.snd instruction because a write might've been performed inside the critical section, and we want the next processor which acquires the lock to be able to see it.

The acquire operation is more complicated and requires a read-modify-write (RMW) operation, which I'll cover in Part 2, but the basic idea is that we can perform a ld.rcv (a special variant of it) and check if the value is 0 (no one's holding the lock). If that's true, we can try to conditionally set the value to 1, which will only succeed if no other processor managed to sneak in and acquire the lock in between our load and the conditional store. If the lock is being held or someone else managed to get in before us, we simply retry, possibly after a small delay. The important part of this operation is the load, which makes sure we receive the latest update for the data from whoever held the lock before.

Here's the entire locking and update procedure in pseudo-assembly:

RETRY:
ld.rcv r1, [lock] // acquire the lock
// Check if r1 is 0 and try to conditionally store 1 if true.
// Go to RETRY if either fails, after a delay.

// Lock acquired. Access the shared data.
st.snd [lock], 0 // release the lock

There are more efficient lock designs and operations for acquiring the lock, but this setup is good enough for now. Can you spot any issues that could arise from this implementation? The problem now is processors executing the instructions out of order.

First, while the release operation already prevents prior stores from completing after the lock value is set, it doesn't constrain loads in the same way. Performing reads outside the critical section will provide no synchronization guarantees whatsoever.

Second, in the current example, we can't allow any store instructions to be executed before the conditional store, because that instruction can fail, and we'd end up modifying shared data without actually holding the lock. The conditional branch typically prevents such reordering from being externally observable, but reading is a bit more tricky. If we perform a read before the conditional store succeeds, we could end up reading data without any synchronization, and then succeeding anyway. Some processors, like Power, respect control (branch) dependencies when it comes to stores, but require additional ordering instructions to prevent speculative reads from completing when the branch is actually taken. In my quest to create the least programmer friendly architecture, I'll also allow speculative loads to ignore branch dependencies. More on this subject in Part 2.

Lastly, it might be tempting to leave the receive load as it is, but there are lock implementations that don't require any conditional store operations between the load and the store. One example is a simple ticket lock, where one variable is used for reservation, and a next variable is used to indicate who can enter the critical section next. Such an implementation would use an RMW operation to reserve a "ticket", and then busy wait with a receive load on the next variable. This means we need to add writes to the list of operations that can't be executed before the receiving load.

I think now is an appropriate time to introduce new load and store instructions to capture these rules. An acquire load will be performed by executing ld.acq and will behave like ld.rcv, except it will also stop subsequent writes from completing before it. In effect, all subsequent instruction processing will halt until the acquire load completes, though we don't need to wait for prior instructions to complete.

Similarly, a release store, st.rel, will behave like st.snd, but it won't allow prior loads to complete after it. Instructions that come after the release store need not wait for anything else to complete, as they're outside the critical section.

This behaviour of acquire and release is sometimes called "Roach Motel optimization", which is a reference to the slogan of a roach trap with that name: "Roaches check in, but they don't check out." The idea is that unrelated operations can move inside the critical section, but nothing inside can get out. Using our "barrier" terminology from before, both ld.acq and st.rel are now one-way read-write barriers, but in opposite directions.

Mutual exclusion, particularly when multiple parallel reads are allowed, is actually much better than people give it credit for, though that will be the subject of another article.

Without implementing some kind of complicated transaction protocol, cache-coherence can only provide unsynchronized access to one cache line worth of data, because our coherence protocol behaves like a reader-writer lock, with some important caveats I'll discuss later. Typically, architectures only provide atomic access - meaning it's safe to access in parallel - up to 4, 8, or maybe 16 bytes at most - enough bytes to pass one or two pointers around. These are the variables we'd be using for send, receiver, acquire, and release instructions.

If atomic instructions span two separate cache lines, processors usually give up and raise an exception, or enforce a heavy interconnect lockdown. x86 is an example of the latter, where instructions with the "LOCK" property (prefix) will actually cause a system-wide lock of the interconnect until the double-line transaction is complete. A userspace program can effectively perform a denial of service attack by abusing this mechanism, which is why the Linux kernel, for example, has options for detecting so-called split locks. The x86 XCHG instruction has an implicit "LOCK" prefix with memory operands, so it's easy for developers to degrade system performance without any malicious intent. Accesses that span two different cache lines without this prefix are silently allowed to tear, meaning you get only parts of what each processor wrote. This is one of many reasons why hardware architectures should have explicit atomic instructions.

If you prefix every atomic operation in x86 with "LOCK", you'll be shooting yourself in the foot in terms of performance, because that prefix will also cause the processor to stall until a snapshot of its store buffer is fully processed - and you rarely want that. For correctness reasons, processors that don't even support locking should just raise an exception when an atomic variable isn't contained in one cache line. This is not possible on x86 - you either get garbage or unnecessary performance degradation.

However, even assuming our atomic synchronization variables are correctly aligned, we're not out of problems to address yet.

7. Repairing Write Causality

We may sometimes want parallel access to data without mutual exclusion or any kind of acquire or release synchronization. This kind of access will be facilitated by relaxed atomic operations, and I'll use the .rlx suffix to denote them in code snippets. In fact, any instruction with that suffix, or a stricter version of it, can be considered atomic. These instructions will not offer any ordering or visibility guarantees for ordinary loads, stores, and even other relaxed operations. Chapter 15. Same-Location Causality will cover why we need dedicated relaxed atomic operations when the reader-writer lock coherence protocol should already provide that for regular operations.

If you've been paying attention so far, you might be wondering why early reads haven't caused any issues yet. Consider the following scenario:

// P0
st.rlx [data], 1

// P1
ld.rlx r1, [data] // reads 1
st.rlx [flag], 1

// P2
ld.rlx r2, [flag] // reads 1
ld.rlx r3, [data] // reads 0

Is it possible to get r1 = r2 = 1, and r3 = 0 at the end? The answer is yes, and I already hinted at the mechanism. Lack of write atomicity means P0's write to data might be observed by P1 before the invalidation even reaches P2. P1's invalidation of the flag could then race P0's to P2's invalidation buffer, and then, assuming P2 already had a cached copy of the data, we get a stale read for data, and a fresh read for flag. We can't even guarantee that the invalidation will be in P2's IB by the time the flag invalidation is processed, so ld.rcv and ld.acq can't prevent this outcome.

Because we didn't give the relaxed loads any ordering guarantees, this outcome is consistent with the operations in either P1 or P2 swapping places, so, while perhaps counter-intuitive, there's no issue with allowing this outcome once we allow local reordering of regular and relaxed operations.

However, the situation becomes more complicated once we introduce synchronization on the flag variable:

// P0
st.rlx [data], 1

// P1
ld.rlx r1, [data] // reads 1
st.rel [flag], 1

// P2
ld.acq r2, [flag] // reads 1
ld.rlx r3, [data] // reads 0

Now we have a problem. Even with local reordering permitted, the rules for acquire and release state that memory operations can't move past the one-way barriers, yet, unless we somehow violated causality, that's the only logical conclusion. We could either prohibit this behaviour, or document it as an exception, and, since the C++11 memory model demands the former, we'll have to figure out how to make sure P0's write is visible to P2 by the time it reads the updated flag.

The simplest, and most efficient, fix in terms of hardware complexity is to make all atomic operations write-atomic. This was Itanium's approach, and chapter 13.2.1.12, "Obeying Causality," of the Intel® IA-64 Architecture Software Developer's Manual Volume 2: IA-64 System Architecture Revision 1.1 uses the same example. However, since the C++ memory model doesn't mandate this, I find it interesting to explore how we can prohibit the undesirable outcome with minimal additional complexity.

7.1 External Cumulativity

A straightforward way to prohibit this outcome is to fix the problem at the source. If P1 is aware that one of its loads has observed an early store, it can stall on the release operation until the observed store is globally visible - it can't know the future, so it needs to be conservative. The processor which produced that store is naturally aware of when the last invalidation acknowledgement arrives, which signifies global visibility, so it can send a message to any processor it serviced early to notify it of the completion.

It would be extremely wasteful to send a notification for every store, because there's no guarantee a release operation will even be executed. Furthermore, accessing regular data like that should be classified as a race condition, so we should only do it for atomic operations. Lastly, we need to decide what to do in cases where the programmer mixes regular memory operations with atomic ones.

One way to let the processor which produced a write (producer) know that an atomic read is being serviced early, is by augmenting the system's network communication protocol with a dedicated ReadAtomic request for line fills. If it services such an atomic read request early, it can use a per-processor bitset, or a similar data structure, to flag that it needs to notify that processor of the global acknowledgement before removing the write from its pending-completion buffer (or wherever that's being tracked). If the write is already globally acknowledged, it can set a "completed" bit in its response.

On the early consumer's side (the processor which issued the atomic read), if the data for the load is supplied, and the "completed" bit isn't set, the load can be placed in a special buffer optimized for lookup and snapshotting, or tagged with a local "epoch" that advances when a release operation is initiated. The release write can then wait until all loads with the previous epoch, or the snapshotted entries, have received a completion notification before issuing the write.

In the literature on memory models, this scenario is called Write-to-Read Causality (WRC), and the term, at least to my knowledge, was first introduced by the seminal Foundations of the C++ Concurrency Memory Model paper by Adve and Boehm. If you're familiar with distributed systems theory, you might recognize WRC as a test of whether the system enforces the "writes follow reads" guarantee of Causal Consistency. The specific feature of the release operation that I developed in this section is also known as A-cumulativity in the IBM Power and later ARM manuals, though I prefer to call it external cumulativity as a direct reference to the fact that we're ordering external writes observed through early loads.

I call this test D-WRC, where the "D" stands for "Direct", because the processor performing the release store observes the value it's propagating directly.

Send operations don't need to provide external cumulativity because they're optimized for point-to-point message passing.

7.2 Internal Cumulativity

Ordering of internal writes, or internal cumulativity (B-cumulativity in IBM and ARM manuals), is a feature the release operation already supports. Consider the following modification of WRC, which is sometimes called ISA2 (a modified version of it appears as the second cumulativity example in the Power ISA manuals):

// P0
st     [data], 1
st.rel [data_ready], 1

// P1
ld.rlx r1, [data_ready] // reads 1
st.rel [flag], 1

// P2
ld.acq r2, [flag] // reads 1
ld     r3, [data] // reads 0

C++11 forbids this outcome, and, given the current implementation of release, it's impossible to manifest it, because, by construction, the release operation doesn't perform the write to data_ready until data is globally visible, so any observation of data_ready = 1 implies the new value for data is obtainable, provided the invalidation buffer is processed. However, notice how P1's release store is completely unnecessary in this test.

In an implementation where P0 didn't have to wait for global acknowledgement of the data modification before issuing the data_ready write, and assuming the architecture guarantees those writes will be observed in the same order by every processor, this outcome would still fail to manifest, because of P1's observation of the modified data_ready flag. External cumulativity would kick in, and P1 will have to make sure its write is issued after the data_ready update is globally visible, which, by extension, guarantees the data write is also globally visible (data has to be processed before data_ready if we want to maintain write ordering).

I'll address the redundant work performed by st.rel in the next chapter, but what's important for now is the fact that external cumulativity is load bearing, and the only difference between ISA2 and D-WRC is that the value we're observing is an indirect causal successor of the write we want to propagate. This is why I call this test I-WRC, which coincides with "I" for both "Indirect" and "Internal [cumulativity]". Global visibility, which is the current solution for external cumulativity, is also an unnecessarily strong condition for external cumulativity too - all we want is for the causal graph of dependent writes to be visible to any processor which observes the most recent write in the graph.

7.3 Error Detection

The additional complexity and storage requirements to support cumulativity might seem excessive given that, seemingly, the only benefit is a faster completion of atomic writes, and only to handle a case that I, quite frankly, haven't seen in real code (would appreciate it if anyone who's encountered this scenario in practical algorithms emails me about the specifics). However, we can use this tracking machinery to detect improper synchronization if writes are tagged with an "atomic" bit. If an incomplete non-atomic write value is requested early, the processor could be dynamically configured to trigger an exception, instead of letting potentially buggy code continue to execute. This is obviously not sufficient to catch all synchronization bugs, but it will increase both reliability and security.

Raising an exception when an early non-atomic read is detected should also raise some concerns about this particular nMCA architecture. We're paying a rather large complexity tax, only to optimize the synchronization operations themselves. Outside of complex coordination protocols or optimistic locking implementations, like a seqlock, early reads will likely be followed by an attempt to write to the same location immediately, which must wait for global acknowledgement anyway. Moreover, seqlock-like scenarios, which are extremely useful in practice, can't be implemented if a racy read triggers an exception. Addressing these concerns will require fundamental changes to the architecture, and will come with non-trivial tradeoffs, so I'll discuss those later.

8. Asynchronous Cumulativity

Analysis of I-WRC revealed that we could potentially optimize the implementation of release stores, because, at least as far as internal cumulativity is concerned, global visibility is not required as long as we can guarantee that the value written by the store is only available after prior local writes become available. This is trivially guaranteed in the case of hardware multithreading, because threads on the same core have access to a shared store buffer, which can easily provide the required ordering.

However, the current behaviour of release stores may still be useful, so I'll split the release implementation into a release store, which only guarantees ordered delivery, and a commit store, st.cmt, which, in addition, waits until prior writes are globally visible, or committed, before issuing its associated store operation.

I'll not cover possible implementations of cross-processor asynchronous cumulativity in this article - that will be the subject of Part 2 - but I'll briefly discuss the inherent challenges such an implementation needs to overcome in a setup where no assumptions can be made about the topology of the interconnect, and the implementation of cache coherence.

8.1 Asynchronous Internal Cumulativity

Suppose you decide to implement internal cumulativity by assigning each invalidation a monotonically increasing, per-processor sequence number. A receiving processor could then stall applying invalidation when it detects there are gaps in the sequences that have been observed. But what if we don't even need to send an invalidation for the release store itself, because the receiver doesn't have a stale copy of its associated cache line. The receiver may have also silently evicted the cache line. We'd need to guard against a cache line fill request overtaking in-flight invalidations, and that's where complexity and overhead start to explode. The receiver can't know who's going to supply the data, so it can't provide a reference "highest observed" sequence number to identify a request we can reject, unless it attaches the observed sequence number vector for all processors, which is expensive. On the producer's side, we need to snapshot the per-processor counter when a release store is issued, so we can attach it to the returned data payload; the receiver can then detect a mismatch and delay installing the new data.

Now what happens when we need to downgrade the write to a shared state? The new data could now be supplied by multiple processors, so we need to replicate the tracking metadata, and make sure it's actually usable by processors with different sequence numbers. The tracking metadata would also need to be replicated if another processor issues a read-modify-write operation, though I'll cover the reason why later. If we decide to deny those request until outstanding invalidations are confirmed, we'd be stalling reads and writes globally, all for the sake of not blocking a single write. Do the tradeoffs seem worth it to you?

8.2 Asynchronous External Cumulativity

Asynchronous external cumulativity is difficult to achieve even in the hardware multithreading case. If an early read is serviced from a local write that's not yet globally visible, we'd need to make sure subsequent release stores by the reader are serviced after that particular write. However, the write which was observed early could, itself, be a release store from another thread that observed one or more writes early. This applies recursively up the chain, until we've built up an acyclic dependency graph of writes. Tracking this metadata efficiently is difficult, and would require heavy compression to avoid large storage requirements (cache is already precious, and we'd need to justify any additional storage), and that also comes with false dependency tradeoffs. If you think this is difficult to implement efficiently in hardware, now imagine implementing the tracking and ordering mechanism across multiple processors. This is orders of magnitude more difficult compared to cross-processor internal cumulativity, and that was already highly non-trivial.

Fortunately, there's an alternative to implementing a full distributed system, with all its associated downsides, in hardware.

8.3 Partially-Ordered Interconnect

The problem of asynchronous cumulativity becomes a lot more tractable in hardware if we drop the "arbitrary topology and routing" assumption about the interconnect. Instead of tracking and enforcing a directed acyclic graph of dependent writes and reads, and all the associated metadata, we could turn those dependencies into transient state inside an interconnect that's, itself, a directed acyclic graph (partially ordered). Moreover, this transient state is provably bounded, provided a suitable backpressure mechanism is in place.

This insight was polished into the operational "Flowing Model (FM), with a more abstract "Partial Order Propagation" (POP) distillation, during the development of the pre-MCA ARMv8 architecture. The paper is excellent, though I'd recommend reading it alongside "Simplifying ARM Concurrency: Multicopy-atomic Axiomatic and Operational Models for ARMv8", which explains the issues with this model, and why ARM ultimately pivoted away from it by requiring MCA architecturally. If you're unfamiliar with the subject, you'll get a lot more out of the papers if you finish this article first. I'll discuss how we can improve upon the Flowing Model in Part 2.

The idea was also explored in the 2013 "A String of Ponies", which introduced Distributed Pony, an experimental extension of the actor-based Pony language.

A similar approach was taken in the distributed systems world with the 2017 Saturn, a metadata service for causally consistent replication, except the partially-ordered topology is dynamically reconfigured in the event of a fault. The authors appear unaware of prior work in the multiprocessor interconnect space, and their related-work review also indicates this idea isn't present in the distributed systems literature, which, while strange, is not surprising given how the work on Flexible Paxos revealed there's still a lot of low-hanging fruit in that space (sorry, Heidi Howard).

Using network topology to maintain order without excessive metadata isn't a particularly new idea, and appears at least as far back as 1991 in the Race-free Interconnection Networks and Multiprocessor Consistency paper. The 1997 Sun Microsystems Enterprise 10000 (Starfire), as presented in chapter 7.7.1 of APMCCC, also uses a tree topology to serialize coherence traffic across 64 processors:

Regardless of how it's accomplished in hardware, I won't require release stores to ensure cumulativity by blocking until causally-prior writes, whether internal or external, are globally visible - only per-processor visibility is necessary. Commit stores, on the other hand, will continue to maintain this property. IBM takes the same per-processor visibility approach when it comes to its lwsync instruction, which can be used to implement release-acquire (RA) synchronization on the Power architecture.

9. Commit-Reconcile Semantics

Adve and Boehm introduced another scenario called Read-to-Write Causality (RWC), which shows up in practice, and often trips up even experienced developers. Here's a slightly modified version of it that introduces more useful context:

// tail = 0, head = 2 at the start

// P0 (Consumer)
st.rlx [tail], 1 // advance the tail

// P1 (Producer)
st.rlx [head], 1  // decrement the head
ld.rlx r1, [tail] // reads 0
// head - tail = 1

// P2 (Consumer)
ld.acq r2, [tail] // reads 1
ld.rlx r3, [head] // reads 2
// head - tail = 1

The C++ memory model architects decided that this outcome shouldn't be prohibited, even if we somehow managed to prevent the P1 store from executing after the load locally, and they had a very good reason for making that choice. We only have 2 variables instead of 3, and, when reading the paper, it seems like a purely theoretical concern.

Now imagine that head and tail are the producer and consumer indices of a ring-buffer-style input-restricted deque. The producer advances the head, the consumer advances the tail, and, at the start, the deque contains two items (head - tail = 2). This particular implementation, called a Chase-Lev deque, allows for multiple consumers, but there's only ever one producer, which is allowed to consume from the front by decrementing the head. Consumers in this design operate speculatively - they copy the entry at the tail first, then attempt to increment the tail index, hoping they were the first to do so.

In this scenario, P0 is a consumer that successfully pops one of the items from the tail. The store would be a RMW operation in actual implementations, but what matters is that a write is performed. The problem arises when there's only one item left in the deque, and both a consumer and a producer attempt to claim it:

If both of them see that there's one item remaining, which happens in this case, they'll both pop it and effectively duplicate the entry. We want one of them to observe head - tail = 0, so the producer can roll back its decrement, or the consumer can give up.

9.1 Store Buffering

The first problem to resolve is ordering the store and the load. The producer needs to make sure its intent to consume an entry from the head is visible to all consumers before it checks the state of the tail. However, we don't have any way of making sure a store is globally visible before executing a load. Conceptually, a commit store executes a commit pseudo-instruction before executing the store, where commit stalls until prior loads and stores are globally committed, meaning local stores and observed external stores (via loads) are globally visible. Similarly, an acquire operation executes an update pseudo-instruction after executing the load, where update performs the usual snapshot of the invalidation buffer and stalling until the snapshotted entries are processed. A commit store followed by an acquire load therefore maps to the following conceptual sequence of instructions:

// st.rel is equivalent to:
commit
st.rlx [head], 1
// ld.acq is equivalent to:
ld.rlx r1, [tail]
update

Notice how the relaxed store and load have no ordering guarantees. In fact, commit and update only stall their associated memory operations, so there are no ordering guarantees for anything that happens before or after them. This is not just a processor-local reordering either - executing the instructions in order provides no guarantees about global visibility because the architecture is nMCA. So what kind of operation do we need to provide to constrain store-load ordering?

Suppose we implemented a commit load (ld.cmt) instruction, which behaves like st.cmt, except it executes a load instead when all prior internal and external stores are globally visible. The store-load ordering case then becomes:

st.rlx [head], 1
commit
ld.rlx r1, [tail]

This might seem sufficient, but consider what would happen if an invalidation arrives before, or while waiting for the store to commit. Without an update, the follow-up load will read a stale value, which would, in effect, reorder the two operations, even if the processor faithfully executed them in program order. The commit load would therefore need to perform an update after the commit:

st.rlx [head], 1
commit
update
ld.rlx r1, [tail]

By the same logic, we can define a reconcile store, st.rec, that performs a store followed by a commit and an update. Notably, the commit part of this operation need only wait for the associated store to become globally visible, and the update part of the commit load should only make sure to apply the invalidation in the IB for the load's cache line, if there is one.

Both options are valid for this litmus test, and which one is a better choice would depend on the context in which they're used in a real algorithm. I'll pick ld.cmt because the instruction will become relevant again in the next section.

The store-load reordering problem is typically called store buffering, because it can be caused by a store buffering unit that allows the processor to continue executing instructions without waiting for cache line arbitration and global acknowledgement. However, as demonstrated, the presence of an invalidation buffer needs to be accounted for as well, both locally and remotely, because a buffered invalidation is the same as non-local store buffering.

The updated snippet now looks like this:

// P0 (Consumer)
st.rlx [tail], 1 // advance the tail

// P1 (Producer)
st.rlx [head], 1  // decrement the head
ld.cmt r1, [tail] // reads 0
// head - tail = 1

// P2 (Consumer)
ld.acq r2, [tail] // reads 1
ld.rlx r3, [head] // reads 2
// head - tail = 1

Do you see any problems with it? While we've considered the ordering issues caused by the invalidation buffers, we've yet to account for issues caused by early loads.

9.2 Reconcile Loads

What would happen if P2's load of tail is early, and the invalidation hasn't even reached P1 by the time it performs its load? P2 may also observe the state of the head before P1's store. This would create the exact same problem we're trying to avoid, and, unlike the external cumulativity issue, this time we can't just shift the responsibility onto the commit operation. Fortunately, we already have the external-cumulativity machinery to detect the early load and stall until the observed write is globally visible. In fact, the commit operation does exactly what we need, so P2 can be modified to:

ld.rlx r2, [tail]
commit // wait for the tail write to be globally visible
update // perform the acquire operation
ld.rlx r3, [head]

Notice how this maps exactly to the ld.cmt pattern:

ld.rlx r2, [tail]
ld.cmt r3, [head]

If we only cared about economy of instructions, we could use ld.cmt to enforce the desired behaviour. However, it's worth considering the impact on the ISA as a whole. If a concrete implementation prohibits early reads, the commit in P2 becomes unnecessary. On the other hand, a ld.cmt operation still needs to guard against potential upstream store buffering, so the commit operation can't be dropped.

The other problem with the commit semantics on the second load is that it doesn't communicate the programmer's intent. Are they guarding against an upstream store, or do they want to work around the lack of write atomicity? It's convenient to have a load that can restore write atomicity, and we'll need such an instruction very soon in a different context.

I'll call the new instruction a reconcile load, or ld.rec; the name is inspired by the 1999 paper on "Commit-Reconcile and Fences" (CRF) semantics, though the similarity is superficial. Using pseudo-instructions, a reconcile load corresponds to the ld.rlx; commit; update sequence of operations, where the semi-colon is used to separate different instructions without a new line in between them. Something important to point out is that the commit in this implementation would only wait for its associated load, not all prior loads and stores. The reason why I'm illustrating it this way is to highlight the commit; update shape, because it will become relevant later. The reconcile load is actually similar to a commit store in that it's a more restrictive version of an acquire operation - it performs an update, just like an acquire, but it doesn't complete the operation until the observed value is globally visible.

Here's the final version of the example:

// P0 (Consumer)
st.rlx [tail], 1 // advance the tail

// P1 (Producer)
st.rlx [head], 1  // decrement the head
ld.cmt r1, [tail] // reads 0
// head - tail = 1

// P2 (Consumer)
ld.rec r2, [tail] // reads 1
ld.rlx r3, [head] // reads 2
// head - tail = 1

Let's work backwards from P1's results to see if the undesired outcome is still possible. If P1 doesn't see P0's update to tail, that means P1's write to head became globally visible before P0's write to tail. This is guaranteed because the update performed by ld.cmt happens strictly after the commit. By the time P2 is able to complete its observation of P0's write, P1's head invalidation must be in P2's invalidation buffer. This is guaranteed because ld.rec stalls until the observed value is globally visible, and we know from P1 that the head update became globally visible before the tail update. Since ld.rec also performs an update, the subsequent ld.rlx can't observe a stale value for head.

I'll cover the full correctness proof together with some optimizations for existing Chase-Lev deque implementations using commit-reconcile semantics, and how both C++11 and C++20 fail to provide efficient cross-platform instruction mappings, in another article.

9.3 Delegated Store Buffering

A variant of RWC that's worth examining is W+RWC, because it also shows up in algorithms such as delegated safe memory reclamation.

Safe memory reclamation (SMR) is the name for a class of techniques that allow memory removed from a concurrent data structure to be safely reclaimed (returned to an allocator or the operating system) and reused (allocated again), despite other threads potentially still holding references to it. Besides reference counting, which requires expensive read-modify-write operations, common approaches differ primarily in how readers announce what they may still access. Implementations based on hazard pointers (HP) explicitly publish addresses which are currently being accessed, while epoch-based reclamation (EBR) requires announcements that a thread is active in a particular "epoch", conservatively protecting memory that might still be reachable by readers from that epoch. Preemptible and software variants of read-copy-update (RCU) use reader-side per-CPU counters or related bookkeeping to determine whether any readers remain active.

Non-preemptible RCU can avoid such explicit reader announcements by preventing a read-side critical section from being preempted; the reclamation machinery instead waits for every relevant CPU to pass through a quiescent state, such as a context switch, transition to user mode, or entering idle state. In general, reclamation algorithms based on quiescent states don't need to announce that they're about to access a shared memory, and are therefore, at least to my knowledge, not relevant for W+RWC.

Here's a distilled example of the problem delegated hazard pointer implementations need to deal with:

// address = OLD_ADDRESS at the start

// P0 (Writer)
st.rel [address], NEW_ADDRESS
st.cmt [clean_up], OLD_ADDRESS

// P1 (Reader)
st.rlx [looking_at], OLD_ADDRESS
ld.cmt r1, [address] // sees OLD_ADDRESS
// Check if the value stored at looking_at matches the value in r1
// Success. Use the address and set looking_at = 0 at the end

// P2 (Garbage Collector)
ld.acq r2, [clean_up]   // sees OLD_ADDRESS
ld.rlx r3, [looking_at] // sees 0
// Concludes P1 can no longer see OLD_ADDRESS and frees the associated memory

The writer updates a shared pointer (address) and stores the old value in the clean_up variable, so a dedicated garbage collector can clean it up when the reader stops using it. The reader stores the old value in its looking_at variable, loads the value from the address variable, concludes that nothing's been updated because the two values match, so it continues to read from the pointer, setting looking_at to 0 when it's done. Meanwhile, the garbage collector sees that there's a new pointer to be cleaned up in the clean_up variable, checks to see if the reader is currently looking at it and concludes it's not because of a stale read. If the reader also read a stale value, the garbage collector may free the memory while the reader is still using it. Can this actually happen with the synchronization already in place?

Following the same strategy as the proof of the final version of RWC, P1's store to looking_at became globally visible before P0's store to address. The store to address became globally visible before the store to clean_up, because of the commit operation. It follows that the looking_at store became globally visible before the clean_up store, and since the corresponding load in P2 has acquire semantics, the follow-up load of looking_at can't be stale.

You might find it suspicious that we needed ld.rec for RWC, but not for W+RWC. The reason why is because the commit store takes care of the global visibility requirement. The correctness proof relies on the ordering between two independent writes, clean_up and looking_at, and the way we obtain the ordering is through the ld.cmt observation establishing a connection between P0 and P1.

A release instead of a commit store is not sufficient to provide any constraints on the ordering between the independent stores. A reconcile load in such an implementation would determine the point at which clean_up becomes globally visible, which implies the address store that precedes it is also globally visible. This is precisely what we need to establish a causal chain of events, because we know from ld.cmt that the store to looking_at became globally visible before the store to address, so the invalidation must be in P2's invalidation buffer by the time the reconcile load completes.

We've just proved that the following snippet is also a valid solution to the W+RWC problem:

// address = OLD_ADDRESS at the start

// P0 (Writer)
st.rel [address], NEW_ADDRESS
st.rel [clean_up], OLD_ADDRESS

// P1 (Reader)
st.rlx [looking_at], OLD_ADDRESS
ld.cmt r1, [address] // sees OLD_ADDRESS
// Check if the value stored at looking_at matches the value in r1
// Success. Use the address and set looking_at = 0 at the end

// P2 (Garbage Collector)
ld.rec r2, [clean_up]   // sees OLD_ADDRESS
ld.rlx r3, [looking_at] // sees 0
// Concludes P1 can no longer see OLD_ADDRESS and frees the associated memory

The second implementation is also more efficient, because the only reason why we're using a dedicated garbage collector is because we want the write to be as fast as possible, and a release store is less constraining compared to a commit store.

I call W+RWC delegated store buffering because an implementation that doesn't use a garbage collecting processor (thread) would need to load the value of looking_at after performing the store to check if it can free the old address, which turns into a Dekker-style store-load problem across two processors.

For comparison, the first implementation corresponds to the Power W+RWC+sync+addr+sync test, and the second implementation corresponds to W+RWC+lwsync+sync+sync; both implementations make the undesired outcome architecturally forbidden.

In the full implementation, P1 needs to read address with an acquire load to obtain the OLD_ADDRESS value before attempting to announce that it will be looking at it, and the acquire is there to ensure the data P0 wrote before publishing the new address with a release store is actually visible to P1. The commit load is only there to verify that the address itself didn't change between the two loads. This check-announce-check-again pattern is quite common in algorithms, so it's important to have efficient operations to support it.

If we turn the P0 commit store in the first solution into a release, and examine the outcome of the equivalent W+RWC+lwsync+addr+sync Power litmus test, we'd find that outcome is observable on Power 6 and 7, but not PowerG5. We can conclude that lwsync doesn't wait for global visibility of prior writes, which matches statements made in their manuals. This is a strong, though not conclusive, indication that Power implements asynchronous internal cumulativity at the very least.

10. Read Causality

If you stare at RWC and W+RWC long enough, you might notice a resemblance with D-WRC and I-WRC.

Suppose a variable x starts out with a value of 0, and processor P0 writes 1 to it. Another processor, P1, which reads 0 can conclude that its read must have happened before P1's write. However, if P1 performed a write of 1 to a variable y (which also equals 0 at the start) before initiating the load, we can't conclude that a third processor, P2, which observes the write of x will also observe the write of y, because those updates can arrive in any order at P2. However, if P2 waits for the write of y to be observable by P1, then the two loads form a causal link, which guarantees that if P1 observes the old value of x, and P2 observes the new value of x, then P2 is guaranteed to observe P1's y write.

In this way, the commit and reconcile loads form a pair similar to the release-acquire pair in WRC, except they establish causality not by reading the same value, but by reading an older and a newer value, respectively, in the modification order of the variable they use for synchronization. Although the mechanism is a little convoluted, the important part is that P1's prior write is guaranteed to become visible to P2's second read as long as P2's synchronizing read observes a store that P1 missed.

The primary difference between RWC and W+RWC is the fact that the synchronization variable is observed directly in the former, and indirectly, via a causal successor, in the latter. This is why I use D-RWC and I-RWC for RWC and W+RWC respectively. D-WRC and I-WRC follow the same pattern, except the direct and indirect observations concern data that's being propagated.

C++11 didn't ship with reconcile loads, commit loads, or reconcile and commit stores; RWC is allotted only a small paragraph in the Adve and Boehm paper, stating it's not clear the RWC outcome should be prohibited by default. This is rather unfortunate, since the RWC family shows up in practical algorithms all the time, and the language makes it impossible to implement efficient synchronization across all platforms, because it requires the programmer to use the blunt hammer of sequential consistency to repair causality. Above all, the missing operations make it difficult to recognize common patterns in parallel algorithms, which also impedes communication of intent, in code and writing, as well as education.

11. Coherence Causality

The external and internal cumulativity tests (D-WRC and I-WRC respectively) have two interesting analogues that involve coherence ordering, and it's worth examining those carefully, because maintaining cumulativity over coherence is necessary to fully recover the appearance of write atomicity when we need it.

11.1 External Coherence Cumulativity

We get DC-WRC ("C" for "Coherence"), when we replace the observation of the new flag value in P2 (via an acquire load) with a write to the flag variable that follows P1's write in modification order (mo), also called coherence order (co). This means that if we read the value of flag after the processors have successfully synchronized with an external observer, we'll see P2's value. We also need a st.rec to ensure proper handling of the invalidation buffer and local ordering:

// P0
st.rlx [data], 1

// P1
ld.rlx r1, [data] // reads 1
st.rel [flag], 1

// P2
st.rec [flag], 2 // succeeds P1 in co
ld.rlx r3, [data] // reads 0

// flag = 2 at the end

This outcome is impossible with the current implementation of external cumulativity, because the release write in P1 is only executed after P0's write is globally visible. This means that when P2's write successfully overtakes P1's, the invalidation should already be in P2's buffer, and the update at the end will ensure a stale value isn't read.

If P1 and P2 are hardware threads executing on the same processor, P2's flag update might overwrite P1's value in their shared store buffer, but, unless the implementation has parallel and completely unsynchronized loads for each thread, the new data should be available to P2, because of the shared cache. I'll cover load ordering in a later chapter, but, for now, you can assume P2's load can read a stale value, even if P1's load got the update.

As stated before, architecturally, we are allowing the flag write to become visible early, and potentially getting overwritten before the data invalidation reaches P2. We can also manifest this outcome by introducing a minor optimization that allows the write to silently complete locally without issuing any invalidations. This would speed up the arbitration part of the write operation, and introduce the problem of a non-silent write invalidating the silent one, and cancelling the visibility obligation. Arbitration communication need not even take place on the data channels used for invalidation, so P2 could easily complete its write before the P0 invalidation reaches it. The commit executed after the store in st.rec can't do anything about the arrival timing of P1's invalidation, and is only there to order the subsequent read - the acknowledgement from P1 might race the invalidation because it took a different route on the interconnect.

To prohibit this outcome, we'd need to replace the release store with a commit store, because the commit implementation is obligated to not make its write visible after all causally dependent writes are globally visible.

On a side note, following the load-load synchronization pairing from the previous section, this time we have a commit store synchronizing with a reconcile store via their coherence ordering to make sure the new data value is propagated to P2.

DC-WRC has the rather cryptic "WRW+WR" name in the literature, and the Power-specific mapping is WRW+WR+lwsync+sync. Interestingly, while the outcome is allowed, it's not actually observable on Power 6 and 7. This is not surprising once you consider the difficulty of making external cumulativity non-blocking. The interconnect might also be sufficiently ordered to prevent the invalidation-acknowledgement race. If you've got access to a modern Power implementation like 8, 9, or 10, and can run this particular test, I'd be curious to see what the results are!

The version of the test with st.cmt instead of st.rel corresponds to the Power WRW+WR+syncs, and, as expected, is also forbidden.

11.2 Internal Coherence Cumulativity

The IC-WRC test follows the same pattern, except this time we replace the P1 load in I-WRC with a store:

// P0
st.rlx [data], 1
st.rel [data_ready], 1

// P1
st.rlx [data_ready], 2 // succeeds P0 in co
st.rel [flag], 1

// P2
ld.acq r2, [flag] // reads 1
ld.rlx r3, [data] // reads 0

With the optimization from the previous subsection in place, this outcome is now possible if P0's early write is invalidated by P1's. It's also possible if P0 and P1 are actually hardware threads executing on the same processor, and thus sharing a store buffer (which won't be split as per our efficient data sharing between hardware threads requirement), though inheriting the causal obligation to propagate data before the flag to P2 is significantly easier in this case.

Replacing the release store in P0 with a commit one may prohibit the outcome, because by the time data_ready is observable, the data invalidation is already sitting in P2's invalidation buffer. However, consider what would happen if P0 and P1 are hardware threads on the same processor and P1's write overwrites P0's value in their shared store buffer. Should the release store inherit the obligation to propagate P0's write now? Such a demand would expand the responsibility for release stores, because now they need to propagate ordering over single-location writes as well, and only for corner cases like this.

One way to prevent this outcome is to halt execution in P0 until causally prior writes are complete, or have a separate buffer for commit stores, making sure they're carefully synchronized with the general store buffer, where P1's first write would be submitted. I don't think this additional complexity is worth it just to prevent coherence ordering issues, so the approach I'll take is to upgrade P1's release store to a commit, and pass the coherence ordering obligation onto it. A simple fix for the SMT scenario is to have the commit store wait for acknowledgement of all stores the core has issued before making it visible, though fine-grained tracking is also possible. Regardless, the following modification should be a solution to the problem:

// P0
st.rlx [data], 1
st.cmt [data_ready], 1

// P1
st.rlx [data_ready], 2 // succeeds P0 in co
st.cmt [flag], 1

// P2
ld.acq r2, [flag] // reads 1
ld.rlx r3, [data] // reads 0

What if we upgrade P2's acquire read to a reconcile? Waiting until the release store is globally complete is the same as waiting for its causal predecessor to be globally complete, and that means that, by the time the reconcile completes, P0 has seen the data_ready update, and for data_ready = 2 to succeed data_ready = 1 in modification order, the data update must already be globally visible, and that would be the case if P0's data_ready update is a commit; remember, commit stores, unlike release ones, are obligated to make their associated writes visible after the causal predecessors are globally visible. This gives us another solution to the problem:

// P0
st.rlx [data], 1
st.cmt [data_ready], 1

// P1
st.rlx [data_ready], 2 // succeeds P0 in co
st.rel [flag], 1

// P2
ld.rec r2, [flag] // reads 1
ld.rlx r3, [data] // has to read 1

Notice how the causal chain between the commit store and reconcile load is propagated by the ordinary release to ensure the data payload is delivered, whereas, in the previous solution, the two commits are sufficient to deliver the data via an ordinary acquire load. As I'll demonstrate throughout this series, this subtle interplay between coherence and write causality is one of the reasons why the informal specification of the C++ memory model struggles so much with how rules are communicated, and why efficient implementation of the rules on real hardware is constantly running into problems.

IC-WRC is usually called Z6.3 in memory models literature, and, as expected, the Power mapping equivalent to the presented snippet is allowed and reproduces the undesirable outcome on Power 6, and 7 (Z6.3+lwsync+lwsync+addr), as well as Power 8 (Z6.3 + pooncerelease + pooncerelease + poacquireonce using the Linux Kernel Memory Model naming convention). Once again, I couldn't find out if this outcome is observable on Power with SMT disabled. The double commit and commit plus reconcile scenarios correspond to Z6.3+sync+sync+addr and Z6.3+sync+lwsync+sync respectively, and both of them are architecturally forbidden.

C++11 doesn't require release operations to maintain cumulativity when it comes to coherence ordering, and we managed to take advantage of this laxness to implement additional optimizations. In Part 2 we'll see how the same issues can materialize even when hardware multithreading isn't involved.

12. Sequential Consistency

You may have noticed that, by now, we've implemented functionality to cancel out almost all ordering side effects caused by local execution reordering, early reads, and invalidation buffering. Reconcile loads prohibit early reads, stops later reads and writes from executing, and the update afterwards makes sure the invalidation buffer doesn't contain any pending invalidations, so the subsequent reads are up-to-date. Commit stores prohibit prior reads and writes from being reordered past them, both locally and globally, in addition to closing off coherence side channels that can expose the lack of write atomicity. The only problem that's left is to close off the store-followed-by-a-load case, which is trivial if we insert a commit;update pseudo-instruction in between them. Once we have that, we'll basically achieving both write atomicity, and strict program order execution, which means executing the instructions in the order they were written.

We can create a new set of sequential consistency loads and stores (.sqc), which, at least for now, will behave like reconcile and commit respectively, except, when a st.sqc is issued, it sets a "pending-serialization" flag, which will be cleared by issuing a commit; update the next time a ld.sqc is executed - commit stores and acquire loads could also clear each of the serialization requirements independently. Similarly to reconcile stores and commit loads, the commit part need only wait for the last SC store, and the update should only make sure there's no invalidation for the load's cache line waiting in the invalidation buffer, and apply it if there is one.

If we turned all loads and stores in our programs into SC ones, we'd effectively restore write atomicity and program order. Obviously that's complete overkill, but as soon as you allow other instructions, like regular loads and stores, you create the possibility of a side channel of communication exposing the architectural lack of write atomicity. An important realization is that this side channel is only opened when regular loads and stores are performed without appropriate synchronization (a data race), so you could mandate that the new set of instructions, together with regular loads and stores, achieve sequential consistency in the absence of data races, or SC+DRF, where "DRF" stands for "Data Race Free". In other words, the DRF part ensures the illusion of write atomicity and program order for SC atomic operations is maintained.

This is quite an attractive model, because the acquire and release semantics built into the SC atomics allow you to create all essential synchronization primitives. The model is also simple to reason about, because programmers can visualize the execution of atomic operations across different processors as a simple interleaving of the instructions they wrote in code. Likewise, proofs and automated verification are straightforward, because they can be modelled with state machines, so reachability of undesired states is a decidable problem.

I'd say SC+DRF is a great model for any language that seeks to expose a simple set of atomic primitives to programmers, and it's a particularly good choice for languages with automatic memory management. This is likely why Java and Go adopted this model, and why computer architectures should have an efficient toolkit for supporting sequential consistency.

12.1 Total Order

If you look at the C++ documentation, you'll find the following definition of sequentially-consistent atomic operations:

Atomic operations tagged memory_order_seq_cst not only order memory the same way as release/acquire ordering (everything that happened-before a store in one thread becomes a visible side effect in the thread that did a load), but also establish a single total modification order of all atomic operations that are so tagged.

I won't get into a mathematical presentation of order theory right now, but, for programmers who are familiar with the language of directed graphs, a total order can be visualized as a sequential chain of nodes with directed edges, while a partial order is a more general directed acyclic graph:

You might think that write atomicity and program order can't possibly achieve a total order across multiple locations, because there's no serialization point anywhere in the system. The reason why that assumption is wrong is because requiring a total order is an unnecessarily strict condition. As long as the operations form a directed acyclic graph, you can topologically sort them into a total order. The topological sorting algorithm serves as a constructive proof of the fact that any finite partial order has a linear extension.

To understand why write atomicity is a prerequisite for acyclicity, you can start by turning memory operations for a single location into a graph:

Reads (R) and writes (W) are represented by function-like notation, where the first argument is the processor, the second is the variable name, and the last is the value read or written - I may omit the processor if it can be inferred from context. Edges between same-location writes are called coherence order (co), or modification order (mo) relations in the formal modelling literature. Edges connecting a write to a read that observes it are called reads-from (rf). Program order edges are typically labelled po or sequenced-before (sb). The most interesting type of edge is reads-before (rb), which is typically called from-reads (fr), though I refuse to use such a terrible name; it connects a read of a prior value to a write that follows the value in modification order. It can formally be specified as going backwards from a read node across an rf edge, and then following the write in modification order to establish an edge between the read and the subsequent write. Formal models literature would also distinguish internal and external operations with an "i" and "e" prefix or suffix, such as "eco", or "rfi". I won't be doing that for now.

As you can see, write atomicity gives us a well-defined cutoff, and past that point the old value is no longer observable by any load.

An execution is called sequentially consistent when the graph formed by all mo, rf, rb, and sb edges is acyclic. Throughout this series, I'll explore this graph-based dataflow analysis in the context of proving correctness and formal modelling.

12.2 Causal Transitivity

But why does write atomicity and program order result in sequentially consistent execution? Feel free to skip this section if you don't care, because I'm about to get a bit more formal.

The per-location graph makes the partial order for each memory location obvious, but, unfortunately, it doesn't provide any insight on how to combine two or more locations into a DAG. A crucial insight is that any cycle which respects the coherence rules of individual memory locations must span multiple locations. Such a cycle necessarily involves at least two writes, including initialization writes. This matters because if we can establish a causal dependency between writes, then every impossible execution can be reduced to a cycle in the write dependency graph, which can't happen unless we've violated causality - that is, unless we have a write causally depend on itself. I call the proof technique I'm going to use causal read projection (CRP), because I'll systematically enumerate all the ways in which two operations sequenced by the same processor create causal dependencies when we project reads onto their causally-dependent writes.

First, the 3 obvious causal dependencies that apply per-location are rf (a read which observes a write), rb (a write that happened after a previous observation), and mo (direct modification order) edges.

I'll denote a read or write event for location x with E(x), with W(x) being a write, and R(y) being a read. An event E(x) connected to an event E(y) by one of the aforementioned edges will be denoted E(x); <edge-type>; E(y), and the starting point for the proof is to considering all pairs of operations that are sequenced before each other: E(x); sb; E(y)

We've got 4 possibilities here: A write followed by a write (WW), a read followed by a write (RW), a write followed by a read (WR), and, finally, the most interesting one: a read followed by a read (RR). Let's carefully examine each one, and how it contributes to establishing a partial order between writes. From now on, I'll use global visibility to reason about causal dependencies.

WW, or W(x); sb; W(y), is the most trivial scenario, because we publish the writes in order, so W(x) became visible before W(y), which I'll denote W(x) < W(y).

The R(x); sb; W(y) RW scenario is also straightforward: Observing a value via a read implies that the corresponding write is already globally visible, and since we publish a write after the read, we can establish that the observed write is globally visible before the second, or W(x) < W(y). The full expression is W(x); rf; R(x); sb; W(y), and if we strip away the events to only highlight the edges, we get our first edge pattern: rf; sb. In this case, we've projected a read sequenced before a write onto the write from which it obtained its value.

A write followed by a read (WR) W(x); sb; R(y) is a little more tricky, because we have to reason about any modification order successor W'(y) of the W(y) write observed by the read. We can write that as W(x); sb; R(y); rb; W'(y), or just sb; rb. Processing the invalidation buffer after W(x) is critical for this relation to hold, because we want to use the read of y as a negative observation. Completion of the x read is evidence that its corresponding write is globally visible, and an observation of the old value of y after that point is evidence that the subsequent write is not globally visible yet. Global visibility here is used to establish causal dependence, so we can conclude that W(x) < W'(y). This is the P1 store buffering issue I covered in the context of RWC, and its corresponding pattern is sb; rb.

The last remaining scenario to address is RR, or R(x); sb; R(y), and the strategy for it is a combination of RW and WR: W(x); rf; R(x); sb; R(y); rb; W'(y), which implies that W(x) < W'(y). RR corresponds to the rf; sb; rb pattern, and it should be familiar from the P2 early visibility problem in RWC. Processing the invalidation buffer after R(x) is critical for the same reason as the store buffering case.

The edge patterns correspond to a notion of causal transitivity for write events, and can be used to iteratively construct a partial order; we can summarize all of them using the following pattern matching rule: rf?; sb; rb?, where the ? syntax is inspired by regular expression (regex) syntax and means "zero or one occurrences".

If we add mo to the pattern, we get mo | (rf?; sb; rb?), and we can apply this pattern expression repeatedly to establish causal dependencies between writes. The | syntax is also inspired by regex, and it means "the expression on the left side, or the expression on the right side", transitively. Note that rf; rb is subsumed by mo, so it's implicitly contained in the pattern. Taking into account the fact that we're constraining the end points to be writes, we get the final pattern:

W(x); mo | (rf?; sb; rb?); W(y)

We have established a projected write-to-write constraint for every program-order bridge that can participate in a cycle. Taking the union of those constraints with modification order gives a global partial order over all writes; causally unrelated writes remain incomparable.

The exhaustive nature of the pattern expression makes it obvious why a cycle in the execution graph, G, would produce a cycle in the write causality graph, K. Suppose G does contain a cycle, and consider every pair of writes in the cycle with no other writes in between. Only the starting edge after the first write can be rf, and only the last edge (before the last write) can be rb. Anything in between could only be sb, and those are transitive, so we can collapse the chain of sb edges into a single edge. Now we're left with either rf; sb; rb or rf; rb, and since the latter collapses to mo, the entire path corresponds to the edge between the writes in K. This means the entire cycle can be mapped onto a cycle in K, which is acyclic by definition, so we get a contradiction. This concludes the proof.

12.3 Global Commit

Reasoning about global visibility is one way to establish causal dependencies, and it's a natural extension of the architecture I've been developing. However, it's an unnecessarily strong condition. Imagine an architecture where all writes are sent to a central arbiter (server), and the arbiter broadcasts the order it has decided on to each processor. Such an implementation achieves an actual total order, but observing a write doesn't mean the list entry has propagated to all other processors, which would imply the write is globally visible.

The two architectures are, in some sense, duals of each other, because the implementation we've been discussing propagates invalidations first, and a global commit point is established after they're acknowledged, so it follows a propagate -> commit (PC) protocol (PCP). The server architecture, on the other hand, can be described as a commit -> propagate (CP) protocol (CPP), because it commits the stores first, and then propagates the established order asynchronously.

Because CPP is sufficient to establish a total order, it's usually seen as the natural way to define sequential consistency. PCP, on the other hand, provides an excellent foundation for synchronizing other atomic operations. Importantly, both can provide an implementation of sequential consistency. I'll demonstrate how to combine the two approaches in Part 2.

For now, it's important to recognize that, in PCP, global visibility implies global commit, which is not true in CPP. The following section will cover the details of how a PCP system can implement sequential consistency.

12.4 Implementation Optimization

Given the usefulness of sequential consistency, it's important to consider various implementation strategies, and how the ISA would be implemented by physical chips that only support simple commit; update instructions, like the serialization fences I'll discuss later - I'll refer to such instructions as serialize from now on. Such an implementation has a very limited choice when it comes to supporting SC atomic operations, assuming it can't provide conditional execution of serialize instructions. SC loads in such an implementation could be implemented as ld.rlx; serialize, and the SC store would then need to be serialize; st.rlx; serialize. The serialize instruction after the relaxed store could also be moved before the load. Regardless of the choice, two consecutive loads or stores will be executing one redundant serialize. The architecture or compiler could fuse those instructions, though other operations being executed in between could make this more cumbersome.

It would be nice to get rid of one of those serializations, and we can do this by recognizing that a reconcile operation is actually unnecessarily strict for what we need. The commit part of reconcile is there to order subsequent atomic reads, but an SC load doesn't need to care about ordering any atomics besides other SC loads. A commit store would already ensure the writes corresponding to prior loads, including the SC one, are globally complete anyway. This means we can change the SC load mapping to a commit; update; ld.rlx; update, while the store could stay the same. More importantly, as long as an implementation can provide an efficient equivalent of update, the mapping could remove a serialize by mapping SC loads to serialize; ld.rlx; update and SC stores to serialize; st.rlx. This is called a leading-sync mapping, and we get the trailing-sync variant by using st.rel; serialize for stores, and ld.rlx; serialize for loads.

The recommended C++11 compiler mappings for ARMv7 and Power are an interesting demonstration of this optimization. The ARMv7 mappings follow the triple-serialization idea discussed earlier. A dmb ish is the equivalent of serialize, and the mappings use ldr; dmb ish for SC loads, and dmb ish; str; dmb ish for stores. The authors of the mappings believed ARM's equivalent to an update, which is teq; beq; isb, is not efficient enough to justify removing a serialization, or they might've been worried about correctness with respect to the C++ model, which I'll cover in the next section. ARM also doesn't have a dedicated instruction or barrier that can emulate the behaviour of a st.rel, hence the triple serialization necessary. On the other hand, the Power mapping does take advantage of the optimization I discussed, and uses a leading mapping: hwsync; ld; cmp; bc; isync for SC loads, and hwsync; st for SC stores, where hwsync is equivalent to serialize and cmp; bc; isync is the equivalent to update (Power doesn't expose the side effects of its invalidation buffering to the programmer unless speculative racy reads are performed).

We can try to take advantage of the same optimization architecturally by removing the commit from ld.sqc, and checking if prior .sqc loads are complete before issuing another SC load; the processor can perform a commit conditionally if the corresponding write is still not globally visible. However, the problem is that we now need to execute, at a minimum, a targeted update (waiting until a pending invalidation for the load itself is processed, if on exists) before each SC load, because we don't know when the prior load completed. Whether this additional complexity is beneficial or not depends entirely on the code that's being executed, but it reveals the relative weakness of ld.sqc compared to ld.rec, so we'll do it to be maximally permissive.

Consider what a correctly synchronized RWC would look like using SC instructions:

// P0 (Consumer)
st.rlx [tail], 1 // advance the tail

// P1 (Producer)
st.sqc [head], 1  // decrement the head
ld.sqc r1, [tail] // reads 0
// head - tail = 1

// P2 (Consumer)
ld.sqc r2, [tail] // reads 1
ld.sqc r3, [head] // reads 2
// head - tail = 1

Notice how the once minimal and efficient solution is now riddled with SC instructions, because we don't know where the commit; update sequence would be inserted in a physical implementation. This awkwardness is not even the major issue with sequential consistency in the presence of atomic operations with fewer ordering guarantees (weaker atomics).

An SC implementation should also be free to make only the SC stores write-atomic, and remove any kind of serialization from the loads, which means the RWC snippet above is incorrect, and the P0 store needs to be replaced with st.sqc instead. This virality of SC operations is one of the reasons why I call them a "blunt tool", and, as you'll see in the following section, the uncertainty around the serialization placement makes them unsuitable for ordering weaker atomic operations.

12.5 Shattering the Illusion

I mentioned the danger of side channels of communication exposing the true nature of the underlying architecture, and, while we can claim side channels opened by data races are the fault of the programmer, other, less restrictive, atomic operations can open the same channels, and those operations, by definition, allow for unsynchronized access.

It took a surprisingly long time for researchers to spot issues with the C++ model. A 2016 paper titled "Counterexamples and Proof Loophole for the C/C++ to POWER and ARMv7 Trailing-Sync Compiler Mappings" by Manerkar et al. used RWC as one of their counterexamples:

// P0 (Consumer)
st.sqc [tail], 1 // advance the tail

// P1 (Producer)
st.sqc [head], 1  // decrement the head
ld.sqc r1, [tail] // reads 0
// head - tail = 1

// P2 (Consumer)
ld.acq r2, [tail] // reads 1
ld.sqc r3, [head] // reads 2
// head - tail = 1

Note how only the first load in P2 is an acquire. Under C++11 rules, this outcome is forbidden, and yet a trailing-sync implementation can't prohibit it. ARMv7 used such a mapping for its SC loads, and the only reason why it could prohibit the two-stale-reads outcome is because implementations chose the ldr; dmb ish mapping for acquire loads over the equivalent ldr; teq; beq; isb - this issue may have been why that mapping was chosen in the first place. The paper also outlined a variant of the same problem with loads, which is typically called IRIW (Independent Reads from Independent Writes):

// P0
st.sqc [x], 1

// P1
st.sqc [y], 1

// P2
ld.acq r0, [x] // reads 1
ld.sqc r1, [y] // reads 0

// P3
ld.acq r2, [y] // reads 1
ld.sqc r3, [x] // reads 0

Once again, the two stale reads can be caused by a lack of serialization points inserted between the loads in P2 and P3, and a trailing-sync implementation would fail to do that. So, the fix appears obvious: Mandate that all implementations strengthen their acquire operations, or use a leading-sync mapping. Not so fast. In "Repairing Sequential Consistency in C/C++11", Lahav et al. point out a problem with the leading-sync implementation for stores, though the corresponding Z6.U litmus test relies on an RMW operation, which I haven't covered yet. C++20 attempted to fix the highlighted problems in a Ptolemaic fashion: Instead of addressing the core issue, which is the unnecessarily strong formulation of RA, it created a bunch of special cases for SC atomics, and complicated the model even more. As is typically the case with such fixes, they failed to address all violations.

Consider I-RWC implemented with the following synchronizations:

// address = OLD_ADDRESS, looking_at = 0 at the start

// P0 (Writer)
st.sqc [address], NEW_ADDRESS
st.rel [clean_up], OLD_ADDRESS

// P1 (Reader)
st.sqc [looking_at], OLD_ADDRESS
ld.sqc r1, [address] // sees OLD_ADDRESS
// Check if the value stored at looking_at matches the value in r1
// Success. Use the address and set looking_at = 0 at the end

// P2 (Garbage Collector)
ld.acq r2, [clean_up]   // sees OLD_ADDRESS
ld.sqc r3, [looking_at] // sees 0
// Concludes P1 can no longer see OLD_ADDRESS
// and frees the associated memory

All memory operations on address and looking_at at are SC, and only the clean_up variable is accessed via RA synchronization. Here's an illustration of the cycle with simplified variable names and values:

If the st.sqc in P0 is implemented with a trailing sync, this outcome becomes impossible. Likewise, if the ld.sqc in P2 is implemented with a leading sync, the outcome is still impossible. However, on an architecture that intelligently places the serialization points dynamically (with the aforementioned flag bits), this outcome is allowed, and yet C++20 forbids it. A compiler implementation that eliminates redundant barriers can't possibly know which barrier to eliminate either, because even with full program visibility, it can't possibly enforce the rules of the model against all possible executions.

To my knowledge, this issue hasn't been raised yet, and it's not surprising given the fact that existing implementations don't exhibit the problem, yet failing to address it would unnecessarily prohibit efficient implementation of SC on nMCA hardware. This may have been one of the reasons why ARM gave up on nMCA for their implementation of dedicated SC an RA operations. C++ is simply too strict due to a flaw in its formulation of RA semantics that doesn't carefully specify that the only guarantee an acquire load can provide is the visibility of writes that causally-precede the release it read from. As the cycle demonstrates, the store that precedes the release operation isn't even read from after the acquire operation, and thus the RA pair does nothing of value besides local execution ordering. Formal models can get around this problem by expanding the edge pattern matching language I introduced in Causal Transitivity to create a path between everything that causally-precedes the release store, as well as including the reads-before relationship of the load after an acquire. Demanding that the path isn't a loop (irreflexive) is sufficient to model import of causal history. C++, with its "happens-before" and now "strong-happen-before" semantics, attempts to define the rules for RA operations via transitivity, which fails to take into account the subtleties of causality.

C++ doesn't even have a notion of cumulativity for release operations - it indirectly forces it through its rules about per-location total order and (since C++20) simply-happens-before relations. However, SC operations, which are also defined to have a total order, expose the same problem and force the implicit promotion of RA into SC. So the shortcut the specification takes to avoid talking about cumulativity ends up restricting valid hardware optimizations.

Relaxed atomics can also expose the illusion when taking advantage of store-to-load forwarding, the optimization that separates MCA from other-MCA:

// P0
st.sqc [x], 1
st.sqc [y], 2
ld.rlx r0, [y] // reads 1

// P1
st.sqc [y],1
st.sqc [x], 2
ld.rlx r1, [x] // reads 1

The cycle should be obvious if you examine the pseudo-instructions involved:

commit
st.rlx [y], 2
ld.rlx r0, [y]

There's no commit in between the two operations to prevent the load of y from being satisfied directly from the local store buffer. This litmus test is called 2+2W or WW+WW in the literature, though I prefer to call it SLF2, for "Store-to-Load Forwarding with 2 Writes". SLF1 is an alternative with only 1 write:

// P0
st.rel [x], 1
ld.sqc r0, [x] // reads 1
ld.sqc r1, [y] // reads 0

// P1
st.rel [y], 1
ld.sqc r2, [y] // reads 1
ld.sqc r3, [x] // reads 0

Once again, the first load in each processor is immediately satisfied from the local store buffer, and the update from the other processor is missed, creating a cycle in the execution graph, even though all the loads are sequentially consistent. SLF1 is also called SB+rfis in the literature, and stands for store buffering with multiple reads-from-internal edges, which correspond to the store-to-load forwarding I've been discussing. You might not be surprised to hear that C++20 forbids this outcome too, which is considered a defect in the standard. This is also the context for the Boehm quote at the beginning of the article.

You can go through all prior litmus tests I've discussed, as well as variations on the problems they highlight, to synthesize other examples where the partial order of SC operations participates in an overall cycle when weaker operations are included.

12.6 The Fundamental Problem

As you can see, defining sequential consistency in terms of a total or partial order within a framework that also permits less strict (weaker) atomic operations is a difficult task, especially when you allow those operations to connect otherwise separate SC graph clusters. C++11 attempted to define how release-acquire atomics can establish ordering between SC operations that are causally connected through a chain of weaker operations, which was proven unsound for some mappings by Manerkar et al. in 2016, and Lahav et al. in 2017. The 2016 paper by Batty et al. correctly recognizes that a partial order is sufficient for achieving sequential consistency, though it had similar issues as C++11, which Lahav et al. pointed out.

C++20 attempted to translate Lahav et al.'s work to the quasi-formal language of the C++ standard, and it kept the idea of a total order. However, the new strong-happens-before semantics are both cryptic and flawed, as they make correct hardware mappings highly inefficient. I won't get into the specifics of the C++11 and 20 models until I discuss formal modelling in a future article. Part 2 will be a stepping stone towards that, because it will cover other issues with the C++20 model.

What I will say is that I believe the weakness of SC loads, at least compared to reconcile ones, as well as the lack of commit loads and reconcile stores, make sequential consistency a highly cumbersome and inefficient tool for more advanced algorithms, where ordering of non-SC atomic operations is required - yet it's all languages based on the C++ memory models currently provide.

It may be tempting to assume that the implementation of SC atomics would be very similar to the implementation of commit-reconcile atomics, and I certainly wouldn't blame if you that's your takeaway from reading this section, because it's possible to implement sequential consistency with commit-reconcile operations, and it may even be desirable on a lot of platforms (for example, ARMv7 SC loads and stores in popular implementations work like reconcile loads and commit stores). However, Part 2 will introduce a synchronization framework that makes it clear the two could have very different underlying behaviour, and SC atomics shouldn't be tied to "global visibility" definitions.

The C++ memory model architects have a genuinely difficult task on their hands, especially when programming language semantics are mixed in. Moreover, it's hard to create useful abstractions without a solid grasp of the underlying machine, which is why I consider that the main contribution of this series.

13. Special Loads

There are several common patterns in parallel algorithms that come up often enough to justify additional hardware support. I'll explore two of them, and discuss how the current instruction set is unable to provide an efficient implementation.

13.1 Consume Semantics

The first pattern is publishing an update via a pointer swap, which is common in algorithms that don't rely on mutual exclusion to achieve synchronization. The entire purpose of the Linux kernel RCU implementation is to provide an efficient and safe way of doing these kinds of updates, but the architecture we've designed requires acquire semantics to ensure the data changes are propagated to whoever sees the pointer update. Issuing an acquire load will trigger a stall until all snapshotted invalidation are processed, but we may only care about a handful of those, especially in a system with a lot of hardware threads that are accessing shared memory. Ideally, we'd want an operation, which is typically called consume (abbreviated .cns) that tracks all loads whose address is computed based on the pointer address we're synchronizing on, and wait only for those invalidations instead of the entire snapshot. If this is too difficult to track architecturally, the processor could enter a mode where the addresses of subsequent loads are checked against the snapshot and stalling if an entry matches. This mode can be exited automatically when all entries in the snapshot are processed.

Consume semantics could also be seen as anti-pessimization technique, which means the implementation can treat it as an acquire operation if address dependencies aren't respected by the platform. This is the case in the LKMM, with its smp_read_barrier_depends(), because, historically, it had to support the infamous DEC Alpha, which didn't hide its unordered invalidation buffers from the user:

Although it is legal C11 to say “atomic_thread_fence(memory_order_consume)”, all known C11 implementations promote this to an acquire fence. Within the Linux kernel, this would have the undesirable effect of promoting rcu_dereference() to acquire as well. Linux therefore needs to continue defining smp_read_barrier_depends() as smp_mb() on DEC Alpha and nothingness elsewhere.

This promotion of consume operations to acquire ones by compilers is a language problem that I'll discuss later in the series, but, suffice it to say that aggressive compiler optimizations cause all kinds of headaches when it comes to establishing an efficient memory model. We won't be worrying about that yet.

Consume semantics are the brain child of Paul E. McKenney, and that's not surprising given the fact he also introduced the RCU mechanism Linux relies on, and is one of the chief architect of both the Linux Kernel Memory Model (LKMM), and the C++ memory model.

13.2 Verify Semantics

Speaking of Linux, and operating systems in general, a common way to publish updates, without potentially stalling the writer, is to issue speculative, racy, reads, and have a mechanism to detect when those reads raced with a writer, so they can be discarded.

This is the idea behind sequence locks, which I mentioned before. The writer in the typical sequence lock implementation increments the seqlock synchronization variable to an odd value to indicate it's performing an update, and then increments it again with a release store, making the value even, to signal a completed update. A reader can issue an receive read on seqlock to get the potentially incomplete data updates, and if it observes an even value, it may assume there's no ongoing write and speculatively read the data. The core problem with this algorithm is that a write may begin at the same time, and the invalidation for the synchronization variable could get stuck in the reader's invalidation queue. If the reader completes its speculative reads and issues a load of seqlockt hat hits a stale cache copy, it may assume the variable didn't change, so it didn't race with a writer.

With the current instruction set, a commit load can provide the necessary behaviour, because it will ensure prior writes aren't completed after the lock read, and it will also make sure the load itself checks the invalidation buffer for its address before completing. However, the commit part of operation is entirely unnecessary.

It's therefore useful to define a verify load, ld.vfy, which waits for prior loads to obtain their values, then searches for the corresponding address in the IB, and, if it finds it, it stall until that entry is processed before completing.

14. Prepare Semantics

We now have all operations needed for implementing a seqlock, except for the writer's version of "acquire". This is valuable in implementations with a single, fixed writer processor. For example, a processor which has been configured to service and publish updates for a specific interrupt request (IRQ), or a software thread pinned to a specific core. A proper operating system implementation of context switching would also ensure updates are globally visible before it swaps the current running thread, so a general writer thread would benefit from the same optimization.

I call this acquire-like operation a prepare store (.prp). The idea is simple: A single-writer sequence lock will first increment the synchronization counter to an odd value with st.prp, modify the protected data structure, and publish an even-value-increment with st.snd:

ld.rlx [r0], [seqlock] // assumed even
inc r0 // increment seqlock by 1
st.prp [seqlock], r0 // begin modification

// Modify the protected data structure

inc r0 // increment seqlock by 1 again
st.snd [seqlock], r0 // end modification

The architecture's job is to make sure that the st.prp modification is visible to any reader processor before any subsequent stores. This is necessary for a seqlock implementation to ensure that if any reads race with a write, the prepare modification is guaranteed to be visible to a corresponding verify load. A reconcile store is already capable of providing this functionality, but it's unnecessarily strict; Part 2 will cover the asynchronous way of handling this functionality.

15. Same-Location Causality

When I first learned about cache coherence protocols, I was very puzzled why you'd need relaxed atomics in the first place. Isn't the reader-writer lock supposed to guarantee that there will be no funny business with same-location ordering of operations?

The answer has to do with the difficulty of enforcing ordering of two consecutive loads on the same processor, especially in the context of pointer aliasing. You may have realized this already, but two consecutive loads are a staple in litmus tests because they act as external observers of write events.

Imagine a scenario where the same address (pointer) is stored in two different variables (ptr0 and ptr1), and one processor dereferences the two pointers in order, while another updates the location the pointers alias:

// P0
ld r0, [ptr0]
ld r1, [r0] // reads 1

ld r2, [ptr1]
ld r3, [r2] // reads 0

// P1
ld r4, [ptr0]
st [r4], 1

What sequence of events could lead to this outcome? Since parallel access is involved, the first problem you should suspect is the pointers residing in two different cache lines. Since, as far as the processor is concerned, there's no ordering that needs to be enforced between two independent locations, the line for ptr0 might not be present in the cache, or got evicted due to a conflict, so the pointer load and the load that depends on it get stalled. Meanwhile, the line for ptr1 is in the cache line, so it immediately completes. The last load is then free to observe the old value of the variable. Meanwhile, P1's invalidation reaches P0 and gets processed immediately. By the time the cache line for ptr0 is retrieved, the variable it points to has changed, so it returns new value.

CPUs have sophisticated memory ordering buffers (MOB) that resolve these types of conflicts, but an architecture might want to dispense with the complexity and not enforce the needed synchronization because it wants to, for example, serve loads in parallel. This was probably the case for Itanium (unconfirmed), which featured out of order execution of VLIW instruction bundles. The Itanium Processor Microarchitecture states:

"Reservation stations, reorder buffers, and memory ordering buffers are all replaced by simpler hardware for speculation."

The IA-64 developer's manual, which I referenced before, is also very explicit about this lack of ordering enforcement:

The dependency rules define the relationship between memory operations that access the same address. Specifically, the IA-64 architecture resolves read-after-write (RAW), write-after-read (WAR), and write-after-write (WAW) dependencies through memory in program order on the local processor.

Note the missing read-after-read (RAR) dependency, which C++11 requires for relaxed atomics. Itanium likely submitted its atomic requests to a separate unit, which did have the more complicated dependency enforcement. Although it took a long time for the Itanium mappings to be corrected, acquire loads were used instead of plain loads from the very beginning, and, as far as I know, Itanium was the only architecture at the time which exhibited this behaviour. This is why a separate relaxed atomic type was needed.

15.1 Load-Load Bugs

However, missing hardware features aren't the only reason to have an atomic type. Sometimes there are hardware bugs which need to be corrected after a lot of software is already written.

The seminal 2013 Herding Cats - Modelling, simulation, testing, and data-mining for weak memory paper, which is a highly recommended read for anyone interested in memory models, uncovered a lot of same-location load-load (load-load coherence) bugs:

Amongst the tests we have ran on ARM hardware, some unveiled a load-load hazard bug in the coherence mechanism of all machines. This bug is a violation of the coRR pattern shown in Sec. 4, and was later acknowledged as such by ARM, in the context of Cortex-A9 cores, in the note [arm 2011].10 Amongst the machines that we have tested, this note applies directly to Tegra 2 and 3, A5X, Exynos 4412. Additionally Qualcomm’s APQ8060 is supposed to have many architectural similarities with the ARM Cortex-A9, thus we believe that the note might apply to APQ8060 as well. Morever we have observed load-load hazards anomalies on cortex-A15 based systems (Exynos 5250 and 5410), on the cortex-A15 compatible Apple “Swift” (A6X) and on the Krait-based APQ8064, although much less often than on cortex-A9 based systems.

In the Cortex-A9 MPCore(TM) Programmer Advice Notice Read-after-Read Hazards, ARM recommended the following fix:

The workaround is to intercept all volatile memory reads in the compiler and issue a DMB instruction immediately afterwards. This ensures that all such accesses are ordered correctly.

The associated 761319: Ordering of Read Accesses to the Same Memory Location Might Be Uncertain errata states that the issue was caused by "some internal replay path mechanisms".

What we can infer from this statement is that if, for some microarchitectural reason, a load fails to complete, it may be sent around a "replay" path and tried again later. This, in effect, could produce the same aliasing race, but without even involving pointers.

Though not directly related, the Arm Cortex-A510 Core errata contain a number of race conditions involving operations by the same processors failing to enforce ordering because of a concurrent write, such as "2218950 Aliased loads and stores might cause an ordering violation", "1976290 A Tag Checked load might forward incorrect data", and "2230537 Load following SVE predicated load/store might cause ordering violation". As you can see, even if the cache coherence protocol functions as a reader-writer lock, it's still possible to have ordering violations.

On a side note, ARM's recommendation to change volatile mappings to preserve load-load coherence isn't unique. Compilers which support Itanium do the same. For example, gcc treats volatile accesses as relaxed atomics and issues ld.acq and st.rel respectively for them (Itanium doesn't have a dedicated relaxed atomic operation):

/* Unless the memory model is relaxed, we want to emit ld.acq, which
    will happen automatically for volatile memories.  */

/* Unless the memory model is relaxed, we want to emit st.rel, which
    will happen automatically for volatile memories.  */

Testing done by PostgreSQL developers revealed the Intel icc compiler does the same, even though the manual states the serialize-volatile option is turned off by default.

All of this is to say that, despite what language lawyers may argue, volatile very much is treated as a relaxed atomic operation by compiler developers, because they have a lot of legacy code to support, which had to be written before atomics were introduced. Linux kernel code, for example, wraps its volatile accesses with READ_ONCE and WRITE_ONCE macros.

16. AI Disclosure

GPT-5.6 Sol was used as a research assistant for this article. It dug up and summarized relevant references for me, so I could evaluate whether it's worth my time to read them. I didn't let it do any of the idea and visualization generation, writing, or proof reading, because its writing style doesn't suit me, and I can't be asked to go through mountains of feedback that I'm going to mostly ignore, so you'll have to bear with any grammar and punctuation mistakes. Above all, letting it do all the work decreases my sense of ownership, and thus motivation to work on articles.

17. Conclusion

I've now introduced all major ideas, and will continue to build upon them in follow-up articles. Read-modify-write operations require dedicated treatment in the context of the synchronization framework I'll introduce in Part 2, so they didn't make the cut for this introduction; same for fences and local dependencies.

Numbers in Base X Part 1: Rising and Falling Division

2025-09-01T00:00:00+01:00

When I was first introduced to polynomial arithmetic, my teacher said it's very similar to regular arithmetic. In fact, because you don't need to worry about carrying, performing addition, subtraction, and multiplication, is significantly easier. Are polynomials really that similar to numbers in base $x$ ? I felt compelled to find out, unaware of the fact that I was about to go down an unfathomably deep rabbit hole. This article will introduce you to various unusual approaches to dividing both polynomials and integers, and the algorithms we'll develop will serve as a foundation for many future explorations.

Preliminaries

In school, we learn that the position of each digit in the standard representation of an integer corresponds to the power of the associated base, usually $10$ , that multiplies it. For example: $123$ is really just $1 \cdot 10^2 + 2 \cdot 10^1 + 1 \cdot 10^0$ . I'll omit $10^0$ , which is just $1$ , and the power of $10^1$ from here on for the sake of brevity. Let's subtract $29$ from $123$ using the expanded representation:

$\begin{align*} 123 - 29 &= 1 \cdot 10^2 + 2 \cdot 10 + 3 - 2 \cdot 10 - 9\\ & = 1 \cdot 10^2 +(2-2) 10 +(3-9) \\ & = 1 \cdot 10^2 + 0 \cdot 10 - 6 \end{align*}$

If we had $x$ instead of $10$ in this example, we'd be done with the subtraction. However, we can only use digits $0$ to $9$ in the base $10$ representation, and we have to deal with the $-6$ by borrowing from the next highest power:

$\begin{align*} 123 - 29 &= 1 \cdot 10^2 + 0 \cdot 10 - 6 \\ & = (9 \cdot 10 + 1 \cdot 10) - 6 \\ & = 9 \cdot 10 + 10 - 6 \\ & = 9 \cdot 10 + 4 \\ & = 94 \end{align*}$

Multiplication is also straightforward, provided we remember its distributive property, which tells us we can multiply each term inside one bracket by the terms in the other:

$\begin{align*} 123 \cdot 29 &= (1 \cdot 10^2 + 2 \cdot 10 + 3) (2 \cdot 10 + 9)\\ & = 1 \cdot 10^2 (2 \cdot 10 + 9) + 2 \cdot 10 (2 \cdot 10 + 9) + 3 (2 \cdot 10 + 9)\\ & = 2 \cdot 10^3 + 9 \cdot 10^2 + 4 \cdot 10^2 + 18 \cdot 10 + 6 \cdot 10 + 27 \\ & = 2 \cdot 10^3 + 13 \cdot 10^2 + 24 \cdot 10 + 27 \end{align*}$

Once again, we're required to normalise the representation, meaning every multiple of a power of $10$ has to be less than $9$ . This is easily accomplished if we rewrite the "digits" as multiples of $10$ with a remainder:

$\begin{align*} 123 \cdot 29 & = 2 \cdot 10^3 + 13 \cdot 10^2 + 24 \cdot 10 + 27 \\ & = 2 \cdot 10^3 + (10 + 3) 10^2 + (2 \cdot 10 + 4) 10 + (2 \cdot 10 + 7) \end{align*}$

Multiply through and carefully regroup to get the final result:

$\begin{align*} 123 \cdot 29 &= 2 \cdot 10^3 + (10 + 3) 10^2 + (2 \cdot 10 + 4) 10 + (2 \cdot 10 + 7) \\ & = (2+1)10^3 + (3+2)10^2 + (4 + 2)10 + 7 \\ & = 3 \cdot 10^3 + 5 \cdot 10^2 + 6 \cdot 10 + 7 \\ & = 3567 \end{align*}$

I would've done this much faster using the notation and algorithms taught in elementary school, but it's not too difficult conceptually.

Division is where my younger self got stuck. Like many students, I had memorised the long division algorithm without understanding why it works, and was also vaguely aware that I could, in principle, subtract repeatedly. The latter option seemed way too inelegant, not to mention tedious, and the only way I could adapt standard long division to the expanded representation was to normalise after each multiplication and subtraction step. This really bothered me because I could perform all the other operations while treating the intermediary results as polynomials, only normalising at the end. What made the problem worse is that the polynomial division algorithm my teacher taught me required the use of rational numbers, and those are nowhere to be seen when dividing integers! I gave up on this approach to division eventually, but the urge to compare integers and polynomials would frequently resurface over the next 20 years. Enough with the navel-gazing, we've got quite a bit of work ahead of us before we get to play!

Polynomial Division

So how do we divide polynomials? The algorithms I was taught were pretty much identical to those presented on Wikipedia, but I'd like to offer an alternative perspective that, in my opinion, provides a lot more insight into what's happening.

Multi-Pass Division

Returning to our running example, this time in base $x$ and with reversed digits, let's try to divide $f(x):3x^2 + 2x + 1$ by $g(x):2x + 9$ . The strategy will be to rewrite $x^2$ and $x$ as multiples of the divisor, which requires us to first subtract all the lower degree (power of $x$ ) terms of the divisor from itself to isolate the leading term (the highest degree). Once we have a single term, we can clear out its coefficient (the leading coefficient) and multiply by an appropriate power of $x$ . Symbolically, the process looks like this:

$\begin{align*} f(x): 3x^2 + 2x + 1 &= 3(\frac{x}{2}(2x+9-9))+2x+1 \\ &= \frac{3x}{2}(g(x)-9) + 2x + 1 \\ &= \frac{3x}{2}g(x) + (2 - \frac{27}{2})x + 1 \\ &= \frac{3x}{2}g(x) -\frac{23}{2}x + 1 \end{align*}$

Take a moment to fully understand the symbolic manipulations because the entire algorithm is just a repeated application of this idea until we have a bunch of multiples of the divisor, and a remainder that has a lower degree. We still have one more term to rewrite:

$\begin{align*} f(x) &= \frac{3x}{2}g(x) -\frac{23}{2}(\frac{1}{2}(2x+9-9)) + 1 \\ &= \frac{3x}{2}g(x) -\frac{23}{4}(g(x)-9) + 1 \\ &= \frac{3x}{2}g(x) -\frac{23}{4}g(x) + 1 + \frac{207}{4} \\ &= \frac{3x}{2}g(x) -\frac{23}{4}g(x) + \frac{211}{4} \end{align*}$

This was the last step because the remaining term, $211/4$ , has a lower degree than the divisor, so we ran out of powers of $x$ to rewrite. We can factor out the $g(x)$ to get our final result:

$\begin{align*} f(x) = (\frac{3x}{2} -\frac{23}{4})g(x) + \frac{211}{4} \end{align*}$

We effectively rewrote $f(x)$ as a multiple of $g(x)$ , the quotient, with an added remainder polynomial of lower degree. Conceptually, this algorithm splits the divisor into the leading term and the lower degree terms, and, at each step, we slide the lower degree terms to the right, and subtract them from the corresponding numerator coefficients. Before the subtraction, each divisor coefficient must be multiplied by the coefficient of the rewritten power of $x$ , and divided by the divisor's leading coefficient. We can bring the resulting numerator coefficient under a common denominator, and, to avoid having to keep track of the various powers of the leading divisor coefficient, we can pre-emptively multiply all numerator coefficients, below the modified terms, by the divisor.

Here's what this algorithm looks like when translated into python code:

from fractions import Fraction # rational numbers support

def polyDivMultiPass(num: list, div: list) -> tuple[list, list]:
    divLen = len(div)
    assert divLen > 0, 'An empty divisor list implies division by 0'
    divDeg = divLen - 1 # the divisor's degree

    leadingZeroTerms = None
    for i in range(divDeg, -1, -1):
        if div[i] != 0:
            leadingZeroTerms = divDeg - i
            divLen -= leadingZeroTerms
            divDeg -= leadingZeroTerms
            break
    assert leadingZeroTerms is not None, 'All divisor coefficients were zero'

    numDeg = len(num) - 1 # numerator degree
    # Not exiting early won't impact correctness, but it can
    # convert integers to rationals unnecessarily.
    if numDeg - divDeg < 0:
        return [], num

    divLc = div[divDeg] # the divisor's leading coefficient
    divLcAccum = divLc
    divLcAccumPrev = 1

    output = num.copy() # copy the numerator to preserve the input arguments

    for i in range(numDeg, divDeg - 1, -1): # decreasing sequence
        prevCoeff = output[i]

        for k in range(1, divLen): # skip the leading divisor coefficient
            # The value of i-k will produce a decreasing sequence of indices,
            # corresponding to the lower degree divisor terms that flow out 
            # out of the rewritten term. Multiplying by divLc is done to
            # bring the values under a common denominator.
            output[i - k] = divLc * output[i - k] - prevCoeff * div[divDeg - k]

        # Multiply all remaining coefficient by the leading divisor coefficient
        for k in range(i - divLen, -1, -1):
            output[k] *= divLc

        output[i] = Fraction(prevCoeff, divLcAccum)
        # Store the previous power of the leading divisor coefficient so we can
        # divide the remainders by it. Eliminating this store requires a 
        # restructuring of the loop that will make the code harder to read.
        divLcAccumPrev = divLcAccum
        divLcAccum *= divLc

    # We didn't get to the remainders in the loop above, so make sure to
    # divide them as well.
    for i in range(divDeg):
        output[i] = Fraction(output[i], divLcAccumPrev)

    return output[divDeg:], output[:divDeg] # split into quotient and remainder

def printQuotRem(quotRem: tuple[list, list]):
    print('Quotient: [',', '.join([str(p) for p in quotRem[0]]),']'
          ' Remainder: [',', '.join([str(p) for p in quotRem[1]]),']')

I added "multi-pass" to the name of the function because we're doing multiple passes over the output list before we compute all coefficients. The input polynomials are specified as coefficient lists in increasing power order, which might look odd if you're not a programmer, but it makes adding higher powers of $x$ more efficient. I've also added a little helper function to print out the output of polyDivMultiPass. Let's divide $6 x^5+5 x^4+4 x^3+3 x^2+2 x+1$ by $7 + 8 x + 9 x^2$ using our code:

printQuotRem(polyDivMultiPass([1, 2, 3, 4, 5, 6], [7, 8, 9]))

Quotient: [ 872/2187, -10/243, -1/27, 2/3 ] Remainder: [ -3917/2187, -1972/2187 ]

Translating the result back to mathematical notation:

$\frac{2 x^3}{3}-\frac{x^2}{27}-\frac{10 x}{243}+\frac{872}{2187}, -\frac{1972 x}{2187}-\frac{3917}{2187}$

If you install the SymPy python library (enter pip install sympy in a command prompt or terminal), you can validate the output against its polynomial division algorithm:

from sympy import *

x = var('x')
n = poly(1 + 2 * x + 3 * x**2 + 4 * x**3 + 5 * x**4 + 6 * x**5)
d = poly(7 + 8 * x + 9 * x**2)
quotient, remainder = div(n, d)
print(quotient.all_coeffs(), remainder.all_coeffs())

[2/3, -1/27, -10/243, 872/2187] [-1972/2187, -3917/2187]

If you're a programmer, try to spot the redundant computations and see if you can come up with a better implementation.

A Change in Perspective

Doing multiple passes over the coefficient list is a straightforward adaptation of how I presented the division algorithm, but we can also look at the computations from the perspective of a single output coefficient! Let's say you wanted to compute the 3rd quotient coefficient directly, and, to make the analysis simpler, we can assume the divisor polynomial is monic, meaning its leading coefficient is $1$ . What information do we need to compute it? Well, if the divisor has degree $1$ , this quotient coefficient will obviously depend on the constant divisor term (the degree $0$ one) from the rewrite of the previous (2nd) quotient coefficient. In other words, we have to multiply the constant divisor coefficient by the previously computed quotient, and then subtract it from the 3rd highest numerator coefficient. Similarly, if the divisor was of degree $2$ , the 3rd quotient coefficient would depend on the previous $2$ quotient coefficients, but, this time, the first (highest) coefficient needs to be multiplied by the constant divisor coefficient, and the 2nd by the degree $1$ divisor coefficient. In general, a quotient coefficient will depend on, at most, $n$ previous coefficients, where $n$ is the degree of the divisor minus $1$ .

Feel free to use this idea to compute the quotient coefficients for the examples above if you're struggling to visualise what's going on. If it helps, what I see in my head is something similar to a linear recurrence, where an array of the negated lower degree divisor coefficients, expressed in increasing power order, are sliding against the numerator coefficients, in decreasing order; each step generates an entry in a quotient array, which is parallel to the numerator one. Where the quotient array and the sliding divisor overlap, we multiply the overlapping coefficient, sum all the products, and subtract the result from the corresponding numerator coefficient. If there are no overlapping entries in the arrays, you can assume the missing values are zero. Let's divide $[1, 2, 3, 4, 5, 6]$ by $[7, 8, 1]$ as an example:


	$6$ ,	$5$ ,	$4$ ,	$3$ ,	$2$ ,	$1$
$-7$ ,	$-8$
	$\boxed{6}$


	$6$ ,	$\underline{5}$ ,	$4$ ,	$3$ ,	$2$ ,	$1$
$-7$ ,	$-8$
	$6$ ,	$\boxed{-43}$

$\underline{5} - 8 \cdot 6 = -43$


$6$ ,	$5$ ,	$\underline{4}$ ,	$3$ ,	$2$ ,	$1$
$-7$ ,	$-8$
$6$ ,	$-43$ ,	$\boxed{306}$

$\underline{4} + 8 \cdot 43 - 7 \cdot 6 = 306$


$6$ ,	$5$ ,	$4$ ,	$\underline{3}$ ,	$2$ ,	$1$
	$-7$ ,	$-8$
$6$ ,	$-43$ ,	$306$ ,	$\boxed{-2144}$

$\underline{3} - 8 \cdot 306 + 7 \cdot 43 = -2144$

The values inside the boxes are the computed coefficients, and the underlined numbers are the numerators we need to subtract from in each step. Unfortunately, this approach doesn't work for the remainder coefficients, besides the highest degree one, because we're not doing any rewriting for them. The solution turns out to be very straightforward if we apply the same analysis we used for the quotient. The constant degree term of the remainder would only be affected by the last rewrite step, so we need to subtract the product of the constant quotient and divisor coefficients. The degree $1$ remainder coefficient will depend on the last $2$ quotients, but this time the constant quotient coefficient is multiplied by the degree $1$ divisor coefficient, and the degree $1$ quotient coefficient by the constant divisor coefficient. At this point you might recognise that the steps are similar to the quotient computation, except we're sliding the divisor against the quotient array in reverse, and are not using the previous output in each step:


$4$ ,	$3$ ,	$2$ ,	$\underline{1}$
$6$ ,	$-43$ ,	$306$ ,	$-2144$
			$-7$ ,	$-8$

$\underline{1} + 2144 \cdot 7 = \boxed{15009}$


$4$ ,	$3$ ,	$\underline{2}$ ,	$1$
$6$ ,	$-43$ ,	$306$ ,	$-2144$
		$-7$ ,	$-8$

$\underline{2} - 306 \cdot 7 + 2144 \cdot 8 = \boxed{15012}$

Depending on your maths background, you might've also noticed that we're effectively performing a partial discreet convolution of the coefficient lists to produce the values we need to subtract from the numerator. You can verify this using the numpy library in python (pip install numpy if you don't have it):

import numpy
print(numpy.convolve([7, 8, 1], [-2144, 306, -43, 6]))

[-15008 -15010      3      4      5      6]

This isn't surprising given how we could, if we wanted to, compute the remainder as $\mathrm{numerator}-\mathrm{quotient} \cdot \mathrm{divisor}$ , with the polynomial multiplication being a discrete convolution. The remainder algorithm above is simply an optimised version of this computation.

Modifying the monic algorithm to support non-monic divisors is also quite easy, because we can factor out the leading coefficient of the divisor, and bring it into the numerator:

$\begin{align*} \frac{3x^2 + 2x + 1}{2x + 9} &= \frac{3x^2 + 2x + 1}{2 (x + \frac{9}{2})} \\ &= \frac{\frac{3}{2}x^2 + \frac{2}{2}x + \frac{1}{2}}{x + \frac{9}{2}} \end{align*}$

The problem is now reduced to monic division with rational coefficients. We can use python's Fraction class to convert all coefficients to fractions before passing them to a monic division function (or implement that function so it uses rationals internally), but I find it more interesting to create an algorithm similar to polyDivMultiPass, which computes the output numerators directly, and only generates the appropriate rational coefficients at the very end.

Using the multi-pass algorithm for inspiration, it's obvious that each numerator should be multiplied by an increasing power of the leading divisor coefficient, which we can label $L$ . We must also bring the previous coefficients under a common denominator, and, because each of their denominators is a consecutive power of $L$ , we can simply multiply each one by an increasing power of $L$ , starting from $0$ for the last rewritten term, and going as far back $L^{k-1}$ , where $k$ is the degree of the divisor. Note how each coefficient of the divisor will always be multiplied by the same power of $L$ , meaning we can precompute and cache their product.

The numerators in the remainder computation should all be multiplied by the same power of $L$ , and the subtracted quotient numerators should be adjusted in the same way as the quotient computation, except the order is reversed. Unfortunately, precomputing the multiplier doesn't work for remainders, except for the leading coefficient. Bringing it all together, we get the following python function:

def polyDivRecurrence(num: list, div: list) -> tuple[list, list]:
    divLen = len(div)
    assert divLen > 0, 'An empty divisor list implies division by 0'
    divDeg = divLen - 1 # the divisor's degree

    leadingZeroTerms = None
    for i in range(divDeg, -1, -1):
        if div[i] != 0:
            leadingZeroTerms = divDeg - i
            divLen -= leadingZeroTerms
            divDeg -= leadingZeroTerms
            break
    assert leadingZeroTerms is not None, 'All divisor coefficients were zero'

    numDeg = len(num) - 1 # numerator degree
    diffDeg = numDeg - divDeg
    if diffDeg < 0:
        return [], num

    divLc = div[divDeg] # the divisor's leading coefficient
    # Precompute the required powers of the leading divisor coefficient and the
    # scaled lower degree divisor coefficients to keep it consistent with the 
    # pen-and-paper notation. Consumes more memory but avoids redundant computations.
    divLcAccum = divLc
    divLcPowers = [1, divLc]
    for i in range(diffDeg):
        divLcAccum *= divLc
        divLcPowers.append(divLcAccum)

    divDegMinus1 = divDeg - 1
    divScaled = [div[divDegMinus1]] # the divisor must be non-empty as a precondition
    for i in range(divDeg - 2, -1, -1): # avoiding list comprehensions for clarity
        divScaled.append(divLcPowers[divDegMinus1 - i] * div[i])

    quotientLen = diffDeg + 1
    # pre-allocate the lists, because we'll fill them backwards
    quotientNums = [0] * quotientLen
    quotients = [0] * quotientLen

    for i in range(numDeg, divDeg - 1, -1):
        iRev = numDeg - i # incrementing index; could `enumerate` the range too
        # Fill the quotient backwards to avoid having to reverse the list; the
        # index is off by 1 to avoid unnecessary subtractions in the next loop.
        quotientIndex = quotientLen - iRev 

        quotNum = num[i] * divLcPowers[iRev] # scale the associated numerator coefficient
        for k in range(0, min(iRev, divDeg)):
            quotNum -= divScaled[k] * quotientNums[quotientIndex + k]

        quotientIndex -= 1 # the quotient index is off by one here, fix it
        quotientNums[quotientIndex] = quotNum
        quotients[quotientIndex] = Fraction(quotNum, divLcPowers[iRev + 1])

    remainders = []
    for i in range(divDeg):
        remNum = num[i] * divLcAccum
        for j in range(i + 1):
            # The values of quotientNums[j] * divLcPowers[j] could be precomputed if the
            # remainder degree tends to be very large. Would use more working memory.
            remNum -= quotientNums[j] * divLcPowers[j] * div[i-j]
        remainders.append(Fraction(remNum, divLcAccum))

    return quotients, remainders

Pen and Paper

If you don't have access to a computer, I find that the notation inspired by this alternate algorithm is much more convenient when dividing large polynomials. The pen-and-paper example should also help clarify what the code does. Let's divide $6 x^5+5 x^4+4 x^3+3 x^2+2 x+1$ by $9 x^2+8 x+7$ as an example. The same "slide the divisor against the quotient coefficients" idea still applies, but we need to scale by the appropriate powers of $L$ in each step. Start by computing all powers of $L=9$ needed for the division, and list them in increasing power order. The degrees of the numerator and divisor are $5$ and $2$ respectively, meaning we'll need $5 - 2 + 1 = 4$ powers of $9$ :

9, 81, 729, 6561

If you have access to a calculator, you can type $9 * 9$ , and then press the equals key $3$ times to get all the powers of $9$ . Next, list all the negated lower degree coefficients of the divisor in decreasing power order, then multiply each by the power of 9 above and one entry to left - the first value will be multiplied by 1 implicitly. This means we get $-8 \cdot 1$ and $-7 \cdot 9 = -63$ . You can do it all in one step, placing the results on the next line, but this time list them in increasing power order to avoid getting disoriented once we start calculating the quotient coefficients. To wrap up the preparations, you should bring the highest degree numerator coefficient down to the third line, resulting in the following:

9, 81, 729, 6561
-63, -8
6

Our first quotient coefficient is $6/9 = 2/3$ , and to compute the next one we evaluate one step of our recurrence relation by multiplying the next highest term of the numerator, which is $5$ , by the lowest power of $9$ in the first row, and then adding the result of the linear recurrence formed by the 2nd and 3rd row. To avoid having to write a bunch of leading zeros in the first row, you can implicitly multiply by $0$ by ignoring the $-63$ entry. This gives us the next value in the third row: $5 \cdot 9 - 8 \cdot 6 = -3$ :

9, 81, 729, 6561
-63, -8
6, -3

This makes $-3/81 = -1/27$ our next quotient coefficient. Continue by bringing down the $4$ from the numerator, multiply it by the next power of $9$ , which is $81$ , and add the result of the linear recurrence: $4 \cdot 81 + 8 \cdot 3 - 63 \cdot 6 = -30$ :

9, 81, 729, 6561
-63, -8
6, -3, -30

The third quotient coefficient is $-30/729$ , or $-10/243$ . Identifying the inputs for the linear recurrence is quite easy, you start from the last computed value in the third row, and pair it with the corresponding value from the second row. For the last quotient coefficient, that's $8 \cdot 30 + 3 \cdot 63$ , which we add to $3 \cdot 729$ to get the last quotient numerator, $2616$ :

9, 81, 729, 6561
-63, -8 
6, -3, -30, 2616

Don't forget to divide by 6561 to get the final quotient coefficient: $2616/6561 = 872/2187$ . I like to underline the denominator value from the first row after each coefficient computation so I don't lose track of where I am.

Computing the leading coefficient of the remainder can be done using the same process as the quotient coefficients. Bring down the 2 from the numerator and add the recurrence: $2 \cdot 6561 - 8 \cdot 2616 + 63 \cdot 30 = -5916$ . I like to place the result in line with the quotient, separated by a vertical bar:

9, 81, 729, 6561
-63, -8
6, -3, -30, 2616    |    -5916

Divide by 6561 to get $-5916/6561 = -1972/2187$ .

I visualise the remainder computation as a table where the first row comes from the low degree coefficients of the quotient, as many of them as there are lower degree terms in the divisor. The second row is comprised of powers of the leading divisor coefficient in decreasing power order, and the third contains the negated lower degree divisor coefficients in increasing power order.

-30, 2616
  9,    1
 -7,   -8

Computing a coefficient of the remainder is now a matter of multiplying all values in the same column and summing the results: $30 \cdot 9 \cdot 7 - 2616 \cdot 1 \cdot 8 = -19038$ This value is then added to the corresponding scaled numerator coefficient: $2 \cdot 6561 - 19038 = -5916$ . It just so happens that this computation coincides with what we did for the quotient. The next coefficient is computed by shifting the bottom row to the right, discarding the coefficient that went out of the table and filling the new empty spot with 0:

-30, 2616
  9,    1
  0,   -7

Notice how the first two rows don't chage. This means the values can be pre-computed! Carry out the same procedure by bringing down the last numerator coefficient: $1 \cdot 6561 - 2616 \cdot 1 \cdot 7 = -11751$ :

9, 81, 729, 6561
-63, -8
6, -3, -30, 2616    |    -5916, -11751

The constant term of the remainder is therefore $-11751/6561 = -3917/2187$ . If the divisor degree was higher, we would've kept shifting the bottom row to the right until we filled it with zeros. With a little bit of practice, you can do this without explicitly constructing the table, though I prefer to write the pre-multiplied quotients in the 4th row, because I find it less error-prone when the remainder has a large degree:

9, 81, 729, 6561
-63, -8
6, -3, -30, 2616    |    -5916 , -11751
-270, 2616

Your phone's calculator can make quick work of each step in the computation, or you can use a separate sheet of paper to keep the bookkeeping notation above nice and tidy. I find this notation significantly easier to work with compared to synthetic division, though, nowadays, I mostly let the computer do the work for me.

If you're planning to teach this notation to someone else, I implore you to not skip the intuition behind its construction, as is often the case in school maths classes. It would be fairly difficult for students to reverse engineer the logic behind such a compact notation, and it encourages mindless memorisation.

Rising Division

One aspect of polynomial division that always bothered me was the focus on the leading term. It makes sense in the context of integer long division, but polynomials are different in one important aspect: $x$ can be less than $1$ , and when it gets very close to $0$ , arbitrarily close, if you will, the contribution of the higher powers of $x$ become vanishingly small. In fact, close to $0$ , the smaller the power of $x$ , the more it contributes to the overall value! The idea behind this vague notion of "close to $0$ " is that, for any polynomial, you can pick a value for $x$ , at and below which the absolute size of every term strictly decreases as the exponent of $x$ increases. I've assumed positive $x$ for simplicity, but the same idea applies for negative values, and, when you include those, it's more appropriate to think of a range centred on zero instead.

So, how do we divide polynomials under the assumption that the constant term is actually the leading one? Simple, we apply the same rewriting process as regular division, but starting from the low degree terms instead. Let's see how that works for our first polynomial division example, where we divided $f(x): 3x^2 + 2x + 1$ by $g(x): 2x + 9$ :

$\begin{align*} f(x): 3x^2 + 2x + 1 &= 3x^2+2x+1(\frac{1}{9}(9 + 2x - 2x)) \\ &= 3x^2+2x+(\frac{1}{9}(g(x) - 2x)) \\ &= 3x^2 + (2 - \frac{2}{9})x + \frac{1}{9}g(x) \\ &= 3x^2 + \frac{16}{9}(\frac{x}{9}(9 + 2x - 2x)) + \frac{1}{9}g(x) \\ &= 3x^2 + \frac{16}{9}(\frac{x}{9}g(x) - \frac{2}{9}x^2)) + \frac{1}{9}g(x) \\ &= (3-\frac{32}{81})x^2 + \frac{16x}{81}g(x) + \frac{1}{9}g(x) \\ &= \frac{211x^2}{81} + (\frac{16x}{81} + \frac{1}{9})g(x) \end{align*}$

An easy way to verify this computation is to reverse the list of coefficients passed to our standard division function, then reverse the output again:

def polyDivRise(n: list, d: list) -> tuple[list, list]:
    quotient, remainder = polyDivMultiPass(list(reversed(n)), list(reversed(d)))
    quotient.reverse()
    remainder.reverse()
    return quotient, remainder

printQuotRem(polyDivRise([3, 2, 1], [2, 9]))

Quotient: [ 1/9, 16/81 ] Remainder: [ 211/81 ]

Even though it's inefficient, polyDivRise is good enough for our current needs, and I like the fact that it explicitly demonstrates its connection to the standard algorithm. Is this type of polynomial division actually useful or merely a curiosity? In my opinion, the most important application of rising division can be found in calculus and analysis, because it allows us to divide truncations of two infinite series. I'll show you an example later on!

Naming

When I first stumbled upon this idea I wanted to find the name mathematicians have assigned to it, but my search yielded no results. I eventually stumbled upon "division by increasing power order", mentioned in the documentation for the Giac computer algebra system, which prompted me to search French Wikipedia, where I finally found a proper description of this type of division. The name was too verbose for my taste, so I decided to call it rising polynomial division instead, because we rewrite starting from the lowest degree term; this line of reasoning suggested an alternative name for standard polynomial division - falling polynomial division. Even though it's effectively the same algorithm, I'll use these names from now on to differentiate between the two rewriting directions.

Expansion

Another polynomial division rule that always bothered me was the fact that we had to stop after rewriting the numerator term whose degree matched the degree of the denominator's leading term. The algorithm for computing decimal expansions of rational numbers was taught in primary school, so, by this point, I was quite comfortable with the idea of an infinite process. As a reminder, the decimal expansion of, say, $1/7$ is:

$\frac{1}{7} = 0.\overline{142857}$

The line above the fractional digits is called an overline and is used to indicate that those digits will repeat indefinitely. For practical reason, the repeating part is often omitted because even relatively small denominators might require the computation of hundreds of digits before the repeating pattern is established. The algorithm is fairly straightforward, and involves repeated multiplication by $1$ , or $10/10$ to be specific:

$\begin{align*} \frac{1}{7} &= \frac{1 \cdot 10}{7 \cdot 10} = \frac{7 + 3}{7 \cdot 10} = \frac{1}{10} + \frac{3}{7 \cdot 10} \\ &= \frac{1}{10} + \frac{3 \cdot 10}{7 \cdot 10^2} = \frac{1}{10} + \frac{4 \cdot 7 + 2}{7 \cdot 10^2} \\ &= \frac{1}{10} + \frac{4}{10^2} + \frac{2}{7 \cdot 10^2} \end{align*}$

Computing the first $2$ digits like this is straightforward but cumbersome, which is why schools usually teach a more compact notation that takes advantage of the fact that at each step we multiply the previous remainder by $10$ and divide by the denominator to decompose it into a quotient and remainder. The quotient is output as a digit, and the remainder is carried into the next step:

$\begin{align*} 1 &= \underline 0 \cdot 7 + \boxed 1 \\ \boxed 1 \cdot 10 &= \underline 1 \cdot 7 + \boxed 3 \\ \boxed 3 \cdot 10 &= \underline 4 \cdot 7 + \boxed 2 \\ \boxed 2 \cdot 10 &= \underline 2 \cdot 7 + \boxed 6 \\ \boxed 6 \cdot 10 &= \underline 8 \cdot 7 + \boxed 4 \\ \boxed 4 \cdot 10 &= \underline 5 \cdot 7 + \boxed 5 \\ \boxed 5 \cdot 10 &= \underline 7 \cdot 7 + \boxed 1 \end{align*}$

The numbers with boxes around them are the remainders that we carry forward into next line, and the underlined numbers are the output digits of the expansion. You can omit the divisor, or replace it with an easy to write symbol, once you're comfortable with the procedure. In the decimal notation, each digit after the dot is implicitly multiplied by a negative power of $10$ that corresponds to the index of the digit relative to the dot. For example, $0.1428$ is a compact representation of the following:

$0.1428 = \frac{1}{10} + \frac{4}{10^2} + \frac{2}{10^3} + \frac{8}{10^4}$

This is very similar to the implicit positive powers of 10 for regular integers. Given how each digit is supposed to be less than $10$ , how can we be sure that quotient after each division will produce a valid digit? Fortunately, the proof is quite simple and it relies on the observation that, by definition, each remainder is strictly smaller than the divisor. Suppose the divisor is $D$ and n-th remainder is $R_n$ . We can form an equality based on the definition of a remainder $R_n < D$ , then divide both sides by $D$ , followed by a multiplication by $10$ :

$\begin{align*} R_n &< D \\ \frac{R_n}{D} &< 1 \\ \frac{R_n \cdot 10}{D} &< 10 \end{align*}$

The left-hand side is how we compute the quotient and remainder, and we can always ensure the previous $R$ is less than the divisor by reducing the numerator with long division first to compute the integer part. This is a rather elementary proof by induction which, sadly, is usually not taught when decimal expansion is introduced. Note how it didn't matter that we multiplied both sides by 10. We could've chosen any other number and the inequality would still hold, which means we can compute expansions in any base! Here's $1/7$ in base $2$ :

$\begin{align*} 1 &= \underline 0 \cdot 7 + \boxed 1 \\ \boxed 1 \cdot 2 &= \underline 0 \cdot 7 + \boxed 2 \\ \boxed 2 \cdot 2 &= \underline 0 \cdot 7 + \boxed 4 \\ \boxed 4 \cdot 2 &= \underline 1 \cdot 7 + \boxed 1 \\ \end{align*}$

which gives us $0.\overline{001}_2$ . The little 2 subscript is the standard notation for specifying the base of the number. Turning this algorithm into code is very easy:

def intDivFalling(n: int, d: int, b: int = 10, termCount: int = 10) -> tuple[int, list]:
    expansion = []

    integerPart = n // d
    n %= d

    for _ in range(termCount):
        n *= b
        expansion.append(n // d)
        n %= d

    return integerPart, expansion

print(intDivFalling(1, 13))

(0, [0, 7, 6, 9, 2, 3, 0, 7, 6, 9])

Given how we can expand in any base $x$ through multiplication by $x/x$ , why not do the same with polynomial falling division?

Rising and Falling Expansions

Let's apply our rational number expansion intuition to the falling division of $f(x): x$ by $g(x): x^2 - x - 1$ :

$\begin{align*} f(x): x &= \frac{x^2}{x} = \frac{(x^2 - x - 1) + x + 1}{x}=\frac{g(x) + x + 1}{x} \\ &= \frac{g(x)}{x} + \frac{(x+1)x}{x^2} = \frac{g(x)}{x} + \frac{x^2 + x}{x^2} \\ &= \frac{g(x)}{x} + \frac{g(x) + 2x + 1}{x^2} = \frac{g(x)}{x} + \frac{g(x)}{x^2} + \frac{2x+1}{x^2} \\ &= \frac{g(x)}{x} + \frac{g(x)}{x^2} + \frac{2x^2+x}{x^3} = \frac{g(x)}{x} + \frac{g(x)}{x^2} + \frac{2 \cdot g(x)}{x^3}+ \frac{3x+2}{x^3} \end{align*}$

You could continue doing this by hand, but, if you're lazy like me, you might already be thinking of ways to reuse our polynomial division code. The good news is that you don't even need to touch the code! Repeatedly multiplying by $x/x$ and dividing is equivalent to multiplying the numerator by a power of $x$ that corresponds to the number of terms you want to compute, and then applying the standard division algorithm. All you have to do at the end is divide the result by the same power of $x$ to preserve the equality:

$\begin{align*} f(x): x &= \frac{x^2 \cdot x^3}{x^4} = \frac{(g(x)+x+1)x^3}{x^4} = \frac{(g(x)x+x^2+x)x^2}{x^4} \\ &=\frac{(g(x)x + g(x)+2x+1)x^2}{x^4} = \frac{(g(x)x^2 + g(x)x+2x^2+x)x}{x^4} \\ &=\frac{(g(x)x^2 + g(x)x+2g(x) + 3x + 2)x}{x^4} \\ &=\frac{g(x)x^3 + g(x)x^2 +2g(x)x + 3x^2 + 2x}{x^4} \\ &=\frac{g(x)x^3 + g(x)x^2 +2g(x)x + 3g(x) + 5x+3}{x^4} \\ &=g(x)(\frac{1}{x}+\frac{1}{x^2} + \frac{2}{x^3} + \frac{3}{x^4}) + \frac{5x + 3}{x^4} \end{align*}$

Multiplying by powers of $x$ is equivalent to prepending zeros in the coefficient list that represents polynomials:

n = [0] * 10 + [0, 1] # compute 10 terms by prepending 10 zeros, equal to a multiplication of x^10
d = [-1, -1, 1] # x^2 - x - 1
print(polyDiv(n, d))

[55, 34, 21, 13, 8, 5, 3, 2, 1, 1]

Applying the division by $x^{10}$ to undo the previous multiplication we get:

$\frac{1}{x}+\frac{1}{x^2}+\frac{2}{x^3}+\frac{3}{x^4}+\frac{5}{x^5}+\frac{8}{x^6}+\frac{13}{x^7}+\frac{21}{x^8}+\frac{34}{x^9}+\frac{55}{x^{10}}$

You might recognise the numerators as the Fibonacci numbers, and it shouldn't surprise you that they came out given the previous discussion about linear recurrences. The division turned into a pure linear recurrence because the leading divisor coefficient is $1$ , and the lower degree coefficients were filled with zeros, so there was nothing to subtract from when computing the quotient coefficients.

Rising division is even simpler because we don't need to worry about adjusting the powers of $x$ at the end. Let's apply it to a special case of the geometric series formula:

$\frac{1}{x-1}$

This time we need to append zeros instead because the list will be reversed:

num = [1] + [0]*10
div = [1, -1]
print(polyDivRise(num, div))

[1, 1, 1, 1, 1, 1, 1, 1, 1, 1]

Which corresponds to:

$x^9+x^8+x^7+x^6+x^5+x^4+x^3+x^2+x+1$

I promised to give you a calculus example earlier, so let's take a few terms from the $\sin(x)$ and $\cos(x)$ series expansions that we've seen before, and use them to compute the first few terms of the $\tan(x)$ series:

$\tan(x)=\frac{\sin(x)}{\cos(x)} \approx\frac{x -\frac{x^3}{6}+\frac{x^5}{120}-\frac{x^7}{5040}}{1-\frac{x^2}{2}+\frac{x^4}{24}-\frac{x^6}{720}}$

from fractions import Fraction as Q

num = [0, Q(1), 0, Q(-1, 6), 0, Q(1, 120), 0, Q(-1, 5040)] + [0]*6 # sin(x) series
div = [Q(1), 0, Q(-1, 2), 0, Q(1, 24), 0, Q(-1, 720)] # cos(x) series
printQuotRem(polyDivRise(num, div))

Quotient: [ 0, 1, 0, 1/3, 0, 2/15, 0, 17/315 ] Remainder: [ 0, 331/15120, 0, -13/6300, 0, 17/226800 ]

The quotient gives us the correct expansion up to the 7th power:

$\tan(x) \approx\ x + \frac{x^3}{3} + \frac{2 x^5}{15}+ \frac{17 x^7}{315}$

The general formula for the tangent series coefficients is quite complicated, because it involves the Bernoulli numbers, and yet we can compute as many terms as we want with a simple algorithm! You can find more efficient formulas for the coefficients of various series by carefully studying the behaviour of rising division, and the patterns in each iteration of its computation. You can also use both rising and falling expansions to approximate the values of tricky integrals.

If you're familiar with calculus you might've realised that our algorithms are producing an infinite series approximation of a rational function. You might also suspect that the series will diverge around the zeros (poles) of the divisor, which, combined with a simple divergence testing method like the ratio test, can give give us one of the oldest polynomial root-finding algorithms - Bernoulli's Method. Optimisations based on this idea underlie many modern root-finding algorithms, which will be the subject of a future article.

We've only scratched the surface of the beautiful structure that underlies these simple algorithms, so feel free to play around with the code, evaluate it symbolically in terms of the roots of the divisor, and look for patterns.

Future Explorations

In future articles, we'll explore how expressing linear recurrences as matrix operations will allow us to create an important bridge between linear algebra and polynomials through the Companion matrix, whose eigenvalues can be used to approximate the roots of a polynomial. We'll also see how any term in a linear recurrence can be directly computed through a formula that depends on the roots of the polynomial division that corresponds to it. This also provides a natural connection between polynomials and a certain class of pseudo-random number generators.

Reversing polynomials expansion will turn out to be important for the efficient implementation of certain error correcting codes, the inversion of Toeplitz matrices, and even rational approximation.

Polynomial division leads to an algorithm for solving systems of polynomial equations, which, in turn, provides the tools necessary for a deep exploration of the theory of algebraic numbers with minimal background knowledge requirements. It turns out that the same ideas can be applied to the problem of partial fraction decomposition.

Division also turns out to be important in explaining what it means to change the basis of a polynomial, and how that relates to various exotic number bases. The subjects of interpolation and approximation theory will also make an appearance during this investigation.

Integers and rational numbers will closely accompany us at each step along the way, providing context and inspiration, particularly when it comes to data compression. This reminds me, before I wrap up, there's one last key tool missing from our division algorithms toolkit.

Rising Integer Division

After playing around with polynomial expansions for a while, I started wondering if there's an integer equivalent for rising division. Falling polynomial division was inspired by the classic integer expansion algorithm I learned in school, so maybe I can go in the opposite direction, and adapt the polynomial rewriting approach for integers! Let's try this idea with our old friend, $1/7$ . Can I rewrite the $1$ as a sum of a digit, which is less than $10$ , multiplied by $7$ , and some remainder? We have a lot of choices if the remainder is negative, but given the fact that we'll continuously rewrite this remainder, it better be a multiple of the base, which will be $10$ in this example, so we can get an increasingly larger base exponent. These requirements correspond to the following symbolic constraints:

$\begin{cases} 7 x + 10 y = 1 \\ 0 \le x < 10 \end{cases}$

Is it even possible to find solutions to the equation? If $7$ and $10$ had a common factor $m$ , greater than $1$ , we could rewrite the equation as $m (a x + b y) = 1$ , for some $a$ and $b$ , which obviously has no integer solutions because $m > 1$ . I picked $7$ and $10$ so they're relatively prime (coprime), meaning their greatest common factor is 1, but if we were trying to find an expansion for, say, $1/15$ in base $10$ , we'd fail, because $15$ and $10$ have $5$ as a common factor. This is a rather serious limitation compared to the usual expansion, though we don't need to worry about it if we pick a prime number as the base, and factor out all multiples of the base from the denominator before we begin the expansion.

Putting this limitation aside, how do we find appropriate values for $x$ and $y$ ? The brute-force approach here would be to simply try out increasingly larger values for $x$ until the product has a lowest digit equal to $1$ . It only takes $2$ attempts to arrive at $7 \cdot 3 = 21$ , which means we can subtract $2 \cdot 10$ to get 1, giving us $x = 3$ and $y = -2$

A more elegant approach relies on the observation that we can rewrite $10$ as $7 + 3$ , which allows us to reduce the size of the coefficients we're considering: $1 = 7 x + (7 + 3) y = 7 (x + y ) + 3 y$

We can repeat this process until one of the outer coefficients becomes $1$ :

$1 = (3 \cdot 2 + 1) (x + y ) + 3 y = (x + y) + 3(y + 2 (x + y))$

One of the multipliers became 1, so at this point we have two obvious options for its corresponding expression: $- 2 + 3 \cdot 1$ or $4 - 3 \cdot 1$

The first gives us:

$\begin{cases} -2 &= x + y \\ 1 &= y + 2 (x + y) = y + 2 (-2) \end{cases}$

which we can solve through simple substitution to find $y = 5$ and $x=-7$ . You can check for yourself that the second option results in $x = 13$ and $y = -9$ . Neither of these satisfies the $0 \le x < 10$ inequality, but we can easily fix that by rewriting $-7$ as $3 - 10$ ; expand and re-collect the terms:

$\begin{align*} (3 - 10) \cdot 7 + 5 \cdot 10 &= 3 \cdot 7 + (5 - 7) \cdot 10 \\ &= \boxed{3 \cdot 7 + (-2) \cdot 10 = 1} \end{align*}$

Depending on your maths background, you might've recognised the rewriting trick as an application of the Extended Euclidean algorithm (EEA), and the equation $7 x + 10 y = 1$ as a special case of Bézout's identity. The EEA will be our constant companion throughout the upcoming series of articles, and we'll explore it in more detail in part 2!

With our $x$ and $y$ coefficients in hand, we can now rewrite $1/7$ as:

$\frac{(3 \cdot 7 - 2 \cdot 10)}{7} = 3 + \frac{(-2 \cdot 10)}{7}$

To proceed from here we need to repeat the same process for $-2$ , which is similar to our remainder term in the regular expansion algorithm. Because we already have a way to express $1$ , let's just multiply that equation on both sides by $-2$ :

$-2 = (-6) \cdot 7 + 4 \cdot 10$

Negative digits aren't allowed in the usual decimal expansion, so let's express $-6$ as $4 - 10$ , which allows us to combine the $-10$ term with the $4 \cdot 10$ one:

$\begin{align*} -2 &= (4 - 10) 7 + 4 \cdot 10 \\ &= 4 \cdot 7 - 10 \cdot 7 + 4 \cdot 10 \\ &= 4 \cdot 7 + (4 - 7) 10 \\ &= 4 \cdot 7 + (-3) 10 \end{align*}$

We've finally arrived at an appropriate expression for $-2$ , which we can substitute back into our expansion:

$\begin{align*} 3 + \frac{-2 \cdot 10}{7} &= 3 + \frac{(4 \cdot 7 + (-3)10) 10}{7} \\ &= 3 + 4 \cdot 10 + \frac{-3 \cdot 10^2}{7} \end{align*}$

From here on we just repeat the same process, but with $-3$ instead:

$\begin{align*} -3 &= -9 \cdot 7 + 6 \cdot 10 \\ &= (1 - 10) 7 + 6 \cdot 10 \\ &= 1 \cdot 7 + (6 - 7) 10 \\ &= 1 \cdot 7 + (-1) 10 \end{align*}$

Rewrite the rational term of the expansion once again:

$\begin{align*} \frac{-3 \cdot 10^2}{7} &= \frac{(1 \cdot 7 + (-1) 10) 10^2}{7} \\ &= 1 \cdot 10^2 + \frac{-1 \cdot 10^3}{7} \end{align*}$

Our next digit is $1$ and rewriting $-1$ is equally straightforward:

$\begin{align*} -1 &= -3 \cdot 7 + 2 \cdot 10 \\ &= (7 - 10) 7 + 2 \cdot 10 \\ &= 7 \cdot 7 + (2 - 7) 10 \\ &= 7 \cdot 7 + (-5) 10 \end{align*}$

At this point you might be starting to see the pattern. The power of $10$ increases by $1$ and the corresponding coefficient (digit) is $7$ . We can express $-5 \cdot 3$ as $5 - 2 \cdot 10$ , which obviously makes the digit after it $5$ . while the next remainder is $5 \cdot 2 - 2 \cdot 7 = -4$ . Our expansion so far looks like this:

$3 + 4 \cdot 10 + 1 \cdot 10^2 + 7 \cdot 10^3 + 5 \cdot 10^4 + (-4) 10^5 / 7$

We can follow a similar logic for $-4 \cdot 3 = (8 - 2*10)$ , giving us digit $8$ and remainder $4 \cdot 2 - 2 \cdot 7 = -6$ . Repeat one more time: $-6 \cdot 3 = 2 - 2 \cdot 10$ , which gives us digit $2$ and remainder $6 \cdot 2 - 2 \cdot 7 = -2$ . Perhaps unsurprisingly, we eventually end up with a remainder that we've seen before, and in this case it goes all the way back when we computed the digit $4$ . This means that the digits $4, 1, 7, 5, 8, 2$ will continue repeating. Here's what we've computed so far: $3 + 4 \cdot 10 + 1 \cdot 10^2 + 7 \cdot 10^3 + 5 \cdot 10^4 + 8 \cdot 10^5 + 2 \cdot 10^6 + (-2)10^7 / 7$

This looks an awful lot like a regular integer, except the powers continue to increase, and the repeating part is similar to repeating expansions. Writing out the powers of 10 is becoming cumbersome, so let's repurpose regular integer notation, and combine it with the overline used for repeated digits in regular expansions:

$\overline{285714}3$

Compare this against the usual base $10$ expansion we computed previously:

$0.\overline{142857}$

Do you see the pattern? Starting from the 2nd digit and going backwards we get 4, and 1, then we loop to the end and continue: $7, 5, 8, 2$ . This a cyclic permutation (rotation) of the repeated digits.

Compact Notation

It would be nice to have a compact notation similar to the one we used for falling integer expansions:

$\begin{align*} 1 &= \underline{3} \cdot 7 + \boxed{-2} 10 \\ \boxed{-2} &= \underline{4} \cdot 7 + \boxed{-3} 10 \\ \boxed{-3} &= \underline{1} \cdot 7 + \boxed{-1} 10 \\ \boxed{-1} &= \underline{7} \cdot 7 + \boxed{-5} 10 \\ \boxed{-5} &= \underline{5} \cdot 7 + \boxed{-4} 10 \\ \boxed{-4} &= \underline{8} \cdot 7 + \boxed{-6} 10 \\ \boxed{-6} &= \underline{2} \cdot 7 + \boxed{-2} 10 \end{align*}$

The equality in each row is easy to verify, but the downside is that you have to perform all the important calculations somewhere else. If the divisor is very large, you can substitute any easy to write symbol for it. Also, feel free to omit the $10$ too, if you feel comfortable with the calculations. An alternative, more computation-friendly, notation can look something like this:

$\begin{align*} 1 &= \underline{3} \cdot 7 + 10 \cdot \underline{\overline{-2}} \\ \boxed{-6} &= \underline{4} - 1 \cdot 10 \rightarrow (\underline{\overline{-2}} - \underline{4} \cdot 7) / 10 = \underline{\overline{-3}} \\ \boxed{-9} &= \underline{1} - 1 \cdot 10 \rightarrow (\underline{\overline{-3}} - \underline{1} \cdot 7) / 10 = \underline{\overline{-1}} \\ \boxed{-3} &= \underline{7} - 1 \cdot 10 \rightarrow (\underline{\overline{-1}}- \underline{7} \cdot 7) / 10 = \underline{\overline{-5}} \\ \boxed{-15} &= \underline{5} - 2 \cdot 10 \rightarrow (\underline{\overline{-5}} - \underline{5} \cdot 7) / 10 = \underline{\overline{-4}} \\ \boxed{-12} &= \underline{8} - 2 \cdot 10 \rightarrow (\underline{\overline{-4}} - \underline{8} \cdot 7) / 10 = \underline{\overline{-6}} \\ \boxed{-18} &= \underline{2} - 2 \cdot 10 \rightarrow (\underline{\overline{-6}} - \underline{2} \cdot 7) / 10 = \underline{\overline{-2}} \end{align*}$

Boxed values in each row are the same as before, but multiplied by the $3$ from our starting identity, which is at the very top. The idea is that you take the rightmost number in each row, the one with both an overline and an underline, an "incomplete box," "complete its box" by multiplying by $3$ , and bring the result down to the next row. The underlined values are just the (positive) remainders when dividing the boxed value by 10. Lastly, the value inside the incomplete box, from which we subtract, is the same as the rightmost value from the row above. We don't make use of the quotient from the first division by 10 so we can omit that too:

$\begin{align*} 1 &= \underline{3} \cdot 7 + 10 \cdot \underline{\overline{-2}} \\ \boxed{-6} &\equiv \underline{4} \rightarrow (\underline{\overline{-2}} - \underline{4} \cdot 7) / 10 = \underline{\overline{-3}} \\ \boxed{-9} &\equiv \underline{1} \rightarrow (\underline{\overline{-3}} - \underline{1} \cdot 7) / 10 = \underline{\overline{-1}} \\ \boxed{-3} &\equiv \underline{7} \rightarrow (\underline{\overline{-1}}- \underline{7} \cdot 7) / 10 = \underline{\overline{-5}} \\ \boxed{-15} &\equiv \underline{5} \rightarrow (\underline{\overline{-5}} - \underline{5} \cdot 7) / 10 = \underline{\overline{-4}} \\ \boxed{-12} &\equiv \underline{8} \rightarrow (\underline{\overline{-4}} - \underline{8} \cdot 7) / 10 = \underline{\overline{-6}} \\ \boxed{-18} &\equiv \underline{2} \rightarrow (\underline{\overline{-6}} - \underline{2} \cdot 7) / 10 = \underline{\overline{-2}} \end{align*}$

The leftmost equals sign is now a triple bar, used to indicate that we're dealing with a remainder. Feel free to replace the arrow with whatever you prefer, and I don't actually draw all the boxes, underlines, and overlines either; those are there to help you get comfortable with the notation.

I've yet to explain where the equation to the right of the arrow comes from.

Alternative Formulation

The fact that $x$ and $y$ are related by the starting equation suggests a potential optimisation:

$\begin{align*} 1 &= 7 x + 10 y \\ (1 - 7 x) / 10 &= y \end{align*}$

So, when computing the $n$ -th digit of the expansion, we multiply the starting equation by the previous remainder, $R_{n-1}$ , and the coefficient in front of $10$ becomes:

$(R_{n-1} - 7 x \cdot R_{n-1}) / 10 = y \cdot R_{n-1}$

On the other hand, the coefficient in front of the divisor becomes:

$x \cdot R_{n-1} = 10 q + r$

for some quotient $q$ and remainder $r$ . We can substitute for $x \cdot R_{n-1}$ in the previous equation to get:

$(R_{n-1} - 7 (10 q + r)) / 10 = (R_{n-1} - 7 \cdot 10 q - 7 r) / 10 = y \cdot R_{n-1}$

Recall that in our original algorithm we eventually add $7 q$ to the multiplied value of $y$ , which we can rewrite as $70 q / 10$ to bring it under common denominator:

$(R_{n-1} - 70 q - 7 r + 70 q) / 10 = (R_{n-1} - 7 r) / 10$

We didn't take advantage of the specific values of the divisor and base, so the expression holds for any divisor, $D$ , and base, $B$ :

$R_{n} = \frac{R_{n-1} - D r}{B}$

This optimisation is useful because it removes a multiplication step from the calculation, and eliminates one of the variables from the symbolic computation, which is useful for proofs. As an exercise, use this formula to prove that the previous step remainder is always negative, and its absolute value is less than the divisor.

Automated Computation

It's time to make the computer do the annoying work for us. Assuming we have a way to compute the coefficients for the starting equation, the rest of the algorithm is trivial to implement. One important observation about that equation is that, if we compute its remainder after dividing by either $7$ or $10$ , we'll get $1$ , and $x$ and $y$ are the values that make that happen. Computations with remainders are so useful that there's an entire branch of mathematics, called modular arithmetic (or clock arithmetic), that studies them. In regular rational arithmetic, if the result of multiplying two rational numbers equals $1$ , we know that one of those numbers is the multiplicative inverse of the other. The exact same idea applies in modular arithmetic, where it's called modular multiplicative inverse instead. As you can imagine, computing such inverses is essential if you want to do modular arithmetic, so you can easily find an implementation that computes $x$ for you. SymPy's implementation is called mod_inverse, and we can use it for our first implementation:

from sympy import mod_inverse

def intDivRisingSimple(n: int, d: int, b: int, termCount: int = 10) -> list:
    expansion = []
    inv = mod_inverse(d, b)
    for _ in range(termCount):
        digit = (n * inv) % b
        n = (n - d * digit) // b
        expansion.append(digit)

    return expansion

print(intDivRisingSimple(1, 7, 10))

[3, 4, 1, 7, 5, 8, 2, 4, 1, 7]

The major problem with intDivRisingSimple is that it will error out if we plug in any divisor that contains multiples of the base, such as $1/30$ . We know we can factor out the $1/10$ , expand $1/3$ , and then divide by $10$ afterwards. This implies something interesting about our notation: There can be a finite number of fractional digits, similar to how decimals can have a finite number of integer digits! To make our function more robust, we can divide out as many multiples of the base from the divisor as we can, and then return the negative power of the base that would need to be applied to the list of coefficients:

def intDivRising(n: int, d: int , b: int, termCount: int = 10) -> tuple[list, int]:
    exp = 0
    while not d % b:
        exp -= 1
        d = d // b

    expansion = []
    inv = mod_inverse(d, b)
    for _ in range(termCount):
        digit = (n * inv) % b
        n = (n - d * digit) // b
        expansion.append(digit)

    return expansion, exp

print(intDivRising(1, 30, 10, 4))

([7, 6, 6, 6], -1)

This would correspond to $666.7$ in our revised notation, or, in expanded form:

$6 \cdot 10^2 + 6 \cdot 10 + 6 + \frac{7}{10}$

Programming Warning

If you're trying to follow along in a different language, you might run into some issues related to how division and remainder operations are implemented. Python is somewhat unusual because its remainder operator, %, always returns a positive value. This is very convenient when implementing maths algorithms, but it comes with the consequence that the its integer division operation can return negative values, and rounds down (floor) instead of the typical truncation towards zero in other languages. For example, the C99 standard explicitly states that integer division will truncate the quotient (section 6.5.5 "Multiplicative Operators"). Following suit, the C++11 standard also defined integer division as a truncating operation (section 5.6.4 "Multiplicative operators"). In practice, truncation was the de-factor standard regardless of what the language standards say, because that's how hardware implementation typically operate. Some languages implement both remainder and modulo to differentiate between truncation and flooring. Fortunately, implementing your own flooring division isn't too difficult if your language doesn't support it out of the box. Here's how you can do it in C:

typedef long long integer;

integer divModFloorConstantTime(integer n, integer d, integer* o_remainder) {
    integer quotient = n / d;
    integer remainder = n % d;
    // Only adjust the output when either the numerator or divisor is negative, which
    // translates to an XOR operation. Clang, gcc, and msvc will optimise isNegative
    // to (n ^ d) >> (BIT_SIZE - 1); I didn't do it for the sake of clarity.
    char isNegative = (n < 0) ^ (d < 0);
    char isApproximate = remainder != 0;
    char shouldAdjust = isApproximate & isNegative;

    quotient -= shouldAdjust;
    remainder += shouldAdjust * d;

    *o_remainder = remainder;
    return quotient;
}

integer divModFloor(integer n, integer d, integer* o_remainder) {
    integer quotient = n / d;
    integer remainder = n % d;

    if (remainder != 0 && ((n < 0) ^ (d < 0))) {
        quotient -= 1;
        remainder += d;    
    }

    *o_remainder = remainder;
    return quotient;
}

Playing Around

Truncating after a specific number of digits in our computation is practical yet boring. Here's a simple rising expansion implementation that won't stop until it finds the repeating digits:

from sympy import mod_inverse

def intDivRisingRepeat(n: int, d: int, b: int) -> tuple[list, list, int]:
    exp = 0
    while not d % b:
        exp -= 1
        d = d // b

    expansion = []
    inv = mod_inverse(d, b)
    remainderIndex = {} # keep track of where we encounter specific remainders
    currIndex = 0

    while True:
        digit = (n * inv) % b
        n = (n - d * digit) // b

        expansion.append(digit)
        if n == 0:
            return expansion, [], exp 

        currIndex += 1
        # `setdefault` will return the existing index, if it finds one during insertion
        prevIndex = remainderIndex.setdefault(n, currIndex)       
        if prevIndex != currIndex:
            return expansion[:prevIndex], expansion[prevIndex:], exp  

print(intDivRisingRepeat(1, 7, 10))

([3], [4, 1, 7, 5, 8, 2], 0)

What else can we throw at this function? Try an integer like $12/1$ :

([2, 1], [], 0)

It just gave us back the 12 (the list is ordered by increasing powers). Can we convert $16$ to base $2$ with this function? Sure:

print(intDivRisingRepeat(16, 1, 2))

([0, 0, 0, 0, 1], [], 0)

How about a negative number like $-13/1$ :

([7, 8], [9], 0)

Interesting, it has a repeating 9! Here's $-127$ :

([3, 7, 8], [9], 0)

Repeating 9 again... If you carefully study the behaviour of the expansion formula we derived above, you'll see why the repeating 9 will continue to appear for every negative integer you input. In fact, the behaviour is the same regardless of the base, you'll just get a repeating $B-1$ , where $B$ is the base. Programmers reading this article might recognise what's going on here: These expansions are just $10$ 's complement representations of negative integers, which behaves the exact same way as the 2's complement used by computers!

We haven't tried any negative rationals yet, so let's see what we get for $-1/7$ , compared to $1/7$ :

 1/7: ([3], [4, 1, 7, 5, 8, 2], 0)
-1/7: ([7], [5, 8, 2, 4, 1, 7], 0)

We get a different fixed term and the repeating part is rotated! Can you come up with an algorithm to negate the expansion of a rational without using the our division algorithm? See if you can do the same for the integers.

At this point you might be wondering why we'd even care about this alternative representation of numbers. Depending on your background, you might be aware of just how useful linear algebra is in a variety of fields. Reducing difficult problems to equivalent, and often approximate, formulations in linear algebra is how a lot of mathematics is done in the wild, because the tools of linear algebra are powerful, well-understood, and well-behaved. I'd argue that the theory of finite fields is very similar in this regard, yet, when it comes to questions involving rational numbers, or approximations with them, we can't use the tools of finite field arithmetic unless we come up with a compatible encoding for our numbers. The representation we came up with appears to be a promising candidate that may allow us to, metaphorically, import the tools of finite fields in our mathematical projects, though, so far, we've only scratched the surface.

Conclusion

As is customary with my writing style, the unavoidable jargon comes at the end. What we've been doing so far is exploring a well-known mathematical field: the p-adic numbers. There are many high quality presentations that feature these curious numbers, some notable examples being 3Blue1Brown's, Eric Rowland's, and, more recently, Veritasium video. However, what all presentation I've seen have in common is a distinct lack of computational grounding. Norman Wildberger was the closest to providing this grounding, yet the clumsy calculations kept getting in the way. What I find most regrettable, however, is that proofs about the p-adics, at least those I've seen, require heavy mathematical machinery, and are practically inaccessible for most people - even though there are elementary alternatives accessible to anyone with a basic understanding of number theory and proof writing. I'll go over those proofs in future articles, and I hope this brief introduction has sparked your interest!

Historically, the invention of the p-adics was, indeed, inspired by what I called rising division, more specifically, Taylor series - we'll explore how those two are connected later on. Kurt Hensel, the inventor of p-adic numbers, was interested in using analysis tools for problems in number theory, and it seems the educational material on the subject hasn't evolved much since the early 20th century, because it's still deeply tied to analysis.

My interests in error correction and random number generators also exposed me to the lacking educational materials in those fields. I've yet to see a single book or paper properly explain why linear recurrences are, formally, represented by rational functions. Generating functions, as is typical for them, are thrown into the mix to sow even more confusion, all while simple ideas are buried under piles of needless abstraction.

Algebraic geometry is no exception, lack of understanding about polynomial division is apparent in most treatments, especially when it comes to elimination theory and Gröbner basis.

The motivation to work on this article came from my frustration with the state of education in fields I find beautiful, and I hope I can properly convey this beauty to people who are interested in mathematics and computer science, yet find the deeper ideas nigh on impenetrable.

Handmade Maths: Cosine and Sine

2021-08-21T00:00:00+01:00

Have you ever wondered how the magical cosine and sine buttons on your calculator work? If so, then this article will not only answer this question, but also develop most of trigonometry and rotation transformations along the way. The only prerequisites are a basic understanding of vectors, and general comfort with algebra. Let's roll up our sleeves and begin exploring the fascinating world of rotations!

1. Foundation

A natural question that comes up after you learn about the basic operations in vector algebra - adding, subtracting, and scaling - is how to create rotated versions of a vector. Given how frequently rotations come up when solving problems, not having them as a vector transformation would make the entire framework seem incomplete.

Before we begin exploring this topic, we'll need to come up with a preliminary method for assigning numerical values to distinct rotation transformations. Suppose you decided to rotate a stick around a fixed (pivot) point on it. If we pick a distinguished point on the stick, different from the pivot, and keep track of its location as we rotate, we'll eventually trace out a circular shape and the point will arrive back to its initial position. This seems like an important and easy to identify characteristic of the transformation, so we can assign the number 1 to a rotation that takes any point on the circle back to its starting location exactly once - the centre point is special in that it never moves. Note that the "1" here is a distinct rotation unit and shouldn't be confused with the units one would use to measure distances - I'll refer to it as a turn (abbreviated by $t$ ) from now on. The distance between the pivot and the distinguished point will be our unit of distance, abbreviated as $d$ . Later on we'll need to figure out a way to convert between turn and distance units, but our most pressing task is to address the problem of interpolation, or how to perform rotations that correspond to different turn values.

An intuitive way to think about turn values between 0 and 1 is to associate them with the boundary traced out by a non-pivot point in a full turn. For example, half a turn would be the point that splits the boundary into two equal pieces, meaning that we can rotate one of the pieces around the pivot to perfectly overlap the other. Constructing this point is quite easy: Draw a line, pick a pivot point on it, and draw a circle centred on that point with the help of a compass. The two places where the circle intersects the line are the 0 (same as 1) and 1/2 turn points.

A quarter (1/4) turn is a bit trickier because we need to figure out how to split the 1/2-turn segment (bounded by 0 and 1/2) in half. Fortunately, there's an ancient compass construction that can help us accomplish this task! Place the needle on either the 0 or 1/2 point and draw a circle with a radius that's larger than the distance between the point and the centre. Repeat the same process with the other boundary point and observe that the new circles intersect in two additional points, and the line going through these points will split the half-turn arcs of the original circle in half:

The 0, 1/4, 1/2, and 3/4 turn points together with the circle centre can be used to set up a two-dimensional Cartesian coordinate system, which would allow us to establish the following connection between turn values and vectors:

Turn	Vector
$0, 1$	$(1, 0)$
$1/4$	$(0, 1)$
$1/2$	$(-1, 0)$
$3/4$	$(0, -1)$

Once again, the Cartesian coordinate values for these vectors are distinct from the turn values despite using the same numeral system - this is an important distinction that is often neglected in school due to the use of implicit units. While this would be a great starting point for developing the ideas behind vector algebra, in the interest of time, I'll assume the reader is familiar with the basics and address the problem of rotation in the context of vectors.

How would you rotate a vector by a quarter turn counterclockwise? It's usually a good idea to begin exploring such questions with the simplest possible configuration you can imagine - the basis vectors. Starting with $(1, 0)$ , rotating a quarter turn counterclockwise results in the vector $(0, 1)$ by definition. Our first observation is that the position of the numbers 0 and 1 was swapped, so we can conjecture that rotating an arbitrary 2D vector will involve swapping its x and y coordinates. Swapping the x and y coordinates of a vector is actually a reflection across the vector $(1, 1)$ (the line $y = x$ ). A single test case is never sufficient so let's continue rotating!

Now that we're at $(0, 1)$ , applying the same rotation would give us $(-1, 0)$ . While the coordinates had to be swapped as we expected, the x value also had to be negated. Fortunately, it's quite easy to reconcile this new finding with the initial conjecture because negating zero doesn't do anything! We can now update our rotation algorithm: Rotating a quarter turn counterclockwise requires a coordinate swap followed by a negation of the new x value. Take note of the fact that negating a vector's x coordinate is the same as reflecting it across the $(0, 1)$ axis.

Does the updated conjecture hold after the third rotation? Indeed, it does! From $(-1, 0)$ we have to go to $(0, -1)$ - the values are swapped and the zero is implicitly negated. The final rotation takes us from $(0, -1)$ to $(1, 0)$ and the conjecture still holds. Marvellous! We've now gone a full circle and our refined initial observation held up beautifully.

Let's check whether this formula works for an arbitrary vector (drag the blue point to move it around):

Looks good to me! The reason why this works is because every vector can be expressed as a linear combination of the basis vectors, so if you rotate the basis, everything else follows. Another way to visualise this transformation is by drawing 2 vectors on a sheet of paper, then rotating the sheet itself. Algebraically, recall that a vector $(a, b)$ can be decomposed into $a(1, 0) + b(0, 1)$ , and, if we rotate the individual vectors by a quarter turn using our formula, we get $a(0, 1) + b(-1, 0) = a(0, 1) - b(1, 0)$ , which is exactly the same as swapping the original vector's entries, and negating the first one.

As it turns out, swapping the coordinates and negating one of them is one of the most important operations in mathematics! Unfortunately, students are almost never exposed to the true power behind it, and to make matters even more confusing, people often refer to this operation as a number! We'll return to this idea later.

If you were reading carefully, you might be wondering why I pointed out the implicit reflections in the rotation formula. While exploring this observation isn't necessary to understand the rest of the article, it provides important insight into more advanced mathematical treatments of rotation. We were somehow able to rotate a vector by a quarter turn counterclockwise with the help of two successive reflections across vectors separated by an eight of a turn (the line $y=x$ splits the 0-1/4 arc into equal pieces). The curious amongst you may wish to investigate this lead on your own because it leads to simple and elegant generalisations of rotation in any dimension!

2. Measurement and Normalization

We only know the coordinates of 4 points on the circle so far. How do we get the rest? Intuitively, the method we used to construct the circle implies that all vectors with a distance of 1 from the origin should lie somewhere on it. It's worth investigating how we can scale any vector so it has length 1, but before we get to that point, how do we even calculate the length of a vector that's not aligned with our coordinate lines? We'll approach this problem in a somewhat roundabout way.

Historically, any notion of distance measurement required a representative object, a unit, to measure against - be that the length of a body part, a stick, or even a metal rod stored in a safe place. It was only recently that we started relying on fundamental physical phenomena, like the speed of light, to define our units. While metrology, the science of measurement, is a fascinating subject, the world of mathematics is much simpler. We already defined our unit of distance, $d$ , but we'll also need an appropriate 2-dimensional object to represent area, which is the space encompassed by triangles, rectangles, and various other shapes. One of the reason why squares are typically chosen as the representative object for area is due to counting. An easy way to count how many, say, coins you have is to arrange them in a rectangular pattern, find how many coins there are along each side, then multiply the results to get the total count (Can you arrange any number of coins in a rectangular pattern?). A square is more appropriate than a rectangle because we need to specify the length of only one side to describe it, so its shape is less arbitrary.

We can define the area of a square with side length $x$ to be $x \cdot x = x^2$ , which is associated with a unit of $d^2$ . This is sufficient to also derive the formula for a rectangle. Consider the area of a square with side length $x + h$ . Geometrically, it looks like this:

We have two squares with area $x^2$ and $h^2$ , and two unknown rectangles with equal area. We can subtract the small squares from the big one and divide by $2$ to compute the area of a rectangle algebraically:

$\frac{(x + h)^2 - x^2 - h^2}{2} = \frac{x^2 + 2xh + h^2 - x^2 - h^2}{2} = \frac{2xh}{2} = xh$

Splitting squares or rectangles along a diagonal gives us two triangles with equal sides which are obviously rotated version of each other, so we can define their area to be $xh/2$ .

2.1 Length

If we find out how to calculate the area of a square whose sides are formed by an arbitrary vector, we can simply take the square root of that formula to get the length of the vector. Conveniently, our quarter-turn rotation allows us to easily construct such squares:

This diagram shows a square constructed by rotating an arbitrary vector $(a, b)$ . Take a moment to convince yourself that, although the signs may differ, and $a$ and $b$ can be swapped depending on the starting quadrant, the lengths will remain the same regardless of the coordinate choice. We know how calculate the area of the big rectangle and the 4 shaded triangles, so we can simply subtract the total area of the triangles from the area of the square to get the area of the inner square. From here on, assume that all mentions of $a$ and $b$ as lengths are referring to the absolute value of those coefficients (removing the sign). If you calculate the side lengths of the big rectangle, you'll find that it is, in fact, a square with side length $(a + b)$ , so its area is simply $(a + b)^2$ . The 4 small triangles also have equal area: $ab/2$ . Putting all of this together we get our length formula:

$\sqrt{(a + b)^2 - 4ab/2} = \sqrt{a^2 + 2ab + b^2 - 2ab} = \sqrt{a^2 + b^2}$

You might recognise the Pythagorean theorem in this derivation, and the formula can easily be extended to compute vector lengths in any dimension. It can also be used to find formulas for constructing rectangles formed by 2 arbitrary vectors, which gives us a notion of projection. Remarkably, this is all we need to calculate parallelepiped (also called parallelotope in higher dimensions) areas, volumes, and higher-dimensional analogues of those in any dimension. If you're curious, you can look into the Gram–Schmidt process, the Gram determinant, and the alternative way of computing those via Exterior Algebra.

Back to our problem at hand, scaling vectors to have length of 1 (unit length; such vectors are also called unit vectors) is as simple as ensuring the value under the square root is 1, which we can accomplish through division:

$\sqrt{\frac{a^2 + b^2}{a^2 + b^2}} = \sqrt{\frac{a^2}{a^2 + b^2} + \frac{b^2}{a^2 + b^2}} = \sqrt{\left(\frac{a}{\sqrt{a^2 + b^2}}\right)^2 + \left(\frac{b}{\sqrt{a^2 + b^2}}\right)^2}$

It looks like all we need to do to scale a vector so it has unit length is divide its coefficients by the length. This operation is usually called "normalization".

3. Arbitrary Rotations

Our simple rotation formula has a glaring deficiency: What if we wanted to rotate by a different turn value? This problem appears to be rather tricky, so let's first reduce it to something that's easier to work with: a right (quarter turn) triangle. We construct this triangle by raising a perpendicular from the horizontal radius vector $(1, 0)$ until the hypothenuse intersects the circle in the turn point we're interested in:

While we don't really know what turn value corresponds to this intersection point - even though we can compute its Cartesian coordinates - we can use the right triangle as an intermediate construction until we develop more sophisticated tools.

For the sake of simplicity, let's assume that the radius of the circle is 1 distance unit, which makes the base of our triangle equal to 1. This simplification also has the convenient side effect of making the triangle's height act as a multiplier - if you multiply the vertical unit vector by this length you'll get the height vector. We're about to make good use of this property!

The height can be arbitrary so let's just call it $h$ , and the length of the hypothenuse will be denoted $l$ . With this setup in place, suppose I asked you to find an algorithm that rotates the base $\vec{e}_1 = (1, 0)$ so that it overlaps the hypothenuse. How would you do it?

If you're comfortable with the idea behind vector scaling, you might suggest multiplying the hypothenuse vector by the reciprocal of its length $l = \sqrt{1 + h^2}$ :

$\vec{e}_r = \frac{1}{l} \cdot (1, h) = \bigg(\frac{1}{l}, \frac{h}{l}\bigg)$

While this is, indeed, the correct answer, such an algorithm only works for starting vectors that lie flat on the horizontal axis. A more general algorithm would first construct the hypothenuse vector before performing the rescaling. Using only the vector we want to rotate as a starting point and the value $h$ , how do we construct a hypothenuse?

Because we ultimately want a quarter turn separation between the base and the height, we can use the handy quarter-turn-counterclockwise rotation formula (represented by the function $\mathrm{rotQ}$ ) to rotate the base vector. Next, we need to rescale it to a unit vector, and then multiply it by $h$ to make its length equal to the height vector. Our starting vector $\vec{e}_1$ had a length of 1 and that's convenient because we can skip the normalization step.

For now, let's assume our more general vector $\vec{v}$ also has a unit length. Rotating and rescaling $\vec{v}$ by $h$ can be expressed as follows:

$\vec{p} = h \cdot \mathrm{rotQ}(\vec{v})$

We then move our new construction from the origin to the correct place by adding the starting vector to it:

$\vec{h} = \vec{p} + \vec{v} = h \cdot \mathrm{\mathrm{rotQ}}(\vec{v}) + \vec{v}$

With the hypothenuse constructed, we can now rescale to get the final rotated vector $\vec{v}_r$ :

$\vec{v}_r = ( h \cdot \mathrm{rotQ}(\vec{v}) + \vec{v} ) \cdot \frac{1}{l} = \frac{h}{l} \cdot \mathrm{rotQ}(\vec{v}) + \frac{1}{l} \cdot \vec{v}$

Let's introduce two helper variables to reduce the visual noise in these equations even further:

$c = \frac{1}{l} \quad s = \frac{h}{l}$

$\vec{v}_r = s \cdot \mathrm{rotQ}(\vec{v}) + c \cdot \vec{v}$

Amazingly simple relation, isn't it? Notice that $c$ and $s$ are the normalized coordinates of the hypothenuse, so we can leverage this finding to overcome $h$ 's limitation of only being able to represent rotations strictly less than a quarter turn (Can you see why that is?). Mapping a turn value to a vector instead of a scalar gives us a lot of flexibility! Setting $\vec{v}$ equal to an arbitrary unit vector $(a, b)$ and unpacking the formula above we get:

$\vec{v}_r = s \cdot (-b, a) + c \cdot (a, b) = (c a - s b, s a + c b)$

If you've taken a course on linear algebra you might recognise how this formula relates to a 2D rotation matrix! It turns out that $(a, b)$ doesn't even need to be a unit vector as long as $c$ and $s$ satisfy the following constraint: $c^2 + s^2 = 1$ . To verify this, all we need to do is bring back the initial normalization step for $\vec{v}$ that we omitted:

$\vec{v}_r = \bigg(\frac{s}{\| \vec{v} \|} \cdot \mathrm{rotQ}(\vec{v}) + \frac{c}{\| \vec{v} \|} \cdot \vec{v}\bigg) \cdot \| \vec{v} \| = s \cdot \mathrm{rotQ}(\vec{v}) + c \cdot \vec{v}$

The double vertical bars $\|$ represent the length of the vector. We're normalizing the offset before adding it to the rotated vector because we want $h$ to apply to all vectors consistently, regardless of their length. We also need to multiply by the length at the end to undo the normalization. As you can see, the vector length value simply cancels out.

Another way to examine the behaviour of our arbitrary rotation formula is to compute the length of the output vector:

$\begin{aligned} \|\vec{v}_r\|^2 &= (c a - s b)^{2} + (s a + c b)^{2} = a^{2} c^{2} - \cancel{{2} a b c s} + b^{2} s^{2} + a^{2} s^{2} + \cancel{{2} a b c s} + b^{2} c^{2} \\ &= a^{2} (c^{2} + s^{2}) + b^{2} (c^{2} + s^{2}) = (a^{2} + b^{2})(c^{2} + s^{2}) \\ \|\vec{v}_r\| \enspace &= \sqrt{a^{2} + b^{2}} \sqrt{c^{2} + s^{2}} \end{aligned}$

It looks like the length of the rotated vector is the product of the lengths of the two input vectors! This is why $(c, s)$ being a unit vector preserves the length of the input when performing the rotation.

Now that we have an algorithm to perform arbitrary vector rotations, we can have some fun with it! Let's generate a bunch of successive rotations:

3.1 Composing Rotations

Now is a good time to reflect on what we've discovered so far. First, the inputs to our arbitrary rotation formula are actually two vectors. Moreover, the order in which they go in doesn't really matter (feel free to check this for yourself) even though we started off under the assumption that one of the vectors will represent a turn value - and will therefore be distinguished from the other. One conclusion we can draw from this is that both vectors can play the role of a turn value representative and we've stumbled upon a rather beautiful way of composing rotations! For example, if we rotated $(1, 0)$ by a vector $\vec{v}_1 = (a, b)$ , and then rotated the result again by a vector $\vec{v}_2 = (c, d)$ , we'd get $\vec{v}_3 = (ac - bd, ad + bc)$ , which is where $(1, 0)$ would end up if we directly rotated it by the sum of the turn values encoded in $\vec{v}_1$ and $\vec{v}_2$ .

This rotation composition property is incredibly useful in practical applications and is not limited to rotations in 2 dimensions. In fact, mathematicians have even discovered an algorithm for constructing such "rotation algebras", though exploring it is left as an exercise for the curious reader.

As you have seen so far, we managed to use our humble quarter turn rotation - and a little bit of basic vector intuition - to develop a more general formula that can aid us in solving problems. However, we're not done yet: The question of converting between turn and distance units is still unanswered! Fortunately, we now have all the tools we need to tackle it, so let's do that next.

4. Travelling Around the Circle

Here's a question: Starting at $(1, 0)$ , if we travelled a distance of $d$ units counterclockwise around the circle, what are the coordinates of the location we'll arrive at?

Perhaps we can repurpose our arbitrary rotation formula to answer this question! The key observation is that we can use multiples of $h$ to approximate a distance of $d$ units along the circle because the smaller the $h$ , the closer it hugs the arc of the circle:

Zoom in on $(1, 0)$ automatically to keep the height constant

If we divide $d$ by $h$ , we'll get the rough number of successive rotations we need to perform to arrive at the final location. But what if $d$ is not a whole number? In that case, we can either round the rotation count to the nearest integer or make $h$ itself a multiple of the lowest power of 10 in $d$ . For example, if $d$ is equal to 1.2345, we can make $h$ a multiple of 0.0001 so that $d/h$ is guaranteed to be a whole number. Either way, non-integer distance values won't really pose a problem. Let's see what we get for $d$ equal to 2 and $h$ equal to 0.001, which would require 2000 rotations:

import math

def circularTravel(d, h):
    c = 1/math.sqrt(1 + h*h)
    s = h*c
    x = 1
    y = 0

    rotations = round(d/h)
    for _ in range(rotations):
        xn = c*x - s*y
        yn = s*x + c*y

        x = xn
        y = yn

    print('-'*32)
    print(f'd={d}, h={h}, rotations={rotations}')
    print(f'x={x}\ny={y}')

circularTravel(2, 0.1)
circularTravel(2, 0.01)
circularTravel(2, 0.001)

--------------------------------
d=2, h=0.1, rotations=20
x=-0.41011187409312255
y=0.9120352244994887
--------------------------------
d=2, h=0.01, rotations=200
x=-0.4160862194310148
y=0.9093251662632378
--------------------------------
d=2, h=0.001, rotations=2000
x=-0.4161462303490977
y=0.9092977042564712

I added two more configurations so we have something to compare against. The brute-force algorithm works but is extremely inefficient, especially when $d$ becomes really large. Can we somehow optimize it to make it faster? We can!

5. Finding Patterns

Recurrence relations are great at providing succinct formulas, but they tend to hide a lot of the patterns behind the underlying computation. Our first step is to create a trace of the repeated rotation by computing the flat equations at each step of evaluation. Once we have that, we can look for patterns in it, and, ideally, come up with an algorithm that generates the $n$ -th rotation directly.

Pattern spotting is a time-honoured practice in both mathematics and programming because it can give you insight into the structure of a system, and understanding the structure is a prerequisite for modifying it without breaking it.

There's one problem, however, which is that creating a trace involves a lot of algebraic manipulations which, while simple, are tedious and error prone. Mathematicians back in the day had to patiently do these by hand (or offload that work to their assistants), but, as inhabitants of the 21st century, we have a much better option - make the computer do the algebra for us! To do this, we'll need a computer algebra system package and I chose Python's SymPy because it's simple and free. While it's not strictly necessary, you can follow along by installing the Python environment and executing pip install sympy in your operating system terminal.

First, let's make SymPy evaluate a trace of the recurrence relation as it is:

from sympy import *

rotations = 12
c, s = var('c, s')
x = 1
y = 0

xs = [x]
ys = [y]
for _ in range(rotations):
    xn = c*x - s*y
    yn = s*x + c*y

    xs.append(xn.expand())
    ys.append(yn.expand())

    x = xn
    y = yn

print('x coordinates:')
for x in xs:
    print(x)

print('\ny coordinates:')
for y in ys:
    print(y)

Table 1

n	x	y
0	$1$	$0$
1	$c$	$s$
2	$c^{2} - s^{2}$	$2 c s$
3	$c^{3} - 3 c s^{2}$	$3 c^{2} s - s^{3}$
4	$c^{4} - 6 c^{2} s^{2} + s^{4}$	$4 c^{3} s - 4 c s^{3}$
5	$c^{5} - 10 c^{3} s^{2} + 5 c s^{4}$	$5 c^{4} s - 10 c^{2} s^{3} + s^{5}$
6	$c^{6} - 15 c^{4} s^{2} + 15 c^{2} s^{4} - s^{6}$	$6 c^{5} s - 20 c^{3} s^{3} + 6 c s^{5}$
7	$c^{7} - 21 c^{5} s^{2} + 35 c^{3} s^{4} - 7 c s^{6}$	$7 c^{6} s - 35 c^{4} s^{3} + 21 c^{2} s^{5} - s^{7}$
8	$c^{8} - 28 c^{6} s^{2} + 70 c^{4} s^{4} - 28 c^{2} s^{6} + s^{8}$	$8 c^{7} s - 56 c^{5} s^{3} + 56 c^{3} s^{5} - 8 c s^{7}$
9	$c^{9} - 36 c^{7} s^{2} + 126 c^{5} s^{4} - 84 c^{3} s^{6} + 9 c s^{8}$	$9 c^{8} s - 84 c^{6} s^{3} + 126 c^{4} s^{5} - 36 c^{2} s^{7} + s^{9}$
10	$c^{10} - 45 c^{8} s^{2} + 210 c^{6} s^{4} - 210 c^{4} s^{6} + 45 c^{2} s^{8} - s^{10}$	$10 c^{9} s - 120 c^{7} s^{3} + 252 c^{5} s^{5} - 120 c^{3} s^{7} + 10 c s^{9}$
12	$c^{11} - 55 c^{9} s^{2} + 330 c^{7} s^{4} - 462 c^{5} s^{6} + 165 c^{3} s^{8} - 11 c s^{10}$	$11 c^{10} s - 165 c^{8} s^{3} + 462 c^{6} s^{5} - 330 c^{4} s^{7} + 55 c^{2} s^{9} - s^{11}$

Looking at these results may be overwhelming if this is your first time doing something like this, so let's take it slow and see if we can make sense of the information.

Before we begin analysing, it's important to have a good idea about what each row represents. For example, row #7 for the x coordinates is a multivariate (multiple variables) polynomial which, when evaluated for any given $c$ and $s$ , will give us the x coordinate of $(1, 0)$ after we rotate it 7 times.

The first step we'll take is check for any patterns in the powers. One observation that immediately stands out to me is the fact that the powers of $c$ and $s$ in each term add up to the same value - the rotation number. Recall that $c$ and $s$ are the coordinates of a normalized direction vector, which means there's a division by a square root involved in their computation. However, on every even rotation this square root cancels out and makes the formula a lot more pleasant. Let's focus on the second iteration, which is effectively a rotation by twice the angle.

Suppose $c$ and $s$ come from normalizing a direction vector $\vec{d} = (x, y)$ . Reading off the x and y values for the second iteration we get $\vec{e}_2 = (c^{2} - s^{2}, 2 c s)$ , which is the vector we get after we rotate $(1, 0)$ by twice the angle $\vec{d}$ encodes. Replacing $c$ and $s$ with their corresponding values from the normalized $\vec{d}$ we get the following:

$c = \frac{x}{\sqrt{x^2 + y^2}}, \quad s = \frac{y}{\sqrt{x^2 + y^2}}$

$\vec{e}_2 = \bigg(\frac{x^2 - y^2}{x^2 + y^2}, \frac{2 x y}{x^2 + y^2}\bigg)$

Remarkably, plugging in any rational numbers for $x$ and $y$ (with the exception of two zeros) is going to give us a pair of rational numbers $(a, b)$ that satisfy the relation $a^2 + b^2 = 1$ . This formula can also be used to create a rational parametrization of the unit circle by fixing one of the inputs (pick any value for it like 1) and varying the other, though the resulting points will not be evenly spaced along the circle's arc. There's a lot more to say about this formula but it would distract us from our task at hand - the curious reader may wish to explore topics like rational circular interpolation between two vectors.

Another pattern you might have picked up on is that all $s$ powers in the x coordinate are even, while $c$ powers alternate between even and odd in both x and y, depending on the row. A similar observation can be made about the y coordinates, except the power of $s$ is always odd. The reason why I find this interesting is because we already have a relationship between $c$ and $s$ : $c^2 + s^2 = 1$ , which can be rearranged into: $s^2 = 1 - c^2$

If we were to replace every occurrence of $s^2$ with $1 - c^2$ in the x coordinate we can eliminate $s$ entirely:

for x in xs[1:]: # Skip the first entry because it's an integer
    print(x.subs(s**2, 1 - c**2).expand())

$\begin{aligned} &c \\ &2 c^{2} - 1 \\ &4 c^{3} - 3 c \\ &8 c^{4} - 8 c^{2} + 1 \\ &16 c^{5} - 20 c^{3} + 5 c \\ &32 c^{6} - 48 c^{4} + 18 c^{2} - 1 \\ &64 c^{7} - 112 c^{5} + 56 c^{3} - 7 c \\ &128 c^{8} - 256 c^{6} + 160 c^{4} - 32 c^{2} + 1 \\ &256 c^{9} - 576 c^{7} + 432 c^{5} - 120 c^{3} + 9 c \\ &512 c^{10} - 1280 c^{8} + 1120 c^{6} - 400 c^{4} + 50 c^{2} - 1 \\ &1024 c^{11} - 2816 c^{9} + 2816 c^{7} - 1232 c^{5} + 220 c^{3} - 11 c \\ &2048 c^{12} - 6144 c^{10} + 6912 c^{8} - 3584 c^{6} + 840 c^{4} - 72 c^{2} + 1 \end{aligned}$

These polynomials are actually quite famous - they're called the Chebyshev polynomials of the first kind and you encounter them a lot in a branch of mathematics called approximation theory. The "first kind" in the name suggests the existence of a second kind, so let's see how to derive it. For the y coordinates we can factor out a single $s$ to get all even powers and perform the same substitution:

for y in ys[1:]: # Skip the first entry because it's an integer
    # Divide by 's' first to make the powers even and then
    # multiply by it after we perform the substitution
    yn = (y / s).expand().subs(s**2, 1 - c**2).expand() * s
    print(yn)

$\begin{aligned} &s \\ &s (2 c) \\ &s (4 c^{2} - 1) \\ &s (8 c^{3} - 4 c) \\ &s (16 c^{4} - 12 c^{2} + 1) \\ &s (32 c^{5} - 32 c^{3} + 6 c) \\ &s (64 c^{6} - 80 c^{4} + 24 c^{2} - 1) \\ &s (128 c^{7} - 192 c^{5} + 80 c^{3} - 8 c) \\ &s (256 c^{8} - 448 c^{6} + 240 c^{4} - 40 c^{2} + 1) \\ &s (512 c^{9} - 1024 c^{7} + 672 c^{5} - 160 c^{3} + 10 c) \\ &s (1024 c^{10} - 2304 c^{8} + 1792 c^{6} - 560 c^{4} + 60 c^{2} - 1) \\ &s (2048 c^{11} - 5120 c^{9} + 4608 c^{7} - 1792 c^{5} + 280 c^{3} - 12 c) \\ \end{aligned}$

The polynomials in the brackets are known as Chebyshev polynomials of the second kind (encountered in algebra and number theory), and, altogether, these formulas are called the multiple-angle formulas in trigonometry. I was required to memorize them in school (at least for the low powers), but, as you can see, they're actually pretty easy to derive with the help of a computer.

We examined the patterns in the powers and all that's left are the coefficients, but before we tackle those, there are a few simplifications I'd like to make in the context of our optimization problem. The first observation is that $l = \sqrt{1 + h^2}$ , which is the denominator for both $c$ and $s$ , is very close to 1 for small values of $h$ , so we might as well set it to 1. This simplification effectively removes the normalization step and will introduce increasing error with each rotation - remember, the length of the output vector is a multiple of the lengths of the input vectors. However, the direction of the output vector doesn't change so we can always normalize the final result if necessary (assuming we had arbitrary precision to work with).

Also, since we're only interested in small values of $h$ , we might as well make that explicit by replacing all its occurrences with $1 / h$ and working with large values of $h$ instead. Over the years, I've found that this tiny adjustment can greatly aid one's intuition.

These modifications result in $c = 1$ and $s = 1 / h$ . Plugging the new values into our multiple-angle formulas we get:

rotations = 12
s = 1 / var('h')

x = 1
y = 0
xs = [x]
ys = [y]

for _ in range(rotations):
    xn = x - s*y
    yn = s*x + y

    xn = xn.expand()
    yn = yn.expand()

    xs.append(xn)
    ys.append(yn)
    x = xn
    y = yn

print('x coordinates:')
for x in xs:
    print(x)

print('\ny coordinates:')
for y in ys:
    print(y)

n	x	y
0	$1$	$h^{-1}$
1	$1 - h^{-2}$	$2 h^{-1}$
2	$1 - 3 h^{-2}$	$3 h^{-1} - h^{-3}$
3	$1 - 6 h^{-2} + h^{-4}$	$4 h^{-1} - 4 h^{-3}$
4	$1 - 10 h^{-2} + 5 h^{-4}$	$5 h^{-1} - 10 h^{-3} + h^{-5}$
5	$1 - 15 h^{-2} + 15 h^{-4} - h^{-6}$	$6 h^{-1} - 20 h^{-3} + 6 h^{-5}$
6	$1 - 21 h^{-2} + 35 h^{-4} - 7 h^{-6}$	$7 h^{-1} - 35 h^{-3} + 21 h^{-5} - h^{-7}$
7	$1 - 28 h^{-2} + 70 h^{-4} - 28 h^{-6} + h^{-8}$	$8 h^{-1} - 56 h^{-3} + 56 h^{-5} - 8 h^{-7}$
8	$1 - 36 h^{-2} + 126 h^{-4} - 84 h^{-6} + 9 h^{-8}$	$9 h^{-1} - 84 h^{-3} + 126 h^{-5} - 36 h^{-7} + h^{-9}$
9	$1 - 45 h^{-2} + 210 h^{-4} - 210 h^{-6} + 45 h^{-8} - h^{-10}$	$10 h^{-1} - 120 h^{-3} + 252 h^{-5} - 120 h^{-7} + 10 h^{-9}$
10	$1 - 55 h^{-2} + 330 h^{-4} - 462 h^{-6} + 165 h^{-8} - 11 h^{-10}$	$11 h^{-1} - 165 h^{-3} + 462 h^{-5} - 330 h^{-7} + 55 h^{-9} - h^{-11}$

The separation between even and odd powers for the x and y coordinates is even more apparent now. We can make SymPy extract the coefficients if we convert the formulas to polynomials. The all_coeffs() method will also insert zeros for all missing powers between the highest and the lowest:

def extractCoefficients(expr):
    # poly will error out if a constant expression is passed in
    coeffs = [expr] if expr.is_constant() else poly(expr).all_coeffs()
    coeffs.reverse()
    return coeffs

for x in xs:
    print(extractCoefficients(x))

for y in ys:
    print(extractCoefficients(y))

Padding the missing columns with zeros results in the following table of coefficients (the x and y coefficients are merged because the powers don't overlap):

$n$	1	2	3	4	5	6	7	8	9	10	11	12	13
$h^{0}$	1	1	1	1	1	1	1	1	1	1	1	1	1
$h^{-1}$	0	1	2	3	4	5	6	7	8	9	10	11	12
$h^{-2}$	0	0	-1	-3	-6	-10	-15	-21	-28	-36	-45	-55	-66
$h^{-3}$	0	0	0	-1	-4	-10	-20	-35	-56	-84	-120	-165	-220
$h^{-4}$	0	0	0	0	1	5	15	35	70	126	210	330	495
$h^{-5}$	0	0	0	0	0	1	6	21	56	126	252	462	792
$h^{-6}$	0	0	0	0	0	0	-1	-7	-28	-84	-210	-462	-924
$h^{-7}$	0	0	0	0	0	0	0	-1	-8	-36	-120	-330	-792
$h^{-8}$	0	0	0	0	0	0	0	0	1	9	45	165	495
$h^{-9}$	0	0	0	0	0	0	0	0	0	1	10	55	220
$h^{-10}$	0	0	0	0	0	0	0	0	0	0	-1	-11	-66
$h^{-11}$	0	0	0	0	0	0	0	0	0	0	0	-1	-12
$h^{-12}$	0	0	0	0	0	0	0	0	0	0	0	0	1

6. Complex Detour

The table form reveals a rather striking pattern: Sign aside, the coefficients in each row are the diagonals of the famous Pascal's triangle!

$\begin{array}{c} 1 \\ 1 \quad 1 \\ 1 \quad 2 \quad 1 \\ 1 \quad 3 \quad 3 \quad 1 \\ 1 \quad 4 \quad 6 \quad 4 \quad 1 \\ 1 \quad 5 \quad 10 \quad 10 \quad 5 \quad 1 \\ 1 \quad 6 \quad 15 \quad 20 \quad 15 \quad 6 \quad 1 \\ 1 \quad 7 \quad 21 \quad 35 \quad 35 \quad 21 \quad 7 \quad 1 \\ 1 \quad 8 \quad 28 \quad 56 \quad 70 \quad 56 \quad 28 \quad 8 \quad 1 \end{array}$

Indeed, if we sum the x and y coordinates in Table 1, we'll get an expression very similar to $(c + (-s))^n$ , except the signs are subtly wrong - this is particularly obvious when you examine the column-sum of the even rows. For $n = 2$ we get $c^{2} + 2 c s - s^{2}$ , which is similar to $(c - s)^2 = c^2 - 2 c s + s^2$ . The sign in front of $s^2$ is negative, which is only possible if $(-s)^2 = (-1)^2 s^2 = -s^2$ , but this implies that $-1$ squares to $-1$ , which goes against the usual rules of arithmetic.

Wouldn't it be nice if we could extend our rules of algebra to accommodate our repeated rotation formula? Focusing on the problematic $(-1)^n$ term, what if we had a special symbol, call it $i$ , that squares to $-1$ instead? Let's try it:

$\begin{aligned} (c + i s)^2 &= c^2 + 2 c i s + (i c)^2 = c^2 + 2 i c s -c^2 \\ (c + i s)^3 &= c^3 + 3 c^2 i s + 3 c i^2 s^2 + i^2 i c^3 = c^3 + 3 c^2 i s - 3 c s^2 - i c^3 \end{aligned}$

The signs are now correct, and, as an added benefit, grouping the terms containing $i$ separates the polynomials for the x and y coordinates! At this point you might be asking yourself an important question: Why does this work? If you've had any exposure to complex numbers you already know that $i$ is the (in)famous imaginary unit, and the geometric interpretation of complex numbers and their connection to rotations in the plane was first presented in 1799 by Wessel, a Norwegian surveyor, and was later popularised by Gauss in 1831, who discovered it independently. I'd like to give you a more modern interpretation, which relies on ideas that were not fully developed until the 20th century.

Around the middle of the 19th century, several mathematicians began experimenting with modifying or repurposing the rules of algebra to encode and perform operations on more abstract data. The search for higher-dimension-analogues of the complex numbers resulted in new imaginary units being added to algebra, developing the rich theory of hypercomplex numbers. At the same time, George Boole used algebra to represent logical operations, advancing Gottfried Wilhelm Leibniz's ideas of computational reasoning. Hermann Grassmann, a German school teacher, developed an algebraic system called Extension Theory, and used it to represent geometric calculations. Underlying Extension Theory was a theory of forms, which advocated for the separation of the computational system from its potential applications or physical interpretations:

"The essential advantages which were attained through this conception were 1) in relation to the form - now all principles which express views of space are entirely omitted, and consequently the beginnings of the science were as direct as those of arithmetic - and 2) in relation to content - the limitation to three dimensions is omitted." - Grassmann's 1844 "Die lineale Ausdehnungslehre, ein neuer Zweig der Mathematik" (translated to "Extension Theory" in English), as presented in Crowe's "A History of Vector Analysis"

Similar ways of thinking about mathematics eventually culminated in the formalist philosophical movement. My only problem with this movement is that its influence on education has been largely negative. While it's true that strong attachments to a particular interpretation can be limiting, stripping away the intuition and ideas that went into a particular formalism can be just as problematic. This is particularly apparent in physics, where practitioners are given highly abstract mathematical tools that they often end up mistaking for what's happening in the real world. Even experienced mathematicians are often forced to reverse engineer the intuition behind obtusely abstract presentations when they veer outside their realm of expertise.

Various mathematical notations were predecessors of modern programming languages, and none were more dominant than algebra. Can you think of ways to represent data structures and operations on them when all you have are abstract letters and the rules of arithmetic? We can convert our vector data structure to an algebraic expression by introducing abstract letters that represent each component, much like fields in a data structure described in a modern programming language. In honour of Grassmann, I'll call these variables $e_0$ and $e_1$ , and we can complete our data structure by connecting them with the plus sign: $e_0 + e_1$ . The coefficients in front of the variables represent the vector components, so a vector, say $(2, 3)$ , can be represented as $2 e_0 + 3 e_1$ . Adding or subtracting two expressions together corresponds to the usual vector linear combination, but multiplication is where we get to exercise our creativity! We can define various kinds of multiplication operations as long as they allow us to expand our data structure expressions like regular arithmetic multiplication (distributive property). Let's multiply $2 e_0 + 3 e_1$ by $4 e_0 + 5 e_1$ :

$(2 e_0 + 3 e_1)(4 e_0 + 5 e_1) = 2 \cdot 4 e_0 e_0 + 2 \cdot 5 e_0 e_1 + 3 \cdot 4 e_1 e_0 + 3 \cdot 5 e_1 e_1$

If you wanted to compute the dot product of two vectors via multiplication, you could define $e_0 e_0 = e_1 e_1 = 1$ and $e_0 e_1 = e_1 e_0 = 0$ , which allows us to clear out the unwanted values:

$2 \cdot 4 e_0 e_0 + 2 \cdot 5 e_0 e_1 + 3 \cdot 4 e_1 e_0 + 3 \cdot 5 e_1 e_1 = 8 \cdot 1 + 10 \cdot 0 + 12 \cdot 0 + 15 \cdot 1 = 23$

How about swapping the x and y components? Here's one way to do it: Multiply a vector expression by $e_1$ and define $e_0 e_1 = e_1$ and $e_1 e_1 = e_0$ :

$(2 e_0 + 3 e_1) e_1 = 2 e_0 e_1 + 3 e_1 e_1 = 2 e_1 + 3 e_0$

The choice of $e_1$ was arbitrary, and we could've used $e_0$ to accomplish the same result. Remember our quarter-turn rotation? Now that we can swap two components, we can just as easily negate the first one by defining $e_1 e_1 = -e_0$ instead, so quarter-turn rotation can now be expressed as:

$(v_0 e_0 + v_1 e_1) e_1 = v_0 e_0 e_1 + v_1 e_1 e_1 = -v_1 e_0 + v_0 e_1$

Adding a scalar value to a vector expression is meaningless, so we can get rid of $e_0$ and rename $e_1$ to $i$ to arrive at the standard complex numbers. I'll leave it up to you to verify that all our rotation formulas can be expressed in this algebraic form. In the spirit of Grassmann, it's important to note that interpreting the operation as a quarter turn rotation is entirely optional, and there are other contexts where swapping two entries in a list and negating one of them can be used for different purposes, like computing determinants to solve linear systems of equations, or calculating (hyper)volumes of parallelotopes, while keeping track of the measurement units. It even shows up in polynomial division, though that's a subject for a future article.

There's a subset of programmers who enjoy finding unintended ways to perform computations, to the point that it gives programming language designers and security-sensitive developers a lot of headaches. Mathematicians are not that different in this regard! Any time they mention the word "formal", you should be on the lookout for hidden data structures, and using polynomials in this way is particularly common. The imaginary unit was originally introduced as a notational device to keep track of square roots of negative values in case they end up being squared or cancelled out later in the computation. It greatly bothered mathematicians because sophisticated notions of computation were not yet developed, and even negative numbers (integers) were viewed with a great deal of suspicion throughout the ages - every "number" had to be associated with something tangible in the real world. Can you imagine programmers agonising over the use of abstract data types in their physics simulations? I don't know about you, but I don't see any hash maps, bounding volume hierarchies, or grids when I knock over a glass of water. I hope this section has convinced you to look at mathematics in a similar way!

7. Interpolation and Approximation

Now that we have a connection with (complex) exponentiation, we can simply use the Binomial theorem to produce symbolic expressions for the $n$ -th repeated rotation. However, in the spirit of exploration, I wanted to show you what we could've done if we didn't recognise Pascal's triangle. Our strategy will involve interpolation - finding a minimal degree polynomial that produces the appropriate coefficient for a given value of $n$ . This is a daunting task if you're not used to this kind of work, but I'll show you that it's not as difficult as it may seem! Feel free to skip this part if you're not interested.

Generating the coefficients for $h^{0}$ and $h^{-1}$ is easy - they're obviously $1$ and $n - 1$ respectively, where $n$ denotes the rotation count. $h^{-2}$ is where the pattern is not immediately obvious unless you have some experience with integer sequences. The easiest thing to do is look up the coefficients in the Online Encyclopedia of Integer Sequences (OEIS), which is an incredible resource that will save you lots of time if you frequently need to crack the patterns behind integer sequences. I tend to look up sequences in the OEIS all the time, but what if you didn't know about it or couldn't find yours in it? I'll show you a surprisingly versatile technique that can be used to analyse all kinds of sequences.

First, you want to find out how the sequence terms change over different values of $n$ , and the most natural way of doing that is computing the difference between consecutive terms. Those differences will form another sequence, and you repeat the same process with it until the consecutive differences become constant (comprised of the same value, e.g. $3, 3, 3 \ldots$ ), increase linearly (e.g. $1, 2, 3, 4, \ldots$ ), or you recognise the pattern in them. Until you get better intuition about the process, I recommend you keep taking consecutive difference until they're constant.

The next step relies on the observation that the $k$ -th integer of a sequence $S$ can be obtained by summing the first $k$ consecutive differences of the sequence, and adding them to the initial integer. Integers in the sequence of differences can, in turn, be computed as sums of their own consecutive differences - it's nested sums of differences all the way down! The skill of manipulating discrete sums is often neglected outside treatments of discrete mathematics, but it's an essential asset even for disciplines like analysis. I may write about it more in the future but, for now, let's return to the task at hand.

Going back to the coefficients of $h^{-2}$ , the difference between $10$ (for $n = 5$ ) and $6$ (for $n = 4$ ) is $4$ . If you repeat this process for all of the coefficients, you'll find that the differences between them form the following sequence: $1, 2, 3, 4, 5, 6 \dots$ In other words, $f(n) - f(n - 1) = n$ , where $f$ is the mystery function we're trying to find. Rearranging the equation we get $f(n) = n + f(n - 1)$ which makes it a little more obvious that getting to the $n$ -th coefficient is as simple as adding $n$ to the $(n-1)$ value. Starting from the initial value we get $f(1) = 0 + 1$ , $f(2) = f(1) + 2 = 0 + 1 + 2$ , $f(3) = f(2) + 3 = 0 + 1 + 2 + 3$ and our mystery function $f(n)$ appears to be the sum of all integers from 1 to $n$ . If you don't remember the formula for that sum, there's an easy way to derive it.

As usual, let's start simple: $S_4 = 1 + 2 + 3 + 4$ Notice that $1 + 4 = 2 + 3 = 5$ . Moving on, $S_5 = 1 + 2 + 3 + 4 + 5$ and $1 + 5 = 2 + 4 = 3 + 3 = 6$ . This observation suggests a general pattern: $S_n = (1 + n) + (2 + n - 1) + (3 + n - 2) \ldots = (n + 1 ) + (n + 1) + (n + 1) \ldots$ How many of those $(n + 1)$ terms do we have? Since we grouped the numbers in pairs of two we have $n / 2$ terms. Putting it all together:

$S_{n} = \frac{n (n + 1)}{2}$

You can check for a few values of $n$ that we do indeed get the same numbers as the coefficients for $h^{-2}$ . However, there's a minor problem: If we plug in $2$ into the formula we'd expect to get a $1$ , because that's the coefficient of $h^{-2}$ for the second rotation, but what we get back instead is $3$ , which is the coefficient for the third rotation. No problem, we can just shift the function by replacing $n$ with $n - 1$ , resulting in the following formula for the coefficients $C_2$ of $h^{-2}$ :

$C_2 = \frac{n (n - 1)}{2}$

Here's a table of successive differences for $n = 3$ :


$1$	$4$	$10$	$20$	$35$	$56$	$84$	$120$	$165$	$220$
$3$	$6$	$10$	$15$	$21$	$28$	$36$	$45$	$55$
$3$	$4$	$5$	$6$	$7$	$8$	$9$	$10$

At this point you can repeat the same process for the other coefficients (don't forget to apply the appropriate shifting where necessary), though you might already have a pretty good guess about the general pattern. Each time we increase the exponent $e$ (technically decrease in our case, but the sign is irrelevant), we multiply the formula for the previous coefficient by $(n - e + 1)$ and divide by $e$ . For example, the formula for the coefficients $C_3$ of $h^{-3}$ is:

$C_2 = \frac{n (n - 1) (n - 2)}{2 \cdot 3}$

and in general:

$C_{k} = \frac{n (n - 1) (n - 2) \ldots (n - k + 1)}{k!}$

where $n$ represents the rotation count and $k$ is the power of $h$ whose coefficient we're trying to determine. Depending on your maths background, you might recognise this as the binomial coefficients formula, usually denoted $\binom{n}{k}$ , which shows up in Newton's generalised binomial theorem. The approach I presented above is often called "method of successive differences", because of the way you repeatedly compute differences between sequences until your new sequence only contains one number, at which point you begin to work backwards to construct a polynomial that goes through every point in the starting sequence. This method lies at the heart of the Gregory-Newton Forward Difference Interpolation method, and it's how Newton originally arrived at his more general binomial formula. The reason why the formula is considered more general is because, as Newton realised, $n$ doesn't need to be a positive whole number anymore - the expressions for $n$ are just polynomials, and you can plug in negative or even rational numbers, though doing so means the sequence no longer terminates - you get an infinite loop! Curious readers may want to investigate interesting cases like $n = 1/2$ for complex numbers.

Now is a good time to write the two functions that generate the $n$ -th rotation polynomials for x and y so we can check whether we made any mistakes along the way:

Symbolic Rotation Formula

h, n = var('h, n')

def rotateX(rotationCount):
    hs = h * h
    accum = 0
    term = 1
    for i in range(1, rotationCount):
        # The sign has to alternate hence the minus in front
        # We want to generate 2!, 4!, 6!, (2*i)!, which can be split into:
        # (1*2) * (3*4) * (5*6) * ... * [(2*i - 1)*(2*i)] = i*(4*i - 2)
        denom = -hs * i * (4 * i - 2)
        # For the numerator we want the following sequence:
        # [(n - 0)*(n - 1)]*[(n - 2)*(n - 3)]*...*[(n - 2*i + 2)*(n - 2*i + 1)]
        term *= (n - 2*i + 2) * (n - 2*i + 1) / denom
        accum += term
    return accum + 1

def rotateY(rotationCount):
    if rotationCount == 0:
        return 0

    hs = h * h
    accum = n / h
    term = accum
    for i in range(1, rotationCount):
        # The sign has to alternate hence the minus in front
        # We want to generate 1!, 3!, 5!, (2*i - 1)!, which can be split into:
        # (2*3) * (3*4) * (5*6) * ... * [(2*i)*(2*i+1)] = i*(4*i + 2)
        denom = -hs * i * (4 * i + 2)
        # Because we start with n already in the term's numerator, we need to
        # generate the following sequence:
        # [(n - 1)*(n - 2)]*[(n - 3)*(n - 4)]*...*[(n - 2*i + 1)*(n - 2*i)]
        term *= (n - 2*i + 1) * (n - 2*i) / denom
        accum += term
    return accum

def rotate(rotationCount):
    print(f'x after {rotationCount} rotations:')
    x = rotateX(rotationCount)
    print(x.subs(n, rotationCount))

    print(f'y after {rotationCount} rotations:')
    y = rotateY(rotationCount)
    print(y.subs(n, rotationCount))

Calling rotate(6) gives us what we'd expect:

$\begin{aligned} x_6 &= 1 - 15 h^{-2} + 15 h^{-4} - h^{-6} \\ y_6 &= 6 h^{-1} - 20 h^{-3} + 6 h^{-5} \end{aligned}$

For our specific algorithm, $n$ , representing the number of rotations, is not an arbitrary number. Recall that the rotation count was determined by evaluating $d/h$ , which becomes $d h$ after we replace $h$ with its reciprocal. To understand how that's going to affect the final output for large values of $n$ , let's substitute $n = d h$ in our rotation formulas for a small rotation count, like 3, and expand:

expanded = lambda expr : expr.subs(n, var('d') * h).expand()
print(expanded(rotateX(3)))
print(expanded(rotateY(3)))

$\begin{aligned} x_3 &= 1 - \frac{d^{2}}{2} + \frac{d^{4}}{24} + \bigg( - \frac{d^{3}}{4 h} + \frac{11 d^{2}}{24 h^{2}} + \frac{d}{2 h} - \frac{d}{4 h^{3}} \bigg) \\ y_3 &= d - \frac{d^{3}}{6} + \frac{d^{5}}{120} + \bigg( - \frac{d^{4}}{12 h} + \frac{7 d^{3}}{24 h^{2}} + \frac{d^{2}}{2 h} - \frac{5 d^{2}}{12 h^{3}} - \frac{d}{3 h^{2}} + \frac{d}{5 h^{4}} \bigg) \end{aligned}$

I grouped all the terms that have $h$ in the denominator because we're interested in huge values of $h$ , which - combined with the factorial in the denominator - would make the terms in the brackets tiny; I'll call those terms the vanishing group. One crucial observation is that $C_{k}$ is a polynomial of the same degree as the corresponding negative power of $h$ , and the coefficient of the highest power term is guaranteed to be 1. This is why it makes sense to have terms where the $h$ values in the denominator cancel out the ones in the numerator, leaving only a power of $d$ on top. Generating the polynomials for 4 rotations makes our observation a bit more obvious:

$\begin{aligned} x_4 = \; &1 - \frac{d^{2}}{2} + \frac{d^{4}}{24} - \frac{d^{6}}{720} + \bigg( \frac{d^{5}}{48 h} - \frac{17 d^{4}}{144 h^{2}} - \frac{d^{3}}{4 h} + \frac{5 d^{3}}{16 h^{3}} + \frac{11 d^{2}}{24 h^{2}} - \frac{137 d^{2}}{360 h^{4}} + \frac{d}{2 h} - \frac{d}{4 h^{3}} + \frac{d}{6 h^{5}} \bigg) \\ y_4 = \; & d - \frac{d^{3}}{6} + \frac{d^{5}}{120} - \frac{d^{7}}{5040} + \bigg( \frac{d^{6}}{240 h} - \frac{5 d^{5}}{144 h^{2}} - \frac{d^{4}}{12 h} + \frac{7 d^{4}}{48 h^{3}} + \frac{7 d^{3}}{24 h^{2}} - \frac{29 d^{3}}{90 h^{4}} + \frac{d^{2}}{2 h} - \frac{5 d^{2}}{12 h^{3}} + \frac{7 d^{2}}{20 h^{5}} \\ &- \frac{d}{3 h^{2}} + \frac{d}{5 h^{4}} - \frac{d}{7 h^{6}} \bigg) \end{aligned}$

Notice that all the terms from the 3-rotation formulas are still present. What would the general structure of the formulas look like as we add more and more rotations? It seems like the group comprised of even or odd powers of $d$ with the factorial of the power in the denominator will continue to expand. Furthermore, the larger $h$ becomes, the smaller the total sum of the vanishing group will be; and if approximate the expression by getting rid of terms containing $h$ , we'd be left with very pleasant formulas for our x and y coordinates:

$\begin{aligned} x_{n} = 1 - \frac{d^2}{2!} + \frac{d^4}{4!} - \frac{d^6}{6!} + \frac{d^8}{8!} - \ldots \\[0.4em] y_{n} = d - \frac{d^3}{3!} + \frac{d^5}{5!} - \frac{d^7}{7!} + \frac{d^9}{9!} - \ldots \end{aligned}$

Mathematicians have a name for these kinds of formulas: series. Take the time to fully process what we've just done because it can be quite disorienting when you first encounter such a transformation.

We converted the coefficients into generating polynomials in $n$ , and that gave us additional information about the structure of the computation. A beautiful pattern was revealed after substituting for $n$ : Each rotation in the brute-force algorithm decomposed into a polynomial in $d$ and a separate group of terms with $h$ in the denominator. This allowed us to determine that for very large values of $h$ - and therefore potentially billions of rotations - the polynomial we'd get in the end would have the same general structure, and so we could simplify the computation by eliminating the part of it that contributes very little to the final result - the vanishing group.

However, we still have a problem: How do we know when to stop? Surely if we continue carelessly adding terms we run the risk of overshooting the target location on the circle! Morever, how do we even know that the final result will actually match - or be close enough to - what the brute-force algorithm would give us?

These are not trivial questions, and over several centuries, mathematicians have developed a lot of tools to help answer them. If you were studying these series in isolation, you'd have to look for insights within a field called analysis. But we might be able to take advantage of the context in which we discovered them to help guide us towards more intuitive answers.

Let's go back to the premise behind the brute-force algorithm: The larger we make $h$ , the more rotations we need to perform, and the closer we'll get to the target location. Performing more rotations would result in generating more and more terms in the series, while the vanishing group would continue to shrink. This directly implies that the two series will not only converge on finite values, but those values will close in on the location that we're looking for! While this is not considered a formal proof, I find this kind of intuitive reasoning remarkably beautiful.

To wrap up this section, let's translate the series to Python functions:

def cosB(x, iters = 11):
    xs = x*x
    accum = 1
    term = 1
    for i in range(1, iters+1):
        term *= -xs/(i*(4*i - 2))
        accum += term
    return accum

def sinB(x, iters = 11):
    xs = x*x
    accum = x
    term = x
    for i in range(1, iters+1):
        term *= -xs/(i*(4*i + 2))
        accum += term
    return accum

Evaluating these functions with $d = 2$ gives us a similar result to our brute force method:

Brute-force:
d=2, h=0.00001, rotations=200000
x=-0.41614683648618206
y=0.9092974268526915

Series:
x=-0.41614683654714246
y=0.9092974268256817

The output values match up to the 9th and 13th decimal respectively (the series x is actually more accurate), yet we only ran a total of 22 iterations compared to the 200000 for the brute-force method!

You might have noticed that I picked rather peculiar names for the functions instead of calling them, say, coordinateX and coordinateY. After all, we're computing the coordinates we'll arrive at if we travelled $x$ distance units along the circle. Unfortunately, mathematics terminology is stuck with these mistranslations of Arabic texts so you'll typically see the functions computing the x and y coordinates being called cosine (or $\cos$ ) and sine (also $\sin$ ) respectively. I decided to use those names as well, even though I dislike them, because they're already in the title of this article and will likely be more familiar to you.

Congratulations if you made it this far! The hardest part is over, but there are a few more optimizations we can leverage.

8. Range Reduction

Suppose you travelled 0.5 turns along the circle, took a short break, and then decided to continue moving for another full turn. Unless you're trying to exercise, this is clearly pointless because you'll end up in the same spot after travelling a total of 1.5 turns from the starting point. In fact, any whole number multiple of one turn will bring you back to the same location. If I asked you to compute the location where you'll end up after travelling 42.5 turns, you don't actually need to plug that value directly into our cosine and sine formulas - assuming you knew how to convert turn units into distances - because you can simply subtract 42 and plug in 0.5 instead.

This simple observation allows us to reduce any arbitrary distance to the range 0 to 1 turns, which is highly desirable from an algorithmic standpoint since it puts a fixed bound on the number of computations we need to perform (given some desired degree of accuracy). However, the pesky turn-to-distance conversion problem is blocking our progress once again! Let's figure out how to measure the length of the unit circle's boundary (its circumference) and put the conversion beast to rest once and for all!

8.1 Unit Circle Circumference

As you might suspect by now, any time I'm faced with a new problem I always attempt to solve it in the simplest way I can think of because that gives me a baseline for comparisons with potential future solutions - it also provides me with initial insight into the structure of the problem, which, as you learned in the previous sections, can inspire clever optimizations. Using the tools we've already developed, what would be your initial attempt at measuring the circumference of a unit circle?

My first thought was to simplify the measurement by relying on the fact that the full length is comprised of 4 equal pieces, belonging to each quadrant (quarter turn slice of the circle), and so computing the circumference can be done by measuring the length of the counter-clockwise arc from $(1, 0)$ to $(0, 1)$ and multiplying the resulting value by 4.

Next, I knew I could rotate in tiny increments until the x coordinate becomes negative, which would indicate that we crossed the $(0, 1)$ vector. Recording the number of rotations we performed before crossing the vertical line and multiplying that value by the chosen $h$ height of the rotated triangle will give us an approximation for the length of the quadrant arc. Alternatively, we could use our cosine formula and slowly increment the distance input until the computed value becomes negative. The latter approach seemed more convenient so I went with it:

x = 0 # starting value and also the upper bound at the end
increment = 0.1 # increment x by this value on each iteration
xprev = x # store the previous value to get a lower bound

while True:
    x += increment
    if cosB(x) < 0:
        break
    xprev = x

print(f'Lower bound: {xprev}\nUpper bound: {x}')

Lower bound: 1.5000000000000002
Upper bound: 1.6000000000000003

I started with a coarse approximation to get a general sense for the interval in which the true values resides. Once I knew it's somewhere between 1.5 and 1.6, I changed x to 1.5, the increment to 0.0000001, and ran the same piece of code again:

Lower bound: 1.5707963000413356
Upper bound: 1.5707964000413357

You could continue substituting the lower bound for x and decreasing the increment to get better and better approximations but that felt too clunky and inelegant so I decided to change my approach. There are two main problems with our current implementation: The increment is fixed so we need to manually adjust it, and we're also terminating the computation as soon as the cosine becomes negative. Can you think of a way to modify the algorithm so the increment is decreased automatically and the negative value check is eliminated?

If you think about the behaviour of cosine around the $(0, 1)$ point, the closer we get to it, the smaller the cosine value becomes. This means that we could use the computed cosines as the increments for each step! Furthermore, as soon as the cosine becomes negative, its value will automatically decrease the approximated distance, and we could use this self-correcting behaviour to get rid of the if statement that terminates the computation. Implementing these adjustments gives us the following quadrant arc length algorithm:

def quadrantArcLength(iters = 2):
    x = 1.57
    for _ in range(0, iters):
        x = x + cosB(x)
    return x

pi = quadrantArcLength()*2
circumference = pi*2
print(f'Pi: {pi}\nCircumference: {circumference}')

Pi: 3.141592653589793
Circumference: 6.283185307179586

Imagine yourself picking up a circle, cutting the boundary curve somewhere, and straightening it out into a line. The length of that line is the circumference (often denoted $\tau$ and pronounced ta-oo) and is roughly 6.28 times longer than the radius (which is 1 for a unit circle). Dividing the circumference by 2 will give us the famous mathematical constant $\pi = 3.141592653589793...$ For the sake of convenience, I'll start expressing circular distances in terms of $\pi$ : The length of a quadrant arc then becomes $\pi/2$ , half a turn would be $\pi$ , the circumference is $2\pi$ , and so on. As a proponent of the $\tau$ notation this wasn't an easy decision, but it is the more common approach.

Knowing the circumference also helps answer the turn value question: One turn is equal to ~6.28 distance units (also called radians) and with the help of our handy cosine and sine functions, we can map turn units to coordinates on the circle by first converting the turn value to its radians representation! Over the years, people have come up with various conventions where the distance unit range $[0, 6.283185307179586]$ is mapped to more convenient values such as $[0, 360]$ (degrees) and the $[0, 1]$ turns we're already familiar with.

If you're curious about the more general principle behind the quadrant arc length algorithm, you can look up the term fixed-point iteration. Briefly, the idea is to leverage structural properties of a given function to iteratively close in on a desired value. The Babylonians, for example, used a similar approach to derive an algorithm for computing square roots.

We now have a good approximation for the circumference of the unit circle so we can divide any given distance value by it to find how many turns around the circle we'll make, and then subtract the distance travelled in those turns from the input to get a range reduced value for our computations:

def rangeReduce(x):
    # In Python you could also do x % circ to compute the same value
    # but I'm being explicit here for the sake of clarity
    k = math.floor(x / circ)
    x = x - k * circ
    return x

Let's augment our base cosine and sine functions:

def cos1(x, iters = 11):
    x = rangeReduce(x) # same as x = x % circ
    return cosB(x, iters)

def sin1(x, iters = 11):
    x = rangeReduce(x) # same as x = x % circ
    return sinB(x, iters)

So far, we've managed to reduce our input range from $[0, \infty)$ to $[0, 2\pi]$ , but we're not done yet!

8.2 Quadrants

Having computed the length of a single quadrant, we have sufficient information to determine which quadrant we'll end up in if we computed the cosine and sine values for a given input. For example, if the input distance (after the initial range reduction) is larger than $\pi/2$ we know it won't be in the first quadrant; if it's larger than $\pi$ , it won't be in the first or second quadrants. We can binary search for the correct quadrant and find it in just two checks, but how does that help us?

If you pick some vector in the first quadrant with coordinates $(\cos(x), \sin(x))$ , you can rotate it around the circle using the familiar swapping of coordinates and negating the x value. Each rotation corresponds to a $\pi/2$ increase in the distance $x$ , which means that, for example, $\cos(x+\pi/2) = -\sin(x)$ because that's the x coordinate in the 2nd quadrant. If we compute all the adjusted values we get the following table:

Quadrant	Coordinate	Adjusted
1	$(\cos(x), \sin(x))$	$(\cos(x), \sin(x))$
2	$(-\sin(x), \cos(x))$	$(\cos(x + \pi/2), \sin(x+\pi/2))$
3	$(-\cos(x), -\sin(x))$	$(\cos(x + \pi), \sin(x+\pi))$
4	$(\sin(x), -\cos(x))$	$(\cos(x + 3\pi/2), \sin(x+3\pi/2))$

Now, suppose you reversed this process and started with an unknown vector whose associated distance value places it in the 3rd quadrant. Instead of computing its location directly using the cosine and sine formulas, you could subtract $\pi$ from the distance, calculate the cosine and sine in the $[0, \pi/2]$ range and then substitute the computed values in the coordinate column for the 3rd quadrant in the table above - in this case, all you need to do is negate them:

def cos2(x, iters = 11):
    x = x % (2*pi)

    if x < pi:
        if x < pi/2:
            return cosB(x, iters)
        return -sinB(x - pi/2, iters)

    if x < pi*3/2:
        return -cosB(x - pi, iters)
    return sinB(x - pi*3/2, iters)

def sin2(x, iters = 11):
    x = x % (2*pi)

    if x < pi:
        if x < pi/2:
            return sinB(x, iters)
        return cosB(x - pi/2, iters)

    if x < pi*3/2:
        return -sinB(x - pi, iters)
    return -cosB(x - pi*3/2, iters)

8.3 Mirror Symmetry

We're almost done with the range reduction optimizations! There's just one more structural property we haven't exploited yet: mirror symmetry (reflection). Let's take a vector $(x, y)$ as an example. Switching the $x$ and the $y$ will give us $(y, x)$ , which is a reflection around the $y=x$ axis. If we set $x = \cos(x)$ and $y= \sin(x)$ we get the following picture:

Note how the line $y=x$ splits the arc bounded by $0$ and $\pi/2$ in half. Maybe we can somehow leverage the symmetry around the half-way distance, $\pi/4$ , to shrink the $[0, \pi/2]$ range to $[0, \pi/4]$ !

More concretely, suppose you were interested in calculating the cosine and sine values for a circular distance value $x$ between $\pi/4$ and $\pi/2$ . Swapping the cosine and sine values will give us the reflected vector that lies somewhere between $0$ and $\pi/4$ . How do we compute the distance value - let's call it $r$ - that corresponds to the reflected vector? If we knew how to calculate $r$ , we could compute $\cos(x)=\sin(r)$ and $\sin(x)=\cos(r)$ !

The key observations that will help us with this problem are that $(1, 0)$ gets reflected into $(0, 1)$ and the fact that reflections preserve distances, so the circular distance between the horizontal base $(1, 0)$ and $(\cos(r), \sin(r))$ is the same as the circular distance between the vertical base $(0, 1)$ and $(\cos(x), \sin(x))$ . This means that if we subtract the value of $x$ from $\pi/2$ , we'll find $r$ ! Putting everything together we arrive at a very handy relation between cosines and sines:

$\begin{aligned} \cos(x) &= \sin\bigg(\frac{\pi}{2} - x\bigg) \\ \sin(x) &= \cos\bigg(\frac{\pi}{2} - x\bigg) \end{aligned}$

To make use of this relation in our cosine and sine code, we could check whether the range reduced $x$ is larger than $\pi/4$ and subtract it from $\pi/2$ if it is - in addition to swapping cosines for sines and vice versa. However, if we apply this range reduction directly, we'll end up with very unpleasant code because we're already alternating between cosB and sinB statements all over the place. Let's clean up the code a bit in preperation for the final range reduction step by rewriting cos2 so it only calls cosB and get rid of all calls to cosB in sin2. We can use the formulas we've just discovered to perform this transformation. For example, we can convert $\mathrm{sinB}(x - \pi/2)$ to $\mathrm{cosB}$ by negating the input value and adding $\pi/2$ to it:

$\begin{aligned} \mathrm{sinB}\bigg(x - \frac{\pi}{2}\bigg) &= \mathrm{cosB}\bigg(-\bigg(x - \frac{\pi}{2}\bigg) + \frac{\pi}{2}\bigg)\\ &= \mathrm{cosB}(\pi - x) \end{aligned}$

This would allow us to make the code a lot more uniform and we can apply the $[0, \pi/4]$ range reduction as the final step:

def cos3(x, iters = 11):
    # [0, 2*pi] range reduction
    x = x % (2*pi)
    sign = 1
    # [0, pi/2] range reduction
    if x < pi:
        if x < pi/2:
            pass # leaving this for clarity
        else:
            x = pi - x
            sign *= -1
    else:
        if x < pi*3/2:
            x = x - pi
            sign *= -1
        else:
            x = 2*pi - x
    # [0, pi/4] range reduction
    c = cosB(x, iters) if x < pi/4 else sinB(pi/2 - x, iters)
    return c*sign

def sin3(x, iters = 11):
    # [0, 2*pi] range reduction
    x = x % (2*pi)
    sign = 1
    # [0, pi/2] range reduction
    if x < pi:
        if x < pi/2:
            pass
        else:
            x = pi - x
    else:
        sign *= -1
        if x < pi*3/2:
            x = x - pi
        else:
            x = 2*pi - x
    # [0, pi/4] range reduction
    s = sinB(x, iters) if x < pi/4 else cosB(pi/2 - x, iters)
    return s*sign

9. Negative Angles

I've been ignoring negative distances so far to keep the incremental development of the algorithms more focused. Fortunately, addressing this omission is fairly straightforward!

First, what does a negative distance mean and does it even make sense to consider it? After all, we could always restrict the inputs to our cosine and sine functions to positive values only. Intuitively, I think of negative distances as rotations in the opposite direction - clockwise instead of the counterclockwise direction we've been working with so far. However, just because that makes sense to me intuitively doesn't mean it will fit the analysis we've done so far, or that our algorithms will produce a meaningful result when a negative value is fed into them.

In the previous section we actually ran into a disguised negative input: $\cos(\pi/2 - x)$ , $\sin(\pi/2 - x)$ Because our rotations compose, we can interpret those expressions as rotating the vector located a circular distance of $\pi/2$ units from $(1, 0)$ - which is $(0, 1)$ - by the vector $(\cos(-x), \sin(-x))$ . Moreover, we already know what the final result is: $(\sin(x), \cos(x))$ . If we're going with the idea that negative distances are rotations in the opposite direction, we can cancel out the $\pi/2$ value in $\pi/2 - x$ by performing a $\pi/2$ clockwise rotation whose formula (I'll call it $\mathrm{rotQc}$ ) can be established with the same thought process we used all the way back in the beginning:

$\mathrm{rotQc}((a, b)) = (b, -a)$

We're still exchanging the coordinate values but this time it's the second one that gets negated. Applying $\mathrm{rotQc}$ to our vector of interest yields the following:

$\begin{aligned} (\cos(-x), \sin(-x)) &= \mathrm{rotQc}((\cos(\pi/2 - x), \sin(\pi/2 - x)) = \mathrm{rotQc}((\sin(x)), \cos(x)) \\ &= (\cos(x), -\sin(x)) \end{aligned}$

This is just a vertical reflection of the original vector so it fits well with our starting idea of $h$ as the height of a right triangle. In fact, simply allowing $h$ to be negative in the repeated rotation calculations already gives us a good idea about the behaviour of a negative input:

The rotation direction is reversed as expected! A negative $h$ would also cancel out the negative sign in the circular distance input of our brute-force cosine and sine algorithm when computing the rotation count. Plugging in a negative circular distance value in the optimized formulas is equally pleasant: The even powers in the cosine will cancel the negative sign while the odd powers in the sine will flip the sign of the final result, which matches the results we obtained from leveraging the compositional property of 2D rotations! What this means is that we can treat negative inputs as positive ones as long as we remember to negate the sine value if the input was negative:

def cos4(x, iters = 11):
    return cos3(abs(x))

def sin4(x, iters = 11):
    if x < 0:
        return -sin3(-x)
    return sin3(x)

9.1 Cosine and Sine Composition

Now is a good time to pause and examine an implicit assumption we've been making throughout the last few sections: The validity of replacing multiples of $\pi/2$ radians with quarter turn rotations. Suppose we wanted to evaluate $\cos(\pi/2 + x)$ and $\sin(\pi/2 + x)$ . Does it make sense to pull the $\pi/2$ out of the function brackets, compute $(\cos(x)$ , $\sin(x))$ , and apply a quarter turn rotation to this vector to produce the final result? In other words, is the following a valid transformation:

$(\cos(\pi/2 + x), \sin(\pi/2 + x)) = \mathrm{rotQ}((\cos(x), \sin(x))) = (-\sin(x), \cos(x))$

Recall our prior discussion of rotation composition in the beginning of this article: We can use vectors to represent rotations and then compose those vectors to produce a new vector that encodes the sum of the input rotations. The vector that encodes the $\pi/2$ rotation is $(0, 1)$ , and the vector corresponding to a rotation by $x$ is $(\cos(x), \sin(x))$ . If we composed these two vectors with the arbitrary rotation formula we'll get an output vector that represents a rotation by $\pi/2$ followed by a rotation by $x$ (or the other way around), which is equivalent to evaluating the cosine and sine functions with $\pi/2 + x$ as input:

$(0 \cdot \cos(x) - 1 \cdot \sin(x), 1 \cdot \cos(x) + 0 \cdot \sin(x)) = (-\sin(x), \cos(x))$

This is precisely what we'd expect! There's actually nothing special about $\pi/2$ here, we can just as easily replace it with an arbitrary value $y$ and compute the composition of $(\cos(x), \sin(x))$ and $(\cos(y), \sin(y))$ :

$(\cos(x+y), \sin(x+y)) = (\cos(x)\cos(y) - \sin(x)\sin(y), \cos(x)\sin(y) + \sin(x)\cos(y))$

Note that this is the more general form of the double angle formula - substitute $y=x$ to check - and you can take the output vector and rotate it by $(\cos(z), \sin(z))$ to produce the general triple angle formula, repeating this process for as long as you want. The algebraic nature of trigonometry is quite beautiful once you start seeing the patterns in it!

10. Final Algorithm

It's been a long journey but we're finally ready to wrap things up! Putting everything together and eliminating some redundant bits results in the following code:

def cosB(x, iters = 11):
    xs = x*x
    accum = 1
    term = 1
    for i in range(1, iters+1):
        term *= -xs/(i*(4*i - 2))
        accum += term
    return accum

def sinB(x, iters = 11):
    xs = x*x
    accum = x
    term = x
    for i in range(1, iters+1):
        term *= -xs/(i*(4*i + 2))
        accum += term
    return accum

def quadrantArcLength(iters = 2):
    x = 1.57
    for _ in range(0, iters):
        x = x + cosB(x)
    return x

pi = quadrantArcLength() * 2

def cos(x, iters = 11):
    # [0, 2*pi] range reduction
    x = abs(x) % (2*pi)
    sign = 1
    # [0, pi/2] range reduction
    if x < pi:
        if x > pi/2:
            x = pi - x
            sign = -1
    elif x < pi*3/2:
        x = x - pi
        sign = -1
    else:
        x = 2*pi - x
    # [0, pi/4] range reduction
    c = cosB(x) if x < pi/4 else sinB(pi/2 - x)
    return c*sign

def sin(x, iters = 11):
    sign = 1
    if x < 0:
        sign = -1
        x = -x
    # [0, 2*pi] range reduction   
    x = x % (2*pi)
    # [0, pi/2] range reduction
    if x < pi:
        if x > pi/2:
            x = pi - x
    else:
        sign *= -1
        if x < pi*3/2:
            x = x - pi
        else:
            x = 2*pi - x
    # [0, pi/4] range reduction
    s = sinB(x) if x < pi/4 else cosB(pi/2 - x)
    return s*sign

10.1 Further Improvements

That was a fair bit of work but how close did we get to the actual implementations you'd find in calculator software or various maths libraries? Surprisingly, while our algorithm will be reasonably accurate for small distance values, the first step in the range reduction will become the source of increasing error for larger inputs. The limited precision of our $\pi$ calculation is the primary culprit, but addressing this limitation would require us to either use an arbitrary precision maths library - which would be simple but slow - or employ more clever - and complex - techniques. This is a common theme in numerical computing: Obtaining accurate results can be relatively easy, but doing that quickly and efficiently is where all the difficulty lies! If you're curious, you can look up the 1983 Payne-Hanek algorithm which is often used as a reliable fallback whenever faster techniques can't handle a particular input value. A variety of improvements on this idea have been made over the years, such as adding fast paths and multi-precision floating point evaluation.

While all this may sound somewhat discouraging, our $[0, 2\pi]$ range reduction is actually good enough for a lot of applications (especially in the real-time domain). The following table shows the absolute error in our implementation - relative to Python's cosine and sine functions - for two fairly large inputs:

$x$	$\cos(x)$	$\sin(x)$
$2^{32}-1$	$8.31^{-8}$	$1.45^{-7}$
$2^{53}-1$	$0.07$	$0.34$

Inputs as large as $2^{53}-1$ have already lost all floating point precision so the large error values aren't particularly significant. $2^{32}-1$ is a far more reasonable input for double precision floats and the error values for it are small enough to be acceptable for most of the software I've worked on.

We could further improve the performance of our algorithm by determining the lowest degree polynomial that would give us accurate results in the $[0, \pi/4]$ range for a desired precision. You can employ the Remez algorithm to find these polynomials, or use a framework like Sollya that will also take care of various floating point pitfalls for you.

Lastly, there's the concept of correctly rounded cosine and sine computations which aim to determine the floating point value closest to the "exact" result. Due to their negative impact on performance and code size, correctly rounded algorithms are typically not used unless calculations need to be fully deterministic across different platforms as part of a specific standard.

I might explore some of these topics in future articles because they're too complex for an introductory presentation. However, you should now have a good sense for how cosine and sine functions are implemented in the real world!

11. Conclusion

Using the tools we developed you can easily derive other famous trigonometry formulas because they're mostly triangle identities in disguise. After all, the word "trigonometry" can be translated to "triangle measurement" from its Greek origin. All you need to do is construct a unit circle on one of the triangle's corners and exploit the fact the circle will intersect the sides in segments with a length of 1 - the rest is just clever use of similarity ratios.

The goal of this write-up was to convince you that all the cryptic trigonometry relations you learned about in school are primarily algebraic in nature and naturally flow out of a small and simple set of ideas. One does not need to have any concept of the cosine and sine functions to do trigonometry, though they are still useful in applied mathematics. I deliberately used $c$ and $s$ instead to avoid needless confusion and the mathematician Norman Wildberger went a step further and demonstrated that you can do trigonometry just fine with rational numbers alone.

I believe the subject would be a lot easier to grasp if schools taught vector algebra earlier and used it as a foundation for developing trigonometry (my teachers developed it on top of classical Euclidean geometry instead). Computer algebra systems would greatly alleviate a lot of the tedium inherent in the exploration of the algebraic structure of rotations, while simultaneously exposing students to the immensely practical field of programming.

Looking back on my maths education, I recall many times when I felt lost and confused. I had a rather complicated relationship with trigonometry in particular because I learned how to apply and manipulate the formulas, yet, regardless of how much "proficiency" I acquired, I could never shake off a certain nagging voice in my head: "You don't really understand what you're doing, you've just learned how to play the trigonometry game!"

I had high hopes that my university maths classes would help answer all the accumulated questions, yet the professor simply assumed that we already understand the subject and only gave us a brief refresher on the opaque formulas I had memorized in my teenage years. This article is one of the maths lessons I wish I could give to my younger self and I hope it provided some value to you too!