2026-165

Transitive dependency management is still hard.

Recently I started replacing SDL windowing with my own. This got me into Wayland. When developing against the Wayland protocols in C or C++, you run wayland-scanner to parse protocol definitions from XML files and generate C code. This C code features many static inline functions, which the Jai bindings generator won't touch, meaning we are left to handroll many convenience function implementations on our own. I gave this a shot, but maintenance was obviously going to be a huge chore, so instead I reimplemented wayland-scanner in Jai, to emit Jai.

At first I did all of this in a single repository, the same one that I was building my own platform-abstracting code in. This was fine and well. But I figure the Jai wayland-scanner is useful. There are several other Jai implementations in existence, but I haven't seen a standalone one yet, or one that doesn't require hand-writing more code than my implementation requires (in my case, it's just forward declarations of opaque types). So I thought to extract wayland-scanner to share it, and then I spent the next couple hours figuring out how to sanely handle the transitive dependencies, as many others have done in this field.

The dependency graph looks like this.

 /-> b -> c -> d
a
 \-> d

And this is all it takes to get us into trouble. (The case of a -> b -> c and also a -> c--that is, removing d--also demonstrates the trouble but is easier to resolve, so we take the more involved case in this post.)

It doesn't really matter, but concretely these are the repos:

a: the game with platform abstraction code
b: Wayland bindings
c: wayland-scanner-jai
d: xml-jai (my own, not the more popular one by the same name)

We want xml-jai to be separate from the scanner, because if I want to parse XML, I don't think to reach for wayland-scanner-jai. We want the scanner to be separate from the bindings, because if I just want to use pre-generated bindings, I don't want the code for the scanner or XML parser. Also, the bindings contain a little bit of glue code required to get the scanner's output compiling, and this code is out of scope of the scanner (following the scoping in the original scanner implementation).

Jai currently has no package manager. While there are already community efforts mulling over an implementation, Jon and Thekla have no intent to build a first-party one. I sympathize with the former and latter.

My philosophy on dependencies is, whenever possible, don't have them. You can accomplish this in 3 ways, from least extreme to most.

The first is questioning whether you really need to do what the candidate dependency does in the first place. It's easier to say "just don't do that" than it is to actually not do that, but it's always worth exploring. For example, when programmatically loading Aseprite assets into my engine, I would have had to deal with JSON since that is what the first-party exporter supports. I explored writing a JSON parser in Jai, and that was interesting, but I wasn't satisfied with it. Instead, I extended the Aseprite-side export tool to emit a binary format that I easily parsed in Jai.

The second is reimplementing a minimal version of the dependency you have in mind. I did this with the above-mentioned XML parser. In about 100 lines of code, I have a parser robust enough to parse everything I need from the Wayland protocol definitions. The code is very simple, and it erased the need to grab a dependency from somewhere else.

The third is bringing the external dependency into your project verbatim. Yes, this is technically still "having a dependency," but the idea is that you avoid all of this complicated machinery required to dynamically fetch the external dependency at build time, with the right upstream source and version, etc.

With that said, the second and third approaches here still face the transitive dependency issue mentioned above once you want to share that part of the code with others, even potentially internally in the same project if your development style is one of those insane ones with hundreds of people living on their own feature branches for months. (If you haven't seen this, this is more common than you might think, for example in traditional automotive, where initial release cycles are on the order of years.) The third approach also has another problem--synchronizing patches--which I'll cover later.

Oh, also, how many XML parsers does the world really need? For real.

On the other hand, we have sophisticated dependency management solutions. Think Bazel, Cargo, pip, etc. with their protocols and remote registries. These solve real problems, but they also introduce new ones.

The biggest problem is one of end-user discipline or even just innocent coincidence, and that is dependency creep. You need this one dependency, and so you bring it in. Someone else needs another, and the diff to bring that one into the codebase is a one-line change in some dependency registry file. Without discipline, you see where this is going. More nefarious is one of your dependency's transitive dependencies. Maybe it only has a single transitive dependency, but then maybe when you pull in upstream changes, the dependency now has 5 new dependencies. This recurses down the dependency tree, with cases of "just adding one dependency" ending up making you suddenly transitively reliant on 10+ projects. In other words, a one-line change in your project is actually a step function jump in project complexity.

No one on your project is reviewing all the new commits across all dependencies, so you're succeptible to vulnerabilities or even malicious code--this becoming even more common in the budding age of agentic AI--in your dependencies.

There's also the amusing situation of asking a package manager for two dependencies, each of which wants a different version of the same transitive dependency. With strict configuration, you can ban this, and you will get a configure or build-time error. But we actually need to build our software, so at best you can leverage the package manager's compatibility resolver to find a version that satisfies both dependencies. Of course, you're still at the mercy of whoever wrote that shared dependency having gotten the versioning and compatibility correct. At worst you end up getting two incompatible versions of the same dependency anyway and using them under the hood.

These aren't digs at package managers. This is a real problem, "how do I build this thing without boiling the ocean" is a valid question, and these three situations are the only answers in the package manager world.

These points have been exhausted before, elsewhere.

Other nuisances are the machinery that is the dependency manager. It's got its own configuration language (ex: Blaze/Bazel's Starlark, Cargo's use of TOML, pip and company's use of ini-like formats, or TOML, or JSON, or even programmatic configuration through Python), caching strategy, proxy nuances, and version resolution algorithm. Now if you want to be a "foo-lang" expert, you also need to be an expert of some foo-lang build tool, even if you found yourself only needing 10% of that tool's functionality across your career. If you work for BigCorp, you will also probably end up with a FooLang Build Tool team.

I digress. The point at hand in this post is the challenge of different parts of a larger codebase depending on different versions of the same dependency. I'd really like to avoid that. Reiterating my above diagram:

 /-> b -> c -> d
a
 \-> d

How does one ensure that both versions of d are the same? This might not be important, until you try to transact at a -> d using data produced at the c -> d interface only to find that data that works with one version of d doesn't work with another in your same codebase. Okay, so maybe the versions don't have to be the same; they just have to be compatible, right? Enter API and ABI compatibility, versioning, semantics guarantees, etc. and take the personal productivity hit. Or, if you're a bigger company, reify all of that and end up spending time and money addressing symptoms instead of causes and have it justified to anyone who questions as adhering to "best practices."

Jai does have a code re-use mechanism, called "modules." In my concrete example, if you want wayland-scanner-jai to be buildable standalone, (let's say the project is managed with git) then you could depend on xml-jai via submodule. You could also just bring in a source copy of xml-jai. The submodule approach has the benefit of tracking exactly which version of xml-jai you depend on and also providing diffs of any changes you've made to your version relative to changes upstream. This allows you to easily bring in upstream changes or submit local fixes upstream.

I promised above I'd say something about it, so briefly, the source-copy approach has none of these benefits, and maintaining synchronization with the original source, if desired, is a chore. This chore only gets more complicated when you leave the project or codebase and someone else has to figure out the current relationship, if any, between your copy and upstream. On the other hand, you are free to do as you please and integrate what you want how you want in the source-copy, and your build very likely stays simpler.

Regarding the scanner, ideally it is buildable standalone, so that it can be tested standalone. That implies the scanner either has a source copy of xml-jai or a submodule for xml-jai. But now say in my game engine, I want to depend on the scanner, build the scanner as part of my project-top-level build procedure, and then run the scanner later in my build for ultimate developer convenience. Do I run

git submodule update --init

and should I add --recursive? That's not terribly different from the package manager situation conceptually. So maybe I should bring in the xml-jai dependency by myself and put it on the module search path in my game engine. But now I'm faced with the compatibility problem again. Or maybe the scanner has its own copy of xml-jai. How many times do I want to debug bespoke implementations of things accomplishing common tasks and also maybe deal with upstreaming fixes?

If I segue for a moment and take a more macroscopic look, we have platforms like Android which say, look, we can't fix this, use whatever version of whatever you want, and now your Android runtime has X versions of Y package in it. iOS is stricter, taking in places the approach of, if you want your software to run on version foo, then you need to build against version U of software/interface V.

Canonical takes a more traditional approach of combining together whatever exists in the software and package ecosystem required to get the job done, albeit somewhat dated versions. When preparing their next operating system release, they select slightly older verions of software and packages that have proven some degree of stability and compatibility in the field already. It's a simple approach, with the downside being you're running on older software and one botched apt install could leave your system in a broken state.

NixOS says, let's do declarative system configuration with atomic updates. Now the "works on my machine" issue is solved, the system-in-broken-state issue is mitigated, and clean rollbacks are possible.

But all of these have one thing in common, which is that they address symptoms instead of causes. I say that our field is still nascent in this regard.

It feels like C runs everywhere with no effort, but if you look under the hood of the various libc distributions, you see all this #ifdef logic testing for host properties, in order to get the user-facing unified interface to "just work." The Windows API is by far the most stable and cross-(Windows-)platform interface at that level, and it also does lots of work behind the scenes to match user-side usage with the actual underlying platform implementation. Things in the digital computing world have always been this way, and abstraction from non-uniformity to some degree is good and necessary for productivity. The solutions we have today in the form of package managers, OS configuration, etc. are reactions to complexity growing at the level above lower-level unified interfaces. The pattern has repeated itself. Perhaps most notably is the web development ecosystem, which seems to have abstraction upon abstraction.

The current complexity of transitive dependencies is a natural phase of evolution of this field. Addressing causes will have us look at the complexity macroscopically and ask, what are the fundamental things we're all trying to do here, and how can we build unified interfaces to do those things? (Regarding unified interfaces, I'm talking about for very common things: display something to the user, get input from the user, play audio, dispatch work to a GPU, transact over the network, etc.) Currently we, especially bigger companies in the field, are incentivized to build walled gardens. Often times these walled gardens grow so large that the complexity problems from lower levels reemerge, and then a simplification and unification attempt is carried out in the walled garden, and this looks like progress, but it is often another stab at symptoms further up the stack of abstractions, not targeting causes in the opposite direction.

So when it comes to my project's dependencies, if I'm building a tool, a game, a packaged experience, then I do the following:

attempt to avoid dependencies
if unable, attempt to reimplement dependencies
if unable, attempt to absorb dependencies
if unable, resort to package-manager-like solutions

If I'm building a reusable library, then I try to depend only on unified low-level interfaces and solve problems people are actually having at scale.

Yes, alas, unified, or "standard," interfaces on *nix means and guarantees something different than on Windows, than on macOS. I am in favor of an alternative, but we'd need to develop it, and that's a subject for a different time.

And yeah, I guess this means I need to bring xml-jai into wayland-scanner-jai in my case. In actuality, my game engine doesn't need xml-jai itself, but I did use the example in this post as the basis for my reasoning.