Value of Trusted Sources

I think when it comes to the "build vs buy" decision for enterprise capabilities, the first questions that need to be asked are:

  1. Does this actually need to be done?
  2. Does the government need to do it, or can we simply purchase it as a commodity.

I'll use e-mail as an example: when I came on acttive duty in 2008, every base had their own local email infrastructure and domain. Around that same time, the Air Force rolled out a program called "e-mail for life" that gave us a us.af.mil email address, and we could log in to a portal and update our profile so that email address would always forward to our current base-specific email address. Fast forward a decade, and the Air Force had moved completely to Office 365 with a single Air Force tenant, meaning email accounts were centrally managed and there were no more base-specific email addresses, in addition to the fact that the local comm squadrons would never again run an Exchange server. E-mail became a commodity the Air Force just buys annually. We decided we still needed email (unfortunate!) but not that the government needed to operate the servers involved.

I'll use Docker Hub as the punching bag in this example that I think illustrates that (1) the government does need to have a centralized repository of container images and (2) the government does actually need to play a direct role in delivering that.

The Vexing Search For Angr

Within the symbolic execution/binary analysis space, there's an open-source tool called Angr1. Circa 2019, I was getting started using it for the first time. At that time, the installation process was fairly complex (there are a number of C dependencies that need to be built in a certain way to work properly with the overall Python engine), and so the general recommendation was to get started by using the angr/angr image on Dockerhub. Dockerhub and Github use a similar strategy for namespacing, so the Github project for Angr is also angr/angr. The angr/angr project on Dockerhub provides a readme that is identical to the readme in the angr/angr project on Github, giving a user quite a bit of confidence that the two projects are related.

Vexingly2, the Angr project on Github doesn't contain a Dockerfile in the top level of the source tree (I believe at one point it did, which was actually more confusing than the current state). So then, where does the Docker image for Angr come from? Well you see, good reader, that comes from an entirely different repository under the angr group on Github: https://github.com/angr/angr-dev. But you just need to know that, or trust that I know that. Nowhere on the Dockerhub page for Angr will you find that information. This was my first real foray into a topic that I would learn is frequenty called dependency confusion, but in this case it isn't so much a dependency as the top-level program I was trying to run, and there was no malicious intent on anyone's part, but it really highlights a critical issue with any of the major container repositories (and to a large extent, package repositories as well): there's just no good way to identify the provenance of a particular image. Just because the Angr Dockerhub image is built out of the angr-dev repository today, who's to say that tomorrow it won't be built out of some other repository? Reproducible builds are still not a solved problem in 2025, so "just rebuild and check the checksum" isn't even a technically viable option, even if we ignored the fact that such an approach would eliminate one of the largest benefits of using a container registry in the first place: having pre-built containers.

Iron Bank's Value Proposition

In my opinion, the core value proposition of the Iron Bank is that you understand the provenance of the container image3. The container images are re-built every night according to their Dockerfile. That Dockerfile is configuration-controlled in a fixed location. Any artifacts that are inserted into the container image via a COPY command in that Dockerfile come from a source with an associated hash in the hardening manifest. They might just be binary blobs, but they are consistent binary blobs. There's not one release out of every 100 that contains a different version of a blob. If that sounds far-fetched, it isn't: purveyors of malware have become incredibly complex. Some malicious JavaScript files are only served something like one out of every 10,000 or even 100,000 times in order to make the malware less detectable. It isn't at all difficult to conceive of an adversary that would serve up a perfectly intact dependency 99% of the time, and then serve up a malicious version of a file at a particular time when they detect the request comes from Iron Bank pipelines or from DoD IP space overall.

Historically (and today), the DoD runs it's own copies of update servers for things like Windows Update and Red Hat Enterprise Linux updates. Why? Because we need to know what we're installing on our machines. It's not that those servers can't possibly be made to serve up malicious or insecure content, it's that we need auditability and traceability and visibility into what was shipped, under what version identifier, and when. The DoD doesn't need to own the data rights to every piece of software we own, or build it all from source, or have custom versions of every piece of software we use, but we do need to control the distibution into and within our environments.

  1. https://angr.io/ ↩

  2. This is a joke on a joke: the intermediate representation (IR) used by Angr is called Vex (it was originally developed for the valgrind tool suite). The name of Angr is in part a play on words because it uses Vex IR. ↩

  3. This often is confused with having provenance for the items inside the container image. I am not extending my claim, at this point, to the actual contents of the image. I am simply claiming that when you find an image in Registry1, you are guaranteed to be able to find the corresponding Dockerfile in Repo1. This is a significantly weaker claim than "Iron Bank secures your sofware supply chain", but the entire point of this sub-chapter is to convince you that even the "weak provenance" guarantee is important and valuable. ↩