FreakTakes

This piece is a part of a FreakTakes series. The goal is to put together a series of administrative histories on specific DARPA projects just as I have done for many industrial R&D labs and other research orgs on FreakTakes. The goal — once I have covered ~20-30 projects — is to put together a larger ‘ARPA Playbook’ which helps individuals such as PMs in ARPA-like orgs navigate the growing catalog of pieces in a way that helps them find what they need to make the best decisions possible. In service of that, I am including in each post a bulleted list of ‘pattern language tags’ that encompass some categories of DARPA project strategies that describe the approaches contained in the piece — which will later be used to organize the ARPA Playbook document. These tags and the piece itself should all be considered in draft form until around the Spring of 2024. In the meantime, please feel free to reach out to me on Twitter or email (egillia3 | at | alumni | dot | stanford | dot | edu) to recommend additions/changes to the tags or the pieces. Also, if you have any ideas for projects from ARPA history — good, bad, or complicated — that would be interesting for me to dive into, please feel free to share them!

Pattern Language Tags:

• Building out an experimental test bed for a research field

• Pushing forward the technological base via speculative machine building contracts

• Putting out contractors for specific “integrators” in addition to prime contractors and basic researchers

• Using customer-relevant applications as a demand pull mechanism for a research field

• Setting up your testing environment in the same machines and locations you will deploy in

DARPA’s customer relationships and the individual agency of its PMs are possibly the two biggest differentiators of the DARPA model from bigger budget science funders like the NSF and NIH. Even in DARPA’s most basic research funding decisions, the question of “how can this be used by the military services?” finds its way into the discussion. In some generations of the organization, being “useful to the military services” is interpreted quite broadly, with justifications such as, “what’s good for the nation will be good for DARPA.” This approach was often taken by DARPA’s Information Processing Technology Office (IPTO) in the 1960s. Arthur Norberg and Judy O’Neill — who wrote an in-depth history of IPTO from its earliest days, including interviews with prominent individuals such as Robert Taylor — summarized the priorities of early IPTO directors as follows: “A contribution to the country which included a contribution to the military is what IPTO directors sought.”  With work such as the development of VLSI computer chips, that ethos has often proved to be an extremely correct heuristic.

However, in different generations of the organization — whether driven by wartime, fiscal austerity in DC, or the preferences of individual directors — being “useful to the military services” has been interpreted more narrowly. The first prominent example of this is DARPA under Director George Heilmeier. Another example of an era of DARPA where this goal was interpreted more narrowly was during Tony Tether’s years at DARPA as Director — starting in 2001. In this era, Tether went as far as to make many funding decisions contingent on a specific armed service putting in a budget wedge to continue developing the project past a certain stage. Neither approach is right or wrong. ARPA-like orgs often need to be able to adapt to changing situations like these.

Regardless, whether your customer is the US military, medical patients, or a certain subset of academics, all ARPA-like orgs serve some customer. This piece will dive into how DARPA’s Strategic Computing Initiative (SC) served its military customers in the 1980s — an era in Washington DC that demanded DARPA justify most of its spending as having clear, near-term value to the armed services. 

The three projects covered in this piece are all examples of DARPA’s attempts to find early military use cases of the exciting advances that had been coming out of their computing portfolio since the 1960s. Some pieces of the IPTO portfolio clearly followed Robert Kahn’s — who headed IPTO at DARPA — vision of “pushing” technology out into the world to find applications. Meanwhile, other areas of IPTO’s portfolio relied more strongly on using projects to stimulate demand “pull” for technological capabilities. 

Robert Cooper — the DARPA Director at the time — was the counterbalance to Kahn and his beliefs. The program that Kahn envisioned was, put plainly, not considered sellable in DC. To sell the program, Cooper helped center the pitch around three new military systems that would result from the program if its individual research programs panned out in a moderately successful way. The three early descriptions of the applications became the centerpiece of the report that pitched SC to Capitol Hill. These applications were meant not just to provide the non-technical politicians a clear sense of what they were paying for, but, also, Cooper wanted to use the applications to help steer IPTO’s research funding in what the SC management hoped would be a coherent and politically explainable direction. 

From an intellectual standpoint, Khan was considered the figurehead of DARPA’s 1980s Strategic Computing Initiative. The initiative was meant to expand on the prior decades of computing research breakthroughs as well as find useful military applications for the growing body of work. However, it would be DARPA Director Robert Cooper  — who had a Ph.D. background but favored industry-focused, goal-oriented engineering research more than Kahn — who would succeed in selling the program to Washington. A large part of the reason he proved to be a better fit for this sales task than Kahn was because he, more than Kahn, was willing and able to outline specific application technologies that SC’s technology portfolio could contribute to once all the work had been done. Khan felt that technological development was uncertain and that doing this was something of a fool’s errand. Cooper also seemed to understand that what they did at DARPA was an uncertain enterprise, but, also, that if DARPA wanted an expanded computing budget, then this was the kind of pitching that DARPA would need to do.

Roland and Shiman — who wrote a history of SC and interviewed many of its key players in the process — described Cooper’s thought process:

“Technology base” sounds fine but it does not mean much to those outside the computer community. Just what will that technology base allow you to do? What is its connection to the real world? How will it serve the defense mission? What new weapons and capabilities will it put in the hands of the commander in the field? These are the real metrics that allow you to measure the worth of such an investment. And any such goals have to be time specific. It is not enough to say that a robotic pilot’s assistant will be possible at some time in the future; there has to be a target date. Between now and then there must be benchmarks and milestones, metrics of progress demonstrating that the program is on course and on time.

Clinton Kelly, who originally joined DARPA as a PM in its Cybernetics Office before moving to the Defense Sciences Office when Cybernetics shut down, was asked by Robert Cooper to help define the applications efforts. Kelly, an engineering Ph.D. who had also run a company related to automating intelligence decision-making, had been funding a variety of somewhat odd mechanical machines while at DARPA — such as a robot that could do backflips. Kelly was known to be a great technology integrator and organizer, so he was very well-suited to the task Cooper had assigned him. The three applications that Kelly helped outline incorporated what Kelly and others saw to be the likely capabilities that would come from Kahn’s technology base and would intrigue the military services. The three major SC applications that Kelly helped jumpstart through his work on Cooper’s report were Pilot’s associate, Battle Management, and Autonomous Land Vehicle.

Progress towards these three applications would often be used to help steer and benchmark the program in the coming years. Many of the benchmarks included in the original SC report — such as real-time understanding of speech with vocabularies of 10,000 words understandable in an office environment or 500 words in a noisy environment — were set with the needs of these applications specifically in mind. Stated goals like helping a fleet commander plan for multiple targets or providing the military the ability to send an unmanned reconnaissance vehicle across a minefield helped ground the vision of the program to its funders and customers who were less likely to see the importance of the stale computer benchmarks — such as those used in the computer architecture section of the SC write-up.

While Khan eventually relented, he still saw these applications as Cooper imposing a NASA-style of operations on DARPA. NASA focused on working towards challenging point solutions rather than broad, flexible technological development — as IPTO historically had. As the years went on and work progressed, work on these projects proved that Kahn and Cooper both had good points. Regardless, the applications proved successful in helping DARPA win over an expanded computing budget for SC. In 1983, work on the program commenced.

The Pilot’s Associate (PA) application was meant to assist aircraft commanders in all cockpit decision-making and planning that could reasonably be augmented by sensors and then-modern computing. The PA planned to incorporate the new hardware, AI/software, and natural language understanding capabilities expected to emerge from DARPA’s funding of the computer technology base. DARPA hoped that the PA would eventually help the aircraft commander perform tasks like preparing and revising mission plans. They hoped for the system to be small enough to fit into a cockpit and outfitted with a natural language understanding system so a pilot could deliver some verbal commands while they used their hands to operate the aircraft. The PA would be a “virtual crew member.” There were even comparisons, at the time, to the system functioning as a pilot’s own R2-D2 unit — although others in the military pushed back on this specific analogy.

Once SC was announced, DARPA’s first step in working on the PA application project was contracting Perceptronics — a research company that DARPA was currently using as a contractor for its SIMNET computerized combat simulation training project — and other contractors to carry out exploratory studies in the area. These studies were meant to more concretely outline how the system would work, as the description of PA in the original SC plan was quite vague. These studies established that, most likely, five separate “expert systems” could work in concert to carry out the functions of the final PA application. 

In the era of strategic computing, “expert system” was a common term to reference software programs — often built by combining the efforts of programmers and specific domain experts — which used codified sets of rules to help carry out tasks in a faster and more accurate fashion. In their simplest forms, expert systems might just be made up of meticulously constructed “if, then” rules. Expert systems were, at the time, considered a branch of AI research — albeit a less exciting one than more generalizable AI systems to AI research purists. However, in an era when generalizable AI was often not living up to any level of commercial promise, expert systems programs, which nowadays would simply be considered “software programs,” were proving interesting to the military and large companies alike. These programs showed real promise in bringing the fruits of the AI research community to a wide variety of sectors and customers. Unlike more pure AI systems at the time, expert systems work had actually even proven profitable in the private market.

So, it is not surprising that PA’s project planning was going to rely on the use of the more application-ready expert systems paradigm. In the case of PA, the preliminary studies determined that the five specific separate expert systems that would make up the final software of the machine would encompass the following areas: mission planning, systems status, situation assessment, tactics, and front-end communication with the pilot. As the PA project went on, most of the direct planning for the PA system was done by scientists and engineers at the Wright Aeronautical Laboratory at the Wright Air Force base in Dayton, Ohio.  The Air Force’s excitement to largely take the reins of the project from its early days was likely due to the fact, in the years leading up to this point, the Air Force had begun developing its own labs to explore the application of AI to Air Force matters. Major Carl Lizza, who had a masters degree in computer science, was the program manager on Wright’s end for much of the project. On DARPA’s end, John Retelle was the program’s PM until 1987, when another DARPA PM took over. With high level buy-in and excitement from the Air Force, minimal direction was needed from DARPA on the direct development and implementation of the PA system. The customer buy-in from the Air Force was a quite positive sign for DARPA. 

Following the study stage of the project, the implementation of the PA program began in 1986. McDonnell Aircraft Company was the prime contractor for the air-to-ground system. Three total companies made up this portion of the contract. Lockheed was the prime for the air-to-air system. Eight companies and university research teams were on that contract. McDonnell was awarded $8.8 million (~$23 million today) from DARPA and Lockheed $13.2 million (~$38 million today). Each company was also willing to cover 50 percent of the costs of the development work because they felt working on the contract would give them an advantage in bidding on the big-budget contract for the Advanced Tactical Fighter which would be awarded in the coming years. These performers would look to take advantage of the equipment and methods that had been coming out of the DARPA portfolio as well as industry in recent years — employing LISP, Symbolics, and Sun machines as well as many of the new expert systems development methods which DARPA often funded.

The project would also make heavy use of expert boards in planning its work schedule as well as work with the customers in developing successive versions of prototypes. As is common in military R&D applications projects, the team leveraged expert advisory boards from academia and industry on an ongoing basis. Help from these sorts of boards seems less common when making small decisions in projects further away from implementation — such as those made in the Defense Sciences Office in comparison to the systems technology offices. Also, as this was absolutely a product for a paying customer, the performer teams heavily involved the users of the eventual product throughout the design process. In fact, much of the development work by the industry contractors looked quite similar to modern software development to modern readers. The project relied heavily on the rapid prototyping approach to managing its workflow and benchmarking its progress. Several demos were set to happen in the first three years of the project that would prove the system’s function and value. After several successful demos and the proof of concept were completed, the work could then progress to doing the development work to ensure that the system worked as needed in live combat situations. From a high level, that was how the project was meant to progress.

Non-PA, but PA-relevant research and engineering work that was in earlier technology readiness stages was overseen by other PMs in the SC portfolio. This bit of bookkeeping is of practical importance to how the projects were managed. The PA-relevant work on the broader technology base — that was still related to the application and might, at some point, make it into the PA system — generally fell to PMs in IPTO. These PMs working on other pieces of the broader portfolio did not “answer to” the PMs of the PA, Battle Management, or ALV applications, in spite of the importance of those three projects to SC. The work in these other PMs’ portfolios, of course, was meant to prove useful to projects like the PA. However, the PA team did not have any direct power to influence other PMs’ priorities to be more in line with PA’s direct needs. The PMs in the broader SC portfolio were meant to help develop better component technology to help projects like PA in the future, but they were generally able to decide on the best possible deployment of their own portfolios. 

As the PA project was in its early planning stages, it seemed clear that a sufficient level of expert systems knowledge and experience was generally available to make an application like PA feasible at the time. But at least two notable technical risks existed that the PA team was relying on developments from the broader SC portfolio to overcome. The first risk was the need for a computer to be developed that was powerful enough — and small enough — to make the system work in real time. It was not going to be clear exactly how much more powerful a computer would need to be to make the PA work until more expert systems development was done by contractors on the project. But it seemed clear that a computer about an order of magnitude more powerful might be necessary before the system could be considered deployable in a cockpit. The second area of notable technical risk was the need to develop a natural language understanding system that could process a basic vocabulary of at least 50 or so words in noisy, cockpit-like conditions. This area of technical risk was likely less important than the issue of computing power, but it was still quite important if the system was going to work the way those in charge of the program wanted it to. 

In spite of there being notable areas of component technology that required improvement, Director Cooper felt it was important that the SC applications programs begin roughly at the same time as its work developing the broader technology base. Politically, this increased SC spending package was meant to be about applications. Waiting years to start working on the applications projects was not really an option. So, work on the three applications projects began, more or less, right after SC was announced. The SC team, however, did go out of its way to ensure that the more basic research projects got off and running sooner rather than later. Meanwhile, the applications projects often had study stages and preliminary planning periods that may have taken a year or two. I do not get the sense that DARPA made similar attempts to speed this process along the way they did to ensure that computer architecture researchers, for example, got the machines they needed for their research sooner rather than later. 

In March 1988, the first major operational milestone for the PA project was hit when the Lockheed Aeronautical Systems Company successfully demonstrated the system it had developed in a non-real-time cockpit simulation. In this demo, the system successfully performed flight tasks like identifying and dealing with a stuck fuel valve, creating a new mission plan as the situation changed, advising the pilot on evasive action, and deploying flares. The machine was able to complete the simulation’s 30-minutes worth of real-time computations in 180-minutes — meaning, in this early stage, the system was about one-sixth as fast as it needed to be. The slow performance was not considered behind schedule by any means. Firstly, much of the software was not even run on the most powerful machines possible at this stage. Additionally, in the early internal writings on the project, it was believed that by around 1990, some parallel computing technology from the DARPA architectures portfolio would be advanced enough to transition the expert systems to run on those machines. 

This bit of planning is quite interesting. Those planning SC’s applications very clearly understood the pace at which computing hardware and architecture was advancing in these years and incorporated a level of anticipation of that progress into their planning. For context, in this period the BBN Butterfly computer — which was a coarse-grained parallel processor and not so powerful yet — was one of the few operational parallel architectures. Massively parallel architectures were still in their very early stages at the time SC was getting underway. By 1993, those at the helm of SC felt that small computers would be able to run the types of systems required to operate the PA in real-time and then some. 

It was the job of the other PMs in the SC portfolio to seed investments and help orchestrate progress via the performer community to make this projected computing progress a reality. I will briefly give the reader an idea of what some of this work looked like. In the period between the mid-1970s and mid-1980s, technological development was occurring elsewhere in the hardware section of the SC portfolio and in industry that would end up becoming far more famous than the PA project itself — but that would, of course, enable many projects like the PA to become a reality in the coming decades. Some of this progress was funded by industry on its own, some by government and DARPA dollars, and often both. While NASA and the Air Force were buying chips from the young Fairchild Semiconductors, DARPA money had been finding its way to Fairchild through projects like the ILLIAC IV (covered in the FreakTakes Thinking Machines piece). Texas Instruments was hard at work on many of its own R&D projects, but many were also pursued in concert with actors like DARPA. For example, in 1983 DARPA approved $6 million to TI to develop compact LISP machine hardware — implementing a LISP processor on a single chip — as long as TI would fund $6 million in simultaneous software research for the device. Projects like the PA were at the top of mind when funding this TI project. Additionally, DARPA had funded much of the early Warp computer research at CMU which the university, later in the 1980s, worked with Intel to help miniaturize in developing the iWarp — whose first prototype was operational around 1990. Since this area of the history of computing progress is extensive and, to some extent, understood by many reading this piece, I will say no more on the progress in computing power in this period. Suffice it to say that the actual progress of the field did turn out to be in line with the lofty expectations of those planning SC as they outlined projects like PA in the early 1980s.

As this computing work was progressing — which would help the PA project team overcome its first major technical risk — SC’s portfolio of work on the second major technical risk, speech recognition systems, was also moving along quite well. While many consider speech recognition systems a relatively modern phenomenon, research in these systems was progressing quite steadily for several different performers in the SC ecosystem in the 1980s. By 1987, CMU had developed a speech system that could recognize 1,000 words with an accuracy of over 90% at real-time speed. BBN had also developed a similar system, and IRUS, which possessed a vocabulary of approximately 4,500 words in 1986 and was generally considered ready for testing in applications like the Navy-relevant Battle Management (the substantially larger vocabulary is because Battle Management’s speech recognition system was meant to be used in less noisy environments). Unsurprisingly, much of the progress occurring in the area was aided by improvements happening in computer hardware — in this period largely due to successes in VLSI development (covered more in the FreakTakes MOSIS piece). 

The progress in component technology was happening on the side of the work being done by the teams working directly on the PA project. Extremely pleased with the 1988 demo which showcased the general feasibility and usefulness of the system in practical situations, and with much of the work on relevant component technology in the SC portfolio going according to plan,  the Air Force elected to have the Lockheed/Air Force development team begin adjusting their development efforts. They hoped to have a system that could be put into aircraft to help them fly missions within three years. After one more successful November 1989 demo which showcased some additional functionality — such as reasoning using more imperfect sensor data, re-implementing programs in faster programming languages, and updating the machines used to run certain expert systems to faster versions —  the Air Force shifted the project into its further development phases. The coming stages were meant to get the machine ready to perform in more battle-like timelines and uncertainty conditions.

Unfortunately, the project would not ever make it across the finish line. The project’s fate seems to have become clear around 1991. Much of the component technology was strong, and some of these component improvements and learnings would find their way into similar future systems. For example, the Lockheed team produced extremely useful new insights when it came to the systems engineering of multiple coordinated expert systems combined in a single application. When the expert systems were combined, there were issues with each expert system following its own objective function and sometimes giving the pilot conflicting advice. The Lockheed team came up with a concept called a plan and goal graph to help the system better infer the pilot’s true wishes and deliver advice that was more coherent and in line with the pilot’s wishes. However, certain vital pieces of functionality were just not coming together the way the team had hoped as they attempted to make the system work in real-world situations. Certain pieces of the project that appeared conceptually feasible in the early stages were proving to be hairier than had been anticipated. The technology base was simply not capable of doing certain key tasks like recognizing certain maneuvers and accurately predicting the geometry that would result. But, in many ways, the program had served its intended purpose in DARPA’s portfolio.

Wright’s chief engineer on the project, Captain Sheila Banks, and its program manager, Major Carl Lizza, seemed to clearly understand that while the PA might not make it into a standard Air Force cockpit, it should not be considered a failure from the point of view of Robert Cooper’s DARPA. The two concluded a 1991 paper they had written on the state of the PA program by writing the following:

In accordance with the program goal to provide a pull on the technology base, the program often needed nonexistent technology to implement the design. Uncertainty-reasoning approaches, pilot modeling techniques, and development tools are areas that lacked sufficient research to implement the Pilot’s Associate vision. Verification and validation of knowledge-based systems constitute another area that is lacking techniques and is quickly becoming a program issue. The actual implementation of a system of this complexity uncovers many gaps in technology still to be addressed by the research community.

The project helped facilitate a level of practical learning in the SC portfolio and provided a level of clarity to many basic researchers trying to understand what contexts, roughly, they were attempting to produce technology for. But the project did not result in any immediate revolution in how Air Force pilots operated in their cockpits.

I will give a slightly more high level accounting of some of Battle Management’s operational details. In many ways, the operational structure of the project was similar to PA’s, but the project resulted in more of a clear-cut win in the short run. Where certain project details are either instructive or interesting, I will be more thorough. But I would rather leave more time for ALV as its messy history and complex legacy are extremely instructive. (For more operational details relating to specific hardware, expert systems, and contractors see this document from Battle Management PM John Flynn.)

The Battle Management application project was launched with the goal of deploying the power of expert systems to help Naval decision-makers make better inferences about enemy forces, keep better tabs on their own forces, generate strike options, come up with operations plans, as well as a variety of other tasks — which any modern reader surely knows computers could assist Naval leadership in carrying out. The vision of the system was quite similar to the PA, but in the context of the Navy rather than the Air Force. As was the case with the PA system, the Battle Management system’s wish list of capabilities would likely require improvements in overall computing power from the rest of the technology base. Initial estimates for the computing power required for the system were around 10 billion operations per second — approximately a one or two order of magnitude increase in peak performance in comparison to what was available at the start of the program. 

While Battle Management was initially conceptualized as one program in SC’s founding document, it quickly became three separate programs: Fleet Command Center Battle Management (FCCBM), Carrier Battle Group Battle Management (CBGBM), and AirLand Battle Management. In this section, I’ll focus on the two projects that focused on the Navy as a customer —  FCCBM and CBGBM — for brevity’s sake. Both systems, similar to PA, were meant to use a GUI, natural language system, and expert systems to help assist Naval commanders in coming up with and executing operational strategies on the ship level (CBGBM) as well as on the theater level (FCCBM). 

CBGBM — the system that would be installed aboard a ship — was considered a continuation of a program by Allen Newell and others at CMU that had been ongoing since the 1970s. With the announcement of SC, CMU had proposed continuing the work area under the well-funded SC umbrella. DARPA believed this was a good idea and helped coordinate Naval buy-in to the project. The Naval Ocean Systems Center in San Diego officially bought-in to the CBGBM project early on. Similarly, for FCCBM — the fleet level battle manager — DARPA was able to secure buy-in from the HQ of the Pacific Fleet at Pearl Harbor. As the Air Force did with PA, the Navy agreed to provide a level of financial support to these projects at this stage. With that support DARPA had early buy-in from its customer into the project. DARPA’s working relationship with the Navy on the project would prove to be an exceptionally functional one — enabling a far more effective prototyping approach than was seen in the PA project.

The FCCBM’s (the system meant to help make plans on the fleet level) rapid prototyping strategy, implemented under PM Al Brandenstein, was executed in coordination with the team at Pearl Harbor. Roland and Shiman describe the project’s prototyping strategy as follows:

A test bed was established at CINCPACFLT [Pearl Harbor] in a room near the operational headquarters. The same data that came into headquarters were fed into the test bed, which was carefully isolated so that it could not accidentally affect real-world activities. Experienced Navy officers and technicians tested it under operational conditions and provided feedback to the developers. The resulting upgrades would improve the knowledge base and interface of the system.

This extremely tight customer relationship in the development process ensured several things in the early stages of the project: faster feedback cycles, increased buy-in from the customer, and extreme clarity on the requirements of the system. 

As the implementation stages of the projects got underway, the FCCBM team installed its computer hardware at Pearl Harbor in 1985. By March 1986, the test bed environment had been completed. With that infrastructure and scaffolding in place, the expert systems contractors could ensure their systems were tested on data with the highest level of fidelity to real-world data possible — because it was real world data. In June 1986, the Force Requirements Expert System — the first of the five expert systems for the project — which had been in development since 1984, was installed and began testing in the test bed. By the end of August the system was successfully demoed to DARPA and Naval leadership. This demo, as well as much of the work on the project, was heavily based on the development of BBN and the Naval Ocean Systems Center’s IRUS natural language generator — a project from DARPA’s technology base. By this time in 1986, IRUS had already achieved a vocabulary of almost 5,000 words and — being an experimental system developed with the knowledge that the Navy would be its initial user — even had a vocabulary and language generation style that could comprehend and respond to queries in language familiar to Naval operations staff.

By 1987, the FCCBM had already proved its worth to Navy staff. The system was capable of performing tasks that used to take fifteen hours in as little as 90 minutes. Research teams frequently seem to generate results like this on paper that either don’t turn out to be true in practice, or, alternatively, are never installed due to fear from customers that the paper results won’t hold up in deployment. FCCBM was able to avoid both of these issues. Since the system was already installed at Pearl Harbor and working on its actual data streams, the results were far from mere paper results. Additionally, the pain and annoyance of initial installation were much less of a factor in the decision than they might usually be because the system was already installed at Pearl Harbor. The first expert system quickly went into operation helping monitor ships in the Pacific while some of the other expert systems — which were further behind in readiness level — were tested until they were deployment ready.

Following a similar approach, CBGBM (the Battle Management system designed to be put on individual ships) was installed for testing on a ship in early 1986. Within a few months, the use of the machine in fleet exercises convinced the Navy that a modified version of the system would be as useful as they’d hoped. Improvements were made to the system before going out on a six-month tour later that year. The technology, at that point, was very much still in the prototyping stage because it could not operate in real time. But as a development project, CBGBM was proving its utility to the Navy. 

While these two projects generally progressed quite smoothly, there was one notable friction on the operations side of the project: classified information. Some individuals who worked with the contractors did not have security clearances because they couldn’t get them. Others just didn’t want them. The university research contractors, in particular, often didn’t want clearances. CMU, as an example, was one of the systems integrators for the CBGBM project and had a very good working relationship with the Navy, but largely did not have security clearances. So, this meant that one of the main systems integrators could not — for the most part — actually look at any of the golden, real-world data in the database at Pearl Harbor. Instead, their workflow had to look something like the following: discuss with the Navy what sort of information would be reasonable to use, do their R&D using fabricated data that they hoped would look like the real thing, and twice a year hand over their updated system to the Navy team and wait to see how it worked. While that was a hassle, it seemed to be a small price to pay for having the best minds on the project — the CMU team's work from campus seemed to port over to the real systems on the ships well enough.  So, the parties made it work. The Navy would even send staffers to CMU for long stays to ensure that as much knowledge exchange as possible was done. 

The work on the Battle Management applications, similar to PA, was enabled by the development work going on in the greater SC portfolio. This is particularly true of the following areas of the portfolio: computer hardware, natural language understanding, and speech recognition. Beyond the improvements in computing hardware going on in the broader SC portfolio — which was explored in the PA section and had similar implications to the Battle Management projects — much of the speech recognition and natural language understanding (NLU) research in the SC portfolio was done explicitly with Battle Management in mind. 

I will provide some clarity as to what some of this work looked like — which I will also cover more in a coming piece on DARPA workhorse computing contractor BBN — since several researchers and ex-DARPA PMs have personally expressed surprise that DARPA’s speech recognition and NLU portfolio has operational successes going back four decades. To many, this area of engineering feels quite young since many of the best methods in use today have been enabled by quite recent technology. However, as far back as the late 1970s, contractors in the speech recognition and NLU areas of the DARPA portfolio were building quite impressive applications with what — by today’s standards — would be considered somewhat quaint computing power and natural language methods.

While DARPA had largely reduced the contracts it let in speech recognition in the mid-1970s under Director Heilmeier, some speech recognition work continued in this era that would prove key to programs like BBN’s IRUS system that was deployed in the Battle Management system in the late 1980s. Several key pieces of work going on in the interim were: Baum et al.’s continued work developing hidden Markov models (HMM) at the Institute for Defense Analysis, Jim Baker’s HMM-enabled Dragon speech recognition system built at CMU, and BBN’s applied work on various speech recognition and compression projects for paying customers. On the natural language understanding front, BBNers and researchers related to BBN — such as the Harvard and MIT grad students of professors who worked part-time at BBN — made new discoveries and first-of-their-kind implementations in augmented transition networks and structured inherited networks in this period. Both of these technologies — which seem to have been replaced by more advanced methods by the late 1990s — proved extremely useful in BBN’s IRUS system which proved the feasibility of certain natural language tasks in the context of applications like Battle Management.

Even while DARPA was not giving contracts for speech recognition, believers in the technology at frequent DARPA contractors like MIT, CMU, and BBN that believed in the technology continued to push the field forward in the meantime. By 1986, the IRUS system already had a speech recognition vocabulary of almost 5,000 words and an accuracy rate of over 90% which was enabled by some of BBN’s work on alternative models. The relatively massive size of IRUS’s vocabulary — in comparison to the vocabulary of under 100 words in the PA’s speech recognition system — was largely due to the noisy conditions of Air Force cockpits. This constraint limited the scope of the PA’s vocabulary to simple vocabs and something like fifty commands along the lines of “Fire!.” In contrast, DARPA knew from the beginning that Battle Management would be a more conducive environment for speech recognition systems. So, from the beginning, SC management had ambitions of building a system capable of utilizing a vocabulary in the 1,000 to 10,000-word range in the next five years or so. 

As the battle management application projects were ongoing, there were eight different contractors across nine projects in the broader DARPA ecosystem pursuing various research projects in the area of speech recognition. To help wrangle all of these findings into technology that could be folded into future generations of battle management systems, CMU was given a contract as an “integrator.”

In the SC portfolio, integrators were sometimes given separate contracts and tasked with the primary responsibility of finding ways to integrate the cutting-edge findings from the performers’ work in the broader technology base — who were supposed to be keeping applications in mind anyway — into the final applications. An integrator contract was given in the PA project as well. In that project, five groups of performers who were working on eight different DARPA programs fed relevant results to TI, who was tasked with integrating the new knowledge into subsequent generations of the PA. In theory, with prime contractors like Lockheed and university researchers doing their jobs, this role should not be necessary. But in practice, the integrators could often make a big difference in the level of ambition of the integration work.  

In the end, the speech recognition work went so well that SC management never needed to change the initial, lofty goal of achieving a system with a vocabulary of roughly 10,000 words. And that's saying something; oftentimes, lofty goals are set at the beginning of projects like this with the understanding that, after a period, the PM, relevant performers, and DARPA management will touch base and agree on more reasonable goals. However, revising the goals was not necessary in this case.

The other area of battle management’s language work — NLU — also had integrators that it leaned on. The broader NLU applications work was made up of two integrators and five other teams of performers. The integrators were both teams. The first team, in charge of integrating text understanding, was made up of NYU researchers and the Systems Development Corporation — a frequent DARPA contractor since its systems engineering work on the SAGE air defense system. The second team, in charge of integrating broader work in natural language query processing, was made up of BBN and the Information Sciences Institute (who is the star of the FreakTakes piece on MOSIS). This natural language understanding work also met the initial, non-quantitative goals that SC’s planners set out: the final users were able to provide the system new information and get understandable feedback in return explaining the system's “thought process.”

The Naval Battle Management projects clearly accomplished their short term goals. The primary goal of the program, on paper, had been to help the Navy with supporting and planning in areas where time constraints are extremely critical. And the systems did just that. For some important tasks done at locations like Pearl Harbor, they reduced the time it took to do certain planning tasks from 15 hours to 90 minutes — along with the increase in computational reliability that comes with using a computer. Additionally, these Battle Management systems were in place and working within five years. This not only meant that the projects were a success as quickly as possible, but also that the projects helped DARPA Directors in doing things like justifying budgets and convincing increasingly austere Washington decision-makers in the late 1980s that their work was saving money for the taxpayers even if it often looked somewhat speculative. In many ways, Battle Management was the PR success that the ALV was hoped to be.

The third application placed at the heart of SC’s founding document was the Autonomous Land Vehicle (ALV). As the SC program wore on, ALV would also come to be one of SC’s biggest disappointments — at least in the short term. The ALV was envisioned as an autonomous recon vehicle that could navigate battlefields — obstacles and all — to conduct reconnaissance missions and report its findings back to soldiers who did not have to leave safer positions to obtain the information. 

From a mechanical perspective, DARPA hoped for a machine that could travel over thirty miles in a single trip with top speeds of almost forty miles per hour. Of course, machinery existed that could mechanically hit these benchmarks. However, DARPA wanted a machine that could do all of this while being autonomously operated. To do this, many computational abilities would need to be developed for the system. The computational tasks required for an ALV to successfully work in battle included: route planning, navigating via visual analysis of physical landmarks, rerouting and avoiding surprising obstacles that were not in the route plan, ingesting useful information on the target, and transmitting that information to a military team. At the time, it was believed that performing these tasks would require an expert system that could run as many as three times more rules per second than current expert systems. Additionally, DARPA believed that the computer required to run the machine would not just need to be small — something like 4 ft x 4ft x 4ft — but would also need to be powerful enough to perform ten to one hundred billion von Neumann instructions per second. This was a tall task, as contemporary von Neumann machines were generally limited to around thirty or forty million instructions per second. This need for a three-order-of-magnitude improvement in computing meant that the ALV project achieving its goals relied even more strongly on improvements in computing power than the PA and Battle Management. 

The project sought to tackle problems in computer vision that modern autonomous vehicle research groups are still working through today. However, Clinton Kelly was confident that enough leg work had been done on the research front for DARPA to begin creating a program as a technology pull to bring DARPA’s vision work to practicality. This confidence derived in no small part from DARPA’s Image Understanding program which had kicked off in the mid-1970s. The Image Understanding (IU) program — sponsored by IPTO and then-Director J.C.R. Licklider — was formally spun up as a cohesive program on the heels of progress in DARPA-funded projects like DENDRAL in the 1960s.

DENDRAL was not computer vision per se — in that it could not actually ingest a photo — but the system could utilize a text representation of some molecular structures and used expert systems to perform analyses on them. Projects like this demonstrated that, once information taken in visually can be coded symbolically, systems were in progress that could make intelligent use of the information. Somewhat naturally, “signal-to-symbol mapping” became a key point of emphasis in the IU program — which would allow visual information from cameras or sensors to be encoded in such a way that systems could act intelligently on the information. Somewhat surprisingly, there were even some practical successes from the early-IU years. One example was DARPA and the Defense Mapping Agency’s (DMA) joint funding of a system — housed at the Stanford Research Institute — to automate certain cartographic feature analysis and assist the DMA in certain mapping and charting functions. However, most of the work done by the performers in IU in this period was quite far from applications largely because the field was young and there were not many clear-cut applications for the work just yet. However, it should be noted that as IU was getting underway in 1975, Licklider and others hoped for the work to progress in such a way that applications were within reach by 1980. 

Even with the ALV project kicking off in 1985, ten years after the start of IU, many veteran vision researchers felt this push for an ALV-type application might be a bit premature. Kelly and Ronald Ohlander — the DARPA PM who oversaw vision-related projects in this era — polled some veteran IU researchers to ask what they felt would be reasonable benchmarks and timelines for the project. Kelly told Roland and Shiman in an interview that he felt the answers were far too conservative and that he “took those numbers…and sort of added twenty percent to them.” When the researchers pushed back, they settled on a  compromise. Ohlander noted to Roland and Shiman that he thought the program to be “a real stretch” — far more than the other applications.

This general sentiment could also be seen with ALV’s lack of early buy-in from its intended customer: the Army. The Army was the intended customer for the ALV, but Army leadership saw no acute need for a robotic vehicle at the time and, thus, did not commit to helping fund its early development. The project, however, was seen by Kelly — and likely Cooper — as a fantastic pull mechanism for DARPA’s image understanding work. DARPA — as it did with other projects in this period such as the ILLIAC IV and Connection Machine — was hoping that providing funding and a contract for a machine with clear goals that required improvements to its component technology could help bring a young area of research to a new level. As Richard Van Atta wrote in one of his technical histories of DARPA:

Before the ALV project, the vision program had centered on defining the terms and building a lexicon to discuss the mechanics of vision and image understanding. The ALV was one of the first real tests of IU techniques in practice.

Even if the technology proved to be out of reach of his generation of researchers, Ohlander acknowledged that the researchers’ work deploying their ideas on a real-world test bed would expose the shortcomings of their approaches and help move the field along faster than it would otherwise.

With the kick-off of SC, funding for all vision-related work in the SC portfolio expanded rapidly. IU’s budget abruptly increased from its usual $2-$3 million a year to $8-$9 million a year. The ALV application program itself was, to some extent, co-managed by Clinton Kelly as well as its official PM William Isler.  They planned for one prime contractor to be in charge of engineering the vehicle and gradually incorporating the new component technology from the technology base into the machine on an ongoing basis. The program managers felt that one of the major aerospace companies from the regular defense contracting pool would be a good fit for the project. While these groups did not have experience in computer vision — almost nobody did — they did have experience overseeing projects on this scale of complexity and had all proven quite adept at large-scale systems integration. Some of these companies, at least, had experience with machines like remotely operated vehicles and basic forms of AI systems. As they were submitting proposals to DARPA’s January 1984 qualified sources sought, these aerospace companies understood that the project would be carried forward with whichever one of them won in partnership with at least one university department. The partnership was intended to help promote the integration of the expanding capabilities of the DARPA technology base into prototypes of the ALV as quickly as possible. 

The ultimate winner was Martin Marietta — which would later become the “Martin” in Lockheed Martin.1  The contract was to last 42 months and was for $10.6 million — with an optional extension of $6 million for an additional 24 months. DARPA chose the University of Maryland (UMD) as the university research partner to assist the company in folding technology improvements from the technology base into the machine in the early stages of the project. UMD was a well-known entity to the AI portion of DARPA and its PMs as it was one of the primary university research centers contributing to IPTO’s Image Understanding work. So, the university’s primary task was to get the practical vision system that they had been working on folded into the machine so the Martin team could begin working towards the goals of its demos as early as possible. 

As implementation work got underway, the ALV quickly adopted an accelerated schedule. Towards the end of 1984, DARPA moved up the date of the machine’s first demonstration to May of 1985 — six months before the previous initial demonstration. The vehicle would have to identify and follow a basic road in a demo with no obstacles of note. As Martin Marietta rushed to quickly assemble its first vehicle, it largely relied on off-the-shelf components that were tried and true. The mechanical parts such as the chassis and engine were standard, commercially available parts. The software and hardware for the machine’s computing were not exactly cutting edge. One of the two types of computers chosen — a VICOM image processor — may have been chosen because one of the University of Maryland Researchers had extensive experience working with the machine and stood the best chance of quickly getting the vision program up and running with familiar equipment. The second type of computer — Intel’s single-board computers, which were more or less personal computers — was used to control the vehicle itself. The camera that fed into the vision system was a color video camera mounted at the top of the vehicle and paired with several charge-coupled devices to transmit the visual information to the computer in a usable format.

The first demo was quite basic. The goal was for the machine to follow the center line of a road at Martin Marietta’s testing facility for about 1,000 meters. To give the reader an idea of how the vision system encoded real-world image data in a usable way, please see the following figure provided in a 1985 Martin Marietta report explaining what a digital scene model would look like for a given road image.

With basic image processing, some engineering cleverness, and standard geometry for calculating things like distance traveled and curve angles for steering, the machine seemed like it might be able to navigate a basic course quite well. The machine was powered by two major categories of visual decision-making software. The first category was the software dedicated to understanding exactly what the ALV’s camera was seeing. This software processed and enhanced the images so it could then use some of UMD’s edge-tracking algorithms to find the various boundaries in the image. Using these boundaries, the software calculated where the center line seemed to be. Another special algorithm would attempt to plot a 3D grid.

This work was passed to the second software system which was meant to perform some basic reasoning tasks using the current 3D grid model as well as the 3D grids from the timestamps immediately preceding the current one. At this stage of the process, the reasoning system largely attempted to assess the reliability of the current image, given what it had seen in the previous images. If the image seemed unreliable, the system attempted to make up a more believable path that fit with the images seen previously. If this switch happened, in that moment the reasoning system was seen as driving the vehicle rather than the vision system — since the vehicle was operating off of a pre-stored model rather than the current, low-confidence model built by the vision system. This backup plan was necessary for achieving the goals set by the demo because, given the slow processing speeds, the vehicle’s speed would often outrun its visual processing. So, this was necessary to keep the vehicle moving without stopping too frequently or going too slowly. 

The first demo — a standard road-following task — went off rather successfully. The vehicle traveled the approximately 1,000 meter course in about 1,000 seconds and did so traveling about 85% of the way under vision control rather than reasoning control. The machine did this while generating over 200 scene models and trajectories to help inform the vehicle along the way. On the heels of this success, the Martin and UMD team were right back to work because the next demonstration was in six months. And this demonstration was a test that was considered less of a layup than the first one had been. The course for the demonstration — which the ALV had to navigate at 10 km/h — required that the ALV navigate a sharp curve at least 3 km/h, progress along several straight stretches of road, and, lastly, stop at a T intersection to turn around and repeat the course. It had to do all of this without the reasoning system overriding the vision system.

To do this, the team would have to add more sensor information than just the single video camera to take in all the required information. Additionally, they had to re-work the vision software to somehow make use of all of this new information. In attempting to navigate this much harder problem, Martin relied on a DARPA-funded laser range finder that had been developed by the Environmental Research Institute of Michigan. The device could be attached to the front of the vehicle and was used to scan — up and down as well as side to side — the terrain in front of the vehicle to identify obstacles and other important features. Those working on the software attempted to find a way to combine this knowledge into a single 3D model of the scene. This knowledge was combined with the reasoning system which now contained a digital map of the track itself which the machine could use as a reference — the Engineering Topographical Laboratory had developed the map. The vehicle, impressively, passed this second test as well in November 1985. At this point, the ALV application seemed to be making extremely satisfactory progress. The technology the ALV used in these early demos — including the laser range finders — was not exactly cutting-edge, but the machine was hitting its practical benchmarks and hopefully could begin subbing in more cutting-edge component technology as the demonstration benchmarks continually raised the bar for the ALV. 

While the work on these early demos was happening at Martin Marietta, Martin and other ALV-related component contractors were also holding meetings at regular intervals — often quarterly or as frequently as monthly — discussing project needs, developments in the technology base, and how to delegate tasks. These conversations with research teams working on broader vision work were particularly important for work on the demos scheduled to happen in later-1986 and beyond — when it seemed like Martin would begin having acute needs to integrate more work from the technology base into the ALV to meet the goals of the demos. Much of what went on at these meetings was not so different from what happens at many DARPA PI meetings — with the standard presentations of progress and teams getting on the same page.

One logistical point of particular note was that, as some of the higher level planning meetings played out, the project structure was almost very heavily shaped based on the arbitrary preferences of the component contractors.  The contractors who found themselves working on the reasoning system vs. the vision system had very different interests and incentives. Most of the reasoning contractors were commercial firms and had the incentive to fight for a broader scope of work and more tasks getting assigned to the reasoning system because that meant better financial returns to the company. Most of the vision contractors, such as the University of Maryland, were academics and largely content to let that happen — or, rather, not as committed to fighting for a broader scope of work as the companies — if it meant they got to focus primarily on specific vision sub-tasks that were more in line with the size and scope of projects academics often undertook. For example, the private companies felt that building up much of the systems mapping capabilities, knowledge base, and systems-level work should fall to them.

CMU and one of its project leaders — Chuck Thorpe — seemed to be the strong voice of dissent on many of these matters from the academic community. Ever since CMU had become a major contractor in DARPA’s Image Understanding work, it had taken a rather systems-level approach to attacking vision problems — working with civil and mechanical engineering professors like Red Whittaker to implement their vision work in robotic vehicles — rather than primarily focusing on the component level. The CMU team far preferred this approach to things like paper studies — in spite of the increased difficulty. So, as the project progressed and these conversations were ongoing, both sides had at least one party fighting to work on component work as well as some amount of integration work prior to Martin’s work integrating components in the last mile — often right before a demo.

While that future planning was taking place in the background, it was becoming clear that Martin Marietta’s pace of actual technical progress was not as positive as the demos had been. Their work implementing basic vision algorithms proved finicky as small changes like the lighting, road conditions, and shadows caused the vehicle to behave very differently from one day to the next. Each new software program or change seemed to cause a new problem which took time to debug. It was messy work. Additionally, to the dismay of Martin with its strict demo timelines, it began to seem like a large number of specific vision algorithms might be required to handle different, niche situations such as even conditions, uneven conditions, etc. This would take significantly more work than if one unified vision approach could cover most situations — as the team had initially hoped. 

Meanwhile, demo timelines began to get tighter and tighter. More demo-like activities kept being added as more PR was brought to the program by Clinton Kelly — news releases were issued, the press was invited to watch some of the vehicle's runs, etc. In the midst of all this demo pressure, the ALV project began to embody more of a negative “demo or die” principle than embodying a pull mechanism to effectively move the technology base forward. As Martin’s actual progress slowed and the demo schedule tightened, the company began to focus only on achieving the benchmarks of the next demonstration at the expense of developing truly useful, new technology. For example, for one of the demonstrations, the obstacle avoidance algorithm — put together by one of the component technology contractor companies — worked in such a narrow set of circumstances that it could not even be used successfully in a parking lot, but it could work on the roads the demonstration took place on. 

As Martin was rushed in these middle years of the program, the ALV was clearly not acting as the test bed for the SCVision projects — the name encompassing much of the vision work in PM Ronald Ohlander’s AI portfolio — that it was meant to be. Much of Martin’s staff time and equipment was dedicated to hitting the next demo and not finding adequate time to serve this more future-looking function. It does seem that the team was truly busy and not just blowing the researchers off. By April of 1986, the 1.5-year-old ALV had logged 100,000 miles and needed to have its engine replaced since it had worn out. In one two-month period of testing prior to a demo, 400 separate tests were conducted using the machine. Preparations for a subsequent demo might begin as much as six months before — often not long after the last demonstration had just finished. In essence, things were becoming very crunched at Martin.

In 1985, in the period between the first and second ALV demonstrations, Ronald Ohlander — a former student of Raj Reddy at CMU — decided it might be necessary to rely on a contractor separate from Martin Marietta to integrate the component technology improvements from the technology base more ambitiously. Ohlander wanted to issue a contract for a scaled-down version of the ALV operating from within the SCVision program. The integration contractor in charge of the machine would be in charge of testing the computer vision algorithms of other research teams that fell under the SCVision umbrella.

Ohlander, in doing this, adeptly observed that the prime contractor on the ALV project might be under pressure to meet its contract milestones and not be sufficiently motivated to fold the most cutting-edge component technology coming out of his SCVision program into the ALV. In addition, he seems to have felt a level of unease relying on a machine, contractor, and program management from a different DARPA office — Isler and Kelly largely operated out of the Engineering Applications Office —  to demonstrate the usefulness of the investments coming out of his own office — IPTO. Ohlander explained the decision to Roland and Shiman in an interview, stating:

It wasn’t to outshine them or anything like that…but we were…used to doing things in universities with very little money and getting sometimes some startling results. So I…had a hankering to just see what a university could do with a lot less money. 

In the end, the contract would go to Carnegie Mellon. CMU had already, unlike most of the component contractors in the SCVision portfolio, been testing its component technology on its own autonomous vehicle: the Terregator. CMU’s used the Terregator — for often slow and halting runs — to test the algorithms developed for its road-following work. The Terregator proved extremely useful to this research, but its use had also helped the CMU team learn enough to outgrow the machine.

In CMU’s ongoing efforts to implement sensor information from a video camera, a laser scanner, and sonar into one workable vehicle, the CMU researchers had come to realize that they needed funding for a new vehicle. CMU needed a vehicle that could not just accommodate the growing amount of sensor equipment, but also one that could carry out its tests with computers and grad students either on board or able to travel alongside the vehicle. This would allow bugs to be fixed and iterations between ideas to happen much faster. The vehicle that the CMU computer vision and robotics community was casting about trying to get funding for would come to be known as the NavLab. Fortunately, what they were seeking money for seemed to be exactly what Ohlander felt the SCVision portfolio needed. The funding for the NavLab began around early 1986. The funding set aside for the first two vehicles was $1.2 million, and it was estimated that any additional NavLab vehicles would cost around $265,000.

As Martin was somewhat frantically taken up with its demo schedule, CMU, with its longer time windows, was  free to focus on untested technology as well as technology that ran quite slowly but seemed promising in terms of building more accurate models of its environment. As the CMU people saw it, if anything proved promising they could upgrade the computing machine used to operate that piece of the system later on. The NavLab team was working closely with CMU’s architecture team that was developing Warp systolic array machines from within Squires’ computer architectures portfolio. The CMU architecture team was developing its equipment with vision applications in mind as an early use case. As a result, they were happy to change the order of functionality they implemented in order to account for the acute needs of the SCVision team. Additionally, if a piece of vision technology proved promising to the NavLab that could not be sufficiently improved with the immediate help of a Warp or some other new-age computing machine, the vision researchers likely figured that a piece of hardware powerful enough would likely exist soon to help push the idea along — even if not in the next year. Lacking the hard metrics of yearly demos with speed requirements loosely matching how fast the vehicle would need to go in the field, the CMU team could afford to think more along the lines of  “how do we build a machine that is as accurate as possible, even if it moves really slowly today?”

As CMU took up the mantle of building SCVision’s true test bed vehicles, it also took over the role of SCVision portfolio’s true “integrator” for Martin’s ALV. Carnegie submitted a proposal to carry out this integration work along with building the NavLab. From a high level — in spite of neither the ALV nor SCVision projects having set aside money for an integration contractor — this made sense. Martin was meant to integrate work from the technology base on its own, but around mid-1985, it was becoming clear that they were not doing an effective job on their own. Academic departments were often close to the evolving frontier of their own field, but often did not have the project management skills and scaffolding to do great integration work. CMU was somewhat exceptional in that it did this sort of thing quite frequently. Clinton Kelly, while apparently not thrilled about the idea since he was a massive supporter of the ALV project, agreed to the arrangement. 

As 1986 went on, Martin Marietta was able to overcome issues with its vision system well enough to continue passing its demos which required the machine to go on larger tracks at faster speeds — such as 20 km/h on a 10km track with intersections and some obstacles. As 1987 approached, plans were being made to upgrade the computing equipment in the machine — some of the new computers were developed in Squires’ architectures portfolio. Additionally, as the demos in the coming years were slated to take a marked step up in terms of the complexity of their missions and terrains, the Martin team’s relations with CMU as the new integration contractor — a bit cold at first — warmed considerably.

Personnel from both orgs kept in regular touch and visited each other’s sites. Martin even went as far as to enroll a member of its staff as a grad student under Chuck Thorpe to serve as a liaison between the two. In mid-1986, CMU began helping Martin install software for map planning, obstacle detection, sensor calibration, and new image recognition functions that it had developed and integrated on its own machines on campus.  This period also saw specialized tools developed so CMU could more easily run its code without alterations on Martin’s machines — which ideally would have been done earlier in the project, but better late than never. The machine was steadily improving. In the summer of 1987, when one of the private sector contractors — Hughes — was at Martin’s Denver testing facility to test its planning and obstacle avoidance software, the ALV made its first extended drive in country conditions on a route it had selected for itself using only map data and sensor inputs. As Roland and Shiman wrote of the rather triumphant run:

The vehicle successfully drove around gullies, bushes, rock outcrops, and steep slopes.

However, that mid-1987 period would prove to be the high water mark of the ALV application program. After the ALV’s November 1987 demo, a panel of DARPA officials and technology base researchers met to discuss phase two of the program. Roland and Shiman described the titanic shift in how the ALV project and the SCVision portfolio operated in the wake of that meeting as follows:

Takeo Kanade of CMU, while lauding Martin’s efforts, criticized the program as “too much demo-driven.” The demonstration requirements were independent of the actual state-of-the-art in the technology base, he argued. “Instead of integrating the technologies developed in the SC tech base, a large portion of Martin Marietta’s effort is spent ‘shopping’ for existing techniques which can be put together just for the sake of a demonstration.” Based on the recommendations of the panel, DARPA quietly abandoned the milestones and ended the ALV’s development program. For phase II, Martin was to maintain the vehicle as a “national test bed” for the vision community, a very expensive hand servant for the researchers.

For all practical purposes, therefore, the ambitious ALV program ended in the fall of 1987, only three years after it had begun. When phase II began the following March, Martin dutifully encouraged the researchers to make use of the test bed but attracted few takers. The researchers, many of whom were not particularly concerned about the “real-world” application of their technology, showed more interest in gathering images from ALV’s sensors that they could take back to their labs.

The cost of the test bed became very difficult to justify. Phase I of the program had cost over $13 million, not a large sum by defense procurement standards, perhaps, but large by the standards of computer research. Even with the reduced level of effort during phase II, the test bed alone would cost $3–4 million per year, while associated planning, vision, and sensor support projects added another $2 million. Perhaps most disappointingly for Kelly, the army showed little interest in the program, not yet having any requirement for robotic combat vehicles. One officer, who completely misunderstood the concept of the ALV program, complained that the vehicle was militarily useless: huge, slow, and painted white, it would be too easy a target on the battlefield. In April 1988, only days after Kelly left the agency, DARPA canceled the program. Martin stopped work that winter.

The Autonomous Land Vehicle hit its agreed upon, quantitative benchmarks in its early demonstrations. It also incorporated many technologies from the SC technology base into successive iterations of the ALV such as BBN’s Butterfly processor, CMU’s Warp, and other vision technologies that CMU and others had helped develop. If the Army had been excited by the far-off sounding application and the ALV’s progress in the early demos, there would likely have been a much higher chance that the program continued on schedule. However, since the Army was not yet interested in autonomous recon vehicles, the ALV’s biggest service to DARPA and its portfolio was as a technology pull mechanism to help move its technology base forward in a coordinated fashion. And since Martin Marietta had proven rather lackluster in the integration department, DARPA’s decision in late 1987 to wind down the ALV project and, instead, put more emphasis on the NavLab makes sense.

Work in the spirit of the ALV largely lived on in CMU’s NavLab efforts. CMU’s somewhat unique setup — running an engineering research program equipped with de-facto program managers from within an academic department — proved extremely effective in carrying out this project. With the organization dedicated to attacking an extremely concrete goal like autonomous driving, the CMU team was able to keep a clarity of vision throughout the project — while maintaining an organized structure that was less strict than a company but more organized than a traditional academic department. CMU would go on to make very noteworthy contributions in the years since the ALV was wound down. In an IEEE oral history of CMU NavLab program manager Chuck Thorpe, Thorpe describes the steady practical progress of the NavLab team, comparing it to his newborn son, saying:

My son was born in ’86, and we had contests to see when NavLab 1 was moving at a crawling speed, my son was moving at a crawling speed. When NavLab 1 picked up speed, my son was walking and learning to run. NavLab 1 got going a little faster, my son got a tricycle. I thought it was going to be a 16-year contest to see who was going to drive the Pennsylvania Turnpike first, because they were each progressing at about the same speed.

Even without the consistent proof of progress demos such as those that drove the ALV project, steady progress was being made by the interdisciplinary, practically-minded, academic CMU team. But the team’s progress very much received one of those unexpected jolts of progress that groups of researchers diving down riskier rabbit holes with some regularity hope to experience. One of CMU’s grad students working on the project, Dean Pomerleau, published a 1988 paper on a system called ALVINN that would be a major step in showing us the future of the field. Thorpe finished the paragraph he started comparing the NavLab and its steady progress to his son, recounting:

...I thought it was going to be a 16-year contest to see who was going to drive the Pennsylvania Turnpike first, because they were each progressing at about the same speed. Unfortunately, this really smart graduate student named Dean Pomerleau came along and built a neural network technique, which managed to be the most efficient use of the computing researchers that we had, and the most innovative use of neural nets at the time, so in 1990, he was ready to go out and drive the Pennsylvania Turnpike.

Thorpe then recounted how, by 1995, a later version of the NavLab running on a newer vision system, RALPH, took things even further. RALPH’s approach — which Pomerleau also came up with —  combined pieces of ALVINN’s neural net approach with a new model-based approach which was similar in spirit to simpler approaches used in the ALV. Using RALPH’s vision system, the NavLab was taken by Pomerleau and another student from D.C. to San Diego, driving 98% of the way autonomously. Things like “new asphalt in Kansas, at night, with no stripes painted on it,” flummoxed the machine. But it was plain for all to see: the technology was taking massive steps.

The legacy of the ALV project is more complicated than that of PA and Battle Management, but it may have done more good in shaping the technology base than the other two projects combined. All three projects had their bright spots as well as their weaknesses.

In terms of customer relationships, PA and Battle Management had significantly better relationships with their armed services customers than the ALV had with the Army. Battle Management’s practical success in terms of positive impact on its customer, the Navy, seems to have outdone the practical success of the PA. But it is hard to make that out to be solely the fault of the PA or its management team. After all, in many situations in ARPA-like orgs, you simply get the customer you get.

That being said, Battle Management certainly provides a great exemplar of how to — with the right customer — integrate a test bed as close to the production process as possible to minimize the number of annoyances that can arise when translating technology built in lab environments to real-world environments. This strategy from the Battle Management team also helped mitigate the organizational/logistical hold-ups that often result when looking to implement technology from a remote test bed.

The PA, in spite of its lack of success in pushing a truly useful product out of the program, proved to be a somewhat successful pull mechanism for the speech recognition and natural language understanding technology base. Let’s remember, to many, it was not crystal clear that speech recognition and NLU research was ready for translation into products. The PA application and associated funding on related problems in the technology base helped harness the efforts of DARPA’s performers to work on common problems. This, in the end, helped push certain areas of research into practice. In other cases, the PA highlighted areas that the field needed to work on.

The ALV, in contrast to these two projects, never had the customer relationship of PA and Battle Management. Not even close, really. The Army did not want the ALV; the Air Force wanted a working PA and the Navy was extremely enthusiastic about Battle Management. However, those at DARPA understood that the Army was not excited about getting involved early on with this project. So to judge the program as a near-term failure as an application is somewhat fair, but also largely misses the point. The ALV was something different than the other two applications: an extreme pull on the DARPA technology base. And, in that, the program did a much better job. A large sum of money and energy may have been spent non-optimally pursuing the Martin Marietta work when it was the prime contractor. But DARPA was also relatively quick to redirect the portfolio after several years of watching its broad array of contractors work and assessing their approaches and their results. In the end, CMU proved to be better suited to help push the vision area of the portfolio forward than Martin and the ALV. Fortunately, DARPA was light on its feet in making that change in strategy happen.

It is somewhat hard to tell what the legacy of the DARPA’s vision work during SC would have been had it not been for the exceptional setup, management, and overall culture at CMU with its DARPA contracts. The CMU computer science and engineering teams involved with DARPA embodied some of the most positive aspects of academia and some of the most positive aspects of industry. I will explore the setup of the CMU department and how they fulfilled their DARPA contracts further in a coming piece. What matters, for the scope of this piece, is that the CMU department was exceptionally equipped and internally motivated to go after a problem like practical vision applications, and a large amount of DARPA money found them and enabled them to go on to do exceptional work.

Each of these three projects provides much to learn from for a PM orchestrating and balancing a portfolio of investments that is looking to show some level of practical success while also hoping to maintain a high level of ambition. Hopefully this piece proves of some use to them.

Thanks for reading. Subscribe and stay tuned for coming pieces.

Subscribe now

Share

• Pilot's Associate: a cooperative, knowledge-based system application

  • A report from the Air Force program managers for the PA detailing the performance of the PA’s subsystems in its demos. 

• DARPA’s John Retelle meeting, presentation, and Q&A on the PA to the Transportation Research Board

  • Contains a mention of being able to transition the program to parallel computing machines around 1990. 

• Rizza, the Wright PM on the PA project, speaking at the 1991 Space Operations, Applications, and Research meeting

  • In the document, Rizza explains how the systems and principles that helped make the PA something of a success could be applied to NASA’s work with managing the movement and maintenance of deployed satellites — which was very man-power intensive at the time

• Bachman, Gunning, and Burke’s Integrated Artificial Intelligence Systems

  • Interesting throughout, but its primary relevance in this section of the write-up is the authors’ clear description of why exactly the machine was not pursued further from a component technology capability standpoint.

• 1986 document from John Flynn outlining some aspects of the early demos

  • DARPA Naval Battle Management PM and then long-time BBN employee goes into some depth on the specific hardware, expert systems, user experience, and contractors on the projects.

• Computer Effort Falling Behind article from Michael Schrage written for the Washington Post in 1984

  • Certain pieces of the article did not age particularly well when it comes to a program like Battle Management — a Pentagon insider even referred to these programs as “duds.” Within two years of the article being written, battle management was becoming a clear success. 

  • But other pieces of the article — like the tension between Kahn and Conway — were very fair and got at something that was a real complication for the program for years.

  • The article provides a clear view of how programs like these might be covered in the press if things do not hit the ground running. 

•  Chapters 14 and 15 of A Culture of Innovation — a book on BBN’s 

  • From a high level, the book dives into BBN’s culture, structure, and details on many of its most famous projects

  • Chapters 14 and 15 provide great detail on the specifics of the researchers, contracts, and work details surrounding BBN’s speech work in the early-1970s all the way through the early-1990s.

• Chapter 14 of Van Atta’s DARPA Technical Accomplishments. Volume 2. An Historical Review of Selected DARPA Projects 

  • This chapter outlines much of the lead-up to this generation of DARPA’s vision work.  Particularly useful is its focus on the Image Understanding program which kicked off in the mid-1970s but had its roots in work from the prior decade. 

  • The work dives into the details of specific projects as well as what specific performers worked on what research areas in different eras. 

• CMU Strategic Computing Vision Project Report: 1984 to 1985  

  • This report outlines the first year of CMU’s work on the application and DARPA-relevant vision work as a whole.

  • Particularly goes into more depth on the technical details of various CMU vision research approaches in the early years of SCVision in using technologies like sonar, laser vision, stereo cameras, and video cameras.

• Martin Marietta road-following write-up: The Autonomous Land Vehicle (ALV) Preliminary Road-Following Demonstration 

  • Martin Marietta describes the finer technical details of the ALV during its first demo.

  • The write-up also dives into how Martin Marietta hoped the system would progress from demonstration to demonstration as well as their planning in this nascent stage.

• Dean Pomerleau’s 1988 paper, ALVINN: An Autonomous Land Vehicle in a Neural Network

  • The paper dives into Pomerleau’s initial implementation and results using a 3-layer backpropagation neural network trained on approximately 1200 simulated images of roads to train the model used to steer the car. At the point of the paper being written, the performance of this system seemed roughly equivalent to CMU’s more manual methods — often utilizing expert systems-style reasoning given the sensor data and clever edge-finding heuristics — of training models to steer the machine to that point.

• Chuck Thorpe IEEE oral history

  • Explores many of the finer details of how the projects came together and progressed at CMU. It is great fun to read for anybody interested. 

• Misc.

  • Developing Technologies for Army Autonomous Land Vehicles — 1985 document written by army engineers on the program area

  • Implementation and performance of a complex vision system on a systolic array machine

  • Norberg’s oral history of Alan Newell

• Strategic Computing: How DARPA Built the Computer Age

• Strategic Computing’s founding document

A team of officials from DARPA and the Engineering Topographical Laboratory (ETL) at Fort Belvoir evaluated the proposals and made the selection. ETL was DARPA's agent for the ALV program as well was the SCVision program.

This piece is a part of a FreakTakes series. The goal is to put together a series of administrative histories on specific DARPA projects just as I have done for many industrial R&D labs and other research orgs on FreakTakes. The goal — once I have covered ~20-30 projects — is to put together a larger ‘ARPA Playbook’ which helps individuals such as PMs in ARPA-like orgs navigate the growing catalog of pieces in a way that helps them find what they need to make the best decisions possible. In service of that, I am including in each post a bulleted list of ‘pattern language tags’ that encompass some categories of DARPA project strategies that describe the approaches contained in the piece — which will later be used to organize the ARPA Playbook document. These tags and the piece itself should all be considered in draft form until around the Spring of 2024. In the meantime, please feel free to reach out to me on Twitter or email (egillia3 | at | alumni | dot | stanford | dot | edu) to recommend additions/changes to the tags or the pieces. Also, if you have any ideas for projects from ARPA history — good, bad, or complicated — that would be interesting for me to dive into, please feel free to share them!

Pattern Language Tags:

• Encouraging modularity and open interfaces

• Democratizing an area of research via cost reduction

• Promoting a coordination/service mechanism to reduce material costs and increase research feedback cycles

• Facilitating tool/hardware improvements in a key technology area far from its suspected theoretical frontier

In the early 1980s, a large amount of money and energy from DARPA and the DoD was shifting towards computing infrastructure and applications. While many involved with those organizations already had one (hopeful) eye on long-term applications such as AI, it was clear that almost all aspects of DARPA’s AI plans, in some way, leaned on ease of access to computer chips for the research community.

So, as a suite of DARPA and DARPA-adjacent projects sprung up — all in the hopes of systematically building up the greater universe of chip capacity and computing applications — DARPA’s MOSIS project was established to ensure that researchers working on computing problems could have faster and cheaper access to computer chips. In this early era of computing research — when the technical capabilities of chips were orders of magnitude more limited than they are today — it was the norm for many researchers and engineers to design bespoke chips for the particular task at hand. MOSIS would provide the middleman service to all researchers who needed their designs fabricated into chips more cheaply, quickly, and discretely than would otherwise be available.

In the late-1970s, printing chips was extremely expensive. For independent researchers paying a separate facility to print their design, the price of a single wafer could have a price tag in the tens of thousands. Since hardware was often specially developed for individual applications or experiments, single research projects often required printing individually designed chips. The cost was a big problem and the process itself was a major annoyance to researchers. Manufacturers who printed chips often had specialized syntax in which they encoded the digital designs into their machines. Going back and forth with a manufacturer was a major tax on researchers’ time and timelines. Also, researchers handing all of their designs to manufacturers was a point of concern. If their project was a success and they attempted to commercialize it, one of the researcher’s now-competitors would have their digital designs in hand. The system of fabricating chips, at the time, was just not set up for convenient access by the community of individual researchers. That is why MOSIS was created.

MOSIS could make things cheaper and easier for these researchers. While individual silicon wafers and runs with manufacturers were expensive, the average chip that needed to be fabricated for a standard research project was quite small — about 1/20th the size of a wafer of silicon. Since the field was new and there was not exactly a mass market of hobbyists who would pay for many chips to be run in parallel, a market/middleman service had not been made for this particular service. Without a solution, new research might be far more expensive, time-consuming, and sparse than it otherwise could be.

In the backdrop of all of this was the belief – encompassed by a 1976 RAND report by Carver Mead, Ivan Sutherland, and Thomas Everhart – that the integrated circuit revolution had only half run its course. Another four orders of magnitude improvement in chip manufacture seemed theoretically possible. Despite this potential, the authors believed that many industry players largely seemed to persist in incremental development. They believed DARPA should take steps to get involved and help speed up ambitious development and research work in the field.

The demand and usefulness of the eventual MOSIS service were first proven through Carver Mead’s chip design course at Caltech — leveraging a more computational, less manual pipeline for designing chips he had developed — and subsequently through courses modeled after Mead’s course at MIT and elsewhere.

Mead wanted to teach his students how to design VLSI (Very Large-Scale Integration) chips. Since there was no single industry standard to teach to because each manufacturer had its own protocols, Mead developed a general set of protocols out of the disparate standards that would allow students to learn the principles in a specific way — but in such a way that the students would still be useful to any arbitrary company they may go work for after graduation. Above all, Mead wanted students to learn to design chips by actually designing their own chips.

To make this goal feasible, printing the chips would have to be cheaper. So Mead figured out a way to put many chip designs onto the same wafer to reduce printing costs. Mead turned to Lynn Conway — Head of LSI System Design at Xerox PARC — and Xerox PARC fulfilled the role of silicon broker in this first iteration of the course. And it was a success. Mead and Conway succeeded in the course's goal of trying to ‘totally isolate the designer from all the trivia that fabrication requires’ to enable students to learn chip design faster.

These learnings and design standards were eventually codified into what became the bible of VLSI design, a book co-authored by Mead and Conway titled Introduction to VLSI Systems. This book and the course's approach were how the course’s teachings began to reach more schools and researchers. The approach demystified chip design and enabled researchers to learn chip design in a course setting — rather than having to consult for a chip manufacturer to learn all the little tips and tricks of the trade as was common at the time. You didn’t need to know everything about chip manufacturing to design workable chips anymore. In 1978, Conway went to MIT to teach MIT’s iteration of the course for the first time. By 1980, things were progressing. In the spring alone, “250 designers from 15 universities and R&D organizations submitted 171 projects for implementation.” These specific chips were fabricated by HP.

With the potential of the service proven at several schools and across multiple manufacturers, Mead, Conway, and others began pushing more strongly for a DARPA project to take over and scale this work. At the time, most DARPA PMs made efforts to be in close contact with as many potential PIs as possible in relevant research areas. In this particular case, Conway — while at Xerox PARC — began pushing Kahn — then head of IPTO — to push this work forward. Although, it is very likely that if Conway had not made this push, Mead could also have reached out to one of the IPTO PMs who frequently worked with CalTech.

The Information Sciences Institute (ISI) — which had an affiliation with USC but was independent of the university in many ways — was granted the role of DARPA “silicon broker.” In the 1980s, they became one of IPTO’s go-to performers to carry out a variety of services like this which required research familiarity as well as the organizational skills of a company — such as orchestrating the Machine Acquisition Program in the adjoining piece. ISI was started by a group of researchers who felt their former employer, RAND, was too geared towards paper studies and did little in the way of directing research and work on technology itself. ISI was an unfunded research arm of USC — paying its own way through (mostly DARPA) grants and contracts. Their regular work included performing services for DARPA, conducting applied research, creating technology transfer programs, and more. ISI had often conducted work somewhere in between applied research and DARPA program administration before, and, thus, was very well-suited to this new role.

Once the program was established, the user experience was quite straightforward. Those who had access to MOSIS — largely NSF researchers, DARPA grantees, and other individuals whose funds were largely from the government — communicated with the MOSIS central office via the ARPANET. MOSIS put together a user manual explaining how electronic mail via the ARPANET worked — those were the early days of the internet — and how to email designs and queries to the mail system in a structured way so a program could sort them.

Researchers sent their designs — in CIF files — to MOSIS. MOSIS, initially using a system designed by Ron Ayres of ISI, took it from there. The system:

• Checked the CIF file to ensure it conformed to the basic design standards outlined by MOSIS in the user manual.

• Packed sets of projects onto a smaller set of dies.

• Translated each die into MEBES format.

• Made bonding diagrams for manufacturers.

• Most importantly, produced tapes that the foundries used to make masks.

Danny Cohen and George Lewecki of ISI, in a 1981 talk on the early MOSIS system, described how the process continued from there, saying, “The next step in the process is mask fabrication. Mask houses expect two types of things from us: tapes with MEBES files and job decks. MEBES files contained the information that the mask houses used to make bitmaps (which are made into masks). A job deck…contains the specifications for each MEBES file — parity, record size, etc.”

The talk continued, “Fabrication itself is very simple because somebody else does it. Once the masks are made, all we have to do is drive three, four, maybe ten miles in Silicon Valley with the masks to a wafer fabricator. (It is wise to drive slowly to make sure the masks don’t break.) After that, if we’re lucky — and typically we are — we end up with a couple of wafers.” MOSIS completed its job by dicing, bonding, packaging, and shipping chips.

The MOSIS staff also took measures to ensure that they were as ‘lucky’ as possible. And when they weren’t lucky, they made arrangements that ensured as few defective chips as possible were handed to users. They accomplished this in several ways. MOSIS carried out basic probing tests on transistors, inverters, etc. ISI kept track of defect rates for any given type of job from all of their fabricators to 1) ensure that they continued to work with the best fabricators possible and 2) that they ordered enough duplicates of any given part for a researcher to ensure a 90% probability that a given print would yield a working chip for the researchers — helping keep research projects on schedule.

ISI also carried out different forms of applied research to ensure progress in a variety of operationally important areas. Some of these areas included exploring the feasibility of expanding the service into new chip types, working to incorporate new coding language into ISI’s workflow, using new varieties of machines in their work; etc. In general, it was ISI’s job to do any research work required to make the service work as efficiently as possible and take advantage of the rapid technological improvements happening in the field of computing. While the ISI team did not set out to do fundamental research in its own right, the ISI team’s skills as researchers may still have been important to the project’s success. In the hunt to make chips smaller, new technical problems constantly arose in fabrication processes. Courses of research were often required to work through these problems.

One ISI research project involved finding a way to translate the different design languages used by researchers into those used by manufacturers. Another case — discussed in the Cohen and Lewecki talk at CalTech — outlined how the ISI team worked around a difficulty in testing new softwares to see which software was best:

The problem arises in comparing two masks, one produced by the old [software] and the other produced by the new system. Are the patterns really the same? A microscope is supposed to help, but it can’t do a good job. We tried many ways and finally worked out a very strange technique. Suppose you want to compare mask A and mask B. What we did was to overprint A and B bar [the reverse of B], and A bar and B. In this way we discovered all the changes. We did all the printing on one plate so we wouldn’t have to use a special microscope.

One final — often forgotten — aspect of MOSIS’ success was that it provided a layer of security and confidentiality to researchers and manufacturers. Researchers came to trust ISI agents and their system for communicating design instructions to manufacturers. As Roland and Shiman — who wrote an in-depth history on DARPA’s 1980s computing efforts — noted, “Only ISI had the electronic version of their designs; versions on masks or even on the chip itself were prohibitively difficult to decode.” Manufacturers also maintained open lines of communication with ISI as they became convinced that ISI respected their proprietary information. Many recognize that ISI’s service in this project was ill-suited to a traditional university department. However, as this remark from Roland and Shiman also indicates, this particular piece of ISI’s service may have been hard to replicate for certain companies in DARPA’s performer pool as well. 

The MOSIS program is widely considered a success.

The number of wafers MOSIS printed rose steadily throughout the 1980s as demand increased and the field developed. The number of wafers printed grew from 258 in 1981 to 1,790 by 1985. The cost of the average chip for a researcher often came in around 5%-10% of what it would have cost if a researcher had to pay for the entire wafer. In 1988, George Lewecki noted that a single 2µ chip in a MOSIS run could cost as little as $258 (~$650 today). For the most part, DoD and DARPA contractors got the service for free, NSF grantees paid for the service out of their research grants (and eventually received the service free), and any commercial actors had to pay full price.

Throughout the 1980s, the average turnaround for chip times from MOSIS gradually decreased from around 10 weeks to about 8 weeks. The ability to print smaller and smaller features for a given cost and time frame continuously improved by large margins as the field evolved. MOSIS’ somewhat stagnant turnaround time was a conscious decision because allowing more projects to fill the queue helped ISI take further steps to reduce costs.

MOSIS was considered a quite efficient operation. A 12-month period from 1987-1988 saw $9 million flow to the MOSIS program. George Lewecki noted that 74.3% of MOSIS expenses went to fabrication costs, 13.1% to salaries, and 11.4% to ISI overhead.

Some in industry were annoyed that MOSIS might support future competitors of theirs. Private researchers could get free access to resources that TI, Intel, and IBM had to pay for. These researchers were not just looking to expand the knowledge frontier, but spin out companies from their university research if things worked out. Conversely, some within the government were annoyed that the MOSIS service — which was primarily targeted at university researchers — could be used by commercial actors at cost. Essentially, they felt the years of infrastructure and know-how MOSIS built up were not being priced in, and the service was essentially providing an industry subsidy.

To this writer, those criticisms are not to be taken too seriously. But a third area of criticism is worth examining. Some didn’t like that MOSIS seemed to be a never-ending program. MOSIS was providing a service that required some research and also helped the research community, but once it settled into its patterns it was not considered high-risk — even if it was seen to be high-payoff. DARPA tended to focus on higher-risk R&D projects. However, MOSIS also did not seem like a sure-fire commercial bet to spin off into its own firm. While the service was successful, it was not considered a good bet for venture capital by many familiar with the program. It was more of a service and contemporary VCs preferred products. Additionally, 98% of the service's customers — in this young and research-focused area — were from DARPA, NSF, or DoD funding sources. Competing firms had little need for MOSIS' main service of subdividing wafers. 

If left on its own in the private sector, one could not be sure MOSIS would survive. For example, it could end up in some sort of ruinous cost-cutting competition with a competing manufacturer and end up out of business. But, still, the service was too good to terminate because it offered a cost-effective way to support a vital area of research and teaching. So, instead, MOSIS was moved under the umbrella of more traditional defense funding to continue operation indefinitely. Funding for the program by the DoD and NSF ended in 1998. However, the program does still exist today under the ISI umbrella at USC. Its continued existence — in an era of very different technological conditions than those that created the original need for the program — may be considered, by some, a data point in favor of “sunset clauses” for certain projects. But I will not address that point further as it is somewhat out of scope of this series.

All in all, the program is considered a success. The MOSIS program succeeded in lowering the barriers to doing ambitious research in a broad research area whose progress was all-important to DARPA and the DoD’s computing goals; microelectronics research could be done by individual researchers for a fraction of the cost it would have previously cost. Additionally, as Kuan and West write, MOSIS “modularized [decomposing a problem into separate modules] the semiconductor ecosystem” by enabling semiconductor design to be done productively on its own. With MOSIS, DARPA was able to help alter the structure of a field in a way that industry incumbents may not have done on their own — but was likely better for the progress of the field.

Previously, design was generally done by those with the skills, experience, and machines to manufacture semiconductors — large industry players. The MOSIS service was able to help unbundle these two separate tasks and pave the way for the semiconductor industry to subdivide into design-only  “fabless” firms and production-only foundries. With so much of this fabless design research being done by the academic community — rather than existing private sector incumbents — the field itself was able to be built out in the open, with researchers making rapid progress by leveraging one another’s discoveries.

Some might believe that rising tides lift all boats and, thus, any infrastructure program in an area that turns out to be explosively productive — like mid-1980s computing hardware — will look good in retrospect. However, one should keep in mind that not all of DARPA’s Strategic Computing infrastructure projects with similar goals from this same era are considered a success in a similar vein. For more on that point, see the accompanying piece on DARPA’s Strategic Computing Machine Acquisition Program. Some feel there could have been a world where the MOSIS investment’s heavy reliance on Mead-Conway standards artificially incentivized researchers to develop λ proportions at the expense of others. The accompanying piece describes a program that made a somewhat analogous error. However, I believe that MOSIS’ placement as more of a broker than a capital purchaser provided it a greater level of flexibility to work around this category of issue that ailed the Machine Acquisition Program.

• Strategic Computing's Machine Acquisition Program

Thanks for reading! Subscribe to continue following along with the series!

Subscribe now

Share

• MOSIS User Manual .txt file

  • Particularly useful if the reader wants to understand how MOSIS instructed self-onboarding onto a complicated technical service.

• Danny Cohen and George Lewecki giving a talk at Caltech on the project in 1981

  • Particularly useful if the reader wants to understand: 

    • How MOSIS ran into problems and solved them using research or other qc measures

    • How the MOSIS service operated in more detail

• Chapter 4 of Strategic Computing: How DARPA Built the Computer Age

  • Particularly useful if the reader wants to understand how MOSIS progressed from just an idea, through its early years, and into a mature program. 

  • Also covers the politics involved in maintaining the program. 

  • Additionally, any details about specific chip design measurements and program decisions are covered at moderate length in this chapter.

• Kuan and West’s Interfaces, modularity and ecosystem emergence: How DARPA modularized the semiconductor ecosystem

  • Particularly useful if the reader would like to understand (in more detail) how MOSIS was able to unbundle design from manufacturing capacity and the effects of this on the commercial and research actors in the space.

• Misc.

  • 1983 ISI report on the present and future of MOSIS

  • MOSIS: A Gateway to Silicon

  • MOSIS 1983 Annual Technical Report

  • Ronald Ayres narrative history of MOSIS software

• Strategic Computing: How DARPA Built the Computer Age 

• Introduction to VLSI Systems

• Keith Uncapher’s Oral History

  • Dives into the founding of ISI as well as the significance of the MOSIS system.

• Robert Kahn’s 1989 Oral History

  • Briefly dives into the overarching need for the program and industry’s trepidation to provide fab capacity to the program at the start.

This piece is an accompaniment to today’s MOSIS piece. So please read the MOSIS piece before starting this one.

Pattern Language Tags:

• Promoting a coordination/service mechanism to reduce material costs and increase research feedback cycles

In the early 1980s, as DARPA’s Strategic Computing Initiative was getting underway, lack of cheap computing power was one of the primary concerns of DARPA performers. The work of MOSIS in reducing the cost of VLSI chips for chip research was one major “infrastructure” investment by DARPA that helped overcome a cost issue in an area of strategic importance for the research community. Another group of performers plagued by a similar problem was DARPA’s corp of software/AI researchers and their need to acquire high cost machines to continue with their current research agendas. PM Ronald Ohlander — who managed many AI projects for IPTO as the Strategic Computing program began — undertook the SC Machine Acquisition Program to help DARPA researchers overcome this cost problem. The program — carried out in a similar period to MOSIS and riding a similar technological wave — would have much more mixed results than MOSIS.

With the launch of SC in 1983, Ronald Ohlander launched the Machine Acquisition Program under the SC umbrella. The program was meant to help boost the productivity of SC’s AI research community. The Machine Acquisition Program would use SC resources to buy those in the AI and software research community LISP machines — the high level programming language popular for AI research at the time — to meet their needs. The program sought to do so with significant cost savings by negotiating bulk discounts with manufacturers. One major drawback was that these specialized LISP machines could only be networked to other LISP machines at the time and could not be time shared, but the AI research community was confident that these machines would be a massive factor in enabling cutting-edge AI research in the coming years.

So, Ohlander went forward with the program.  ISI — the org that had won the MOSIS contract — also won the right to run this much lower touch contract. Approximately six months after ISI first proposed to serve as the machine acquisition center for SC, it had won a sole-source justification to service the early part of the contract and made its first purchase — in March 1984. With ISI’s first purchase of 30 Symbolics 3600 LISP computers — which cost a total of $2.8 million total at the time (~$8 million today) — the program was off and running.

ISI charged no overhead rate to DARPA for the contract and only asked to be compensated for the cost of the machines purchased and the time and resources ISI invested into the contract. The primary service provided by ISI — besides buying machines for researchers on SC’s behalf — was working with the staff at computer manufacturers to negotiate bulk discounts for the researchers. Given how limited the market for LISP machines was — which is primarily what the program would specialize in buying — the demand for LISP machines by DARPA researchers represented a substantial portion of the market. So, ISI’s job was primarily to find a way to negotiate deals so DARPA researchers’ collective demand could translate to lower prices. 

Beyond this function, ISI carried out a handful of services to enhance the user experience of the researchers and limit the headaches that are sure to arise when acquiring new machines in a young manufacturing area. ISI would operate the machines at its office for about thirty to sixty days to ensure there were no problems. And, when there were problems, they would coordinate with the manufacturer to either repair the machine or have a replacement sent. 

Ohlander and DARPA required that Common LISP be offered on every machine purchased by the program and pushed the research community and industry to settle on the specification of a common LISP language. Ohlander had taken the temperature of the AI research community and they felt that LISP was clearly the best language for their work now and would continue to be moving forward. So, Ohlander and DARPA did their part to ensure that these researchers had more LISP machines sooner rather than later, that they could be acquired as cheaply as possible, and that the community settled on common LISP standards so they could seamlessly build on each other’s work and communicate findings. 

By the end of 1985, when the program was in full swing, the Machine Acquisition Program had spent over $8 million on LISP machines for the SC research community. These machines would have retailed for around $50,000 to $100,000 but were acquired for about a 35%-45% discount by the ISI team. Additionally, ISI often secured massive discounts for additional services such as follow-up maintenance from manufacturers. On one occasion, ISI even secured a maintenance fee of $76,176 from Symbolics for services with a book value of $495,000. DARPA received all these cost savings for only the service cost paid to ISI — which was equivalent to about 2% of the cost of what they spent on these massively discounted machines. 

However, the program quickly encountered a change in the technical landscape that caused some headaches. Ohlander had set up the program to give the researchers as much choice as possible. DARPA was not going to tell the researchers which computers were best suited to pursue their research goals. So, ISI planned to buy whichever machines DARPA’s performers asked them to buy. The program was set up with the four main 1983 manufacturers of LISP machines in mind — Symbolics, TI, LISP Machines Inc, and Xerox. The LISP machines from these vendors were supposed to represent the near future of the field — so the researchers had said. But as early as 1986, the research community had almost entirely shifted its preferences, now preferring the program to acquire them the new general-purpose workstations such as Sun’s Sparc, which retailed for three or four times less than the LISP machines. These machines, made possible by substantial commercial advances in computer architecture and manufacture, were powerful enough to run LISP and more versatile than the more expensive, specialized LISP machines.

The program’s activities contracted substantially during the budget crisis in 1987 and 1988. When the program revived, preferences had once again begun to shift. Performers were beginning to become interested in the new lines of parallel processing machines — which SC had spent significant resources helping develop. 

Between 1984 and 1992 the Machine Acquisition Program spent over $15 million on equipment for its research contractors. From a cost-savings perspective, the program was absolutely a success. And from a short-term point of view, all parties had their needs satisfied. Researchers got the machines they wanted when they wanted them. ISI even did a lot of installation and debugging for them. Manufacturers in the relatively small market for LISP machines received a much appreciated demand boost. And DARPA was able to speed up the velocity in which research was done in its AI community, receive its products and services at a substantial discount, and get the research community to agree on common LISP standards to ensure its money was spent on researchers who could iterate on each others’ work as quickly as possible.

But does all of that overshadow the fact that the majority of the money the program spent was on machines that were essentially obsolete within a couple years of their initial purchase? To use an objective term like “mistake” with the benefit of hindsight is probably unfair. The research frontier is constantly shifting and the role of people like DARPA PMs is to shape that frontier, adapt rapidly when things change, and also make peace with the fact that money will sometimes be spent on dead ends. Whether or not the decision to fund the program was considered a mistake internally would depend on whether Ohlander and DARPA management had adequately considered this possible outcome and accepted the risks, or if they were somewhat blindsided by the outcome.

Looking at the landscape as they would have seen it, in 1983, is a useful exercise for PMs. In 1983, those who worked on SC programs generally understood that most of the AI/software applications coming out of the work SC was funding would build on top of the computer hardware and architecture developments DARPA was funding in the early 1980s. While projecting technological developments can be messy, it seemed quite clear that orders of magnitude improvements in computing were coming and would impact the researchers who made use of computing equipment in five years or less. In a very active way, SC was explicitly aiming to help achieve the orders of magnitude improvements in computation that they knew to be possible as rapidly as possible. In fact, as I explore in my piece on SC’s three main applications (coming out this Tuesday), some of the programs were counting on one or two orders of magnitude improvements coming by the end of the decade. The rapidly evolving hardware landscape was an area that many SC PMs kept even more abreast of than those in the AI research community. Yet, in spite of the rapid changes occurring in the field, the Machine Acquisition Program invested a yearly amount similar to what SC was spending on MOSIS to buy machines that were only useful to the AI research community as long as the current state-of-the-art only improved along a very foreseeable path — foreseeable to the AI researchers in particular. If Ohlander and DARPA knew this might happen, but felt $15 million was a small cost to pay for getting the AI researchers moving on their projects as fast as possible, it would be hard to call the work this program did a mistake. But if DARPA simply placed too much faith in the foresight of the AI research community when it came to near-term hardware developments, then this is surely a lesson to learn from. 

Taking a step back and comparing the similar MOSIS program to Machine Acquisition, it does seem that the MOSIS budget had less downside risk in spite of also being a DARPA program dedicated to investing in subsidized computing hardware for researchers. The MOSIS budget was spent on infrastructure that was less likely to be rendered obsolete by technological advances elsewhere in the SC portfolio or industry. In addition, MOSIS also put itself in the (less risky) position of broker, rather than the (more risky) position of large equipment purchaser. The Machine Acquisition Program was on the hook for a lot of obsolescent, big-ticket hardware that cost $50,000 to $100,000 per unit. MOSIS, on the other hand, was simply paying others for runs on their big-ticket hardware. Given this difference in capital ownership, the MOSIS program was better structured to adapt to unforeseen developments such as those that befell the Machine Acquisition Program. The ISI team, surely, would have been forced to scramble in response to new research developments showing that VLSI standards or λ proportions were no longer useful and the future looked different than they expected. But it does not seem that they would have been in as much trouble as the Machine Acquisition Program if they guessed wrong. 

MOSIS had its risks, but it did not share with the Machine Acquisition Program the additional risk of government-purchased big-ticket research equipment sitting in the corner of a lab and serving minimal purpose to the research movement. For MOSIS, if certain specs proved obsolete, certain manufacturers might be similarly hit hard, but MOSIS could simply switch runs to those who were better suited. Of course, MOSIS could only make this switch once it re-wrote some of its software and re-designed some of its services. And while that is inconvenient, this sort of reworking, such as re-writing software or adjusting the Mead-Conway standards,  was already built into the organization’s processes. Minor changes of this sort were routine. It is likely that MOSIS could, within reason, likely do whatever internal research work was required to adapt their service to the new world and continue on — shaken but in decent standing. 

In summary, the Machine Acquisition Program, in comparison to MOSIS, both incurred more of a financial burden in terms of expensive machine ownership while also being more liable to having those machines rendered obsolete by other investments in SC’s portfolio. The program could still be considered a worthwhile investment if program managers believed one of the following:

• It was important to get the AI research community working on DARPA applications projects sooner rather than later. This was likely at least partially a factor since DARPA Director Robert Cooper is said to have insisted that SC begin work on applications programs concurrently with the programs developing the technology base.

• There was a chance that very useful near-term applications and learnings would arise from the AI researcher community’s current lines of work with LISP machines — even if they did go obsolete quite quickly.

• The PMs considered the money spent and the known risks incurred a small price to pay to seed early work in an interesting research area.

Assessing the project’s execution, it is unclear if the researchers — who were allowed to purchase whatever equipment they wanted — were effectively incentivized and informed enough on the state of hardware research to properly make purchases with DARPA’s money. From one POV, the researchers are the experts and should have final say on what equipment is best to carry out their own research. But, on the other hand, all of the financial downside — and incentives for making prudent decisions like waiting three or so years before spinning up a program like this in the AI community — lied with DARPA. Some could accuse those in charge of the program of not properly weighing SC’s own internal knowledge of the developing technological frontier in hardware research when purchasing millions in soon-to-be-obsolete machines. In retrospect, Ohlander and DARPA could have weighed their own knowledge of the evolving microelectronics landscape more heavily than they did when making decisions on this program. 

But it is probably not worth dwelling on what could have been. PM work is often about making “big if true” bets and being willing to incur the losses as long as the wins are big wins. But it is still helpful to know about and learn from DARPA bets that did not pan out. And Machine Acquisition is, to most, a bet that did not pan out. To be sure, the Machine Acquisition Program should probably be remembered for the money it saved as well as the money it had a part in spending fruitlessly.

The Machine Acquisition Program had similar goals and (somewhat) similar strategies to wildly successful DARPA projects like MOSIS (with its goal of reducing chip costs with shared fabrication runs), the ARPAnet (with its early goal of using the network to share research machines and reduce costs), and DARPA’s computer time sharing program (which enabled more researchers to use a single machine). In the end, the Machine Acquisition Program’s outcomes were simply a victim of the risks that Ohlander and SC leadership (hopefully knowingly) took when they acquired infrastructure for researchers in a field whose equipment was rapidly evolving. 

While the rising tide of computer chip development did raise all boats in the computer research field, the tide did not have the same positive effect on all existing projects within the SC portfolio. The Machine Acquisition Program provides a useful case study for those making investments that build on top of a rapidly evolving field of technology.

Thanks for reading! Subscribe if you’d like to continue following along with the series.

Subscribe now

Share

• Chapter 4 of Strategic Computing: How DARPA Built the Computer Age

  • Particularly useful if the reader wants to understand how Ohlander attempted to subtly nudge researchers away from inordinately selecting Symbolics machines. Symbolics, in the beginning, was the only vendor that had a track record but also was the only vendor that did not offer ISI a steep discount.

  • Also contains further details about the program’s early history and various operational details.

• ISI’s Annual Reports

  • Section 16 of the 1985 Annual Report outlines the early goals and work of the program. Additionally, this section briefly outlines some areas of military application ISI believed the research area could eventually contribute to.

  • Summer 1983 - Summer 1984 Annual Technical Report contains brief descriptions of ISI’s first year of acquiring machines and operating the program.

• Strategic Computing: How DARPA Built the Computer Ag

This piece is a part of a FreakTakes series. The goal is to put together a series of administrative histories on specific DARPA projects just as I have done for many industrial R&D labs and other research orgs on FreakTakes. The goal — once I have covered ~20-30 projects — is to put together a larger ‘ARPA Playbook’ which helps individuals such as PMs in ARPA-like orgs navigate the growing catalog of pieces in a way that helps them find what they need to make the best decisions possible. In service of that, I am including in each post a bulleted list of ‘pattern language tags’ that encompass some categories of DARPA project strategies that describe the approaches contained in the piece — which will later be used to organize the ARPA Playbook document. These tags and the piece itself should all be considered in draft form until around the Spring of 2024. In the meantime, please feel free to reach out to me on Twitter or email (egillia3 | at | alumni | dot | stanford | dot | edu) to recommend additions/changes to the tags or the pieces. Also, if you have any ideas for projects from ARPA history — good, bad, or complicated — that would be interesting for me to dive into, please feel free to share them!

Pattern Language Tags:

• Pushing forward the technological base via speculative machine building contracts

• Gray coding

• Incentivizing a group of related goals by organizing them all into a single contract

• Selecting multiple industry contractors for a trial period before choosing the final contractor

• Funding a conservative project to use as a measuring stick for more ambitious projects in the area

DARPA’s multi-decade investment in developing early versions of parallel processing computers provides a lot to learn from for ARPA-style PMs. Two of DARPA’s most prominent, long-standing investments in this area took two completely different approaches to push the technology base forward in a coordinated fashion. Both are viewed as successes to some and abject failures to others. In this piece, I will dive into the operational structures that informed DARPA’s investments both in the ILLIAC IV parallel processing computer in the late 1960s — the project was based out of the University of Illinois under a researcher named Daniel Slotnick — and in the young Thinking Machines Company in the 1980s and their Connection Machine computer — which brought them fame and infamy in the computing world in the years in which DARPA was most heavily involved with the company. 

To start, I’ll dive into DARPA’s investment in the ILIAC IV. The cost overruns and time delays for which this project became internally famous very heavily informed the approach that Stephen Squires took to managing DARPA’s investments in Thinking Machines and his computer architectures portfolio as a whole in the 1980s.

When DARPA Director Charles Herzfeld took over in the mid-1960s, computing research was not nearly as pressing of a defense research issue as others like ballistic missile defense or nuclear test detection. The previous director, Jack Ruina, often only met with the then-IPTO Director J.C.R. Licklider once or twice a month. Now, that is in large part because he both trusted Licklider and attempted to have a hands-off management style. However, it also speaks to the lack of priority that the Information Processing Techniques Office (IPTO) had within the DARPA portfolio at the time. PMs in hotter areas of the portfolio were said to have quickly met with Ruina as many as five times a day to discuss small issues. 

As the reader surely knows, great work was done by Licklider and his portfolio of performers in the early 1960s with this approach. As Herzfeld took over, the progression of the field was apparent to him and he thought it had the potential to help improve programs all across DARPA and the DoD. He recounted his approach to managing the IPTO as he took over, saying:

I thought that my job vis-a-vis the computing program, when I was Deputy Director and Director, was first of all to make sure to get the best office directors we could get, and second, to help them do what they thought they needed to do, and third, to look for applications across the projects for computing, when you apply the new computing ideas, technology and its capabilities as widely as possible.

Ivan Sutherland — Licklider’s young, hand-picked replacement who had gotten his Ph.D. at MIT and had been at CMU for several years since graduating — became IPTO’s Director under Herzfeld. Sutherland, while young, had been making a name for himself in his field. His Sketchpad program, written for his MIT thesis, would later help win him a Turing Award.

Herzfeld also lived up to his word of helping good ideas from this office find funding. One example of this was when, in 1965, Robert Taylor brought forward the network proposal to Herzfeld that would become the ARPANET. Herzfeld, thinking the idea had promise, quickly determined that ARPA should fund the idea and took some money that had been set aside from a lower-priority program — almost $500,000 (~$5 million today) — to get started immediately. They would get funds from Congress to continue the program in the following fiscal year. This kind of administrative move, while not rare in managing the DARPA budget at the time, was more likely to be done for higher-priority projects.

As IPTO’s technology base was becoming more mature, its budget began to increase and PMs were looking to fund projects that were further along in the development process — IPTO’s early-1960s budget had been almost entirely allocated to early-stage research.1 One development area of interest was the idea of building a more efficient computer leveraging hardware that was more parallel in nature. If implemented successfully, it was thought that the machine could help the DoD solve many pressing problems that required more computing power — such as calculating nuclear weapons effects or real-time processing of radar data.

In 1965, Sutherland reached out to Daniel Slotnick a month after Slotnick had arrived at the University of Illinois to set up a visit at Slotnick’s new office in Urbana-Champaign. Slotnick had become acquainted with Sutherland while he was working on his SOLOMON parallel computing machine at Westinghouse; this area of work at Westinghouse had many DoD ties. In their meeting, Sutherland would ask Slotnick if he was interested in developing a large parallel computer at Illinois. 

At least, that is how the events unfolded according to Daniel Slotnick’s IEEE memoir. Sutherland, in an oral history conducted by William Aspray, seems to remember things differently. According to his memory, Slotnick came into his office while he was still at Westinghouse and funding for his project was drying up, and an interaction resembling the following ensued:

He came into the office and said," Basically, I want to do these things and I cannot do it at Westinghouse, because they will not let me do it. Where should I go?" I introduced him to some of the university folk. So that one was fairly a long time in the doing.

This, also, is a quite believable ordering of events given my understanding of how things worked at IPTO in the 1960s. Whether Slotnick’s account or Sutherland’s is more accurate does not matter much. Regardless, in 1965 Slotnick and Sutherland had a conversation about building a parallel computing machine — which they had clearly discussed before — and Sutherland let Slotnick know that this might be a good time to submit a proposal now that he was at the University of Illinois. When the proposal was submitted, other IPTO staff as well as Herzfeld supported funding the proposal.2 In the next few paragraphs describing the details of how the finer points of the contract materialized, I’ll rely on Slotnick’s more clear account of the events. Certain small details might differ between the two accounts, but none that should matter much for the takeaways of this piece.

The project would be quite similar to Slotnick’s work at Westinghouse, and it was an area of obvious interest to IPTO’s mission. After Slotnick thought it over, he agreed. Sutherland initially proposed starting the project with a small study phase. But this tentative approach was not agreeable to Slotnick. Arriving at Illinois, the disappointment of the SOLOMON project was fresh on Slotnick’s mind. Slotnick’s intensive work on the SOLOMON project jointly funded by the DoD and Westinghouse for two years, Slotnick’s Westinghouse team steadily grew to around 100 people. In spite of the team’s promising early work on the technology, the project’s steady progress came to an abrupt halt when the project’s main sponsor in the DoD suffered a tragic drowning accident. The man’s replacement did not back the project in the same way. Quickly, the project went from pushing for the funds to build a full-scale version of the machine to having no funding at all. Not long after, Slotnick found himself at Illinois.

With this disappointment fresh on his mind, Slotnick said he was “anxious to do something else.” He only wished to pursue the project with DARPA sponsorship if Sutherland meant business.  Slotnick described his reaction to Sutherland’s proposal of a modest seed funding stage as insistent:

I absolutely refused. I wanted a $2 million payment at the outset and a contract for a total of $10 million. I did this to make sure that the ARPA commitment was real and had passed the highest levels of review.” Sutherland must have understood. Slotnick continued,  “Ivan agreed. I wrote the proposal and a few weeks later we had our contract.”

The proposed machine was heavily inspired by Slotnik’s time spent developing the SOLOMON prototype machines. The newly proposed machine — the ILLIAC IV, named after three earlier computers built at Illinois — sought to achieve over one billion operations per second. Beyond the project’s aim to produce a machine with a substantial improvement in computing power, its management sought to roll out improvements to much of the component technology in the machine. When all was said and done, Slotnick and the DARPA team involved believed the machine would have several useful areas for military application. DARPA and the armed services research organizations planned to deploy the resultant machine to help advance research problems in climate dynamics, anti-submarine warfare, ballistic missile defense, and the army’s multiple target problem. Additionally, a nuclear research office was interested in the capability of the computer to help solve the climate modification problem. Lastly, IPTO itself was particularly interested in the computer’s applications to researchers in cryptanalysis, simulation of fluid flow, seismic array processing, economics, and other problems ideally structured to utilize the computer’s parallel processing capabilities.

Sutherland — as the IPTO Director — was very explicitly attempting to use the ILLIAC IV project to push the boundaries of several component fields of computing simultaneously. As later-DARPA Director Stephen Lukasik described the ILLIAC IV team’s full frontal assault approach, IPTO and DARPA were “pushing machine architecture, software for parallel machines…medium scale integrated circuits — and then the application of that to a variety of important defense problems.”

In addition to this project helping push forward the technology base in the area of parallel computing, Sutherland hoped that it could help spark some efforts from industry in building new computer components relevant to this general area of technology. So, while the overarching contract for the project was issued to the University of Illinois team under Slotnick, the work of an industry subcontractor to build the machines the Illinois team designed would also be vital to the project’s success. In addition, contracts for the research and engineering development of various components of the machine would be given out to other parties from the IPTO performer pool. However, the work of these additional parties would generally be less important to the project’s success than the work of the Illinois team and the computer manufacturer chosen to produce the final machine.

The Illinois team’s initial design specified a machine with four modules of 64 processors under the control of a single construction unit which would allow them to simultaneously work on the same problem. As Slotnick began to assemble a team at Illinois, the team also began to figure out which computer manufacturer to partner with. Slotnick was not upset by the project’s partnership structure. From the beginning, he felt this was a natural approach to doing a project like the ILLIAC IV. Slotnick had come from industry and understood the comparative advantages of industry and academia when it came to designing and building new computers at that time. As Slotnick put it, “The days when a university could design and fabricate a big machine by itself were over, and we decided that while we would do the architectural design and most of the software and applications work, we would rely on industry for detail design and fabrication.”

Since selecting the right contractor to build the machine was pivotal to the success of the contract, Slotnick and the DARPA team held a competition. Slotnick and the DARPA team outlined the major approaches and incorporated them into a bid set. In August 1966, after “many months of intensive contacts with industry,” RCA, the Burroughs Corporation, and Univac were all awarded 8-month contracts to continue working on the early stages of the contract and collaborating with the Illinois team. After this period, in 1967, the Burroughs Corporation was selected as the best fit to continue on as a partner in building the ILLIAC IV. Burroughs had partnered with Texas Instruments in its bid. The plan was for TI to develop the integrated circuits for the machine’s all-important process element (PE) array. With that trial period over and the DARPA and the Illinois teams having gotten the chance to work with all three companies for a time, the industry partner for the project was decided upon.

Either Larry Roberts or Robert Taylor — Taylor had become the new IPTO Director in the intervening period and Roberts was the PM who focused on the ILLIAC IV project — and the Illinois team awarded the related contracts to work on other component parts of the ILLIAC IV to the following non-U of I teams: 

• The design and implementation of the FORTRAN compiler for the ILIAC IV to the Applied Data Research Corporation

• Various hardware and software to help link the ILLIAC IV and a trillion bit memory with the ARPANET to the Computer Corporation of America

• And work on the development of the interactive front end processor to BBN.

As the project progressed, the design and construction phase was riddled with research problems and setbacks that forced design changes on the research team, delays, and cost overruns. Two main issues contributed to the early cost overruns and time delays in the first couple of years of the project. The first was TI’s inability to produce the 64-pin emitter-coupled logic (ECL) packages. This caused a delay of over a year and a $4 million cost overrun. Slotnick documented that TI was not exactly forthright in taking the blame, writing, “A great deal of inconclusive finger-pointing went on between TI and its suppliers.” Regardless, the problem — and loss of cash and time — remained. The second major problem was Burroughs’ inability to make workable magnetic thin-film PE memories for the evolving design of the machine. This problem caused an additional year of delay and another $2 million cost overrun. Additional problems existed with packaging, circuit design, and interconnections for the large machine. But these two problems, in the early years of the project, caused the most headaches.

Slotnick, in his writings, explains how these very early setbacks began to affect project planning. Certain goals became more modest and the team had to substitute in and out certain component technologies to fit in better with the evolving design and changing circumstances. Slotnick writes: 

We retreated from the 64-pin packages to standard 16-pin dual in-line packages. In doing so, however, everything got bigger and more expensive. A lot of logic design was salvageable as a consequence of making the 16-pin packages logically derivative of the abandoned 64-pin packages, but board layout, back-panel wiring, and all system level hardware, of course, had to be restarted from scratch. The memory situation was even messier. Burroughs had a large investment in its thin films and didn’t want to give up on them even after my own independent review had concluded they still represented an intolerable development risk. Semiconductor manufacturers were just beginning to gear up for memory chip manufacturing; the manufacturing means were clearly at hand, or at least so it seemed to me, but the chips were not. I made the painful decision to drop films and go with semiconductors.

In making this semiconductor pivot, Slotnick believed the young Fairchild Semiconductor was easily the most qualified supplier. While Burroughs railed against the selection of the young company, a panel of independent ARPA experts — assembled due to Burroughs’ ‘furor’ at Slotnick’s selection of Fairchild — confirmed the decision. Fairchild seemed to be one of the only — if not the only — suppliers delivering high-speed semiconductor memory at the time. 

As the project wore on, the projected cost of the project roughly tripled and Slotnick was now only promising to deliver a system a fourth the size of the one originally promised. In spite of these shifts, Larry Roberts — who would become IPTO Director himself and who was already known as the “father of computer networks” — remained behind the project. Not only did Roberts, as Slotnick jokingly put it, share in his “sick attachment to really big pieces of hardware,” he was very interested in the list of applications which the machine was set to work on — including numerical weather prediction, sonar, radar, seismic signal processing, and an assortment of other computations array computers tended to do well. But, as Slotnick saw it, to justify the escalating costs, making the machine available to as large a community of users as possible became even more essential. The larger the scale of users the machine achieved, the more justified the escalating costs would be. Slotnick believed that this was a major factor that drove making the ILLIAC IV easily available over the ARPANET to be a much stronger point of emphasis for the project as the years wore on.

In 1970, five years into the constantly delayed project, the Illinois team was finally beginning to hone in on a workable machine. But, even then, there was still an element of instability in the project’s operations. From a technical standpoint, things were beginning to get on more sure footing. The PE had been successfully redone with the new TI integrated circuits, PE memory chips were being delivered by Fairchild, and other pieces of equipment were getting produced on schedule. However, cultural conflicts were now coming to a head on the Illinois campus. Not only were stresses between Slotnick and those faculty not involved with the million-dollar-a-month project being run out of an academic department coming to a head, but Vietnam War-era turmoil from the students was making it more difficult to peacefully continue the defense-funded project on campus. In the midst of all of these non-scientific issues, the project was first moved from within the academic department to become its own free-standing center at the university. Then, in January 1971, ARPA decided to move the system to the NASA Ames Research Center when it became feasible. The move was carried out by Burroughs in 1972 and may have happened earlier in the technological development process than DARPA would have preferred.

Hardware problems plagued the machine in its early years at Ames. Early circuit batches failed at high rates and in ways that were often difficult to detect. Additionally, many of the back-panel connections and terminating transistors functioned poorly. While successful runs were made as early as 1973, the team at Ames did not have the machine working in a way that could be considered reliable until 1975. However, even after the machine was completed, one problem continued to cause severe headaches: producing software to run on the machine. This would be an area of lasting difficulty that plagued the machine for its entire lifetime — which was admittedly not so long. The difficulty in writing software for the machine would be remembered when the next generation of IPTO leaders sought to push the area of parallel computing to the next level.

The ILLIAC IV would only be used until about 1981 when NASA replaced the machine with a more reliable and easy-to-use successor

By the time ILLIAC IV was finally constructed, the total cost was $31 million (over $200 million today) — substantially higher than the original, expected price tag of $8 million. Beyond the cost problem, the final machine only had one quadrant of 64 processors rather than the originally planned four. Several of the many experimental components of the machine could not be made to work and existing components had to be used in their place. The resulting machine only achieved one-twentieth of the computing power that the researchers originally sought to achieve — the initial goal was one billion operations per second. Also, even though the ILLIAC IV was considered finished in 1972, much of the 1972 to 1975 period was spent in an intensive effort to correct problems and improve the reliability of the machine as researchers affiliated with the armed services were attempting to use the machine for practical applications. 

Notwithstanding the setbacks, once the ILLIAC IV became operational in 1975, it was thought to be the fastest machine in the world for certain types of problems — calculations of fluid flow being one example. In the late 1970s, the computer was used in service of many of the armed services and DARPA applications on which it was intended to be used. This includes the army’s multiple target problem, research in cryptanalysis, seismic array processing, economics, and more. For the few years in which the machine was in operation, it may have been the best computer in the world for solving problems whose calculations could be performed in an “all at once” manner and were, thus, ideally suited to being solved by a parallel processing computer. 

From the standpoint of developing the technology base, the component technology of the computer helped improve many pieces of the technology base. Some of the prominent examples:

• The first large-scale computer to employ emitter-coupled logic in its processors — taking the place of transistor-transistor logic. This technology would go on to be used in many high-speed computers in subsequent years. 

• Its circuit boards were the first to be designed with the aid of a computer. Computer design automation is now widely used in industry.

• The design of its storage technology — which consisted of 64 disks that could be read from and written to concurrently — led to higher speed input/output.

• The machine’s laser memory system as an additional, tertiary memory had a capacity in the trillion-bit range and read-in and read-out rates in the million bits per second range.

• ILLIAC IV’s architecture employed a single instruction stream to control the multiple data streams involved in interprocessor communication.

• ILLIAC IV was the first large system to employ a semiconductor primary store.

On top of all of the new functionality, the program also succeeded in (eventually) producing a machine built by a joint industry-university collaboration. Slotnick and the Illinois team did not believe they’d have been better off working on the project with no help from industry. In fact, Slotnick later reflected that he believed running such an operation with him and the university as primary contractor could have been a mistake — or something to learn from at the very least. He referred to trying to administratively manage such a technically complex operation with that sort of headcount, scale, budget, and operational needs as having “all the sense of trying to build  a battleship in a bathtub.”

Joining all of these non-obvious developments into a project meant to produce a single, useful machine caused substantial time delays and cost escalation. But the field of computing got more than nothing in return for the tradeoff. The ambitious design of the machine created a technology “pull” and stimulated the development of new component technologies that had the potential to find broad use. 

For example, when the project ran into issues incorporating the originally planned film memories into the design, Slotnick turned to the nascent Fairchild Semiconductors to design the project’s all-important memories to fit with its new design. The new design was seen to be quite risky since it required advances in semiconductor art, engineering design, and production. But Slotnick was convinced the Fairchild team could do the work. This confidence may have come from the fact that Slotnick’s old mentor, Rex Rice, headed memory system operations at Fairchild. It also may have been due to Fairchild’s rapidly growing reputation as the new provider of semiconductors to the NASA program and certain Air Force programs. Regardless, in the end, Slotnick was right to put his faith in the Fairchild team because they successfully produced the required memory chips and the ILLIAC IV was the first to use chips of this design. Some believe that this project helped speed up the pace with which semiconductor memories became commercially available — although given the field was growing so quickly, it is hard to tell if such a claim is true. 

Taking a step back, it is reasonable to question whether or not the ILLIAC IV’s approach is the ideal strategy to orchestrate new technological developments. As always, the suitability of the approach depends on the situation and what the PM thinks is appropriate given the context. Developing so many new pieces of component technology to use, for the first time, in one machine seems to leave one obviously more prone to cost overruns and time delays than most projects. The upside is that the approach provides new funding and a focused direction to help bring what could become very useful component technology into existence. In the best case, the approach can push a field forward much faster than if one had taken a more measured approach. But the approach does also run the substantial risk of generating demand for and funding pieces of component technology that might be useful to the current project, but will not be useful to the field as a whole in later years. For example, if the components required by the design of the machine are not reflective of the needs of the field as it progressed outside of that individual project.

In essence, the  approach of the ILLIAC IV team should be considered a high-risk, high-reward strategy. It probably won’t achieve outcomes as stable as those of the gray coding approach covered in the following Connection Machine sections, but there are still situations where the full frontal assault approach of the ILLIAC IV has a high enough expected value to justify the risks. With this approach, one may run the risk of cost overruns, delays in producing a working machine, and developing components that the broader field turns out to not need. But the approach also might help a technological area make discontinuous leaps in component technology and overall system performance that would not happen by changing one component at a time. Particularly in the early days of a technological field, changing multiple design components at once is often an attractive approach given that existing approaches might be extremely far from what researchers know to be theoretically possible.3 (See this earlier FreakTakes piece on the field of early molecular biology to read more about this strategy.)

Depending on the political and technical context of the project, the full frontal assault approach might be just what is required. But, in other situations, it might be considered far too risky. Stephen Squires, with his computer architectures program in the more mature DARPA computing portfolio of the 1980s, considered the approach far too risky.

While the final ILLIAC IV machine did work, Squires took the management of the program as a cautionary tale of what not to do when developing machines within IPTO. In his eyes, the cost overruns and delays in producing the machine were entirely foreseeable given the structure of the project. Squires believed the “full frontal assault” approach of the ILLIAC IV project — with novel approaches to the machine’s architecture, processors, memory, storage, and other components — pushed too many untried technologies at once. Problems in developing or implementing any one of the components could negatively impact the entire project. In the uncertain enterprise that is research, Squires believed this was an untenable risk profile. 

Squires wanted to encourage projects funded through his architectures program to limit the number of new technologies a project worked on at a time — usually to one. He called the approach “gray coding.” Roland and Shiman — who wrote the most extensive history on DARPA’s Strategic Computing Initiative of the 1980s and extensively interviewed Squires — expanded on the thinking behind the approach as follows:

That way, problems could be easily traced, isolated, and corrected, and the chances of catastrophic failure of the system would be minimized if not eliminated. Advance of the field would not occur in one fell swoop by ‘frontal assault’ of some ‘point solution,’ but through the selected development and installation of key technologies, all well-timed and well-coordinated. Advance would occur over a series of generations. Once a technology or key concept had proven reliable, a new component or design would be implemented and tested.

Squires portion of DARPA would still pursue several technological trajectories at once — which Squires called the ensemble model —  to allow the research ecosystem to prove out which one held the most promise. 

Additionally, learning from the ILLIAC IV example and his own experience working on systems software at the NSA, Squires insisted that software development be emphasized as much as hardware work for the architectures being developed. In the case of the ILLIAC IV, the development largely focused on getting the hardware working and later shifted efforts towards generating useful applications via software development. Many believe the ILLIAC IV team emphasizing the hardware first and software later was a large part of what took the machines so long to finally get working at the back half of the project. Squires was going to do things differently.

Squires planned for the architectures program to progress in three phases — each lasting roughly three years. The projects funded in the first phase would generally seek to refine promising architecture ideas and develop microelectronics components that would later be used in building parallel machines. Projects funded in the second phase would generally seek to build full prototypes of the most promising architectures. The projects funded in the third phase would integrate technology that now (hopefully) existed into composite systems that could support DARPA–relevant applications — such as military applications in computer vision, natural language understanding, simulations, and more. Squires hoped that the machines being developed would achieve 100 million floating point operations per second by 1987, a billion by 1989, and a trillion by 1992. In service of this goal, all of the early work on smaller-scale prototypes had to be easily scalable — for example, being designed in such a way that they could be easily expanded by simply increasing the number or power of processors in the prototype. As all of this work progressed, work on new programming languages, compilers, tools, and concepts that better served parallel programming would be also funded.

With the high level plan in place, Squires began by traveling around and further familiarizing himself with the pool of potential performers that might be funded and meeting with those in industry — such as computer manufacturers. In addition, this familiarizing process also entailed a February 1984 call for qualified sources asking for potential contractors to explain what they could do in the areas of signal, symbolic, or multifunction processing if they were funded by Squires’ architectures program.

Squires, at the time, primarily issued the formal solicitation for informational purposes — as a way to see what was out there, not necessarily planning to immediately award any contracts to the proposals submitted. But one proposal jumped off the page from the pool of submissions — two-thirds of which from industry and one-third from academia. That proposal was a big black notebook sent in by a newly formed company called Thinking Machines Corporation (TMC) outlining their technical and business plans for a computer called the Connection Machine.

The proposed machine was exactly what Squires was looking for. The machine the company proposed was an ambitious, truly parallel hardware architecture that seemed like it could scale up by simply adding more or better processors. In spite of the ambitious design, Squires still saw the approach as conservative — in a positive way. Not only had the Thinking Machines Corporation (TMC) already developed a workable VLSI design for the processor chips — which was one notable source of technological risk already mitigated — but the company also set out to construct the architecture using as many commonly used components as possible. In fact, in several areas they went as far as to use components that were far simpler than the ones commonly used in other machines at the time. One example is that, for its first machine, TMC planned to use the simplest 1-bit processors instead of the more powerful 4, 8, or 16-bit processors commonly in use at the time. In addition, the Connection Machine would not even be a stand-alone machine in the early iterations. The front end used to access the machine would be a more commonly used computer — a LISP machine in this case. In this early stage, SC’s main goal was to demonstrate the feasibility of basic parallel processing concepts while not adding needless complications to projects that did not serve the main goal. TMC — maybe not surprisingly since its founder came from a lab that was quite used to working with DARPA PMs — clearly got the memo and submitted a proposal that perfectly complied with the gray coding approach while maintaining an ambitious vision.

After some meetings between DARPA staff and TMC executives, it seemed clear to DARPA that TMC as a venture was well thought out from both the business and technical perspectives. In early 1984, $3 million (~$9 million today) was approved to fund TMC’s work on a small prototype of the machine containing 16k processors — with the option for an additional $1.65 million to scale up the prototype to 64k processors. Raising an additional $16 million in private investments around this time from private investors, TMC was well-capitalized and ready to begin work on its first machine.

Even having laid out a plan of action that complied with Squires’ gray coding framework, the work was daunting for the company. Danny Hillis, the CEO of TMC who came up with the idea for the Connection Machine during his Ph.D. under Marvin Minksy at MIT, said the early work building the simplified version of the Connection Machine underway in 1984-1985 was “overwhelming.” He continued, describing the early work of the company in a 1989 Physics Today article:

We had to design our own silicon integrated circuits, with processors and a router. We also had to invent packaging and cooling mechanisms, write compilers and assemblers, devise ways of testing processors simultaneously, and so on. Even simple problems like wiring the boards together took on a whole new meaning when working with tens of thousands of processors. In retrospect, if we had had any understanding of how complicated the project was going to be, we never would have started.

This reflection from Hillis probably lends some credibility to Squires’ gray coding approach having been a good fit for this particular project. The design and need to develop new software approaches for making use of the novel design would also bring with it more difficulties. There was no need to compound on these difficulties by ensuring the initial prototype had the most cutting-edge component tech possible. Novel engineering development projects like this were sure to run into plenty of issues that required novel solutions to overcome. As Hillis stated, they ran into so many of these in the early days that they might not have even undertaken the project had they understood how painful it would be. As the project grew more complicated, Hillis seemed thankful to have Manhattan Project alum Richard Feynman around as an intern for some summers — Feynman’s son Carl had worked with Hillis when Hillis was a grad student and Carl was an undergrad at MIT. Feynman seemed relatively influential in pushing the company to take a page out of the Manhattan Project’s management playbook and designate a group lead in each area of technology — such as software, packaging, electronics, etc. Feynman — who was initially brought on to help brainstorm applications for the computer — also pushed the company to hold regular seminars of invited speakers, often scientists, who the company believed might have interesting use cases for the machine.

While the project did have its difficulties in the early years, in spite of Hillis’s statement about the project being “overwhelming,” the project was considered the gem of the architectures portfolio in the early years of SC. Squires’ program had also invested in other prototype-stage projects for at least a two year trial stage — which all had some positive developments as well as some downsides. Each of the general-purpose, prototype-stage projects was meant to help in the architectures program’s larger goals of identifying promising approaches to parallel computing and providing experience working with the technology. Simultaneously, smaller projects called “accelerators” were funded with the goal of doing development work to produce components that could later improve the performance of the general-purpose machines. Besides the Connection Machine, some of the architectures program’s other general-purpose prototype projects included:

• BBN’s Butterfly computer series

  • The Butterfly computer had been in the DARPA funding pipeline since 1977, with funding going to one of DARPA’s workhouse computing research contractors, BBN. At the start of the architectures program, a model of the Butterfly with 10 processors already existed. BBN received funding to continue to scale the prototype to 128 processors. The butterfly computer was a coarse-grained shared-memory machine — coarse-grained machines have fewer but more powerful processors capable of running entire programs in a moderately parallel but not truly parallel fashion. This technology was both further along in development than smooth-grained parallel computers — like the Connection Machine — were. Additionally, machines like the Butterfly could easily make use of traditional programming methodologies rather than developing new ones. For those reasons, the Butterfly was meant to serve as the benchmark against which progress on other machines in the portfolio would be measured. 

• CMU’s Warp

  • The architectures program also contracted CMU to build Warp. Warp was a systolic array machine that sought to achieve more efficient performance using an approach in which processors are connected sequentially and data flows from one processor to the next with each performing a different operation — analogous to an assembly line. CMU had received this contract based on its prior demonstration of the systolic approach having built a programmable systolic chip.

• Columbia’s DADO

  • DADO was designed as a coarse-grained parallel machine. DADO was one of two tree-structured machines — in which a central processing unit was connected to two others, and each of those to two others, etc. — started at Columbia University in 1981. DARPA discontinued funding for this project after the early prototype stage.

• Columbia’s Non-Von

  • The second of Columbia’s two tree-structured machines, the Non-Von, was somewhere in between the DADO and the Connection Machine in terms of how fine-grained it was. A machine like this one would likely prove simpler to program than a Connection Machine, but would also not achieve quite the level of parallelization. The Non-Von sought to prove useful in managing and manipulating large databases. DARPA also discontinued funding for this project after the early prototype stage.

• TI’s compact LISP machine

  • TI sought DARPA funding to fund its development of a miniature version of a LISP machine. To do this, TI planned to implement LISP processors onto individual chips. TI hoped that the entire machine would fit on nine small printed circuit cards and, in the end, would be small enough to be embedded in computers of military systems such as within a pilot’s cockpit to help in expert decision making. DARPA agreed to the $6 million in hardware costs and TI, sharing the costs, funded the $6 million in software development.

But none of these projects seemed to carry as much promise as the Connection Machine and its model of performing identical operations simultaneously across its many, many processors. However, each of the other machines did often have at least one piece of component technology in development that DARPA was intrigued to pursue further. For example, the Non-Von project developed a way of using multiple disk heads in operating its storage system. TMC later incorporated this idea into one of its later prototypes — all SC-funded developments were eligible to be used by other SC contractors. This is just one case of discontinued portfolio projects still finding help push the technology base forward in Squires’ portfolio.

As the Connection Machine project got underway, Hillis had arranged for DARPA funding to install a LISP machine at the chip foundry TMC worked with in California. With that machine installed, TMC could test chips at the foundry rather than having them shipped to Massachusetts to test them there. By mid-May 1985, a little over a year after TMC had received its first DARPA grant, the 16k processor prototype had been finished — more than a month ahead of schedule. After DARPA checked out the machine in the TMC offices, DARPA immediately invoked the option for the 64k processor machine. This scaled-up prototype of the machine, too, would be finished ahead of schedule — by the end of 1985. Hillis’ design seemed to be scaling the way he and DARPA had hoped. And for all of the sub-problems that the initial design could not account for, the very talented (and growing) TMC research team — largely drawn from the staff and students of MIT, CMU, Yale, Stanford, and other elite pools of computer engineering development talent — was proving more than up to the task of solving. 

By the spring of 1986, the first Connection Machine — the CM1 — was being manufactured and was available for sale. The machine’s chips each contained 16 customized 1-bit VLSI processors. Thirty-two of these chips would be printed on a circuit board and mounted next to each other — eventually making up four separate quadrants of 16k processors. Each quadrant could stand alone as a cheaper, scaled-down model of the 64k processor machine. In spite of the novelty of the truly parallel design of the architecture, each of the components remained quite safe and reliable to use — as the gray coding approach intended. The 1-bit processors were basic and slow compared to the state-of-the-art processors at the time, but they also produced so little heat that the machine did not require much cooling. Installation of the machine on a buyer’s site was easy and could be readily set up with a LISP machine on-site — which customers knew how to use. However, customers did have to learn new, Connection Machines-workable, versions of C and LISP to make full use of the machine’s capabilities while programming. The user did not need to fully understand how memory assignment and other aspects of the machines worked with these languages, but learning the languages was a necessity.

With the project going well, in the beginning of 1986 DARPA funded TMC for an additional $3.8 million over two years to find ways to exploit the CM1 and CM2 on various scientific and military applications. While the Connection Machines were still in the process of becoming truly useful machines, this particular generation of DARPA’s computing programs carried a strong political emphasis on pushing successful research projects into application mode as early as possible. This iteration of DARPA funding for TMC included covering work on projects to further improve software development practices for the machine, develop the machine so it could be used as a network’s server, increase storage capabilities to aid in work on data-intensive problems, and developing a training program to help others learn to use the machine. Beyond this $3.8 million, DARPA would also serve as TMC’s biggest buyer in the coming years. 

With this DARPA funding, the work continued on into more mature prototyping and scaling stages. As of 1987, around twelve CM1s had been purchased from TMC — about half of them purchased by DARPA for its contractor community. The machines were typically purchased for around $5 million a piece in 1987 (~$14 million in 2023). As the TMC team sought new applications for its machines, under Hillis’ leadership, they eagerly sought out projects with many in the research community. This included partnering with physicists, astronomers, geologists, biologists, and chemists to understand the details of their applications. Someone like Feynman worked in application areas at TMC such as problems in database searches, geophysical modeling, protein folding, analyzing images, and simulated evolution. Establishing partnerships with researchers proved, unsurprisingly, quite natural to the organization. It should be remembered that, during his graduate work, Hillis designed the machine to be something like an optimal tool for researchers studying artificial intelligence — as well as other complex research problems. People like himself, his former lab mates in Minsky’s lab, and other researchers from similar environments were the kinds of people who staffed the company as well as its initial, intended customers.

The company did not prove nearly as eager to establish profitable partnerships with industry as it did partnerships with researchers. Of course, not all of the problems the company explored applying the machine to were research related. For example, Feynman also worked on one application area related to reading insurance forms. However, this sort of application work did seem to be a less natural fit for Hillis and the company.  Hillis, after all, saw the Connection Machine as a machine for science. In practice, TMC very clearly preferred not to make compromises to the goal of pushing science forward in search of increased revenue. For now, DARPA was TMC’s guardian angel when it came to buying machines, helping broker sales with research consumers of the machines that DARPA didn’t buy itself, and funding the company’s R&D work.

The CM1 — which TMC largely relied on DARPA to facilitate purchases for — did not only have limited market potential outside of the DARPA universe because of the price tag of its machines, but also, simply, because of what they just couldn’t do. From a consumer standpoint, the first major deficiency was that the machines could not yet run the standard computer language of the scientific community: FORTRAN. The second major deficiency was even though the CM1 could perform ten times as many computations per second as many contemporary, commercial supercomputers — 2.5 billion operations per second vs. a few hundred million — the 2.5 billion operations it could do per second were not floating-point operations. And floating-point operations were vital to most computationally intensive research problems at the time. Nevertheless, DARPA likely felt comfortable facilitating these purchases because much of the SC program emphasized folding other SC-developed component technologies into projects sooner rather than later. At the beginning of SC, DARPA had also facilitated the purchases of the young BBN Butterfly computer to some of its contractors in order to ensure they had the most up-to-date technology available to them and their development work. So, TMC was not the only contractor in the architectures portfolio to have received this sort of favorable treatment from DARPA.

With the deficiencies of the CM1 in mind, TMC got to work incorporating the clearly needed functionality into its CM2. Surely this lack of functionality, at this stage in the development process, was not considered a failure by any means. The first successful prototype — the CM1 — lacking much needed functionality from the user’s perspective was likely just a symptom of the gray coding approach. The project was likely unfolding as Squires hoped successful architectures projects would — in successive generations. By this time, the architectures program, headlined by TMC, was clearly proving that parallel computing could be very workable. From a technology development standpoint, that was no small thing, even if the tech was not the most powerful or user friendly yet. 

The CM2 was brought to market in April 1986 and could run FORTRAN as well as do floating-point operations. This version of the machine was much more usable to the scientific community and the machine's very novel, massively parallel architecture still required its customers to learn to use special software and new programming techniques. Using these programming languages proved difficult for even the machine’s quite savvy researcher-customers and far more trouble than it was worth for almost all commercial customers. An additional factor that made the machine more trouble than it was worth for many corporate customers was that the machine's rate of breakdowns and subsequent downtime, while not an issue to academic work schedules, was far more than corporate customers were comfortable with.

Some in the research community found useful applications for the new machines in areas such as fluid flow dynamics and large database searches. Yet, in spite of these successes, the machines were not yet considered very usable to their potential customers. Years later, Dave Waltz — the head of TMC’s AI group — recounted that most of the CM2’s early customers were not using the machine correctly. In his eyes, TMC had built a machine that worked far more like a human brain than a sequential computer like the Cray — the most successful commercial supercomputer company at the time — but, he noted, that people actually knew how to write programs for a Cray! Many of the customers were running  their computations in such a way that they used the floating point processors in the CM2s in a standard way and largely ignored the machine’s novel, 64,000 single-bit processors. But, with the technology continually progressing, DARPA continued to help TMC facilitate sales. As one of TMC’s research directors later recounted,  "Our charter wasn't to look at a machine and figure out the commercial profit. Our charter was to build an interesting machine."

As the Cold War waned and the acute need for military supercomputing and near-term AI applications lessened, the Bush administration and the president’s main science advisor — nuclear physicist Allan Bromley — turned its eye towards helping solve “grand science challenges.4” These challenges included problems related to climate modeling, analyzing protein folding patterns, mapping the human genome, earthquake prediction, and learning more about the underpinnings of quantum mechanics. Bromley seemed to believe that these problems required enormous computing power rather than artificial intelligence. From the modern perspective, smart people could argue whether or not Bromley was right in that idea. But, surely, massive increases in computing power paved the way for specific solutions in those problem domains — even if novel AI methods were what pushed some of those things across the finish line. Regardless, that general ethos was what drove the government’s new High Performance Computing and Communications (HPCC) program. DARPA was the lead agency in this work and received an additional budget of several billion dollars through 1996 for the program. One of the primary goals of the new program was a computer that could achieve one trillion computations per second.

Naturally, DARPA turned to TMC and gave the company some of this funding to help the company expand on its work that was so successful thus far. DARPA granted the company a $12 million initial contract to produce a scaled-down machine that could theoretically hit the trillion computations per second benchmark by 1992. It was in this period when TMC was working on the CM5 — TMC CEO Sheryl Handler skipped “CM3” and “CM4” in the hopes that the naming curveball would make espionage harder for would-be infiltrators — that the music stopped for the young technology darling.

In this period, as TMC was hitting its objectives and in a great place from a technology development standpoint, they began to spend as if they were having equivalent levels of commercial success — which they were not. Company executive Sheryl Handler had been brought in to co-found the company to bring a level of business maturity to the operation, but that proved to be a mistake. Her corporate spending habits were, putting it charitably, a bit decadent. This habit showed itself early on in the company’s lifespan and continued to worsen as the 1980s wore on. As early as 1984, the very young TMC moved into the top two floors of the expensive Carter Ink building and began spending money on modern, luxurious tech company-like amenities. A couple of these expenses included shelling out for a custom design plan and a gourmet chef. In 1989, as TMC’s work on the CM5 was getting underway, Handler signed a whopping 10-year lease with the Carter Ink building which cost the company $6 million a year — the $37 per square foot for the lease was reportedly 4.5 times higher than Lotus Development Corp. was paying just down the road. Additionally, management rapidly increased headcount by around 40% around this time — resulting in a staff of around 400. The company’s spending was more in line with the company’s growing status in the computing world than its actual financial standing. As Stephen Wolfram described the company’s status at the time, the company was “the place that foreign trade delegations would come to visit to see where American business was at these days.” An IBM computer scientist put it a little differently when he described the company as having cornered the market “on sex appeal in high-performance computing.”

While TMC was in the process of developing machines that were on their way to doing things that no other computer could seemingly do at the time, they were not producing machines that the market wanted as measured in total dollars of actual market demand. With the launch of the government’s HPCC initiative, the company’s research emphasis shifted even more strongly into building the most powerful computer possible by 1992. As it did this, the company continued to put off making the developments it needed to make to find some level of product market fit. The company had produced a machine that cost millions, but was almost exclusively sourcing its customers from the research community. The number of customers and the budget per customer in this community was far more limited than it was in the corporate sphere.

As work on the CM2 and later CM5 progressed, it does seem that the company had at least some opportunity to slowly ramp up efforts in the commercial sector. For example, in the late 1980s TMC sold two Connection Machines to American Express as database mining for improved customer analytics was a growing trend at the time. There was some internal debate about whether or not to start a business supercomputing group at the company — which would be an obvious “yes” for most young and growing companies with shareholders — but nothing much came of it. The company, in practice, clearly preferred to serve the research vision that drove its founder and employees to pursue computing over maximizing shareholder value.

As the work got underway on the CM5, progress was not as smooth as it had been on the previous two generations of machines. The first CM5 would not be completed until October 1991 — and even then it was not quite as powerful as promised because there was a delay in producing chips that fit the machine’s design. However, throughout this 1988 to 1991 period, TMC was peaking from a revenue standpoint. In 1989, thanks to the company’s good reputation within DARPA’s walls, the company achieved its first profit — $700,000 profit on revenues of $45 million. Then, in 1990 the company saw profits of $1 million on revenues of $65 million. This would be the peak of TMC’s profits.

Sadly, for TMC and the world of early 1990s parallel computing, that was as good as things would ever get for the firm. The company’s introduction of an early version of the CM5 in late 1991 was lauded by Hillis as possessing the highest theoretical peak performance if you added enough processors to it — but that particular version of the machine was less powerful than the CM2. It’s very possible that with more DARPA funding and further work upgrading the prototype, this branch of the technology base would have continued to progress from within the expensive walls of the Carter Ink building, but that never came to be.

In August 1991, in the midst of the HPCC initiative’s leaders beginning to make plans for how to deploy most of its sizable budget moving forward, the Wall Street Journal released a piece diving into DARPA’s subsidies of at least two dozen Connection Machines in addition to the machines it had bought. While this in and of itself was not necessarily illegal or even nefarious, this particular political era was one in which anything that looked like “industrial policy” or government interference in markets was extremely scrutinized. DARPA buying its contractor pool BBN Butterfly machines and some machines that resulted from CMU’s Warp project does not seem to have received negative press in the same way. This disparity in PR outcomes might be because buying machines that might soon become obsolete due to technology in development within your portfolio did not come off as bad as subsidizing machines that were not yet considered to be truly finished, market-competitive products. 

Regardless of why the negative press came to be, the Bush administration made a point of ensuring that “industrial policy”-like help to TMC did not continue, and the company did not fare so well when it was abruptly thrust out into the competition of the commercial market. In 1992, TMC reported a $17 million loss. Massive layoffs and spending cuts began around this time. A new CEO was brought on who attempted to broker some kind of merger or partnership deal with Sun Microsystems and IBM, but TMC’s balance sheet was more toxic than its tech was impressive. To give the reader an idea: TMC owed $36 million in rent to the Carter Ink Building over the next six years for its ill-advised ten-year lease. And that was just one of the apparently many dark spots on its balance sheet. A deal like the one the new CEO sought could surely have been brokered around 1989 or 1990, when the company still seemed several years ahead of the field in the rapidly developing field of parallel computation and was beginning to show profits. However, the company had rejected offers for mergers and acquisitions at that time.

TMC filed for Chapter 11 bankruptcy in late 1993 and re-emerged from this as a small software company attempting to sell programs to run on their former competitors' parallel computers.

Thinking Machines had been capable of hanging onto the dream of building a machine that helped raise all boats in the research community when it was fueled by DARPA funding. With DARPA funding, it was able to largely ignore the commercial market and commit itself to pushing the technology base forward with DARPA’s relatively rare form of risk-tolerant, farther-sighted capital. And they were doing a good job of meeting DARPA’s needs. However, TMC was spending its privately raised capital as if they were doing an equally good job courting commercial customers as they were hitting computing benchmarks. And that was absolutely not the case. Of the 125 machines the company sold in its entire history, only ten percent were to commercial customers. So, when the DARPA relationship took a turn, the company inevitably had to file for Chapter 11 bankruptcy after some rushed attempts at solvency.

The company surely succeeded in its primary goal: producing a machine that proved truly parallel machines could work. Also, the firm did this while developing its technology, step-by-step, in a way that fit with DARPA’s preferred approach to moving the architectures program’s technology base along at the time: gray coding. From a technological standpoint, for the first five or six years it seems the company did an effective job of moving technological progress forward — in the form of a machine with steadily increasing functionality — on a consistent timeline that DARPA was very happy about. From a commercial perspective, however, the company was clearly a failure. Hillis himself admitted as much. At the time of the company’s bankruptcy, Hillis was asked how he felt. He replied, “Sad, that the people who put so much in — the employees and the investors — won't necessarily be the ones who benefit. And sad that I haven't been able to build a technology success into a business success.” DARPA surely got more out of the program for what it spent than the company’s investors did. 

In retrospect, DARPA’s general thinking on the problem area seems to have proven quite prescient. A major application area like image processing was exactly what they believed building a truly parallel processing machine like the Connection Machine could do. In the seminar series that Feynman encouraged TMC to put on, they even had a speaker from a not-so-popular field called neural networks come and talk. This story is surely proof that the DARPA ecosystem was, in many ways, functioning quite well — even if no commercial success came from TMC. Hillis and individuals at DARPA were in contact, talking about securing money for the idea contained in his Ph.D. thesis, at least as early as 1983. DARPA’s awareness of his idea and the young company almost from the moment of inception — as well as its recognition of Hillis’ ability to lead such an operation from a technical perspective — should surely be seen as a feather in DARPA’s cap. This awareness, of course, was possibly facilitated by Hillis’ prior connection with Minsky’s DARPA-funded lab at the heavily DARPA-funded MIT. But these kinds of interpersonal networks at key universities that were very good at certain sub-areas of research — such as MIT and CMU in AI research — were what DARPA PMs made it their business to maintain. That sort of relationship management, first and foremost, will always be one of the primary pieces of the puzzle that determines whether or not ARPA-like organizations succeed or fail.

A final note on TMC’s results: it is very possible that even a change as small as Hillis partnering with a much more responsible, business side co-founder than Handler — who proved adept at fundraising, but deficient in many key areas that TMC required — could have been enough to shepherd the company to a longer lifespan. From a commercial strategy standpoint, an individual like this may have helped TMC find commercial uses that fit in with the company’s goals and culture. From a financial standpoint, a better CEO might have ensured that the company either did not raise money from misaligned capital pools or did not spend the money it did raise as poorly as it did. 

One quote that encapsulates several of the PM-relevant lessons from the Thinking Machines Corporation’s work and death was said by Hillis as the company was filing for bankruptcy, “The real money is in handling Wal-Mart’s inventory rather than searching for the origins of the universe.” From the perspective of a pure venture capitalist, this quote might read as the complaint of an academic wishing that the financial markets rewarded what researchers found important. Of course, that is not the ideal perspective most venture capitalists would not want in their portfolio founders. But the quote can also be read as Hillis calling attention to what might be a very major market inefficiency. Research companies like Thinking Machines are great vehicles for building new, exploratory technology, but there is often a very salient tradeoff between a company like this moving the technology base forward and profit-maximizing for shareholders in the near term.

Let’s not forget, the ILLIAC IV project did not only run into difficulties because its team was attempting to build a machine with too many novel components at once. Slotnick’s “battleship in a bathtub” remark was explicitly describing how ill-suited modern universities had become to building big, new machines. Hillis did not make the identical mistake of trying to build the Connection Machine from within a university, but he still did make a mistake in choosing his organizational setup. TMC ran into the conundrum of raising money from private investors — who would naturally prefer the company do things like build a computer to handle Wal-Mart’s inventory — as well as a funder (and initial customer) in DARPA that primarily cared about building a computer to move technology forward. In most cases, a company is going to make one of its two (rather different) funders unhappy in a situation where their incentives are only partially aligned.

Had Thinking Machines only pursued DARPA funding and spent more frugally — as an entity like ISI, which ran DARPA’s MOSIS program, would have done — it is very possible the company would have continued to progress as the darling of DARPA’s architectures program for the foreseeable future. If the company would have taken this approach, it could still have raised money from private markets at some point, just later on in its existence when it had a product that was closer to market-ready and had a more natural customer. Additionally, the company could have attempted to find a variety of customers so it would not have all its eggs in the DARPA basket the way BBN (covered at length in a coming piece) was known to have done in the 1960s and 1970s. BBN, in its early existence, did not seem to be purely profit-maximizing. Rather, it sought a variety of contracts — from DARPA, private philanthropies, engineering industry, and more — that its researchers found new and useful, but that still paid enough to facilitate a rather profitable operation.

Of course, none of this alternative history brings any assurance of success either. The political winds may have blown against TMC one way or another regardless. It should be noted, though, that Hillis’ Wal-Mart quote was predictive in some sense. The company that did make possibly the biggest parallel computing breakthrough in the late 1990s — NVIDIA with its new GPU — found a way to fund its breakthrough research by servicing a market that required high-tech solutions and was surprisingly large given how unimportant it was in the grand scheme of things: gaming graphics. 

It is also important to note that, in the first five or six years of TMC’s existence, it seems that the gray coding approach was working just as Squires had hoped. TMC was often putting out versions of Connection Machines that did not have the functionality or usability that many users would have wanted. But that could be considered a marketing issue as much as anything else. In bringing new functionality to the machines in this steady way, this gradual decrease in user dissatisfaction seems par for the course. The rate of progress of the Connection Machine seems, generally, far preferable to the messy process that was the ILLIAC IV development. Of course, individual PMs will know best when either approach is most suitable to a given project or portfolio. A variety of factors can impact the decision and there is no one-size-fits-all solution. One factor that could impact a decision like this, for example, is the higher the probability of failures or surprises in developing components in a given project, the more gray coding might be the clearly responsible approach. However, there are obviously also cases in which multiple components need to be changed between iterations of a machine for anything truly new to come of the process. When to apply either approach is, as always, best left up to a smart PM.

While the legacy of the Connection Machine is a complicated one, it was still the gem in an architectures portfolio that accomplished what it sought to do. As Roland and Shiman put it:

The first stage of the SC architectures program proved unquestionably that parallelism was viable and did have promise in solving many real-world problems…According to Professor John Hennessy of Stanford University, a respected authority on microelectronics and computer architectures, computer designers before SC, in their quest for greater speed and performance, kept running into the wall posed by the fundamental physical limits to the improvement of conventional systems and components. Each architect would try, and each would join the pile at the base of the wall. It was Squires, he said, who showed the way, and, through parallelism, led designers over the wall.

Hopefully the collection of history and lessons in this post prove of some use to PMs thinking through decisions like those that arose in the ILLIAC IV and CM1-CM5 projects.

Share

Subscribe now

Stay tuned for next week’s pieces! They explore DARPA’s ~1980 work in speech and computer vision applications, expert systems, and its programs for moving the field of chip research forward.

• Coming Sunday

  • MOSIS: The 1980s DARPA 'Silicon Broker'

• Coming Monday/Tuesday

  • The Autonomous Land Vehicle, Pilot's Associate, and Battle Management: Three North Star Application Projects from DARPA's Strategic Computing Portfolio

ILLIAC IV

• Slotnick’s The Conception and Development of Parallel Processors: A Personal Memoir

  • An extremely useful source. Short, but quite thorough. Dives further into Slotnick’s journey from grad student to eventually running the ILLIAC IV project and his perspectives and thoughts on the events as they happened. 

• Chapter 17 of Van Atta’s DARPA Technical Accomplishments: An Historical Review of Selected DARPA Projects, Volume 1

  • A 1990, third party perspective on the planning, operations, and lessons learned from the ILLIAC IV project

• Chapter 5, pages 160-164 of Strategic Computing: How DARPA Built the Computer Age 

  • Particularly emphasizes the DARPA internal memory and different interpretations of the ILLIAC IV projects at DARPA during the 1980s.

• Chapter 6, pages 264-268 of Transforming Computer Technology: Information Processing for the Pentagon, 1962-1986

  • Particularly emphasizes how IPTO first got into parallel computing and the plans and early project planning of the ILLIAC IV project from a high level

• Chapter 2 of Transforming Computer Technology: Information Processing for the Pentagon, 1962-1986

  • Particularly dives into much of the early administrative history of IPTO — including things like budgets, norms for discussing and sourcing projects, etc.

• Slotnick’s Scientific American article, The Fastest Computer

  • A 1971 article outlining the project’s goals and achievements for a public audience. 

  • The article also contains the performance metrics and specs of various aspects of the machine as of 1971. 

  • The article also spends several pages walking through an example problem from mathematical physics, walking the reader through how the ILLIAC IV — or any parallel processor for that matter — would perform its operations and solve the problem faster than a traditional computer at the time. 

Connection Machine

• Chapter 5, pages 158-184 of Strategic Computing: How DARPA Built the Computer Age 

  • Provides more detail on the history of Squires’ approach to the architectures program and other projects that made up the portfolio. Some details about how the projects were run can also be found there. 

• The Rise and Fall of Thinking Machines

  • A thorough article diving into the lifespan of the company from a business and cultural perspective. The piece covers Thinking Machine’s corporate decision-making and relationships from its early years until its bankruptcy.

• Richard Feynman and the Connection Machine

  • This Physics Today article, written by Hillis himself, not only discusses Feynman and his work with the company, but also paints a picture of what went on in the early days of the company when it was a bunch of technical researchers working out of an old mansion outside of Boston.

• Thinking Machines Rather Than Markets

  • This Washington Post article, written in the midst of TMC’s demise, provides additional facts about the situation, but will be primarily useful to readers of this piece in helping them understand how reasonable-seeming help provided by an actor like DARPA can be portrayed depending on the given political climate.

• Strategic Computing: How DARPA Built the Computer Age

• Transforming Computer Technology: Information Processing for the Pentagon, 1962-1986

In 1962, IPTO's $9 million budget was entirely earmarked for category 6.1 research.

For the rest of the interpersonal aspects of the accounts in the next few paragraphs, I’ll rely on Slotnick’s memoir since he seemed to have a more clear recollection of the order in which things happened than Sutherland.

Another FreakTakes piece dives into the analogous case of early molecular biology researchers — many of whom would eventually win Nobel Prizes — taking risks and changing multiple parameters between experiments. The piece explores what they did and why they thought that was an optimal strategy for a field as young as molecular biology.

For those curious about what "AI applications" looked like in the mid-1980s, check out next week's post on the Autonomous Land Vehicle, Pilot's Associate, and Battle Management

I’ve taken to calling the pieces I write for FreakTakes “administrative histories.” The reason I opted for that name instead of “progress studies histories” or “metascience histories” is that it was the name that drew the fewest confused stares and/or eye rolls from the scientists and engineers whom I hope to attract. Some think this classification is a bit boring…but I have no problem with that! After all, I have — to some extent — made my name diving deeper into the boring details of science and engineering history than most would ever feel incentivized to do just to answer specific questions like “How did Bell Labs choose its research questions?” or “Why in the hell was Warren Weaver confident enough to shift 80% of the Rockefeller Foundation’s natural sciences budget to some no-name field halfway between biology and physics?”

Luckily, it turns out those who run research orgs and fund research believe administrative histories like these help them do their jobs better. Some in the community have even reached out to me asking if I could expand my work to cover a specific sub-area of particular importance: ARPA history.

In the coming months, I plan to do jus that. I will be writing administrative histories on specific DARPA/ARPA projects throughout history just as I have been doing with industrial R&D labs. Each individual write-up will be a bit shorter and there will be a bit less storytelling, but there will be many more of them to make us of. The hope is that these will prove useful to PMs at organizations similar to DARPA as well as the broader science and engineering research community. The work is being done with the help of former ARPA PMs, making full use of those who have written books containing details on specific ARPA projects, and in my role as a Fellow with the Good Science Project.

Please subscribe to follow along with the work! And, of course, I would be happy to talk through any of the write-ups with practitioners who have questions. Later this month, I’ll be releasing the first batch of pieces covering the history of 8-10 DARPA projects in about 40-50 pages of writing. This first batch will largely cover computing projects from the late 1970s into the early 1990s. The following batch will largely cover projects from DARPA’s earlier decades of computing projects. As the project wears on, I hope to cover projects from all eras and general subject areas of DARPA history.

This project will take a deep look at projects that are considered unequivocal successes as well as those that are remembered more for their lessons learned than scientific outputs. The goal of the project is, first and foremost, to produce materials that help those who manage research funding portfolios and labs to do their jobs even better. The tone of the administrative histories on this Substack has always been a positive one and I intend to keep it that way even when writing about projects that did not work out.

Please subscribe to follow along! If you know any great ARPA alumni who I should talk with or projects from history that I should look into, please ping me and let me know! Pointing me to any useful reading materials is always appreciated as well. My Twitter is right here and my email is egillia3 | at | alumni | dot | stanford | dot | edu.

Please subscribe and follow along! Thanks for reading.

Subscribe now

Share

Don Swanson’s career started on a path familiar to many who read this blog. A BA in Physics from CalTech in 1945. A Ph.D. in Theoretical Physics from UC Berkeley in 1952. A decade working in scientific labs. Then his career took a turn when, after a decade of working in traditional scientific labs, Don took a professorship in the Graduate School of Library Science at the University of Chicago. For those keeping track, Don is the first Professor of Library Science mentioned on FreakTakes. But, as weird as it is to say, thanks to some of his trailblazing work and recent progress in the world of computing, it’s very possible he won’t be the last.

Don would become known for developing computing methods to take advantage of the increasingly online nature of scientific papers — leading to discoveries in a field he knew little about, biology.

The first paper in a series that would come to be his career-defining work was released in the April 1986 edition of The Library Quarterly. The paper, titled “Undiscovered Public Knowledge,” is a fascinating piece of first principles thinking that is even more interesting today than it was in 1986 — Swanson’s big idea was not dependent on the technology of his time, but limited by it.

Subscribe now

Share

This post is a much longer writeup to an abbreviated response I gave to Michael Retchin when, at coffee, he asked I had any ideas for incorporating LLMs into a research workflow. It is also a bit of a break from my pieces on industrial R&D labs and applied research — which are my primary focus and will be resuming in the coming weeks.

In the first section, I tell the story of Don Swanson and his use of computers to identify undiscovered public knowledge. In the second section, I detail how Mullainathan and Ludwig, recently, have devised a pipeline — enabled by modern ML — to generate novel social science hypotheses.

Then, in the third and final section of this piece, I enlist the help of an actual life scientist — Corin Wagen. Corin gives a practical breakdown of how one could deploy the general ideas covered in the first two sections of this piece to the field of organic chemistry.

The first two sections and the conclusion are written by myself; the third section was written by Corin. Back to the action!

In spite of Swanson's 1986 paper being the first in a very practical series, 22% of the citations in the article cite a non-scientific researcher: Karl Popper. Popper, the great philosopher of science, outlined a model of how science works in practice defined by an interplay of three worlds. Popper's model of these three worlds is what set the stage for Swanson’s simple idea which, in Popper’s era, would not have been at all feasible.

Popper’s model, as Swanson saw it, saw the truth-seeking that made up the real scientific process as a messy and failure-prone process. This messy process is what Gerald Holton referred to as ‘science-in-the-making’ or S₁. While science-in-the-making is much messier than the clean write-ups that appear in journals, it is how great science is actually done.

Swanson further broke down his reading of Popper and how he viewed the body of scientific knowledge contained in existing research, saying:

The question of what is or is not real can in part be fought out on the battleground of critical argument, with various scholars corroborating or correcting the perceptions of others.

Knowledge begins therefore with conjecture, hypothesis, or theory, all of which mean about the same thing. Scientific knowledge grows through testing and criticizing theories and through replacing theories with better ones that can withstand more severe tests and criticism. Thus knowledge is constructed of conjecture, and, though filtered through reality, remains forever conjectural. The sine qua non of science is not objectivity or even "truth," as is often thought, but a systematically self-critical attitude. Scientists are expected to propose testable theories and to be diligent in seeking evidence that is unfavorable to those theories. If they do not criticize their own work, someone else will.

Public criticism and published argument are crucial in helping the scientist create a product that can rise above his own prejudices and presuppositions. Although scientists and scholars may never be objective, the published products they create, shaped and weeded by criticism, can move ever closer to objectivity and truth.

Popper’s model of the three worlds of science contained:

• World 1, the physical world. This is the actual world which scientists and engineers concern themselves with.

• World 2, researchers’ swirling thoughts about the physical world. This is the subjective world of “mental states and mental processes” that happen in individuals’ heads. World 2 can be taken to mean many things. But, for the sake of this piece, we can take World 2 to mean the collection of personal experiences, ideas, and individual thought processes that pertain to the physical world (World 1) but are not exactly formal scientific theories or fully fleshed-out ideas. The beginnings of theories are based on some vast collection of experiences with the physical world present in any individual researcher’s head. This collection is contained in World 2.

• World 3, the (mostly) solidified body of scientific knowledge. This world contains problems, theories, and other products of the human mind. As Swanson noted, “World 3 is real in that, through interacting with World 2, it can influence World 1.”

Popper pointed out the interesting implication that World 3 was created by man, but it could give rise to problems unforeseen by its creators. As one example, man created a number system, but with the number system came “an infinity of unintended and unforeseen consequences, including prime numbers.” Each of these unforeseen consequences awaited discovery. Many of these relevant unforeseen consequences have been discovered, but a countless number of them have also not been.

Swanson reiterated this point, saying:

World 3, while created by man, contains far more than man has ever thought of or dreamed about. World 3 indeed must contain ever increasing quantities of undiscovered knowledge.

The objective state of World 3, and what we subjectively know about World 3, are quite different concepts. So far as either certainty or "truth" is concerned, I shall show that World 3 must be in principle unknowable in the same sense that World 1 is unknowable. In either case, we cannot know, we can only guess. Only a small portion of World 3 is known to any one person. Some things we can perhaps know reasonably well, such as a theory that we ourselves have invented. But, once our invention becomes public knowledge-a bona fide resident of World 3 — it takes on a life of its own.

There are many implications of this conclusion, but the one most relevant to this piece (and Swanson’s work) is that many great, new ideas can be found by a researcher primarily concerning themself and their research process with only the observations in World 3 — rather than in World 1. In Swanson’s eyes, only so much of World 3 is ever knowable to any one researcher or research team’s brain. Computers and new methods of information retrieval, in 1986, were starting to change that. This led to practical implications that Popper's model did not explore — but which Swanson's work would.

Swanson had learned in his now two decades in his adopted world of library science that the role of librarians and their indexing methods were vital to the scientific process. Individual researchers were only familiar with certain aspects of the scientific literature, and librarians had developed methodologies to, as best they could, help researchers identify and gain access to the public scientific knowledge which they did not know how to find. When in this position, a researcher often does not know exactly what they're looking for, how to explore the literatures of related fields, how to account for the fact that some concepts go by multiple terms, etc.

Swanson — and anyone who has ever used a library index — knew the librarians' methodologies were better than nothing, but also extremely limited. Interestingly, he noted, “Information retrieval is necessarily incomplete, problematic, and therefore of great interest — for it is just this incompleteness that implies the existence of undiscovered public knowledge.”

He then went on to explain several hypothetical examples in which advancements in information retrieval could have extremely practicable scientific implications. One of these examples was titled, ‘A Missing Link in the Logic of Discovery:’

A more complex, and perhaps more interesting, example will further illuminate the idea of undiscovered public knowledge. Suppose the following two reports are published separately and independently, the authors of each report being unaware of the other report: (i) a report that process A causes the result B, and (ii) a separate report that B causes the result C. It follows of course that A leads to, causes, or implies C. That is, the proposition that A causes C objectively exists, at least as a hypothesis. Whether it does or does not clash with reality depends in part on the state of criticism and testing of i and ii, which themselves are hypotheses. We can think of i and ii as indirect tests of the hidden hypothesis "A causes C."

In order for the objective knowledge "A causes C" to affect the future growth of knowledge-that is, in order for it to be tested directly, or to reveal new problems, solutions, and perhaps new conjectures, the premises i and ii must be known simultaneously to the same person, a person capable of perceiving the logical implications of i and ii. If the two reports, i and ii, have never together become known to anyone, then we must regard "A causes C" as an objectively existing but as yet undiscovered piece of knowledge — a missing link. Its discovery depends on the effectiveness with which information can be found in the body of recorded knowledge. Notice that "A causes C" is not something once known but forgotten; it is genuinely new knowledge that awaits discovery by explorers of World 3.

Swanson, in this paper, proceeds to put this theory to the test, highlighting an A→B→C relationship which he used his own computing pipeline to identify. In this case, A was fish oil, B was the reduction of platelet aggregability and blood viscosity, and C was the risk of Raynaud’s disease. In the literature, it seemed that A caused B. In the literature, it seemed that B might counteract C. But there were no papers implying that A counteracted C.

Maybe it had been tried, failed, and not published due to publication bias. Maybe it was a silly idea for a reason that Swanson, not a biologist, was unaware of. Or maybe this was an idea that was brand new to World 3 and worthy of empirical testing by scientists who concerned themselves with World 1.

So, Swanson wrote the idea down and proposed that any interested life sciences research team test the hypothesis. It turned out that he was somewhat onto something! A paper published in the following years concluded:

We conclude that the ingestion of fish oil improves tolerance to cold exposure and delays the onset of vasospasm in patients with primary, but not secondary, Raynaud's phenomenon. These improvements are associated with significantly increased digital systolic blood pressures in cold temperatures.

Swanson’s most well-known finding using this A→B→C methodology, published two years after this initial paper, pertained to magnesium and the alleviation of migraines. The paper, titled “Migraine and Magnesium: Eleven Neglected Connections,” explored eleven possibly unnoticed connections between the pair of literatures on migraine and magnesium.

In the introduction of this paper, Swanson breaks down the necessity of this style of work even more clearly:

Undocumented connections arise neither by chance nor by design but as a result of the inherent connectedness within the physical or biological world; they are of particular interest because of their potential for being discovered by bringing together the relevant noninteractive literatures, like assembling pieces of a puzzle to reveal an unnoticed, intended, but not unintelligible pattern. The fragmentation of science into specialties makes it likely that there exist innumerable pairs of logically related, mutually isolated literatures.

Swanson then goes on to point out a litany of A→B→C style connections that connected A, magnesium levels, to C, the presence/severity of migraines. In spite of a number of connections, the number of papers in the literature that mentioned A and C together was vanishingly small. The eleven factors relevant to both migraine physiopathology and the physiological effects of magnesium were: type A personality, vascular tone and reactivity, calcium channel blockers, spreading cortical depression, epilepsy, serotonin, platelet activity, inflammation, prostaglandins, substance P, and brain hypoxia.

So, naturally, Swanson recommended that researchers in the medical community that concern themselves with World 1 look into this relationship to see if there was causality. And, once again, it seems that Swanson and his methods were vindicated. While magnesium treatments did not prove to be a silver bullet for migraine sufferers — obviously migraines still exist — magnesium has since become a remarkably common treatment to help alleviate migraine symptoms. A paper as recently as 2012, titled “Why all migraine patients should be treated with magnesium,” argued, “The fact that magnesium deficiency may be present in up to half of migraine patients, and that routine blood tests are not indicative of magnesium status, empiric treatment with at least oral magnesium is warranted in all migraine sufferers.”

Even if the reader is not impressed with those findings in and of themselves, the findings should surely be taken seriously as an extremely convincing proof of concept of the idea that undiscovered ideas are out there and can be practically discovered with the right scientific area and computationally-augmented workflow. These new hypotheses done using data from World 3 can then be tested and verified/rejected using traditional scientific methods on World 1.

These pipelines will never be perfect. And using them well may always be — as is the case for much of computing — more of an art or messy engineering process than a pure science. As Swanson pointed out, the problem of information retrieval is inevitably incomplete and is destined to not be perfectly efficient. But the areas of computing relevant to this problem area have improved by many orders of magnitude in the years since Swanson…and, as readers have surely noticed, to a shocking degree in the past year or two.

New functionality exists, and much is possible now that was not before. The pipeline which Swanson used to generate his insights was remarkably primitive by modern standards.

I'll (briefly) break down his process to give the reader an idea. For various diseases, such as migraine, Swanson scanned a Medline file in which 2,500 article titles contained the word “migraine.” One of those titles referred to a possible relation between spreading depression to the visual scotomata of classic migraine. Then, as Swanson detailed:

The phrase “spreading depression” is next explored on its own, by examining titles in which it occurs and reading the literature itself is necessary. One can discover among other things that magnesium in the extracellular cerebral fluid can prevent or terminate spreading depression. Combining this new fact with the above connection, one is led to the conjecture that magnesium deficiency might be a causal factor in migraine.

A surmise that migraine might be a vasospastic disorder led to another connection with magnesium. By examining several hundred titles containing the word “vasospasm,” one can notice a few titles indicating that magnesium deficit can cause vasospasm. That discovery can be made more efficiently if one guesses at the outset that a deficiency of some kind might be implicated. By requiring the Medline subheading “deficiency” to appear in the descriptor field, one can narrow the several hundred titles with “vasospasm” down to four, of which three are about magnesium.

Swanson points out that, of course, this methodology did not lead directly or exclusively to magnesium. It led to many other guesses, most of which were discarded. The biggest reason for hypotheses being weeded out was simply that two terms tended to be mentioned in the same article frequently. Swanson’s heuristic took that to mean the hypothesis had already been examined.

Swanson later put together a computer system called Arrowsmith which attempted to make carrying out these steps easier, but it never became widely used. This system was inevitably quite manual. And the use of computers in day-to-day knowledge exploration tasks was both not as common today and the machines themselves were much less capable.

But Swanson’s core idea should live on! Yes, his methods were simple. And, yes, his findings were not Nobel-winning (or anything close). But one can only imagine what he may have attempted had modern text processing, tf-idf scoring, or cosine similarity existed in 1986. Swanson only had keyword searching, manual skimming, and the faint idea that this should be possible available to him.

We now live in a different world — one Swanson only could have dreamed of. The task of rooting out undiscovered public knowledge and aiding human researchers in assembling this information into real-world scientific discoveries is absolutely something that, now, should be very seriously re-visited.

With the improvements in LLMs, it seems we are living in a world where language models — whether or not they ‘understand’ in a literal sense — have capacities that often mimic human capacity to understand conceptual phenomena beyond the literal words on a page. Creative minds can likely find ways to make use of this fantastic new tool in the hypothesis-generating process. Surely, at the minimum, Swanson's A→B→C methodology to find undiscovered public knowledge in World 3 is one viable way to do this. But I imagine even more fruitful frameworks for deploying modern computing tools to accomplish similar goals are out there to be discovered as well.

This will be explored in more detail in Corin's section which contains some ideas on how this could apply to modern organic chemistry research. But, first, I'd like to explore a recent paper on machine learning as a tool for hypothesis generation. The paper seeks to generate new hypotheses in a common area of computational social science, causes of bias in bail decisions, using non-LLM methods in modern ML — image recognition, morphing, and GANS.

Jens Ludwig and Sendhil Mullainathan recently released a working paper on machine learning as a tool for hypothesis generation. The paper demonstrates that a very different future of hypothesis generation could already be upon us — in very specific areas at least.

The authors open with a beautiful description that outlines how messy, human, and creative hypothesis generation usually is. It is a description that does a good job of unpacking the kind of activities that tend to reside in Popper’s World 2. The authors open the paper, writing:

Science is curiously asymmetric. New ideas are meticulously tested using data, statistics and formal models. Yet those ideas originate in a notably less meticulous process involving intuition, inspiration and creativity. The asymmetry between how ideas are generated versus tested is noteworthy because idea generation is also, at its core, an empirical activity. Creativity begins with “data” (albeit data stored in the mind), which are then “analyzed” (albeit analyzed through a purely psychological process of pattern recognition). What feels like inspiration is actually the output of a data analysis run by the human brain. Despite this, idea generation largely happens off stage, something that typically happens before “actual science” begins. Things are likely this way because there is no obvious alternative. The creative process is so human and idiosyncratic that it would seem to resist formalism.

That may be about to change because of two developments. First, human cognition is no longer the only way to notice patterns in the world. Machine learning algorithms can also notice patterns, including patterns people might not notice themselves. These algorithms can work not just with structured, tabular data but also with the kinds of inputs that traditionally could only be processed by the mind, like images or text. Second, at the same time data on human behavior is exploding: second-by-second price and volume data in asset markets, high-frequency cellphone data on location and usage, CCTV camera and police “bodycam” footage, news stories, children’s books, the entire text of corporate filings and so on. The kind of information researchers once relied on for inspiration is now machine readable: what was once solely mental data is increasingly becoming actual data.

We suggest these changes can be leveraged to expand how we generate hypotheses. Currently, researchers do of course look at data to generate hypotheses, as in exploratory data analysis (EDA). But EDA depends on the idiosyncratic creativity of investigators who must decide what statistics to calculate. In contrast, we suggest capitalizing on the capacity of machine learning algorithms to automatically detect patterns, especially ones people might never have considered. A key challenge, however, is that we require hypotheses that are interpretable to people. One important goal of science is to generalize knowledge to new contexts. Predictive patterns in a single dataset alone are rarely useful; they become insightful when they can be generalized. Currently, that generalization is done by people, and people can only generalize things they understand. The predictors produced by machine learning algorithms are, however, notoriously opaque — hard-to-decipher “black boxes.” We propose a procedure that integrates these algorithms into a pipeline that results in human-interpretable hypotheses that are both novel and testable.

The problem area on which the authors chose to test their new method is one they know well: biased bail decisions.

In recent years, Mullainathan, Jens Ludwig, and a handful of other researchers who straddle the worlds of social science and computer science have been producing a widely-followed series of papers on algorithmic bias. Many of the papers in this area focus on bail decisions. It is a somewhat ideal environment in which to do this kind of research, all things considered. The outcome variables are public, there is a lot of (bias-prone) human discretion from judges’ decisions, statistical algorithms are sometimes used to determine risk scores to jail or release defendants, both numerical data and text are available on the crimes, and additional data is available about the defendants themselves — even their photos in some cases.

Researchers have been utilizing this data to ask and answer questions such as: Can we produce bail algorithms that are more accurate than judges?, Can we produce bail algorithms that minimize seemingly unfair racial bias?, or What features seem to be most associated with unfair treatment from judges?

The latter question is the one that Ludwig and Mullainathan chose to be their subject of investigation in this paper: Can we use machine learning to help generate new theories for what features of a defendant seem to bias a judge one way or another? This question is particularly interesting because, as in many areas with large datasets available, the machine learning algorithms far outperform the hypothesis-based statistical models put together by people like economists — which are often regression models.

The machine learning algorithms are clearly seeing some patterns that are just not captured by existing hypotheses. So, Ludwig and Mullainathan set out to see if they could get similar neural net-based approaches to unpack what these complex models were seeing.

And…it worked. Their methods discovered hypotheses that were interpretable, novel, and seemingly right.

When the researchers built deep learning models to predict judges' decisions, the defendants mugshot photos proved to be extremely predictive of judge behavior. This, on its own, accounted for a quarter to half of the predictable variation in detention — depending on whether one uses R-squared or AUC. Contextualizing this for the readers, the authors write, “Defendants whose mugshots fall in the bottom quartile of predicted detention are 20.4 percentage points more likely to be jailed than those in the top quartile. By comparison, the difference in detention rates between those arrested for violent versus non-violent crimes is 4.8pp.”

Of course, some percentage of what the algorithm was ‘seeing’ in the photo was inevitably going to be things that have already been hypothesized and accounted for in the existing social science literature and their regression models. Some of these prior hypotheses include things like: gender, race, race, skin color, and other facial features suggested by existing research.

Panel A of Figure I in the paper demonstrates that the authors' ML algorithm is finding far, far more than what the causally-focused models account for. Even if one adds in a column that accounts for human guesses of judge behavior based on the photos, only statistically accounting for those factors does not come close to the algorithm’s prediction of judge behavior. In the figure below, the panel on the left indicates which variables are being controlled for in the rightmost panel where the R-squared achieved by the existing hypotheses in the literature is compared to the R-squared of a machine learning algorithm using that information plus the mugshot pixel information.

Now, while impressive, that result is not in and of itself groundbreaking. We have known for a while that basic statistical methods such as OLS often cannot touch the performance of machine learning algorithms when dealing with high-dimensional data such as cell phone data, online clickthrough data, images, text, etc.

What has vexed us is the ability to interpret what’s happening in a human-explainable way. Even attempting to understand what neural nets are doing as they learn primitive tasks such as basic addition is a handful for remarkably intelligent researchers. Explainable hypotheses on applications like this, many feel, have seemed quite out of reach in the near term.

But that is exactly what the authors succeed in doing a convincing job of. In the end, they are able to produce a very similar chart to the one above — demonstrating marked improvements based on new, human-interpreted, ML-based hypotheses rather than just a black box algorithm.

Their new pipeline, in part, leverages M-Turkers learning what the algorithm was seeing. The M-Turkers' descriptions of what the algorithm was seeing — in word form — became hypotheses. These hypotheses were fed to separate M-Turkers who added those hypotheses — as text labels — to the test set. And the basic statistical model — similar to the simple ones often used in social science literature — was able to achieve substantial increases in performance by controlling for these new machine-discovered and human-interpreted features.

Let’s explore how exactly they did this in more detail. (Feel free to only skim the steps of the pipeline section if you don’t care to know the details.)

This section contains a condensed version of the steps that made up the authors' workflow for generating human-interpretable hypotheses leveraging machine learning methods. Please check out the paper if you’d like to know the minutiae of how certain individual steps were performed. (The paper plus appendix came out to 130 pages, so the authors really do get into a lot of detail on all the steps for those curious.)

In this case, specifically, the authors built a supervised learning algorithm — with an outcome variable based on judges' bail decisions — and identified that a surprising amount of the predictable variation in judges’ decision-making came from the defendants' faces. They did this using an ensemble approach with gradient-boosted decision trees on the structured administrative data (current charge, prior record, age, gender, etc.) and a convolutional neural net trained on the raw pixel values from the mugshots.

This is the step displayed in the graphic in the preceding section — comparing the R-squared of the ML approach to the simpler models which control for hypotheses from the social science literature. The authors ran an OLS regression controlling for the variables that represent positive hypotheses in the existing literature to see how much of the variation, if any, that the ML ensemble method identifies that is not accounted for by these variables. The answer was: a lot.

To do this, for this particular use case, the authors needed to employ people to label some images with things like race, skin tone, etc. What this step looks like will vary based on the application. But what matters for this high-level overview is that the authors did what they needed to do to label the data so previously proven hypotheses could be controlled for. With that done, the algorithm was still finding a ton of predictable variation that those variables did not account for.1

This step is the most pivotal (and technically complex) of the paper. The authors write:

The algorithm has made a discovery: something about the defendant’s face explains judge decisions, above and beyond the facial features implicated by existing research. But what is it about the face that matters? Without an answer, we are left with a discovery of an unsatisfying sort. We will have simply replaced one black box hypothesis-generation procedure (human creativity) with another (the algorithm).

The authors need to find a way to unpack this in a human-understandable way. That is the crux of the paper. First, they explain why common computer science methods like saliency maps do not do the job for cases like this.

A common solution in computer science is to forget about looking inside the algorithmic black box and focus instead on drawing inferences from curated outputs of that box. Many of these methods involve gradients…The idea of gradients is useful for image classification tasks because it allows us to tell which pixel image values are most important for changing the predicted outcome.

For example, a widely used method known as “saliency maps” uses gradient information to highlight which specific pixels are most important for predicting the outcome of interest (Baehrens et al., 2010; Simonyan et al., 2014). This approach works well for many applications like determining whether a given picture contains a given type of animal, a common task in ecology (Norouzzadeh et al., 2018). What distinguishes a cat from a dog? A saliency map for a cat detector might highlight pixels around, say, the cat’s head: what is most cat-like is not the tail, paws or torso, but the eyes, ears and whiskers. But more complicated outcomes of the sort social scientists study may depend on complicated functions of the entire image.

The authors elaborate:

In the cat detector example, a saliency map can tell us that something about the cat’s (say) whiskers are key for distinguishing cats from dogs. But what about that feature matters? Would a cat look more like a dog if its whiskers were, say, longer? Or shorter? More (or less?) even in length? People need to know not just what features matter but how they must change to change the prediction. For hypothesis generation, the saliency map under-communicates with humans.

For something like a human face’s age, age is a function of almost all parts of our faces. So, a saliency map highlights almost everything. That isn't helpful.

So, the authors pursue a “morphing” procedure. Instead of highlighting pixels, they change them in the direction of the gradient of the predicted outcome. This, essentially, creates a new synthetic face that has a different predicted outcome. They describe the idea as follows:

Our approach builds on the ability of people to comprehend ideas through comparisons, so we can show morphed image pairs to subjects [M-Turk workers] to have them name the differences that they see.

The top row of the following graphic shows how, for a factor like age, the saliency map covers the entire image. The bottom panel of the graphic demonstrates the authors' morphing procedure done in a way that seeks to increase predicted age — just as an example.

The authors constrain their morphing procedure in such a way that it attempts to change as little as possible about the images while maximizing the detention risk of the synthetic images. In addition, they also construct the procedure in such a way that the morph produces differentiation orthogonal to traits that are already accounted for by hypotheses in the existing literature. In short, these are new faces are minimally morphed to achieve a maximal increase in the probability of certain bail decisions. And they have been morphed in such a way that the traits that have been changed are not traits accounted for in the existing literature.2

The researchers then ask M-Turkers to look at image pairs — with one morphed image morphed to have a higher predicted harshness of sentence than the other. These subjects, after being given a short set of images and correct answers to have a chance to ‘learn’ the pattern, were then shown many pairs of images and asked to guess which image expresses a higher likelihood of a harsh bail decision. The M-Turkers, on average, improved from more or less 50/50 guessing at the start of the round to a 67% accuracy rate by the end of the round.

You can imagine the following image — with the caption under the photos showing the correct answer — representing what the training process looked like.

As a point of reference: the authors also generated the following ‘intermediate steps’ for that particular photo to give us an idea of what the image morphs look like for detention probabilities between .41 and .13

Next, given a set of humans clearly just demonstrated that they were learning the differences between the morphs somehow and using the knowledge to make more accurate predictions, the researchers simply asked them to describe what they were doing.

The M-Turkers would have just looked at many pairs of images that look like those below and been asked to describe what traits separate one column from the other.

The authors gave these text descriptions to separate research assistants and asked them to create basic category labels that captured the mostly synonymous human descriptions. The humans' unstructured text comments were obviously a bit noisy, but the authors show that — even in a word cloud — a pretty clear first hypothesis starts to emerge. Some of the most common words are things like “cleaner,” “shaved,” “scruffy,” “moustache,” and “shorter.”

The researchers called this feature “well-groomed.” According to the humans who were making accurate predictions, being well-groomed helps a lot!

Now, with this new hypothesis in place, the next step is to check if it holds up in practice. So, in the next step, the researchers asked a new set of 343 humans to label over 32,000 mugshots in the training and validation set according to how well-groomed they were on a 9-point scale. Once they did this, the authors found that this well-groomed variable was very highly correlated with the algorithm's predictions. A 1-point increase in standard deviation was associated with a 1.74% decline in predicted detention risk. A 1.74% decrease in detention risk is a 7.5% reduction in the base rate.

The new variable also significantly increased the predictiveness of the new version of the full model. The authors break down the explanatory power of the new variable, saying: “Another way to see the explanatory power of this hypothesis is to note that this coefficient hardly changes when we add all the other explanatory variables to the regression…despite the substantial increase in the model’s R-squared.”

With that, it seems they've found a new, ML-enabled, human-interpretable hypothesis that is likely correct: being well-groomed seems to substantially bias judges in your favor.

Next, the authors did it again!

The cool thing about this procedure is that it's iterable. This well-groomed variable explains some of the variation in the algorithm’s prediction of the judge, but not all of it. So, the authors continue with the above steps — now with a defendant’s well-groomed-ness labeled in the dataset alongside things like race and charge — and attempt to find another novel hypothesis orthogonal to what has already been labeled and accounted for in the data.

The authors write:

Note that the order in which features are discovered will depend not just on how important each feature is in explaining the judge’s detention decision, but also on how salient each feature is to the subjects who are viewing the morphed image pairs. So explanatory power for the judge’s decisions need not monotonically decline as we iterate and discover new features.

To isolate the algorithm’s signal above and beyond what is explained by well-groomed, we wish to generate a new set of morphed image pairs that differ in predicted detention but hold well-groomed constant. That would help subjects see other novel features that might differ across the detention-risk-morphed images, without subjects getting distracted by differences in well-groomed.

At this point, they note that this iteration procedure raises several technical challenges because, if they used the same procedure as in the first step, they would be orthogonalizing against predictions of well-groomed rather than objective well-groomed-ness — and orthogonalizing against a prediction can be an error-prone process. So, instead, they built a new detention prediction algorithm curated on a training set that was limited to pairs of images matched on the features that they were looking to orthogonalize against. In this particular step, that meant that they could, as they put it, “use the gradient of the orthogonalized judge predictor to move in GAN latent space to create new morphed images that have different detention odds but are similar with respect to well-groomed.”

These new images, shown below, were then put through the same steps as before. The authors also note that — as you will see — the morph generator did not do a perfect job of orthogonalizing against well-groomed-ness. But what is important is that there is still what they believe to be “salient new signal.”

With these image pairs, a second new salient facial feature comes to light in the eyes of the M-Turkers: how “heavy-faced” or “full-faced” a defendant is.

In fact, heavy-faced seemed to be even more important than well-groomed was. A one standard deviation improvement in the well-groomed variable was associated with a 7.5% reduction in the base rate. A one standard deviation improvement in the full-faced variable was associated with a 9.3% improvement.

The authors, once again, have the M-Turkers go through and add this label to the relevant dataset so they could run the regression model controlling for this new variable as well. Once again, it yielded an improvement.

The authors conclude their section on ‘Iteration,’ saying:

In principle, the procedure could be iterated further. After all, well-groomed, heavy-faced plus previously known facial features all taken together still only explain 27% of the variation in the algorithm’s predictions of the judges’ decisions. So long as there is residual variation, the hypothesis-generation crank could be turned again and again. But since our goal is not to fully explain judges’ decisions but rather to illustrate that the procedure works and is iterable, we leave this for future work.

Before this can be considered a success, one has to check if these new hypotheses are good predictors of actual judge decisions. Up until this point, the models were largely being validated on the (imperfect and noisy) ML predictions of judge predictions rather than actual judge decisions. So, this step is the one that really matters. Well-groomed and full-faced could be correlated with some part of the algorithm that is actually unrelated to actual judge behavior.

When the authors did this, building a regression model using all of the original variables as well as well-groomed and full-faced, they noted that “the explanatory power of our two novel hypotheses alone equals about 28% of what we get from all the variables together.”

The gray dot in the bottom-most row in the right panel demonstrates the improvement of the regression model when well-groomed and heavy-faced are controlled for. The magnitude of improvement when including both of these variables is similar to when skin tone was found to matter more than race in many cases — which was considered a very significant and not necessarily intuitive finding made by humans in this field of social science.

After running an experiment at a public defender’s office and legal aid society, the authors determined that it did not seem that the practitioners previously understood the extent to which these newly-discovered characteristics correlated with outcomes. The authors noted that they were mentioned here and there by some of the practitioners, “but these were far from the most frequently mentioned features.”

The researchers also had the practitioners guess which of a pair of images was more likely to be detained — just as the authors did with M-Turkers. They found that the effect of the practitioners' guesses in the performance of the full model was four times as large as the M-Turker guesses. In fact, the practitioners' guesses were about as accurate as the algorithm itself. The researchers elaborated on the relationship between the practitioners' predictions and the algorithm's predictions, saying:

Yet practitioners do not seem to already know what the algorithm has discovered. We can see this in several ways in Table VI. First, the sum of the adjusted R-squared values from the bivariate regressions of judge decisions against practitioner guesses and judge decisions against the algorithm mugshot-based prediction is not so different from the adjusted R-squared from including both variables together in the same regression…We see something similar for the novel features of well-groomed and heavy-faced specifically as well. The practitioners and the algorithm seem to be tapping into largely unrelated signal.

This pipeline — leaning heavily on neural nets and a human-feedback loop leveraging basic human competencies — generated a new, good idea.

While Swanson’s idea was an example that used primitive technology in the general area to which LLMs can contribute — text-based idea discovery — Mullainathan and Ludwig demonstrate how modern neural nets can be leveraged to generate clever theories in a completely different realm of application.

Is what the computer is doing creativity? I’d say no…but I’d also say it doesn’t matter. The increased capacity of modern computing tools simply is. It's up to us to find ways to use the new capacities well. If researchers use them well, it should free up human minds to focus their efforts on entirely new areas of creativity than they have before. This won't impact all hypotheses. Many hypotheses will come to be in the same ways that they always have. But, hopefully, many hypotheses will also come to be through new combinations of humans and computing. Both Don Swanson's work from the 1980s and the morphing work of the more recent authors lay a fantastic base for understanding what is now possible in this realm.

Researchers should be taking advantage of these ideas wherever they can! In line with this goal, the following section contains one researcher's ideas for possible applications of these ideas which could be fruitful in his own field: organic chemistry.

The ideas are largely inspired by Swanson’s A→B→C methodology and the work that led to the (recent) second Nobel Prize in Chemistry for Karl Sharpless.

Tools that accomplish Don Swanson-like tasks are particularly useful in areas of science in which the capacity to read and remember a large volume of research papers — from all languages and time periods — is a distinct advantage. You could think of this as having a bunch of really dumb research assistants who are very well-read and have great memories — but they're still dumb. One current research area in which this would be a distinct advantage is the field of organic chemistry.

The rest of this section was written by Harvard Ph.D. organic chemist, blogger, and current founder, Corin Wagen. Corin was the first person to tell me about Karl Sharpless' idea-generation process. Corin, after reading a draft of the Don Swanson and Sendhil & Ludwig sections, set out to write a section describing how he thought those ideas could fit into his own field in the near-term and make a big difference.

Several factors make organic chemistry a promising domain for LLM-assisted hypothesis generation. Organic chemistry is a very old field, with almost two centuries of publications to sift through, and there are a lot of venerable results which have been forgotten and unearthed with great fanfare (vide infra). Additionally, it’s relatively easy to validate the accuracy of previous results in organic chemistry by reproducing the reactions—few organic reactions require bespoke instrumentation, and there’s an expectation that one lab’s reactions ought to be easily reproducible by other labs (e.g.).

The work of K. Barry Sharpless illustrates how knowledge of the old literature can allow for important breakthroughs today. Since around 2000, Sharpless has been advocating for a new paradigm in chemical synthesis, termed “click chemistry,” for which he was co-awarded the 2022 Nobel Prize. In his words:

The goal [of click chemistry] is to develop an expanding set of powerful, selective, and modular “blocks” that work reliably in both small- and large-scale applications. We have termed the foundation of this approach “click chemistry,” and have defined a set of stringent criteria that a process must meet to be useful in this context. The reaction must be modular, wide in scope, give very high yields, generate only inoffensive byproducts that can be removed by nonchromatographic methods, and be stereospecific (but not necessarily enantioselective).

This approach aims to avoid many of the problems that render chemical synthesis so inefficient and capricious today, and thus accelerate chemical discovery. But it’s not easy to find reactions with the desired characteristics: to date, Sharpless has focused on two reactions: copper-catalyzed azide–alkyne cycloaddition (CuAAC) and sulfur(VI) fluoride exchange (SuFEx). Importantly, both of these reactions were discovered long ago: CuAAC was developed by Rolf Huisgen in the 1960s (ref), while SuFEx dates from the 1920s German literature (ref).

These reactions, thus, are prime examples of undiscovered public knowledge. In both cases, the actual experimental results had been published for many decades, yet their relevance to modern synthesis wasn’t realized until someone (Sharpless) put the pieces together. And it wasn’t just the novelty of the click chemistry concept that led to these results being forgotten: the original click chemistry paper was published in 2001, yet SuFEx wasn’t rediscovered until 2014, meaning that there was about a decade in which anyone could have recognized that these 1920s sulfur(VI) reactions satisfied the demands of click chemistry, and yet no one did.

What makes Sharpless so successful? In part, it’s his vision for how chemical synthesis could be different — having the creativity and courage to be iconoclastic and push the field in a new direction. But it’s also his ability to discover and remember old results that fit with his vision that’s made him so successful. It’s not really possible to sit down and design a reaction that satisfies a list of desired properties; it’s much more effective to simply look through all previous reactions and find one that meets your criteria.

If we consider a cartoonish version of the Sharpless production function, then, we might arrive at the following list:

1. Have a bold vision for the future of the field.

2. Read a lot of old papers and remember them.

3. Speak German.

4. Have a good “eye” for promising old results that might fit into your vision.

Attributes 1 & 4 are likely to remain the domain of humanity for the foreseeable future — it’s silly to expect LLMs to develop good taste or a keen scientific vision. But attributes 2 & 3 are ripe for automation: an LLM can digest papers much more quickly than a human, and in far more languages than any scientist can learn. Perhaps there are future potential Sharpless-type scientists out there who have been rate-limited by the speed at which they read, or their inability to speak German!

Unfortunately, application of LLMs to these questions is hamstrung by the fact that ChatGPT and its ilk don’t have access to most research papers, which are stuck behind publisher paywalls. But one could imagine LLMs being very useful research assistants if this limitation were removed. To demonstrate, I asked ChatGPT to suggest a click reaction of allenes, another bench-stable but high potential energy functional group that (to my knowledge) no one has proposed a click reaction for. ChatGPT responded with a very promising lead, the “allene–amine coupling reaction”:

This answer is quite good: ChatGPT identifies that the reaction is highly chemoselective, displays excellent functional group tolerance, and proceeds in water as a solvent, all qualities that match the criteria for inclusion as a click reaction. Unfortunately, the reaction doesn’t exist: Google doesn’t return any results for “allene–amine coupling,” the closest examples I can find don’t match the descriptions given at all (poor functional group tolerance, not compatible with water), and the references ChatGPT gives me upon further interrogation are completely made up.

Despite this failure, the response is encouraging. ChatGPT clearly understands what it’s supposed to be looking for (type of reaction, behavior with different functional groups, solvent compatibility), but is hamstrung by the fact that it doesn’t actually seem to know chemistry beyond the level of Wikipedia. This is a pretty fundamental limitation of LLMs on their own, but not an insurmountable one: the use of retrieval augmentation and similar techniques has been demonstrated to allow ChatGPT and other LLMs to reason much more accurately based on extant data, and there’s no reason that scientific data ought to behave any differently.

While there are plenty of (proprietary) ways to search the literature based on chemical structures (like SciFinder and Reaxys), the ability of LLMs to perform semantic search opens the door to a wider variety of applications. A chemist who knows the properties they wish their reaction to have, but not the reagents involved, would be hard-pressed to use state-of-the-art tools to find an answer. But one could imagine a version of ChatGPT equipped with the ability to semantically query a database of patents and scientific publications (with the information stored in a machine-friendly way) being an extremely useful research assistant to the practicing organic chemist. Armed with such a tool, a chemist could ask plenty of useful questions:

• “I’m looking for photochemical cyclization reactions that generate piperidines, published before 1960.”

• “Robust and functional-group tolerant reactions that couple allenes to another molecule and work in water.”

• “Regioselective cycloadditions of bench-stable starting materials that occur at room temperature”

These prompts are presently very challenging to answer, limiting the ability of scientists to locate relevant literature results, but hypothetically ought to be easy enough to answer based on the data that exists. And while the answers to these particular questions probably won’t lead to any Nobel prizes, enabling scientists to query arcane corners of the literature quickly and easily would allow for more workflows like the one demonstrated by Sharpless, and thus be beneficial for scientific innovation.

What prevents this from happening today? Developing a suitable database of chemical data would entail scraping through tens of thousands of publications and patents and extracting the data contained therein, both structured (reaction conditions, yields, selectivities, etc.) and unstructured (the actual text of the papers). While efforts in this direction have been made (e.g. the Open Reaction Database for structured data), doing so for all the old journals that Sharpless and others know so well would be a sizable undertaking — but not insurmountable, contingent on IP issues being worked out.

Eric taking over again.

In my eyes, the work of both Swanson and Mullainathan & Ludwig serve as massively important proofs of concept that computationally-augmented hypothesis generation is much more than sci-fi-inspired wishful thinking.

One ideal place to start might be a scientific philanthropy scraping together the resources to put together a minimum viable dataset in an area ripe for experimentation with the idea — say, organic chemistry — and getting some researchers to see just how much they can do with the basic tool.

If it works, you can scale the hell out of it. If it doesn't, we can iterate, learn, and hopefully try again. Building up resources and know-how in the area of computer-augmented hypothesis generation seems to be an area of significant promise that is obviously worth pursuing.

Thanks for reading:)

Share

Subscribe now

Of course, the chance always exists that performance disparities are coming from the different statistical procedures and not one method “discovering” variables that the other is not.

As a note: To ensure that the morphed faces looked like realistic faces — rather than images produced by following the gradient that have different predicted outcomes but no longer look like realistic faces — the authors use a GAN. If you want to know more about this, check out page 25 of the paper, Appendix C, or Figure V.

This post is an accompaniment to Tony Kulesa’s excellent piece on the history of Y Combinator.

Many in Silicon Valley think of Y Combinator (YC) as the sum of its services. The services are, in short: cash, a network, and guidance for early-stage (mostly software) founders. The YC model and its effectiveness have become widely known and understood in Silicon Valley. The organization has become a staple of the software VC community, applying many principles that have made software VC a success to founders and companies at the earliest stages of company formation. Many of YC’s bets are made so early that the org has been known to make bets on attractive founding teams with no workable company idea yet. 

But Tony Kulesa’s Substack piece provided enough perspective to see the YC model in a different light than it is usually seen in: YC as a sum of its goals. Tony writes the following on what early YC was gearing itself to do:

One could summarize YC’s structure: 

• Invest in young, smart, energetic, and determined hackers

• Give them enough money to pay for living expenses, but not much more

• Give them a few months to build something, launch it, and see some evidence of growth

And, since finding good ideas is hard, get statistics on your side by batch processing – run as many of these experiments as you can in parallel.

The specific resources that YC chose to provide were those that allowed for fast experimentation and proof-of-growth. Considering the 'YC model' as a set of goals — rather than a set of resources —  has interesting implications for one notable area in which modern VC approaches have generally not worked well: deep tech. Satisfying YC's original goals in the realm of deep tech, for most market sizes and technology areas, has proven significantly more challenging than in the realm of software entrepreneurship. Some areas, such as biotech, have found some level of success — although not on the level of software VC. But, in general, VCs have had limited success in the realm of deep tech in comparison to their software efforts.

Tony’s piece helps the reader understand exactly how the early goals of YC and the principles on which the org was founded subsequently led to the org growing into the basket of services and resources it is today. YC’s initial goals paired with the specific market and specific technologies that surround software businesses led to the YC in its current form. But, it should be remembered, had the nature of the markets and technologies that surround software businesses looked different, then the YC model could have turned out completely differently. It is this framing that I’d like the reader to keep in mind as they read today’s piece.

Today’s piece proposes a new model of deep tech VC that seems to satisfy the same goals YC had at its inception, but looks completely different from the current, largely software VC-inspired models of deep tech VC. The specific structure I propose is inspired by some of the research organizations that helped drive the effective deep tech research and commercialization pipeline of the mid-20th C. United States.

The model I propose seeks to pair the practical problem selection and cost-efficient problem solving of entities like the early GE Research Lab, Bell Labs, and MIT’s 1920s Technology Plan with a modern understanding of risk capital. In the natural course of operation in each of those three labs — and many like them in the era — each ended up with technologies that could have been ripe for spinoff as their own independent companies. In many cases, the course of research en route to one of these potential spinoffs was a cash-flow-generating activity in and of itself. In this era, while America did have refined systems of deep tech problem discovery and technological development that are jealousy-inducing by most modern standards, it often did not have financiers willing to give money to businesses with moderate chances of failure the way modern venture capitalists are quite comfortable doing.

The model I propose builds on the formulas of these three research organizations and seeks to facilitate the creation of companies that originate with non-zero customer bases and few technical risks. I am calling the model the William Walker model of deep tech VC. Walker was a key leader of MIT’s Technology Plan and  a pioneer in the field of chemical engineering practice. In both an applied scientific and an administrative capacity, Walker’s career makes him an ideal namesake. 

The William Walker model is not dependent on a few bets “returning the fund” as is often the case in software VC. Instead, it depends on starting good, moderately sized businesses and ensuring that as much of the R&D process as possible can generate its own cash-flow. 

The rough order of the piece will go as follows:

• I’ll introduce the problem further.

• I’ll (briefly) outline some market dynamics that separate deep tech companies from software companies.

• I’ll explore some of the history of the three research organizations I mentioned — and 20th C. deep tech in general — that inform this model.

• Lastly, I’ll outline how the William Walker model for incubating deep tech companies can satisfy YC’s original goals given the more constrained financial fundamentals of deep tech companies. 

Subscribe now

Share

If any of Tony’s readers are visiting FreakTakes for the first time today, great to have you! If you’re a regular FreakTakes reader who hasn’t read Tony’s work before, check out his Substack!

This piece, as always, is done in partnership with the Good Science Project.

Back to the action.

It's no secret that the deep tech VC industry is still looking for a model that sticks. Their 21st C. efforts, largely inspired by software VC approaches, have not generally achieved the desirable returns or societal impacts they were looking for. One possible reason for this might be that software VC-type approaches were just not built to service the vast majority of deep tech companies. The structure of many deep tech VCs — when considering Tony’s piece — may be more in line with a formula that mirrors YC's current offerings than a formula put together to accomplish YC’s original goals for deep tech founders.

But I don’t think this means we can’t hope to start deep tech companies in bulk the way we’ve learned to do with software companies. It just seems that the model has to look different. In finding a model of deep tech VC that seeks to satisfy the spirit of YC's initial goals for deep tech founders, the approach of the William Walker model is very different from software-focused VC. But that should not be surprising. Achieving similar goals in two very different problem areas can often require two very different solutions.

Proposing this model does not mean it can or should be a replacement for all current models of deep tech VC. For certain (specific) problem areas, modern deep tech VC approaches work well. But, as Tony pointed out in his piece, the status quo in most areas of engineering and science is many thousands of PhDs entering into job markets with few opportunities to use their expertise and energy to start companies. 

My solution is one aimed at putting as many of them to work profitably solving problems for American industry as possible.

The two major things early YC hoped to provide its founders shelter to pursue were: 

1. Fast experimentation

2. Proof-of-growth

In the case of YC and software startups in general, it turns out that a modest financial package and access to a network of ongoing advisory relationships are more than enough to facilitate both of these. In fact, they work so well that, to many in the Valley, a pile of money and advice is what has become synonymous with the ‘YC model.’ 

And, while that is what YC is doing now, thinking only of these specific resources as the formula to achieve YC’s two goals is probably overfit to the special case of software startups.

When we take a step back, it’s maybe a bit strange to assume that some variant of the financing model that proved optimal for software startups would prove optimal in the general case. The (simplified) examples of products that can be scaled with the most negligible costs that one finds in Econ textbooks are software and certain chemicals/drugs. Additionally, most software companies have the added benefit of negligible technical risks when compared to startups in the hard sciences. Frankly, most scientists would be insulted if you compared the technical risks involved in constructing something like Uber’s early product to what they were working on. 

It’s wonderful that a lever as straightforward as a modest amount of capital can facilitate new company formation in an area like software entrepreneurship — as long as the potential market is large enough. But the dual factors of negligible technical risk (when compared to the hard sciences) and (relative) ease of scaling make software a relatively easy industry to enable via venture capital. Much of the modern software VC industry has structured itself with these dynamics in mind. In fact, as I understand it, the closer the fundamentals of a would-be biotech company are to that of a VC-attractive software startup (in terms of market size and ease of scaling), the more likely it is to get funding. And that makes sense. These dynamics make for fantastic VC investments! When markets have these dynamics, the current VC model works great.

Even a deep tech company like Varda is heavily leaning on these dynamics in its monetization strategy. Varda, the space bio-manufacturing company which Not Boring Capital announced an investment in last month, aspires to be a generalized space manufacturing company. In general, companies in the defense/space spaces are already more VC-investable because something like a $60 million Air Force contract can offset much of your early-stage R&D costs — if you can clear the bureaucratic hurdles to winning a government contract. Even with that contract in place, the initial product that Varda is looking to manufacture that makes their future balance sheet more ‘venture-scale’ is drugs. Drugs have the extremely high margins (and low weight) needed to make this a solid VC investment. And, with all that and a great team in place, Varda does seem to be an extremely compelling investment for deep tech VCs.

But that is also a lot of pieces that needed to come into place. The markets Varda is serving in the short and long run are all very special cases. The joint vertical of  “space drug manufacturer that wins defense contracts to offset R&D costs” is an unbelievably specific category that was seemingly made in a lab to be a VC-investable bet. (It should probably be unsurprising that Delian Asparouhova, Varda’s co-founder, was a VC before starting Varda.)

The number of potential deep tech companies that can meet the financial criteria that venture-scale software companies are able to meet is absolutely minuscule. This group of possible venture-scale deep tech companies is absolutely dwarfed in size by the number of hypothetical companies out there that technical researchers are suited to pursue. The following chart from a 2018 MIT report will not shock you. Certain kinds of companies in certain fields are flourishing. But it is a special set of companies that utilize a specific sliver of MIT’s talented alumni.

As you can see, there is a very large (and growing) pool of ambitious, talented scientists and engineers that have few opportunities to use their degrees to start companies. More/slightly different structures on current VC models is one solution that could certainly solve a portion of the problem — which is what often gets talked about. One such example of a small modification to current models would be earlier/more frequent engagement with applied professors at land grant universities who frequently work on applied research contracts for industrial customers, but are not otherwise involved with the VC ecosystem. However, for most would-be founders and companies, deep tech VCs — in their current form —  are just not offering the right set of resources to start a company in their field.

‘Jet fuel’ is not right for every problem in the short run. But YC’s initial principles, which led them to the specific resources they currently offer software founders, may lead us to the right bundle of resources for a larger pool of would-be deep tech founders.

An effective model to support these technical founders and their ventures does not necessarily need to be as profitable as software VC in as short of a time horizon, but it should aspire to be profitable. Additionally, the model should make use of both the learnings from the last five decades of development in the VC industry as well as the well-functioning deep tech pipeline of the early/mid-20th C. United States. After all, before the boom in software/EE/biotech startups, deep tech was not even a phrase. For the most part, all tech was deep tech.

As Tony framed it, YC’s goals were to provide founders with the right resources for fast experimentation as well as proof of growth. For the case of deep tech research, we can maybe fiddle with these to instead be labeled 1) many rounds of experimentation and 2) product-market fit.

In most areas of science, of course, rounds of experimentation are both longer and more expensive than in software. In most cases, it’s not profitable for a venture capital firm to rain experimentation money on a researcher and see what happens. Additionally, many PhDs are not schooled enough on the business side of technical ventures to know how/where to shepherd their resources and efforts. This is noteworthy because false starts and wide pivots are far more expensive learning experiences in deep tech than in software.

Interestingly, labs in the early/mid-1900s US dealt with these issues quite well. These labs possessed great systems for exposing top researchers to all sorts of nitty-gritty problems and profitably carrying out numerous rounds of experimentation on those problem areas.  These systems — while they did go away for various reasons unrelated to their profitability — were generally seen to be effective and profitable. The two most notable examples are large industrial R&D labs and the portions of universities (particularly land grant universities) dedicated to applied research. Entities I’ve covered on FreakTakes such as the early GE Research Laboratory, Bell Labs, and early MIT’s Technology Plan typify the models to which I refer.

I encourage you to check out each of the pieces to understand the approach of each lab in depth. But, in short:

• Bell Labs had a fantastic system of exposing its researchers to problems all across Bell’s operations — including the day-to-today problems of its manufacturing entities, implementation staff, and elsewhere. Researchers’ regular interactions with people and problems from all parts of the enterprise were not serendipitous, but an intentional outcome of Bell Labs’ management. Researchers at Labs had it instilled in them that it was “a matter of individual responsibility” to find problems that paid off for Labs. Bell also did not rely on researchers to find just the right problem by themselves. Bell maintained a corps of “systems engineers”  to ensure that its researchers were shepherded towards the profitable/useful problems that researchers found interesting (and not just any random problem they found interesting). Bell’s engineers and metallurgists being systematically exposed to ideal research problems — such as the fact that a billion maintenance dollars a year were being spent on fixing a specific kind of wire that degraded in certain climates — was no accident. Bell’s leaders knew their systems of problem exposure and selection were the secret sauce that allowed them to effectively deploy their research talent.

• The GE Research Laboratory, similar to Bell Labs, had excellent methods of problem identification. Like many applied research groups at the time, they were also excellent at brute force experimenting their way to the solutions to the right problems. But the lab also had a fantastic system of managing its basic researchers — such as Irving Langmuir — by granting them what I call a “long leash within a narrow fence.” Langmuir was allowed to work on any basic research project he wanted…so long as it was directly related to a project already underway on the applied side of the lab. His arbitrary selection of bulb vacuums — even though he could care less about the practicality of the project — led him to findings that would not only win him a Nobel Prize in what would come to be known as surface chemistry, but also cut the energy usage of GE’s bulbs in half while tripling their shelf life. 

• MIT, while short on cash in the 1920s, implemented a program called the Technology Plan. MIT’s Technology Plan carried out industry research contracts and consulting work in order to subsidize the Institute’s other research activities. The work was not ‘consulting’ in the style of today’s ‘management consulting,’ but, rather, more in the style of early-1900s engineering or research consulting projects. Companies came to the departments with concrete technical needs, and MIT researchers would often deploy their experimental skills and scientific knowledge to find a solution to the problem. The program was enormously profitable. At one point, 6/7ths of MIT’s Applied Chemistry department’s funds were coming from Technology Plan contracts. Many researchers even preferred this work. Industry’s problems were often interesting. And MIT staff got to use their brains to figure things out for actual businesses with pressing problems that were so important that the businesses were willing to pay a premium to have them solved. 

The (very) general ingredients that these programs had in common were: 

1. They had a pool of talented researchers excited to work on interesting, applied problems.

2. They each had a system of exposing their researchers to profitable, interesting problems.

3. They effectively paired researchers to short-term problems they were well-tooled to solve.

The labs deployed the day-to-day efforts of their researchers on near-term problems that were profitable to some business. The researchers were shepherded to these problems and, over time, learned to identify good problems for themselves in their industry. Each lab encouraged its researchers to do additional exploratory research related to potential customer problems. Each lab also had technically competent, business-savvy individuals who sourced customers, contracts, and problems for the researchers.

While working on problems in this ecosystem, the researchers learned the ins and outs of their industry and how their work could profitably fit into it. Many times, when attempting to solve a smaller-market problem for a customer, researchers would stumble onto a course of research that solved a much larger-market problem. Often, this also happened when researchers used excess funds to pursue problems of their own interest. Those same individuals who sourced problems/consulting work often sourced initial customers for new discoveries that came from the labs’ exploratory research.

Of course, none of these research organizations were explicitly aimed at creating startups. The industrial R&D labs were dedicated to servicing the research needs of the large organizations to which they were attached. MIT’s Technology Plan carried out industry research contracts and consulting work in order to subsidize the Institute’s other research activities. But the scaffolding and organizational structures these labs erected provide a clear template for a modern organizational structure that could meet both of the goals of the original YC model in the deep tech space. 

By pursuing continuous research consulting style contracts, an organization could offer hopeful technical founders the chance to continuously experiment in areas that many businesses objectively need help in and are willing to pay for. One-off problems that require research consulting that are brought to the org could provide industry experience and experimentation resources to the pool of researchers — who would all be aspiring deep tech founders. And any solution that members of the org develop that has a more general use can be taken by interested lab members and spun off into its own company. The goal of the contracts would be as a ‘break-even search function,’ not a profit generator in and of itself. Contracts should be viewed as ways to learn an industry or provide excess funds for experimentation. The endless ‘selling work’ that management consultants do should not be incentivized. Uninteresting and needless projects are unlikely to lead to new company formation.

The 20th C. models I have outlined went away, but they were financially sound and made sense. William Walker, who headed the MIT department that coordinated research contracts and brought in customers, once said, “There could be no more legitimate way for a great scientific school to seek support than by being paid for the service it can render in supplying special knowledge where it is needed.” This was not Walker paying lip service to some concept he didn’t wholly believe in. He would go down as a pioneer in many areas of the young field of chemical engineering practice largely because of his applied contracting work with industry. 

Emphasis on the Technology Plan program faded over the years because, to many in the MIT administration, it came to be seen as a program of necessity. MIT was going broke, but a time came when they no longer needed the money. Most modern universities would not be well-suited to attempt to bring this model back. As Corin Wagen — who received his PhD in Chemistry at Harvard — noted in a blog post last year, nowadays it is common that “policies like [Harvard's] prevent companies from hiring research labs on a purely transactional basis, forcing academics to decouple their incentives from those of industry…research groups cannot simply remake their interests to suit whichever employer they want to attract.”

Universities are not pursuing goals that are anti-industry. But, at many top universities, it has become much much harder, over the decades, to explicitly align one’s research goals to be directly in service to industry — as the entire institution of MIT once proudly did.

But an entity like the model of deep tech VC I describe could be ideally suited for such a task. 

These early-1900s orgs clearly had effective scaffolding for sourcing problems and profitably coming to solutions in markets that did not have software-style fundamentals. But what their researchers absolutely did not have was access to risk capital that is anything close to what we have today. A 1997 BankBoston report on the status of MIT-related startups demonstrates the extreme extent to which, even in the later-1900s,  technical founders usually had to bootstrap their companies through immediate cash-flow positivity or their own savings.

If a researcher wanted to leave MIT or some company’s R&D department to start a company related to their work, financial institutions were not much help. This was often true even if the IP on which the business was based had been technically de-risked and warm relationships with potential customers were in place. (Also, on the point of IP, to leave Bell Labs to spin off a piece of technology was no small thing either. On one’s first day at Bell Labs, employees signed an IP agreement signing over all IP created over to Labs for $1…)

70 years ago, it was extremely difficult to spin off many great and (mostly) technically de-risked ideas because the state of capital financing was not where it is today. Today, we have a different set of problems. Our corporate R&D labs in most industries have taken a step back in how “basic” their research is. Meanwhile, what universities call ‘applied’ research has become much less applied than it used to be. This ‘middle’ of the deep tech pipeline has been hollowed out.

The mere availability of $2 million VC checks is not enough to bring most could-be companies into existence. Practical entrepreneurial knowledge gaps aside, most technical ideas are not meant for “jet fuel.” In the BankBoston dataset of MIT companies, 7/8ths of them remained under 100 employees 15 years into their existence. Many were planning expansions in various ways, but in a modest sense. With friendlier capital markets, some surely would choose to expand more aggressively. But most probably wouldn’t. Most markets are only so big. This doesn’t mean technical ventures with modest initial market caps can’t grow massive, they just might do so on a different time horizon. As the report pointed out, “Of the 17 largest firms [with 10,00+ employees in the dataset], all but 5 were founded by students who left MIT more than 50 years ago and none were founded by those graduating in the last 30 years.”

This was the traditional route to mega-unicorn status. Companies learned to effectively serve some target market for 5-10 years and gradually expanded as the decades wore on — some until they were as massive as Koch Industries. Modern risk capital can surely speed up this process, but it is a process that surely will take longer than the 10 to 15-year time horizon of many current VC funds.

Also, most likely, the base rate of unicorns in deep tech will be smaller than in software. But there seems to be a way to mitigate that issue from a financial standpoint. Even if the base rate of unicorns and mega-unicorns in deep tech is smaller, it’s very possible that the mid-20th C. inspired model could lead to a much higher base rate of break-even and moderately profitable companies than is seen in software VC. Looking at the 2018 MIT report (with data on MIT-related companies from 1940 onward), it seems only a minority of these MIT businesses failed in the first 5-10 years (possibly as little as 20% of businesses after 5 years and 30% after 10). While the data are possibly crude and no specific number should be trusted too closely, a failure rate this low may not be so surprising given that so many of the companies were being started by those who knew an industry well and began the companies’ life cycle with immediate cash-flow positivity. 

In summary, while the base rate of unicorns in deep tech is likely lower and it will likely still take at least 20 years for most potential unicorns to become unicorns with modern risk capital, there are ways to drastically reduce the number of investments that go to zero — with the median case being a moderately profitable company. That limited downside, combined with the upside being companies like Koch Industries, could make this model workable.

Most portfolio companies would never grow as large as Koch Industries, and that should be ok because the median case in this model is still extremely compelling. The median case is far more ambitious and interesting than the mere $300,000 yearly profit after expenses and salaries on its balance sheet would have you believe. A model like this has the potential to create many small, profitable technical companies that could effectively utilize the skills of otherwise underemployed PhDs and engineers. The model could help rebuild the chasm between many areas of STEM research and American industry that has formed over the past 50 years. These companies could be bastions of profitable engineering and scientific excellence, leveraging Bell-style problem finders and effective sales operations. 

If they grow big, great! But if all most of the portfolio companies end up doing is collectively solving as many problems as possible and their researchers make a good living doing it…the right kind of investors might find that exciting! For that mission, they might be willing to have their money held for a longer term or be open to a different style of returns than in software VC. After all, few investors partake in deep tech VC because it is the fastest or surest way to get rich. Many, frankly, want to do their part to help America recover its status as a powerhouse of great industrial research in all technical areas.

(The basic models and discussion that went into this section were worked on jointly with Aakash Pattabi, who understood better than I what kinds of simplified models would be productive for this brief exploration.)

The return profile of a VC model like the one I describe, as you’d expect, looks very different from traditional software VC. The following graphic shows a simplified model of what has come to be normal in something like Series A software VC. In this style, as many know, the hits make the portfolio. Investors are happy to write off 85+% of their checks as most likely going to zero. This is ok because the big hits make or break the fund's success. In the following case, a 5% home run rate — in this case over a ten-year time horizon for a fund that writes 100 $1 million checks — can help the fund return $510 million after ten years.

That case, of course, is a case of things going extremely well. Most funds do not achieve anything close to that kind of success on average. If a VC does not find those few companies that “return the fund,” things can look much more bleak. 

In the William Walker model, one can “return the fund” over a ten-year time horizon with quite modest assumptions. The returns in this basic model are much less sensitive to ‘home run’ performance than in the software model because, not unreasonably, I assume that a deep tech home run in the William Walker model might be a company 10Xing its initial valuation in the first ten years — rather than 100Xing. What performance for a William Walker model fund is much more sensitive to, on the other hand,  is the growth rate of more standard companies in the portfolio. But these standard companies need not be extremely large companies. In this case, I imagine:

• 50% of companies achieve the goal of 50% total growth (over the first ten years) from their modest initial valuations

• 15% of companies achieve moderately ambitious growth goals, 3Xing their initial valuations

• 5% achieve 10X valuations

• And 30% of companies crash and burn

With these assumptions, we observe a return profile that is almost comparable to buying and holding index funds.

The assumptions and outcomes in this hypothetical scenario are only to give the reader an idea of the thought process and goals that one would have if they were looking at this model as an investor who wanted to match stock market returns. Of course, one would need to deploy this “experiment” in the real world to observe what truly realistic assumptions actually are.

It could go wildly well. The following is a version of the model that  would justify scaling the model with reckless abandon if proven true. These (still not crazy) assumptions imagine the following over a ten-year time horizon:

• Home runs were 1/5th the growth rate of software companies rather than 1/10th

• Moderately ambitious growth multiples are 5X rather than 3X

• And the base growth rate was a company 2Xing in 10 years rather than 1.5Xing

With that, the ten-year return on the fund is $275 million on a $100 million investment.

Things could go even better than this model. For example, the Fraunhofer-Gesellschaft — Germany’s system of applied research institutes that serve a similar function to what the Max Planck institutes are for German basic research — report a survival rate of 90% after five years for their spinoff companies. A crash and burn rate far below the 30% number I used in the model could make the financial outcomes even more compelling.

Of course, there exists a world where this could also go very poorly from an ROI perspective. As you’d expect, the worst possible scenario is one in which the companies have a crash and burn rate similar to the hit-or-miss world of software VC — where companies are encouraged to swing big by design.  The following graphic shows a version of this going badly, with half of investor money disappearing. With the same return multiples that netted a $170 million total fund return on $100 million invested, the following assumptions yield a very different financial outcome:

• There are no home runs

• 5% of companies grow moderately ambitiously (rather than 15%)

• 25% of companies grow by 50%

• 70% of companies fail

If one picks investments that go to zero too often, then this is not viable. But if one picks businesses that — as the old MIT-style companies were — are almost immediately cash-flow positive and launch with customer bases and warm leads, then the model is extremely compelling. This is why the pivotal first step in making this a profitable investment class is a style of labs and consulting contracts similar to MIT’s Technology plan, Bell Labs, or early GE Research. These environments produced engineers and scientists whose ideas and ventures could be considered ‘safe bets.’

There would naturally be many small differences in the operation of this model that differ from software VC. One would imagine that — as in the early era of Silicon Valley when VCs were funding more chip/hardware companies — William Walker funds might take a higher percentage of the companies they invest in than is normal in software and, if need be, write fewer checks to ensure companies have the resources they need from the start. This is a reasonable approach when one assumes that far more of its companies will survive the early years than in software VC. Additionally, dilution of one’s equity is a much more negligible factor in this style of VC. Lastly, the styles of businesses being launched would be much more capable of issuing things like yearly dividends to investors. So, even if exits to financial actors like search funds of low-to-mid-market PE firms take a while to materialize, some liquid returns can be seen in the interim.

There is a chance that the William Walker model turns out to be attractive in its own right from the POV of a self-interested investor. But this “experimental” fund idea should be extremely compelling, in particular, to the affluent groups that have, to this point, been losing money in an effort to fund technological growth. Many of these individuals have either been directly funding research through philanthropic giving or been contributing large amounts to ambitious deep tech VC projects knowing the investments are not as profitable as alternative investments.

Investing in this model has the potential to: 

1. At the minimum, bring into existence a new style of org where under-employed, top minds can start companies that solve the problems of American industry even if they are only modestly profitable.

2. Possibly prove out an extremely profitable model of VC. And, even if it doesn't, the organization would have an extremely high chance of breaking even as an experiment — as opposed to many experiments that have no chance of returning the donor’s money back to them under any circumstance.

Maybe this is an experiment that launches a new area of venture, or maybe it just turns out to be a new and clever way to enable top research minds to generate industry solutions — but doesn’t work as a for-profit company. Regardless, it is an experiment worth running.

This alternative approach to deep tech VC has the potential to marry the best of the old and new. On the day-to-day, doing consulting project work like that of MIT’s Technology Plan projects or Bell Labs-style assignments could bring in money and experience. This provides would-be deep tech founders runway to experiment and on-the-job industry know-how. In general, the goal of the org should be to produce profitable new companies in the long run while (at least) breaking even in the short run on its contracts. When they spin off, the companies would ideally be further along than many deep tech investments currently are — originating with a non-zero customer base and fewer technical risks.

In that case, providing risk capital to the spinoff for 25% of its equity could be (very roughly) one-third as likely to go to zero as the standard deep tech check currently. Each of these successful, break-even companies is one less hole in your balance sheet and one more chance at a company that, in the longer run, can grow into a mega-unicorn like Koch, Gillette, or (even) Campbell’s Soup did in the MIT data.

Not all university labs, of course, have forgotten how to do this style of work. Certain PIs and their labs do large amounts of applied research and continuously graduate grad students who spin off work that originated in the lab. And that’s great! Individuals just like that are the ones who should help guide an initiative like the one I’m describing. 

But what those labs are doing is not the default in academia. Incentives often lead researchers elsewhere. Lab heads like these are proving that it’s doable. But we could do with a hundred more of them across many fields.  The other week I spoke with a Big Ten Mechanical Engineering Professor who brings in industry contracts and whose students have gone on to collectively raise significant amounts in VC rounds. He felt that he had a much better sense of industry needs and how VCs thought than most of his departmental colleagues. I asked him if there were colleagues in his department whose labs he believed he could take over and get to the point that they were raising as much as his lab from VCs. He replied, “Several.”

I’ve also spoken to several professors at land grant schools who took over academic labs after having worked in industrial research for a portion of their careers. Even after transitioning into academia, the industry from which they came continued to lean on their research and hiring the grad students they trained. In various ways, they’ve all (politely) said that most researchers in academia just don’t really understand how to make themselves and their work genuinely useful to industry while working at a university. Their incentives and training seem to lead them elsewhere. 

All of these stories point to an opportunity! The William Walker model can take advantage of top research minds to solve very different problems in a very different organizational structure that is built to train future founders, preserve investor capital, and serve industry as its main goal. 

The right combination of older researcher/advisors and (the right) big-name financial backers could surely provide sufficient ‘signal’ so that some early customers with interesting projects would be willing to come on board. After all, the brands of people like Eric Schmidt have come to extend beyond the borders of Silicon Valley. And there is also a subset of applied-minded professors already known to produce novel, profitable research within their own industries. This William Walker model of deep tech VC, in its early years, could leverage both of these reputational signals to bring in early projects and get the experiment off the ground. (Of course, the young entrepreneur/researchers would do the lion’s share of the actual work on the projects, but an initial stream of projects would be vital to get the experiment off the ground.)

It would be ideal to set up a full William Walker-inspired org, from the start, that was equipped with full-time research heads to lead contracts, a small research facility,  a pool of capital to invest in spinoffs, and full-time research staff who aspire to be entrepreneurs. However, if it were more workable, there are more measured approaches to help the org “ramp up” to full capacity. At the minimum, the org should start with:

1. Salaries and facilities for the full-time research staff (who are hopeful entrepreneurs). The number of research staff could be small — around three to five people — if the org decided to focus on one specific area of research consulting and spinoffs.

2. (At least) part-time research heads — who would often be university PIs — who have experience working with industry contracts and could bring in initial contracts on which the full-time research staff could get started working.

3. Access to some form of capital that is excited about the experiment and willing to invest in the spinoffs of the research staff. If the William Walker org does not have a fund itself, it should establish some form of relationship to friendly capital pools that lead the org to believe investments in deserving spinoffs should materialize. However, since outside capital may be timid, it would be best if the org could initially raise at least a small fund to ensure it has enough money to invest in the first couple of spinoffs.

Other variations of this could work, but as much as is possible, no half-measures should be taken when it comes to the research staff. The potential entrepreneurs should be spending all of their time on this — it shouldn’t be some side-show to other work they are doing.

One could try to change academia to better correspond with these goals so professors and labs strive towards them more often — rather than tenure or filling amorphous ‘holes in the literature.’ One could also try to restructure large companies to more effectively pursue exploratory research and, when appropriate, cleanly spin the work off with a friendly deal and minimal IP annoyances.

But an entirely new organization dedicated to the task is probably best. Specifically, an org that is:

• Inspired by early YC’s goals

• Deploys the structures of early/mid-1900s applied research orgs

• And leverages the learnings of modern risk capital

If Tony’s article is correct, it seems that YC’s original thesis was about how to “solve the money problem” for hackers, not building huge companies. Talented engineers, instead of working for some pointy-haired boss on uninspired engineering work, could take a risk and earn themselves a chance to work on cool problems with a level of financial security. That was the idea.

The same history that leads me to propose this specific structure also leads me to believe this can be done profitably. Will it be as profitable as software VC, index funds, or real estate? I’m not sure. But, as it stands, current deep tech VCs cannot promise anything like that either. This approach offers a fresh perspective, but a perspective that is based on a historically profitable approach.

As it stands, for many LPs who invest in deep tech, it is about more than just money. Many hold the staunch belief that this country is meant to be a place that feverishly produces technical innovations across all scientific and engineering areas — not just two or three! And I think so too. For the most part, it was our consistently fantastic feats of applied science throughout the 20th C. that lead us to this mindset in the first place. That’s why, to me, when attempting to structure an organization that can enable the creation of technical business ventures across all industries, America’s 20th C. systems of applied science are the first place one should turn to for inspiration.

That history, combined with our expanded systems of risk capital, lead me to this model — which I’m calling the William Walker model of deep tech VC.

It might not be as profitable as YC. But it does allow technical individuals to work on hard, useful problems. And it can give them a level of financial security in doing so. It’s a very legitimate way to satisfy the spirit of both of YC’s original goals. 

So, it’s an idea worth entertaining.

Thanks for reading:)

But one last thing!

I absolutely believe an organization like this should exist and would happily work with individuals excited to make it happen. I’m starting an offline list of potential markets in which researchers in the William Walker model could potentially be profitably deployed. So, if any folks do think this is interesting but don’t know exactly what specific markets the organization could start in, I’m already working on that and would love to share what I’ve come up with.

On that note, if anybody reading this works in some area like low-to-mid market industrial PE, I’d love to pick your brain on some things. In my market exploration so far, I’ve found individuals in these sorts of roles useful starting points. (My Twitter can be found here)

Lastly, since most of today’s piece was aimed at describing a new model of scientific organization, I’ve included an additional End Notes section with a couple of tamer takeaways that might be useful to existing deep tech VCs who do not intend to wholly change their model, but are compelled by the historical details contained in today’s piece.

Of course, there are also ways to more easily fold the learnings from this piece into a modern deep tech VC to achieve at least some of the effect — with significantly less effort. But I wanted the chance to pitch the most extreme version of the idea first. But, of course, the learnings that inform this atypical idea can have more standard VC applications as well.

My first piece of advice is this: Deep tech VCs should establish much closer relationships with researchers who work more on applied contracts at land grant universities. Many of these professors continuously generate profitable and commercializable IP that plugs into the industrial process, but they are not necessarily plugged into the deep tech VC community.

I recently spoke to one Big Ten Material Science professor who said that when they hosted an ‘industry day,’ many companies who eventually became customers of the department’s services attended. But he couldn’t recall a single VC as having attended. Maybe one was there, but this professor couldn’t recall any. Maybe these professors’ work was not venture-scale, but I got the sense that many of their projects had never even come to the attention of a deep tech VC.

These labs are often sitting on know-how/skillsets that industries find usable and continually work with them to solve problems, but they are generally not plugged into the deep tech VC community. Relying on them to find a VC to share their learnings with/pitch ideas to is not a reliable approach.

Given how many VCs spend the majority of their time sourcing deals, having discussions with people in industries that they know have a low probability of leading to a deal, etc., it seems like these individuals should be actively sought out. If I worked at a deep tech VC fund as an analyst, I would spend a large amount of my time sourcing deals through this alternative channel if I could.

On that note…

Many of these researchers are working on potentially commercializable problems on somebody else’s dime…use that! A VC can shape a project for many years before they are the one funding it.

I could imagine a world where a deep tech VC does something like ingratiate itself with the MIT master’s cohort. Many of those students, in a perfect world, would love to be deep tech founders. In a hypothetical world, a VC could attempt to shape the master’s thesis projects in the cohort — which the university is not as protective of as it is of its PhDs — as their projects are underway. For the (many) interested students, a VC could attempt to shape their research projects to be more VC-investable at the end of the two-year program. I could imagine this generating at least one or two noticeably de-risked investments for a VC firm. After all, as things stand, many of these students do not have much knowledge or direction in choosing initial project areas or choosing the ‘right’ next steps as the project goes on to make the project a commercial success.

These students seem to be one of the purest examples I can think of as having extreme talent and energy, but limited industry knowledge. Servicing them and their projects seems to be extremely aligned with the initial goals of YC — but for the deep tech space — as anything I can think of.

Of course, in the role of a VC, I would also attempt to informally do this with as many interested professors and their research directions as I could — years before I was even considering writing a check. Many professors are ‘over’ academia and would be happy to shape their research directions (on the NIH/NSF dime), over the course of a handful of years, to be more about solving the technical risks of a hypothetical company than upping their h-index. If I were a VC, I’d spend a massive amount of time working with them. If the issue many deep tech VCs have is that the deals coming across their table have too many technical risks than the VC is able to profitably fund experimentation to mitigate, this seems like a way to clearly be proactive about improving the quality of deals that cross your desk — if the VC takes a sufficiently long-term view. 

These are just a few of the tamer learnings that come to mind that could be folded into interested VC operations right now

Thanks for reading:)

As always, I would love to speak with any readers interested in applying any of these learnings to their own organization.

Subscribe now

Share

• Gilliam, Eric. “An Alternative Approach to Deep Tech VC.” FreakTakes Substack. 2023.

Check out my guest post for the ChinaTalk Substack on recent innovation in the sport of jiu-jitsu and how it can help us think about OpenAI’s future. In a world where progress in GPT research drastically slows, what should we expect from OpenAI?

Some think it spells doom!

In my eyes, even if something like that happens, OpenAI and DeepMind are still the odds-on favorites to make the next big AI breakthroughs as well. Check out the piece to learn why! It draws on recent innovations in jiu-jitsu as well as the history of mid-20th-century industrial R&D labs.

Enjoy!

Thanks for reading FreakTakes! Subscribe for free to receive new posts and support my work.

From a historical perspective, the burden of knowledge hypothesis is flimsy. For those who don’t know, I’m referring to the hypothesis that scientific ideas get significantly more difficult to find as knowledge continually progresses. My lack of faith in the hypothesis does not stem from some knee-jerk reaction against hypotheses that imply human progress will slow down. In fact, I used to assume that the burden of knowledge hypothesis was true.

But as I’ve continued to explore the history of early-to-mid 1900s science — which largely occurred in a period pre-dating the best micro-level data we see in the economics of innovation literature — I’ve become much more skeptical.

I’m not saying that as more and more things are known in a field, it doesn’t take longer to learn each of them. That is, of course, true. It takes longer to learn more things than it does fewer things. But what I am saying is that the accumulation of additional knowledge as the primary driver of our slowing rate of scientific progress is flimsy as a hypothesis. There are other theories that can fit the trends in the data just as well; one of these theories was put forward by scientists living through the mid-1900s who felt they were observing the onset of new branches of science becoming harder and harder to create.

Accounts I’ve come across of WW2-era physicists complaining about academia becoming a worse place to do creative or interdisciplinary work do not point to the growing size of the literature as the reason. Interestingly, their complaints had nothing to do with the growing size of hypothetical ‘idea space’ at all; their complaints often blamed administrative changes. They point to things like growing conference sizes or scientific meetings being held for narrower and narrower sub-branches of work rather than broad areas — human organization problems.

Today’s piece will put forward an alternative hypothesis as the primary driver of the scientific slowdown: the human systems hypothesis.

To inform different aspects of the hypothesis, I’ll examine:

• What the existing evidence from the economics literature does and does not say about the burden of knowledge

• Why many great mid-1900s scientists blamed human organization problems for the growing difficulty in asking new, big questions

• Why the exact nature of math and physics’ ‘divorce’ supports the human systems hypothesis

• And how the field of poetry — whose rate of progress is likely not vulnerable to burden of knowledge issues — suffered a similar fate to that of science when institutions like those in late-1900s science were introduced

Let’s get into it.

Subscribe now

Share

• For those who listened to the Edison podcast, please feel free to comment or DM me topics that you’d like to see covered on the podcast!

• For the sake of this piece, ‘interdisciplinary work’ should be defined as work that draws from and/or is relevant to multiple disciplines.

• This piece, as always, is done in concert with the Good Science Project

I should state upfront that I don’t expect this hypothesis to win over a majority of the economists. The reason is that there is not exactly a single, clean ‘identification strategy’ to verify the theory. Most imaginable ‘natural experiments’ in the modern context would not approach any kind of clean answer. An unwieldy number of changes occurred in the culture, organization, and incentives of science that affected an entire generation of researchers and have continued to affect each generation afterward. Any single accidental change to one component would still leave most others intact.

So, instead, I employ a style of argument more familiar to later-1900s political science literature in which I attempt to fold in all available data into the theory, but am very willing to let relevant qualitative information inform pieces of the hypothesis when it makes sense to do so. This style of work was often used in areas such as later-1900s comparative economic development work. In a case like this, they had before them:

• A very specific and biased set of countries that had achieved 'developed country' status

• Some high-level data on most countries

• Some more granular data on some — mostly developed ones

• Detailed qualitative histories from many countries going back centuries

• But no data that was nearly specific enough or numerous enough to begin answering the question in a purely quantitative way

But the stakes were large and the questions were worth answering as best they could, so those studying economic development tried and used all the information at their disposal. For those readers familiar with the subject, works by people like Mancur Olson exemplify this tradition and are examples of this style done at the highest level.

With that explanation out of the way, let’s explore the argument.

To start, I should briefly go over what has been confirmed by data in the existing economics of innovation literature and what has not on the question of the burden of knowledge.

Whether ideas are getting harder to find or not is very much an open question — or at least has some notable caveats. What researchers like Bloom et al. have confirmed is that our research productivity across the board very likely seems to have been decreasing since about 1970 — give or take. As we’ve spent more money and more researchers than ever have poured into fields, we are getting less and less productive.

The following graphics from Bloom et al.’s famous paper demonstrates this disappointing trend on the micro-level, within fields. Some of the y-axes variables are more convincing than others — i.e. many would believe that transistor count is more robust than counting the number of clinical trials — but the general trends the paper observes seem quite robust and clear: we’ve been getting steadily less impressive outcomes compared to what we’ve been putting in since about 1970.

Now, all of this micro-level data only goes back to around 1970-ish — a little before in a few cases. A portion of the introduction of their paper paper — which introduces the topic using very macro, national trends — goes back as early as 1930. However, many question the ability of economy-wide metrics like TFP growth on one axis and total US expenditure in R&D on the other to be useful in understanding these questions since the metrics capture so many other trends that one can’t control for — particularly in a mid-1900s American science landscape where how research was used and invested in was rapidly transforming. The authors were aware of this, which is why they dove into the micro-level data shown above. The result was a fantastic and robust paper identifying a massive problem in American science from about 1970 onward.

As those micro-level charts show, in almost all research areas explored in the literature, we seem to be getting as little research productivity per dollar as ever. Of course, these fields do not cover everything. But they were selected because they were generally measurable. And I think that approach makes a lot of sense. From those graphs, it seems very likely that we’re not as productive as we used to be. That is hard to argue with — I wouldn’t argue it.

The natural next question is, “Is this our fault?”

Are ideas getting harder to find by some almost natural force? Or have we gotten worse at doing science somehow?

The ‘natural force’ supporters often believe that ideas just get harder to find because previous generations have somehow picked all the lower-hanging fruit. This implies that the lack of research productivity is not humanity’s fault, it’s just that we’re increasingly fighting an uphill battle.

Does the economics of innovation literature disentangle this question at all?

In short, not exactly.

When you trace ‘burden of knowledge’ up the citation tree in the economics literature, all roads lead you to Ben Jones at Northwestern. His big initial paper on the topic, for which this section is named, and a later paper called Age dynamics in scientific creativity confirm some things about the burden of knowledge, to be sure, but they don’t do anything to pin down the burden of knowledge as the primary factor in the slowdown of scientific ideas.

The first paper is not causal. It is a structural modeling paper with a portion dedicated to the analysis of some patent data from a high level. The paper, in a nutshell, posits what a good model of burden of knowledge-like dynamics would be if the dynamics existed. The data don’t confirm strong causality, simply that the theory put forward more or less can mesh with a reality in which the trends in the data exist. The data that the author used to fit his theory to was basic patent data noting things like team size and age of invention.

Even with this basic data, the theory hardly fit cleanly. The biggest reason I say this is that, even though he acknowledged some patent areas were more complex than others, he noted that:

Innovators in the same cohort choose the same amount of education across different areas of application. What is particularly surprising is that this result holds even though some areas may feature a greater difficulty in reaching the frontier of knowledge.

He attempts to fit this observation into the theory with a hypothetical explanation of how innovators could allocate themselves in a way that could make this consistent with the theory.

You get the idea. The paper surely empirically shows certain relationships — and I do genuinely love the paper — but it’s far from a ‘nail in the coffin’ for competing theories.

However, Jones did extend the idea further. He, along with Bruce Weinberg, wrote a fascinating follow-up paper using all of the physics, chemistry, and medicine Nobel Prize-winning discoveries from 1900-2008 as their dataset. Some relevant trends the authors observe are:

• The proportion of Nobel-winners who make their Nobel-winning discovery by the age of 30 or 40 is going down drastically.

• Theoretical contributions are associated with younger researchers.

• Physics Nobel winners showed a substantial burst in youthful productivity after quantum mechanics burst onto the scene in their field.

The following graph displays this youth boom in physics. The lower line is the percentage of Nobels given out to researchers who made their winning discovery by the age of 30, and the upper line by the age of 40. The quantum era seems to have initiated a steadily rising level of outstanding discoveries from young researchers, peaking in the 1930s.

I’ll re-examine more specific evidence from this paper later in the piece — once I’ve further fleshed out the historical argument for the human systems hypothesis. For now, I’d like to bring attention to a key caveat that the authors noted in explaining how this new branch of physics affected the trend of Nobel-prize winners getting increasingly older:

Age dynamics might be associated with changes in a field’s foundational knowledge, which may typically expand with time but also may contract in cases where new knowledge devalues old knowledge. Heisenberg [who won his Nobel for work done at 23 and 25], for example, nearly failed his PhD examinations at age 21, because he knew little of classical electromagnetism; his contributions in the subsequent 4 years suggest that training in classical physics may have become less salient.

This is a fantastic observation from the authors and one that often gets lost in discussions of the burden of knowledge. The concept of the burden of knowledge and its relationship with scientific productivity is one that cannot be disentangled from the concept of scientific branch-creation. The older the branch, the more the 'burdensome' of the burden of knowledge is; the younger the branch, the more negligible it becomes.

This observation, along with the oddity that fields of varying complexity have similar training times in Jones’ first paper, will fit very cleanly into the human systems hypothesis.

The reasons fields seem to grow apart, when one reads the historical sources, are quite interesting. The growing apart process often seems much more abrupt than the rate of change in the scientific fields themselves.

The opinions of many of the researchers themselves are the first point that lead me to this observation. The growing apart of fields in the years leading up to 1970 did not go unnoticed by researchers at the time. Researchers who lived through great eras of their fields in the early-1900s and continued to research through the 1970s point the finger at things like conference sizes and not the size of the literature as responsible for it becoming harder to do truly exploratory research. This group is interesting because many of them started out in fields that were smaller but explosively growing in terms of ideas and continued to research into the era in which the number of publications exploded but the rate of big ideas in their fields did not seem to be exploding.

To be clear, the growth in the size of the literature was surely noticed…

It’s material that great minds who thought critically about their fields chose not to complain about this growth in the size of the literature as much as things like conference sizes and incentives. Just because the early-1900s was an era in which one could try to keep up with all the literature in related fields just by reading it, that doesn't mean that's what researchers usually did. The accounts in this section paint a picture of researchers who largely kept up with science by keeping up with people.

The second point that leads me to this observation relates to the importance of how fields grow apart. The exact nature of how fields come together and grow apart often has more to do with people and incentives than ideas themselves. A supporter of the burden of knowledge hypothesis would likely expect that fields that were formerly very interrelated would grow apart somewhat gradually as keeping all the ideas in one's head became more and more untenable. Fields separating, usually, should not be for some interpersonal reason or internal politics. But this abrupt separation is often exactly what one finds.

I’ll use the accounts of mid-1900s physicists to explore both of these points.

Many World War II-era physicists were not only practicing researchers in the small and bustling field of early-1900s physics, but continued on as researchers well into the 1970s. As there was an explosion of publications in their field, the researcher accounts I’ve come across do contain complaints from the old-guard researchers. Several prominent examples note the feeling that academia was increasingly becoming a worse place to do creative or interdisciplinary work, but the complaints don’t pertain to the number of publications themselves — although they did grumble at the changing state of publications and how smaller findings were published more often.

In practice, they more so blamed the human organization problems — essentially administrative issues — that they saw all around them. The growing conference sizes made it much more difficult to keep up with adjacent fields and scientific meetings. Seminars began to cater to narrower and narrower sub-branches of work rather than broad ones.

These were the places that many researchers leveraged to actually keep up to date on new work and problems in their fields as well as others. But, as money began to funnel into their field in the post-War era, there were more and more researchers and logistical decisions had to be made on how to do things like run conferences and decide who sits in what seminars.

The following Richard Feynman excerpt — taken from a 1973 oral history interview, which was one of a series of interviews between Charles Weiner and Feynman — goes into why, in the early 1970s, Feynman felt physics conferences had begun to grow far less useful than they were during the initial interviews for the series — where Feynman had told positive stories about the state of conferences as recently as 1956:

Weiner: How are they [conferences] going from what you’ve seen over the years? Is this the same kind of continuing tradition of the same kinds of people coming together? Because in the early period — we talked about the meeting where you got up and said: “Mr. Block has an idea that I would like to tell you about,” and you talked at the following meeting on that, and these were very exciting things. Has it continued in that same tradition?

Feynman: No, they’ve gotten too big. For example, they have parallel sessions which they never had before. Parallel sessions means that more than one thing is going on at a time, in fact, usually three, sometimes four. And when I went to the meeting in Chicago, I was only there two days before I broke my kneecap, but I had a great deal of trouble making up my mind which of the parallel sessions I was going to miss. Sometimes I’d miss them both, but sometimes there were two things I would be interested in at the same time. These guys who organize this imagine that each guy is a specialist and only interested in one lousy corner of the field. It’s impossible really to go — so it’s just as if you went to half the meeting. Therefore half is not much better than nothing. You might as well stay home and read the reports.

This is no small thing. Feynman, who never religiously kept up with the literature, was one of many who used these conferences as a primary way of keeping up with the changing state of things.

We find a similar account in Warren Weaver’s autobiography.

In Weaver’s book, once again, we hear a 1970-ish account of a researcher who lived through and was active during the golden era of physics complaining about growing conference sizes, not the growing number of publications, making it more difficult to keep up with related scientific areas. Not only were conferences growing in size, but, in response, they separated various disciplines into their own conferences to make conference logistics more manageable.

Many economists tend to believe that it is the sheer size of the literature causing the problem. But both Weaver’s and Feynman’s accounts seem to more directly blame the highly statistically correlated, but fundamentally different, change in conference dynamics and logistics that came with more and more researchers flooding into the community.

Weaver wrote of the early AAAS, which he was once the President of:

This vast organization, with the largest membership of any general scientific society in the world, embraces all fields of pure and applied science. In the earlier days each branch of science conducted intensive sessions at which its own specialized research reports were given, and there was also some mild attempt to organize interdisciplinary sessions of general interest. Then the meetings got so large that they almost collapsed under their own weight. One group after another, first the chemists, then the physicists, the mathematicians, the biologists found it necessary to hold other and separate meetings. Attendance fell and the function of serving as a communication center between the various branches of science became less effective.

At a time when publications had roughly 10x’d, both Feynman and Weaver chose to talk about the masses of people flooding their conferences (and then conferences being held separately) rather than papers flooding their journals.

While the two were not active in the research community throughout the entirety of the decades-long explosion in journal articles — a scientific career is only so long — they were around for enough of the explosion for their accounts to be quite telling. If you take seriously the ability of individuals like Feynman and (especially) Weaver to diagnose issues in their fields — Weaver ‘saw the whole board’ well enough that he was the primary driver in jumpstarting the field of molecular biology as a grant funder — these accounts are surely not evidence to be disregarded.

Fields that used to be largely integrated can grow apart for a variety of reasons. On one end of the spectrum, fields can grow slowly grow apart. On the other end, they can rather abruptly divorce. As with a real divorce, divorce does not mean that the two fields do not at all interact anymore, but the frequency and amicability of the interactions do change drastically and abruptly.

The nature of how two fields that frequently produced interdisciplinary research come to separate is very relevant to how much faith we should have in the burden of knowledge hypothesis. If the sheer number of publications is the driving force behind the drop in research that combines disciplines, then a strict burden of knowledge hypothesis supporter would probably expect the amount of interdisciplinary research work to drop (more or less) in lockstep with the number of papers in those fields. However, that is often not what we see.

Let’s continue with the case of math and physics in the mid-1900s. These two once-married fields did grow apart, but it had little to do with the size of the literature.

Robert Dijkgraaf — a mathematical physicist, current Dutch Minister of Education, and former Director of the Princeton Institute for Advanced Study — recently published a book chapter in a volume about the late Freeman Dyson. The chapter talks about the ever-exploratory Dyson’s presence at the Institute and several areas of change during his tenure that he decried. One noteworthy change the mathematician-turned-physicist fought against was the divorce of the two fields within the Institute.

Dyson, a dear friend and mentee of Feynman, said this of the growing propensity of mathematicians and physicists to contribute to each other’s field in the 1972 Gibbs Lecture to the American Mathematical Society. The lecture was titled “Missed Opportunities”:

I happen to be a physicist who started life as a mathematician. As a working physicist, I am acutely aware of the fact that the marriage between mathematics and physics, which was so enormously fruitful in past centuries, has recently ended in divorce.

Dyson was the mathematician who proved the equivalence of Feynman diagrams and the operator method of Schwinger and Tomonaga. His style of work, answering physics questions that required the deep toolkit and experience of a solid mathematician, was not uncommon in his early years. One could view the work of von Neumann or Dirac in a similar vein.

Robert Dijkgraff elaborated on the events leading up to the divorce as follows:

As physics became messy and muddy, mathematics turned austere and rigid. After the war, mathematicians started to reconsider the foundations of their discipline and turned inside, driven by a great need for rigor and abstraction. The more intuitive and approachable style of the “old-school” mathematicians like von Neumann and Weyl, who eagerly embraced new developments in physics like general relativity and quantum mechanics, was replaced by the much more austere approach of the next generation. This movement towards axiomatization and generalization was largely driven by the French Bourbaki school, started by a group of young mathematicians in Paris in the 1930s. They had set themselves the task of rebuilding the shabby house of mathematics from its very foundations, under the motto “structures are the weapons of the mathematician.”

This arbitrary shift in taste — surely fueled, in part, by a post-war glut in American funding for research much more basic than before — encouraged young mathematicians to pursue problems in those areas rather than working on physics-related problems. The relationship between mathematicians and theoretical physicists began to change in noticeable ways. Dyson described how this affected his own life at the Institute in an interview with Natalie Walchover:

We had a unified school of mathematics which included natural sciences; it included Einstein; it included Pauli and other physicists. So, the mathematicians and physicists in the first 20 years of the Institute worked together. But at that time when I became a professor it just coincided with the time when Oppenheimer became director and there was a divorce largely occasioned by the fact that Oppenheimer had no use for pure mathematics, and the pure mathematicians had no use for bombs. So, there was a temperamental incompatibility. For one year, after I became a professor, we still had meetings of the whole faculty, but after that they were separate. We had meetings of the physicists and meetings of the mathematicians, no longer speaking to each other. So, that was sort of a local manifestation of the divorce.

Things were changing.

For a while, Dyson was exempt from exclusion on either side — he was a ‘real’ mathematician who simply had success plying his craft in particle physics, just as von Neumann once won fame for plying his mathematical craft in various non-math fields. But, eventually, this new wave of mathematicians did not want him either. He recalled in an interview with his son and science writer, George Dyson:

But I had the advantage, of course, that I was also a mathematician And the mathematicians knew that I had done some useful stuff in pure mathematics. And so, when I appeared here as a professor, I was actually invited to the faculty meetings in mathematics. And for several years [in the mid-1950s] I used to go to the mathematicians’ faculty meetings. And they were quite friendly. They considered me as one of them…So, for the first two years or so I functioned as a mathematician. But at some point, I was, I don’t remember how this happened, but I was dropped, and it was made clear that they didn’t want me at their meetings. So, they regarded me, at first, as being on their side, but then afterwards they found I wasn’t.

In this Dyson description, we see strong hints of academia beginning its march toward the siloed beast we know it as today.

In general, a divorce like this is not symptomatic of a ‘burden of knowledge,’ but, rather, a changing set of incentives and systems taking hold. One could say ‘maybe the fields would have grown apart (at some point) anyway if it weren’t for this systems-related divorce’ just as they could say ‘maybe Weaver and Feynman would have eventually started complaining about the size of the literature as it went from 10x to 30x the original size’ — even though they didn’t complain about it 10x’ing. If one wants to play counterfactual history in that fashion, they can. But we should take the events as they actually happened much more seriously.

With the American boom in post-war basic research funding, obligations for American researchers to have specific, short-term applications in mind for their work became lighter than ever. So, if a field was made up of individuals whose propensity leaned towards mathematical elegance and rigor over real-world applications, they now had more leeway to do that kind of thing. In fact, it might even be encouraged.

This style of thinking can quickly lead to intra-disciplinary work — and the creation of inward-looking sub-branches of science — rather than work that connects fields of separate departments or builds towards concrete applications. American academia was becoming a different world: more people, more funding, more bureaucracy, more silos.

Gerald Holton, writing at the end of a boom period in several scientific subjects in the mid-1900s, felt that ideas were getting easier to find. He was not the only researcher to think this. It seemed that whenever scientists found a new idea, in line with certain combinatorial theories of innovation, they now had a new hammer to use on existing nails. Alternatively, they could re-combine the idea with other existing ideas to see what happened. Doing this was speculative and prone to more misses than hits, but, to the researchers, the positive expected value of the work was obvious and many were excited to do the work. More importantly, they felt free to do so.

The era recounted in Dyson’s stories of the IAS corresponds with the onset of the modern, increasingly discipline-oriented organization of science. This is an era in which many feel publishing in multiple areas and extreme novelty are disincentivized in various ways due to current journal, grant, and tenure processes.

All of this has happened, coincidentally, as we’ve eased into a world in which it feels like ideas are getting harder to find rather than easier to find. My previous piece, When do ideas get easier to find?, documents Gerald Holton’s mid-1900s work on what seemed to be driving the productivity of early-to-mid-1900s physics. His model of the contemporary ‘physics production function’ placed an extremely high valuation on the importance of incentivizing the workflow of an individual like John von Neumann or the branch-creating inventions of a researcher like I.I. Rabi — whose work many subsequent Nobel prizes depended upon.

Theirs were results worth failing towards.

Holton documented — both anecdotally and with a graph documenting the productivity of research on particle accelerators — that the productivity of fields he was familiar with seemed to be almost entirely driven by the consistent creation of new branches of research. Holton, who was a physicist before he became something of a progress studies researcher, was telling the story of his own field.

As the following graphic shows, exponential growth (logged y-axis) in accelerator energies over several decades was achieved almost entirely by the consistent creation of new technologies and approaches to attack the same problem. The rate of progress for any individual branch of technology usually curtailed quite rapidly after its first 5 years or so, but new branches took its place.

He noted that the creation and extension of these new branches tended to be the domain of the young. Beating a long-tenured professor in an area of knowledge in which they have been building up minute knowledge for several decades is quite difficult. Knowing only 80% as much as the elder statesman doesn’t count for much on older branches of science. But understanding the old ideas, in a nutshell, is plenty when looking to apply the learnings of an old branch to another area of knowledge. When doing that sort of work, enthusiasm and willingness to take some risks count for much more. Most importantly, the young researchers tended to be the most tolerant of this failure-prone and open-ended exploration. They were often excited by it. It was a great way to make a name for yourself!

Holton did not choose this accelerator example because he felt the underlying dynamics driving its growth were any different from those that people believed to be present in the rest of the field. Similar to the choices made in the Bloom paper, it was just the most readily measurable example.

Building on his point, Holton noted that, at the time of writing in around 1962, work by M.M. Kessler had recently found that 50% of the references cited in the Physical Review — the top physics journal at the time — were less than three years old. And only 20% were more than seven years old. Decades on from the paradigm-changing discoveries of relativity and quantum mechanics, physics researchers had still been rapidly leapfrogging each other in the race for human progress, constantly citing young papers and letters. Science was moving so fast that a non-negligible number of citations in the journals were citing personal correspondence and conversations with other physicists as opposed to published papers.

I don’t believe Holton’s theory contradicts the evidence in the Jones and Weinberg paper, either. Even though the authors only examined the quantum revolution by name, there was a second, generally similar burst in theoretical knowledge that their data captured that they did not discuss — the birth of the field of molecular biology. I’m not sure why they omitted this, but it might be that the emergence of molecular biology and the magnitude of its importance is generally paid less attention to than quantum mechanics.

This young branch is one that I covered at length in a previous piece on this Substack. The piece details the dynamics of the burst in branch-creating science in the field of molecular biology between around 1933 and 1953. As the following graph shows, the chemistry Nobel Prize also experienced a resurgence in youth around this time.

The bump would probably look even starker if molecular biology didn’t split its Nobel Prizes between the chemistry and medicine categories.

Looking at the author’s Nobel in Medicine charts, things are less stark. But that should not be surprising. Medicine is a very broad field, handing out Nobel prizes to things like molecular biology discoveries that add to our theoretical understanding of medicine as well as applied findings such as “discoveries concerning heart catheterization and pathological changes in the circulatory system.” So, while the aggregate age trends in medicine don’t exhibit a similar bump (as I show below), the growth in the frequency of theoretical findings responsible for Nobel prizes in this period did experience a noticeable bump. And, as the authors pointed out, in the data theoretical findings were very strongly associated with youth. But, in general, making sense of the mixed bag of research projects that qualify for Nobels in medicine as a single statistical category is maybe not worth looking into too deeply.

Next, we look further into the figures to assess the validity of Holton’s observation that, decades after the start of the quantum revolution, the research being cited in the Physical Review was still extremely young. The larger citation dataset in the appendix of the Jones and Weinberg paper does nothing to make us doubt Holton’s observation. While the smaller, Nobel-prize-only data in the main paper observed shows steady increases in the age of citations, a different trend entirely comes to light in the more robust datasets used in the appendix — which include the 100 most cited papers per year in each field rather than just the Nobel-winning discoveries. In physics and chemistry, it seems that there is more or less a plateau — or extremely marginal decrease — in average citation age from around 1900 to 1970. Then, somewhat abruptly, there begins a significant and steady climb in citation age. One could interpret this, simplistically, as knowledge beginning to progress more slowly.

Interestingly, this late-1960s/early-1970s period is also when a lot of the data in papers like Bloom et al’s tends to begin. This is also about the time when Weaver and Feynman’s statements — pointing to a noticeable problem forming in science’s systems — take place. The historical perspective, when examining a question like this, is imperative.

Holton’s model — along with Feynman and Weaver’s statements — came right at the end of a bit of a golden era. Holton’s model describes a different world, but one that is very relevant to the question of how to re-ignite scientific productivity.

Reflecting on how there were a growing number of problem areas where headway could be made by individuals from as many as five fields, Holton asserted that:

It is becoming more and more evident that departmental barriers are going to be difficult to defend.

He believed that we might see the end of departmental barriers and a system that evolved to incentivize all-important, branch-creating science. Young branches were where the most productive science was done. Young branches are where the burden of knowledge isn’t so burdensome. He believed we would build institutions and systems focused on creating new branches above all else because that’s what was driving our productivity.

We built the exact opposite.

As you know, we cannot re-run recent scientific history, changing individual parameters as we wish. What if institutions and incentives hadn’t changed 50+ years ago? What if we could set the ‘burden of knowledge coefficient’ to zero and re-run the simulation with the same institutional changes? This inability is one that haunts social scientists every day.

But, luckily, we do have a case study available to us that provides a lot of perspective — and lends a lot of credibility to the human systems hypothesis. There is a non-STEM field where institutions and incentives similar to those introduced to American STEM research in the late-1900s were introduced. The field, to most people’s eyes, is subject to rather negligible burden of knowledge issues. And when similar institutions were introduced into the field, similar changes occurred.

It is an imperfect analogy, but one that comes close to setting the burden of knowledge coefficient to zero and re-running the simulation with similar institutional changes to observe how much of the slowdown persists.

The field: poetry.

Many within the field of poetry and creative writing in general, apparently, believe they are living through an extreme slowdown in work of the highest quality as well. This is something I was unaware of until recently when an unusually broad-ranging mathematician friend of mine, Chen Lu, began telling me about it and shared the piece on which this section is based. The piece is called Poetry and Ambition. It is a long, part essay/part poem published by Donald Hall who was a practicing poet from 1950 through the early-200s.

In Poetry and Ambition, Hall highlights the changing state of the field over his career in terms of goals, outcomes, incentives, and overall ambition. In many of the sections, if one replaces words like ‘writer’ and ‘poem’ with ‘researcher’ and ‘finding’, the excerpt sounds eerily similar to complaints from those in the progress community railing against science’s stagnation.

For example, Hall writes the following on the changing ambitions of the poet:

5. True ambition in a poet seeks fame in the old sense, to make words that live forever. If even to entertain such ambition reveals monstrous egotism, let me argue that the common alternative is petty egotism that spends itself in small competitiveness, that measures its success by quantity of publication, by blurbs on jackets, by small achievement: to be the best poet in the workshop, to be published by Knopf, to win the Pulitzer or the Nobel. . . . The grander goal is to be as good as Dante.

Let me hypothesize the developmental stages of the poet.

At twelve, say, the American poet-to-be is afflicted with generalized ambition. (Robert Frost wanted to be a baseball pitcher and a United States senator; Oliver Wendell Holmes said that nothing was so commonplace as the desire to appear remarkable; the desire may be common but it is at least essential.) At sixteen the poet reads [Walt] Whitman and Homer and wants to be immortal. Alas, at twenty-four the same poet wants to be in The New Yorker. . . .

To be clear, Hall is not some rogue in his beliefs. Mark McGurl wrote the following in his book, The Program Era, on postwar fiction and its effects on the field of creative writing:

While the existence of degree-granting entities like the Iowa Writers’ Workshop was the result, in part, of a new hospitality to self-expressive creativity on the part of the progressive-minded universities willing to expand the boundaries of what could count as legitimate academic work, the founders and promoters of these programs more than met the institution halfway, rationalizing their presence in a scholarly environment by asserting their own disciplinary rigor.

Halls words simply attempt to (eloquently) describe a wider trend.

Hall goes on to refer to the type of poem that the exploding poetry and MFA programs have geared themselves to train poets to produce as the ‘McPoem.’ “To produce the McPoem, institutions must enforce patterns, institutions within institutions.” These misguided forms of quality controls, in Hall’s eyes, were how most large American organizations had adapted themselves to meet the needs of larger and larger audiences who wanted to buy what they were selling. Similarly, this is how MFA programs had structured themselves to meet the increasing need of young people seeking out the degree programs. To him, the approach was far more suited to mass-market goods than his craft.

Hall continues, diving into what the work of the ‘poetic heroes of the American present’ often looked like:

When Robert Lowell was young, he wrote slowly and painfully and very well. On his wonderful Library of Congress LP, before he recites his early poem about “Falling Asleep over the Aeneid,” he tells how the poem began when he tried translating Virgil but produced only eighty lines in six months, which he found disheartening. Five years elapsed between his Pulitzer book Lord Weary’s Castle, which was the announcement of his genius, and its underrated successor The Mills of the Kavanaughs. Then, there were eight more years before the abrupt innovation of Life Studies. For the Union Dead was spotty, Near the Ocean spottier, and then the rot set in.

Now, no man should be hanged for losing his gift, most especially a man who suffered as Lowell did. But one can, I think, feel annoyed when quality plunges as quantity multiplies: Lowell published six bad books of poems in those disastrous last eight years of his life.

He continues:

9. Horace, when he wrote the Ars Poetica, recommended that poets keep their poems home for ten years; don’t let them go, don’t publish them until you have kept them around for ten years: by that time, they ought to stop moving on you; by that time, you ought to have them right. Sensible advice, I think — but difficult to follow. When [Alexander] Pope wrote “An Essay on Criticism” seventeen hundred years after Horace, he cut the waiting time in half, suggesting that poets keep their poems for five years before publication. Henry Adams said something about acceleration, mounting his complaint in 1912; some would say that acceleration has accelerated in the seventy years since. By this time, I would be grateful—and published poetry would be better—if people kept their poems home for eighteen months.

Poems have become as instant as coffee or onion soup mix. One of our eminent critics compared Lowell’s last book to the work of Horace, although some of its poems were dated the year of publication.

…

If Robert Lowell, John Berryman, and Robert Penn Warren publish without allowing for revision or self-criticism, how can we expect a twenty-four-year-old in Manhattan to wait five years—or eighteen months? With these famous men as models, how should we blame the young poet who boasts in a brochure of over four hundred poems published in the last five years? Or the publisher, advertising a book, who brags that his poet has published twelve books in ten years? Or the workshop teacher who meets a colleague on a crosswalk and buffs the backs of his fingernails against his tweed as he proclaims that, over the last two years, he has averaged “placing” two poems a week?

Hall then goes on to describe the MFA and how it has specifically, in his eyes, tainted the process of great poetry. Looking at the passage, one could imagine an eerily similar statement coming from a scientist who practiced from the mid into the late 1900s, talking about how the research process had become much more about detailed methods and learning to conduct a research pipeline slightly more rigorously and less about taking proper time and intellectual effort to do true exploration to find properly new ideas. Hall continues, furiously:

10. Abolish the MFA! What a ringing slogan for a new Cato: Iowa delenda est! [Iowa must be destroyed!] (Iowa possesses the most prominent MFA program.)

The workshop schools us to produce the McPoem, which is “a mold in plaster, / Made with no loss of time,” with no waste of effort, with no strenuous questioning as to merit. If we attend a workshop we must bring something to class or we do not contribute. What kind of workshop could Horace have contributed to, if he kept his poems to himself for ten years? No, we will not admit Horace and Pope to our workshops, for they will just sit there, holding back their own work, claiming it is not ready, acting superior, a bunch of elitists. . . .

When we use a metaphor, it is useful to make inquiries of it. I have already compared the workshop to a fast-food franchise, to a Ford assembly line. […] Or should we compare Creative Writing 401 to a sweatshop where women sew shirts at an illegally low wage? Probably the metaphor refers to none of the above, because the workshop is rarely a place for starting and finishing poems; it is a place for repairing them. The poetry workshop resembles a garage to which we bring incomplete or malfunctioning homemade machines for diagnosis and repair. Here is the homemade airplane for which the crazed inventor forgot to provide wings; here is the internal combustion engine all finished except that it lacks a carburetor; here is the rowboat without oarlocks, the ladder without rungs, the motorcycle without wheels. We advance our nonfunctional machine into a circle of other apprentice inventors and one or two senior Edisons. “Very good,” they say; “it almost flies. . . . How about, uh . . . how about wings?” Or, “Let me just show you how to build a carburetor. . . .”

Whatever we bring to this place, we bring it too soon. The weekly meetings of the workshop serve the haste of our culture. When we bring a new poem to the workshop, anxious for praise, others’ voices enter the poem’s metabolism before it is mature, distorting its possible growth and change.

He later moves on to examine the ill effects of the overt bureaucratization of the poetry profession — where, increasingly often, the university’s creative writing department or adjacent grant funders provided the best chance for poets to earn a steady income practicing some version of their craft. The effects could easily be seen. One only needed look at the industry’s newsletters:

14. A product of the creative writing industry is the writerly newsletter which concerns itself with publications, grants, and jobs—and with nothing serious. If poets meeting each other in 1941 discussed how much they were paid a line, now they trade information about grants; left wing and right united; to be Establishment is to have received a National Endowment for the Arts grant; to be anti-Establishment is to denounce the N.E.A. as a conspiracy.

…

Associated Writing Programs publishes A.W.P. Newsletter, which includes one article each issue—often a talk addressed to an A.W.P. meeting—and adds helpful business aids: The December 1982, issue includes advice on “The ‘Well Written’ Letter of Application,” lists of magazines requesting material (“The editors state they are looking for ‘straightforward but not inartistic work’”), lists of grants and awards (“The annual HARRY SMITH BOOK AWARD is given by COSMEP to…”), and notices of A.W.P. competitions and conventions. . . .

Really, these newsletters provide illusion; for jobs and grants go to the eminent people. As we all know, eminence is arithmetical: it derives from the number of units published times the prestige of the places of publication. People hiring or granting do not judge quality—it’s so subjective!—but anyone can multiply units by the prestige index and come off with the product. Eminence also brings readings. Can we go uncorrupted by such knowledge? I am asked to introduce a young poet’s volume; the publisher will pay the going rate; but I did not know that there was a going rate. . . . Even blurbs on jackets are commodities. They are exchanged for pamphlets, for readings; reciprocal blurbs are only the most obvious exchanges. . . .

I may need to add ‘Donald Hall’ to my list of ‘people who did progress studies before progress studies existed’…

Academics and the system they operated within used to understand that they were in a hit-making game. John Nash, with an h-index of 7, was employed by the profession for decades because they felt he, at any point, had the potential to produce a massive hit. Richard Feynman left new discoveries in his desk constantly. He felt they wouldn’t change much in the field, so they weren’t worth the hassle of publishing. He had more ambition than that. When he did publish interesting and ambitious papers, journals would not reject them because pieces of them were shaky. This Substack covered a case of Feynman publishing a finding, that turned out to be right, in spite of Bohr and Dirac openly saying beforehand they thought it was completely wrong. It was an amicable disagreement — most were. The journals were the best place to debate that sort of thing, so it was published!

Hall’s final paragraph, in a way, touches on all of this:

16. There is no audit we can perform on ourselves, to assure that we work with proper ambition. Obviously, it helps to be careful; to revise, to take time, to put the poem away; to pursue distance in the hope of objective measure. We know that the poem, to satisfy ambition’s goals, must not express mere personal feeling or opinion—as the moment’s McPoem does. It must by its language make art’s new object. We must try to hold ourselves to the mark; we must not write to publish or to prevail. Repeated scrutiny is the only method general enough for recommending. . . .

And of course repeated scrutiny is not foolproof; and we will fool ourselves. Nor can the hours we work provide an index of ambition or seriousness. Although Henry Moore laughs at artists who work only an hour or two a day, he acknowledges that sculptors can carve sixteen hours at a stretch for years on end—tap-tap-tap on stone—and remain lazy. We can revise our poems five hundred times; we can lock poems in their rooms for ten years—and remain modest in our endeavor. On the other hand, anyone casting a glance over biography or literary history must acknowledge: Some great poems have come without noticeable labor.

But, as I speak, I confuse realms. Ambition is not a quality of the poem but of the poet. Failure and achievement belong to the poet, and if our goal remains unattainable, then failure must be standard. To pursue the unattainable for eighty-five years, like Henry Moore, may imply a certain temperament. […] If there is no method of work that we can rely on, maybe at least we can encourage in ourselves a temperament that is not easily satisfied. Sometime when we are discouraged with our own work, we may notice that even the great poems, the sources and the standards, seem inadequate: “Ode to a Nightingale” feels too limited in scope, “Out of the Cradle Endlessly Rocking” too sloppy, “To His Coy Mistress” too neat, and “Among Schoolchildren” padded. . . .

Maybe ambition is appropriately unattainable when we acknowledge: No poem is so great as we demand that poetry be.

Ambition is not something to be metered and rewarded accordingly — it’s not possible. But everything possible should be done to, at the very least, not disincentivize it.

Poetry is only so similar to science. But it does seem that — even though in the American tradition writers and scientists do not co-habit the same areas of the intelligentsia as they do in Russia — many in both groups that lived through similar systems changes come together in the belief that the new institutions and incentives tainted their fields.

It’s not so silly to think that what happened to the field of poetry could inform how we think about counterfactual scientific history. As much as there are great scientists who learn by reading, the number of stories of great science as a predominantly learning-by-doing exercise abound. It’s not so silly to give serious consideration to the case of writers and the state of their field. Writers, after all, also spend substantial amounts of time formulating new ideas, perusing the works of peers, and attempting to deploy both new methods and ideas into their own work.

Surely, the burden of knowledge is more burdensome in science than in writing. But, it should not be forgotten, that many of the great scientists I’ve covered on this Substack didn’t place nearly as high a priority on ‘keeping up with the literature’ as modern researchers.

Edison spent time on it, but he spent far more time at the lab bench learning things for himself. He both kept up with and was extremely skeptical of the literature.

And if the reader feels that Edison’s era of electrical science was somehow primitive and is not representative, Willis Whitney — a former academic — eventually instituted a rule at the GE Research Lab where you’d get in trouble for going to the library too frequently. He learned, through early failures in relying too heavily on the literature, that it had limited effectiveness in solving problems — compared to experimentation and working with colleagues.

Not to belabor the point, but Feynman was also notorious for not exactly keeping up with the literature. He kept up to some extent, but often worked things out for himself, later checking the literature to see if he was the first one to have done that. He felt that was the best way — both to understand things and also to practice the process of figuring things out.

A different world — with different systems — yielded a different kind of researcher.

To me, the idea that the slowdown in scientific ideas might be driven far more by changes in human systems and incentives than some force of nature related to growing ‘idea space’ is great news. While solutions can likely be found to mitigate issues associated with the more standard burden of knowledge hypothesis, we’re still working against constraints of nature. In the world where it’s a human systems issue, we’re mostly working against people!

Working against people isn’t easy, but it’s doable. We’re already making great first steps. Progress studies funders have been very forward-thinking in funding new, experimental science orgs such as Arc Institute, Arcadia, and Convergent Research (with more to come) that provide invaluable learning opportunities on a small scale. Any learnings that prove successful on that small scale can grow into medium-term efforts — possibly even whole research institutes — which can scale, continue to experiment, and (hopefully) find even more success. In the longer run, the federal government — an effective copycat of ideas that have been de-risked and aren’t too politicized — could fold these learnings into their existing NIH and NSF policies. Or, as is often the case in government scientific history, they may start a new department to take advantage of the new learnings.

The pressing question then is: what are the best things these new science orgs can do?

The most general and boring answer is twofold:

1. Make sure you’re trying something truly different from existing efforts. We need novelty.

2. Document as much as possible about your novel attempt so we can learn from it!

My more prescriptive (and interesting) answer is also twofold:

1. Do more truly applied research! This was largely America’s biggest comparative advantage in research throughout the early-1900s. We found ways to let this process drive great basic research and vice-versa. I’ve written up detailed accounts of many different ways to do this that proved successful on this Substack:

   1. How did places like Bell Labs know how to ask the right questions?

   2. Irving Langmuir, the General Electric Research Laboratory, and when applications lead to theory

   3. A Progress Studies History of Early MIT— Part 2: An Industrial Research Powerhouse

   4. Tales of Edison's Lab & Thomas Edison, Technical Entrepreneur

2. If you’re doing basic research, focus on creating new scientific branches at the expense of all other things! It’s a right-tailed game, and it’s extremely worth it. As the pieces listed above detail, these new branches can often result from applied research projects if managed the right way — with researchers given certain problems and certain freedoms. The following pieces document processes that yielded branch-creation in a more (traditional) basic research setting:

   1. A Report on Scientific Branch-Creation: How the Rockefeller Foundation helped bootstrap the field of molecular biology

   2. John von Neumann: A Strange Kind of Bird

   3. When do ideas get easier to find?

To conclude, I implore the reader to remember something that Weaver, Feynman, Dyson, and Holton would agree upon: the way science is done is indelibly impacted by the way scientists are organized and incentivized. If you spoil their excitement and ambition, the whole enterprise becomes entirely different. The magnitude of change that can result should not be underestimated.

Thanks for reading:)

Share

Subscribe now

In the near future, I would love to document — one by one — the historical details of why fields that used to work together closely grew apart. That kind of brute force project would go a long way in further addressing the question in today’s piece. If you’d like to help on that project, please reach out — it will take more people than just me!

Also, Chen Lu — the mathematician I mentioned — is finishing his Ph.D. in applied math at MIT and is looking for a job. Something like ML roles where he can use his math and coding toolkit would be ideal. You can check out his LinkedIn here and some of his publications here. If you’d like an intro, DM me on Twitter!

When showing this paper around, one economist of innovation asked how I felt about Agarwal et al.’s paper on the growing size of publishing teams in mathematics in the aftermath of the Iron Curtain falling. I thought this was a very fair point and worth including, but one that would break up the flow of the main piece as it is a bit of an aside.

Before speaking briefly on the Agarwal paper, I should first start by explaining the previous and more well-known Borjas and Doran paper whose data Agarwal et al. used in their analysis. Borjas and Doran put together a dataset on the publications, citations, university affiliations, and more of both Soviet mathematicians as well as American mathematicians before and after the Iron Curtain falling. They pay special attention to the different effects the shock had on American mathematicians who were publishing in fields that the Soviet’s were particularly strong in vs. those where they were comparatively weak.

As the authors note, in spite of the fact that there were around 2.4x as many American mathematicians and Americans published 3x as many papers overall, there were areas in which the Soviets were far stronger, publishing 1.4x as many papers as Americans and having the undeniable world leaders in the space. There were also areas where the Soviet Union was essentially a non-factor, publishing around .05x as much as Americans. The following graphic gives you an idea of just how wide the difference was across fields — the bar measures the ratio of Soviet papers to American papers in a topic area.

Paying close attention to the labor market effects in the aftermath of the shock, the authors find (among other things):

• The overall number of papers written by professors at American universities was more or less unchanged after the Iron Curtain fell.

• The number of papers written per American mathematician and by American mathematicians overall went down, but this was counteracted by the effective increase in total Soviet-descendant professors publishing.

• American professors who published in the weakest Soviet disciplines were more or less unaffected, whereas those who published in Soviet areas of strength ended up publishing less, received fewer citations, were more likely to end up unemployed or at a worse university, etc.

  • However, if an American professor in an area of Soviet strength co-authored with a Soviet professor, it seems this was correlated with being unaffected.

• Around 1/8th of new professorships were being given to Soviet mathematicians immediately after the Iron Curtain fell, and these immigrants were particularly strong.

In general, I believe the Borjas and Doran observations in all of these areas are quite strong. As a dataset informing the labor market effects of the Soviet influence in American math academia from around 1975-2010 — which is what the paper claims to be — the methods and takeaways seem interesting and robust.

In the author’s own words, they summarize their results as follows:

Soviet mathematics developed in an insular fashion and along different specializations than American mathematics After the collapse of the Soviet Union, over 1,000 Soviet mathematicians (nearly a tenth of the pre-existing workforce) migrated to other countries, with about 30 percent settling in the United States. As a result, some fields in the American mathematics community experienced a flood of new mathematicians, theorems, and ideas, while other fields received few Soviet mathematicians and gained few potential insights.

Our empirical analysis unambiguously documents that the typical American mathematician whose research agenda most overlapped with that of the Soviets suffered a reduction in productivity after the collapse of the Soviet Union. Based solely on the pre-1992 age-output profile of American mathematicians, we find that the actual post-1992 output of mathematicians whose work most overlapped with that of the Soviets and hence could have benefited more from the influx of Soviet ideas is far below what would have been expected. The data also reveal that these American mathematicians became much more likely to switch institutions; that the switch entailed a move to a lower quality institution; that many of these American mathematicians ceased publishing relatively early in their career; and that they became much less likely to publish a “home run” after the arrival of the Soviet émigrés. Although total output declined for the pre-existing group of American mathematicians, the gap was “filled in” by the contribution of Soviet émigrés

I think that is a great paper and I value the results a lot.

Agarwal et al., four years later, use mostly the same data and attempt to tell a burden of knowledge hypothesis story with it. I don’t think it would be an unfair interpretation to say that their work surely approximates — to some extent — something of an upper-bound on the ‘burden of knowledge coefficient,’ but I believe it would be a stretch to say it establishes any kind of well-identified lower-bound. I’ll begin to tell you why in the next paragraph. But, before I do, it is probably worth pointing out that — in spite of Ben Jone’s paper on the burden of knowledge having been published several years before the Borjas and Doran paper — Borjas and Doran did not choose to make claims about their data as demonstrating proof of the burden of knowledge in their paper.

Agarwal et al., however, do attempt to make this claim using largely the same data. I don’t think it’s unreasonable for the journal to have published the paper, but I do think the results are not nearly as robust as Borjas and Doran’s and should be used with caution. Agarwal et al. attempt to exploit the difference in team size trends between the three sub-fields in which the Soviets were strongest and the three sub-fields in which the Soviets were the weakest. The trend they’re looking at is visualized below — with the orange line representing American team size in Soviet-rich areas and the blue the team size of areas of Soviet weakness.

As is common, the starting point of the data is in 1970. From 1970 to 1991, there is a slow and steady rise in both lines, then — and around the fall of the Iron Curtain — the author count in Soviet-rich sub-fields begins to climb at a higher rate. The authors wrote, “For the treated subfields (Soviet-rich), the mean team size for the 20-year period before 1990 was 1.34 compared to 1.78 for the 20-year period after. By comparison, for the control subfields (Soviet-poor), the mean team size was 1.26 before compared to 1.55 after.” That increase in .29 authors per paper in the ‘control’ group vs. .44 in the ‘treatment’ group is observational and not to be disputed. In relation to how this affects the human systems hypothesis, the grains of salt I take their solutions with — as they do/don’t apply to the hypothesis — have to do with 1) what the authors could not control for that also change around this period and 2) the time period of American academia in which the shock occurred.

The first grain of salt I take this chart in with is that all of the changes resulting from this shock are happening in a post-1970 American system which I’ve never disputed has certain norms and systems for accounting for new ideas that would make things like team size grow. That is my biggest issue, but it is also unsurprising, I’m sure.

The second grain of salt I take the results in with is that the effects of a knowledge shock that result from an entire country’s ideas — that have been built of over decades — reaching the world are probably not the same as when individuals or small teams produce an idea. One major difference being: if it’s just one researcher’s big and new idea, there is only one of them to go around. In the Iron Curtain case, there were 12,000+ other researchers who had been familiar with many of the ideas for a while. 1,000+ of the most talented ones abruptly left to research abroad in other country’s. When combined with the first grain of salt, this could impact the trends we see in the data in a way that is not representative of the standard case of idea creation.

The third grain of salt I take the paper with is that it does not control for the labor market influx of Soviet researchers to the US.1 And the influx was non-negligible. In fact, the labor market shock itself was what Borjas and Doran exploited in their paper. Borjas and Doran noted that around 1/8th of new professorships in the post-Iron Curtain falling years were going to Soviets — and quite inordinately talented Soviets. In an attempt to show that their results were robust to this, Agarwal et al. show a chart of Japan because Japan had almost no Soviet mathematics immigrants. They ran other regressions as well, but, first, I’ll address the Japan chart — the only other country’s chart they chose to show.2

Looking at the chart, I found it strange that the Soviet-poor fields had such steady team size numbers pre-fall of the Iron Curtain but then began a noticeable and steady climb after the Iron Curtain fell. Agarwal et al. did not mention this which I found strange, because whatever happened there is significantly larger than the effect noted by the distance between the orange and blue lines. This distance between the two lines is what the authors’ regressions concern themselves with. I asked an economics of innovation-related researcher who works with the Japanese government if anything happened in the early 1990s as well. He laughed and said that the years around 1995 marked extremely important changes. This period saw substantial increases in government input, funding, and planning for many types of scientific grants.

I asked some questions about how these worked, and it seems like the new policies were often very NIH-like — large government employee oversight, sizable panels, proposal processes, etc. So, I don’t personally take that Japan chart very seriously — unless you are very compelled by the first three pairs of dots post-Iron Curtain falling, in which they noisily grow apart and back together again before the onset of the government changes. If I did take the chart seriously, accounting for the change in human systems, it very well could tell a story that points the finger more towards the human systems hypothesis as driving the overall changes more so than the burden of knowledge hypothesis.

The regressions the authors ran on other countries — which they did not visualize — are essentially tracking the same thing — the statistical significance of the growth in the gap between those two lines before and after the iron curtain fell. (Half of these countries’ regression coefficients did not come out as significant at the 95% confidence interval.) As with the Japanese chart, the paper does not seem to pay attention to the fact that there could be much more significant changes happening elsewhere in the system that the methods (and authors) are unaware of and don’t account for. I don’t blame them because it is extremely burdensome — on the order of months — to figure out exactly what happened in even a handful of countries' scientific grant-funding ecosystems over that timeline. But, what matters for our interpretation is that they didn’t. Here is a 1995 Nature article titled Japan opens a new era in university funding very publicly exploring some of these changes.

(Also, see Figure 2-10 at this link, Trends in Budget for Grants-in-Aid for Scientific Research, showing related funding changes impacting trends as early as 1991.)

Speaking frankly, anything could have been happening around 1991 in any number of those countries that the authors do not know about. In the best case, the method only identifies the existence of a burden of knowledge and does nothing to identify the burden of knowledge effects as anything more than existing and being statistically significant — saying nothing about its presence as a primary driver of stagnation. And, once again, that is if you buy into the premise of their natural experiment as a natural experiment at all.

I read this paper as doing an okay job (at best) of isolating the presence of a burden of knowledge-type effect in the US ecosystem conditional on the shock being introduced to the post-1970 US ecosystem. I believe the coefficient estimates themselves are very likely overestimated. It is highly likely that they capture the effects of things that the authors did not/could not control for, such as: the new system itself interacted with the math field type, the full effects of new Soviet colleagues on their American colleagues, and the effects of the “knowledge” coming from 12,000 people all at once rather than a handful.

Natural experiment papers can only be as strong as the natural experiment itself. And there’s just a lot of moving parts here. Throughout this piece, I’ve used many papers whose results maybe get over-interpreted by others, but that I think are great and well-done papers that are very thorough. This paper — unlike the Borjas and Doran paper written using the same event as a shock — is one that I think should be used more carefully.

The authors attempt to run one of their regressions dropping all Soviet last named papers and said their regressions were still robust. But they acknowledged that this was not enough to control for all of the effects Soviets could have on their sub-field by having been incorporated into an American researchers hallway and social circle. The authors acknowledged the Japan evidence as their primary reasoning for their findings being robust to this labor market confounder, saying, “Secondly, and perhaps most importantly, we show the same pattern persists in Japan, a country that experienced very little immigration of the Soviet mathematicians.”

Much of the observational work in the Borjas and Doran paper exemplifies why merely dropping the Soviet-named publications only scratches the surface on the effects the Soviet emigré mathematicians could have on a mathematician and the department/field as a whole. That is why this Japan analysis was so vital to do.

For some reason, Figure 6 in the paper shows China as insignificant at the 95% confidence interval, but their table has three stars indicating it significant at the 99% interval. But, looking at the table, the standard error and coefficient make me believe that China’s results in Figure 6 are more likely to be insignificant at the 95% interval rather than significant at the 99% interval. But I wanted to note this (what I think might be a) discrepancy.

Works in Progress released a piece from me diving into Thomas Edison’s Herculean contributions as a technical entrepreneur. He was not often the first to invent the technologies which his name has become synonymous with, but he did not even see himself as an ‘inventor.’

We have a word for people like him now: technical entrepreneur And he was the best ever.

• He took tech that barely worked and made it useful

• He took products to market at prices people were willing to pay

• He made things that could be manufactured at scale

• He set outrageous deadlines AND hit them

• He built systems with others' inventions that the original inventors didn't dream of

So no, he was not an inventor. He was something even better

Also, tragically, I don't think the modern US could make use of someone of Edison's talents. Check out the piece to learn how Edison did what he did and why a modern-day Edison might be wasted on today's US.

Check it out here!!

For those interested, check out the podcast I put together exploring what it was like to work in Edison’s lab:

I have another piece coming out Thursday diving into why the historical evidence leads me to believe there is a better hypothesis than the ‘burden of knowledge’ hypothesis to address the question, ‘What is the primary driver of the scientific slowdown?’ Stay tuned for that!

Subscribe now

Share