Comment: Critical Point Physics World  October 2015
(Mikkel Juul Jensen / Science Photo Library)

From wrong to right

The Yang–Mills theory is central to particle physics, but why did something that was once thought to be so wrong prove to be so right? Robert P Crease investigates

The early 1950s was the start of new period for physics. The first accelerators to surpass energies of 1 GeV were coming online. No longer would experimentalists have to climb mountains or set sail in hot-air balloons to hunt cosmic rays and strange particles; they could now create and study them conveniently and copiously in the safety of the laboratory. The experimental workplace in particle physics was suddenly booming, which made it a thrilling time for theorists as well.

What sense, the theorists wanted to know, did the new particles all make? What schemes could be devised to organize a particle equivalent of the periodic table? It wasn’t clear which path to take, and a variety of tools were developed for handling the strong, weak and electromagnetic interactions. Theorists spoke different mathematical languages, making their workshop a kind of Babel.

High-energy physics is unthinkable without Yang–Mills

Then, in 1954, Chen Ning “Frank” Yang and Robert Mills proposed a mathematical scheme that might be useful for the strong interaction, which (among other things) binds protons and neutrons together in the nucleus. The two theorists – and anyone acquainted with the details of field theory – knew that the scheme contained a glaring defect, which I’ll mention in a moment. But over six decades Yang–Mills, as it is known, has become fundamental not only to the Standard Model of particle physics – which describes all known particles and forces bar gravity – but also to efforts to go beyond it. Yang–Mills is the loom on which modern particle physics is woven. High-energy physics is unthinkable without it.

What fascinates me about the genesis of Yang–Mills theory, though, is how it could go from being wrong to being right – even indispensable – in a mere 20 years. In an attempt to get a grip on the answer, and to become acquainted with the possibilities of Yang–Mills in post-Standard Model theories, I attended an international conference entitled “60 years of Yang–Mills gauge field theories” held in May at the Nanyang Technological University in Singapore. Attended by Yang himself (Mills died in 1999), the conference made it clear how and why Yang–Mills has become such a seminal event in the history of physics.

The Pauli snag

Yang–Mills theory did not have an auspicious beginning. In 1953 Yang went to Brookhaven National Laboratory for a year, where his officemate was Mills, who was then in his last year of getting his PhD at Columbia University. Yang shared with Mills his fascination with the possible connection between symmetries and particle interactions; might forces in nature, in other words, arise somehow from the conservation of symmetries?

Others had also explored this thought. In 1918, for instance, Hermann Weyl had tried to explain electromagnetism as arising from a symmetry of the phase of the wave function. Yang and Mills now wondered if they could find some symmetry among particles that would dictate their interactions, and found a promising-looking candidate called “isotopic spin”, first described by Werner Heisenberg in 1932. Just as the phase of the wave function in electromagnetism can be shifted arbitrarily in space and time because the interaction with the electromagnetic field cancels out the effect of the alteration, so Yang and Mills proposed to do the same for isotopic spin, hypothesizing the existence of a “B field” to counteract the change.

In February 1954 Robert Oppenheimer invited Yang to present the work at a seminar at the Institute for Advanced Study (IAS) in Princeton, New Jersey. Wolfgang Pauli was present. He had been pursuing a similar thought, but had quit after encountering what seemed to be a show-stopping issue: in such theories, the mass of such a field has to be zero. Pauli knew that in “Abelian” theories, such as quantum electrodynamics (QED), it’s alright if the force-carrying particle is zero; this is indeed what lies behind our understanding of the zero mass of the photon. But extending field theory to hadrons required a “non-Abelian theory”, in which nature requires that the force-carrying particles are massive.

The cranky perfectionist interrupted Yang’s presentation demanding to know the mass of the B field. Yang said he didn’t know, and resumed the presentation. Pauli cut him off again with the same demand, to which Yang responded that he and Mills had reached “no definite conclusions”. Pauli snapped back, “That is not sufficient excuse”, in such a hostile way that Yang, distressed and uncertain, sat down. An awkward silence ensued, with the seminar effectively at a halt. Oppenheimer then said, “We should let Frank proceed.” He did, but with the rest of the presentation having an awkward flavour in the shadow of Pauli’s unanswered, and obviously all-important, question.

Pauli may have been abrupt and inconsiderate, but he was merely channelling the voice of the quantum field theory of the day, and his question was on the money. Yang–Mills theory required massless force-carrying particles. But nature said that such particles are massive. This defect meant that the Yang–Mills theory was obviously wrong.

When Yang returned to Brookhaven following his IAS talk, he and Mills decided to publish their work anyway. Years later in his Selected Papers, trying to express their rationale for doing so despite the obvious defect, Yang wrote simply: “The idea was beautiful and should be published.” The paper appeared in October 1954 (Phys. Rev. 96 191). In it, the authors wrestled with the nature of the B field in the final section, which in his Selected Papers Yang wrote was “more difficult to write than all earlier sections”. In regard to its mass, the authors added, “we do not have a satisfactory answer”. Small wonder that their work was initially regarded as merely a mathematical curiosity.

Overcoming the snag

Today, the phrase “Yang–Mills” can be used in two ways: to refer (1) to the specific scheme proposed by Yang and Mills in 1954, or (2) as shorthand for any non-Abelian gauge theory of the sort that is now fundamental to the Standard Model. The Pauli snag had to be overcome for (1) to turn into (2). Precisely how this transformation happened is a complicated, 20-year saga with unexpected twists, dramatic moments, and tangled plots and sub-plots. For example, in electroweak theory, which unifies the electromagnetic and weak interactions, spontaneous symmetry breaking – discovered in the 1960s – was found to create a loophole around the Pauli snag. Still, Yang–Mills theories of the electroweak interaction were not taken seriously until the 1970s, when they were shown to be “renormalizable”, meaning essentially that they were mathematically sound with no unwanted infinities.

Meanwhile, in the physics of the strong interaction, a Yang–Mills-like theory had long been assumed to be a useless tool. In 1968, however, the results of “deep” inelastic electron–proton scattering at the Stanford Linear Accelerator Laboratory suggested that quarks, the constituents of hadrons, were not mere mathematical entities but free particles at short distances. This work motivated physicists to take the possibilities of quantum field theory for the strong interaction more seriously, provided that a version of quantum chromodynamics could be developed in which the coupling gets weaker at short distances.

After yet more plot twists, such an “asymptotically free” field theory was eventually found in 1973. Here the Pauli snag was overcome because while Pauli’s question was about things that today we’d call quarks and gluons, quantum chromodynamics showed that the relevant physical states are actually “colour singlets”. These are pairs or trios of quarks bearing a quantum number called colour, the combination of which cancels out. A non-Abelian gauge theory of the sort Yang and Mills were trying to build turned out to apply to entities different from the neutrons and protons and pions that had inspired their effort.

The most remarkable aspect of the tortured transformation story of how Yang and Mills went from their original 1954 proposal to an indispensable tool of modern physics is – to paraphrase Lochlainn O’Raifeartaigh in his book The Dawning of Gauge Theory – not that it took so much time, but that it came together at all.

A new era

The development had a tremendous impact on high-energy theory. In the 1950s theorists were segmented into different ethnicities, with those who worked on strong, weak and electromagnetic interactions each using different tools and speaking different mathematical languages. Just 20 years later, all had changed. Yang–Mills established field theory as the dominant theoretical language and unified the mathematical Babel. It’s also had longer-term structural effects, kick-starting the era when looking at constraints at low energies let you make predictions at very high energies. Over time the landscape of high-energy or short-distance physics has been dramatically reshaped by Yang–Mills theory and is likely to remain a key part of future developments.

Celebrated The International Conference on 60 Years of Yang–Mills Gauge Field Theories (left) was held in May in Singapore. Chen Ning Yang himself attended (right). (Institute of Advanced Studies, Nanyang Technological University)

Some discoveries not only contribute to science but can also tell us about science, and Yang–Mills is one. How was it possible for a theory to be not yet true of the world? It is impossible to pin down to any specific date between 1954 and, say, 1975 the exact moment when Yang–Mills went from a theory that did not apply to the world to one that did. It required a gradual shift in ideas about the world, to which the Yang–Mills proposal itself contributed and even made possible. Sometimes, it seems, the world has to change before a theory will fit it.

Theory-making is therefore not always a matter of seeking something provable and applicable to the world. It involves articulating a sense of the world that’s yet to take shape. In some cases, therefore, theory-making involves summarizing and organizing some pre-existing sense of the world that is not yet explicitly stated, before any proof or evidence. (Afterwards, of course, we can say it was this way all along.) The 1954 Yang–Mills theory laid out what would make it possible for the resources of quantum field theory to apply. It illustrates that some perfectly respectable theories can be open to being filled in rather than matching the world in every detail – the theory of evolution being another, though rather different, illustration.

The critical point

On the last day of the Singapore conference, Da Hsuen Feng of the University of Macau asked me an intriguing question. “Sixty years after Maxwell’s equations,” he said, “you could easily get the public to appreciate the importance and relevance of his work. The same is true of general relativity and quantum mechanics. But how can we get the public to share the same appreciation for Yang–Mills?”

The obvious snappy comeback is that Yang–Mills is just impossibly obscure and arcane. But to non-physicists, so are the other theories that Feng mentioned. Part of the answer is surely that those theories relate differently to public interests. Maxwell’s theory has many practical applications – from mobile phones to radio. General relativity has fewer applications, but a dramatic discovery story and a charismatic discoverer (Einstein). Quantum mechanics has weirdness, as well as a cast of interesting co-discoverers and esoteric devices on the horizon.

Feng’s question left me with the feeling that one of the biggest unsolved challenges of Yang–Mills theory is simply finding ways for outsiders to get a hint, at least, of the stunning achievement it represents.