The Invisible Hand Behind Innovation

Editor’s note: Thanks to Hon Dr Andrew Leigh MP for his presentation at the University of Melbourne some weeks ago.

Despite the mythology of garages, lone geniuses, and venture capital origin stories, breakthroughs rarely emerge from thin air. They often grow from environments carefully shaped by policy decisions.

This is one of the central arguments in Andrew Leigh’s The Shortest History of Innovation. Leigh’s work traces the long arc of human ingenuity to show that the most innovative societies did not simply produce more brilliant individuals; they constructed systems that allowed ideas to spread, combine, and scale.

This can be seen throughout history - the most successful innovation policies focused on creating the conditions for experimentation to flourish, rather than attempting to identify specific technologies, companies, or industries that would be successful. 

Innovation is not just a product of individual brilliance; it is also the product of policy design.

Throughout history, several government policies have proven particularly powerful in accelerating innovation. Three of the most important are: competition policy, public investment in research, and labour mobility. Each addresses a fundamental challenge in the innovation ecosystem and together, they illustrate the broader lesson that governments rarely succeed when they attempt to engineer innovation directly. They succeed when they build the systems that allow innovation to emerge.

Firms facing capable rivals have strong incentives to improve their products, adopt new technologies and find more efficient ways to operate. Without that pressure, the motivation to innovate weakens.

Economists have long observed that monopolies often become complacent. When a firm dominates a market, it can maintain profits without needing to improve what it produces. By contrast, competitive markets reward those who find better solutions and punish those who fail to adapt. 

Competition policy is a delicate balancing act. Markets that are too tightly controlled can stifle entry and experimentation, but markets with too little regulation can allow dominant players to entrench themselves and block new competitors. 

The goal is not competition for its own sake, but a market structure where new firms can enter, ideas can be challenged, and successful innovations can scale.

In the 1990s, the Australian government implemented a sweeping set of reforms known as the National Competition Policy. These reforms expanded the reach of competition law, opened previously protected sectors to market competition, and reduced regulatory barriers to entry.

They affected industries ranging from telecommunications and electricity to transport and agriculture. Many sectors that had once been shielded by regulation or public monopolies were exposed to competitive pressure for the first time. The result was a measurable increase in productivity across large parts of the economy.

Greater competition forced firms to modernise operations, adopt new technologies, and innovate to survive. A large portion of Australia’s productivity growth in the 1990s was attributed to these reforms, proving that innovation thrives when firms cannot afford to be complacent. 

Competition drives firms to innovate for survival; however, markets alone rarely generate enough investment in fundamental research. The benefits of basic research are difficult for any single company to capture.

Discoveries in physics, chemistry, or biology spill across industries and take years to develop. The organisation that funds the original research may only capture a small portion of the economic value it ultimately creates. For private firms operating under commercial pressures, such investments can appear too uncertain or too distant. This makes government investment critical.

Public funding has played a decisive role in the development of many technologies that underpin the modern economy. From the internet and GPS to advanced medical treatments, a large share of foundational innovation originated in publicly funded laboratories and universities.

For decades, Australia’s Commonwealth Scientific and Industrial Research Organisation (CSIRO) has conducted long-term research in fields such as agriculture, environmental science, telecommunications, and engineering. Its work has generated thousands of patents and contributed to innovations ranging from improved crop varieties to wireless networking technology.

Importantly, institutions like CSIRO rarely operate with the narrow commercial objectives that guide private firms. Their mandate allows them to explore long-term scientific challenges. These discoveries often become the building blocks for private sector innovation. The relationship between public discovery and private application has repeated throughout modern history - government investment lowers the initial barriers to discovery, and the private sector focuses on turning ideas into products. In this sense, public research funding acts as the foundation of innovation. It cultivates knowledge that markets alone might never produce.

Knowledge spreads through people. 

Even when strong competition exists and groundbreaking research is produced, innovation can stall if ideas remain trapped inside organisations. Scientists, engineers, and entrepreneurs carry insights from one company to another, combining ideas from different fields and building new ventures around them. When labour markets allow skilled individuals to move freely between firms, ideas recombine at a faster rate. Restrictions on labour mobility can slow this process and one of the most significant barriers is the use of non-compete clauses. 

For decades, California has largely refused to enforce non-compete agreements; employees can leave a company to join a competitor or start a rival firm with little-to-no repercussions. This legal environment has allowed ideas and expertise to circulate rapidly among companies.

This policy played a significant role in the development of Silicon Valley. Engineers and entrepreneurs frequently moved between firms or started their own. Each move carried technical knowledge, management practices, and networks of collaborators.

Over time, this circulation created a dense ecosystem of startups, investors, and specialised expertise. Regions with stricter non-compete enforcement often see slower knowledge diffusion as ideas remain siloed inside firms, and the formation of new companies becomes more difficult. It is important to note the need for intellectual property (IP) to be protected however there is a difference between tangible IP and knowledge. Innovation systems function best when knowledge can travel with the individuals who create it. When people move, ideas move with them and combine in ways that produce innovation.

Competition, research investment, and labour mobility are the backbone of innovation and all need to be addressed.

Competition provides incentives to innovate, public research expands the pool of knowledge, and labour mobility ensures that knowledge spreads through the economy. Together, they create conditions for inevitable innovation. Innovation rarely responds well to attempts at centralisation.

It is difficult for governments to predict which technologies will succeed or which industries will dominate the future, what they can do is design environments for experimentation to flourish. When markets remain competitive, research institutions remain strong, and people remain free to move between ideas, innovation becomes less dependent on extraordinary individuals. Once successful, innovation stops being a rare stroke of genius and becomes an inevitability. 

P.S. Congratulations to Hon Dr Andrew Leigh MP on the publication of his book, the Shortest History of Innovation.