By Ernest Lefner, Chief Product Officer, Gluware
“Fail fast” is one of the defining philosophies of modern software engineering.
It reshaped startups, Agile development, DevOps, and cloud-native applications. Deploy early. Learn quickly. Iterate. Reduce the cost of experimentation.
It was the right philosophy for software. It never was for NetOps.
Nobody in network operations has ever believed that production is the right place to discover whether a network change was correct. That’s not because network engineers are more conservative than software developers. It’s because they’re operating under a fundamentally different set of realities.
The first is that enterprise networks are autonomous distributed systems.
Every router, switch, firewall, and controller continuously participates in distributed algorithms that collectively determine how the network behaves. Routing protocols exchange state. Security policies determine reachability. Forwarding decisions change dynamically as the network responds to new information.
A configuration change on one device doesn’t simply affect that device. It changes the inputs into a distributed system that immediately begins recalculating its behavior. The network itself propagates the effects.
That’s why even a seemingly localized change can have consequences far beyond where it was made.
The second reality is that network changes are operationally expensive.
Software organizations have spent decades making deployment inexpensive. A new release can often be deployed to a single server, a canary environment, or a small percentage of users before being rolled out more broadly.
Network changes don’t generally work that way.
Rolling out an operating system update or a configuration change across hundreds or thousands of devices is a coordinated operational event. Devices have dependencies.Changes have to be sequenced carefully. Health has to be monitored throughout the rollout.Maintenance windows have to be coordinated. Validation has to occur continuously as the deployment progresses.
Even in highly automated environments, large-scale network changes are measured in hours or days, not seconds.
Rollback isn’t free, either. If a deployment fails halfway through, recovery is another production change with its own planning, execution, and operational risk. Sometimes the failed deployment has already disrupted the very management paths needed to recover cleanly.
For decades, NetOps has optimized around one simple idea: production is an expensive place to discover mistakes. That’s why “fail fast” never became a networking philosophy.
As AI becomes part of network operations, though, a different assumption has started to emerge:
“Fail fast” has become “Probably fine.”
Large language models are probabilistic reasoning systems. That’s one of the reasons they’re so powerful. They generate likely answers from incomplete information. For many tasks, that’s exactly what’s needed.
- A meeting summary can be probably right.
- A first draft can be probably right.
- A brainstorming session can be probably right.
A production network change cannot, because “probably” isn’t an operational standard.
Every routing policy, segmentation rule, access control list, operating system upgrade, and configuration change becomes an authoritative change on infrastructure that everything else depends on.
That doesn’t mean AI has no place in NetOps. Quite the opposite. AI has enormous potential to accelerate network operations by proposing solutions, reducing manual effort, and helping engineers solve increasingly complex problems.
But proposing a change and proving a change are not the same thing.
If production isn’t where we discover mistakes, then validation isn’t optional. It’s the foundation of trustworthy automation.
And validation means more than governance.
First, it has to establish that the proposed change actually accomplishes the intended operational objective.
Second, it has to establish that the intended outcome has been expressed correctly in the native semantics of each vendor’s implementation across a heterogeneous network.
Those aren’t optional checks. They’re prerequisites for treating an AI-generated recommendation as an authoritative production change.
Which raises a critical question: If AI is responsible for proposing network changes, who is responsible for proving they’re correct?
One answer it can’t be is, “The AI validated itself.” That’s not independent validation. That’s asking a probabilistic system to endorse its own probabilistic conclusions.
Confidence has to come from a proven validation capability that exists independently of the reasoning engine itself.
NetOps has always understood that production isn’t the place to discover mistakes. AI doesn’t change that principle. If anything, it raises the standard for proving a network change is correct before it ever reaches production.
If you’re ready to build automation and agentic network operations around the industry’s most powerful, multi-vendor validation engine, learn about DIAL and then request a demo to start your journey.
