Edge AI: Why moving intelligence to the device changes everything

In 2024, John Deere's See & Spray technology helped farmers save an estimated 8 million gallons of herbicide mix across more than 1 million acres. Every spray decision, treat or skip, was made on the machine, in real time, with no network involved.

The system runs on embedded NVIDIA Jetson hardware and cut herbicide use by an average of 59% versus broadcast spraying, per Deere's September 2024 announcement. An Iowa State University Extension study measured 76% average savings and $15.7 per acre. By 2025, coverage reached 5 million acres and 31 million gallons saved.

What the See & Spray numbers demonstrate

Behind those figures is a computer vision system distinguishing weeds from crops and firing individual nozzles as the machine moves, the kind of decision that has to happen on the device, not in the cloud.

Three engineering details matter more than the headline figures:

• The inference budget is tiny. Cameras scan more than 2,100 square feet of crop per second, leaving the model a window measured in low tens of milliseconds.
• Connectivity is unreliable or absent. The machines work in fields, and the system was designed to keep deciding without an uplink.
• Models are updated over the air across a distributed fleet. Every machine is a deployment target with its own version state.

Latency, autonomy, and fleet lifecycle management are what edge AI is really about. Whether a model can be made to run on a device is rarely the constraint anymore. What happens to it after it gets there is.

Edge AI vs. cloud AI: what decides placement

Dimension	Cloud inference	Edge inference
Latency	Round trip plus queueing, tens to hundreds of milliseconds	Bounded by local compute, single-digit to low tens of milliseconds
Privacy and residency	Raw data crosses a boundary and must be governed in transit and at rest	Raw data stays local; only derived signals move upstream
Connectivity	Availability is a hard dependency	Degrades gracefully; state reconciles when the link returns
Cost model	Scales linearly with request volume	Amortized against hardware already deployed

The cost row is the one that changes product strategy. Cloud inference means a successful product becomes more expensive to run in direct proportion to its success, and few other lines in a software P&L behave that way.

Where edge AI is winning right now

• Healthcare devices. GE HealthCare received FDA 510(k) clearance in December 2023 for Critical Care Suite 2.1, which runs pneumothorax detection on the mobile X-ray system itself. The company cites a 57% reduction in reporting times for clinically actionable cases and a 17.7% increase in clinician detection of small pneumothoraces. Triage happens without patient images leaving the device. 
• Autonomous systems. Robotics, drones, and vehicles have a hard real-time deadline between perception and actuation, with a safety consequence for missing it. Cloud assistance is useful for fleet learning and unusable inside the control loop. 
• Industrial IoT. Visual inspection and process control generate continuous high-bandwidth sensor data, almost none of which is worth transmitting, feeding a control loop on the same floor. 
• Retail. Shelf monitoring, loss prevention, and checkout automation run against in-store camera feeds. Sending events rather than video changes both the bandwidth bill and the compliance surface.

Why on-device deployment finally became practical

Capable edge silicon has existed for years. NPUs on mainstream SoCs, NVIDIA Jetson modules, Hailo accelerators, and Google Coral were all shipping well before edge AI projects started reaching production at scale. Compute was not the blocker.

The blocker was the path from a trained model to a deployable artifact, which meant fragile conversion pipelines, numerical mismatches between the training graph and the deployed graph, and in some cases rewriting the model in C++.

That path improved substantially over the last two years. PyTorch shipped ExecuTorch 1.0 in October 2025, with a base runtime of roughly 50KB and delegates for more than a dozen hardware backends, and Meta reports it running in production across its own apps and devices. LiteRT, ONNX Runtime, Core ML, and llama.cpp cover adjacent ground with different tradeoffs around framework neutrality and mobile optimization. Deploying to constrained hardware became a repeatable engineering task rather than a bespoke project each time.

Model compression and what each technique costs

Compression is where model teams and platform teams have to agree on a budget, and all three mainstream techniques trade something away.

• Quantization reduces weight and activation precision from FP32 to int8 or int4, cutting memory footprint and increasing throughput. Post-training quantization applies quickly and can degrade accuracy unpredictably on sensitive layers; quantization-aware training recovers most of that at the cost of a retraining cycle. A format only pays off if the target accelerator implements it natively.
• Distillation trains a smaller student model against a larger teacher and usually gives the best capability per byte. DistilBERT established the pattern, cutting BERT's size by 40% and running 60% faster while retaining 97% of its language understanding on GLUE. The cost is a full training pipeline and a teacher model, which makes it an investment rather than a deployment step.
• Pruning removes redundant weights or structural units. Structured pruning produces real speedups because it yields dense smaller tensors. Unstructured sparsity often does not, because most edge accelerators lack hardware support for sparse execution.

The sequencing that tends to work: distill first to set the architecture, quantize to fit the memory envelope, and treat pruning as a targeted optimization rather than a default.

Hardware constraints that shape architecture

• The accelerator dictates the operator set. NPUs and TPUs are efficient on the operations they implement and fall back to CPU on anything they do not. A single unsupported operator can erase the entire performance advantage, so operator coverage on the target delegate is worth checking before the architecture is finalized. 
• Memory ceilings cap model size before compute does. Available RAM, and whether the platform uses unified or discrete memory, usually determines the feasible parameter count and quantization format. 
• Power and thermal envelopes bound sustained throughput. Benchmarks measure short bursts; devices in enclosures under continuous load throttle. Gartner's Predicts 2026 research on physical AI projects fivefold growth in per-node embedded compute by 2029 against flat power envelopes, which makes this a permanent constraint.

Operational challenges teams underestimate

• Fleet lifecycle management. Updating a model in a cloud service is a deployment. Updating one across thousands of devices means staged rollouts, version skew between model and application code, rollback paths that work offline, and updates that survive a power loss mid-write. Immutable container-native operating systems paired with lightweight Kubernetes distributions such as K3s, MicroShift, and KubeEdge have become the common substrate, with KubeEdge splitting the control plane between cloud and device so nodes keep running when the API server is unreachable.
• Observability without data exfiltration. Inference often moved to the device because the data could not be centralized, which sits in direct tension with knowing how the model performs in the field. Detecting drift requires signals, and deciding which signals can leave the device without recreating the original privacy problem is a design-time decision that is expensive to retrofit.
• Hardware heterogeneity. A fleet accumulates generations with different accelerators, memory, and drivers. The workable pattern is a set of compiled model variants and a device capability registry that decides which variant lands where.

Edge-first is becoming a design philosophy

What makes edge-first a philosophy rather than a deployment choice is the order of reasoning. Edge-first teams start from the constraint, meaning the latency budget, the residency boundary, and the power envelope, then decide what can be centralized. Cloud-first teams build the system and later discover which parts cannot survive a round trip. The first approach produces an architecture. The second produces a migration project.

The escalation boundary deserves the most engineering attention in either case. A local model that handles the common case and routes uncertain inputs to a cloud model captures most of the latency and cost benefit while preserving a ceiling on capability. Instrumenting that confidence threshold so it can be tuned after deployment matters more than the choice of accelerator.

Building edge AI that reaches production

Edge AI succeeds or fails as a platform engineering problem. The model is the smallest component in the system. The compression pipeline, the update path, the device fleet, and the observability plane are where the effort concentrates, and they decide whether a working prototype becomes a shipped product.

Opcito is always open to conversations with teams evaluating edge deployments or working through fleet, lifecycle, and observability problems on hardware already in the field.

One dimension deserves separate treatment. Once model weights, inference, and decision logic sit on hardware an attacker can physically hold, the security assumptions carried over from cloud AI stop applying. That is the subject of the next post: securing edge AI, a different threat model entirely.