Running machine learning near a sensor can be useful, but “edge” is not automatically the right answer. Some IoT workloads benefit from local inference, others from central computing, and many need a combination. The decision should follow the operating requirement: what data enters the model, when an answer is needed, who uses it and how the system will be maintained after the demonstration.
1. Define the complete response-time budget
Start with an end-to-end requirement rather than a model benchmark. For image inspection, the path may include exposure, image transfer, preprocessing, inference, result delivery and operator display. A fast model cannot compensate for a slow or unreliable stage elsewhere.
State the acceptable delay and what should happen when that limit is exceeded. Local inference may reduce dependence on a remote round trip, while cloud inference introduces a network path that needs measurement. Neither arrangement automatically provides deterministic, safety-critical control.
Also distinguish continuous analysis from occasional requests. A model running once per hour has different hardware and connectivity demands from one processing an ongoing stream.
2. Place each part of the workload deliberately
Consider local execution where data volumes, response needs or connectivity make sending every input upstream impractical. An illustrative inspection setup might analyse images on site and forward selected results for review. That still requires a decision about which images to retain, who can access them and when they are deleted.
Cloud processing can be considered where central management, larger compute requirements or information from several sites is important. A hybrid design might keep immediate inference local while performing approved model development and fleet-level analysis centrally.
Compare the full operating cost: devices, installation, connectivity, cloud resources, update work and field maintenance. Edge hardware is not free to manage, and a cloud endpoint is not the whole cost of a cloud workflow.
3. Benchmark the model on the intended device
Test representative inputs on the actual hardware class, including preprocessing and output handling. Measure response time, memory use, sustained operation and behaviour under expected environmental conditions. Confirm that required model operations are supported by the chosen runtime and accelerator.
Model optimisation is a tradeoff to validate, not a checkbox. Google’s LiteRT quantization guidance explains how reduced numerical precision can affect deployment, with calibration needs for integer approaches. Establish a working baseline and re-evaluate task performance after conversion rather than assuming the smaller model behaves identically.
Include difficult but legitimate inputs. For vision, this can mean changed lighting, partial obstruction or an inspection view near the acceptance boundary. A tidy demo set is not enough.
4. Test connectivity and dependency failures
List every dependency needed for the local workflow: model files, licences, credentials, clocks, storage, configuration and any remote services. Local inference does not make a remote document lookup or cloud authentication step available during an outage.
Microsoft’s IoT Edge offline guidance shows why local operation and message retention must be considered separately. Buffering depends on available storage and configured retention. For the selected platform, test what continues, what queues and what stops when connectivity is lost.
Make degraded operation visible. Users should be able to distinguish a current result from an old one and a model fault from a network fault. Define recovery behaviour before a backlog reaches production systems.
5. Prepare the model lifecycle checklist
A deployment needs traceable releases, validation and ongoing monitoring. Google Cloud’s MLOps guidance describes these as parts of the production process, not just model training. Apply them at a scale appropriate to the project.
- Record the model version, input schema, preprocessing and compatible device configuration.
- Retain a representative evaluation set and agreed acceptance criteria.
- Test a release on a controlled device group before wider rollout.
- Confirm update integrity, device access controls and a tested rollback route.
- Monitor data quality, runtime health and reviewed examples of model performance.
- Assign responsibility for investigating drift and approving any retraining or replacement.
Do not assume that detecting a distribution change should trigger an automatic production update. Investigate its cause and follow the agreed approval process.
6. Choose a design you can operate
Common pitfalls include optimising only inference speed, overlooking physical device maintenance and deploying a model without a recovery plan. Data kept on an edge device still needs protection; moving it out of the cloud does not remove access or retention responsibilities.
A realistic first outcome is a measured deployment choice for one workload, with known limits and an owner for its lifecycle. Explore Skymics’ edge gateway and on-device ML services and AI capabilities, or discuss your device and inference requirement.