Edge Computing Gateway - When Something Goes Wrong

Edge Computing Gateway - When Something Goes Wrong

The one habit

Automatic recovery lands on your last backup. A fresh backup recovers to yesterday; no backup recovers to factory state.

With a fresh backup
Recovery lands you somewhere recent.
Recovers to yesterday
With no backup
Recovery restores the factory baseline.
Recovers to factory state

RobustOS Pro devices are built to recover from failures on their own. A failed install simply aborts — the device keeps running its old software. A device that cannot boot recovers automatically — it restores your last backup and starts again.

There is one thing the device cannot do for you: create that backup. The device never makes a backup automatically — not even before an update. Whatever backup you created last is where automatic recovery will land. A device with a fresh backup recovers to yesterday; a device with no backup recovers to its factory state.

The one habit
The whole recovery story reduces to one habit: back up before every change.

The recovery ladder

Six stages, mild to severe. The first three need no one on site.

1
An install fails
Aborts cleanly. The old software keeps running.
2
The device cannot boot
After three failed attempts it restores your last backup.
3
The kernel is damaged
Falls back to the backup kernel copy. Designed to be invisible.
Hands on the device required from stage 4 on.
4
RST button
Unreachable or misconfigured: a state restore, not a re-installation.
5
Base OS corrupted: the last resort. It starts the device over.
6
Verified-boot devices
Boot code the device cannot verify has no override. Prevention is the mitigation.

Read each stage in detail

Failures are handled in stages, from mild to severe. Each stage below tells you what the device does on its own, and what — if anything — you need to do.

1
An install fails
Aborts cleanly — the device keeps running its old software.
2
The device cannot boot
Automatic after three failed attempts — restores your last backup and starts again.
3
The kernel is damaged
The boot software falls back to the backup kernel copy — designed to be invisible.
Hands on the device required
4
Unreachable or misconfigured — RST button
Two paths — while running (reboot / default configuration / deep restore) and during power-on (deep restore). A state restore, not a re-installation.
5
Base OS corrupted — USB re-flash
Full re-image from a USB stick. The last resort; starts the device over.
6
Verified-boot devices: unverifiable boot software
A locked device still accepts correctly signed boot-software updates — but boot code it cannot verify has no override. Prevention is the mitigation.
The recovery ladder, mild to severe. Each stage is described below.

Stage 1 An install fails

What happens: The installation aborts and the device keeps running the software it had. Nothing is left half-applied at the package level.

What you do: Check the device logs for the reason (a corrupted download and insufficient space are common causes), resolve it, and try again. No recovery action is needed — the device never left its working state.

Stage 2 The device cannot boot

What happens: After three failed boot attempts, the device recovers automatically: it restores your most recent backup — boot software, kernel, system state, and containers — and starts again. If no backup exists on the device, it restores the factory baseline instead.

What you do: Nothing to trigger it — it is automatic. Afterwards, verify the device state and re-apply anything you changed after that backup was created. The backup itself is kept, not consumed — it remains available for the future.

The one exception: a factory restore clears the device’s on-device backup store. Automatic recovery depends on that store, so a device that has just been factory-restored has nothing to recover to until you create a new backup. Make a backup part of your post-restore checklist.

This is why the backup habit matters: your last backup is the exact point automatic recovery returns you to.

Web UISystem → Backup & Restore
Drop a screenshot of the Backup & Restore page
System → Backup & Restore holds two tabs — Snapshot and Backup. Pictured: the Backup tab, the one automatic recovery restores from, with the Automatic Failback setting that drives it. No backups exist on this device yet — the state in which automatic recovery has nothing to fall back to.

Stage 3 The kernel is damaged

What happens: The device keeps a separate backup copy of the kernel. If the operational kernel is damaged, the boot software falls back to the backup copy; if more is wrong, the automatic recovery from stage 2 takes over.

That kernel copy is stored with the on-device backups, which means a factory restore clears it too — this stage is unavailable on a device that has just been factory-restored, until the next firmware update or backup puts a fresh copy in place.

What you do: Usually nothing — this stage is designed to be invisible. If the device came back up after an unexpected reboot, check the logs and consider re-applying your latest firmware update.

Stage 4 The device is unreachable or misconfigured — the RST button

What happens: The RST button has two paths — one while the device is running, one during power-on — and on the running path what you get depends on how long you hold the button and whether you did the same thing a moment ago.

While the device is running:

How long you hold RSTWhat happens
Under 2 secondsNothing — the press is ignored.
2–5 secondsThe device reboots.
5–10 secondsThe device restores its default configuration. This resets settings; it does not clear the device's software. The device generally does not reboot on its own afterwards.
5–10 seconds again, within a minute of the firstDeep restore — the device rewrites its software state from the factory package, then reboots. This route is enabled by default and can be switched off in the device configuration (devcfg.deep_restore_enable).
Longer than 10–15 secondsNothing — the action times out. Release inside the window you want.

The RUN light tells you where you are while you hold: slow flashing from about 2 seconds, fast flashing from about 5 seconds, steady on at about 10–15 seconds, and back to normal indication after that.

During power-on: power the device off, hold RST while powering it back on, and release 2–7 seconds later — the LEDs cycle while you hold, and speed up once you release inside the window. This performs the same deep restore, then reboots.

What the deep restore reaches: your configuration, installed software and data are returned to the factory baseline — what that baseline contains depends on your unit (see below). It also clears the device's on-device backup store, including the backup copy of the kernel. The base operating system itself is not re-flashed: this is a state restore, not a re-installation. A default-configuration restore is narrower — it resets settings and leaves your software in place.

What you do: Use RST when you have lost access to the device — a forgotten configuration, a broken network setup — and remote recovery is not possible. You need physical access to the device. Decide first which of the two outcomes you want — settings back to default, or the whole software state back to the factory baseline — because the second one takes your installed software with it. Watch the RUN light and release inside the window. Afterwards, restore your configuration from a backup or re-configure the device, and create a new backup: after a deep restore the device has nothing to recover to until you do.

Before you press it, know what "factory state" means for your unit — see below.

Stage 5 The base operating system is corrupted — USB re-flash

What happens: If the base OS itself is damaged (for example, a full-image install failed beyond retry), the stages above cannot repair it — they restore state, not the operating system. The recovery path is a full re-image from a USB stick, with physical access to the device.

What you do: This is the last resort, and it starts the device over — everything you installed is erased (see What an Update Changes for exactly what that means). Restore from a clone package you wrote to external storage beforehand. If you reach this stage on a production device, involve your support contact — a corrupted base OS is rare and worth a root-cause look.

Stage 6 Verified-boot devices: boot software damaged after Secure Boot is locked

What happens: On devices provisioned for verified boot, once the Secure Boot fuses are locked, the device only boots software signed with the correct key. If the boot software is ever damaged or replaced with something the device cannot verify, the device cannot be recovered — there is no override. This is not a defect: it is the security guarantee doing its job. A device that could be talked out of verification would not be secure.

What you do: Prevention is the only mitigation, and it is straightforward:

The RST button, at a glance

How long you hold decides what happens. Watch the RUN light and release inside the window.

HoldWhat happens
Under 2 secondsNothing.
2–5 secondsThe device reboots.
5–10 secondsDefault configuration. Settings reset; software stays.
5–10 seconds again, within a minuteDeep restore to the factory baseline, then reboot.
Longer than 10–15 secondsNothing: the action times out.
A deep restore takes your installed software with it, and clears the on-device backups. Create a new backup afterwards.

Know what "factory state" means for your unit

Recovery can land on the factory baseline. What that contains depends on your unit.

Standard unit
Clean default: empty device, default settings.
Custom factory image
Your delivery state, pre-configured at production.
After a clone restore
The clone: it became the new baseline.
Enrolled in RCMS? After a reset or re-image the device re-registers by serial number, as long as RCMS is still enabled.

Stages 2, 4, and 5 can land your device on its factory baseline — and what that contains depends on your unit:

Standard unit

Clean default — empty device, default settings

Custom factory image

Your delivery state — pre-configured at production

After a clone restore

The clone — it became the new baseline
"Factory state" is whatever baseline your unit carries — check with /etc/version-clone.
  • On a standard unit, the factory baseline is a clean state: default settings, no customer software.
  • On a unit ordered with a custom factory image, the baseline is your company's pre-configured image — recovery restores your intended delivery state, not an empty device. That is by design.
  • After a clone restore, the clone is the new baseline — recovery returns to the clone.

You can check which baseline your unit carries from the command line (/etc/version-clone is present on units carrying a custom or cloned image).

One reassurance for fleet operators: if the device is enrolled in RCMS, its cloud identity is derived from its serial number, so after a reset or re-image it re-registers automatically — as long as RCMS is still enabled. A reset that returns configuration to defaults also switches the RCMS agent off; re-enable it and the device comes back online in your fleet by serial number, with no re-enrollment.

The three habits that make recovery painless

A current backup is what keeps recovery in the automatic stages.

1
And keep a copy off the device: a clone package on external storage.
2
Know your factory baseline
Before you need the RST button, not after.
3
Plan physical access
RST and USB re-flash need hands on the device.

1. Back up before every change — updates, package installs, configuration experiments. The device does not do it for you, and your last backup is both your manual restore point and the automatic recovery's landing point. Keep a copy off the device too — a clone package written to external storage — because on-device backups do not survive a full-image install.

2. Know your factory baseline — check whether your unit carries a clean state or a custom image before you need the RST button, not after.

3. Plan physical access for the last resorts. Automatic recovery (stages 1–3) needs no one on site. The RST button and USB re-flash (stages 4–5) need hands on the device. For remote and unattended sites, this makes the backup habit count double — a current backup is what keeps recovery in the automatic stages.

What an Update Changes
What each operation keeps, replaces or erases.
Updating Your Device
How to perform updates, and the three uses of USB.
Securing Your Device
Securing your device at deployment.

What this guide does not cover

Next · step 10 of 13How-to: step by step →Every task the guides ask of you, click by click.