Watch Duty case study

| Security, Customer story
Download now

Protecting emergency fire intel with a robust OTA update infrastructure

 

THE CUSTOMER

image 1

Public Safety | United States
WatchDuty.org

 

Delivering verified situational awareness in emergency situations to stay safe when seconds count.

Nearly every wildfire fatality occurs within the first hour, yet official notifications are often delayed, sparse, or non-existent. Watch Duty, a 501(c)(3) nonprofit, alerts the public of nearby wildfires and firefighting efforts in real-time when every second counts.

The Watch Duty service is powered by real people – active and retired firefighters, dispatchers, and first responders – who monitor radio scanners and collaborate around the clock to bring up-to-the-minute information. Watch Duty’s IoT product, nicknamed Echo, was launched in 2022 to expand capabilities to hear first responder communications deep within forests. This service allows not only the Watch Duty team but first responders and emergency services to hear farther and clearer than ever before – helping to keep all of us safer.

400+

countries

1MM+

downloads

 

Objectives

Group 8455 (2) Reduce resources spent on device management infrastructure
Group 8454 Shorten device diagnosis and remediation processes
Group 8456 (1) Reduce security risk throughout operations and across the Echo fleet
Group 8457 (1) Minimize the risk of significant fleet downtime
Group 8458 Enable more team members to manage the device fleet

 

Watch Duty’s Echo radios deliver remote listening, tone detection, and early warning to improve radio information in remote areas.

The Challenge

Robust and secure management of devices in remote areas

Watch Duty’s Echo radios are deployed far and wide to alleviate the radio dead zones that hinder reporting, first responders, agencies, and volunteer firefighters. With some existing in very rural areas, Echo radios are designed and deployed to remain evergreen.

However, managing, improving, and maintaining Echo radios – remotely, at scale, and in the field – created significant new challenges. Initially, the Watch Duty team built its own remote management infrastructure solution based on remote terminal access (SSH), custom scripts (download updates), and a VPN infrastructure (connectivity).

At dozens of devices, its homegrown remote management solution suffered from scalability and robustness issues. Dealing with individual device issues consumed significant time and effort. Only one employee (its creator) knew how to use the solution, resulting in a single point of failure (SPOF). The employee needed to be available 24/7 – creating internal risks, delays, and pressures – and he was also the only person who could manage the Echo devices. In scenarios where seconds matter, these scalability and SOPF factors could incur massive repercussions, such as radio frequencies changes during a wildfire requiring a device update.

The homegrown management solution also had inherent security risks. The sole employee using the solution, the only one with access, meant his credentials were present on all devices. If his credentials were lost, no devices could be managed. Or worse, if this single credential was compromised, the attacker could take control of all Echo radios. Lastly, the VPN infrastructure created another major attack vector. Any of the Echo radios could see the network traffic of the other radios; a single compromised Echo could impact the entire fleet.

While the Echo radios required a remote management solution, Watch Duty’s homegrown solution was fragile, high risk, and already consumed significant time and effort. The Watch Duty team knew it was just a matter of time until something disruptive would happen as its Echo fleet continued to scale. Watch Duty required a robust, secure, and scalable solution to avoid unplanned fleet downtime and manage future security issues.

 

OTA Software Update Solution Technical Requirements

Tick Integration with GitHub Actions for streamlined CI/CD
Tick Image-based updates for consistent and repeatable software builds
Tick Scalable and repeatable device provisioning
Tick Robustness, even in low and intermittent network connectivity

 

The Solution

Leveraging a scalable and secure OTA update infrastructure

The Watch Duty team knew its Echo fleet required a new remote management solution. A team member experiencing similar challenges in his past company with its homegrown solution recommended Mender, stating, “Every good company in the world uses Mender.” With a solid reputation and track record of managing devices for some of the largest enterprises in the world, Mender offered a battle-tested, secure, robust solution for the Echo product.

With Mender, Watch Duty now has a standardized DevOps release process. Every code change triggers a GitHub Action to build a software package (Mender Artifact) that is automatically uploaded to hosted Mender for deployment. With a few clicks, this package can be deployed to test devices, tested, and rolled out to all production Echo radios.

Watch Duty replaced its entire homegrown infrastructure with Mender, immediately streamlining operations. Using Mender, Watch Duty seamlessly upgrades and maintains Echo radios without physically accessing them, including initiating a power cycle if the system is unresponsive. The Mender Troubleshoot Add-on enables Watch Duty to diagnose and troubleshoot devices in the field; its remote terminal and file transfer features allow Watch Duty to easily download log files or audio snippets from the radios and quickly diagnose any issues.

End-to-end security and encryption protect devices and the entire fleet from security threats, whether unintentional or malicious. The easy interface and role-based access control allow the whole Watch Duty team to utilize the solution in a secure manner. The Mender Configure Add-on securely and uniquely provisions credentials, such as for WiFi, without exposure to other devices, lowering the security attack surface.

“Our deployment process was initially limited, relying on a VPN mesh and custom scripts. Switching to Mender has streamlined our operations, enabling standardized setups with OS images, one-click GitHub PR deployments, and enhanced security through unique user accounts.”

Nick Russell, VP of Operations

 

The Mender Difference

Group 8455 Deploy to test and production environments with a few clicks Group 8456 Standardized and documented software deployment process
    Lightning Rapid diagnosis and remediation of device problems high-level 1 (1) Fast and repeatable new device onboarding

 

Technology Ecosystem

Tick Raspberry Pi 4- Headless Processor: Broadcom BCM2711, Quad core Cortex-A72 (ARM v8) 64-bit SoC @ 1.8GHz
Tick 2GB LPDDR4-3200 SDRAM w/ 8GB SanDisk Industrial SD-card
Tick Diamond D130NJ Super Discone Antenna
Tick RX Frequency Range 25 - 1300 MHz
Tick Active vented IP65 NEMA 3X enclosure designed to withstand harsh weather conditions

 

“This shift has markedly improved our deployment’s robustness, speed, and scalability without the complexities of our previous system."

Nick Russell, VP of Operations

 

 

Benefits

Realizing robustness and speed at scale

With a streamlined and automated software release process using Mender, Watch Duty now enjoys short and low-effort software development iterations – submit code, deploy it, test it, and repeat – all with just a few clicks in the Mender UI, and completing the cycle in less than 5 minutes. Compared to the homegrown setup, this is a major time saver, enabling Watch Duty to move faster and focus on improving its user experience.

As an additional benefit, the whole team can make key changes to the Echo fleet, such as changing the radio frequency of devices with very little effort or know-how. The team can now collaborate due to the lower skill barrier, reducing the risk and stress on single employees.

Scalability is now built-in. New Echo devices can be easily built using a standard process. There is no longer a scalability risk from the infrastructure itself.

With Mender, Watch Duty now spends less time on infrastructure and more time on its products. Watch Duty now has the peace of mind of using a robust and proven OTA update solution.

 

Key Benefits

Tick Reduced time and spend with a fully automated CI/CD process
Tick Mitigated the risk of compromise with new security layers on device and fleet levels
Tick Decreased organizational risk and stress through team-wide fleet visibility and control
Tick Ensure scalability with a future-proof solution capable of meeting requirements for the next decade and beyond

 

 

 

Tags:

Download the case study