This paper describes the development and implementation of IoBT-MAX, a multimodal analytics experimentation testbed designed to support research and evaluation of Internet of Battlefield Things (IoBT) technologies. The testbed consists of a distributed set of edge nodes with multimodal sensing and compute capabilities coupled with a high-precision GPS localization system, and a remote monitoring and control platform. The testbed is designed to support research on multiple analytic tasks including object classification, object detection, multi-object tracking, data compression, and communication efficient inference and scheduling. The testbed has been deployed at the Robotics Research Collaboration Campus (R2C2), a DEVCOM Army Research Laboratory (ARL) facility, and is a key research instrumentation project of ARL’s Internet of Battlefield Things Collaborative Research Alliance.
With the rise of hailing services, people are increasingly relying on shared mobility (e.g., Uber, Lyft) drivers to pick up for transportation. However, such drivers and riders have difficulties finding each other in urban areas as GPS signals get blocked by skyscrapers, in crowded environments (e.g., in stadiums, airports, and bars), at night, and in bad weather. It wastes their time, creates a bad user experience, and causes more CO2 emissions due to idle driving. In this work, we explore the potential of Wi-Fi to help drivers to determine the street side of the riders. Our proposed system is called CarFi that uses Wi-Fi CSI from two antennas placed inside a moving vehicle and a data-driven technique to determine the street side of the rider. By collecting real-world data in realistic and challenging settings by blocking the signal with other people and other parked cars, we see that CarFi is 95.44% accurate in rider-side determination in both line of sight (LoS) and non-line of sight (nLoS) conditions, and can be run on an embedded GPU in real-time.
Heteroskedastic Geospatial Tracking with Distributed Camera Networks
Colin Samplawski, Shiwei Fang, Ziqi Wang, Deepak Ganesan, Mani Srivastava, Benjamin M. Marlin
Proceedings of the 39th Conference on Uncertainty in Artificial Intelligence (UAI 2023). July. 2023.
Visual object tracking has seen significant progress in recent years. However, the vast majority of this work focuses on tracking objects within the image plane of a single camera and ignores the uncertainty associated with predicted object locations. In this work, we focus on the geospatial object tracking problem using data from a distributed camera network. The goal is to predict an object’s track in geospatial coordinates along with uncertainty over the object’s location while respecting communication constraints that prohibit centralizing raw image data. We present a novel single-object geospatial tracking data set that includes high-accuracy ground truth object locations and video data from a network of four cameras. We present a modeling framework for addressing this task including a novel backbone model and explore how uncertainty calibration and fine-tuning through a differentiable tracker affect performance.
Optimizing Intelligent Edge-clouds with Partitioning, Compression and Speculative Inference
Shiwei Fang, Jin Huang, Colin Samplawski, Deepak Ganesan, Benjamin Marlin, Tarek Abdelzaher, and Maggie B. Wigness
Military Communication Conference (MILCOM). IEEE, DEC. 2021.
Internet of Battlefield Things (IoBTs) are well positioned to take advantage of recent technology trends that have led to the development of low-power neural accelerators and low-cost high-performance sensors. However, a key challenge that needs to be dealt with is that despite all the advancements, edge devices remain resource-constrained, thus prohibiting complex deep neural networks from deploying and deriving actionable insights from various sensors. Furthermore, deploying sophisticated sensors in a distributed manner to improve decision-making also poses an extra challenge of coordinating and exchanging data between the nodes and server. We propose an architecture that abstracts away these thorny deployment considerations from an end-user (such as a commander or warfighter). Our architecture can automatically compile and deploy the inference model into a set of distributed nodes and server while taking into consideration of the resource availability, variation, and uncertainties.
Exploiting Scene and Body Contexts in Controlling Continuous Vision Body Cameras
Shiwei Fang, Ketan Mayer-Patel, Shahriar Nirjon
Ad Hoc Networks Journal, Elsevier, Volumn 113, Mar. 2021.
Ever-increasing performance at decreasing price has fueled camera deployments in a wide variety of real-world applications—making the case stronger for battery-powered, continuous-vision camera systems. However, given the state-of-the-art battery technology and embedded systems, most battery-powered mobile devices still do not support continuous vision. In order to reduce energy and storage requirements, there have been proposals to offload energy-demanding computations to the cloud Naderiparizi et al. (2016), to discard uninteresting video frames Naderiparizi et al. (2017), and to use additional sensors to detect and predict when to turn on the camera Bahl et al. (2012) . However, these proposals either require a fat communication bandwidth or have to sacrifice capturing of important events.
In this paper, we present — ZenCam, which is an always-on body camera that exploits readily available information in the encoded video stream from the on-chip firmware to classify the dynamics of the scene. This scene-context is further combined with simple inertial measurement unit (IMU)-based activity level-context of the wearer to optimally control the camera configuration at run-time to keep the device under the desired energy budget. We describe the design and implementation of ZenCam and thoroughly evaluate its performance in real-world scenarios. Our evaluation shows a 29.8%–35% reduction in energy consumption and 48.1-49.5% reduction in storage usage when compared to a standard baseline setting of 1920x1080 at 30fps while maintaining a competitive or better video quality at the minimal computational overhead.
EyeFi: Fast Human Identification Through Vision and WiFi-based Trajectory Matching
Human sensing, motion trajectory estimation, and identification are central to a wide range of applications in many domains such as retail stores, surveillance, public safety, public address, smart homes and cities, and access control. Existing solutions either require facial recognition or installation and maintenance of multiple units, or they lack long-term re-identification capability. In this paper, we propose a novel system -- called EyeFi -- that combines WiFi and camera on a standalone device to overcome these limitations. EyeFi integrates a WiFi chipset to an overhead camera and fuses motion trajectories obtained from both vision and RF modalities to identify individuals. In order to do that, EyeFi uses a student-teacher model to train a neural network to estimate the Angle of Arrival (AoA) of WiFi packets from the CSI values. Based on extensive evaluation using real-world data, we observe that EyeFi improves WiFi CSI based AoA estimation accuracy by more than 30% and offers 3,800 times computational speed over the state-of-the-art solution. In a real-world environment, EyeFi's accuracy of person identification averages 75% when the number of people varies from 2 to 10.
This paper introduces SuperRF – which takes radio frequency (RF) signals from an off-the-shelf, low-cost, 77GHz mmWave radar and produces an enhanced 3D RF representation of a scene. SuperRF is useful in scenarios where camera and other types of sensors do not work, or not allowed due to privacy concerns, or their performance is impacted due to bad lighting conditions and occlusions, or an alternate RF sensing system like synthetic aperture radar (SAR) is too large, inconvenient, and costly. Applications of SuperRF includes navigation and planning of autonomous and semi-autonomous systems, human-robot interactions and social robotics, and elderly and/or patient monitoring in-home healthcare scenarios. We use low-cost, off-the-shelf parts to capture RF signals and to train SuperRF. The novelty of SuperRF lies in its use of deep learning algorithm, followed by a compressed sensing-based iterative algorithm that further enhances the output, to generate a fine-grained 3D representation of an RF scene from its sparse RF representation – which a mmWave radar of the same class cannot achieve without instrumenting the system with large sized multiple antennas or physically moving the antenna over a longer period in time. We demonstrate the feasibility and effectiveness through an in-depth evaluation.
In this paper, we present - ZenCam, which is an always-on body camera that exploits readily available information in the encoded video stream from the on-chip firmware to classify the dynamics of the scene. This scene-context is further combined with simple inertial measurement unit (IMU)-based activity level-context of the wearer to optimally control the camera configuration at run-time to keep the device under the desired energy budget. We describe the design and implementation of ZenCam and thoroughly evaluate its performance in real-world scenarios. Our evaluation shows a 29.8-35% reduction in energy consumption and 48.1-49.5% reduction in storage usage when compared to a standard baseline setting of 1920×1080 at 30fps while maintaining a competitive or better video quality at the minimal computational overhead.
Low Swing TSV Signaling using Novel Level Shifters with Single Supply Voltage
Shiwei Fang, Emre Salman
IEEE International Symposium on Circuits and Systems (ISCAS), May 2015.
Low swing TSV signaling is proposed for three-dimensional (3D) integrated circuits (ICs) to reduce dynamic power consumption. Novel level shifters are designed to lower the voltage swing before the TSV and to pull the voltage swing back to full rail at the far end of the TSV. Proposed level shifters operate with a single supply voltage, thereby reducing the overall cost. Critical TSV capacitance beyond which the proposed scheme saves dynamic power is determined. Up to 42% reduction in overall power is demonstrated with a voltage swing of 0.5 V, where the supply voltage is 1 V.
Workshop, Demo, Poster, arXiv:
Augmenting Vibration-Based Customer-Product Interaction Recognition with Sparse Load Sensing
Yue Zhang, Shiwei Fang, Carlos Ruiz, Zhizhang Hu, Shubham Rohal, Shijia Pan
In Proceedings of Cyber-Physical Systems and Internet of Things Week 2023 (CPS-IoT Week '23). May 2023.
This paper introduces a multimodal solution for autonomous retail customer-product interaction recognition using a combination of vibration and load sensing. Scalable and robust customer-product interaction recognition is important for autonomous retail. Current efforts focus on computer vision-based approaches, which are prone to occlusion from both customers and shelves. A densely deployed load cell array can mitigate this issue, however, the high cost and maintenance effort make it difficult to scale up for large deployments. Vibration-based approaches are also explored to detect such interaction, however, the robustness over noisy environments is limited.
We propose a multimodal solution with sparse vibration and load sensing on the shelves. These two modalities are complementary in terms of information – load sensing can effectively detect the weight changes on the shelf while vibration sensing can recognize detailed interaction. Preliminary results are presented to demonstrate the complementarity of these two modalities. Our system is able to augment product recognition performance with the combination of sparse vibration and load sensing.
Efficient IoT Inference via Context-Awareness
Mohammad Mehdi Rastikerdar, Jin Huang, Shiwei Fang, Hui Guan, Deepak Ganesan
While existing strategies to execute deep learning-based classification on low-power platforms assume the models are trained on all classes of interest, this paper posits that adopting context-awareness i.e. narrowing down a classification task to the current deployment context consisting of only recent inference queries can substantially enhance performance in resource-constrained environments. We propose a new paradigm, CACTUS, for scalable and efficient context-aware classification where a micro-classifier recognizes a small set of classes relevant to the current context and, when context change happens (e.g., a new class comes into the scene), rapidly switches to another suitable micro-classifier. CACTUS features several innovations, including optimizing the training cost of context-aware classifiers, enabling on-the-fly context-aware switching between classifiers, and balancing context switching costs and performance gains via simple yet effective switching policies. We show that CACTUS achieves significant benefits in accuracy, latency, and compute budget across a range of datasets and IoT platforms.
With the rise of hailing services, people are increasingly relying on shared mobility (e.g., Uber, Lyft) drivers to pick up for transportation. However, such drivers and riders have difficulties finding each other in urban areas as GPS signals get blocked by skyscrapers, in crowded environments (e.g., in stadiums, airports, and bars), at night, and in bad weather. It wastes their time, creates a bad user experience, and causes more CO2 emissions due to idle driving. In this work, we explore the potential of Wi-Fi to help drivers to determine the street side of the riders. Our proposed system is called CarFi that uses Wi-Fi CSI from two antennas placed inside a moving vehicle and a data-driven technique to determine the street side of the rider. By collecting real-world data in realistic and challenging settings by blocking the signal with other people and other parked cars, we see that CarFi is 95.44% accurate in rider-side determination in both line of sight (LoS) and non-line of sight (nLoS) conditions, and can be run on an embedded GPU in real-time.
Design and Deployment of a Multi-Modal Multi-Node Sensor Data Collection Platform
Sensing and data collection platforms are the crucial components of high-quality datasets that can fuel advancements in research. However, such platforms usually are ad-hoc designs and are limited in sensor modalities. In this paper, we discuss our experience designing and deploying a multi-modal multi-node sensor data collection platform that can be utilized for various data collection tasks. The main goal of this platform is to create a modality-rich data collection platform suitable for Internet of Things (IoT) applications with easy reproducibility and deployment, which can accelerate data collection and downstream research tasks.
Dataset: Person Tracking and Identification using Cameras and Wi-Fi Channel State Information (CSI) from Smartphones
Shiwei Fang, Sirajum Munir, Shahriar Nirjon
The 3rd International Workshop on Data: Acquisition To Analysis (DATA ’20) (SenSys + BuildSys). ACM, Nov. 2020.
Human sensing, motion trajectory estimation, and identification are crucial to applications such as customer analysis, public safety, smart homes and cities, and access control. In the wake of the global COVID-19 pandemic, the ability to perform contact tracing effectively is vital to limit the spread of infectious diseases. Although vision-based solutions such as facial recognition can potentially scale to millions of people for identification, the privacy implications and laws to banning such a technology limit its applicability in the real world. Other techniques may require installations and maintenance of multiple units, and/or lack long-term re-identification capability. We present a dataset to fuse WiFi Channel State Information (CSI) and camera-based location information for person identification and tracking. While previous works focused on collecting WiFi CSI from stationary transmitters and receivers (laptop, desktop, or router), our WiFi CSI data are generated from a smartphone that is carried while someone is moving. In addition, we collect camera-generated real-world coordinate for each WiFi packet that can serve as ground truth location. The dataset is collected in different environments and with various numbers of persons in the scene at several days to capture real-world variations.
Demo Abstract: Fusing WiFi and Camera for Fast Motion Tracking and Person Identification
Shiwei Fang, Sirajum Munir, Shahriar Nirjon
ACM Conference on Embedded Networked Sensor Systems (SenSys), Nov. 2020.
Human sensing, motion tracking, and identification are at the center of numerous applications such as customer analysis, public safety, smart cities, and surveillance. To enable such capabilities, existing solutions mostly rely on vision-based approaches, e.g., facial recognition that is perceived to be too privacy invasive. Other camera-based approaches using body appearances lack long-term re-identification capability. WiFi-based approaches require the installation and maintenance of multiple units. We propose a novel system - called EyeFi - that overcomes these limitations on a standalone device by fusing camera and WiFi data. We use a three-antenna WiFi chipset to measure WiFi Channel State Information (CSI) to estimate the Angle of Arrival (AoA) using a neural network trained with a novel student-teacher model. Then, we perform cross modal (WiFi, camera) trajectory matching to identify individuals using the MAC address of the incoming WiFi packets. We demonstrate our work using real-world data and showcase improvements over traditional optimization-based methods in terms of accuracy and speed.
Non-Line-of-Sight Around the Corner Human Presence Detection Using Commodity WiFi
Devices
Shiwei Fang, Ron Alterovitz, Shahriar Nirjon
Workshop on Device-Free Human Sensing (DFHS). ACM, Nov. 2019.
As robots penetrate into real-world environments, practical human-robot co-existence issues such as the requirement for safe human-robot interaction are becoming increasingly important. In almost every vision-capable mobile robot, the field of view of the robot is occluded by the presence of obstacles such as indoor walls, furniture, and humans. Such occlusions force the robots to be stationary or to move slowly so that they can avoid collisions and violations of entry into the personal spaces of humans. We see this as a barrier to robots being able to optimally plan motions with reasonable speeds. In order to solve this problem, we propose to augment the sensing capability of a robot by using a commodity WiFi receiver. Using our proposed method, a robot can observe the changes in the properties of received signals, and thus be able to infer whether a human is present behind the wall or obstacles, which enhances its ability to plan and navigate efficiently and intelligently.
Demo Abstract: AI-Enhanced 3D RF Representation Using Low-Cost mmWave Radar
Shiwei Fang, Shahriar Nirjon
ACM Conference on Embedded Networked Sensor Systems (SenSys), Nov. 2018.
This paper introduces a system that takes radio frequency (RF) signals from an off-the-shelf, low-cost, 77 GHz mm Wave radar and produces an enhanced 3D RF representation of a scene. Such a system can be used in scenarios where camera and other types of sensors do not work, or their performance is impacted due to bad lighting conditions and occlusions, or an alternate RF sensing system like synthetic aperture radar (SAR) is too large, inconvenient, and costly. The enhanced RF representation can be used in numerous applications such as robot navigation, human-computer interaction, and patient monitoring. We use off-the-shelf parts to capture RF signals and collect our own data set for training and testing of the approach. The novelty of the system lies in its use of AI to generate a fine-grained 3D representation of an RF scene from its sparse RF representation which a mmWave radar of the same class cannot achieve.
Distributed Adaptive Model Predictive Control of a Cluster of Autonomous and Context-Sensitive Body Cameras
Shiwei Fang, Ketan Mayer-Patel, Shahriar Nirjon
Workshop on Wearable Systems and Applications (WearSys). ACM, June 2017.
Increasing deployment of body cameras by the law enforcement agencies makes us rethink the relation between the camera and the public. In contrast to current implementations of a body camera that use a power-hungry default configuration and can only be turned on and off by an officer, we propose an idea that the camera should be autonomous and active all the time. By leveraging the information from an on-board inertial measurement unit (IMU), these autonomous cameras should dynamically adjust their configuration in order to keep the device under the desired energy budget. To enable such a system, we propose a distributed adaptive model predictive controller for a system of body cameras, which allows the collaboration between multiple cameras which is currently not available in existing implementations.