In this study, we address the critical need for enhanced situational awareness and victim detection capabilities in Search and Rescue (SAR) operations amidst disasters. Traditional unmanned ground vehicles (UGVs) often struggle in such chaotic environments due to their limited manoeuvrability and the challenge of distinguishing victims from debris. Recognising these gaps, our research introduces a novel technological framework that integrates advanced gesture-recognition with cutting-edge deep learning for camera-based victim identification, specifically designed to empower UGVs in disaster scenarios. At the core of our methodology is the development and implementation of the Meerkat Optimization Algorithm—Stacked Convolutional Neural Network—Bi—Long Short Term Memory—Gated Recurrent Unit (MOA-SConv-Bi-LSTM-GRU) model, which sets a new benchmark for hand gesture detection with its remarkable performance metrics: accuracy, precision, recall, and F1-score all approximately 0.9866. This model enables intuitive, real-time control of UGVs through hand gestures, allowing for precise navigation in confined and obstacle-ridden spaces, which is vital for effective SAR operations. Furthermore, we leverage the capabilities of the latest YOLOv8 deep learning model, trained on specialised datasets to accurately detect human victims under a wide range of challenging conditions, such as varying occlusions, lighting, and perspectives. Our comprehensive testing in simulated emergency scenarios validates the effectiveness of our integrated approach. The system demonstrated exceptional proficiency in navigating through obstructions and rapidly locating victims, even in environments with visual impairments like smoke, clutter, and poor lighting. Our study not only highlights the critical gaps in current SAR response capabilities but also offers a pioneering solution through a synergistic blend of gesture-based control, deep learning, and purpose-built robotics. The key findings underscore the potential of our integrated technological framework to significantly enhance UGV performance in disaster scenarios, thereby optimising life-saving outcomes when time is of the essence. This research paves the way for future advancements in SAR technology, with the promise of more efficient and reliable rescue operations in the face of disaster.

The performance of the robotic manipulator is negatively impacted by outside disturbances and uncertain parameters. The system’s variables are also highly coupled, complex, and nonlinear, indicating that it is a multi-input, multi-output system. Therefore, it is necessary to develop a controller that can control the variables in the system in order to handle these complications. This work proposes six control structures based on neural networks (NNs) with proportional integral derivative (PID) and fractional-order PID (FOPID) controllers to operate a 2-link rigid robot manipulator (2-LRRM) for trajectory tracking. These are named as set-point-weighted PID (W-PID), set-point weighted FOPID (W-FOPID), recurrent neural network (RNN)-like PID (RNNPID), RNN-like FOPID (RNN-FOPID), NN+PID, and NN+FOPID controllers. The zebra optimization algorithm (ZOA) was used to adjust the parameters of the proposed controllers while reducing the integral-time-square error (ITSE). A new objective function was proposed for tuning to generate controllers with minimal chattering in the control signal. After implementing the proposed controller designs, a comparative robustness study was conducted among these controllers by altering the initial conditions, disturbances, and model uncertainties. The simulation results demonstrate that the NN+FOPID controller has the best trajectory tracking performance with the minimum ITSE and best robustness against changes in the initial states, external disturbances, and parameter uncertainties compared to the other controllers.

Implementing and deploying advanced technologies are principal in improving manufacturing processes, signifying a transformative stride in the industrial sector. Computer vision plays a crucial innovation role during this technological advancement, demonstrating broad applicability and profound impact across various industrial operations. This pivotal technology is not merely an additive enhancement but a revolutionary approach that redefines quality control, automation, and operational efficiency parameters in manufacturing landscapes. By integrating computer vision, industries are positioned to optimize their current processes significantly and spearhead innovations that could set new standards for future industrial endeavors. However, the integration of computer vision in these contexts necessitates comprehensive training programs for operators, given this advanced system’s complexity and abstract nature. Historically, training modalities have grappled with the complexities of understanding concepts as advanced as computer vision. Despite these challenges, computer vision has recently surged to the forefront across various disciplines, attributed to its versatility and superior performance, often matching or exceeding the capabilities of other established technologies. Nonetheless, there is a noticeable knowledge gap among students, particularly in comprehending the application of Artificial Intelligence (AI) within Computer Vision. This disconnect underscores the need for an educational paradigm transcending traditional theoretical instruction. Cultivating a more practical understanding of the symbiotic relationship between AI and computer vision is essential. To address this, the current work proposes a project-based instructional approach to bridge the educational divide. This methodology will enable students to engage directly with the practical aspects of computer vision applications within AI. By guiding students through a hands-on project, they will learn how to effectively utilize a dataset, train an object detection model, and implement it within a microcomputer infrastructure. This immersive experience is intended to bolster theoretical knowledge and provide a practical understanding of deploying AI techniques within computer vision. The main goal is to equip students with a robust skill set that translates into practical acumen, preparing a competent workforce to navigate and innovate in the complex landscape of Industry 4.0. This approach emphasizes the criticality of adapting educational strategies to meet the evolving demands of advanced technological infrastructures. It ensures that emerging professionals are adept at harnessing the potential of transformative tools like computer vision in industrial settings.

This paper introduces DAC-HRC, a novel cognitive architecture designed to optimize human-robot collaboration (HRC) in industrial settings, particularly within the context of Industry 4.0. The architecture is grounded in the Distributed Adaptive Control theory and the principles of joint intentionality and interdependence, which are key to effective HRC. Joint intentionality refers to the shared goals and mutual understanding between a human and a robot, while interdependence emphasizes the reliance on each other’s capabilities to complete tasks. DAC-HRC is applied to a hybrid recycling plant for the disassembly and recycling of Waste Electrical and Electronic Equipment (WEEE) devices. The architecture incorporates several cognitive modules operating at different timescales and abstraction levels, fostering adaptive collaboration that is personalized to each human user. The effectiveness of DAC-HRC is demonstrated through several pilot studies, showcasing functionalities such as turn-taking interaction, personalized error-handling mechanisms, adaptive safety measures, and gesture-based communication. These features enhance human-robot collaboration in the recycling plant by promoting real-time robot adaptation to human needs and preferences. The DAC-HRC architecture aims to contribute to the development of a new HRC paradigm by paving the way for more seamless and efficient collaboration in Industry 4.0 by relying on socially adept cognitive architectures.

Soft robots exhibit complex nonlinear dynamics with large degrees of freedom, making their modelling and control challenging. Typically, reduced-order models in time or space are used in addressing these challenges, but the resulting simplification limits soft robot control accuracy and restricts their range of motion. In this work, we introduce an end-to-end learning-based approach for fully dynamic modelling of any general robotic system that does not rely on predefined structures, learning dynamic models of the robot directly in the visual space. The generated models possess identical dimensionality to the observation space, resulting in models whose complexity is determined by the sensory system without explicitly decomposing the problem. To validate the effectiveness of our proposed method, we apply it to a fully soft robotic manipulator, and we demonstrate its applicability in controller development through an open-loop optimization-based controller. We achieve a wide range of dynamic control tasks including shape control, trajectory tracking and obstacle avoidance using a model derived from just 90 min of real-world data. Our work thus far provides the most comprehensive strategy for controlling a general soft robotic system, without constraints on the shape, properties, or dimensionality of the system.

Introduction: Robots present an opportunity to enhance healthcare delivery. Rather than targeting complete automation and nurse replacement, collaborative robots, or “cobots”, might be designed to allow nurses to focus on high-value caregiving. While many institutions are now investing in these platforms, there is little publicly available data on how cobots are being developed, implemented, and evaluated to determine if and how they support nursing practice in the real world.

Methods: This systematic review investigates the current state of cobotic technologies designed to assist nurses in hospital settings, their intended applications, and impacts on nurses and patient care. A comprehensive database search identified 28 relevant peer-reviewed articles published since 2018 which involve real studies with robotic platforms in simulated or actual clinical contexts.

Results: Few cobots were explicitly designed to reduce nursing workload through administrative or logistical assistance. Most included studies were designed as patient-centered rather than nurse-centered, but included assistance for tasks like medication delivery, vital monitoring, and social interaction. Most applications emerged from India, with limited evidence from the United States despite commercial availability of nurse-assistive cobots. Robots ranged from proof-of-concept to commercially deployed systems.

Discussion: This review highlights the need for further published studies on cobotic development and evaluation. A larger body of evidence is needed to recognize current limitations and pragmatic opportunities to assist nurses and patients using state-of-the-art robotics. Human-centered design can assist in discovering the right opportunities for cobotic assistance. Committed research-practice partnerships and human-centered design are needed to guide the technical development of nurse-centered cobotic solutions.

The Expanded Endoscopic Endonasal Approach, one of the best examples of endoscopic neurosurgery, allows access to the skull base through the natural orifice of the nostril. Current standard instruments lack articulation limiting operative access and surgeon dexterity, and thus, could benefit from robotic articulation. In this study, a handheld robotic system with a series of detachable end-effectors for this approach is presented. This system is comprised of interchangeable articulated 2/3 degrees-of-freedom 3 mm instruments that expand the operative workspace and enhance the surgeon’s dexterity, an ergonomically designed handheld controller with a rotating joystick-body that can be placed at the position most comfortable for the user, and the accompanying control box. The robotic instruments were experimentally evaluated for their workspace, structural integrity, and force-delivery capabilities. The entire system was then tested in a pre-clinical context during a phantom feasibility test, followed up by a cadaveric pilot study by a cohort of surgeons of varied clinical experience. Results from this series of experiments suggested enhanced dexterity and adequate robustness that could be associated with feasibility in a clinical context, as well as improvement over current neurosurgical instruments.

Social technology can improve the quality of social lives of older adults (OAs) and mitigate negative mental and physical health outcomes. When people engage with technology, they can do so to stimulate social interaction (stimulation hypothesis) or disengage from their real world (disengagement hypothesis), according to Nowland et al.‘s model of the relationship between social Internet use and loneliness. External events, such as large periods of social isolation like during the COVID-19 pandemic, can also affect whether people use technology in line with the stimulation or disengagement hypothesis. We examined how the COVID-19 pandemic affected the social challenges OAs faced and their expectations for robot technology to solve their challenges. We conducted two participatory design (PD) workshops with OAs during and after the COVID-19 pandemic. During the pandemic, OAs’ primary concern was distanced communication with family members, with a prevalent desire to assist them through technology. They also wanted to share experiences socially, as such OA’s attitude toward technology could be explained mostly by the stimulation hypothesis. However, after COVID-19 the pandemic, their focus shifted towards their own wellbeing. Social isolation and loneliness were already significant issues for OAs, and these were exacerbated by the COVID-19 pandemic. Therefore, such OAs’ attitudes toward technology after the pandemic could be explained mostly by the disengagement hypothesis. This clearly reflect the OA’s current situation that they have been getting further digitally excluded due to rapid technological development during the pandemic. Both during and after the pandemic, OAs found it important to have technologies that were easy to use, which would reduce their digital exclusion. After the pandemic, we found this especially in relation to newly developed technologies meant to help people keep at a distance. To effectively integrate these technologies and avoid excluding large parts of the population, society must address the social challenges faced by OAs.

This study proposes a modular platform to improve the adoption of gamification in conventional physical rehabilitation programs. The effectiveness of rehabilitation is correlated to a patient’s adherence to the program. This adherence can be diminished due to factors such as motivation, feedback, and isolation. Gamification is a means of adding game-like elements to a traditionally non-game activity. This has been shown to be effective in providing a more engaging experience and improving adherence. The platform is made of three main parts; a central hardware hub, various wired and wireless sensors, and a software program with a stream-lined user interface. The software interface and hardware peripherals were all designed to be simple to use by either a medical specialist or an end-user patient without the need for technical training. A usability study was performed using a group of university students and a group of medical specialists. Using the System Usability Scale, the system received an average score of 69.25 ± 20.14 and 72.5 ± 17.16 by the students and medical specialists, respectively. We also present a framework that attempts to assist in selecting commercial games that are viable for physical rehabilitation.

Introduction: Flow state, the optimal experience resulting from the equilibrium between perceived challenge and skill level, has been extensively studied in various domains. However, its occurrence in industrial settings has remained relatively unexplored. Notably, the literature predominantly focuses on Flow within mentally demanding tasks, which differ significantly from industrial tasks. Consequently, our understanding of emotional and physiological responses to varying challenge levels, specifically in the context of industry-like tasks, remains limited.

Methods: To bridge this gap, we investigate how facial emotion estimation (valence, arousal) and Heart Rate Variability (HRV) features vary with the perceived challenge levels during industrial assembly tasks. Our study involves an assembly scenario that simulates an industrial human-robot collaboration task with three distinct challenge levels. As part of our study, we collected video, electrocardiogram (ECG), and NASA-TLX questionnaire data from 37 participants.

Results: Our results demonstrate a significant difference in mean arousal and heart rate between the low-challenge (Boredom) condition and the other conditions. We also found a noticeable trend-level difference in mean heart rate between the adaptive (Flow) and high-challenge (Anxiety) conditions. Similar differences were also observed in a few other temporal HRV features like Mean NN and Triangular index. Considering the characteristics of typical industrial assembly tasks, we aim to facilitate Flow by detecting and balancing the perceived challenge levels. Leveraging our analysis results, we developed an HRV-based machine learning model for discerning perceived challenge levels, distinguishing between low and higher-challenge conditions.

Discussion: This work deepens our understanding of emotional and physiological responses to perceived challenge levels in industrial contexts and provides valuable insights for the design of adaptive work environments.

Force is crucial for learning psychomotor skills in laparoscopic tissue manipulation. Fundamental laparoscopic surgery (FLS), on the other hand, only measures time and position accuracy. FLS is a commonly used training program for basic laparoscopic training through part tasks. The FLS is employed in most of the laparoscopic training systems, including box trainers and virtual reality (VR) simulators. However, many laparoscopic VR simulators lack force feedback and measure tissue damage solely through visual feedback based on virtual collisions. Few VR simulators that provide force feedback have subjective force metrics. To provide an objective force assessment for haptic skills training in the VR simulators, we extend the FLS part tasks to haptic-based FLS (HFLS), focusing on controlled force exertion. We interface the simulated HFLS part tasks with a customized bi-manual haptic simulator that offers five degrees of freedom (DOF) for force feedback. The proposed tasks are evaluated through face and content validity among laparoscopic surgeons of varying experience levels. The results show that trainees perform better in HFLS tasks. The average Likert score observed for face and content validity is greater than 4.6 ± 0.3 and 4 ± 0.5 for all the part tasks, which indicates the acceptance of the simulator among subjects for its appearance and functionality. Face and content validations show the need to improve haptic realism, which is also observed in existing simulators. To enhance the accuracy of force rendering, we incorporated a laparoscopic tool force model into the simulation. We study the effectiveness of the model through a psychophysical study that measures just noticeable difference (JND) for the laparoscopic gripping task. The study reveals an insignificant decrease in gripping-force JND. A simple linear model could be sufficient for gripper force feedback, and a non-linear LapTool force model does not affect the force perception for the force range of 0.5–2.5 N. Further study is required to understand the usability of the force model in laparoscopic training at a higher force range. Additionally, the construct validity of HFLS will confirm the applicability of the developed simulator to train surgeons with different levels of experience.

Despite robots being applied in various situations of modern society, some people avoid them or do not feel comfortable interacting with them. Designs that allow robots to interact appropriately with people will make a positive impression on them resulting in a better evaluation of robots, which will solve this problem. To establish such a design, this study conducted two scenario-based experiments focusing on the politeness of the robot’s conversation and behavior, and examined the impressions caused when the robot succeeds or slightly fails at a task. These two experiments revealed that regardless of whether the partner is a robot or a human, politeness not only affected the impression of interaction but also the expectations for better task results on the next occasion. Although the effect of politeness on preference toward robot agents was smaller than those toward human agents when agents failed a task, people were more likely to interact with polite robots and human agents again because they thought that they would not fail the next time. This study revealed that politeness motivates people to interact with robots repeatedly even if they make minor mistakes, suggesting that the politeness design is important for encouraging human-robot interaction.

Soft grippers are garnering increasing attention for their adeptness in conforming to diverse objects, particularly delicate items, without warranting precise force control. This attribute proves especially beneficial in unstructured environments and dynamic tasks such as food handling. Human hands, owing to their elevated dexterity and precise motor control, exhibit the ability to delicately manipulate complex food items, such as small or fragile objects, by dynamically adjusting their grasping configurations. Furthermore, with their rich sensory receptors and hand-eye coordination that provide valuable information involving the texture and form factor, real-time adjustments to avoid damage or spill during food handling appear seamless. Despite numerous endeavors to replicate these capabilities through robotic solutions involving soft grippers, matching human performance remains a formidable engineering challenge. Robotic competitions serve as an invaluable platform for pushing the boundaries of manipulation capabilities, simultaneously offering insights into the adoption of these solutions across diverse domains, including food handling. Serving as a proxy for the future transition of robotic solutions from the laboratory to the market, these competitions simulate real-world challenges. Since 2021, our research group has actively participated in RoboSoft competitions, securing victories in the Manipulation track in 2022 and 2023. Our success was propelled by the utilization of a modified iteration of our Retractable Nails Soft Gripper (RNSG), tailored to meet the specific requirements of each task. The integration of sensors and collaborative manipulators further enhanced the gripper’s performance, facilitating the seamless execution of complex grasping tasks associated with food handling. This article encapsulates the experiential insights gained during the application of our highly versatile soft gripper in these competition environments.

Companion robots are aimed to mitigate loneliness and social isolation among older adults by providing social and emotional support in their everyday lives. However, older adults’ expectations of conversational companionship might substantially differ from what current technologies can achieve, as well as from other age groups like young adults. Thus, it is crucial to involve older adults in the development of conversational companion robots to ensure that these devices align with their unique expectations and experiences. The recent advancement in foundation models, such as large language models, has taken a significant stride toward fulfilling those expectations, in contrast to the prior literature that relied on humans controlling robots (i.e., Wizard of Oz) or limited rule-based architectures that are not feasible to apply in the daily lives of older adults. Consequently, we conducted a participatory design (co-design) study with 28 older adults, demonstrating a companion robot using a large language model (LLM), and design scenarios that represent situations from everyday life. The thematic analysis of the discussions around these scenarios shows that older adults expect a conversational companion robot to engage in conversation actively in isolation and passively in social settings, remember previous conversations and personalize, protect privacy and provide control over learned data, give information and daily reminders, foster social skills and connections, and express empathy and emotions. Based on these findings, this article provides actionable recommendations for designing conversational companion robots for older adults with foundation models, such as LLMs and vision-language models, which can also be applied to conversational robots in other domains.

Navigation of mobile agents in unknown, unmapped environments is a critical task for achieving general autonomy. Recent advancements in combining Reinforcement Learning with Deep Neural Networks have shown promising results in addressing this challenge. However, the inherent complexity of these approaches, characterized by multi-layer networks and intricate reward objectives, limits their autonomy, increases memory footprint, and complicates adaptation to energy-efficient edge hardware. To overcome these challenges, we propose a brain-inspired method that employs a shallow architecture trained by a local learning rule for self-supervised navigation in uncharted environments. Our approach achieves performance comparable to a state-of-the-art Deep Q Network (DQN) method with respect to goal-reaching accuracy and path length, with a similar (slightly lower) number of parameters, operations, and training iterations. Notably, our self-supervised approach combines novelty-based and random walks to alleviate the need for objective reward definition and enhance agent autonomy. At the same time, the shallow architecture and local learning rule do not call for error backpropagation, decreasing the memory overhead and enabling implementation on edge neuromorphic processors. These results contribute to the potential of embodied neuromorphic agents utilizing minimal resources while effectively handling variability.

We recently developed a biomimetic robotic eye with six independent tendons, each controlled by their own rotatory motor, and with insertions on the eye ball that faithfully mimic the biomechanics of the human eye. We constructed an accurate physical computational model of this system, and learned to control its nonlinear dynamics by optimising a cost that penalised saccade inaccuracy, movement duration, and total energy expenditure of the motors. To speed up the calculations, the physical simulator was approximated by a recurrent neural network (NARX). We showed that the system can produce realistic eye movements that closely resemble human saccades in all directions: their nonlinear main-sequence dynamics (amplitude-peak eye velocity and duration relationships), cross-coupling of the horizontal and vertical movement components leading to approximately straight saccade trajectories, and the 3D kinematics that restrict 3D eye orientations to a plane (Listing’s law). Interestingly, the control algorithm had organised the motors into appropriate agonist-antagonist muscle pairs, and the motor signals for the eye resembled the well-known pulse-step characteristics that have been reported for monkey motoneuronal activity. We here fully analyse the eye-movement properties produced by the computational model across the entire oculomotor range and the underlying control signals. We argue that our system may shed new light on the neural control signals and their couplings within the final neural pathways of the primate oculomotor system, and that an optimal control principle may account for a wide variety of oculomotor behaviours. The generated data are publicly available at https://data.ru.nl/collections/di/dcn/DSC_626870_0003_600.

Teleoperation allows workers to safely control powerful construction machines; however, its primary reliance on visual feedback limits the operator’s efficiency in situations with stiff contact or poor visibility, hindering its use for assembly of pre-fabricated building components. Reliable, economical, and easy-to-implement haptic feedback could fill this perception gap and facilitate the broader use of robots in construction and other application areas. Thus, we adapted widely available commercial audio equipment to create AiroTouch, a naturalistic haptic feedback system that measures the vibration experienced by each robot tool and enables the operator to feel a scaled version of this vibration in real time. Accurate haptic transmission was achieved by optimizing the positions of the system’s off-the-shelf accelerometers and voice-coil actuators. A study was conducted to evaluate how adding this naturalistic type of vibrotactile feedback affects the operator during telerobotic assembly. Thirty participants used a bimanual dexterous teleoperation system (Intuitive da Vinci Si) to build a small rigid structure under three randomly ordered haptic feedback conditions: no vibrations, one-axis vibrations, and summed three-axis vibrations. The results show that users took advantage of both tested versions of the naturalistic haptic feedback after gaining some experience with the task, causing significantly lower vibrations and forces in the second trial. Subjective responses indicate that haptic feedback increased the realism of the interaction and reduced the perceived task duration, task difficulty, and fatigue. As hypothesized, higher haptic feedback gains were chosen by users with larger hands and for the smaller sensed vibrations in the one-axis condition. These results elucidate important details for effective implementation of naturalistic vibrotactile feedback and demonstrate that our accessible audio-based approach could enhance user performance and experience during telerobotic assembly in construction and other application domains.