This paper focuses on the characterization of the state of a driver, which is the first key step for driver monitoring (DM) and driver monitoring systems (DMSs). It surveys (in Section 3) the relevant scientific and technical literature on driver-state characterization, and subsequently provides a synthesis (in Sections 4-10) of the main, published techniques for this characterization.
The survey yielded 56 publications in scientific/technical journals and conference proceedings. Their examination led to the conclusion that the state of a driver should be characterized according to the five main dimensions—called here “(sub)states ”—of drowsiness, mental workload, distraction (further divided into four subconstructs, e.g., visual), emotions, and under the influence.
In comparison to standard physical quantities such as voltage and power, these states are not well defined and/or are very difficult—if at all possible—to quantify or to label, not only in a validated way, but also in real time and non-invasively, as is required in the driving context. The only reasonable approach, found almost universally in the literature, is to have recourse to indicators (of each of these states), the value of which can be obtained in a practical and validated way. Examples of indicators are the eye-blink rate, the standard deviation of lane departure (SDLP), and the outside temperature. The values of many indicators (but not all) are obtained by applying algorithms, often complex, to data (typically signals and images) collected from sensors.
The last paragraph brings to light the three ingredients that, in our view, lie at the heart of DM and DMSs, i.e., the triad of states, indicators (of these states), and sensors (providing data, which is the source of the values of these indicators). Figure 2 links these three ingredients.
Our survey confirmed the intuition that one should monitor, not only the driver (D), but also the (driven) vehicle (V) and the (driving) environment (E). Accordingly, we partitioned both the indicators and the sensors into D, V, and E categories, leading to the phrases “X-based indicators” and “X-centric sensors”, where X can be D, V, or E. For the D-based indicators, we further distinguished between 3 types: physiological, behavioral, and subjective. The three examples of indicators given earlier correspond to D, V, and E, respectively.
The major outcome of the paper is the pair of interlocked tables “states vs indicators” (Table 4) and “sensors vs indicators” (Table 5), where each cell contains zero, one, or more references. These tables bring together, in an organized way, most of the useful information found in the literature, up to the time of this writing, about driver-state characterization, for DM and DMSs. These tables constitute an up-to-date, at-a-glance, visual reference guide for anyone active in this field. They provide immediate answers to key questions that arise in the design of DMSs, such as the four questions posed in Section 5.
The pair of tables and the references they contain lead to the following main conclusions:
The next two paragraphs respectively elaborate on the last two points.
For driving safety, it is paramount that the processing and decisions made by any algorithm used in a vehicle, including for DM, be fully explainable (to a human being) at the time of design and certification of this algorithm. Most algorithms using ML do not, however, have this necessary feature of explainability, or interpretability, and this is certainly the case for ML-based algorithms that would learn on-the-fly during one or more trips. Therefore, while ML algorithms and, especially deep-learning algorithms, often provide, on specific datasets, stellar performances in comparison to other types of algorithms, they will almost certainly not be acceptable to an equipment provider or a car manufacturer. There is, however, a trend toward designing ML algorithms that produce results that can be explained (123; 245). The above remarks apply, not only to ML, but also to any approach whose operation cannot be explained simply. Our framework, which implies the use of indicators and states, supports the desired explainability. It indeed prevents any algorithm from going, in one fell swoop, from (nearly-)raw sensor data to driver characterization, by forcing it to estimate both the values of indicators and the levels of states, as a stepping stone toward the ultimate characterization.
The literature on DM focuses almost exclusively on characterizing the “present” state of the driver. We use quotes because the characterization is typically based on data from the recent past, e.g., in a window that extends over several tens of seconds and butts against almost the present time. This results in a characterization of the “recent-past” state of the driver. If the driver is in control, a DMS using this characterization may not have sufficient lead time to take proper emergency action (to issue an alarm and/or to take back the control) and, if the car is in control, such a DMS may hand the control over to the driver even though he/she might be falling asleep or getting distracted in a few tens of seconds or more. A major missing link in current DMS research and development is thus the true prediction of the future state of the driver, at least a few tens of seconds into the future.
On the one hand, Tables 4 and 5 show, at a glance, which areas of driver-state characterization have been the object of research and with what intensity (measured by the number of references listed in each cell). For example, Table 4 shows that significant research has been performed to analyze the emotions of the driver using the driver-based, physiological indicators of heart rate, breathing activity, and electrodermal activity. On the other hand, the two tables show, also at a glance, where little or no research has been performed to date, thereby suggesting new, potentially-fruitful research areas. The two tables should thus prove to be a rich source of information for both research and product development.
Starting from a set of 56 initial references, our exploration of the field of DM led us to examine a total of 254 references. While our criss-crossing of the field, at several different times, led us to identify many relevant publications, our search cannot, obviously, be exhaustive. In any case, the two histograms of “number of references vs year” of Figure 4 (for the 56 and 254 references, respectively) constitute a clue that the research activity in DM has been accelerating over the past decade.
The methodology used in this paper can be applied to update the tables at various times in the future to take into accounts new developments. This can be done by adding and/or removing rows, columns, and/or references, as appropriate.
Characterizing the state of a driver and, more generally, DM will remain important despite the progressive increase in vehicle automation. SAE Level 3 enables vehicles to drive by themselves under certain conditions such as on a highway in sunny weather, but a driver must still be present and able to take back the control of the vehicle at anytime and in a relatively short lapse of time. In order to ensure that the driver is able to take back the control, technologies for monitoring the state of the driver will become even more critical. These technologies are also needed to monitor him/her during the time he/she is driving, and to possibly allow the vehicle to take back the control if necessary.
Currently, some vehicle manufacturers offer DMSs based on the behavior of the driver and/or the behavior of the vehicle, such as the detection of steering-wheel movements and lane deviations, respectively. These systems can be useful in current vehicles with automation up to (SAE) Level 2, but will become obsolete at higher levels of automation. Indeed, when a vehicle drives autonomously, monitoring its behavior does not give any information about the state of the driver, and technologies that directly monitor both the driver and the driving environment are a necessity as long as the driver is involved in the driving task, at least partially.
To date, the development of driving-automation systems (DASs) has moved at a faster pace than the development of DMSs has. This is, in major part, a consequence of the long-held belief by some automotive-industry players that they would be able to easily leapfrog Levels 3 and 4, and move on directly to Level 5, where there is no need to monitor the driver. But, most experts now agree that it will be decades before most privately-owned vehicles are fully automated, if ever. Along the long and winding road to Level 5, the automotive industry will need to significantly boost the research and development on DMSs. For Levels 3 and 4, the same industry will need to develop automated-driving systems (ADSs) and DMSs in full synergy. The future could thus not be brighter for the field of DM and DMSs.
This work was supported in part by the European Regional Development Fund (ERDF).
The authors declare no conflict of interest.