3 Survey of literature on driver monitoring

This section describes our survey of the literature on DM and DMSs. The subsections below successively describe (1) our strategy for building an initial set of references, (2) some conclusions drawn from these references, (3) the design of a table for organizing them, (4) comments about the content of this table, and (5,6) trends observable in it or in some references. The analysis performed here guides the developments in subsequent sections.

3.1 Strategy for building an initial set of references, and number of these

To build an initial set of relevant references, we used an approach inspired from Gutiérrez et al. (70). The block (or flow) diagram of Figure 1 describes it.

Our search focused on surveys, reviews, or similar studies about DM and DMSs. We independently performed two searches during February 2021. The first focused on publications from IEEE, Sensors, and ScienceDirect, and the second on publications from ResearchGate; these four databases appeared well-suited for providing a useful set of initial references. We used the search engine specific to each database and a boolean query equivalent to (“survey” OR “review”) AND (“driver” OR “driving”) AND (“detection” OR “detecting” OR “behavior” OR “state” OR “monitoring”). We limited the search to publications in English, and did not place any constraint on the dates of publication. The two searches yielded 124 and 30 items, respectively. After removing 16 duplicates, we obtained a set of 138 references. We manually screened these, and only kept the ones satisfying the two criteria of (1) being in scientific journals or conference proceedings, and (2) providing a survey, review, or similar study of one or more aspects of the domain of interest. This screening led to 56 references, which appear in the first column of Table 2³ , and in the References section, which contains additional references quoted later.

3.2 Conclusions from preliminary analysis of 56 initial references

The preliminary analysis of the 56 initial references led to the following high-level conclusions:

1.

To characterize the (global) state of a driver, one should consider the five main substates of drowsiness, mental workload, distraction, emotions, and under the influence.

2.

A wide variety of parameters, which we call “indicators”, are used to characterize each of these substates, and some indicators are applicable to more than one substate.

3.

Ideally, a DMS should monitor, not only the driver, but also the (driven) vehicle and the (driving) environment.

4.

A value for each indicator is obtained by processing data (mainly signals and images) obtained from sensors “observing” the driver, the vehicle, and the environment.

5.

A DMS generally involves one or more types and/or instances of each of the following: substate, indicator, and sensor.

These conclusions guided the structuring and writing of the bulk of the paper.

When the context is clear, we use “state” for the global state and each of the five substates. The plural “states” and the phrase “state i” each imply that one is talking about one or more substates.

3.3 Design of the structure of the table organizing the initial references

We used the above conclusions to design the structure of a table—namely Table 2—for organizing the 56 initial references in a useful way, in particular for the later synthesis in this paper.

The 56 references are listed in the first column, labelled “References”, by alphabetical order of first author. The three megacolumns following the first column successively correspond to the three key items above, and are accordingly labelled “States”, “Indicators”, and “Sensors”. The last column, labelled “Tests”, indicates whether the technique or system described in a reference was tested in the laboratory, or in real conditions (“in the wild”), or both.

The “States” megacolumn is divided into 5 columns corresponding to the 5 (sub)states of interest. Each of the “Indicators” and “Sensors” megacolumns is divided into 3 columns corresponding to the 3 previously-listed items that a DMS should ideally monitor, i.e., the driver, vehicle, and environment. The column corresponding to the indicators for the driver is further divided into 3 subcolumns corresponding to the qualifiers “physiological”, “behavioral”, and “subjective”. Some other columns could be further subdivided, such as for “Distraction”, but the table deals with such additional subdivisions in a different way.

3.4 Description of the content of the table of references

We successively describe the three megacolumns of Table 2.

States For each of the 56 papers, we indicate which particular (sub)state(s) it addresses. If a paper addresses drowsiness, we place the checkmark “V” in the corresponding column, and similarly for mental workload. For the three other states, we either use a general “V” or give more specific information, often via an abbreviation. There are four types of distraction, i.e., manual, visual, auditory, and cognitive, respectively abbreviated via man, vis, aud, and cog. These types are self-explanatory, but they are addressed later. For emotions, we indicate the type, i.e., stress or anger (ang). For under the influence, we also indicate the type; in all cases, it turns out to be alcohol (alc).

As an example, the second paper, by Alluhaibi et al. (6), addresses drowsiness, distraction, and the emotion of anger.

All abbreviations used in Table 2, for this and other (mega)columns, are defined in Table 3.

Indicators The indicator(s) used by a paper is (are) indicated in the same way as above.

Sensors The sensor(s) used by a paper is (are) indicated in a similar, but not identical, way. If a sensor is embedded in a mobile device (typically a smartphone), rather than in the vehicle, we add a “*”, leading to cam*/mic* for a camera/microphone of a mobile device. In the vehicle column, “V” indicates that the sensor is integrated in the vehicle, whereas “V*” indicates that it is part of a mobile device. For example, the vehicle speed can be obtained via the controller-area-network (CAN) system/bus or a mobile device.

3.5 Trends observable in the table

Table 2 reveals the following trends.

Drowsiness is the most covered state (with 44 references among the total of 56), distraction is the second most covered (with 20 references), and more than one (sub)state is considered in only 19 references.

Indicators are widely used in most references, in various numbers and combinations. Subjective indicators are not frequent (which is to be expected given the constraints of real-time operation). While several authors, such as Dong et al. (44) and Sahayadhas et al. (186), emphasize the importance of the environment and of its various characteristics (e.g., road type, weather conditions, and traffic density), few references (and, specifically, only 6) take it into account.

While the three “Sensors” columns seem well filled, several references either neglect to talk about the sensor(s) they use, or cover them in an incomplete way. Some references give a list of indicators, but do not say which sensor(s) to use to get access to them. References simply saying that, e.g., drowsiness can be measured via a camera or an eye tracker do not help the reader. Indeed, these devices can be head- or dashboard-mounted, and they can provide access to a variety of indicators such as blink dynamics, PERCLOS, and gaze parameters.

Many systems are tested in real conditions, perhaps after initial development and validation in a simulator. Many papers do not, however, systematically document the test conditions for each method that they describe.

3.6 Other trends observable in references

Other trends are not directly observable in Table 2, but can be identified in some individual references.

Experts agree that there does not exist any globally-accepted definition for each of the first four states that we decided to consider. For example, even though many authors try to give a proper definition for drowsiness, there remains a lot of confusion and inconsistencies about the concepts of drowsiness and fatigue, and the difference between them. There is thus a need to define, as precisely as possible, what the first four states are, and this is done in the sequel.

In the more recent references, one sees a trend, growing with time, in the use of mobile devices, and in particular of smartphones (6; 29; 31; 51; 95; 96; 130; 141; 153; 219). A smartphone is relatively low-cost, and one can easily link it to a DMS. This DMS can then use the data provided by the smartphone’s many sensors, such as its inertial devices, microphones, cameras, and navigation system(s). A smartphone can also receive data from wearable sensors (e.g., from a smartwatch), which can provide information such as heart rate (HR), skin temperature, and electrodermal activity (EDA). A smartphone can also be used for its processing unit.