The previous section shows the key role played by the triad of states, indicators, and sensors (which are also emphasized in Figure 2) in driver-state characterization, which is the first of two key steps in DM, and the object of this paper. The present section describes our approach to synthesize, in terms of this triad, the techniques for driver-state characterization found in the literature.
Our approach aims at answering, in a simple, visual way, the two following questions: (1) For a given state, what indicator(s) can one use? (2) For a given indicator, what sensor(s) can one use? We achieve this goal by naturally providing two tables (or matrices) of “states vs indicators” and “sensors vs indicators”. These two tables can be viewed as being two-dimensional (2D) views of a 3D table (or array) of “states vs indicators vs sensors”, as illustrated in Figure 3, where the positions shown for the three dimensions and for the “dihedral” they subtend make the tables on the right appear in numerical order from top to bottom. The figure shows visually that the tables share the “Indicators” dimension, and are thereby interlocked. It gives a simplified representation of each of the tables that are progressively filled in Sections 6-10, i.e., Tables 4 and 5.
In Figure 3, the simplified representations of Tables 4 and 5 give the high-level structure of these tables.
In Table 2, the megacolumn “Indicators” is partitioned into the three columns “Driver”, “Vehicle”, and “ Environment”. Figure 3 shows, via the simplified representations, that Table 4 and Table 5 are also partitioned in this way, but in megarows and with the corresponding abbreviations D, V, and E. In Table 2, the megacolumn “Sensors” is partitioned in the same way as the megacolum “Indicators”. This is reflected in Figure 3 by the partitioning of Table 5 into the megacolumns D, V, and E. The figure shows that Table 4 is partitioned into the 5 megacolumns corresponding to the five states, denoted here by S1, ..., S5, where Si stands for “State i”, which appear in the titles of the next five sections.
Each lowest-level cell in both tables is destined to contain 0, 1, or more related references.
The pair of tables allows one to answer other questions such as: (1) If one invests in the calculation of an indicator for a particular state, what other state(s) can this indicator be useful for? (2) If one invests in a particular sensor for a particular state, what other state(s) can this sensor be useful for?
The rows and columns of Table 4 and Table 5 are further divided as follows. The D-megarows of Table 4 and Table 5 are subdivided as the D-megacolumns of Table 2 are, i.e., into the rows “Physiological”, “Behavioral”, and “Subjective”.
The D-megacolumns of Table 5 are subdivided in a way that does not already appear in Table 2, i.e., into the columns “Seat”, “Steering wheel”, “Safety belt”, “Internal camera”, “Internal microphone”, and “Wearable”. Observe that the D-megarows and D-megacolumns are not divided in the same way, even though they correspond to the driver.
The V- and E- rows and columns are also further divided as necessary.
We give examples for the various categories of indicators and sensors that are further discussed in the next five sections. We use the above terminology of X-based indicators and X-centric sensors, where X can be replaced by driver (or D), vehicle (or V), or environment (or E).
Indicators D-based indicators relate to the state of the driver. They include physiological indicators (e.g., heart activity, brain activity, electrodermal activity (EDA)), behavioral indicators (e.g., eye blinks, gaze direction, hands positions), and subjective indicators (which are not suited for real-world operation, but can be used for validation at some point in the development of a DMS).
V-based indicators relate to how the driver control his/her vehicle, e.g., how he/she controls the speed, steers, and brakes.
E-based indicators relate to the state of the environment, viewed here as consisting of three parts: (1) the outside environment (outside of vehicle), (2) the inside environment (inside of vehicle), and (3) the contextual environment (independent of the previous two). Examples of characteristics of these parts of the environment are, respectively, (1) the road type, weather conditions, and traffic density, (2) the temperature and noise, and (3) the time of day and day of year. Each of these characteristics (e.g., road type) can be used as an E-based indicator.
Sensors Some D-centric sensors are placed in the seat (e.g., radar for breathing activity), steering wheel (e.g., electrodes for electrocardiogram (ECG)), and safety belt (e.g., magnetic induction (MI) sensors). Some D-centric sensors, in particular cameras (e.g., RGB) and microphones, are appropriately placed in the cockpit to monitor the driver. We qualify these sensors of “internal”, to distinguish them from similar sensors monitoring the external environment, and qualified of “external”. Some D-centric sensors are wearables (e.g., a smartwatch measuring HR and/or skin temperature). Since the aim is to monitor the state of the driver, we assume throughout this paper that the seat, safety belt, and similar items are related to the driver.
V-centric sensors are mostly the sensors—whether integrated in the vehicle or not—that allow for the acquisition of vehicle parameters such as speed, steering angle, and braking level. Such parameters are often obtained via the CAN bus. But sensors (e.g., accelerometers, gyroscopes) built into recent mobile devices can also provide some of this information.
E-centric sensors are the sensors that allow for the acquisition of parameters related to the environment. Cameras and radars can provide, e.g., information about the driving scene.
The next five sections successively cover the five selected states in detail. In general, each section defines a state, the indicators that characterize it, and the sensors that allow access to them, and progressively fills Table 4 and Table 5 with relevant references.
At the end of the last of these five sections, both tables are complete. They, together with the explanations in the five sections, constitute the main contribution of this paper.
The structures of Tables 2, 4, and 5 were obtained after a significant number of iterations. This implies that the ultimate structure of Table 2 was informed by the content of Sections 4 to 10.