A generalized methodology for evaluating alerting systems is presented. The methodology is used to construct System Operating Characteristic curves that describe the tradeoffs between unnecessary alerts and collisions based on probabilistic models of sensors, vehicle dynamics, and human response. An appropriate alerting threshold location can then be selected and parametric studies can be performed to examine the relative benefits of alternative sensor types and accuracies. Different avoidance maneuver options (e.g., swerving or braking) can be evaluated in terms of their ability to reduce the probability of a collision. A simplified example application is presented for a ground vehicle rear-end collision alerting system. The relative benefits of increased sensor accuracy vs. improved driver response time and braking deceleration are examined. It is shown that uncertainty in human response time is the key factor affecting the performance of the alerting system. In contrast, sensor accuracy specifications are shown to be entirely adequate: system performance is not significantly impacted by expected sensor errors.

Keywords: alerting, warning, collision avoidance, modeling, false alarm

Alerting systems are important components of human-operated vehicles that provide warning of hazards in time to allow the operator to avoid an undesirable incident. To date, alerting system design for vehicles has focused primarily on aerospace applications, and a number of complex alerting systems are currently in place on aircraft, including systems that warn of internal hazards (e.g., engine fire) and external hazards (e.g., terrain or other traffic on a collision course). Recent interest in developing alerting systems for ground vehicles provides an opportunity to translate experience gained from aerospace applications to the ground vehicle domain [1,2]. Example applications include systems to warn operators of collisions with other vehicles or to alert the operator that the vehicle is drifting off the roadway. Ground vehicle applications are especially challenging because accidents generally occur much more quickly than in aircraft and the operators are more diverse and do not have training specific to the alerting system in their vehicle.

The design of alerting systems requires a systems-level study of sensors, alerting logic, human performance, and the dynamics of an encounter with a threat [3]. There is a tradeoff between alerting too early (and having frequent unnecessary alerts) and alerting too late (and having a high accident rate). These interdependencies are often difficult to visualize, leading to the fact that current design methods typically follow an ad hoc, evolutionary process. Incidents and unnecessary alerts have traditionally been resolved as they arise through modifications in alerting thresholds followed by numerical simulation to predict changes in safety [4,5].

This paper presents a methodology to model hazard situations and to produce curves that directly show the tradeoffs between unnecessary alerts and incidents. Alerting thresholds can then be selected based on these tradeoffs. Additionally, parametric studies can be performed to determine, for example, the relative impact that a change of sensor accuracy has on overall system performance. The methodology has previously been demonstrated in aerospace applications including analyses of current and proposed traffic collision alerting systems and the Ground Proximity Warning System (GPWS) [3,6,7]. In this paper, the methodology is demonstrated in a simplified ground vehicle rear-end collision warning system application. The relative benefit of increased sensor accuracy is contrasted against the relative benefit of shortened driver reaction times and improved knowledge of braking deceleration rates.

The first step in developing an alerting system is to model the hazard situations that are of interest. Figure 1 shows a simplified situation involving a vehicle and a generic hazard. At the current moment in time shown in the figure, the hazard is located in a certain estimated position relative to the vehicle, . The estimated position may contain errors, due primarily to sensor inaccuracies, which are shown by an error ellipse surrounding the vehicle. From this viewpoint, the alerting system considers the vehicle to be located in the position shown in Figure 1, though the true position of the vehicle could lie elsewhere as described by the error ellipse.

Figure 1. Generalized Hazard Encounter Situation

The situation in Figure 1 is not static: the vehicle is traveling along some Nominal Trajectory whose direction relative to the hazard can be estimated. This Nominal Trajectory is based on current knowledge of the intended paths of the vehicle and hazard. In the example shown in Figure 1, the Nominal Trajectory is a line, though in general it may follow a more complex, curved path.

Because of uncertainties, the Nominal Trajectory is probabilistic: the shaded region in Figure 1 represents the set of trajectories that could be followed with some probability. In general, the uncertainty in the future location of the vehicle relative to the hazard will grow as the extrapolated time increases. The uncertainties in the current state () and in the Nominal Trajectory combine to result in the case that a collision with the hazard will occur in the future with some probability. However, the alerting system must decide whether an alert should be issued based only on current information.

The approach taken here is to examine the hazard situation in Figure 1 from the standpoint of the alerting system. That is, the state estimate (which may contain errors) is provided to the alerting system, and the system will issue an alert if and only if exceeds an alerting threshold defined by the state vector . For simplicity, it is assumed that the alerting system is always functioning: if exceeds the threshold an alert is issued with probability 1. If is below the threshold, an alert is not issued, with probability 1. However, because contains errors, unnecessary alerts may still occur. Whether the alert is considered to be unnecessary is one metric of alerting system performance and can also be a design constraint.

Assume now that an alert is issued at the time shown in Figure 1 (that is, = ). The alert is defined to be unnecessary if, assuming the alert had not been issued, an incident would not occur. That is, there would have been no collision had the alert not been issued. With this definition, the probability of an unnecessary alert, P(UA), depends on the probability that an incident will occur along the Nominal Trajectory [denoted PN(I | )]. As described earlier, at the moment that an alert is issued, = . If an alert is issued when PN(I | = ) is small, the alert is likely to be unnecessary because it is likely that an incident would have been avoided without the alert. In particular, when an alert is issued, the probability that it is unnecessary is given by:

P(UA) = 1 - PN(I | = ) (1)

As the vehicle continues along the Nominal Trajectory, PN(I | ) will generally approach either 0 or 1 as it becomes more clear whether the hazard is a true threat. To minimize unnecessary alerts, the alert should be delayed until PN(I | ) is close to 1. If the alert is delayed too long, however, there may be insufficient time and space to avoid an incident. It is therefore necessary to also consider the trajectory that is followed after an alert is issued.

Figure 2 shows two hypothetical outcomes of the decision to alert at a given time. If no alert is issued, the vehicle will follow the probabilistic Nominal Trajectory described previously. If an alert is issued, however, the vehicle will instead follow an Avoidance Trajectory along which the vehicle may attempt to avoid the hazard using some procedure. The Avoidance Trajectory is also probabilistic, and in general may include some probability that no action is taken. Because there is some latency in the reaction to an alert, there may be a region near the vehicle where the Nominal and Avoidance Trajectories coincide. If a hazard is within this region, an incident could occur before the operator is able to respond to an alert.

Figure 2. Outcome Trajectories of an Alerting Decision

In Figure 2, the Avoidance Trajectory involves a simple turn to the left following some delay in the reaction to the alert. In general, more complicated Avoidance Trajectories can be considered. It is also possible that the Avoidance Trajectory accounts for actions taken by the hazard in response to the vehicle (if the hazard also has an alerting system).

There is a corresponding probability of an incident along the Avoidance Trajectory, denoted PA(I | ). If PA(I | ) is close to 1, then it is likely that an alert is too late or is issued at an inappropriate time and an incident may occur even with (or because of) the alert. Another important metric can now be defined: the probability of successful alert, P(SA). An alert is classified as successful if no incident occurs (regardless of whether the alert was necessary). Thus, when an alert is issued, it is a successful alert with probability

P(SA) = 1 - PA(I | = ) (2)

Note that with these definitions, an alert could be classified as unnecessary but also as successful. An example is an alert issued for a hazard that is far away: the alert may be unnecessary because an incident would not have occurred anyway, and the alert may be successful because an incident is still avoided.

P(UA) and P(SA) are important metrics of alerting system performance. To maximize performance, it is necessary to minimize unnecessary alerts and maximize successful alerts. To minimize unnecessary alerts, the alert should be delayed, until PN(I | ) is large. To maximize safety, the alert should be issued early, when PA(I | ) is small (and P(SA) is large). These goals cannot generally be met simultaneously and some tradeoff must be made.

Figure 3. Example SOC Curve

The tradeoff between unnecessary alerts and successful alerts can be shown using a System Operating Characteristic (SOC) curve, shown in Figure 3. SOC curves are similar to Receiver Operating Characteristic (ROC) curves in Signal Detection Theory (SDT) and allow the alerting decision to be recast as a conventional signal detection problem [8,9]. This enables the use of established SDT methods to determine an optimal alert threshold.

The shape of an SOC curve is a function of the probability density functions that describe sensor accuracy, human performance (such as response time), and the situation and avoidance maneuver being performed. Each point on the SOC curve represents an alerting threshold setting. For example, in Figure 3 one potential alerting threshold location is shown as 1. This threshold setting corresponds to alerting when the hazard is a certain distance ahead of the vehicle. When issued, an alert in this case will be unnecessary with probability 0.70 and will be successful with probability 0.98. That is, when an alert is issued there is a 70% chance that the alert was not necessary and a 2% chance that a collision will still occur. As the alerting threshold is moved, the operating point moves along the SOC curve to reflect the tradeoff between unnecessary and successful alerts. For example, if the alerting threshold is moved to delay alerting (thus alerting when the hazard is closer to the vehicle), the operating point moves from location 1 toward location 2: unnecessary alerts are reduced but at the expense of fewer successes.

If the threshold is moved too far (to location 3), P(UA) becomes equal to P(SA): this indicates that the alert is of no benefit. An example is a situation in which the time before impacting a hazard is less than the response delay to an alert: the outcomes are the same whether or not an alert is issued because there is insufficient time to respond.

An ideal alerting system would operate at the upper-left corner of the plot, but in general alerting systems are constrained to operate on SOC curves that do not pass through the ideal location. However, as uncertainties in the situation are reduced or as the avoidance maneuver becomes more aggressive, the SOC curve will approach the ideal operating point.

P(UA) and P(SA) can be calculated using several techniques. If the uncertainties in the situation are gaussian, the probability that an incident will occur may be calculated through the propagation of error covariance matrices. If the probability density functions are more complex, numerical integration or Monte Carlo simulation can be used to estimate the probabilities. In general, Monte Carlo simulation is relatively simple to implement and can provide accurate results if the probabilities of interest are not small or if a large amount of computer processing power is available.

The methodology is demonstrated here through an example application to a rear-end collision situation. In this example, a host vehicle is equipped with an alerting system that has access to vehicle state information including velocity and distance from a stationary hazard in the roadway ahead. It is desired that the alerting system warn the driver when the vehicle is in danger of colliding with the hazard, but to remain silent if the driver could brake normally. Thus, based on predicted braking deceleration, an alert is issued some distance from the hazard. In this example, the relative degradation in alerting system performance is compared parametrically for sensor accuracy, driver response time, and braking deceleration.

3.1 Situation

To simplify the problem, only one rear-end situation is examined here, though the methodology could straightforwardly be extended to cover a range of potential situations. In this example, the host vehicle is assumed to be traveling at v = 35 miles/hr (15.6 m/s) and is approaching a stationary hazard a distance r ahead (Figure 4). When an alert is issued, the driver will react after an amount of time, t, and decelerate at a constant rate, a. Whether the host vehicle hits the hazard can be determined from simple kinematics. The distance taken to stop, d, is given by:

(3)

If d is greater than r, then a collision occurs.

Figure 4. Braking Distance

Next, uncertainty in the state variables is introduced, and Monte Carlo simulation is used to estimate the probability of a collision given that an alert has occurred, PA(I | = ). The probability density functions for the state variables are defined in the following sections.

In order to estimate PN(I | = ), it is necessary to define situations in which an alert is considered unnecessary. It is assumed here that if an alert is issued in a case in which a collision would not occur with t = 1.5 s and a = 10 ft/s2, that alert is unnecessary. That is, it is assumed that a typical unalerted driver responds to roadway hazards within 1.5 s and decelerates at 10 ft/s2. Again, by using probability density functions and Monte Carlo simulation, the probability of a collision can be estimated. At this point, the definition of unnecessary alert is rather arbitrary, though a more rigorous definition could be obtained through a study of what constitutes nominal driving behavior. Additionally, the methodology presented here can be used to examine the sensitivity of alerting system performance to this definition.

The time at which an alert is issued, talert, defines the alerting threshold setting. This alert time is referenced to the projected time to impact:

(4)

As defined, talert is the time remaining before braking is required to begin to avoid a collision. Thus, if talert = 0, then the alert is issued such that braking must begin immediately in order to avoid a collision. As talert becomes larger, the alert is issued earlier, providing more time before braking must be applied. If talert = t, the alert is issued such that exactly the required time to stop remains if the assumed response delay and deceleration level are correct. If talert is greater than t, the alert is issued earlier than this nominal time; talert < t means that the alert is issued later than this nominal alert time. Because of uncertainties, a collision may be possible even if talert is greater than t. Similarly, it may be that a collision is still avoided when talert is less than t.

3.2 Perfect sensors

To begin, a deterministic situation is examined. In particular, it is assumed that the sensors are perfectly accurate and that the driver responds in 1 second to an alert and begins braking at a deceleration of 16 ft/s2. These assumptions will be relaxed subsequently to examine the relative impact of uncertainty on performance.

When there are no uncertainties, there are only three possible operating points as shown in Figure 5. In this case, the SOC curve collapses into three points because the problem is deterministic (all probabilities are 0 or 1). If talert is greater than 2.5 s, the alert is unnecessary based on the assumed nominal delay of 1.5 s and deceleration at 10 ft/s2. This corresponds to the point at the upper right in Figure 5 [P(UA) = 1 and P(SA) = 1]. If talert is between 2.5 and 1.0 s, then the alert is necessary and successful. This corresponds to the point at the upper left on the SOC curve [P(UA) = 0 and P(SA) = 1]. If talert is less than 1.0 s, there is not enough time to avoid a collision [lower left corner: P(UA) = 0 and P(SA) = 0]. Ideal performance for this system can be obtained by setting the alert threshold anywhere between 2.5 and 1.0 s.

Figure 5. Perfect Sensor SOC Curve

3.3 Sensor uncertainty

Next, uncertainty is added to the problem through sensor inaccuracies. In particular, it is now assumed that there are normally-distributed range and velocity sensor errors equal in magnitude to current specifications for rear-end alerting systems.3 Namely, the standard deviations of range and velocity errors are 0.7 m and 0.4 m/s respectively.

The performance of the system is no longer deterministic as was the case in the previous example. Instead, for each possible value of talert, there is some probability of unnecessary alert and some probability of successful alert. The results from this case, determined through Monte Carlo simulation, are shown in Figure 6 as an SOC curve. Several example thresholds are marked in terms of talert in italics. If talert is set between 1.75 and 1.45 s, the system will perform nearly ideally: within the accuracy of the calculations (over 10,000 Monte Carlo evaluations), P(UA) = 0 and P(SA) = 1. As talert moves outside this window, unnecessary or unsuccessful alerts can occur. For example, if the alert is triggered earlier than 1.75 s, the probability of unnecessary alert rises (e.g., when talert = 2.5 s, P(UA) = 0.60). Likewise, if the alert is issued later than 1.45 s, then the probability of successful alert decreases (e.g., when talert = 1.0 s, P(SA) = 0.50).

Figure 6. SOC Curve with Sensor Uncertainty

The SOC curve in Figure 6 shows that sensor accuracy does not place a major limitation on system performance. This is because the current specifications for sensor accuracy are stringent; higher sensor uncertainty would certainly produce a larger negative impact on system performance.

3.4 Uncertain driver response latency

The previous example is unrealistic in that a human driver will not respond consistently in exactly 1 second to an alert or brake at precisely 16 ft/s2. To examine the impact of these issues on system performance, uncertainty in the driver's response latency is now added to the sensor errors.

Previous work has shown that a surprised driver's response latency can be modeled as a lognormal distribution with a mode of 1.07 s and a dispersion parameter of z = 0.49.2 This lognormal probability density function was incorporated into the Monte Carlo simulation to produce the SOC curve shown in Figure 7. Note that the curve now clearly departs from the ideal operating point.

Figure 7. SOC Curve with Sensor and Response Time Uncertainty

(threshold values of talert in seconds shown in italics)

As can be seen by the SOC curve, P(SA) decreases rapidly as talert is decreased from 2.0 s, but P(UA) increases rapidly as talert is moved above 1.5 s. One reasonable threshold setting could be talert = 1.6 s. Such a setting would provide a relatively high P(SA) with a relatively low P(UA). Alternatively, the system could be designed with talert = 2.4 s, in which case successful alerts would be issued 95% of the time [P(SA) = 0.95], and unnecessary alerts would be issued approximately 46% of the time. If the system were instead designed so that unnecessary alerts occurred with probability 0.05 (talert = 1.6 s), then the probability of successful alert would be 0.78. In this way, the SOC curve can be used to examine the tradeoffs in system performance.

Figure 7 clearly shows that uncertain response latency has a major impact on the performance of the system. A system designed to protect slowly-responding drivers would have a high unnecessary alert rate for rapidly-responding drivers. If the system were designed to have few unnecessary alerts, however, there would also be a relatively high accident rate. The only way to combat this tradeoff is to decrease response time uncertainty. This could be achieved through driver training, improved alerting displays, or by tuning each system to the driver that is using it. For example, perhaps the system could be designed such that the threshold is set to alert earlier for slowly-responding drivers than for quickly-responding drivers.

3.5 Uncertain braking deceleration

The uncertainties impacting system design do not end at driver response. Because roadway conditions and the aggressiveness of driver braking are also uncertain, it is furthermore necessary to recast a as a random variable. For this example, it is assumed that a is normally-distributed with a mean equal to the previously assumed values (16 ft/s2 for the alert condition, or 10 ft/sec2 for the unnecessary alert condition) and a standard deviation of 3 ft/s2. Again, SOC curves and performance values were calculated and shown in Figure 8.

Figure 8. SOC Curve Uncertain Sensors, Response Time, and Deceleration

(threshold values of talert in seconds shown in italics)

Comparing Figure 8 to Figure 7, the relative magnitude of the effect of uncertain deceleration can be estimated. For example, for the system in Figure 8 to provide a successful alert rate of 95% (i.e., talert = 2.6 s), the unnecessary alert rate would have to be 0.48 as compared to 0.46 when deceleration is known perfectly. Similarly, if the system in Figure 8 were designed so that P(UA) is 0.05 (i.e., talert = 1.2 s), then P(SA) would be 0.53 as opposed to 0.74 when deceleration is known perfectly.

This additional decrement in alerting performance could be reduced by increasing the accuracy of the deceleration estimate. For example, through driver training it could be possible to better predict the aggressiveness with which a driver will brake. Additionally, the uncertainty in deceleration due to roadway conditions could be reduced through sensors that determine whether the roadway is dry, wet, or icy.

This paper presents a generalized methodology for the development and evaluation of alerting systems. Based on probabilistic models of sensors, dynamics, and the human operator, the probability of collisions or unnecessary alerts can be determined and used to create System Operating Characteristic (SOC) curves. The SOC curves aid in placing alerting thresholds and can be used to examine the impact of changes in sensors or other components of the system.

The methodology is demonstrated using a simplified example of a ground vehicle rear-end collision alerting system. The relative impact of three types of uncertainty (sensors, driver reaction latency, and braking deceleration) are discussed. Using SOC curves, it is shown that driver reaction latency has a major impact on the performance of the alerting system. Because of variations in driving style and ability, some drivers may perceive alerts to be unnecessary while other drivers may react too late to avoid a collision even with an alert. Braking deceleration is also shown to reduce system effectiveness, suggesting that a means to predict roadway conditions is important. In contrast, current sensor specifications are shown to be entirely adequate: the reduction in system performance due to sensor errors is significantly less than that due to driver latency or braking deceleration. It appears, therefore, that efforts would be better spent on improving driver reaction times and on providing accurate estimates of roadway conditions and braking ability than on improved sensor accuracy.

Although the methodology presented in this paper is applied to a simplified example, it can be straightforwardly extended to treat alerting design problems in more depth. This would require detailed dynamical models and probability distributions backed by more complete data than used in this example. Additionally, it would be necessary to apply the methodology to a range of potential hazard situations at different driving speeds and roadway conditions. However, the use of numerical simulation and Monte Carlo evaluation is well suited to automating this process, and the extension of the methodology to a more thorough analysis should be reasonably simple.

1. Ervin, R., "Evaluating Active Safety Technologies for the Motor Vehicle", Collision Avoidance Systems Issues and Opportunities Proceedings, ITS America and National Highway Traffic Safety Administration, pp. 59-97, Reston, VA, March 21-22, 1994.

2. Najm, W., Mironer, M., Koziol, J., Wang, J. S., and R. Knipling, "Synthesis Report: Examination of Target Vehicular Crashes and Potential ITS Countermeasures", US Department of Transportation Final Report DOT-VNTSC-NHTSA-95-4, Cambridge, MA, June, 1995.

3. Kuchar, J. K., "Methodology for Alerting-System Performance Evaluation", AIAA Journal of Guidance, Control, and Dynamics, Vol. 19, No. 2, March-April, 1996.

4. Miller, C., Williamson, T., Walsh, J., Nivert, L., Anderson, J., and A. Zeitlin, "Initiatives to Improve TCAS-ATC Compatibility", Journal of ATC, July-September, 1994.

5. Drumm, A. C., "Lincoln Laboratory Evaluation of TCAS II Logic Version 6.04a", MIT Lincoln Laboratory Report ATC-240, Vols. 1 and 2, Lexington, MA, February 21, 1996.

6. Pritchett, A., Carpenter, B., Asari, K., Kuchar, J. K., and R. J. Hansman, "Issues in Airborne Systems for Closely-Spaced Parallel Runway Operations", Proceedings of the 14th AIAA/IEEE Digital Avionics Systems Conference, pp. 140-145, Cambridge, MA, November, 1995.

7. Carpenter, B. D., and J. K. Kuchar, "A Probability-Based Alerting Logic for Aircraft on Parallel Approach", MIT Aeronautical Systems Laboratory Report, ASL-96-3, Cambridge, MA, June, 1996.

8. Sheridan, T. B. and W. R. Ferrell, Man-Machine Systems: Information, Control, and Decision Models of Human Performance, MIT Press, Cambridge, MA, 1974.

9. Barkat, M., Signal Detection and Estimation, Artech House, Boston, MA, 1991.

10. Wilson, T., "IVHS Countermeasures for Read-End Collisions", Workshop on Peer Review of the National Highway Traffic Safety Administration Program, Baltimore, MD, December 11-12, 1995.