Appendix A. Delphi Survey Methodology to Prioritize Work Zone Performance Measures
The Delphi Survey Method is a systems analysis technique designed to allow individuals to converge on opinions, ratings, priorities, etc. without having to be in face-to-face or video/teleconference communications. In fact, opinions of individuals are kept anonymous so as to avoid creating any biases in the data. The Delphi process itself works best with a limited number of participants, and so the research team, in consultation with FHWA, reached out to 10 practitioners to participate as an advisory panel for the project and assist in the Delphi survey process. Five of the panel participants were state work zone traffic control engineers or equivalent, two were from research institutions and had both performance measurement and work zone safety and mobility backgrounds and expertise, two others were private-sector consultants with performance measurement backgrounds and expertise, and the final panelist was a city traffic engineer from a moderate-sized city on the west coast. For this project, an initial on-line survey instrument was created to obtain initial panel member ratings of the various work zone performance measures and measurement categories, and to provide any thoughts on additional measures or categories that could prove useful to agencies and other practitioners for evaluating and monitoring work zone safety and mobility impacts. The survey is included as Appendix A. Each participant was asked to consider each proposed performance measure in terms of its importance or value they would place upon it (assuming the measure accurately reflected the data upon which it was calculated). Panelists rated each measure on a scale from 1 to 5, with a “1” indicating that they considered the measure to be of low importance, and a “5” indicating that they considered the measure to be of high importance. Any comments they wished to offer in terms of their thought processes were welcomed as well.
Once the initial ratings were obtained, researchers computed the median rating value, and both the 25th and 75th percentile rating values submitted. The median (50th percentile) value was where one-half (5) of the panelists rated the measure higher, and one-half rated it lower. The 25th and 75th percentile rating values were calculated in a similar manner. If all panelists gave the measure the same rating value, the 25th, median, and 75th percentile value would be the same.
All ratings that fell in this range were judged to indicate general agreement with the median rating value, whereas values outside of that range were considered to not be in general agreement. A second iteration of the survey was then developed that was tailored to each individual panel member. The panelist received a reminder of their initial rating value for that measure or category, the median value given by the overall panel, and a notice if their rating did not fall within the inter-quartile range for the group (i.e., the value was an outlier). The panel member then decided if they wanted to change their rating value to something within the inter-quartile range, or keep the rating outside the range and provide a justification for their rating value (i.e., why they rated the measure or category higher or lower than the group). This was repeated a third time so that everyone could see the justifications provided and offered a chance to change their ratings as well.
Overall, the process worked extremely well, with the median rating values stabilizing after the second iteration, and the inter-quartile ranges for almost all of the measures and categories decreasing dramatically between the first and second iterations. Only a few final changes occurred during the third iteration of the survey, which had no effect on the median values and small reduction in inter-quartile ratings for a few of the measures and categories.
< Previous
