RNS
Browse Projects > Detailed View

Transportation Research Culture--Synthesis of Practices

Description:

Scope:

The Fixing America’s Surface Transportation (FAST) Act of 2015 highlighted a keen interest in improving the responsiveness, utility, and quality of transportation research. Research culture underpins the transportation research community. Interest in the subject of research culture was prompted by two papers presented at a recent TRB Annual Meeting. Both papers dealt with the same research question, yet reached entirely opposite conclusions. Immediate questions were obvious. How could this be? What does this outcome reveal about approaches to transportation research? How do we, in the transportation research community, resolve divergent findings from two high quality research projects?

A subsequent TRB paper (1) reviewed these two studies and explored the “culture” of transportation research. Several additional questions that deserve attention and further exploration were identified. These questions suggest that there are likely ways in which transportation research can be bolstered and improved and this subject was further explored at a session at the 2017 TRB Annual Meeting (2). Additionally, recent efforts have begun to improve the quality of papers and publications of the TRB and the National Academies. This Synthesis proposal is designed to help answer these emerging questions and provide input to the TRB administrative efforts.

Proposed tasks described below will answer many questions about the culture of transportation research and identify ways that research practices can be further improved. This proposed Synthesis is notable in that the findings would apply to virtually all areas and specialties of transportation research. It is a broad and overarching topic that reaches across all disciplines. The problems are widespread and of general interest. The topic is timely and the findings will be instrumental in improving research. There is no known other ongoing, or recently completed, research on this subject.

Objective:

An assessment of current research practices and procedures.

Benefits:

This synthesis will help the transportation research community better understand the character of our research culture and, in turn, provide essential insights on how transportation research can be bolstered and improved. These findings will be of value to state departments of transportation and managers of their research programs.

Related Research:
  1. Upchurch, Jonathan. “The Importance of Experimental Study Design in Traffic Control Device Research”. Paper Number 15-3385. 2015 Annual Meeting Compendium of Papers, Transportation Research Board.

  2. Event 766, Transportation Research Board 2017 Annual Meeting. Presentations on research culture by Katie Turnbull, John Milton, Darryll Dockstader, and Jonathan Upchurch.

  3. Upchurch, Jonathan. “The Culture of Transportation Research: What Do We Need to Change?” Draft article for potential publication in TR News.

Tasks:

Task 1: Identify examples of research projects generating divergent results

Occasionally, two (or more) research studies seeking to answer the same question yield contradictory findings. When this happens decision-makers and policy-makers often have a difficult time evaluating the conflicting results. This task will identify sets of research studies that have produced divergent results. These examples should represent a range of specialty areas within transportation research.

Task 2: Determine how, in the transportation research community, decision makers resolve divergent findings from two high quality research projects

Probing the examples identified in Task 1, interview researchers, research managers, decision-makers, and policy-makers to determine how widely differing conclusions from two (or more) research projects were resolved. What were the means of debating the merits of the two projects and resolving the conflict? Were the differing findings and conclusions the result of differing choices in: 1) research methodology; 2) experimental design; 3) measures of effectiveness (MOEs); and interpretation of results? Was resolution achieved through dialogue and debate among researchers, by an independent third party review, or some other mechanism? What is the venue and medium for accomplishing this resolution? Document practices that have been used for resolution.

Task 3: Determine extent of peer review of research proposals

Survey transportation research organizations (those that sponsor research) to determine whether proposed research methodology, experimental design, and measures of effectiveness (MOEs) are routinely peer reviewed prior to conducting research. In TRB’s Cooperative Research Programs these aspects are – in essence – peer reviewed by project panels when reviewing competing research proposals. But in many other transportation research venues a peer review – by independent and well-qualified third parties – is not part of the process. Determine the extent of peer review across a wide range of organizations that sponsor transportation research and a wide range of agencies. Document existing practices.

Task 4: Determine the character of review of research results by research sponsors

When sponsors of research (such as state departments of transportation, or federal agencies) review draft research reports, those reviews tend to be conducted by the employees of the sponsoring organization who have a vested interest in the research results and recommendations. In TRB’s Cooperative Research Programs draft reports are reviewed by project panels composed of individuals knowledgeable in the specialty area. Survey transportation research organizations (those that sponsor research) to determine the typical composition of teams that review draft research reports. To what extent are reviewers knowledgeable of the specialty area? To what extent are reviewers from inside versus outside the sponsoring organization? Document existing practices.

Task 5: Survey transportation research organizations--both sponsors of research and performers of research--to determine whether replication of research is valued

In many other research fields – medicine is one example - results are not considered valid or trusted until they are replicated and validated by multiple research projects. The recent observation that transportation research projects sometimes generate divergent results has brought attention to the value of replicating results. Replication is a means to confirm the soundness of research findings and can be one way to help resolve divergent results. Thus, there is a high interest in synthesizing state DOT – and other research agency – outlooks on the value of replication. Does the transportation research field value replication of research to confirm research findings? Survey transportation research organizations and document existing practices.

Task 6: Quantify rates of dialogue and debate about research finding in multiple fields as measured by "letters to the editor" or the equivalent

Preliminary findings show that different fields of medicine, science and engineering have widely varying rates of “letters to the editor” commenting on published research. As examples: the New England Journal of Medicine publishes about one letter to the editor for each article published; the journal Science has one published comment for every 13 papers published; and Transportation Research Record has one published Discussion for every 700 papers published. Gather data from additional highly regarded journals in science, medicine, and engineering to expand the preliminary findings.

Task 7: Identify individuals in the fields of medicine and science who could offer observations about research culture in those fields and interview them

The preliminary findings cited in Task 6, as well as other evidence, suggests that the culture in other research fields (science, medicine, other fields of engineering) is very different from transportation research. Different fields of research are very different in the extent to which they vigorously debate divergent research results and competing scientific theories. In the medical community researchers are much more inclined to openly engage in review, critique, and debate in questioning the findings of one another’s research – including published research that has already been peer reviewed. Astronomers debate the meaning of research findings, such as the age of the universe. Geologists debate competing theories on how and when the Grand Canyon was formed. Medicine and other fields appear to place greater value on replication of research. To gain a better understanding of research culture in other fields, interview researchers from those fields and gather their insights. Determine what practices are used in other research communities to critically review and debate research (beyond the initial peer review and publication of papers).

Sponsoring Committee:ABG10, Conduct of Research
Date Posted:04/27/2018
Date Modified:05/03/2019
Index Terms:Research, Research management, Research projects, Researchers, Business practices, Culture (Social sciences),
Cosponsoring Committees:AHB50, Traffic Control Devices; ANB25, Highway Safety Performance; AHB40, Highway Capacity and Quality of Service; AND10, Vehicle User Characteristics
 
Subjects    
Administration and Management
Research
Transportation (General)

Please click here if you wish to share information or are aware of any research underway that addresses issues in this research needs statement. The information may be helpful to the sponsoring committee in keeping the statement up-to-date.