What to Expect for Independent Evaluation Team Site Visits
By ICF
Feb 28, 2022
Introduction
An independent evaluation, or IE, is required for demonstration projects receiving funding from the Federal Transit Administration’s Public Transportation Innovation program. In addition to satisfying this requirement, an independent evaluation also supports FTA’s research program goals and advances the state of the transit industry by documenting the successes and impacts of demonstration projects and the national impact of the Integrated Mobility Innovation (IMI) and Accelerating Innovative Mobility (AIM) grants.
ICF is leading the independent evaluation of the IMI and AIM grants with support from the UC Berkeley Transportation Sustainability Research Center (TSRC). This team is experienced in conducting independent evaluations of technology-enabled mobility demonstration programs such as FTA’s Mobility on Demand (MOD) Sandbox.
The IE team will coordinate with the project team and FTA to determine an appropriate method for evaluating each demonstration project during the evaluation planning phase. Some evaluation plans may require site visits by the IE team to conduct expert interviews, group discussions with users of the service or other project stakeholders, and/or ethnographic research methods such as ride-along interviews. During expert interviews, the IE team will conduct interviews with principal stakeholders (referred to as experts) to capture key insights, lessons learned, and challenges gathered throughout the duration of the project. The goal of group discussions is to capture experiences, insights, and recommendations for future work from those who experience the outcome of the demonstration project first-hand. Where applicable, ethnographic interviews will intercept users in the field as they are interacting with the demonstration project and will be used to develop user vignettes for the IE report.
The following section describes these site visit activities and related logistics in further detail.
Site Visits
Where site visits are indicated by the evaluation plan, the IE team will coordinate with the grantee in advance of the site visit to ensure adequate time is allotted to secure a venue, recruit participants, and avoid scheduling conflicts to the extent possible. The site visit will consist of multiple steps, including outreach, selecting a venue, a pre-screening process, scheduling and holding expert interviews and group discussions, and ethnographic interviews if applicable. All human subjects research conducted as part of the independent evaluation process will be governed by an approved Institutional Review Board (IRB) Human Research Protection Program (HRPP) protocol, and consent will be obtained before engaging with human subjects.
For outreach, IMI/AIM grantees and the IE team should coordinate to determine what support is needed to secure participants for the group discussions and expert interviews. The IE team and the IMI/AIM grantee will also coordinate in advance to schedule all evaluation activities.
The IE team will provide incentives to users of the system for participating in a group discussion or other independent evaluation activities such as ethnographic interviews. Details regarding user incentive types, values, and media will be specified in the site’s evaluation plan. Additionally, the recipient agency can support the independent evaluation process by providing no cost rides to and from the group discussion venue for participants who have identified such a need during the pre-screening process. Agencies who wish to provide supplementary incentives to users of the system, beyond those provided by the IE team, should coordinate with the IE team to ensure such incentives avoid biasing participant responses.
Group Discussions
For group discussions, discussion invitees will also be asked to complete a pre-discussion questionnaire. The pre-discussion questionnaire will capture basic information about each participant and their experience using the service, including standard demographic characteristics, and information such as perceptions of the impacts of the service, frequency of use of the service, and types of trips made. The questionnaire may be administered remotely in advance of the group discussion by phone or through a web survey link. Alternatively, the questionnaire may also be administered on-site prior to the initiation of the group discussion. The results of the pre-discussion questionnaire will supplement findings from the group discussion in the evaluation report.
The IE team will coordinate in advance of the site visit to determine group discussion scheduling and logistics with the site. Typically, the best time to hold group discussions is shortly after typical work hours (e.g., starting between 5:15 – 6:15 PM) Tuesday through Thursday. Group discussions should also ideally be scheduled in 90-minute blocks. In some cases, these discussions will need to be broken up into separate events to ensure adherence to public health guidelines.
Group discussions may take the form of focus groups, which include at least eight participants in discussion at the same time and place. If there are fewer than eight attendees, or if sessions must be separated by time or location for public health or logistical reasons, the forum will be referred to as a small group discussion in recognition that such discussions are likely to produce less robust results. These discussions will be conducted in person, where possible. Qualitative findings from these discussions will be used in the evaluation report.
Expert Interviews
For expert interviews, the recipient agency will identify core project team members that can be available to interview with the IE team. Interviewees should have enough knowledge on the project to talk candidly and in detail about its successes and challenges. Grantees can find the expert interview protocol in their independent evaluation plan. This protocol is a semi-structured base guide for the interview; however, the interviewer may ask follow-up questions that are not listed in the protocol to further probe additional topics of interest that may arise.
Ethnographic Interviews
Ethnographic interviews will be planned for select deployments, as documented in each recipient’s evaluation plan. Where designated, these interviews will be conducted in the field and will often take the place of “ride-alongs,” where a member of the IE team will join a trip in progress and engage willing participants to capture additional experiences and insights. Participating individuals will be eligible to receive a $5 – $10 gift card to a national restaurant chain or retailer. These interviews will occur at the place of service during regular service hours. From start to finish, the ride-along interview process is anticipated to take between 4 – 8 hours, depending on individual site conditions. Individual ride-along interview lengths will be dictated by the duration of the journey and other factors. The IE team will coordinate with the site in advance of the interviews to ensure they are conducted in a safe and efficient manner. In the event a day scheduled for ethnographic interviews need to be rescheduled (e.g., due to weather, public health conditions, etc.), the IE team will coordinate with the site on a strategy for rescheduling or alternate approach.
Health & Safety Requirements and Covid-19 Precautions
For expert interviews and group discussions, the IE team will also request the grantee’s assistance in providing a suitable venue. For the venue to be considered suitable for the evaluation process it should meet University of California Berkeley Committee for Protection of Human Subjects (CPHS) requirements to ensure the health and safety of all participants.
- These requirements stipulate that focus groups and interviews should be conducted in an outside setting when possible.
- If an outside location is not possible, the venue should have exterior windows with access to outside air and/or be highly ventilated.
- An inside venue should also be large enough to maintain social distancing among participants.
Some of these requirements may go above and beyond minimum local, state, or federal health and safety protocols. This can include requiring masks, screening for potential risk factors and COVID symptoms, and requiring proof of vaccination to participate in a focus group setting. The IE team kindly requests that the site assist the evaluation team in emphasizing that all individuals involved must adhere to wearing a mask over their mouth and nose during these activities.
In addition to CPHS venue requirements, the IE team will also conduct a pre-screening process of participants as an additional health and safety measure. All individuals invited to participate in expert interviews or group discussions must correspond with the IE team to be appropriately screened prior to participating in-person. The pre-screening will take place up to 14 days before the event and again immediately prior to the event. The pre-screening process will ensure that the number of people attending in-person will adhere to social distancing guidelines in the chosen venue. Additionally, the IE team will screen participants for: (1) verification that participant has received a full course of vaccination for COVID-19, (2) symptoms, exposure, or positive COVID-19 test, and (3) compliance with wearing a mask for the duration of the evaluation process. Invitations to participate will include the IE team’s contact information but not the location of the venue. Once an individual has passed the pre-screening process, has been determined to meet the target participant criteria, and has indicated suitable availability, they will be notified of the venue location. This process will ensure that no one will arrive in-person that has not been pre-screened. The above public health precautions are subject to change based on updated CPHS requirements.
Conclusion
To date, the IE team has completed two site visits and plans to resume conducting site visits in spring of 2022. We look forward to conducting site visits to your region and discussing the impacts of your agency’s demonstration project with stakeholders and users. For agencies who have received a completed evaluation plan, the tentative site visit date will be indicated in the included schedule for the independent evaluation. If your agency has not yet begun the evaluation planning process with the IE team, we will be in touch in the first or second quarter of 2022 to discuss site visit logistics and other evaluation plan details.
For questions related to the IE site visit or other IE related questions, please feel free to reach out to the following IE team contacts:
- Les Brown, Project Manager, les.brown@icf.com
- Mike McQueen, Lead Evaluation Coordinator, mike.mcqueen@icf.com