CAPP Component 2 Evaluation

Stay Tuned!

Component 2 survey documents are being developed to reflect new Component 2 programming. Several document links below are simply placeholders — the documents will be available soon.

The goal of Component 2 programming is to provide youth-led, multi-dimensional (educational, social, vocational, economic, and recreational) opportunities for adolescents to provide alternatives to sexual activity and to develop skills that can support a successful transition into healthy young adulthood. As with Component 1 programming, how do we know that what we are doing for Component 2 programming is working? Unless we actually measure change within individual participants, we can only make educated guesses. Surveys that ask the same questions of students before and after the program content is delivered can give some indication of program success.

Component 2 programs will be evaluated using pre-post surveys. These surveys will ask questions about competencies your programming aims to build, such as self-efficacy or healthy decision making. By asking these questions before the program begins and again after the program ends, we hope to identify positive gains in these core outcomes for youth participants.

Note

Permission must be secured from each program site before the surveys can be administered, and permission from a parent or guardian (via a parent information/opt-out form) may also be necessary. Surveys will be administered using a URL or QR code on a device that is connected to the internet, and a process that guarantees anonymity.

Surveys are currently being developed to reflect upcoming Component 2 programming and will be available soon.

Selecting a Program for Evaluation

It's important to note that not all Component 2 programming needs to be evaluated. Each agency will select a program for evaluation that fulfills the following criteria:

• A group of youth attends programming for a minimum of two months (8-12 sessions). Shorter programs are excluded because a program of longer duration is more likely to achieve Component 2 outcomes than a shorter, more concentrated program. For example, a one-week program that meets five hours a day is likely to be less effective than a program that meets once a week for two hours over several months.
• Programming is focused on skill building.  
• The same program leader is present for all program sessions. A supportive, stable relationship between youth and an adult program leader is a critical feature of effective youth development programming.

Administering Surveys: An Overview

Administering the surveys takes planning and time — time that may already be tight in any given program cycle. Preparation is key.

Survey Links

Educators administer surveys to participants through URLs and/or QR codes on a device that is connected to the internet. Here are some points to consider when preparing to administer surveys:

• Do participants have internet access?
• Do participants have cell phones?
• Do program facilitators have an electronic method to contact participants in order to send links such as email, social media messaging, or texting?
• Tablets previously purchased for survey administration can be used to access survey links as long as they are connected to the internet.
  • Mobile hotspots are an option to help with internet access &mdash please contact your DOH program advisor for more information.

Health Educator Supervisors can request the URLs and/or QR codes from their ACT for Youth evaluation team contact. Their ACT for Youth evaluation team contact will send separate URLs and/or QR codes for each specific combination of survey type (Pre/Post), age group (high school/middle school), setting (NYC DOE/Non-NYC DOE), and language (English/Spanish). Each URL and/or QR code can be used as many times as necessary for that particular survey type, age group, setting, and language combination.

Informed Assent

It is important that youth program participants know why you are asking them to complete this survey. Be sure to verbally state this information and check to see if anyone has questions. Here is a sample script for informed assent:

Sample Script

"We want to learn more about how well this program works, and that means we need to ask you, the experts, about how your thoughts and opinions may have changed from the beginning to the end of this program. I'll ask you to take a short survey now and again on the last day of the program. We want to learn how your thoughts and behaviors may be different at the end of the program. Both surveys will take just a few minutes. Both surveys are anonymous. That means your name does not go on the surveys and we will never be able to link your answers to you. I will not know which answers are yours, and your teacher/program leader and parents will never see your responses. Your answers will be combined with those from other young people. Both surveys are voluntary. While we hope you will finish both of them, you do not have to participate. You can skip any question you do not want to answer. The information you tell us will be used to help make this program better in the future."

Obtaining Permission

In order to conduct the surveys, you will need permission from the organization that is hosting the EBP, and you may also need to allow parents to opt their children out of the survey.

When introducing the survey, be sure to let administrators know that this survey and protocol have gone through rigorous review processes by Cornell University's Institutional Review Board and the New York City Department of Education (NYC DOE) Institutional Review Board. These boards have high standards for protecting the privacy, rights, and welfare of participants in research and evaluation projects, and they have approved this survey and protocol.

You may wish to emphasize:

• CAPP is a statewide initiative; survey responses from your sites will be combined with those from around the state as part of the evaluation of the CAPP initiative.
• Surveys are anonymous. Names are never collected with surveys. Program facilitators will not see individual survey responses, only aggregate data.
• Individual participation is optional. Parents and youth participants can decide NOT to take the surveys. Youth participants may also skip any question on the surveys.

The process for obtaining permission differs depending on whether the cycle is being implemented in or outside of New York City public schools.

In sites OUTSIDE New York City public schools

1. Discuss the use of surveys with the site administrator or principal. ACT for Youth has developed a template you can use to initiate this conversation or to draw talking points from. The letter may be sent with the parent information/opt-out form below. Feel free to modify the template to fit your needs. Use of the template is optional.

   CAPP Component 2 survey letter to administrator

   CAPP Component 2 survey letter to principal
2. Send documentation of the site administrator's or principal's permission to use the survey to your ACT for Youth evaluation contact. This can be as simple as a forwarded email exchange. This is required.
3. If desired by the site, send the parent information/opt-out form home prior to survey administration. If a signed form is returned, be sure the youth participant returning the form does NOT complete surveys. Note: a youth participant without permission to complete surveys may still participate in EBP programming, if permission to attend the EBP was granted.

In sites WITHIN New York City public schools

1. Discuss the use of surveys with the principal. ACT for Youth has developed a template you can use to initiate this conversation or to draw talking points from. Feel free to modify this letter to fit your needs. Use of the template is optional.

   CAPP Component 2 survey letter to NYC principal
2. Have the principal complete the approval to conduct research form, making sure that the school ATS code is included. Send the completed form to your ACT for Youth evaluation contact. This is required.
3. Send the parent information/opt-out form home at least one week prior to survey administration. This is required. If a signed form is returned, be sure the student returning the form does NOT complete surveys. Note: a student without permission to complete surveys may still participate in EBP programming, if permission to attend the EBP was granted.

NYC Schools Fingerprinting and Security Clearance

In New York City public schools, the NYC DOE requires anyone who distributes survey links to students to go through the fingerprinting and security clearance process for NYC schools and to be added to the IRB protocol. We provide detailed information on the process:

Implementing CAPP Programs in NYC Schools: Fingerprinting and Security Clearance FAQ

If you need a copy of the Cornell University or NYC DOE Internal Review Board letters of exemption/approval, please contact Paula Aiken (psa2@cornell.edu).

Data, Reports, and Feedback

Once an organization has reached a threshold of 50 completed pre-and post-surveys, ACT for Youth can create a report from the results. For questions about survey reports, or the number of surveys completed at your organization, please reach out to your evaluation team contact.