CAPP Component Two

The CAPP initiative strives to enhance adolescents' social and emotional development, as well as promote environments that support young people's health and safety. With this focus on broader youth development outcomes, some CAPP agencies received Component Two funding to:

• implement multi-dimensional educational, vocational, economic, and recreational opportunities for youth on multiple health and development topics that introduce them to new situations, ideas, and people, and challenge them to build or learn skills; and/or
   
• implement mechanisms to refer individuals to other federal, state, county, city, school district, and local community service providers for physical, social, emotional, educational, and developmental support and services as necessary.

Agencies that received Component Two funding proposed a wide range of programming, including youth leadership programs, peer education, social activism and social media production, and training institutes focused on work readiness, life skills, and relationship skills. Programs varied from long-term involvement of one year to short, intensive summer programs.

Best Practices for Youth Development Programming

Effective youth development programs build competencies in many areas, increase self-efficacy, expand opportunities and recognition for youth, and increase healthy bonding with others. Generally, research has established that effective interventions last nine months or longer. These findings are reflected in the features of effective youth development settings, which currently serve as a guideline for youth programs.

To help CAPP agencies design effective Component Two programs, Jutta Dotterweich presented findings from youth development research in a learning community meeting in 2017:

CAPP Component Two: Best Practice

Webinar Recording (4/17/17)

Outcome Measures

Working together, CAPP agencies and ACT for Youth developed logic models to narrow down the outcomes we are trying to achieve, which enabled us to identify the core outcomes we will measure: self-efficacy, healthy decision making, and youth-adult connectedness. These three measures appear to be relevant to Component Two proposals across sites and will be used to evaluate efforts.

CAPP Component Two: Measuring Outcomes

Webinar Recording (6/16/17)

Introducing Outcome Measures for Component Two

Webinar Recording (11/16/17)

Evaluation

Component Two programs that meet certain criteria will be evaluated using pre-post surveys. CAPP Component Two youth surveys ask questions about self-efficacy, healthy decision making, and youth-adult connectedness. By asking these questions before the program begins and again after the program ends, we hope to identify positive gains in these core outcomes for youth participants.

Component Two Survey Questions (sample only)

Permission must be secured from each program site before the surveys can be administered, and passive consent from a parent or guardian may also be necessary (as described below). Surveys will be administered using a URL or QR code on a device that is connected to the internet, and a process that guarantees anonymity.

Selecting a Program for Evaluation

It's important to note that not all Component Two programming needs to be evaluated. Each agency will select a program for evaluation that fulfills the following criteria:

• A group of youth attends programming for a minimum of two months (8-12 sessions). Shorter programs are excluded because a program of longer duration is more likely to achieve Component Two outcomes than a shorter, more concentrated program. For example, a one-week program that meets five hours a day is likely to be less effective than a program that meets once a week for two hours over several months.
   
• Programming is focused on skill building.
   
• The same program leader is present for all program sessions. A supportive, stable relationship between youth and an adult program leader is a critical feature of effective youth development programming.

At a Component Two learning community meeting, we discussed the selection process.

C2: Selecting a Program for Evaluation

Webinar Recording (5/18/18)

Securing Permission

Usually, if a site administrator has granted permission for you to implement a program, he or she is likely to be interested in the results of the survey. Before you begin, be sure to secure permission in writing from the school principal, program director, or other authority.

For programs in sites OUTSIDE New York City public schools:

1. Discuss the use of surveys with the site administrator or principal. ACT for Youth has developed a template you can use to initiate this conversation or to draw talking points from. The letter may be sent with the parent information/opt-out form below. Feel free to modify the template to fit your needs.

   Template for letter to principal (Word)

   Template for letter to site administrator (Word)
    

2. Send documentation of the site administrator's or principal's permission to use the survey to your ACT for Youth evaluation contact. This can be as simple as a forwarded email exchange. This is required.
    
3. If desired by the site, send the parent information/opt-out form (PDF) home prior to survey administration. This is a "passive consent" process: parents will need to return a form only if they are opting their children out of the surveys. If a signed form is returned, be sure the youth participant returning the form does NOT complete surveys. Note: a youth participant without permission to complete surveys may still participate in programming.

For programs WITHIN New York City public schools:

In New York City public schools, the New York City Department of Education (NYC DOE) requires additional steps.

1. Ensure all educators who will administer the surveys have been fingerprinted by the NYC DOE. (Fingerprinting by other organizations is not accepted.)

   Any educators who have not yet been fingerprinted by the NYC DOE will need to submit copies or photos of both a signed social security card and driver's license to the Institutional Review Board (IRB) Personnel Eligibility Tracking System (PETS) via email (PETSAdminSupport@schools.nyc.gov). The PETS team will enter the educator's information into the IRB system and the educator will then receive a nomination email within 24 hours. All instructions will be in the nomination email. If you have any questions, please contact Vicki Baum.
    

2. Send documentation of fingerprinting to the NYC DOE Internal Review Board. This is required.
    
3. Discuss the use of surveys with the principal. ACT for Youth has developed a template you can use to initiate this conversation or to draw talking points from. Feel free to modify this document to fit your needs. Use of this document is optional.

   Template for letter to NYC principal regarding Component Two surveys (Word)
    

4. Once approved, have the principal complete the Approval to Conduct Research Form (making sure that the school ATS code is included) and send it to your ACT for Youth evaluation contact. This is required.

   Approval to Conduct Research form (Word)
    

5. Send the Component Two survey parent information/opt-out form home prior to survey administration. This is required. If a signed form is returned, be sure the student returning the form does NOT complete surveys. Note: a student without permission to complete surveys may still participate in programming.

   Parental Information and Permission Form

If you need a copy of the Cornell University letter of exemption/approval, please contact Amanda Purington (ald17@cornell.edu).

Administering the Surveys

Administering the surveys takes planning and time -- time that may already be tight in any given program. Preparation is key.

Educators administer the surveys to participants through a URL or QR code on a device that is connected to the internet. Here are some points to consider when preparing to administer the surveys:

• Do participants have internet access?
   
• Do participants have cell phones?
   
• Do program facilitators have an electronic method to contact participants in order to send the link, such as email, social media messaging, or texting?
   
• Tablets previously purchased for survey administration can be used to access the survey link as long as they are connected to the internet.
   
  • Mobile hotspots are an option to help with internet access. Please contact your DOH program advisor for more information.

Health Educator Supervisors can request the URL/QR code from their ACT for Youth evaluation team contact. The URL or QR code can be used as many times as necessary.

Informed Assent

It is important that youth program participants know why you are asking them to complete this survey. Be sure to explain the following information verbally and check to see if anyone has questions. Here is a sample script for informed assent:

"We want to learn more about how well this program works, and that means we need to ask you, the experts, about how your thoughts and opinions may have changed from the beginning to the end of this program. I'll ask you to take a short survey now and again on the last day of the program. We want to learn how your thoughts and behaviors may be different at the end of the program. The survey will take just a few minutes. The survey is anonymous. That means your name does not go on the survey and we will never be able to link your answers to you. I will not know which answers are yours, and your teacher/program leader and parents will never see your responses. Your answers will be combined with those from other young people. The survey is voluntary. While we hope you will finish it, you do not have to participate. You can skip any question you do not want to answer. The information you tell us will be used to help make this program better in the future."

Data, Reports, and Feedback

Once your organization has gathered completed pre- and post-surveys from at least 50 participants, you can request a summary report from ACT for Youth. For questions about survey reports, or the number of surveys completed at your organization, please reach out to your evaluation team contact.