Afterschool Universe

Description

Afterschool Universe is a hands-on astronomy program targeted at middle school children in out-of-school-time settings. It explores basic astronomy concepts through engaging hands-on activities, and focuses on the Universe outside the solar system. Afterschool Universe is available to out-of-school-time program providers who are looking to include engaging and high-quality science programming in their organization's portfolio, and is widely recognized within the informal education community as a high quality and effective STEM education curriculum. The overarching goals of the Afterschool Universe program are to increase participant comfort and engagement with STEM, and to increase awareness and understanding about concepts in astronomy. Additional goals and objectives have been set for specific efforts during the course of program development and dissemination.

Afterschool Universe consists of 12 sessions. Each session typically requires 45-60 minutes for implementation. The program is flexibly structured and can be run in a variety of settings that include astronomy days, summer camps, or year-long afterschool programming. Although many sessions build on previous ones, each session is intended to be free-standing, as not all children will attend every session in informal education programs. Activities for this program were developed by drawing from a wide array of existing resources and adapting them for the out-of-school-time environment. Responding to pilot feedback and best practices in informal education, the team developed a comprehensive manual for the program facilitators that provides background information and detailed recipes for running program sessions. The manual has been written at a level that is appropriate for facilitators with little science background. It has passed the rigorous NASA Product Review and is approved for wide-spread dissemination.

Afterschool Universe is an education/outreach program of the Astrophysics Science Division at NASA's Goddard Space Flight Center. It was originally sponsored by NASA's Beyond Einstein program (now called Physics of the Cosmos) and was developed in partnership with the Imagine the Universe! EPO program, in response to demand from the informal education community for quality astronomy out-of-school-time programming. Afterschool Universe was pilot tested under the name Beyond Einstein Explorers' Program (BEEP) in the summers of 2006 and 2007, at a number of sites in the Washington, DC, area. The summer 2007 training and pilot site implementation was formally assessed by external evaluators at Magnolia Consulting. This evaluation informed further refinement of the curriculum and professional development as the program moved into nationwide dissemination. 2008-2009 marked a big push to get Afterschool Universe recognized and disseminated. Over those two years, the Goddard team ran 11 trainings around the country, and our first two certified trainers ran 11 more. During this time, Afterschool Universe was added to the Consumers Guide to Afterschool Science Resources and the approved curriculum list for Great Science for Girls, featured on the NASA portal, and the Afterschool Universe team won a NASA Honor Award for Public Service Group Achievement.

In 2010, the team changed focus to developing an infrastructure of certified Afterschool Universe trainers around the country. Funded by a NASA EPOESS 3-year grant, they ran a special train-the-trainer workshop at NASA Goddard, and trained 30 participants from competitively-selected informal education organizations and networks around the country. The goal of this project was to build strategic partnerships with well-established and diverse networks of afterschool programs across the country, and build the capacity of these networks to integrate Afterschool Universe into their portfolio so that they could offer training and implementation within their networks for years to come. Since the train-the-trainer workshop, the attendees have been running their own training workshops around the country, increasing the reach of the Afterschool Universe program. This model was evaluated externally at all three levels by Cornerstone Evaluation Associates – the initial NASA-delivered training workshop, the informal educators trained by partners, and the youth reached by subsequent implementation of the curriculum – to assess the impact and integrity of the train-the-trainer model for professional development.

The latest phase of the Afterschool Universe project will take place during FY2014 with the development of an elementary school adaptation of the program curriculum. There has been significant demand for astronomy out-of-school-time curricula for this age group, and implementers frequently ask if the existing program can be offered for younger audiences. This adaptation will be funded by the Astro-H mission and conducted in partnership with pre-service educators at Fort Hays State University in Kansas, and evaluated in partnership with Cornerstone Evaluation Associates.

Lead Organizations

NASA Goddard Space Flight Center

Partnerships

During the development of the Afterschool Universe (AU) program, the AU team worked with the Children and Youth Investment Trust Corporation of Washington, DC, to pilot test the sessions at various informal education sites in DC and inform refinement of the curriculum and professional development delivery. Throughout the dissemination of the program, the team has worked with a variety of organizations and networks to extend the program’s reach. The full list of partners would number into the hundreds, but our largest, farthest-reaching, and longest-lasting partners include: Afterschool Alliance, Great Science for Girls, Explore! Program at the Lunar and Planetary Institute, Girl Scouts of Central Maryland, Challenger Center for Space Science Education, Center for Radiophysics and Space Research at Cornell University, Build the Out-of-School Time Network, Central Texas Afterschool Network, Coordinated Child Care of Pinellas, Dallas AfterSchool Network, Girl Scouts of the Northwestern Great Lakes, Illinois AfterSchool Network, Maryland Department of Juvenile Services, Massachusetts Afterschool Partnership, OregonASK, The After-School Corporation, THINK Together, Tri-State Education & Technology Foundation, and the Missouri AfterSchool Network. Many of these partners participated in a strategic train-the-trainer effort in 2010, sending their trainers to NASA Goddard to learn about the program’s curriculum, philosophy, and pedagogy, returning to their organizations to train others locally and implement the program with youth. In addition, the AU team is currently partnering with the Astro-H mission and Fort Hays State University in Kansas to adapt the program during FY2014 for the elementary school aged audience, which will broaden the program’s reach substantially due to the demand for quality STEM afterschool content for that age group. Throughout the program’s development and dissemination, the AU team has worked with external evaluators (Magnolia Consulting and Cornerstone Evaluation Associates) to assess the impact and integrity of the program.

Metrics

Through local, regional, national, and partner-run trainings, nearly 1,000 program implementers have participated in single and multi-day training institutes that familiarize them with the curriculum as well as increase their familiarity and comfort with teaching basic astronomy concepts. Approximately half of these implementers have been trained by a network of 58 certified trainers working independently and within partner organizations around the country to more efficiently disseminate the curriculum, increase program reach, and meet increasing demand for program professional development.

Effectiveness and Impact

Evaluation Findings

Each phase of the Afterschool Universe program effort has been evaluated, and the findings of those assessments have been used to refine the program and inform future direction of efforts.

In pilot test evaluations in 2006 and 2007, program leaders reported that the training led to increases in both their knowledge and confidence in leading the sessions. Surveys conducted with the leaders one month after training, when they had begun to implement the program, show that even those leaders with no previous experience teaching astronomy learned and retained a very large percentage of the information presented in training. Other key findings included:

Overall perceptions of the BEEP program were very positive. The majority of leaders indicated they would be happy to lead the program again.
Leaders found the BEEP manual to be clearly organized, detailed and easy to follow. Background material was especially helpful in building a knowledge base for both leaders and students.
Leaders found the materials needed for the sessions easy to obtain. However, several leaders mentioned that it is extremely time consuming to gather all of the materials needed for the individual sessions.
Leaders felt that the BEEP program met their expectations including exposing students to science and astronomy, fostering interest for these subjects in their students, and teaching their students about basic astronomy concepts.
Several leaders mentioned how much they had learned as a result of leading the BEEP sessions. Since the majority of leaders had little science background, this is an important finding.
BEEP activities that were most successful were those that allowed for hands-on exploration and the chance for students to work in groups, such as the creation of models of the Universe.
Students who participated in BEEP activities showed an increase in understanding of astronomical concepts and the tools used by astronomers to study the Universe.
Student interest at the start of the program was positive and did not change over the course of participation in BEEP activities.
Students particularly liked activities that allowed them to create, explore and “experiment.”

The information obtained through these pilot evaluations drove a number of program refinements, leading to a high level of confidence in the quality of the program and in the training currently offered to front-line implementers by our group.

The final evaluation report prepared by Magnolia Consulting is available below along with these metrics and impacts (BEEP_Evaluation.pdf).

The multi-level evaluation of the train-the-trainer dissemination model addressed questions about the program’s accountability as well as its outcomes. Both quantitative and qualitative data were collected over a 1 1/2 year period (2010-2012) with the evaluative lens focused on all three tiers of program participation— trainers (Tier 1), front-line implementers (Tier 2), and students (Tier 3). The data unreservedly support the success of AU’s model revealing that...

AU content knowledge was successfully conveyed across all three tiers of participants and retained by trainers and implementers
Assessment of knowledge gains ranged from an average of 26% for implementers to 32% for students, with trainers’ assessment of knowledge showing a 29% gain from pre- to post-assessment
Feelings of comfort, confidence and preparedness in using the AU curriculum were imparted by the AU leaders to Tier 1 trainers and likewise by the trainers to Tier 2 front-line implementers

The train-the-trainer model has clearly demonstrated its efficacy in delivering content knowledge throughout all three tiers of the model. Furthermore, the evidence indicates that the ‘expert’ leaders can trust trainers not only to maintain high quality in transferring AU content knowledge to implementers, but also to instill in the implementers the same levels of comfort, confidence and preparedness that the ‘experts’ imparted to them.

The final evaluation report prepared by Cornerstone Evaluation Associates has been uploaded along with these metrics and impacts (AU_Evaluation.pdf).

Evaluation of the elementary school adaptation of Afterschool Universe is projected to begin in FY2014.

Impact Statements

Evaluation during the development and initial dissemination of the Afterschool Universe program, as well as of the train-the-trainer model of dissemination, sought to assess the impacts on participant knowledge and attitudes. Evaluation findings showed positive outcomes for each of these impacts.

In an effort to assess changes in students’ attitudes toward science as a result of participating in the Afterschool Universe pilot program in 2007, students were given a survey in the first and last week of program participation. Students were asked to respond to 15 items on a 4-point scale (1= really don’t agree, 2 =don’t agree, 3= agree, 4= really agree). An overall score was calculated for each student. Scores had a possible range of 15 (if a student rated all items 1) to 60 (if a student rated all items 4). There was no change in mean scores from the first to the second administration of the attitude survey. Students began with a somewhat positive attitude toward science and astronomy, and ended with virtually the same measured attitude.

A second way to view the student attitude results was to examine the percentage of positive responses to the survey items. This was determined by calculating the frequency of positive responses (either agree or really agree) for all items combined. In this analysis, a value of 50% would indicate that students were neutral in their attitudes toward science. Examining the student data from this perspective again indicates that students began with a fairly positive attitude and ended with relatively no change, but maintained the positive attitude throughout the program.

Eight additional items were included in the post survey in an effort to gauge changes in students’ perceptions about their knowledge about astronomy. Students were asked to rate their level of agreement with statements concerning how much they learned about various aspects of Astronomy through participating in the sessions. These items were rated on a 4-point scale (1= really don’t agree, 2 =don’t agree, 3= agree, 4= really agree). All statements elicited a positive response (agree or really agree) from a high percentage of the students, indicating that they perceived their learning over the course of the program to be significant. In addition, the statement “I know a lot about how scientists study stars and planets” was included in both the pre and post surveys. Positive responses to this item increased significantly (from 48% to 70%) as a result of participation in the pilot program.

The train-the-trainer model performed pre- and post-assessment of participants at all three tiers of the effort – trainers (Tier 1), front-line implementers (Tier 2), and students (Tier 3). Of utmost concern was the effectiveness of the model in ensuring the conveyance of AU content knowledge throughout its three tiers. Growth in knowledge was measured in two ways—1) the trainers’ and implementers’ perceptions of their own knowledge and 2) the pre-post knowledge assessments/tests given to participants at all three tiers.

In order to gather information about the trainers’ and implementers’ perceptions of knowledge, each group was asked to rate statements using a 5-point scale with ‘1-strongly disagree’ through ‘5-strongly agree’. Ratings were collected from trainers and implementers alike at the end of their workshops and in online follow-up surveys. The mean ratings for Tier 1 and Tier 2 reveal an average initial gain (from prior to the workshop to its end) of 1.7 and 1.5, respectively. Both tiers continued to retain their gains in knowledge following their workshop experiences.

Additional evidence supplied by the pre-posttests gives weight to the assertion that knowledge was successfully conveyed throughout all three tiers. A critical piece of the evaluation was the series of questions (multiple choice, fill-in-the-blank, ranking, and open ended) designed to assess the AU-related content knowledge of trainers and implementers both before and after their workshops. A separate assessment was developed for students. In both cases, questions were designed to reflect the content of the Afterschool Universe curriculum and ‘test’ common misconceptions about AU content that the program seeks to dispel.

AU program participants in each tier demonstrated a positive change in average percentage of correct answers ranging from 26% to 32%. These positive gains confirm the perceptions of trainers and implementers that they not only gained a good deal of AU-related content knowledge during their workshops, but also retained this knowledge over the subsequent year. For those implementers who taught the curriculum, their students also achieved a positive gain in knowledge. The assertion can, therefore, be made that the model allowed for knowledge to be successfully transferred within the ‘train-the-trainer’ model.

Employing the ‘train-the-trainer’ model, the AU program also sought to ensure that implementers—ultimately bearing the responsibility for delivering the program to their network youth—were not only knowledgeable, but also felt comfortable with the AU curriculum and had confidence in their ability to present it. Findings show that the trainers emerged from their workshop exhibiting high levels of comfort and confidence. More importantly, they were then able to instill the same high levels of these qualities in the implementers for whom they conducted workshops.

At the end of their workshop at NASA Goddard, trainers rated statements about their ‘confidence in understanding AU content’ and in their ‘ability to present it’. Trainers’ mean ratings for these questions on ‘confidence in understanding AU content’ was a 3.6 on a 4-point scale, while their ‘confidence in presenting AU content’ was a 4.0 on a 5-point scale. Clearly, trainers left their workshop feeling comfortable and confident about embarking on their journeys to conduct their own workshops.

Rating scales were used again to measure comfort and confidence in the follow-up surveys for both trainers and implementers. That is, once trainers had had an opportunity to conduct workshops and implementers had had time to instruct students in their afterschool networks, they were asked how their respective workshops and instructional sessions had gone, how their workshops/instruction were received, how comfortable they felt conveying AU content and if they saw evidence that their respective audiences would be able to use what they had taught them. These data were remarkable in their consistency between trainers and implementers—both rating their comfort and confidence at the same high level, trainers at 4.2 and implementers at 4.3 (both on a 5-point scale).

Trainers and implementers were also asked to rate a series of questions about their sense of preparedness for the tasks at hand—conducting workshops for the trainers and instructing youth for implementers. They were presented with the same questions both at the end of their respective trainings and in the follow- up surveys. These questions asked how sufficient they believed their training to be and whether they thought they had enough content knowledge and pedagogical knowledge to teach AU content.

The data show that both trainers and implementers have high levels of ‘a sense of preparedness’ once they are trained and that they maintain these high levels once they go out into the ‘real world’ to teach the AU program to their respective audiences. For trainers, a mean cluster rating of 4.4 after training stays steady after conducting workshops, while a mean cluster rating of 4.2 after training for the implementers bumps up slightly to 4.3 after instructing their students.

Other

The Afterschool Universe program has been one of our greatest success stories within Goddard’s Astrophysics E/PO – the curriculum and professional development have consistently been in high demand, the evaluation findings at each phase have been positive and highly informative to both the AU effort and other programs, and the program has achieved significant name recognition among informal educators. The program has demonstrated flexibility and we are interested in further exploring its adaptability (to a lower grade level audience) as well as expandability (to add additional sessions on other content, within astrophysics and other astronomy topics). We continue to explore innovative strategies and partnerships to supply professional development in response to demand, and to meet the ever-changing needs of the informal education community. We aim to build upon this effective and successful program with future efforts, to keep it fresh, accessible, and useful to our audiences.

Awards

NASA Honor Award for Public Service Group Achievement

In 2009, the Afterschool Universe team won a NASA Honor Award for Public Service Group Achievement “for dedication and innovation in bringing NASA into the lives of thousands of individuals who might not otherwise be engaged in space science.”