Project Evaluation

Building Capacity for Disciplinary Experts in Math and Science Teaching

Final Evaluation Report

August 2016

Estelle Raimondo, eraimondo@gwu.edu, Steffi Renninger, srenninger@gwu.edu, Cheryl Beil, cbeil@gwu.edu

1.    Background

• Problem Definition

The George Washington University and its partners (DCPS, CASE, and the Smithsonian National Museum of Natural History) identified the shortage of math and science teachers in the greater DC-Metro area as a serious challenge that has characterized the region for many years. The Noyce-funded program “Building Capacity for Disciplinary Experts in Math and Science Teaching” was thus designed to establish various pathways to assure continuous supply of math and science educators.

• Program objectives

The program has the following three overarching objectives:

• Increase the number of DC-area STEM professionals and undergraduate STEM majors seeking comprehensive pedagogical preparation for careers in math and science teaching;
• Increase alignment among teacher preparation coursework and field experiences, and DCPS standards for science and math education; and
• Form a nexus of collaboration between GW STEM and education faculty, local educational organizations, and DCPS centered on the preparation and mentoring of secondary math and science teachers in the DC-metro region’s high-needs schools.
  • Program Theory of Change

Based on program documents and a discussion with the PI(s), a program theory of change was reconstructed as the basis for program activities. Figure 1 lays out the logical framework that forms the basis for the evaluation.

2.    Evaluation Scope & Methodology

The evaluation of the program plays both a formative and a summative function. Through periodic meetings with the project teams, the evaluators’ analysis and findings are feeding into strategic decisions made within the program cycle. This intermediary report is thus primarily meant to inform the program’s implementation during its last phase. A more summative report will be submitted at the end of the program’s implementation with an assessment of the program’s effectiveness in meeting its intended outcome.

Figure 1: Program Logical Framework

• Evaluation Questions

This evaluation addresses the three following questions:

1. To what extent has the Noyce Capacity Building work contributed to increasing the number of STEM professionals and undergraduate STEM majors, especially those from underrepresented groups, recruited into teaching and considering a career in DCPS?
2. How dynamic is the nexus of collaboration between GW faculty, DCPS and non-profit organizations? To what extent has this network been instrumental in enhancing the quality and alignment of the math and science teacher preparation at GW?
3. As a result of the Noyce Capacity Building work are graduating math and science teachers well prepared to engage in math and science teaching aligned with in Common Core State Standards in Mathematics (CCSSM) and Next Generation Science Standards (NGSS)?

• Evaluation Timeline

Months	Tasks	Deliverables
September 2014	Preparation	Evaluation plan presented at Advisory Panel I
October 2014- January 2015	Data collection
March 2015	Analyze data collected	Intermediary report
April 2015-May 2015	Further data collection
June 2015	Data analysis and formulation of preliminary findings and recommendations	Draft Final evaluation report for presentation at Advisory Panel II
August 2015		Final evaluation report submitted
June-July 2016	Data collection for no-cost extension
August 2016	Data analysis and summary of findings and recommendations	Final evaluation report submitted

 

• Evaluation Methods

The evaluation was conducted by GW’s Office of Survey Research and Analysis in close collaboration with the project team. The findings of the evaluation are corroborated by multiple sources of data and information collected throughout the evaluation process as much as possible. The key building blocks of the evaluation are the following:

• A review of important documents, including Common Core State Standards in Mathematics (CCSSM) and the Next Generation Science Standards (NGSS)
• Longitudinal data analysis of applications, admissions, matriculations, and graduations both for the M.Ed in secondary education (concentration in math and science) and for the undergraduate student populations with a STEM Major.
• Analysis of graduating senior survey data
• Face-to-face conversations with members of the project team and its supporting network
• Interviews with GW Career Center personnel
• Focus groups with students
• Participatory observation at two Advisory Panels
• Desk review of a number of activity reports
• Regular check-in with Principal Investigators in person or via email
• Analysis of teacher-candidates’ Q?rius field experience journal
• Analysis of teacher-candidates’ observation by instructors during field experience at Q?rius
• Analysis of pre and post interviews with teacher-candidates regarding their field experience at Q?rius
• Analysis of program impact on students graduating prior to NOYCE and post-NOYCE
• Analysis of research goals and dissemination

Table 1 sums up the data collection strategy for each evaluation question. The remainder of the report consists of the evaluative findings based on the framework laid out above. These findings, while being substantiated by multiple sources of evidence remain preliminary because little time has elapsed between the completion of certain activities (e.g., awareness raising and recruitment) and the evaluation. The full range of outcomes is likely to materialize later. The first section of the findings outlines the activities conducted and results gathered until August 15^th, 2015. The team applied for, and received a no cost extension for the award to complete some of the remaining activities. The findings from this extension are outlined in the second half of Section 3.

Table 1: Data collection strategy

Evaluation Questions	Indicators of Performance	Means of Verification
1a. To what extent is the recruitment strategy effectively tapping into the potential pools of STEM pre-teachers applicants?	Relevance and comprehensiveness of the recruitment strategy	Mapping of potential pool of applicants; Needs assessment through interviews (including with undergraduate and career service advisers) and focus groups with undergraduates; Use of relevant Institutional Research and Survey Research and Analysis data
1b. To what extent has the Noyce Capacity Building work contributed to increasing the number of STEM professionals and undergraduate STEM majors, especially those from underrepresented groups, recruited into teaching and considering DCPS?	Number of inquiries about the program (referring to outreach element)	Record of Inquiries (emails, phone-calls)
	Number of recruited students (referring to outreach or new curriculum elements)	Application and enrollment data (disaggregated by socio-demographic characteristics)
	Number of students attending recruitment events	Attendance records at teaching panels and other outreach events
2a. How dynamic is the nexus of collaboration between GW faculty, DCPS and non-profit organizations?	Satisfaction of network members with collaborative process	Observation of Advisory Panels and other meetings
		Interviews with PI/co-PI and advisory panels
2b.To what extent has this network been instrumental in enhancing the quality and alignment of the math and science teacher preparation at GW?	Effectiveness of the field experience at Q?rius	Analysis of teacher-candidates’ -      Q?rius field experience journals -      observation by instructors during field experience Analysis of pre and post interviews with teacher-candidates regarding their field experience at Q?rius
	Effectiveness of Methods coursework	Feedback on teaching, teaching practices, and engineering course development from students participating in Methods class
	Trace of collaboration in the curriculum and synergy in co-teaching	Video of co-taught class
3. As a result of the Noyce Capacity Building work, are graduating math and science teachers well prepared to engage in math and science teaching aligned with CCSSM and NGSS?	Readiness of pre-service teachers to start their field work	Evaluation of pre-service teachers using the Candidate Impact on Student Learning Assessment (CISLA)
	Satisfaction/self-assessment of students	Focus groups with students and/or online course evaluations
	Effectiveness of alumni database for STEM teachers into induction year	Data from alumni database

3.    Findings

• Recruitment strategy
• Progress to date

The program relied on a two-pronged approach to its recruitment and to creating pipelines for recruiting pre-service teachers. First, a range of program activities were dedicated to raising awareness among the GW undergraduates enrolled in a STEM major. The planned activities included:

• Organization of two teacher panels
• Outreach to academic advisors
• Outreach to student associations
• Outreach with students in the learning assistant (LA) program

Another set of activities was dedicated to raising awareness about the program outside of GW among consortium students as well as local professionals in a STEM-related job who would be interested in a career change. The planned activities included:

• Distribution of information material in various professional organizations and government agencies
• Online marketing material through social media

The program achieved more progress with the recruitment within GW compared to outside recruitment. One of the two planned teacher panels was held in November. Outreach material for GW academic advisors and student associations were prepared and deployed. Very limited progress on the outreach to the non-GW population has taken place so far.

The success of the recruitment strategy is assessed against two evaluation criteria: its relevance and effectiveness.

• Relevance of recruitment strategy

Most of the recruiting efforts were focused on raising awareness within the GW undergraduate population who were majoring in a STEM field. Yet, the evaluators found that at the beginning of the recruitment effort, very few students enrolled at GW were considering a career in secondary STEM education. Figure 2 depicts a reconstructed pipeline as it currently exists within GW, beginning with the number of undergraduate applicants who expressed an interest in STEM to the number of graduating students in STEM majors. The second portion of the pipeline focuses on the GSEHD M.Ed in secondary education with a specialization in STEM.

Figure 2: GW pipeline for STEM Secondary Education (2013-2014)[1]

 

The pool of potential GW undergraduate candidates

As displayed in Figure 3, the number of GW students graduating with a major in STEM has steadily increased over the past six years. The actual numbers, disaggregated by race, ethnicity, and gender, are presented in Appendix A.

Figure 3: Number of graduating students in a STEM major over the past six years

At the same time as the pool of GW students graduating with a STEM major is increasing, the number of applicants, matriculates, and graduates in the Master of Education in the field of secondary education with a STEM focus has steadily decreased over the past six years, as displayed in Figure 4.

Figure 4: Trends in applicants, admits, matriculates, and graduates in the Master of Education in the field of secondary education with a STEM focus in the past six years

While there are many distinct reasons why the two trends are not necessarily linked, these inverse patterns can be interpreted in two different ways which are relevant to the NOYCE active recruitment strategy:

1. The need to increase awareness among the GW student population about the possibility of teaching careers is obvious and thus NOYCE Capacity Building has a large window of opportunity to make a difference.
2. The possibility of structural issues that may explain why at the same time as an increasing number of students graduate with a STEM-related degree from GW, the interest in a secondary education degree with a specialty in science and math seems to decrease.

To probe into these structural issues, the evaluators conducted a number of focus groups and interviews with students and key informants.

• Findings from focus groups

Table 2 summarizes the list of factors motivating and deterring students from going into a teaching career. These findings are based on two focus groups. The first was conducted with a group of first year students enrolled in the Research in Science course in the fall 2014. The second was conducted with a group of students enrolled in the JUMP program; these students span the sophomore, junior, and senior grade level. The total number of students participating in focus groups was 27. Among these, one student expressed her interest in pursuing a career as a high-school teacher.

While students recognized the importance of acquiring teaching skills while undergraduate students at GW, the overwhelming majority of students taking part in the focus groups did not consider a teaching career as one of their future career plans. A range of personal, financial, and structural factors were put forth to explain why a teaching career was not a desirable option for them.

At the personal level, some students indicated they did not have the personal traits to be a good teacher (e.g., patience, communication skills, and pedagogy). Other students expressed aspirations for careers that they considered “more prestigious” such as physicians, engineer, medical researchers, or university professors. These aspirations were echoed by adults in their immediate environment, whether parents or current professors and advisors. Closely related to their personal motivations were financial considerations. Many students reiterated that given the high tuition of GW at the undergraduate level, they had to consider well-paying jobs to have a good return on their educational investments. The general perception of the students was that high-school teaching positions were not well paid, with limited upward mobility. Many students also brought up some systemic factors that make the teaching profession not palatable: notably, the instability of the job market in case of budget cuts, the bad reputation of teachers in the American culture, as well as their own experience with unmotivated teachers in high school.

Table 2. Factors motivating and deterring undergraduate STEM students to consider a teaching career

Motivating factors about “teaching”	Deterring factors about a “teaching career”
Teaching skills are important to have because it is likely that at one point in a career any professional will have to train others	Not financially rewarding given: ·       high tuition at GW ·       amount of work teaching requires
Teaching skills are important insofar as they help convey complicated scientific ideas to people who are not experts	Lack of job security and upward mobility (both financially and professionally): ·       Even in a university or college context: once tenure is obtained there is no upward mobility ·       It is difficult to get a teaching job and teachers get laid off when there are budget cuts
It is motivating to see the positive contributions of a good teacher on the life of a student	By being STEM majors at GW students are not set up for a teaching trajectory. Instead they are expected to pursue a more “prestigious path”: ·       Teaching in the American culture is not glorified ·       GW students are expected to “do better than that”
Recognition that good teachers in their own schools made a difference in their lives and influenced them in their choice of STEM majors	Experience of their relatives who have teach jobs is not positive
Teaching as an opportunity to study behaviors and psychological patterns	Experience with bad teachers in secondary school did not inspire them to become teachers
Enthusiasm around the idea of having a course or a seminar where teaching skills are taught (e.g., LA program)	The overall US education system is flawed: past teachers could not change the system, so students don’t want to be part of it.
	Not “having what it takes” in terms of patience, pedagogy, communicative skills: “the best scientists are not the best teachers”

• Findings from interviews with GW career center

The limited interest in teaching careers by GW undergraduate STEM students was corroborated by professionals from the GW career center. The career advisor specializing in education explained that it is very rare to have students in STEM majors consult with her about career options. Money is a very important factor, along with the fact that STEM majors are in high demand in other markets and have the choice to consider other higher-paying options.

Given the high cost of tuition and some students’ high level of indebtedness, they feel pressure to earn high salaries after graduation. Students might not be aware of the “loan-forgiveness” program that could contribute to decreasing the pressure of having a high salary after graduation. Students seem to have limited awareness of the multiple graduate fellowship possibilities in some schools of education that give graduate students in education the opportunity to be put into the classroom early on in their career and the opportunity for tuition benefits.

An additional explanation lies in the fact that the inspiration to become a teacher often takes root when students have had experience working with children through service work or activities linked to the Centers for Civic Engagement. Once they have had this ‘first spark’ they might be more willing to consider a career in education when this was not on their map before. Yet, so far, students in STEM majors have had more limited opportunity to have this first key interaction in a teaching-like career.

Furthermore, GW does not offer an education track (e.g., teaching major, preparation for teaching certificate) at the undergraduate level. This may impact students’ decision-making process at two distinct levels:

• Students who have an interest in STEM education coming out of high school are likely not to enroll at GW because of the lack of teaching certificate preparation at the undergraduate level
• Given that GW does not have a teaching certificate at the undergrad level, students who are interested in teaching tend to go for the alternative route such as Teach for America, or Americorps. This initial experience might further convince or deter some students to continue into a teaching career.

 

• Results from Graduating Senior Survey and Career Center Post-Baccalaureate Survey Data

Each year, the Office of Survey Research and Analysis conducts a survey of graduating seniors to better understand their overall experience at GW as well as to learn about their post-graduation plans. The Graduating Senior Survey (GSS) 2014 had an overall response rate of 90%[2]. A targeted analysis of the survey results was conducted by filtering responses from graduating seniors in a STEM major[3]. Among the respondents, only one indicated participating in a volunteer program, such as Teach for America, Americorps, or Peace Corp after graduation. No respondent indicated starting a graduate degree in education, and only one respondent indicated “teacher” as a job title. These numbers are captured in Figure 2.

• Results from GSEHD Alumni survey

In the fall 2014, GSEHD conducted an Alumni Electronic Survey using the Taskstream Assessment Survey System. For the purpose of this survey report, information was collected from graduates of secondary education teacher preparation programs from 2005 through 2014. The response rate was 24.8% after excluding 66 unreachable respondents with no contact information. It should be noted that the results should be interpreted with caution as the response rate from the survey is below the national average of 30% for online survey administration. Of 435 participants, 108 respondents responded to the alumni survey.

Table 3 provides a pictures of the graduates’ fields of interest from 2005-2014 as a baseline before the Noyce program was implemented.

Table 3. Results of GSEHD Alumni survey

Content areas	% of total sample	N
Math	6.48	7
Biology	7.41	8
Chemistry	3.7	4
Physics	4.63	5
General Science	0.93	1
	100	108

Not disaggregated by specialty, GSHED Second Education Alumni have the following profile:

• 41 % teach in DCPS
• 78% teach in DC Charter schools
• 22% teach in STEM focused school
• 62% teach in high needs school
• 62% reported being completely prepared to implement standards of their school district including common core.

Many alumni reported teaching in STEM-focused or high-needs schools.

• Evidence from the awareness campaign for GWTeach and Noyce

Notwithstanding the above evidence, a possible turning point in the long-term recruitment potential of the program is taking shape with the introduction of the UTeach program at GW. As laid out above, one of the main constraints to recruiting candidates for teaching STEM in secondary education among the GW undergraduate population is a selection effect. Given that GW did not offer a path towards teaching, students that were considering such career option, may have selected out of GW. Moreover, once on campus, the absence of such a path reinforced the ingrained idea that STEM majors ‘should aspire for another type of career’. The introduction of GWTeach opens such a path to teaching. Not only does this program have the potential to attract students who are already thinking about teaching as they select into colleges, but it also can cause a ripple effect among the STEM major student population. In the medium-term, the GSEHD secondary education program in science and math teaching has the potential to benefit from such a change.

GWTeach is a new program for GW that was introduced in fall 2015, based on a successful model called UTeach. National data from UTeach shows that 80% of graduates of the program across the nation are still teaching after five years. GW recently received a large grant to implement the program. The objective is to create an undergraduate level pipeline to teaching in STEM secondary education.

Students completing the GWTeach program will have a degree in their field — math, science, or engineering — and the option to teach in school. By completing the GWTeach curriculum graduates will have a bachelor’s degree in a STEM field and be eligible for licensure by the District of Columbia as a middle or high school teacher. That means that graduates can go directly into the STEM workforce, teaching profession, or graduate school.

The introduction of this pipeline to teaching within GW undergraduate population has the potential to alleviate a major structural roadblock for the recruitment of future graduate students in secondary STEM education, the key target audience for the NOYCE Capacity Building grant. When recruiting for the GWTeach program among the undergraduate population, the number of students showing interest in enrolling in the first step (a one-credit course) was higher than anticipated. This shows that there is a pool of interested students who are interested in acquiring teaching skills in STEM education. The creation of a pathway to teaching at the undergraduate level has the potential to raise the profile of secondary STEM education and overcome some of the students biases sensed in the focus groups with students.

• Effectiveness of recruitment strategy

Finding 1: Limited short-term but high long-term relevance of the recruitment strategy

Given the current limited pool of potential applicants at GW, the short-term program’s recruitment strategy would need to redirect its efforts towards increasing its outreach to non-GW population. At the same time, an active awareness-raising campaign on campus to attempt to change both the minds of students and faculties on the prestige and worthiness of a secondary education career is a worthwhile effort that may play out in the long run. This effort in information-sharing and awareness-raising seems more relevant in light of the upcoming start of the GWTeach program which is dedicated to creating a curriculum to prepare STEM majors to take the certification exam at the end of their senior year.

Applications and admissions

At this stage, it may be premature to assess the effectiveness of the recruitment strategy as only one admission cycle has been completed. Nevertheless, one of the key objectives of the program was to increase enrollment within a year. The objective for the Master’s program has not been met within the year, total new admits for the summer and fall of 2015 combined is 5. However, at the undergraduate level the combined effort of this project and the GWTeach start-up has exceeded expectations with more than 50 freshman and sophomore students enrolled in the GWTeach program for fall 2015.

A range of recruitment activities were undertaken over the course of the year as illustrated in Table 4. Through Listserv or flyers the information about the program was disseminated to the entire undergraduate population enrolled in a STEM major.

Additionally the LA program participants were a core audience for the recruitment. The GW LA Program places talented undergraduates into classrooms as Learning Assistants (LAs), where they work closely with the lead instructor to plan and enact research-based teaching practices. Learning Assistants were seen as most likely to be interested in teaching as a career and/or have more knowledge than the average major. The rationale is to reach a certain level of saturation in this group such that by the end of the project year more LAs are thinking about teaching and more of them know about the GW graduate program as an option.

The online marketing presence through Google Ad words and Facebook posts yielded 261 clicks. There is no data available for the previous years for comparison.

Table 4. Recruitment activities and estimated audience reached

Recruitment activity	Estimate of number of students reached
Listserv of STEM-related Student organization	30 organizations with memberships ranging from 3 to 60 students
Listserv of STEM-faculty advisors with invitation to disseminate to their students	Entire STEM major population
Teaching Panel	3 students attended in the 2014-2015 school year
List of DC-Based businesses related to STEM or STEM education	Dropped activity
Flyers on quick fact about teaching in STEM	Entire STEM major population
Tabling events in student centers	Reached 20 students
Career Fair	Reached 20 teachers/administrators
In person meeting with STEM-Faculty advisors
In person meeting with student organizations	7 students organizations

Table 5 lays out the baseline data that is the basis for determining the effectiveness of the present year recruitment.

Finding 2: Effectiveness of the recruitment strategy

The recruitment strategy has not led to the expected increase in enrollment for the summer and fall 2015 at the Master’s level. That being said, awareness raising and capacity-building are processes that bear fruit in the medium to long-terms. It is expected that the next cycle of recruitment should be a better reflection of the results of the full-range of Noyce recruitment activities. The effect of the strategy, in combination with the change in capacity offered by GWTeach has had a significant positive effect on the incoming freshman and sophomore classes.

Box 2. Insight from the Teach for America experience at GW

While GW does not currently have a teaching major (notwithstanding the introduction of GWTeach), it is one of the main nation-wide providers of Teach for America (TFA) fellowships in second-tier level schools. Some lessons can be drawn from the TFA recruitment success. However, it is unclear whether students who apply for TFA are truly interested in a teaching career, if they use the program as an interim (or gap year) while deciding what career they are interested in pursuing, or if they rely on the prestige of the program as a spring board for a different career. This is a difficult question to answer given that GW does not systematically track the activities of its alumni more than 6 months after their graduation.

That said, from a recruiting perspective, TFA can be regarded as a successful model for GW campus. Based on an interview with Career Center staff, the evaluators found that TFA recruiters have a sustained presence on GW campus. Not only are they represented at the career fairs and various forums but the TFA representative for GW is also continuously present on campus. For instance, he meets with students informally on a one-on-one basis to talk about the program and their interests in applying. Additionally, TFA brand is very strong and highly visible on campus. Leaflets and posters are present in residence halls and spaces where students gather to eat or spend time, such as the Marvin center

• Network and collaboration between partners

The second overarching goal of the program was to establish and sustain a dynamic collaboration among education and STEM faculty, DCPS, the Smithsonian, Carnegie Academy for Science Education, and GW’s Center for Civic Engagement with the view to better align the teacher preparation course work and field experience with DCPS standards and needs

• Dynamism of the Network

The Advisory Panels organized on September 18^th and June 2^nd 2015 were successful gatherings for the partners. All partners were represented except for DCPS. Participants presented their planned contribution to the program and ideas were brainstormed on how to make the most of each other’s resources for the benefit of the program. It is too early to establish whether the connections made through the program will be sustainable. Nonetheless, given that the partnership relies on the comparative advantages of each partner and is mutually beneficial to members, it started on sound grounds. For instance Figure 5 shows how the Noyce goals are connected to the Smithsonian Q?rius goals, as well as through the pre-teacher summer internship program.

• Effectiveness of the Network

 

• Cooperation in needs assessment of DC schools for graduate and undergraduate internships and service activities

One of the important parts of the pre-service teachers’ curriculum is the field placement. The quality of the field placement is particularly important; there is evidence that placing pre-service teachers in schools that have a particular focus on math and science and where there is an institutional commitment to disciplinary practices as reflected in NGSS and CCSSM is a key contributing factor to the success of the field experience.

As a result the program aimed to identify STEM-focused schools that are willing to host pre-service teachers. Through collaboration with DCPS’ STEM integration managers and various Charter schools, as well as advice from CASE, the program conducted a needs assessment.

The needs assessment started by emailing 20 schools, eight of which responded. The responses obtained were used to set up further connections with GW’s Center for Civic Engagement on finding opportunities for GW undergraduate STEM majors to volunteer or intern at the schools. Concrete discussions on setting up internships did not take place. However the program can still be on track to generate quality school placements by spring 2016, which is when most of the students in the upcoming cohort will do their internships. This timing change is due to a recent modification in the overall curriculum of the secondary education Master’s program, which has been compressed to 30 credits.

With regard to the development of opportunities for GW undergraduate STEM majors to volunteer or intern at STEM-focused DC public schools, the program, with the cooperation of the GW Center for Civic Engagement, aimed to create a dedicated webpage on its roster of service-learning and volunteer opportunities. The dedicated webpage was created and has been live as of late February and has information about 7 schools with contact information and a brief description of the service learning opportunity. The use and internet traffic associated with this page, as well as the number of actual placements due to this page, is unknown at this point.

• Experience at the Smithsonian Q?rius Center

One of the most innovative elements of the program is the development of a field experience at the Smithsonian National Museum of Natural History’s Q?rius Center. The focus of this original field experience was to expose pre-service teachers to STEM facilitation in informal settings. The pre-service teachers placed at Q?rius over the summer of 2015 had direct contact with the visitors in the Q?rius collection area. An added component of the internship had to do with the observations of how visitors interact with particular objects or stations within Q?rius.

On December 10^th, there was a meeting between the main partners and the evaluators to discuss research and evaluation. This fruitful conversation led to a clarification of goals and expectations regarding the intended outcomes of the Q?rius internship experience. Figure 5 summarizes the goals of the Q?rius internship programs and how they intersect between Noyce program’s overarching goals and Q?rius overarching objectives, as well as how they relate to NGSS main practices. Appendix B lays out the measurement framework with a more refined set of objectives and evaluation criteria for the assessment of teacher-candidate internship experience. The next section assesses whether the goals have been achieved. An update for participants in the summer 2016 can be found in section 3.5.4.

Figure 5: Field experience at the Smithsonian

• Progress on scaffolding induction follow-through

Another mechanism through which the program intended to increase alignment between teacher preparation and DCPS standards, as well as ensuring the success of alumni/ae of the program working in DCPS, was by developing an induction year mentoring plan. Two streams of activities fed into this part of the program: (i) the creation of a mentoring plan around three core components (job placement, relationships and professional networking, and pedagogy); (ii) developing an alumni database through which graduates can be identified, contacted, and supported during the early years of their career as science and math teachers.

The mentoring plan will be inspired by the well-established model from Math for America (MfA) via consultation with CASE. Informal consultations with CASE have led to a fruitful exchange of good practices.

As of August 2015, little progress on the development of a dedicated TaskStream database and identification of Alumni from the program has been achieved. That said, in the summer 2016, a graduate student has begun tracking graduates’ employment history from 2002. Although it remains a challenge to figure out how to fund a mentor program, this database is being created with the hopes of being able to better identify potential mentors. Furthermore, as a result of the needs analysis, the PIs have connected with two alumni in leadership roles within DC public and public charter schools and are working with them to establish placements for 2015-16.

Finding 3: Effectiveness of the network

Noyce program relied on a rather sound division of tasks based on the comparative advantage of various partners. The advisory panels held in September 2014 and June 2015 were effective ways to allow the various partners to present their respective contribution to the rest of the network. Activities have progressed unevenly. While the preparation and implementation of the internship with the Smithsonian Q?rius moved steadily and was completed successfully, limited progress was achieved in developing a mentoring plan for alumni and in developing an effective tracking system. The needs assessment with DCPS and Charter Schools within DC for placement of both undergraduates and graduates has yet to show concrete results. Little progress has been achieved on identifying alumni of the program who are currently teaching in DCPS with the view of building a database as a platform for implementing the mentoring plan.

• Impact of the community engaged teaching experience at Q?rius
  • Participation/Attendance

Three teacher-candidates participated fully in the field experience at Q?Rius: two new full-time students and one continuing student. The PIs found that in general the new (summer start, full-time) Secondary Education students were better able to immerse themselves in the Q?rius experience as compared to the continuing, part-time student. The continuing students have jobs and personal commitments that significantly limited their availability this summer.  Some teacher-candidates regretted that due to scheduling issues they spent a good portion of their experience being a “greeter” in the space. From there they could not observe as well as the learning process of visitors. On the contrary when their shifts consisted in animating a defined set of activities (e.g., rocks and minerals or forensic mystery lab) they found it to be a very rich experience.

Box 3. Activities included in Q?rius field experience

Q?rius volunteer training:

The Teacher-candidates took part in 17 hours of general and specific trainings prior to starting as volunteers in the collection zone of Q?rius. The training touched upon the following themes: (i) Discovering behind-the-scenes scientific research; (ii) exploring activity content as learners; (iii) understanding the museum education approach; and (iv) working with multi-generational visitors; (v) examining scientist communication techniques.

Volunteering in the Collections Zone and the Forensic Anthropology Lab:

Once trained, teacher-candidates completed a number of volunteer hours in one of the two zones of Q?rius. The goal was for each teacher-candidate to complete 8 shifts of 4 hours.

School program observations

Pre-service teacher volunteers observed two hours of school programs. Q?rius organized visits for up to 35 students in each session. During their visits, the students can use objects, data and equipment, as well as digital media to complete a number of tasks based on Smithsonian research. The objective is for them to investigate on their own core ideas related to their school curriculum. Their experience helps them develop research skills. The observation of the school program by pre-service teacher was aimed to help candidates see how the museum approaches inquiry-based science instruction and how this can be done in a classroom setting.

• Change in teacher-candidates’ perspective around teaching in informal learning spaces :

In order to observe any changes in the thought process, attitude and awareness of teacher-candidates that would have been triggered by their field experience at Q?rius, a questionnaire was sent to students before their month of volunteering at the museum. They were also interviewed at the end of their experience. Their reaction should be interpreted as short-term but they are insightful in understanding the preliminary impact of a first teaching experience in an informal space.

Table 6. Findings from pre and post interviews with teacher-candidates

Pre-experience questionnaire	Post-experience Interviews
What do you expect visitors to do in the collection zone in Q?rius?
·       Candidates are focused on visitor’s information need ·       Materials exposed in the collection zone are regarded as a source of knowledge, while visitors are portrayed as knowledge-seekers and receivers ·       One candidate went beyond this conception of the space as an information field and thought of it as a reflection space and a go-to for future inquiries	·       The candidates emphasizing questioning and reasoning. They recognized that it is possible to convey information while probing reasoning and inquiry. ·       One candidate explained that Q?rius is about removing the barriers of inquiry that are typical in museum where observation is the primary. In Q?rius specimens can be touched, observed under a microscope, “Bringing hands-on learning at the forefront” ·       Nonetheless the focus on “information gathering” remains important. For example, one candidate proposed to measure “successful Q?rius” experience with the amount of data gathered in the field-book during the visit.
How do you think this experience in an informal setting will affect /affected your future teaching?
·       The teacher-candidates conceived their role as teachers, first and foremost as being able to answer questions. Their primary expectations from their experience at Q?rius was to sharpen this set of skills ·       The teacher-candidates also mentioned other types of skills that they intended to develop while in Q?rius: o   Skills to engage students in both formal and informal learning spaces o   Develop ways to bring in the interaction that is inherent in an informal space, into their classroom o   Skills on how to ask questions that “hook students” o   How to think outside of the box in approaching their lesson plan	·       The teacher-candidates emphasized the importance of facilitating inquiry and asking questions that feed the curiosity of the learner. Without asking the right questions and triggering further inquiry, the learners may have quickly moved on from one specimen to another. ·       One teacher-candidate emphasized that “it is the questioning-strategy that makes a great teacher.” ·       Engaging students to explore a specimen not only visually, but also tactile, comparison, microscope observations, recognitions of patterns is an important skill.

Themes from teacher-candidates’ Reflective Journals

Additionally, each teacher-candidate kept a reflective journal during their field experience at Q?rius. After each shift, they reflected back on their experience, looking in particular at learners’ ideas and interactions, learner engagement and overall impressions and insights. The themes that were mentioned most frequently are summarized in Table 7.

• Achievement of field experience objectives

Objective 1: Teacher- candidates attending to diverse learner’s thinking

The three teacher-candidates that took part in the Q?rius field experience demonstrated a willingness to seek the opportunity to facilitate the inquiry of different group of students. They incrementally gained confidence shift after shift. While at the beginning two of them shied away from engaging with learners for fear of not having the correct answer and the appropriate knowledge base, these two students both had an important turning point and became much more engaged.

The three teacher-candidates used their journaling to reflect, compare, and contrast their interactions with various groups of learners. Some reflected on different learners based on age, on their learning styles (e.g., visual learners), based on the type of questions they asked (‘how questions’ vs. ‘why questions’).

The three teacher-candidates demonstrated reflective practice. As illustrated in Table 7, two candidates in particular identified early on a couple of challenges that they were facing when facilitating inquiry (e.g., engaging with teenagers who did not seem to want help, asking questions while not knowing the answer) and kept reflecting on these challenges, how to overcome them, and different trial and errors strategy that they attempted during their various volunteering shifts.

Objective 2: Teacher-candidates facilitate questioning

The area where teacher-candidates seem to have made most progress lies in developing their appreciation and skills for ‘inquiry-based learning’ and asking questions to engage the Q?rius visitors. All student-candidates reflected on the challenges of engaging learners by asking them questions in areas where they themselves have little knowledge, and might not be able to answer. They each developed different strategies to get around this initial obstacle. They observed how parents would engage their own children, how other volunteers would engage visitors, and attempted to mimic these behaviors.

One of the teacher-candidates distinguished between two types of learners in the Forensic Lab, the learners who ask “why” questions and those who ask “how” questions. Some questions were also seen as introductory questions that had the potential to ‘break-off’ the engagement. For example, one teacher-candidate was asked “are these real?” as a generic question and felt embarrassed not to be able to answer for the various specimens. They also distinguished between various attitudes around questioning: they recognized that asking the right question is not sufficient; it needs to be accompanied with prompting, clueing, and accompanying the learner through the questioning process.

Table 7. Q?Rius Field Experience: Teacher-Candidates Journals thematic analysis

Themes	Teacher-Candidate 1	Teacher-Candidate 2	Teacher-Candidate 3
Anxiety about not being knowledgeable about all the specimens	·       At the outset, expressed anxiety about lack of familiarity with specimens: “What is the point of a question with no answer?” ·       Turning point: an interaction where the candidate knew enough about a theme to engage a learner and ask questions to facilitate inquiry	·       Intimidation about not knowing much about the specimens ·       Turning point: when observing a mother asking questions to her own child. And confidence boost when realizing that other volunteers did not necessarily have more expertise.	·       Did not seem to experience the same anxiety, maybe because the student was assigned a specific activity in a well-defined task environment (the rock and mineral activity)
Anxiety about engaging learners who are not directly asking for help	·       Uneasiness in engaging visitors who are not explicitly asking for help. ·       Compounded by the fact that the candidate was not confident in their knowledge-base: “nerve racking” ·       Overtime, the anxiety was not as strong with young children but persisted with adolescents who “never seem to want any help”
Appreciation for “inquiry-based learning”	·       Found comfort in inquiry based learning where finding the answer is not as important as asking the question. ·       Eagerness to become a good facilitator as well as a good lecturer ·       Several moments during field experience when the teacher-candidate enjoyed “learning together” or “learning from a 6 year old”	·       While the candidate has general appreciation for the process of asking questions, this is a difficult process. The candidate gained confidence as they acquired more knowledge about specimens, which enabled them to be more engaging and ask more questions. ·       Comparing their questioning style with another more experienced volunteer, led the candidate to realize that “something was missing” in their style. That asking a question is not sufficient to engage learners, they need time to answer, hints and clues, active ·       One particular interaction with a couple of visitors who had a large knowledge-base, made an impression on the candidate and had them reflect on the importance of “being an expert” (content-expert)	·       Not much reflection on the ‘inquiry-based’ part of learning. Focus remains on “knowledge”, “content”, the “correct answer”. Satisfaction when the “students left more informed” ·       The teacher-candidate was destabilized when they did not have the complete set of material to carry-out their activity ·       Interested in the learners’ reasoning behind a particular answer.
Environmental conditions (open space, number of visitors, resources, etc.)	·       Overwhelmed when the number of visitors was high ·       Found the Collection Zone (bigger and more open space) more disturbing than the Forensic mystery lab which is more confined and bounded ·       More comfort in leading a well-defined activity without having to “grapple with the awkwardness of intruding to offer unwanted help everyone in the lab was looking for help in completing the analysis.” ·       Complaints about lack of information on computer: invalidating feeling of referring learners to a source that is not informative	·       Also found the Forensic mystery lab easier to navigate and a space where their contribution as volunteer can be greater. This good experience was a jumping point to gain confidence in the collection zone. ·       Appreciation for resources, tips, curriculum items, websites to which they were exposed during the dinner workshop.         ·       Same complaints about the computers.
Diversity in learners’ thinking and learning process	·       Identified two types of learners, categorized into the “why” and the “how” categories. The two groups approach the mystery with different techniques ·       Easier interactions with younger learners, perceived as more open to help	·       This teacher-candidate had a harder time engaging with younger learners (under 10). The candidate found it hard to ‘do science’ with visitors who did not have basic skills (e.g., conversion)	·       Young children focus on staring and then touching the specimen ·       Characterized young children mostly as visual learners. One learner relied on the weight of a rock as opposed to its visual characteristic to identify properties.

 

This quote from one of the teacher-candidate’s journal is particularly insightful to better understand the transformational impact that Q?rius has had:

“My last week at Q?rius came so soon. If I look back when I first started Q?rius volunteering, I was shy and afraid to talk to any visitor. I just opened boxes when they asked me to, and responded to questions from visitors by saying mostly “I don’t know. Sorry!” But after eight shifts, 32 hours at Q?rius made me more confident to ask and facilitate visitors to think, as well as keeping me to think and seek for more knowledge. I know that this is only a start. I have to improve more and more until I can be like other numerous experienced volunteers at Q?rius and be a good teacher at school.”

Objective 3: Teacher-candidates sustain curiosity and engagement

The teacher-candidates described in great details their strategies for overcoming some of the challenges to sustaining learner’s engagement. For them successful engagement could sparkle from giving the right information, or asking the right question at the beginning of the interaction. Once a visitor found interest in one ‘activity’, it was easier to sustain their engagement, by comparing and contrasting different specimens, by looking at different rocks under the microscope, by attempting for example to identify the age and sex of a person based on a few clues from their skulls.

Finding 4: Merit and effectiveness of the ‘Community engaged Teaching Experience at Q?rius 2015

The field experience at Q?rius achieved all of its main objectives. Through volunteering in the Collection Zone and in the Forensic Lab, through keeping a reflective journal on their experience, and through observations and debrief with their instructors, the teacher-candidates were able to show their capacity to engage in reflective practices and attend to diverse learner’s thinking, facilitate visitors’ inquiry and sustain curiosity and engagement. While it is too early to establish whether this experience will have a sustained impact on how the teacher-candidates engage with their future students and think about their role as teachers, it is undeniable that this experience has triggered a series of critical ‘ah-ha moments’ that are likely to leave a mark on teacher-candidates’ practice.

• Secondary Education in Math and Science Curriculum Changes

A number of curriculum improvement activities were at the core of the program’s strategy to increase its alignment with CCSSM, NGSS and DCPS requirements. Three sets of activities were part of the original proposal:

1. Modify the curriculum for science and math methods courses, with a focus on rich disciplinary-specific engagement. Each course will be co-taught by a member of GSEHD and CCAS faculty
2. Create a discipline-specific version of the Instructional Model and Classroom Management course to be co-facilitated by a DCPS science and math teacher
3. Identify another DCPS teacher to host, co-design and co-teach a new Engineering in the Math and Science Classroom for summer 2015

• Science Method (Fall 2014)

In the fall semester of 2014, five students were enrolled in the “Teaching Science” course co-taught by a member of the science faculty from CCAS and a member of the science education faculty at GSEHD. The focus of the course was to have pre-service teachers “do science” to help them better learn how to engage their own students in doing science. The curriculum for this course was aligned with the NGSS standards by focusing on key concepts, on cross-cutting ideas, and on the disciplinary practice of “inquiry.” For example, pre-service teachers were asked to build models, to development scientific arguments and create their own experiments to solve a scientific problem.

This course provided the opportunity for in-depth research on how the co-facilitation and the opportunity to “do science” impacted the students. With the following overarching research question: how is the experience consequential? The course sessions were thus video-taped and the recording analyzed. Findings are not yet available but will be presented in a conference in the spring 2015.

Students were asked to write a reflection piece on their experience with the course. The main messages from these pieces are as follows:

• The course helped pre-service teachers reflect in-depth on student thinking and idea formation, and how to make them the focal points of unit plans development
• Through the activities around “doing science,” the pre-service teachers discovered the significant impact that modeling and asking questions can have on student thinking
• Through a visit to Q?rius they learned about the importance of informal science teaching and to look for existing resources to do science outside the classroom
• The pre-service teachers also gained the skills of developing complete unit plans
• The presence of two instructors in the class and, for some, the field observations of multiple science courses in a DC high-school, gave students a unique insight into how the same basic topic can be approached and taught in different ways
• They gained appreciation for considering the full process of scientific inquiry in science teaching, instead of solely focusing on content.

Research stemming from the course observation material

The course and the video-taping of the process generated rich research material that was successfully exploited by the PI and presented in academic conferences.

Weekly 1 hour meetings were organized to discuss and collaboratively analyze the video data from the two co-taught courses. The PIs collected and logged approximately 75 hours of video footage of our teacher candidates in their science methods and STEM perspectives courses. Approximately 30 hours of that footage consists of the STEM/STEM Education faculty co-facilitation of doing math and doing science.

The research was presented in both national and local conferences as follows:

• Sikorski, T., & Doebel, H. (2015). A disciplinary practices-oriented rationale for science and science education faculty collaboration in pre-service methods courses. Paper presented at the 2015 Annual Meeting of the National Association for Research in Science Teaching, Chicago, IL.
• Sikorski, T., Pyke, C., & Smith, K. (scheduled July 7, 2015). Presentation at the University of Maryland, College Park Science Education/Physics Education Research Group.
• Sikorski, T. (2014, September). "Doing science" with LAs and future science teachers. Presentation at the GW DBER Group, Washington, DC.

Box 4. Excerpt from Research presentation

The purpose of the research was presented in these terms: “Drawing on video recordings and artifacts from pre-service science teaching methods course focused on modelling the circulatory system, we explore potential affordances of bringing science and science education faculty together to facilitate scientific inquiry in our methods course. “

Overarching Question: What happens when we bring science and science education faculty together to collaborate on “doing science” in a methods course for pre-service teachers that foregrounds disciplinary practices?

• By what criteria are ideas assessed? (Coffey, Levin, Hammer, & Grant, 2011)
• How are next moves offered, negotiated, and selected? (Ball, 1993)
• Videotaped all sessions of the methods course (3 hours/week; 14 weeks)
• Focused on 6 sessions of co-taught “doing science” (~12 hours total)
• Audiotaped 1 hour debriefs the day after each course meeting.
• Digital copies of planning notes and classroom artifacts.

 

• Perspectives in research in math and science (Spring 2015)

Six students were enrolled in the Perspectives in Research in Math and Science course taught during the spring semester. Two students were from the Math concentration and four from the science concentration. The curriculum also focused on modeling as one of the main disciplinary practices in the NGSS. The students are “doing math and science” by working through five problem-sets.

• Other curricular activities

It was decided to postpone the development of an engineering course that will be co-taught with a DCPS teacher. Originally, the engineering course was supposed to run within the STEM Master Teacher certificate during the summer 2015. However, independent of the Noyce project, the start of the STEM Master Teacher certificate was delayed by one year due to the launch of a redesigned Secondary Education program. The STEM Certificate will now be launched in the summer 2016, which is when the team also expects the engineering course to run.

Finding 4: Preparation of pre-service teachers aligned with NGSS and CCSSM

The shift to a disciplinary approach in the methods courses of the secondary education STEM program was piloted in two courses of the curriculum. It is too early to determine whether the changes in the curriculum under the program have a long-lasting effect on pre-service teachers’ preparation. However, the short term feedback from students participating in the Science Methods class was largely positive with evidence that they have gained appreciation for the importance of experiential learning by “doing science” and focusing on student-thinking. This was reinforced for the teacher-candidates participating in the Q?rius field work.

3.5      Findings from No-Cost Extension Year

3.5.1      Updates on Evaluation Objective 1: Analysis of recruitment efforts

Master’s in Education

The conclusions drawn regarding the effectiveness of the recruitment strategies to meet this objective remain limited, as only two academic years have been completed. With regards to the Master’s level degree, the total number of committed students for summer and fall 2016 is five. Among these 5 attendees, a number of different math and/or science majors are represented, including: mathematics, chemistry, biology, and health sciences. This number is the same as last year, indicating no change.

A range of recruitment activities were continued over the course of the no-cost extension year. For example, funds from the NOYCE grant were used to distribute banner advertisement in local publications. The findings from these efforts are illustrated in Table 8. Additionally, a STEM webpage was created for the GSEHD website. Individuals in the GSEHD department also met individually with each of the undergraduate chairs of math and science department to establish relationships. Finally, the marketing coordinator developed a STEM placard for distribution at admission fairs and appointments.

At this point, it is hard to quantify the impact of these efforts, as the admissions office receives numerous inquiries on a daily basis. However, the Admissions office hopes that changes in the application tracking system will provide the means to keep track of this type of data in the future.

Table 8. Recruitment activities for local publications

Recruitment activity	Estimate of number of students reached
Arlington Connection (local publication)	11 clicks to website from banner ad
Arlington now	219 clicks to website
Fairfax Times	24 clicks to website from banner ad
Alexandria Gazette	17 clicks to website from banner ad

 

GW Career Center Updates

Graduating Senior Survey 2015

The Graduating Senior Survey (GSS) 2015 had an overall response rate of 91%[4]. A targeted analysis of the survey results was conducted by filtering responses from graduating seniors in a STEM major[5]. Among the respondents, two indicated participating in a volunteer program, such as Teach for America, Americorps, or Peace Corp after graduation. No respondent indicated starting a graduate degree in education, and only one respondent indicated working as a teacher. Thus, there do not appear to be any large changes in career plans among graduating seniors. This is not surprising, however given that a majority of students’ participating in programs like GWTeach are freshmen and sophomores, thus it may take one or two more years to see a significant increase in future plans.

3.5.1      Updates on Evaluation Objective 2: Network collaboration

GWTeach Program

At the undergraduate level, the combined effort of this project and the GWTeach start-up continues to demonstrate positive outcomes. The 2015-16 academic year marked the start of GWTeach, with 57 students (33 in the fall; 24 in the spring) enrolled in the first required 1-credit course. Thirteen of these students have continued in the program, enrolling in the next two required courses as part of the minor program. An additional 34 new students are enrolled for the first required course for the fall 2016 semester.

Although there does not appear to be an increase in participation in the GWTeach program, there are a number of explanations for similar enrollment numbers between this year and last. First, there were unforeseen difficulties in spring 2016 recruitment strategies, such as limited faculty/staff, which resulted in less classroom visits to advertise GWTeach across STEM courses. Second, the physical classroom capacity was cut in half due to laboratory availability, thus although some students have expressed interest in taking the introductory course, there is no space. There are plans for continued recruitment efforts at the start of the school year (e.g. information table at Kogan Plaza, ice cream social during Welcome Week, and bulletin boards/flyers posted in dorms throughout campus) that may increase enrollment for the fall. This, of course, is contingent upon gaining more space to allow increased for additional students.

In addition to recruitment plans, there are two important updates in the past year for the GWTeach program. First, staff expressed confidence that the official application process for the program will be ready this coming fall. In the past year, the program was offered to any interested students without having to officially apply for the GWTeach minor. Additionally, the faculty/staff hope to have some of the coursework be a part of the Service Learning program at GW, given the field experience aspect of the program, which may increase student interest.

Finally, student responses to the program suggest continued success as a result of GWTeach implementation. For example, one chemistry major wrote the following about the program:

“Someone like me, who feels passionately about teaching and knows what she wants to do, can just right into a classroom after college.”

Field Experience

Once again, students were given the opportunity to participate in the field experience at the Smithsonian National Museum of Natural History’s Q?rius Center. The internship was also expanded to include experience in an interactive exhibit, the Butterfly Pavilion. The teacher-candidates who participated fully in the program kept journals, with one student even starting an online blog regarding her experiences at the Q?rius Center.

Math and Science Curricular Development  

In fall 2015, four additional students (2 math candidates, 2 science candidates) enrolled in the Science Methods course introduced fall 2014. Since its introduction, faculty and students continue to analyze video-taped data regarding the course. Analyses of the video case studies (one for science, one for math) are still underway. Clips have been selected and reviewed by participants. Although currently behind schedule, finishing the write up of the text to accompany the clips is a major priority as the project comes to an end. The cases are about 30% complete and targeted for completion and publication by the end of September.

In addition to these video case studies, there has been significant progress in other research activities including: (1) detailed logs of all 40 hours of video collected during the study, (2) transcripts of selected clips, (3) a NARST 2015 conference paper, (4) a NARST 2017 conference paper under development to be submitted on August 15th, and (5) a review of research on the use of museums in science teacher preparation which will be submitted for review soon. One of the CO-PIs also served as discussant on a session at AERA on the topic of disciplinary practices, which is one of the themes of the Noyce project. Furthermore, there have been a few presentations to STEM education groups in the DC area. Evidence of research dissemination can also be seen in the opportunity to publish research on "doing science" with future teachers in the PhysTEC fall issue and the invitation to submit project data for inclusion in the Case Studies in Science Project associated with Tufts University.

The research was presented both nationally and locally as follows:

• Sikorski, T., & Nogay, W. (in preparation).Doing science to learn to teach science.PhysTEC 2016 Fall Newsletter.
• Sikorski, T. & Pyke, P. (in preparation). “Can we do an experiment?”: Sensemaking and Agency in Next Moves Conversations. To be submitted to the 2017 Annual Conference of the National Association for Research in Science Teaching, San Antonio, TX.
• Sikorski, T. & Lau, M. (in preparation).Science teacher learning in museums: A review of alignment with effective professional development and dimensions of science learning.
• Pyke, C., & Sikorski, T. (in preparation). Teachers’ Learning to Teach Practices in Museums. In S. Uzzo (Ed.), Research to Practice for PCK in STEM. 
• Pyke, C., Sikorski, T., Smith, K., Doebel, H., & Ullman, D. (2016). Catalysts for creating curricular change and alignment. Poster presented at the 2015 NSF Noyce Summit. Washington, DC. July 20, 2016.
• Sikorski, T. (2016). Discussant for Learning Science through Engaging in the Practices of Science.Division C Learning and Instruction section 1d: Science, American Association of Educational Research annual meeting. Washington, DC. April 11, 2016.
• Sikorski, T. & Popson, C. (2015). Community-Engaged Teaching Field Experience at Q?rius: 2015. DC STEM Summit. Breakout Session 1: Elements of Successful Pathways—Engaging the Public and Private Sector. Gallaudet University, Washington, DC. November 5, 2015.
• Doebel, H., Pyke, C., Sikorski, T., Smith, K., & Ullman, D. (2015). "Doing math" and "doing science" in the context of a teacher preparation program. Presentation to the GW Discipline-Based Education Research Group, November 30, 2015, Washington, DC.
• Smith, K., Sikorski, T., Pyke, C., Ullman, D., & Doebel, D. (2015). "Doing math" and "doing science" in the context of a teacher preparation program. Presentation to the UMD Science Education Research Group, August 25, 2015, Washington, DC.

The development of an engineering course that will be co-taught with a DCPS teacher was pushed back once again, but final development of the course materials is currently in progress, and funding has been secured for a cohort of students to take the course in the 2016-2017 academic year.

4.    Conclusions and recommendations

Overall, the program continues to make good progress towards implementing the full range of activities included in its ambitious Noyce proposal. Even with the no-cost extension that was rewarded, it is clear that the findings and conclusions laid out above remain preliminary. The program continues to rely on a sound division of labor and a dynamic network that tapped into each partner’s comparative advantages. It remains unclear whether GW undergraduate population enrolled in STEM majors is the right target for the program, but it appears that interest in activities associated with the grant may be among younger students, thus more time is needed to see whether such activities have a meaningful impact on career choices.

That being said, the fact that GW has received a UTeach grant to set up a curriculum and prepare STEM majors to obtain a teaching certificate upon graduation is a possible game-changer. This program has the potential to become the missing link in the envisioned pipeline to science teaching at GW. First, it has the potential to increase awareness about science teaching as a career option for GW STEM majors. By preparing undergraduates in STEM majors to obtain a teaching certificate upon their graduation, GWTeach also has the potential to attract applicants from high-school graduates with an interest in teaching who did not consider GW as a viable option in the past. This would remove one of the main structural obstacles in the recruitment of GW undergraduates for the program. The consistent interest in GWTeach over the course of the no-cost extension year suggests that this program holds a lot of promise in the coming years.

Progress on the various activities involving network partners was somewhat uneven. While the preparation and implementation of the Smithsonian internship program was very successful, limited progress on the needs-assessment and the creation of an induction program were achieved.

On the basis of the progress achieved to date and the preliminary findings of the evaluative analysis, four main areas for improvement have been identified. (The bullet points under the first area constitute more specific suggestions.)

1. Enhance scope and intensity of recruitment efforts

• Increase the outreach to potential applicants outside of GW and increase collaboration with the Admission office to better cater recruitment material to both GW and non-GW audience.
• Increase the visibility of the program in student spaces (e.g., Marvin Center, Residence Halls) as well as via a more active use of social media. The successful TFA recruitment strategy on GW campus can serve as a useful model.
• Secure adequate space for GWTeach course enrollment
• Interview students recently admitted into the Teach for America and Americorp programs to better understand why they applied to the program and when they thought about applying. Use this information to help inform a better recruitment plan for this program.
• Promote the teaching certificate option to parents and potential applicants to GW. Parents are concerned about job opportunities for their children after graduation.
• Promote the teaching certificate option to current students as a way for them to obtain leadership and communication skills, and a means to support themselves while they think about future career options.

2. Think more strategically about the relationships between Noyce Capacity Building activities and other existing initiatives to raise the profile of STEM teaching (e.g., GWTeach, Learning Assistant program, JUMP program)

• In the past few years, the George Washington University has developed and implemented a range of programs aimed at generating interest and capacity among students for teaching STEM. These initiatives have each achieved important results independently, but they would have a higher potential for impact if they were articulated more strategically. For example, the very recent start of GWTeach could be strategically positioned as a possible incubator for the GSEHD Secondary Education in Sciences and Math program that this grant is supporting.
• The recruitment for GWTeach among the undergraduate student population, the LA and the JUMP program have already been astutely leveraged to spread the word about the Master and Certificate program offered by GSEHD, but more synergies could be exploited.

3. Explore avenues for offering scholarships for students enrolling in the program as well as funding for internships (e.g., via the work-study scheme)

• As laid out in the evaluation, one of the primary factors explaining why some of the undergraduate students in STEM major do not consider teaching as a viable career option has to do with financial constraints. Supporting students through scholarships would be a way to alleviate some of these constraints and may work as a powerful incentive to convince interested students.
• The research team is aware of this and is already taking step to address this constraint. They have submitted a new proposal to Noyce for funding a scholarship program for students interested in teaching in high-need schools.
• Integrating the GWTeach program with the Service Learning programs may also provide a means to motivate involvement and ultimately spark an interest in teaching careers.

4. Renew efforts to create an induction program into DCPS for the graduates of the program

• As demonstrated by one of the partners’ experience (Math for America, MfA), supporting and mentoring teachers beyond their graduation from the program is critical to their professional success. Building a tracking system from scratch is a difficult but necessary endeavor to be able to identify both potential mentees and mentors. However, once this is in place, the team has received useful advice and material from MfA that can be readily implemented.
• This project is currently underway, with a research assistant working to assemble and migrate data on math and science teacher graduates from 2002 forward into a single database. Significant progress has been made in the past few months, and will provide a means to identify potential mentors in the teaching profession (see Alumni Tracking Survey Report and Database).

APPENDIX A: STEM UNDERGRADUATE POPULATION AT GW

Table 4: Number of graduating seniors in a STEM major between over the last 6 academic years

STEM Major	2009-10	2010-11	2011-12	2012-13	2013-14	2014-15	Total
Applied Science & Technology	9		2	1	4	5	21
Biology	86	85	86	78	84	136	555
Biomedical Engineering	22	17	35	37	33	61	205
Biophysics	5	2	2	1		7	17
Chemistry	15	17	15	20	16	34	117
Civil Engineering	9	18	17	22	32	37	135
Clin Lab Sci: Microbiology		1		2	1	1	5
Computer Engineering	8	7	5	7	5	10	42
Computer Science	24	21	20	23	26	42	156
Electrical Engineering	5	7	4	14	15	20	65
Information Systems	14	19	19	17	20	25	114
Integrated Info,Science,&Tech			9	15	13	21	58
Mathematics	4	13	14	9	21	35	96
Mechanical Engineering	21	20	27	21	47	54	190
Pharmacogenomics	7	2	7	5	6	6	33
Physics	6	9	4	6	8	9	42
Statistics	8	9	5	8	10	18	58
Systems Engineering	8	10	8	8	17	35	86
Grand Total	251	257	279	294	358	558	1997

Source: extracted from Banner July 2016

Table 5: Number of graduating seniors in a STEM major between over the last 6 academic years by race/ethnicity

STEM Major	American Indian	Asian	Black	Hispanic	Int’l	Pacific Islander	Two or More Races	Unknown	White	Total
Applied Science & Technology	0	5	3	1	2	0	0	2	8	21
Biology	0	138	42	30	27	1	11	50	256	555
Biomedical Engineering	1	53	18	10	8	1	3	11	100	205
Biophysics	0	4	0	0	0	0	1	1	11	17
Chemistry	1	31	8	6	10	1	3	6	51	117
Civil Engineering	0	16	3	9	14	0	1	10	82	135
Clin Lab Sci: Microbiology	0	1	0	2	0	0	0	1	1	5
Computer Engineering	0	7	3	1	4	0	1	2	24	42
Computer Science	0	19	5	5	23	1	1	18	84	156
Electrical Engineering	0	8	10	4	8	0	0	3	32	65
Information Systems	0	17	6	8	16	2	1	7	57	114
Integrated Info,Science,&Tech	0	9	4	2	7	0	1	14	21	58
Mathematics	0	13	3	3	22	0	0	5	50	96
Mechanical Engineering	0	22	7	10	11	0	8	12	120	190
Pharmacogenomics	0	9	6	1	2	0	0	1	14	33
Physics	1	8	0	1	2	0	0	1	29	42
Statistics	0	8	2	2	16	0	2	5	23	58
Systems Engineering	0	8	7	3	16	0	2	7	43	86
Grand Total	3	376	127	98	188	6	35	156	1006	1995

Source: extracted from Banner July 2016

Table 6: Number of graduating seniors in a STEM major between over the last 6 academic years by gender

STEM Labels	F	M	Total
Applied Science & Technology	10	11	21
Biology	331	224	555
Biomedical Engineering	105	100	205
Biophysics	8	9	17
Chemistry	61	56	117
Civil Engineering	61	74	135
Clin Lab Sci: Microbiology	4	1	5
Computer Engineering	10	32	42
Computer Science	38	118	156
Electrical Engineering	12	49	65
Information Systems	29	85	114
Integrated Info,Science,&Tech	20	38	58
Mathematics	54	42	96
Mechanical Engineering	44	146	190
Pharmacogenomics	22	11	33
Physics	9	33	42
Statistics	28	30	58
Systems Engineering	43	43	86
Grand Total	634	805	1995

Source: extracted from Banner July 2016

APPENDIX B: PLAN FOR MEASURING OUTCOMES OF Q?RIUS FIELD EXPERIENCE

Goal 1	Attending to diverse learners’ thinking
Description	Teacher candidates actively seek the opportunity to practice facilitating inquiry among a diverse population of learners and demonstrate increased capacity to attend to different learners’ thinking.
Evidence	GWU teacher candidates will: ·       actively seek the opportunity to facilitate the inquiry of different groups of learners (e.g., age, gender, ethnicity/race, primary language, ability)
	·       compare and contrast how different groups of learners interact with one or multiple objects and describe what adaptive strategy they adopted to facilitate each learner’s inquiry
	·       demonstrate reflective practice and identify one or more challenge(s) that they encountered while facilitating diverse learners’ inquiry
Goal 2	Facilitating questioning
Description	Teacher candidates strengthen their facilitation skills and value the opportunity to engage with learners in an informal setting
Evidence	GWU teacher candidates will: ·       listen to and observe learners and adjust questioning strategies accordingly ·       model how to ask generative questions with the learners
	·       identify one or more question(s) that learners generated as a result of their interaction
	·       identify one or more Q?rius questions that learners investigated in Q?rius and describe what they did to help the learners explore their questions.]
Goal 3	Sustaining curiosity and engagement
Description	Teacher candidates identify the challenges of sustaining curiosity and engagement and develop a strategy to address these challenges
Evidence	GWU teacher candidates will: ·       identify one or more challenges to sustaining learners’ engagement and describe what they did to overcome these challenges identify and document one or more patterns that emerged from their observation of learners’ interaction with artifacts and/or Q?rius space ·       identify and document one or more feature(s) of Q?rius that make(s) the space well-suited for inquiry

[1] This figure is not meant to be interpreted dynamically. The numbers represent a fixed picture of the situation in each category at one point in time: the academic year 2013 2014. Additionally, there is no assumption that the undergraduate and graduate levels are automatically connected.

[2] A 90% overall response rate does not mean that 90% of the graduating class answered each questions of the survey, but that 90% of the graduating class answered the required questions and submitted their answers.

[3] The list of STEM major used in every analysis in this report is based on the official list laid out on the website of the United States Department of Homeland Security list of STEM degree: http://www.ice.gov/sites/default/files/documents/Document/2014/stem-list.pdf

[4] A 91% overall response rate does not mean that 91% of the graduating class answered each questions of the survey, but that 91% of the graduating class answered the required questions and submitted their answers.

[5] The list of STEM major used in every analysis in this report is based on the official list laid out on the website of the United States Department of Homeland Security list of STEM degree: http://www.ice.gov/sites/default/files/documents/Document/2014/stem-list.pdf