EDUCATION AND OUTREACH www.csiro.au Indigenous STEM Education Project Second evaluation report September 2014 – September 2017 Zane Ma Rhea, Michael Tynan, Christopher Banks, Shane Phillipson, Kirsten Sadler and Celia McNeilly COPYRIGHT © Commonwealth Scientific and Industrial Research Organisation 2018. CITATION Ma Rhea, Z., Tynan, M., Banks, C., Phillipson, S., Sadler, K., & McNeilly, C. (2018) Indigenous STEM Education Project Second Evaluation Report: September 2014 – September 2017. CSIRO. Canberra. ii Indigenous STEM Education Project Acknowledgements Acknowledgement of Country Other acknowledgements Aboriginal and Torres Strait Islander peoples have longstanding scientific knowledge traditions. These traditions have developed knowledge about the world through observation, using all the senses; through prediction and hypothesis; through testing (trial and error); and through making generalisations within specific contexts. These scientific methods have been practised and transmitted from one generation to the next and contribute to particular ways of knowing the world that are unique as well as complementary to western scientific knowledge. CSIRO wishes to acknowledge the significant knowledge and leadership of Aboriginal and Torres Strait Islander scientists, educators and program leaders that have made the development and implementation of the Indigenous STEM Education Project possible. This respect encompasses recognition of traditional contexts for technologies and concepts; their application in the past, present and future, including supporting modern STEM career pathways for Aboriginal and Torres Strait Islander students; and reaffirming the ingenuity and creativity of Aboriginal and Torres Strait Islander peoples’ knowledge systems. The authors would like to thank the external peer reviewer and CSIRO Performance and Evaluation for their peer review of this report. Their valuable suggestions have made a significant contribution. The Project Steering Committee’s feedback on this report is also gratefully acknowledged. We would also like to thank the program element leaders for their constructive feedback. CSIRO acknowledges the contributions of former and current members of the program teams and evaluation team who led or supported the program monitoring and evaluation methodologies, data collection, and analysis. A deep respect for these Aboriginal and Torres Strait Islander cultural practices and knowledge underpin the philosophy and practice of the Indigenous STEM Education Project. Recognition of traditional contexts for technologies and concepts; their application in the past, present, and future; including supporting modern STEM career pathways for Aboriginal and Torres Strait Islander students; reaffirm the ingenuity and creativity of Aboriginal and Torres Strait Islander peoples’ knowledge systems. The Indigenous STEM Education Project team acknowledges the Traditional Owners of the lands with whom this Project is collaborating and their vibrant living cultures and knowledge systems. We pay our respects to Elders past and present and thank all community members who are providing the leadership to ensure meaningful and effective engagement with Aboriginal and Torres Strait Islander communities for the six distinct but complementary STEM education programs that make up this Project. CSIRO acknowledges that Aboriginal and Torres Strait Islander peoples make extraordinary contributions to Australia in cultural, economic and scientific domains; for example, incorporating Indigenous knowledge of ecological and social systems is vital to the achievement of sustainable development. iii Contents Acknowledgements................................................................................................................ iii Executive summary..................................................................................................................1 Introduction........................................................................................................................... 2 Inquiry for Indigenous Science Students (I2S2) program.......................................................... 5 PRIME Futures.......................................................................................................................10 Aboriginal Summer School for Excellence in Technology and Science (ASSETS).........................14 Bachelor of Science (Extended) program.................................................................................18 Science Pathways for Indigenous Communities...................................................................... 20 Indigenous STEM Awards.......................................................................................................21 Indigenous STEM Education context...................................................................................... 22 Conclusions.......................................................................................................................... 26 References........................................................................................................................... 27 1 Appendix A: CSIRO response to EEGL recommendations.....................................................31 2 Appendix B: first evaluation report key findings................................................................ 33 3 Appendix C: Indigenous STEM Education Project research framework................................ 34 4 Appendix D: PRIME Futures principals’ responses............................................................. 37 5 Appendix E: PRIME Futures teachers’ responses................................................................ 39 6 Appendix F: student responses to ASSETS survey questions at time 1 and time 2 (2016-17).........................................................................................................41 7 Appendix G: student responses to PISA self efficacy questions at time 1 and time 2 (2016-17)........................................................................................................ 43 iv Indigenous STEM Education Project List of acronyms ASSETS Aboriginal Summer School for Excellence in Technology and Science – one of the Project’s six program elements PISA The Programme for International Student Assessment CEdO CSIRO Education and Outreach PRIME Purposeful Rich Indigenous Mathematics Education CSIRO Commonwealth Scientific and Industrial Research Organisation QUT Queensland University of Technology EEGL Education Experts Group Limited – consultancy that independently reviewed this report STEM Science, Technology, Engineering and Mathematics ICSEA Index of Community Socio-Educational Advantage – a scale which allows for fair and reasonable comparisons among schools with similar students. YDC YuMi Deadly Centre – A research centre at QUT that delivers the PRIME Futures program I2S2 Inquiry for Indigenous Science Students – one of the Project’s six program elements YDM YuMi Deadly Maths – a mathematics pedagogical framework that covers all strands of the Australian Mathematics Curriculum developed by Queensland University of Technology and delivered by QUT as the PRIME Futures program one of the six program elements of the Indigenous STEM Education Project. i ii Indigenous STEM Education Project Executive summary The Indigenous STEM Education Project aims to increase participation and achievement of Aboriginal and Torres Strait Islander students in STEM, it consists of six program elements that cater to the diversity of students as they progress through primary, secondary and tertiary education, and into employment. The First Indigenous STEM Education Evaluation Report (Tynan & Noon, 2017) concluded that the initial implementation of program elements had been successful. However, as the first report focused on implementation, it was not intended or able to provide conclusive evidence that program elements were contributing to an improvement in the engagement, attendance and improved academic achievement of Aboriginal and Torres Strait Islander students in STEM subjects. This Second Evaluation Report begins to provide positive evidence that the goal of increased engagement and achievement of Aboriginal and Torres Strait Islander students is being met across the program elements. Analysis of the data also highlights the need to be cautious in interpreting the findings as these programs are still in the early phase of their implementation and that it will require ongoing monitoring and continual improvement of these programs in both content and in methodological design to ensure that they are meeting the STEM aspirations of Australia’s Aboriginal and Torres Strait Islander students and their families.. The key findings in this Second Evaluation Report include: • The substantial impact of the I2S2 program on increasing Indigenous and non-Indigenous students’ achievement and engagement. Although attendance decreased slightly for all students, likely due to seasonal factors, it did not preclude engagement or achievement increases. • The benefits of I2S2 were particularly pronounced for the Indigenous and non-Indigenous students who were achieving below level prior to the I2S2 program. Sixty three per cent of these students improved their grades. • PRIME Futures is demonstrating sustained student engagement, and improved learning and understanding. • ASSETS is demonstrating increases in student engagement, including aspirations for university and STEM studies, and some students choosing more STEM subjects in Year 11. • Initial data showing high levels of transition of former ASSETS students to university, including strong representation in STEM or STEM-related degrees. • The Bachelor of Science (Extended) is experiencing variable recruitment (between 5 and 12 students per year), with retention (between 58 – 100 per cent) and average subject completion rates (67 per cent) broadly comparable to national rates for Indigenous university students and for all science degree students in Australia. The evaluation findings of the Indigenous STEM Education Project should be interpreted within the context of contemporary research on participation in STEM by Aboriginal and Torres Strait Islander peoples. A number of potential biases could affect the methodologies and the findings. A discussion of these biases and how the evaluation design has, and will, manage or minimise them are included in this report. Forthcoming in-depth case study research on I2S2, Bachelor of Science (Extended), ASSETS, and Science Pathways for Indigenous Communities will provide further key evidence on the impact and effectiveness of these elements. Education Experts Group Limited (EEGL), an education consultancy based in Melbourne, was commissioned by CSIRO to make independent recommendations and verify the findings included in this Second Evaluation Report. A summary of the findings and the corresponding recommendations are included in Section 1. CSIRO’s response to the EEGL recommendations are outlined in Appendix A. 1 Introduction The Indigenous STEM Education Project is a partnership The BHP Billiton STEM Indigenous Project is a partnership between CSIRO, Australia’s national research science between CSIRO, Australia’s national research science agency, and BHP Billiton Foundation, an independent agency, and BHP Billiton Foundation, an independent charity established by BHP Billiton to support large, charity established by BHP Billiton to support large, longlong‑term community projects by not-for-profit term community projects by not-for-profit organisations. organisations. The Project’s overarching goal is to provide The Project’s overarching goal is to provide supported supported pathways that improve the participation and pathways that improve the participation and achievement achievement of Aboriginal and Torres Strait Islander of Aboriginal and Torres Strait Islander students in STEM students in STEM subjects and professions. The Project subjects and professions. The Project consists of six consists of six programs. Three of these are universal program elements. Three of these are universal programs: programs: Inquiry for Indigenous Science Students Inquiry for Indigenous Science Students (I2S2) and PRIME (I2S2) and PRIME Futures which are science inquiry Futures which are science inquiry and maths programs and maths programs implemented in metropolitan implemented in metropolitan and regional communities; and regional communities; and Science Pathways for and Science Pathways for Indigenous Communities which Indigenous Communities which uses Traditional Ecological uses Traditional Ecological Knowledge as the basis for Knowledge as the basis for teaching science in remote teaching science in remote communities, which is the communities, which is the evolving knowledge acquired evolving knowledge acquired by Aboriginal and Torres Strait by Aboriginal and Torres Strait Islander peoples through Islander peoples through thousands of years of contact and thousands of years of contact and ongoing relationship ongoing relationship with the local environment. Three of with the local environment. Three of the programs are the programs are targeted: the Aboriginal Summer School targeted: the Aboriginal Summer School for Excellence for Excellence in Technology and Science (ASSETS) and the in Technology and Science (ASSETS) and the Indigenous Indigenous STEM Awards which support, celebrate and STEM Awards which support, celebrate and extend high extend high achievers; and the Bachelor of Science (Extended) achievers; and the Bachelor of Science (Extended) which which provides an alternate pathway to a university science provides an alternate pathway to a university science degree for students requiring additional assistance. degree for students requiring additional assistance. Purpose of this evaluation This Second Evaluation Report draws on data from the Project’s start until September 2017. It reports on the evaluation of progress towards ‘success’ outcomes which are largely quantitative indicators of engagement, attendance, and improved academic achievement of Aboriginal and Torres Strait Islander students in STEM subjects. The core hypothesis is the program elements are delivering innovative programs that are leading to improved engagement, attendance and improved academic achievement of Aboriginal and Torres Strait Islander students in STEM subjects. Key findings and recommendations are made based on analysis of these data. Revision of program element program logics and Project Theory of Change The First Evaluation Report (see Key Findings from the First Evaluation Report at Appendix B) summarised evidence that supported the program elements’ program logics and the Project Theory of Change as being sound and based on the best practice literature. However, peer reviewers of that report identified that the Project’s Theory of Change 2 Indigenous STEM Education Project lacked clarity with respect to identifying the key drivers of change at the Project level. A subsequent review of both the Project Theory of Change and the individual program element logics has identified five activities that all program elements utilise, in part or whole, to effect change: • Academic excellence in curriculum development; • Teacher professional development in culturally relevant pedagogies and curriculum content; • High expectations extra-curricular engagement; • Stakeholder engagement; and • Personalised support. Outcomes and impacts The First Evaluation Report also argued to more clearly distinguish between ‘outcomes’ and ‘impact,’ which aligns with CSIRO’s broader evaluation approach. This distinction allows a better understanding of the outcomes that are under the direct influence of the Project, and its intended future impact (typically beyond the Project’s funding period). As part of this revision process, it was also decided to refer to the Project Theory of Change and program element logics as ‘Impact Pathways’ in order to be consistent with broader CSIRO evaluation terminology. This Second Evaluation Report focuses on the ‘Outcomes’ elements because there is not yet sufficient data to report on the ‘Impact Pathways.’ The Indigenous STEM Education Project research framework, including each element’s outputs, targets, indicators of success, and outcomes, is in Appendix C. The Impact Pathways for each program element and the overall Indigenous STEM Education Project are available at www.csiro.au/en/ Education/Programs/Indigenous-STEM/Monitoring-Evaluation Table 1 outlines the measurable achievements of each program element against the agreed indicators and the related EEGL recommendations. CSIRO’s responses to the recommendations are in Appendix A. What is evident from the findings is that the overall program of work is an inductive endeavour. Each program element has been contextualised to the needs of the Indigenous students in supporting their interest in STEM and improving their academic achievement in STEM-related subjects. For some agreed indicators, there are still areas where there are insufficient data to measure outcomes but efforts have already been made to ensure that jurisdictional access and research design issues are being addressed for the benefit of future understanding. Based on CSIRO’s continual improvement processes, the Third Evaluation Report will report on both indicators/success (level 1 outcomes) and pathways to success (level 2 outcomes). Table 1: Key findings and recommendations of the Indigenous STEM Education Project KEY FINDINGS (OUTCOME INDICATORS) EEGL RECOMMENDATIONS I2S2 Improved student results (academic achievement) 38 per cent of Aboriginal and Torres Strait Islander students demonstrated an increase in academic achievement (51 per cent maintained their achievement levels) (statistically significant increase, 0.24 effect size). There was variability across the two time periods among all sub-groups in terms of academic achievement; high achieving students showed the least amount of variability (63 per cent of all students remained at the same achievement level, compared to 33 per cent of low achieving students). Improved student engagement 34 per cent of Aboriginal and Torres Strait Islander students had increased levels of engagement (50 per cent maintained engagement levels) (statistically significant increase, 0.17 effect size). Improved attendance 1. As teachers are required to both implement the program and assess students, the viability of other skill and knowledge assessment processes should be explored to confirm current findings e.g. access jurisdictional data. 2. Further data collection and analysis be undertaken to better understand variability in the effectiveness of the program by year level, geographical location and socio-economic disadvantage (ICSEA). Although 36 per cent of Aboriginal and Torres Strait Islander students improved their attendance, 54 per cent of Aboriginal and Torres Strait Islander students’ attendance levels decreased (statistically significant decrease, 0.12 effect size); however, this did not preclude the improvements in achievement and engagement. Students choosing STEM subjects The first I2S2 students will enter Year 11 in 2018 after which data will be able to be collected. PRIME Futures Improved student results The perceived increase in student test results is trending in the right direction. 19 per cent of teachers reported better test results after six months, increasing to 24 per cent after 12 months. Improved student engagement The changes in student outcomes in all three areas are positive. 81 per cent of teachers reported high levels of student engagement after 6 months, increasing to 90 per cent after 12 months. 63 per cent of teachers reported improved learning and understanding after six months, increasing again after 12 months to 70 per cent. In terms of student interest in STEM, 13 per cent of teachers reported greater interest in these subjects. Teachers agree that all students demonstrate each of the four dimensions of engagement. Furthermore, the teachers’ perceptions do not change from Survey 1 to Survey 2. This result is consistent with teachers’ broad assessments of engagement across different student groupings including Aboriginal and Torres Strait Islander students and non-Indigenous students. 3. As current forms of data collection do not include direct student achievement, engagement or attendance data, it should be triangulated with other data sources that have established validity and reliability (e.g. NAPLAN) in future reports. 4. Teachers be encouraged to use online platforms provided by YuMi Deadly Centre to share their lesson plans more broadly with teachers beyond their school. 5. Continued monitoring by YuMi Deadly Centre of the teachers’ perceived lack of support from school leadership and/or information about Indigenous resources. There was an indication that some teachers perceived a lack of support from school leadership and a lack of information regarding local Indigenous resources. 3 ASSETS Student choice of STEM subjects in Years 11 and 12 ASSETS participants are high achieving and have an existing interest in STEM, however six out of the twenty-one students (29 per cent) who had selected one or two STEM subjects for Year 11 intended to change to more STEM subjects for Year 11 due to the experience of ASSETS Summer School. In contrast, only one out of twenty-six students (four per cent) who had selected three or four STEM subjects for Year 11 intended to change to additional STEM subjects. Students choosing STEM at university Seven of the eight 2014-15 cohort respondents are studying at university, and three of these were studying science subjects. Improved student engagement ASSETS pre- and post-summer school surveys are showing statistically significant changes in student aspirations to pursue STEM studies at a university (increasing from 63.7 per cent to 84.8 per cent) and their aspirations to pursue a career in STEM (increasing from 48.5 per cent to 81.8 per cent). Improved student results Not yet sufficient data – will be available for analysis in the next report. 6. T he survey of participants would benefit from development to ensure that it focuses on collecting data that is closely linked to Project outcomes. 7. An instrument be developed to better reflect student variability in their STEM knowledge and skills to replace the PISA questions. 8. An instrument be developed to better reflect student variability in their personal development, including aspects of leadership, knowledge of university and career options, the desirability of STEM, and the desirability of nonSTEM courses. 9. T he Summer School activities could be rated to the extent that they meet the cognitive, social, cultural and aspirational needs of students. 10. A gender and site analysis be undertaken to establish areas of best practice and areas that require improvement. 11. Further ASSETS data collection and analysis should pay particular attention to attribution. Bachelor of Science (Extended) The Bachelor of Science (Extended) is experiencing variable recruitment (between 5 and 12 students per year), with retention (between 58 – 100 per cent) and subject completion rates (67 per cent) across the different student intakes being comparable to national university rates for Indigenous students. 12. Research should focus on the reasons that students chose to study the University of Melbourne Bachelor of Science (Extended), choose to remain in the course as well as the reasons that they depart. Such research should include data from both students and the teaching staff. Science Pathways for Indigenous Communities program The Science Pathways for Indigenous Communities program has exceeded its target quota of participating schools, teachers and students. Data has not been collected on student attendance, engagement, achievement and teachers teaching more science as the research program is yet to receive jurisdictional approval for undertaking the research. 13. An instrument be developed for students to measure their behavioural, emotional, cognitive and social dimensions of engagement with the materials. 14. Teacher evaluation of attendance should include school records. 15. Multimodal opportunities for assessments of student academic achievement should be used to supplement teacher evaluations of academic achievement. 16. An understanding of student backgrounds such as year level and gender would be useful to establish predictive relationships to determine effect sizes of the program on student outcomes. 4 Indigenous STEM Education Project Inquiry for Indigenous Science Students (I2S2) program Program elements, outputs and targets The Inquiry for Indigenous Science Students (I2S2) program develops and implements Indigenous inquiry resources targeting middle school students (Years 5-9). Students are from mainstream metropolitan and regional schools. The inquiries are delivered as part of a school’s regular science curriculum. The inquiries utilise multimodal delivery and assessment techniques. These allow Aboriginal and Torres Strait Islander students to demonstrate their cognitive science skills through a diversity of modalities that are not necessarily dependent on [English] literacy skills. The I2S2 team also trains science teachers in their delivery and broader Indigenous cultural awareness relevant to their implementation. At the time of writing this report, I2S2 was working in 15 clusters, 74 schools (cumulative total of 82 schools), with 275 teachers, 2,895 Aboriginal and Torres Strait Islander students and 8,491 non-Indigenous students. This exceeds its targets of 168 teachers and 2,100 Aboriginal and Torres Strait Islander students and is close to the original target of 84 schools. For this report, 2016 data is analysed for 17 schools in New South Wales and Queensland for which both principal and jurisdictional consents have been obtained. This is 46 per cent of a total of 37 participating schools in New South Wales and Queensland in 2016. Unfortunately, principal approvals for the remaining schools were not obtained in time for inclusion in this report. However, they are expected to be obtained for subsequent reports. The number of students participating in the I2S2 program in these 17 schools is 421 Aboriginal and Torres Strait Islander students and 1,308 non-Indigenous students. This represents 45 per cent of 936 Aboriginal and Torres Strait Islander students and 41 per cent of the 3,193 non‑Indigenous students across the 37 participating schools. Indicators of success 1. Improved student results 2. Improved student engagement 3. Improved attendance 4. Students choosing STEM subjects Research methods: program monitoring and jurisdictional administrative data I2S2 program monitoring involves two sets of data. The first is student related, consisting of student results (Grades A-E and N – not assessed), engagement (on a scale of 1-5) and attendance (percentage of classes attended) in the term prior and term during inquiry delivery. Participating teachers completed the student data collection template. Each inquiry has a detailed assessment rubric to assist teachers in the process. Teachers were instructed to refer to class and school data to assist with this task. The second, the engagement scale, is a simple five-point scale and has been conceptualised as similar to the grading of ‘effort’ which is a common school reporting practice. Jurisdictional administrative data - by March 2016, ethics and jurisdictional approval had been obtained in Queensland, New South Wales and South Australia for principals to provide permission for aggregated data to be made available for the evaluation. Negotiations in Western Australia are ongoing to obtain an extra level of parental/ caregiver consent. Because of the complexities involved in obtaining the data, not all students have both the pre- and during data for attendance, engagement, and academic achievement recorded. This reduces the number of students for which results are reported to 321 for attendance (34 per cent), 299 for engagement (32 per cent), and 347 for academic achievement (37 per cent). While permission was also obtained from schools involved in the 2015 pilot, data collection methods at that stage were not as effective and for this reason not included in this report. Key findings of the I2S2 program OVERALL Data on changes in teachers’ perceptions of student attendance, engagement, and academic achievement is presented in Table 2 (Aboriginal and Torres Strait Islander students) and Table 3 (non-Indigenous students). Wilcoxon Signed Rank tests were conducted to statistically compare differences in student attendance, engagement and achievement before and after participating in the inquiry-based learning. Indications of the impact of the I2S2 program on the rate of attendance, engagement, and academic achievement of Aboriginal and Torres Strait Islander students is outlined in the following sections. Given that the basis of the I2S2 is the use of resources that include awareness of Indigenous cultures and teacher professional development, the impact of the I2S2 program is expected to be positive for Aboriginal and Torres Strait Islander students. 5 Table 2: Teacher perceptions of attendance, engagement and academic achievement of Aboriginal and Torres Strait Islander students participating in the I2S2 program TIME 2 COMPARED TO TIME 11 STUDENT ACHIEVEMENT LEVEL2 (N)3 Attendance4 Engagement5 Academic achievement6 Z7 DIRECTION EFFECT SIZE (r)8 -0.092 ↓ 0.01 -2.223* ↓ 0.13 -2.934** ↓ 0.12 -2.846** ↑ 0.16 -3.484** ↑ 0.21 -4.125*** ↑ 0.17 -0.476 ↑ 0.03 -7.827*** ↑ 0.44 -5.943*** ↑ 0.24 PER CENT IMPROVED (N) SAME (N) DECLINED (N) A/B/C Level 41 14 45 (N = 155) (64) (21) (70) D/E Level 35 7 58 (N = 139) (49) (10) (80) All Levels (N = 321) 36 10 54 (116) (32) (173) A/B/C Level 37 45 18 (N = 149) (55) (67) (27) D/E Level 34 53 13 (N = 137) (46) (73) (18) All Levels 34 50 16 (N = 299) (102) (149) (48) A/B/C Level 19 65 16 (N = 159) (31) (103) (25) D/E Level 56 38 6 (N = 160) (90) (60) (10) All Levels 38 51 11 (N = 319) (121) (163) (35) 1Time 1 data was obtained prior to the beginning of the program. Time 2 data was obtained after the completion of the program. Declined = Negative Rank; Improved = Positive Rank; Same = Ties. 2Students were rated as A/B/C (high achieving) or D/E (low achieving) at Time 1. See 6 3All values of N are valid pre- and post-data. Note, however, that Overall values of N do not reflect the subtotals. 4Student attendance was recorded by teachers, which was converted to a scale from 0 to 100 per cent; teachers were instructed to refer to class or school attendance information when entering these data to ensure accuracy. Differences in attendance between Time 1 and Time 2 were tested using a Wilcoxon Signed Ranks Test as the data were not normally distributed. 5Student engagement was assessed by teachers using a five-point Likert scale (1 = avoids engagement, 2 = inconsistent engagement, 3 = participates, 4 = engaged, 5 = highly engaged). 6Student achievement was assessed by teachers using six categories (N = insufficient evidence, E = very limited, D = limited, C = sound, B = high, A = very high). Students who were not assessed a grade (N) at either pre- or post-test were excluded from the Academic achievement analyses, but were included in the Attendance and Engagement analyses (All Levels). 7Wilcoxon Signed Rank Tests were conducted, due to non-normal distribution of data (Attendance) and ordinal data (Engagement and Achievement). Significant levels: * p < .05 ** p < .01 *** p < .001 8Effect size r was calculated using a procedure similar to the Mann-Whitney U test: r = Z / √N, where N is the total number of observations (students x 2). Although there are existing classifications of effect size (Cohen’s (1988) impressionistic criteria (0.2 small, 0.5 medium, 0.8 large) and Gignac and Szodorai’s (2016) empirically derived criteria (0.15 small, 0.25 medium, 0.35 large)), Lipsey et al. (2012) warn of the inappropriateness of using general classifications of effect size for education interventions. 6 Indigenous STEM Education Project Table 3: Teacher perceptions of attendance, engagement and academic achievement of non-Indigenous students participating in the I2S2 program TIME 2 COMPARED TO TIME 11 STUDENT ACHIEVEMENT LEVEL2 (N)3 Attendance4 Engagement5 Academic achievement6 Z7 DIRECTION EFFECT SIZE (r)8 -3.767*** ↓ 0.10 -3.536*** ↓ 0.18 -4.991*** ↓ 0.11 -6.921*** ↑ 0.17 -5.938*** ↑ 0.32 -8.955*** ↑ 0.20 -1.015 ↑ 0.02 -10.149*** ↑ 0.53 -6.947*** ↑ 0.15 PER CENT IMPROVED (N) SAME (N) DECLINED (N) A/B/C Level 33 20 47 (N = 748) (247) (151) (350) D/E Level 30 15 56 (N = 200) (59) (29) (112) All Levels 32 19 48 (N = 962) (312) (184) (466) A/B/C Level 28 59 13 (N = 824) (229) (487) (108) D/E Level 49 40 12 (N = 169) (82) (67) (20) All Levels 31 56 13 (N = 1,009) (316) (561) (132) A/B/C Level 20 63 17 (N = 827) (164) (523) (140) D/E Level 68 30 2 (N = 186) (127) (55) (4) All Levels 29 57 14 (N = 1,013) (291) (578) (144) 1Time 1 data was obtained prior to the beginning of the program. Time 2 data was obtained after completion of the program. Declined = Negative Rank; Improved = Positive Rank; Same = Ties. 2Students were rated as A/B/C (high achieving) or D/E (low achieving) at Time 1. See 6 3All values of N are valid pre- and post-data. Note, however, that overall values of N do not reflect the subtotals. 4 Student attendance was recorded by teachers, which was converted to a scale from 0 to 100 per cent; teachers were instructed to refer to class or school attendance information when entering these data to ensure accuracy. Differences in attendance between Time 1 and Time 2 were tested using a Wilcoxon Signed Ranks Test as the data were not normally distributed. 5 Student engagement was assessed by teachers using a five-point Likert scale (1 = avoids engagement, 2 = inconsistent engagement, 3 = participates, 4 = engaged, 5 = highly engaged). 6 Student achievement was assessed by teachers using six categories (N = insufficient evidence, E = very limited, D = limited, C = sound, B = high, A = very high). Students who were not assessed a grade (N) at either pre- or post-test were excluded from the Academic achievement analyses, but were included in the Attendance and Engagement analyses (All Levels). 7Wilcoxon Signed Rank Tests were conducted, due to non-normal distribution of data (Attendance) and ordinal data (Engagement and Achievement). Significance levels: * p < .05 ** p < .01 *** p < .001 8Effect size r was calculated using a procedure similar to the Mann-Whitney U test: r = Z / √N, where N is the total number of observations (students x 2). 7 Impact of I2S2 on attendance As shown in Tables 2 and 3, overall attendance levels decreased for all students and sub-groups (low and high achieving, and Aboriginal and Torres Strait Islander students and non-Indigenous students). Specifically, 54 per cent of Aboriginal and Torres Strait Islander students’ attendance decreased from Time 1 to Time 2 (Z = -2.934), which was statistically significant (p < .01); however, the effect size was relatively small (r = 0.12). Similarly, a statistically significant decrease in attendance for non-Indigenous students was observed (48 per cent of students’ attendance decreased) (Z = -4.991, p < .001), although the effect size was also relatively small (r = 0.11). I2S2 program team discussions with teaching staff indicate this relatively prevalent reduction in attendance is likely related to seasonal factors such as greater illness in winter terms and students being absent for school camps. In line with this, although attendance decreased for many students, it did not preclude overall improvements in engagement and achievement, as discussed below. Impact of I2S2 on academic achievement Impact on engagement Aboriginal and Torres Strait Islander students’ achievement increased from Time 1 to Time 2, particularly for those students initially assessed as low-performing. In addition, the data suggests that this program element is also proving beneficial for non-Indigenous students. This is a promising finding as program staff report that mainstream schools seek to implement programs that are beneficial for all students, to justify their inclusion within a busy curriculum and multiple demands on teachers’ time. Tables 2 and 3 show that there were statistically significant increases in achievement among Aboriginal and Torres Strait Islander students (Z = -5.943, p < .001, r = .24) and non-Indigenous students (Z = -6.947, p < .001, r = .15). The largest effect size (r = .53) for any sub-group was for non-Indigenous low-performing students (Z = -10.149, p < .001), which saw 68 per cent of students improve their academic achievement. It should be noted that there is a ceiling effect when comparing Time 1 and Time 2; that is, students assessed as ‘A’ at Time 1 have no ability to increase their achievement level, which reduces the variability in the high-performing group of students. The overall engagement of Aboriginal and Torres Strait Islander students increased between Time 1 and Time 2. When comparing changes in engagement, the analysis shows that the increase in the engagement of Aboriginal and Torres Strait Islander students is statistically significant (Z = -4.125, p < .01); however, the effect size was relatively small (r = .17). Specifically, 34 per cent of Aboriginal and Torres Strait Islander students’ engagement increased, while 50 per cent of Aboriginal and Torres Strait Islander students maintained their engagement levels. Both low and high-performing Aboriginal and Torres Strait Islander students had overall statistically significant increased levels of engagement. The largest increase in engagement observed was among low-performing non-Indigenous students: 49 per cent of those students had increased engagement (Z = -5.938, p < .001, r = 0.32 effect size). An examination of the changes in proportions of students at each level of achievement is shown in Figure 1. The results show that the proportion of Aboriginal and Torres Strait Islander students achieving levels E and D has reduced (from 51 per cent to 35 per cent) and the proportion of students achieving levels A, B and C increased (from 49 per cent to 66 per cent). This result is remarkable given the short duration of the intervention. It points to the possible importance of the Indigenous context of the resources and/or the usefulness of multimodal delivery and assessment procedures, which will be further explored in future evaluation work. It also supports the notion that improving Indigenous student learning outcomes can occur in a relatively short space of time through concerted efforts (McRae et al., 2000). The data also suggests that improvements in academic achievement for higher performing students was mixed. This is an aspect of I2S2 that should be monitored because it indicates the possibility that some students have achieved success through existing ways of teaching and may find it unnecessary or difficult to adapt to alternative ways of teaching and learning. Further analyses will be reported on in future reports in relation to how student achievement, engagement and attendance are related to other variables, such as ICSEA, geography and year level. 8 Indigenous STEM Education Project ABORIGINAL AND TORRES STRAIT ISLANDER STUDENT ACHIEVEMENT Per cent of Students achieving at each level 50% 46 % 45% 39 % 40% 37 % 35% 30% 26 % 25% 20% 15% 16 % 12 % 9% 10% 9% 4% 3% 5% 0% E D C B A Achievement Level time 1 time 2 Figure 1. I2S2 Aboriginal and Torres Strait Islander student achievement levels (Time 1 and Time 2) Note: the percentage of Aboriginal and Torres Strait Islander students at each achievement level at Time 1 and Time 2, where E = very  limited, D = limited, C = sound, B = high and A = very high. Students that were rated N (insufficient information) were not included in this analysis. 9 PRIME Futures Program elements, outputs and targets PRIME Futures contributes to the overarching Indigenous STEM Education Project goal through the delivery of YuMi Deadly Maths (YDM), developed and delivered by the YuMi Deadly Centre at the Queensland University of Technology (QUT). YDM is a cohesive mathematics pedagogical framework covering all strands of the Australian Mathematics Curriculum from Foundation to Year 9. It seeks to achieve whole-school change in the teaching and learning of mathematics, by adopting a systemic approach to changing the whole school over a period of two or more years, working with the principal, or another senior leader, and a core group of teachers using a train‑the-trainer model (QUT YuMi Deadly Centre, 2018). PRIME Futures has been deployed in three phases: • Phase One commenced in Term 4, 2015 with two clusters in Queensland comprising 14 schools and principals, 57 teacher-trainers and 1,958 Aboriginal and Torres Strait Islander students. • Phase Two commenced in 2016 with two additional clusters in Queensland (commenced Term 3, 2016) and two clusters in South Australia (commenced Term 4, 2016) comprising a further 26 schools and principals, 133 teacher-trainers and 1,871 Aboriginal and Torres Strait Islander students. • Phase Three commenced in Term 2, 2017 with two additional clusters in Queensland, and two clusters in Western Australia comprising a further 30 schools and principals, 125 teacher-trainers, and 4,080 Aboriginal and Torres Strait Islander students. This totals 70 schools and 315 teacher-trainers which is well in excess of the targets for PRIME Futures across all three phases of 60 schools and 120 teachers. The target of 1,500 Aboriginal and Torres Strait Islander students was also likely met, although it was not possible to ascertain conclusively. An estimated total of 7,909 Aboriginal and Torres Strait Islander students attend the 70 schools involved in the program. However, fewer than this number would actually be receiving YDM teaching at this stage because only some staff in each school have been trained to date. Changes in teacher capacity have the potential to extend beyond the four-year life of the program, increasing the potential number of students who would benefit from PRIME Futures. PRIME Futures differs from other program elements as it targets all students in a large number of schools, rather than individual classes, small groups, or individuals. It follows that the lead teachers need time to understand the YDM pedagogy, try it in their own classes, and then train other 10 Indigenous STEM Education Project teachers in their school, who, in turn, make changes to their pedagogy. The changes advocated by YuMi Deadly Centre staff are intended to lead to changes to teaching program and practices. This requires some time to achieve. Indicators of success 1. Improved student engagement 2. Improved student results 3. Improved teacher capacity Research methods: surveys and teacher reflective journals Surveys - YuMi Deadly Centre has approvals from the QUT ethics committee and the relevant State education jurisdictions to conduct research enabling the evaluation of the PRIME Futures program. It collects six-monthly survey data from the teachers and principals leading the implementation of the program in the participating schools. The teacher participants are asked a number of questions in relation to student engagement and academic achievement on an aggregate basis. As pre-implementation data was not collected, and the intervention is a gradual introduction of a complete mathematics pedagogy, statistically significant changes are not expected over a six-month period. Instead, the establishment and maintenance of a positive trajectory will be used as evidence of success. Student achievement data / teacher reflective journals – As YuMi Deadly Centre does not have access to direct student data it uses two sources of data to monitor student achievement and attendance: information provided by principals (see Appendix D) and teachers (see Appendix E) about their students, including teacher reflective journals; and publicly available data such as NAPLAN results. NAPLAN data has been used previously to evaluate YDM (which commenced in 2010). Impact evaluation of schools involved in earlier programs has shown “students attending YDMactive schools between 2012 and 2014 outperformed their similar school counterparts from Year 5 to Year 7 by more than 30 per cent” (Spina, 2017, p.5). In a case study of one school, the first cohort of students who had experienced YDM during Prep, Year 1, 2 and 3 sat NAPLAN in 2015. The school achieved above the average for similar schools for the first time. The results also represented the first time that 100 per cent of Aboriginal and Torres Strait Islander students achieved above the national minimum standard (Spina, 2017, p. 7). NAPLAN data will, therefore, be used in this evaluation to report against the attendance and achievement success indicators in future reports. Key findings of the PRIME Futures program Overall, teacher perceptions of improved academic achievement and engagement were positive over the 12 months of data. IMPROVED STUDENT ENGAGEMENT 81 per cent of responding teachers (Survey 1: N = 73 and Survey 2: N = 68) in the first four clusters of schools reported increased levels of student engagement after six months in the program, rising to 90 per cent of responding teachers after 12 months (Figure 2). Teachers were also asked to differentiate this increase in engagement across different population groups on a five-point scale (Figure 3). They reported a modest and similar level of increased engagement (averaging between ‘somewhat’ and ‘moderately’ - third and fourth points of a five-point scale) across a variety of cohorts including gender, ability, and Aboriginal and Torres Strait Islander students. Again this data is encouraging, indicating that an increase in engagement due to YDM is being shared across all population groups. WHAT STUDENT OUTCOMES HAVE YOU OBSERVED WHEN USING THE YDM APPROACH? 100% 90% Per cent of Responding Teachers 90% 80% 81% 70% 70% 63% 60% 50% 40% 30% 24% 19% 20% 13% 10% 8% 8% 7% 1% 0% Increased student engagement Improved learning/understanding Better test results survey 1 More interest in STEM subjects/pathways/careers No perceived change in outcomes 4% Negative outcomes survey 2 Figure 2: PRIME Futures teacher perceptions of student outcomes using the YuMi Deadly Maths approach 11 When looking more closely at teachers’ perceptions of student engagement (see Figure 4), the results show that teachers agree that all students demonstrate each of the four dimensions of engagement (behavioural, emotional, social, or cognitive engagement). Furthermore, the teachers’ perceptions do not change statistically from Survey 1 to Survey 2. This result is consistent with teachers’ broad assessments of engagement. IMPROVED STUDENT RESULTS The perceived increase in student test results is trending in the right direction: 19 per cent of responding teachers reported better test results after six months, increasing to 24 per cent of responding teachers after 12 months. Sixty-three per cent of responding teachers reported improved learning and understanding after six months increasing again after 12 months to 70 per cent (Figure 2). In terms of student interest in STEM, 13 per cent of teachers reported greater interest in these subjects (see Figure 2). Principals responded positively to a question about whether the PRIME Futures program has increased the local Aboriginal and Torres Strait Islander community’s support for increased school attendance. A mean increase from ‘very little’ in the first survey (2.1) to close to ‘somewhat’ (2.8) in the second survey was observed. 12 Indigenous STEM Education Project IMPROVED CAPACITY OF TEACHERS IN MATHEMATICS This information is based on surveys of school principals and teachers. Principals reported a positive adoption of YDM by their teachers, and their perceptions of teachers’ levels of improved confidence, knowledge, pedagogical skills, Indigenous knowledge, and student expectations remained relatively steady between Survey 1 (n = 20) and Survey 2 (n = 21). Principals reported that YDM methods were used in K-Year 9 between ‘occasionally’ and ‘moderately’. They also reported that PRIME Futures moderately influenced their planning. Furthermore, the most effective strategies for implementing YDM were the train-the-trainer model, followed by peer support through classroom help, and informal discussions. Teachers found the various components of the PRIME Futures program as generally useful, with the books rated as good. Of the YDM approach, there was evidence that there was a growing adoption of the materials over the 12-month period since commencing in the program. In particular, the activities and lesson plans were most often used and many teachers appeared to use the examples as a starting point and then developed and shared lesson plans among each other. At the same time, half of the teachers perceived that the preparation time required to implement YDM in the classroom was an obstacle. Furthermore, there was an indication that some teachers perceived a lack of support from school leadership and a lack of information about the local Indigenous culture and community, and lack of suitable classroom resources. This aligns with principal feedback that suggested that the greatest inhibitors of the implementation of YDM were changing school priorities, followed by the time allocation for planning and implementation. HOW HAVE DIFFERENT GROUPS IN YOUR CLASS(ES) INCREASED THEIR ENGAGEMENT IN MATHEMATICS? 5 4 3.84 3.75 3.69 3.56 3.72 3.60 3.68 3.63 3.68 3.56 Mean Rating 3.40 3.46 3 2 1 Indigenous students Boys Girls Students in the upper ability range survey 1 Students in the lower ability range Other survey 2 Figure 3: PRIME Futures teacher perceptions of increased student engagement in mathematics Note: 1 = not at all; 2 = very little; 3 = somewhat; 4 = moderately; 5 = extensively THINKING ABOUT YOUR RECENT MATHEMATICS LESSONS, PLEASE GIVE YOUR OPINION ABOUT THE FOLLOWING STATEMENTS CONCERNING THE EXTENT OF YOUR STUDENTS' ENGAGEMENT IN THOSE LESSONS 5 4 3.88 3.87 4.06 4.12 4.07 3.85 Mean Rating 3.60 3.63 3.65 4.18 3.88 3.81 3.76 3.71 3.74 3.50 3.44 3.81 3.79 3.50 3 2 1 Students Students have a Students are Students are Students are willing to extended and empowered demonstrate positive attitude go’ 'have a ago' challenge with a vareity confidence towards learning ‘have themselves of strategies and selfmathematics esteem Students teach and learn from each other survey 1 Students are generally willing to persist with challenging tasks Students are Students show able to enthusiasm in verbalise their mathematics thoughts and lessons strategies mathematically Students' behaviour in class is positive survey 2 Figure 4: PRIME Futures teacher perceptions of student engagement in recent mathematics lessons Note: 1 = strongly disagree; 2 = disagree; 3 = undecided; 4 = agree; 5 = strongly agree 13 Aboriginal Summer School for Excellence in Technology and Science (ASSETS) Program elements, outputs and targets Indicators of success ASSETS supports high achieving Year 10 Aboriginal and Torres Strait Islander students with an interest in STEM to explore the study and career options available to them. This is achieved through the participation in a nine-day residential summer school at the end of Year 10, followed by a two‑year leadership and support program in Years 11 and 12. The three key components of the summer school are a rigorous academic program; a strong focus on personal development; and these two components being overlaid with an integrated cultural program. During the summer school, scientists share their research, guide and mentor students in an open inquiry, and discuss study and career possibilities. 2. Students choosing STEM at university Students learn the inquiry process within an Indigenous context using one of the I2S2 inquiries. They complete a group inquiry project and present their findings at the closing ceremony. Students experience a rich cultural environment allowing them to reflect on their cultural identity, overseen by a local cultural patron. Aboriginal and Torres Strait Islander mentors and leaders act as role models throughout the summer school. After the summer school, the leadership and support program works with students through Years 11 and 12 to develop leadership skills and access work placements, mentoring and tertiary education opportunities. Building on the single summer school in 2014 and three summer schools in 2015-16, three summer schools were successfully held in Townsville, Newcastle and Adelaide in 2016-17. These students were selected from 175 applicants. This was a substantial increase on the previous year’s applications of 119. One hundred and one students (n = 66 female, n = 35 male) attended the summer schools, exceeding the target of 100 students per year. 71 students completed the questionnaire before the beginning of the program (time 1) and 80 students completed the post-program questionnaire after the completion of the program (time 2). Students received and returned e-copies of the instrument by email. The leadership and support program for the 201617 cohort was initiated during these summer schools, connecting with students via Facebook, email and phone. The ASSETS Facebook group, for all three cohorts, has grown and has been used to engage with 199 students regarding opportunities and information. Of the alumni on this page, 180 (90 per cent) are classed as ‘active’ group members with 1,829 positive reactions to posts. 14 Indigenous STEM Education Project 1. Student choice of STEM subjects Years 11 and 12 3. Improved student engagement 4. Improved student results Research methods: student surveys (pre and post-summer schools, years 11 and 12, and post year 12) ASSETS has developed pre and post-summer school surveys, longitudinal surveys in Years 11 and 12, and a destination survey for the immediate years after Year 12. These monitor several aspects of student achievement and engagement. This report includes results from the pre and post survey for the 2016-17 ASSETS cohort. Some of the responses to these questions are in Appendix F. Key findings of the ASSETS program STUDENT CHOICE OF STEM SUBJECTS YEARS 11 AND 12 Students from the 2015-16 cohort were asked whether they had enrolled in STEM subjects in Year 11. The results are shown in Table 4. Table 4: Number of STEM subjects (Physics, Maths, Chemistry and Biology) selected for Year 11 NUMBER OF STEM SUBJECTS FREQUENCY PER CENT 0 1 1.6 1 11 18.0 2 21 34.4 3 20 32.8 4 8 13.1 Total 61 100.0 Note: Only includes data from students that responded (n = 24 non-responders) Sixty-one students responded to this question. Of these 61 students, 60 students indicated that they had enrolled in one to four STEM subjects, with 1 student indicating they had not enrolled in any STEM subjects. Of the 61 total respondents, 48 identified their STEM subject selection. The most common STEM subject that respondents enrolled in was maths, followed by biology, chemistry and physics. In response to the question of whether they intended to change their subject selection as a result of attending ASSETS, 7 of the 48 said they intended (or had already changed) to increase the number of STEM subjects they would study1. As outlined in Table 5, those who selected only one or two of these subjects before attending ASSETS were much more likely to change their subjects than students who had already chosen three or four. This indicates the value of ASSETS in better preparing some of this cohort of Aboriginal and Torres Strait Islander students for studying STEM at university through inspiring them to undertake more STEM studies in Year 11. Participants’ understanding of what is required to enter university in terms of pre-requisite subjects also increased. This was particularly the case in their understanding of the pre-requisite subjects needed for their desired study which increased from 50 per cent to 86 per cent. This increased understanding was accompanied by qualitative responses by 13 students (16 per cent) who identified that they intend to change their Year 11 subject choice. As one student said: “I would like to change my geography and psychology ATARs for chemistry and physics ATARs. So I have contacted my principal and he is changing them for me.” Similarly, “I’ve changed drama to maths and I changed physical education to chemistry and I’m also doing biology”. Other reported changes included replacing dance with biology, hospitality with chemistry, modern history with physics or chemistry. These positive results for the 2016-17 summer school are further supported by the results from the 2014-15 summer school cohort of 28 students. In 2017, this cohort was contacted to take part in a destination survey. However, with the leadership program in the early years of ASSETS being underdeveloped many contact details were no longer correct. Supplemented by contact via Facebook a total of eight responses were received – a relatively low response rate of 29 per cent. Three of the eight respondents (38 per cent) reported that they changed their subjects to include more science in Years 11 and 12 as a result of attending ASSETS. STUDENTS CHOOSING STEM AT UNIVERSITY The destination survey of the 2014-15 cohort found that seven of the eight respondents are studying at University and one is studying at TAFE. Six out of eight of these students are currently pursuing STEM or STEM-related studies. Three of the seven at university are studying science, one technology and two education, with one of the education students mentioning their passion for science education. The student stated: “Even though I want to be an educator, I want to teach science and give students the same love that I have for the subject.” The TAFE student is studying information technology. The other student is studying English and history. Two of the eight students did not complete Year 12. The complexity of some students’ pathways is captured by the technology student who did not complete Year 11 and 12: “After ASSETS I decided to study evolutionary developmental biology, I left Year 11 and went to TAFE to study chemistry. Part way through this course, however, I realised I was more passionate about gaming and graphic design.” Table 5: Number of STEM subjects selected for Year 11 by student intention to change subject selection due to the experience of ASSETS summer school NUMBER OF STEM SUBJECTS (PHYSICS, MATHS, CHEMISTRY AND BIOLOGY) SELECTED FOR YEAR 11 NUMBER OF STUDENTS INTENDING TO CHANGE TO STEM SUBJECTS FOR YEAR 11 YES NO TOTAL 0 0 1 1 1 3 5 8 2 3 10 13 3 1 18 19 4 0 7 7 Total 7 41 48 1A further six students also identified that they were planning to change their subject selection, however, they did not provide details of their subject choice. 15 IMPROVED STUDENT ENGAGEMENT There are also some promising indications that students were more committed to pursuing STEM education pathways and careers in the future as demonstrated by changes in students’ answers to questions about their understanding of STEM careers, requirements for university, and their aspirations for the future as shown in Table 6. There were positive increases in the number of students who agreed or strongly agreed that they intended to study STEM at university and to have a career in STEM. Having a good understanding of STEM careers increased from 65 per cent to 99 per cent (Table 6). All differences for each question were statistically significant using a paired t-test (p < .001), with most exhibiting relatively large effect sizes (between 0.41 and 0.76). While these changes are encouraging it is important to note the context of the intense experience of ASSETS. It will be important to monitor these intentions longitudinally to see if the change is sustained. Table 6: Number and percentage of students agreeing or strongly agreeing to questions about university and their future pre and post ASSETS NUMBER AGREE AND STRONGLY AGREE (PER CENT) MEAN (STANDARD DEVIATION) QUESTIONS PRE-SURVEY I intend to study a STEM field at university POSTSURVEY PRE-SURVEY 42 56 3.98 4.39 (63.6) (84.8) (0.94) (0.74) 32 54 3.79 4.35 (48.5) (81.8) (0.89) (0.77) I have a good understanding of STEM careers 43 65 3.77 4.50 (65.2) (98.5) (0.78) (0.53) 48 61 3.97 4.53 (72.7) (92.4) (1.12) (0.77) I know what the prerequisite subjects for what I want to study at university T-VALUE EFFECT SIZE (COHEN’S D) +0.41 -3.58*** 0.41 +0.56 -5.08*** 0.59 +0.73 -7.09*** 0.76 +0.56 -5.18*** 0.58 +0.61 -4.76*** 0.53 POSTSURVEY I intend to have a career in STEM I know what a prerequisite subject is DIFFERENCE 33 57 3.64 4.24 (50.0) (86.4) (1.15) (0.90) Note: n = 66 for all questions. Responses were on a scale from 1 to 5 (1 = strongly disagree; 2 = disagree; 3 = neither agree or disagree; 4 = agree; 5 = strongly agree). Significance levels based on Paired T-Test: *** p < .001. 16 Indigenous STEM Education Project IMPROVED STUDENT RESULTS The longitudinal research of ASSETS alumni is obtaining data on student results. These will be available for analysis in the next survey. Self-efficacy is one of a number of student attributes that can impact academic achievement in science (Frawley, Ober, Olcay, & Smith, 2017). The student surveys included questions regarding their self-efficacy in science (see Appendix G). The items are based on the PISA Self-Efficacy Scale and are designed to measure student’s beliefs that they can confidently explain scientific processes or phenomena. For one question (Recognise the science question that underlies a newspaper report on a health issue), the percentage of students that responded “I could do this easily” increased by 23.3 per cent, from 31.7 per cent at Time 1 to 55.0 per cent at Time 2. Similarly, another question (Identify the science question associated with the disposal of garbage) resulted in a 14.8 per cent increase in the number of students who felt “I could do this easily,” from 19.7 per cent at Time 1 to 34.4 per cent at Time 2. The reason for the change in these two questions could be due to the specific nature of activities covered by the ASSETS program. For example, health could have been the focus of the activities and/or project. Across all eight PISA Self Efficacy questions, the percentage of students who felt “I could do this easily” increased from 38.8 per cent to 45.4 per cent, and the percentage who felt “I would struggle to do this on my own” decreased from 17.2 per cent to 10.5 per cent. In the future, questions in the survey designed to test their capacity to solve actual problems would yield some evidence as to the veracity of students’ beliefs and provide a useful comparison of the relationship between their beliefs and their actual capability. In addition, it would provide evidence that the ASSETS program is having an impact on student cognitive knowledge and skills. 17 Bachelor of Science (Extended) program Program elements, outputs and targets Indicators of success The Bachelor of Science (Extended) program is designed to support the Aboriginal and Torres Strait Islander students wanting to pursue a program of study in STEM at the University of Melbourne. In supporting these students to meet the Bachelor of Science requirements within the program, it offers foundational subjects in mathematics and science. The additional components of the extended program are offered over an 18-month period, including a residency at the University. In assisting these students to meet the enrolment requirements, the objectives of the program are to enhance student engagement, retention and academic achievement. In meeting these objectives the program aims to establish a seamless transition into standard STEM awards at the university. Data from the university includes beginning enrolment, retention, and subject completions. 2. Retention 1. Number of students enrolled 3. Success rates (completed subjects) Research methods: program monitoring As a program with a small cohort, Bachelor of Science (Extended) program monitoring processes allow regular three monthly reporting from University of Melbourne to CSIRO. Reports provide detail of student enrolments, retention and subject completions. Key findings of the Bachelor of Science (Extended) program Enrolment, retention and subject completion rates for each of three cohorts are shown in Table 7. Table 7: Enrolment, retention and subject completions for the Bachelor of Science (Extended) program for 2015-2017 COHORT TOTAL MEAN 1 2 3 2015 2016 2017 Target enrolment – number 5 10 15 30 10 per year Offers of enrolment – number 12 6 11 29 9.7 per year Actual enrolment number 12 5 8 25 8.3 per year NA NA 9 NA2 12 Retention number 2016 (per cent) 2017 Subject completion rates (per cent) 9 (75%) 7 5 (58%)1 (100%) 2016 51% 74% NA 62.5% 2017 67% 48% 86% 67% 5 0 Unav.1 Total discontinued 5 1Three students enrolled in other STEM courses. 2As at September 2017, 7 out of 8 students (87.8 per cent) were still enrolled. 18 Indigenous STEM Education Project 6 (79%) 2.5 per year Number of students enrolled Success rates (completed subjects) These data show that over the three years, the program has attracted 25 students, which is less than the target of 30. Over the three years student numbers have fluctuated. The First Evaluation Report noted that in the years from 2009 – 2016 student enrolment numbers also fluctuated in the Bachelor of Arts (Extended), which this program is modelled upon (Tynan and Noon, 2017, p. 38). The reasons why this has been the case for the Bachelor of Science (Extended) program will be explored in a case study and reported on in future reports. In 2017, 67 per cent of Cohort 1 students had completed the first year’s subjects. Three of the five students in Cohort 2 made satisfactory progress and enrolled in the second year of their Science degree. The subject completion rate for Cohort 3 students was 86 per cent in 2017. Retention In 2017, the retention rate for Cohort 1 was 58 per cent (seven of twelve students) and 100 per cent of cohort 2. Three of the five students who did not complete the Bachelor of Science (Extended) in Cohort 1 enrolled in other courses that were STEM-related. Four of the remaining students in Cohort 1 have taken leave of absence in 2017. It is too early to assess the retention rate for Cohort 3. Although based on a small sample of students, and at a relatively early point in its implementation, the efficacy of the Bachelor of Science (Extended) can be broadly compared to general data on retention and completion rates, as shown in Table 8. These rates are not specific to Bachelor of Science (Extended) degrees for Aboriginal and Torres Strait Islander students, but do provide a context for understanding the findings. The forthcoming case study research on the Bachelor of Science (Extended) will investigate in detail the experiences and perceptions of a sample of Bachelor of Science (Extended) students. Table 8. Data on university completion and retention rates COMPLETION / RETENTION / SUCCESS RATES COHORT YEAR(S) SOURCE 69.9 completion rate All domestic natural and physical science university bachelor students in Australia 2010-15 Australian Government (2017) 79.2 completion rate Bachelor of Science students at University of Melbourne 2010-15 92.8 success rate All bachelor degree students at University of Melbourne 2015 40.5 completion rate All Aboriginal and Torres Strait Islander university bachelor students in Australia 2010-15 71.2 retention rate All Aboriginal and Torres Strait Islander university bachelor students University of Melbourne 2014 Australian Government (2016) Australian Government (2017) Universities Australia (2017) Note: Success rate is the proportion of units of study passed divided by all units of study attempted. 19 Science Pathways for Indigenous Communities Program elements Science Pathways for Indigenous Communities uses On‑Country projects as the context for learning science for primary and secondary school students in remote Aboriginal communities. Participating communities are located in the Northern Territory and Western Australia. The program supports schools to develop curriculum and education plans that integrate western science and Traditional Ecological Knowledge. These are built around On‑Country projects developed through strong community partnerships with Elders and, where they exist, ranger and mainstream research organisations. The Science Pathways for Indigenous Communities program is building upon the work initiated by Tangentyere Council in Central Australia, and is funded by Bank Australia and supported by CSIRO. The First Evaluation Report noted that, while Science Pathways for Indigenous Communities was behind in its deliverables due to delays in implementation, there was positive evidence of short-term outcomes being achieved. In 2017 the program was exceeding its deliverables having engaged with ten schools (seven in Western Australia and three in Northern Territory), 45 teachers, 61 Aboriginal and Torres Strait Islander teacher assistants and 547 Aboriginal and Torres Strait Islander students. Outcome indicators 1. Improved student attendance 2. Improved student engagement 3. Improved student results 4. Teachers teach more science 20 Indigenous STEM Education Project Research methods: program monitoring and jurisdictional administrative data It is proposed that the Science Pathways for Indigenous Communities program monitoring processes will include reviewing student school attendance, student engagement and the quantity of On‑Country and related classroom activities delivered by participating schools. The program evaluation will draw upon jurisdictional data and case studies to analyse achievement of the outcomes. Key findings of the Science Pathways for Indigenous Communities program This program element was one of the last to be established, and involved a co-development approach among CSIRO staff, school principals and teachers, and local stakeholders, including the development of contextual specific activities, and monitoring and evaluation processes. It is also the most challenging program to access data given availability constraints and the privacy and confidentiality requirements associated with remote and small communities. Ethical approval has been provided to undertake evaluation of the program, however jurisdictional approvals are still in the process of being sought. Both Western Australia and Northern Territory program coordinators report strong engagement in the program by teachers and students. They have developed on-country curriculum-linked teaching resources to be used in the field and in the classroom and provided teacher professional development and classroom modelling. This provides a rich context for the proposed case studies. Initial anecdotal reports from program coordinators indicate that there is strong support among the school community for the program and the outcomes are being achieved, however these reports need to be validated in future reports. Indigenous STEM Awards The Indigenous STEM Awards recognise, reward and celebrate the achievements of Aboriginal and Torres Strait Islander students and STEM professionals who are studying and working in the STEM field, as well as the integral role schools, teachers and mentors have in supporting Aboriginal and Torres Strait Islander students in pursuing STEM education and careers. The 2016 Awards were implemented on a pilot basis, and it is planned that the Awards will be gradually scaled up, and in 2018 will involve advertising all Award categories on a national basis. In 2016, 28 nominations were received, and 6 winners were announced in December 2016. The 2016 Award finalists and winners were selected by a range of professionals from CSIRO, BHP Billiton Foundation and other organisations, and presentations were made at a range of ceremonies across Australia. Details on the Award winners and other information can be found at: www.csiro.au/en/ Education/Programs/Indigenous-STEM/AWARDS Because the Indigenous STEM Awards element involves a relatively smaller level of investment (about seven per cent of the total budget of the Indigenous STEM Education Project), it was decided not to prioritise evaluation work for this element. CSIRO staff will continue to focus on continuous improvement of the operations of the Awards, for example through feedback from event participants and judges, and monitoring of the quality of applications. 21 Indigenous STEM Education context The purpose of this section is to place the evaluation findings within a broader context of STEM, Aboriginal and Torres Strait Islander peoples’ participation in STEM, and culture. • There are compounding effects on successful outcomes of university students belonging to multiple cohorts, such as Indigenous students from non-metropolitan and low socioeconomic areas (Edwards & McMillan, 2015). POLICY CONTEXT The most recent Report on Government Services (Productivity Commission, 2018) allows a finer-grained analysis of the educational achievements of Indigenous primary and secondary students in numeracy, science and information technology. While the overall picture suggests that levels of achievement are static, closer analysis by gender, age group, school level, and geolocation show some promising trends in the primary cohorts of larger cities and regional centres. However the data also suggests that educational programs need to be reviewed to address declining rates of science literacy (Connolly, 2015). Capability in STEM is currently seen as key to productivity, technological adaptation and research-based innovation (Commonwealth of Australia, 2018; Productivity Commission, 2018). The Australian Government is committed to increasing Australia’s STEM capacity and ensuring equitable opportunity for all Australians to quality education (Australian Government, 2015). ABORIGINAL AND TORRES STRAIT ISLANDER PEOPLES’ PARTICIPATION IN STEM Indigenous Australian involvement with STEM-related developments is growing, in part because there are more successful Aboriginal and Torres Strait Islander role models and there is a greater willingness in Australian universities and businesses to proactively seek out Aboriginal and Torres Strait Islander peoples to take up leadership in this important growth sector of the Australian economy (Ball, 2015; Matthews, 2015; Sarra, 2011). Some of the key research findings in relation to Aboriginal and Torres Strait Islander peoples’ STEM participation include: • Indigenous students do not perceive STEM subjects as being welcoming (McKinley, 2016) • Indigenous students’ interest in science is equal or greater than that of non-Indigenous students (McConney, Oliver, Woods-McConney, & Schibeci, 2011) • Engagement in science is most strongly associated with the extent to which students participate in science-related activities outside school (WoodsMcConney, Oliver, McConney, Maor, & Schibeci, 2013) • Some students and parents believe there is limited awareness shown by teachers of the linguistic, social and behavioural capital that is necessary for success in the classroom (Lewthwaite, Lloyd, & Boon, 2015). • Early intervention at the school level supports students’ increasing engagement, participation and ultimate career choice (Gale et al., 2010) • The factors that contribute to the success of Aboriginal and Torres Strait Islander students in universities include supported pathways from high school; enrolment assistance; smooth transitions into university life; and targeted cultural, social, academic and financial support (Lampert, Burnett, Patton, Lee Hong, & Anderson, 2013) 22 Indigenous STEM Education Project CULTURE AND STEM Culture has an impact on the confidence of Aboriginal and Torres Strait Islander peoples in taking an interest in STEM-related subjects at school and university (Ball, 2015). For example, self-confidence in science, mathematics, and engineering courses (STEM confidence) varies at the intersection of race/ethnicity (Litzler, Samuelson, & Lorah, 2014); specifically, ‘personal, environmental, and behavioural factors have different relationships with STEM confidence levels for different groups’ (p. 810). Another matter raised by Aboriginal and Torres Strait Islander peoples is that they are not convinced that there is a viable career pathway available to them in STEM careers (Paige, Hattam, Rigney, Osborne, & Morrison, 2016). The key concept of motivation has two aspects of importance when designing programs that attempt to take into account the factor of ‘Indigenous identity’: expectancy and value. Again, while no large quantitative studies have specifically addressed the motivation of Aboriginal and Torres Strait Islander students in STEM in Australia, research has examined expectancy theory in cross-cultural perspective, finding that ‘This work indicates that individuals’ beliefs about their ability relate to their performance, and their values have an impact on activity choice’ (Wigfield, Tonks & Eccles, 2004, p. 191). Potentially, a lack of expectancy of succeeding because of an anticipated individual or structural racism may overwhelm any perception of value that a program might develop in potential Indigenous scholars of STEM. Methodological considerations The purpose of this section is to outline some of the key methodological issues and challenges faced by the evaluation, and to briefly describe how the overall evaluation design will minimise or overcome them. CHALLENGE: MEASURING ENGAGEMENT, ATTENDANCE AND ACADEMIC ACHIEVEMENT Scholars have acknowledged that concepts such as engagement, attendance, and academic achievement can be multi-faceted, overlapping, and complex to theorise and measure (Baxter & Meyers, 2016; Boekaerts, 2016; Dunstan, Hewitt & Tomaszewski, 2017; Eccles, 2016; Lam et al., 2014; Prout, 2009). The challenge is partly definitional, particularly as it applies to the concept of engagement. Engagement is used too broadly, to ‘describe everything…’ (Azevedo, 2015, p. 84). There is also a lack of consistency with the engagement construct in terms of the nature and number of dimensions used, and distinguishing between indicators, contextual factors or outcome variables (Skinner, Kindermann, & Furrer, 2008; Lam et al., 2014). There is growing consensus that student engagement consists of four dimensions: • Behavioural: refers to student involvement in academic and class-based activities, the presence of positive conduct, and absence of disruptive behaviour. • Emotional: refers to students having positive emotional reactions to teachers, peers, and classroom activities. Valuing learning and having an interest in the learning content can also be indicators. • Cognitive: is defined in terms of self-regulated learning, the use of deep learning strategies, and the exertion of cognitive strategies to comprehend complex ideas. • Social: refers to the quality of peer and teacher interactions, and the willingness to build and maintain relationships while learning (Wang, Fredricks, Ye, Hofkens, & Linn, 2016). According to Briggs (2016) and of specific importance to this evaluation, ‘Evidence from Australian-based programmes to improve Indigenous attendance is not strong’ (p. 34). Similarly, there are few high-quality evaluations of programs aiming to increase attendance or retention of Aboriginal and Torres Strait Islander students (Purdie & Buckley, 2010). Important to the current evaluation, relatively little is known about programs that are conducted outside the formal primary and secondary education system. There is some limited evidence of successes in the context of boarding schools (Australian Indigenous Education Foundation, 2015), but this provides only general insights rather than comparative data by which to assess the current Project. One evaluation that is comparable is of an older version of ASSETS (Aldous, Barnes, & Clark, 2008), which is now part of the Indigenous STEM Education Project. One of the challenges of this type of educational research is to balance the objectives of the research with the constraints of each program. In attempting to measure the impact of the intervention on student engagement, for example, the ideal is to determine changes in the individual student based on the principles of objective measurement (i.e. define the concept of engagement, develop an instrument with appropriate items and ratings scale, control for bias in the data collection processes, convert student responses to interval measurement scale, analyse dataset using appropriate statistical techniques). In practice, however, data gathering sometimes relies on teachers to both implement the intervention and to evaluate its outcomes, introducing the possibility of bias in their assessment of student engagement. The potential for bias increases if teachers are asked to provide an overall impression of student engagement rather than rate the engagement of each individual student. In other words, the unit of measurement becomes a group of students (Aboriginal and Torres Strait Islander students) rather than the individual student. Any conclusions based on broad assessments of student behaviour and performance should be viewed cautiously and confirmed by triangulation with available jurisdictional data, which will form part of future evaluation reports. RESPONSE The respective elements in the Indigenous STEM Education Project have approached this challenge in different ways. Choices about how to measure student engagement, attendance, and academic achievement have been made in response to contextual differences between program elements, and time, scope and ethical constraints. The success indicators collected through program monitoring and evaluation processes have been (or will be) provided by teachers (PRIME Futures), students (ASSETS), or teachers and students (I2S2, Bachelor of Science (Extended), and Science Pathways). ASSETS in particular will draw on student self-assessment across a range of engagement indicators. Where student achievement data has been requested from teachers for program monitoring and evaluation purposes, such as I2S2, program staff have developed and provided rubrics (where relevant) to inform this process. In part, this is recognition of the limitations of standardised testing (see for example Gorur and Wu’s (2015) discussion of PISA). A key feature of the Indigenous STEM Education Project program elements is the use of innovative pedagogies. In particular, I2S2 has identified alternative assessment modalities that allow greater scope for students to show 23 their cognitive understanding of science which is not reliant on high levels of written [English] literacy. This aligns with contemporary research highlighting the importance of more culturally responsive assessment practices, including seeing knowledge and learning in terms of the relationship between an individual and their environment (Friesen and Ezeife, 2009), more closely matching learning goals and assessment tasks (Delaney et al., 2018), and integrating Indigenous knowledge into existing assessment techniques (Dupuis & Abrams, 2017). One example relevant to I2S2 is the use of tablet computers, with which “students have responded well to the tactile, flexible and more relational format available” (Australian Council for Education Research, 2016). In contrast, attendance is a relatively straightforward measure. With I2S2, teachers provide it as a percentage of I2S2 classes attended. In PRIME Futures a much larger ‘grain size’ is used in asking the principal to rate the effects of the program on community engagement and school attendance in general. In the future, we will seek to access jurisdictional data on attendance for I2S2, Science Pathways and PRIME Futures to triangulate data provided by teachers. CHALLENGE: BIAS AND ITS POTENTIAL EFFECT ON BOTH DATA COLLECTION AND INTERPRETATION The methods employed in this evaluation, as with much social research, are susceptible to biases that need to be recognised and addressed. Two specific biases relevant to this evaluation are: Observer biases (cultural bias and the Halo Effect) and Participant biases (the Hawthorne Effect, the Pygmalion Effect, and Gratuitous Concurrence). Cultural bias: The generic term ‘cultural bias’ refers to the phenomenon of interpreting and judging phenomena by standards inherent to one’s own culture. When program evaluation relies on the perceptions of non-Indigenous teachers and principals about Indigenous student behaviour (or attitudes) there could be general cultural bias involved if the principal or teacher knows very little about Indigenous lifeways or the history of colonisation of Australia. Halo effect: The potential impact of the Halo Effect is necessary to consider in research design because any evaluations need to be able to control for the bias of principals and teachers affecting judgements of the performance of Indigenous students. The Halo Effect proposes that if an individual, in this case, an Indigenous student, does well in one aspect then this is likely to trigger favourable reports on other aspects from the same principal or teacher (Abikoff, Courtney, Pelham, & Koplewicz, 1993; Foster & Ysseldyke, 1976). Rasmussen (2008, p. 458) explains that “a teacher who sees a well-behaved student might tend 24 Indigenous STEM Education Project to assume this student is also bright, diligent, and engaged before that teacher has objectively evaluated the student’s capacity in these areas. When these types of halo effects occur, they can affect students’ approval ratings in certain areas of functioning and can even affect students’ grades.” Hawthorne Effect: This bias is of importance for the current evaluation because the evaluations need to be able to control for the bias of self-reporting of students. The Hawthorne effect recognizes that individuals behave differently when aware that they are being studied (also known as the Experimenter Effect). Pygmalion Effect: The Pygmalion effect recognizes that individuals are influenced by the expectations that others have of them. This is important because if the student experiences cultural bias against them as an Indigenous person, for example, portrayed as low expectation, then they may respond by conforming to that expectation. Conversely, a perception of high expectation by a teacher or principal might encourage meeting that expectation. Gratuitous Concurrence: This is the inclination of a speaker to agree with a question put to them, regardless of whether they truly agree or have understood the question. This can occur especially with ‘yes/no’ questions, or when a questioner holds a position of power or authority (Smykowsky & Williams, 2011). Fryer-Smith (2002) notes, in a legal context, that an “Aboriginal person is likely to ‘gratuitously concur’ with a proposition put to him or her by a non-Aboriginal person, especially when the questioner is (or appears to be) in a position of authority” (p. 5:8). RESPONSE Cultural bias is and will be minimised in the evaluation through a cultural relativism approach, including avoiding a ‘deficit discourse’ (Fogarty, Lovell, Langenberg, & Heron, 2018). This means that the evaluation team has and will continue to be cognizant of their own cultural assumptions, which will periodically be checked by critical friends and peer reviewers, many of whom are Aboriginal and Torres Strait Islander people and/or specialists in Aboriginal and Torres Strait Islander cultures. In addition, the evaluation team will ensure participant voices are strongly represented in future reports through direct quotes and case studies, unfiltered by any pre-conceived evaluation framework. For the case study approach being developed for several elements of the program, a pilot study approach will be undertaken to test research instruments and ensure their cultural appropriateness with local stakeholders. The evaluation team have high levels of cultural competency, and will ensure participants and community are honoured, in part by taking a place-specific approach to the research (Howard, 2017). In terms of the Halo Effect, although there is evidence of relatively high correlations between teacher judgement and students’ actual achievement generally (Südkamp, Kaiser, & Möller, 2012), there is also evidence of stereotypes of Aboriginal students biasing teacher assessments (Dandy, Durkin, Barber, & Houghton, 2015). To address the Halo Effect, teacher perceptual data will be triangulated with other objective measures (administrative data), other participants (students, principals, school staff), and through collection of work samples in the Science Pathways for Indigenous Communities element case study. For this Second Evaluation Report, evaluation strategies and the program design were employed to minimise the Halo Effect, for example, in I2S2, teachers were instructed to refer to objective administrative data when completing assessments of students’ achievement. In addition, a standard I2S2 rubric was provided to teachers for making assessments of student achievement and engagement, which encouraged teachers to pay attention to assessment validity, and was intended to encourage and support teacher competence in making science assessments. An example of this was the use of ebooks on mobile devices in some assessments, which removed students’ English literacy as a barrier to demonstrating science understanding. The Hawthorne Effect (see McCambridge, Witton, and Elbourne (2014) for a detailed definition) will be minimised through using multiple methods and triangulating the findings, for example, using teacher perception, jurisdictional administrative data, focus groups, interviews, and direct observation to provide a comprehensive insight into the impact of the Indigenous STEM Education Project elements. elements. In addition, data from similar schools and students that were not involved in the Indigenous STEM Education Project will be collected to form a comparison group. The Pygmalion Effect will be minimised in the evaluation through case study research that directly asks teachers, principals and students about expectations, that is, whether there are high expectations of teachers and students. The potential Pygmalion Effect among teachers in the classroom, which may impact on the evaluation results, will be uncovered through similar methods, although unconscious biases will be difficult to detect using evaluation methodologies alone (Blank, Houkamau, & Kingi, 2016). It should be noted that the Pygmalion Effect in the classroom can have the benefit of improved student results through positive expectations (McLeod, 1995). Gratuitous Concurrence will be addressed through using local Aboriginal and Torres Strait Islander stakeholders to facilitate focus groups and interviews involving Indigenous students and teachers (as part of the case study research). Yes/no questions will be avoided and more indirect approaches to seeking information will be employed. Silence and pauses to allow interviewees to put their thoughts together will also be used, and an overall sympathetic understanding and attentive listening will be used. In general, the evaluation team will seek to elicit participants’ stories rather than simple responses to questions that may be prone to gratuitous concurrence. CHALLENGE: ATTRIBUTION AND CONTRIBUTION Measuring attribution is another key challenge in evaluation research (White, 2010). For the purposes of this evaluation, attribution is defined as ascribing observed changes to the intervention being studied. Because there are almost always many different factors at play, attribution is often thought of as the amount of the observed changes that can be attributed to the intervention. The related term of contribution refers to an approach of providing plausible evidence to reduce the uncertainty about the difference an intervention is making, rather than definitely proving sole attribution to the intervention (Mayne, 2012). Taking the ASSETS program as an example, given participants are high performing Year 10 students, the question may be asked: how many of them would pursue STEM studies if they did not attend ASSETS? Some preliminary data in this regard are the reflections from the eight students who responded to the destination survey in relation to the impact ASSETS had on their career direction based on a five-point Likert scale ranging from ‘Not at all’, ‘A little’, ‘Moderately’, ‘A lot’, and ‘Life-changing’. Four respondents rated the impact as a lot with one identifying it as life-changing. This latter respondent stated: “I would not have known about the possibility of a Cadetship, let alone applied for it without help from ASSETS.” The remaining three indicated that it had a little impact. While this is encouraging it is a very small sample. RESPONSE It will be important in the future collection of data for evaluation to be able to triangulate the statistical data with qualitative data drawn from case studies, including individual interviews and focus groups with the range of participants across all the programs. Continuing with ASSETS as an example, because the decision to pursue STEM studies is the result of a myriad of factors, it is more likely that the contribution of ASSETS to participant outcomes will be qualitatively understood through the case study research, rather than being able to quantitatively measure the amount of student outcomes directly attributable to ASSETS. The ASSETS Year 11 survey was implemented in 2017 with the 2016-17 cohort to see if the changes post summer school were maintained, which will add to the evidence of contribution. 25 Conclusions The Indigenous STEM Education Impact Statement identifies two levels of outcomes that are measured by indicators of success (level 1 outcomes) and indicators of pathways to success (level 2 outcomes). This report focuses on the reporting on the level 1 outcomes of five program elements (the sixth element, Indigenous STEM Awards, may be included in future reports). The data analysed in this report was collected in the 2016-17 year. I2S2 is having a substantial positive impact on participants, principally in terms of academic achievement and engagement. Low-achieving students in particular are benefitting from involvement in the inquiries, demonstrated by relatively large effect sizes. Although overall attendance declined, likely due to seasonal factors, the slightly reduced exposure to the inquiries did not impede the more important and relevant gains in achievement and engagement. Based on available evidence, PRIME Futures is increasing student engagement and improving learning and understanding of students, across upper and lower ability ranges. There has been growing adoption of the YDM materials by teachers, particularly the activities and lesson plans. The reported issues with the lack of teacher preparation time and a perceived lack of support from school leadership will need to be explored and addressed. Participation in ASSETS is increasing participants’ intention to study a STEM field in university, intention to have a career in STEM, and understanding of STEM careers. Based on a small sample from a destination survey, many former participants have gone on to study STEM subjects at university or TAFE, indicating the potential sustainable impact of the program. 26 Indigenous STEM Education Project The Bachelor of Science (Extended) program is attracting slightly below the target number of enrolments per year but retention rates are positive and compare favourably to national data. The factors affecting Aboriginal and Torres Strait Islanders students’ engagement and success in science degrees is complex and will need to be investigated further through the case study research. Science Pathways for Indigenous Communities has been fully established in 10 schools across Western Australia and the Northern Territory, with anecdotal reports suggesting strong support among the school community for the program. The inaugural Indigenous STEM Awards were successfully held in 2016, including a range of ceremonial events across Australia. This element has not been prioritised for evaluation but a continuous improvement process will be undertaken by the program team. This report begins to provide positive evidence that the goal of increased engagement with STEM and STEM achievement of Aboriginal and Torres Strait Islander students is being met across the programs. Analysis of the data also highlights the need to be cautious in interpreting the findings as these programs are still in the early phase of their implementation and that it will require ongoing monitoring and continual improvement of these programs in both content and in methodological design to ensure that they are meeting the STEM aspirations of Australia’s Aboriginal and Torres Strait Islander students and their families. References Abikoff, H., Courtney, M., Pelham, W.E., & Koplewicz, HS. (1993). Teachers’ ratings of disruptive behaviors: The influence of Halo Effects. Journal of Abnormal Child Psychology, 21(5), 519–33. Aldous, C., Barnes, A., & Clark, J. (2008). Engaging excellent Aboriginal students in science: An innovation in culturally - inclusive schooling. Teaching Science, 54(4), 35-39. Australian Council for Education Research. (2016). Assessment for Indigenous learners. Retrieved from https://rd.acer.org/article/assessment-for-indigenous-learners Australian Government. (2015). National innovation and science agenda report. Retrieved from www.industry.gov. au/national-innovation-and-science-agenda-report Australian Government. (2016). Higher education attrition, success and retention rate tables for the 2015 full year. Retrieved from https://docs.education.gov.au/ documents/2015-appendix-4-attrition-success-and-retention Australian Government. (2017). Completion rates of higher education students – cohort analysis, 2005-2015. Retrieved from https://docs.education.gov.au/node/46126 Australian Indigenous Education Foundation (AIEF). (2015). AIEF Compendium of best practice for achieving successful outcomes with Indigenous Students in Australian Boarding Schools. Surry Hills, New South Wales: AIEF. Azevedo, R. (2015). Defining and measuring engagement and learning in science: Conceptual, theoretical, methodological, and analytical Issues. Educational Psychologist, 50(1), 84-94. Ball, R. (2015). STEM the gap: Science belongs to us mob too. In G. Mills (Ed.) Australian Quarterly volume 86, Jan–Mar 2015, pp. 13–19. Baxter, L. P., & Meyers, N. M. (2016). Increasing urban Indigenous students’ attendance: Mitigating the influence of poverty through community partnership. Australian Journal of Education, 60(3), 211-228. Blank, A., Houkamau, C., & Kingi, H. (2016). Unconscious bias and education: A comparative study of Maori and African American students. Oranui Diversity Leadership. Retrieved from http://apo.org.au/node/65536 Boekaerts, M. (2016). Engagement as an inherent aspect of the learning process. Learning and Instruction, 43(2016), 76-83. Briggs, A. (2016). Links between senior high school Indigenous attendance, retention, and engagement: Observation at two urban high schools. Australian Journal of Indigenous Education, 46(1), 34-43. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.).Hillsdale, NJ: Lawrence Earlbaum Associates. Commonwealth of Australia. (2018). Closing the Gap Prime Minister’s Report 2018, Canberra: Department of the Prime Minister and Cabinet. Retrieved from https://closingthegap. pmc.gov.au/sites/default/files/ctg-report-2018.pdf Connolly, N. (2015). NAP Sample Assessment: Science Literacy Report 2015 – Public Report. Sydney: Australian Curriculum and Reporting Authority. Dandy, J., Durkin, K., Barber, B., & Houghton, S. (2015). Academic expectations of Australian students from Aboriginal, Asian and Anglo backgrounds: Perspectives of teachers, trainee-teachers and students. International Journal of Disability, Development and Education, 62(1), 60-82. Delaney, C., Doughney, L., Bandler, L., Harms, L., Andrews, S., Nicholson, P., Remedios, L., Edmondson, W., Kosta, L., & Ewen, S. (2018). Exploring learning goals and assessment approaches for Indigenous health education: A qualitative study in Australia and New Zealand. Higher Education, 75(2), 255-70. Dunstan, L., Hewitt, B., & Tomaszewski, W. (2017). Indigenous children’s affective engagement with school: The influence of socio-structural, subjective and relational factors. Australian Journal of Education, 61(3), 250-269. Dupuis, J., & Abrams, E. (2017). Student science achievement and the integration of Indigenous knowledge on standardized tests. Cultural Studies of Science Education, 12(3), 581-604. Eccles, J.S. (2016). Engagement: Where to next? Learning and Instruction, 43(2016), 71-75. Edwards, D., & McMillan, J. (2015). Completing university in a growing sector: Is equity an issue? Australian Council for Education Research Report. Retrieved from https://research.acer.edu.au/cgi/viewcontent. cgi?article=1045&context=higher_education Fogarty, W., Lovell, M., Langenberg, J., & Heron, M-J. (2018). Deficit discourse and strengths-based approaches. Changing the narrative of Aboriginal and Torres Strait Islander health and wellbeing. The Lowitja Institute, Melbourne. Retrieved from www.lowitja.org.au/sites/ default/files/docs/deficit-discourse-strengths-based.pdf Foster, G., & Ysseldyke, J. (1976). Expectancy and halo effects as a result of artificially induced teacher bias. Contemporary Educational Psychology, 1(1), 37–45. Frawley, J., Ober, R., Olcay, M., & Smith, J.A. (2017). Indigenous achievement in higher education and the role of self-efficacy: Rippling stories of success. National Centre 27 for Student Equity in Higher Education (NCSEHE), Perth: Curtin University. Retrieved from www.ncsehe.edu.au/wpcontent/uploads/2017/05/Frawley_Rippling-Stories.pdf Friesen, J. B., & Ezeife, A. N. (2009). Making science assessment culturally valid for Aboriginal students. Canadian Journal of Native Education, 32(2), 24-37. Fryer-Smith, S. (2002). Aboriginal Benchbook for Western Australian Courts. Melbourne: The Australian Institute of Judicial Administration Incorporated. Retrieved from https://aija.org.au/wp-content/uploads/2017/07/ Aboriginal-Benchbook-for-WA-Courts-2nd-Ed.pdf Gale, T., Hattam, R., Comber, B., Tranter, D., Bills, D., Sellar, S., & Parker, S. (2010). Interventions early in school as a means to improve higher education outcomes for disadvantaged students. Canberra, ACT.: Department of Education, Employment and Workplace Relations. Retrieved from http://dro.deakin.edu. au/eserv/DU:30040776/gale-interventions-synopsis-2010.pdf Gignac, G.E., & Szodorai, E. (2016). Effect size guidelines for individual differences researchers. Personality and Individual Differences, 102(2016), 74-8. Gorur, R., & Wu, M. (2015). Leaning too far? PISA, policy and Australia’s ‘top five’ ambitions. Discourse: Studies in the Cultural Politics of Education, 36(5), 647-64. Howard, E. (2017). Context and culture in evaluation: A case study of evaluation anthropology. Evaluation Journal of Australasia, 17(1), 30-38. Lam, S., Jimerson, S., Wong, B.P.H., Kikas, E., Shin, H., Veiga, F.H., Hatzichristou, C., Polychroni, F., Cefai, C., Negovan, V., Stanculescu, E., Yang, H., Liu, Y., Basnett, J., Duck, R., Farrell, P., Nelson, B., & Zollneritsch, J. (2014). Understanding and measuring student engagement in school: The results of an international study from 12 countries. School Psychology Quarterly, 29(2), 213-32. Lampert, J., Burnett, B., Patton, W., Lee Hong, A., & Anderson, J. (2013). Improving retention and graduation of Aboriginal and Torres Strait Islander students in initial teacher education programs. Journal of Australian Indigenous Issues, 16(4), 27-42. Lewthwaite, B. E., Lloyd, N., & Boon, H. J. (2015). Effective teaching practices reported by Aboriginal parents, students and their teachers: Comparisons and contrasts. Paper presented at the Australian Association of Research in Education (AARE), 29 Nov. - 2 Dec., University of Notre Dame, Fremantle, Western Australia. 28 Indigenous STEM Education Project Lipsey, M.W., Puzio, K., Yun, C., Hebert, M.A., Steinka-Fry, K., Cole, M.W., Roberts, M., Anthony, K.S., & Busick, M.D. (2012). Translating the statistical representation of the effects of education interventions into more readily interpretable forms. (NCSER 2013-3000).Washington, DC: National Center for Special Education Research, Institute of Education Services, U.S. Department of Education. Paper. Retrieved from https://files.eric.ed.gov/fulltext/ED537446.pdf Litzler, E., Samuelson, C., & Lorah, J. (2014). Breaking it down: Engineering student STEM confidence at the intersection of race/ethnicity and gender. Research in Higher Education, 55, 810–832. Matthews, C. (2015). Maths as storytelling, maths is beautiful. In K. Price (Ed.), Aboriginal and Torres Strait Islander Education: An introduction for the teaching profession (Second Ed.). (pp. 102-120) Melbourne: Cambridge University Press. Mayne, J. (2012). Contribution analysis: Coming of age? Evaluation, 18(3), 270-80. McCambridge, J., Witton, J., & Elbourne, E.R. (2014). Systematic review of the Hawthorne effect: New concepts are needed to study research participation effects. Journal of Clinical Epidemiology, 67(3), 267-277. McConney, A., Oliver, M., Woods-McConney, A., & Schibeci, R. (2011). Bridging the gap? A comparative, retrospective analysis of science literacy and interest in science for Indigenous and non-Indigenous Australian students. International Journal of Science Education, 33(14), 2017–35. McKinley, E. (2016). STEM and Indigenous students. Paper presented at the Australian Council for Education Research Conference, 7-9 August, Brisbane, Queensland. Retrieved from www.acer.org/files/2016_ Research_Conference_Proceedings_150.pdf McLeod, S. (1995). Pygmalion or Golem? Teacher affect and efficacy. College Composition and Communication, 46(3), 37-41. McRae, D., Ainsworth, G., Cumming, J., Hughes, P., Mackay, T., Price, K., Rowland, K., Warhurst, J., Woods, D., & Zbar, V. (2000). What works? Explorations in improving outcomes for Indigenous students. Commonwealth Department of Education, Training and Youth Affairs. Retrieved from https:// ncis.anu.edu.au/_lib/doc/visitors/what_works_2000.pdf Paige, K., Hattam, R., Rigney, L. I., Osborne, S., & Morrison, A. (2016). Strengthening Indigenous participation and practice in STEM: University initiatives for equity and excellence. Adelaide: University of South Australia. Productivity Commission. (2018). Report on Government Services. Canberra, ACT: Australian Government. Retrieved from www.pc.gov.au/research/ongoing/ report-on-government-services/2018/child-careeducation-and-training/school-education Tynan, M., & Noon, K. (2017). Canberra, A.C.T: CSIRO. Retrieved from www.csiro.au/~/media/Education-media/ Files/Indigeneous-STEM/Evaluation/Indigenous-STEMEducation-Project---First-Evaluation-Report.pdf?la=en&h ash=82A47EE87A5143E55F18D9E54091CCA066C375A0 Prout, S. (2009). Policy, practice and the ‘revolving classroom door’: Examining the relationship between Aboriginal spatiality and the mainstream education system. Australian Journal of Education, 53(1), 39-53. Universities Australia. (2017). Indigenous Strategy 2017-2020. Retrieved from www. universitiesaustralia.edu.au/ArticleDocuments/212/ Indigenous%20Strategy%20Web.pdf.aspx Purdie, N., & Buckley, S. (2010). Closing the gap: School attendance and retention of Indigenous students. Issues Paper No. 1. Retrieved from www.aihw.gov.au/uploadedFiles/ ClosingTheGap/Content/Publications/2010/ctg-ip01.pdf Wang, M.T., Fredricks, J.A., Ye, F., Hofkens, T.L., & Linn, J.S. (2016). The math and science engagement scales: Scale development, validation, and psychometric properties. Learning and Instruction, 43(2016), 16-26. QUT YuMi Deadly Centre. (2018). PRIME Futures Program. Retrieved from https://research.qut.edu. au/ydc/projects/prime-futures-program/ White, H. (2010). A contribution to current debates in impact evaluation. Evaluation, 16(2), 153-164. Rasmussen, K. (2008). Halo effect. In N. J. Salkind (Ed.) Encyclopaedia of Educational Psychology, Volume 1 (pp. 458-9), Sage Publications. Sarra, C. (2011). Transforming Indigenous education. In N. Purdie, G. Milgate and H. R. Bell (Eds.), Two-way teaching and learning: Towards culturally reflective and relevant education. Camberwell, Victoria: ACER. Skinner, E., Kindermann, T., & Furrer, C. (2008). A motivational perspective on engagement and disaffection: Conceptualization and assessment of children’s behavioral and emotional participation in academic activities in the classroom. Educational and Psychological Measurement, 69(3), 493-525. Wigfield, A., Tonks, S., & Eccles, J. (2004). Expectancy-value theory in cross-cultural perspective. In D. M. McInerney and S. Van Etten (Eds.) Big Theories Revisited, Volume 4, Research on Sociocultural Influences on Motivation and learning (pp. 165-198), Information Age Publishing. Woods-McConney, A., Oliver, M., McConney, A., Maor, D., & Schibeci, R. (2013). Science engagement and literacy: A retrospective analysis for Indigenous and non-Indigenous students in Aotearoa New Zealand and Australia. Research in Science Education, 43, 233–52. Smykowsky, A., & Williams, A. (2011). Improving communication with Indigenous people. Australian Nursing Journal, 19(3), 28. Spina, N. (2017). YuMi Deadly Centre Faculty of Education Impact Evaluation Report: Executive Summary. Brisbane: Queensland University of Technology, YuMi Deadly Centre. Retrieved from http://ydc.qut.edu.au/documents/ YDC_Impact_Evaluation_Report_March_2017.pdf Südkamp, A., Kaiser, J., & Möller, J. (2012). Accuracy of teachers’ judgments of students’ academic achievement: A meta-analysis. Journal of Educational Psychology, 104(3), 743–62. 29 30 Indigenous STEM Education Project 1 Appendix A: CSIRO response to EEGL recommendations EEGL RECOMMENDATIONS CSIRO RESPONSE I2S2 1. As teachers are required to both implement the program and assess students, the viability of other skill and knowledge assessment processes be explored to confirm current findings e.g. access jurisdictional data. Supported. Future evaluation reports will involve triangulation of data through in-depth case study methodologies and accessing jurisdictional administrative data on student achievement, engagement and attendance, where applicable. 2. Further data collection and analysis be undertaken to better understand variability in the effectiveness of the program by year level, geographical location and socio-economic disadvantage (ICSEA). Supported in principle. Future data collection and analyses will be undertaken to ascertain differences in effectiveness by year level, geographic location and other relevant factors. PRIME Futures 3. As current forms of data collection do not include direct student achievement, engagement or attendance data, it should be triangulated with other data sources that have established validity and reliability (e.g. NAPLAN) in future reports. Supported. Publicly available NAPLAN results at the school level will be used to report against the achievement success indicators. 4. Teachers be encouraged to use online platforms provided by YuMi Deadly Centre to share their lesson plans more broadly with teachers beyond their school. Supported. All teachers in PRIME Futures schools are being encouraged to share lesson plans with teachers in other participating schools using the online platform as a tool. 5. Continued monitoring by YuMi Deadly Centre of the teachers’ perceived lack of support from school leadership and/or information about local Indigenous resources. Supported. YuMi Deadly Centre will continue to monitor through the use of surveys, reflective journals and site visits. ASSETS 6. The survey of participants would benefit from development to ensure that it focuses on collecting data that is closely linked to Project outcomes. Supported. The ASSETS survey(s) will be reviewed to ensure questions are directly related to the outcomes of the program element, and more directly linked to students’ tangible experiences with ASSETS. Any items that are not providing useful and necessary information will be removed. 7. An instrument be developed to better reflect student variability in their STEM knowledge and skills to replace the PISA questions. Supported in principle. The ASSETS survey(s) will be reviewed to ensure it is appropriate for respondents’ knowledge and skills, and the appropriateness of the PISA questions will be re-examined. 8. An instrument be developed to better reflect student variability in their personal development, including aspects of leadership, knowledge of university and career options, the desirability of STEM, and the desirability of non-STEM courses. Supported in principle. The ASSETS survey(s) will be reviewed to ensure it is measuring different aspects of respondents’ personal development. 31 ASSETS 9. The Summer School activities could be rated to the extent that they meet the cognitive, social, cultural and aspirational needs of students. Supported in principle. The analysis of case study data (to be completed in late 2018) will include a broad investigation of these needs. In addition, program monitoring tools will be reviewed to ensure they are measuring different aspects of the activities, including whether they are meeting a range of student needs. 10. A gender and site analysis be undertaken to establish areas of best practice and areas that require improvement. Supported in principle. The analysis of case study data (to be completed in late 2018) and future program monitoring data will include investigation of gender and place. 11. Further ASSETS data collection and analysis should pay particular attention to attribution. Supported in principle. Existing and future program monitoring data will be subject to further analyses. The ASSETS case study work, to be finalised in late 2018, will provide additional insights into understanding the role ASSETS played in participant decisions and outcomes through in-depth interviews with a sample of participants. Bachelor of Science (Extended) 12. Research should focus on the reasons that students chose to study the University of Melbourne Bachelor of Science Extended, choose to remain in the course as well as the reasons that they depart. Such research should include data from both students and the teaching staff. Supported. The case-study methodology for Bachelor of Science (Extended) will include interview data from students and University staff, including on factors in relation to recruitment and retention. Science Pathways for Indigenous Communities program 13. An instrument be developed for students to measure their behavioural, emotional, cognitive and social dimensions of engagement with the materials. Supported in principle. The case study research will explore the dimensions of engagement through focus groups and interviews; however, a standalone measure will not be developed. 14. Teacher evaluation of attendance should include school records. Supported. Jurisdictional administrative data will be sought as part of the program monitoring and case study research. 15. Multimodal opportunities for assessments of student academic achievement should be used to supplement teacher evaluations of academic achievement. Supported in principle. In 2018, for the first time, schools in Western Australia have been mandated to assess science. Science Pathways supports schools to develop ‘Two-way Science’ assessment strategies that include Indigenous knowledge systems and ways of expression, however, school principals ultimately decide on the methods of assessing academic achievement. 16. An understanding of student backgrounds such as year level and gender would be useful to establish predictive relationships to determine effect sizes of the program on student outcomes. Supported in principle. Basic demographic information will be requested from jurisdictions as part of a request for administrative data. However, overall numbers of participants in Science Pathways is relatively small, which limits the ability to conduct multivariate analyses. 32 Indigenous STEM Education Project 2 Appendix B: first evaluation report key findings Key finding 1: Initial results of the implementation of the Indigenous STEM Education Project are positive. However, more substantial quantitative and qualitative evidence is required as to the extent, effectiveness and sustainability of outcomes. Key finding 2: Four common principles were identified as being central to all program elements: 1) being place-based; 2) having strong cultural engagement; 3) being strength-based and 4) being built on high expectations. A deeper understanding of how each program element enacts these principles should be a priority for future monitoring and evaluation as well as enriching the individual program element program logics and the overall Project Theory of Change. Key finding 4: The Indigenous STEM Education Project is well placed to contribute to building the evidence base for high expectation, strength-based Indigenous STEM programs, in particular a better understanding of the pathways for effective teacher professional development, school capacity building, and the integration of Indigenous-focused curriculum content. Key finding 5: The complex issues and policy environment in remote communities, including high turnover of nonIndigenous school staff, requires particular attention to the partnerships required at all levels of the system, from the community to schools to policy makers to maximise the chances of developing a sustainable model or models. Key finding 5: The complex issues and policy environment in remote communities, including high turnover of nonIndigenous school staff, requires particular attention to the partnerships required at all levels of the system, from the community to schools to policy makers to maximise the chances of developing a sustainable model or models. Key finding 6: The Project’s Theory of Change is well grounded in the literature and the individual program element logics have provided a robust conceptual base for assessing the implementation phase of the program elements, notwithstanding their diverse contexts and approaches. There are, however, revisions needed to more clearly identify the Project Theory of Change Impact Pathways and refine the individual program element logics as discussed in the relevant parts of Sections 3 and 4. 33 3 Appendix C: Indigenous STEM Education Project research framework The indicators developed to measure the two overarching outcome levels of the research framework, along with the high-level methods used to obtain them, are outlined in the table below. This table also identifies for which of these indicators data is currently available. It summarises the key results for the success outcomes (highlighted) that are discussed in detail in Section 2. To provide context for these outcomes the scope of the program element is referenced in the first column with a summary of outputs. These outputs are referenced against the targets set by the funding body. While there is some limited data available for CSIRO’s Pathways Outcomes, this will not be presented as it is not yet substantial enough to provide an accurate picture of the pathways. LEVEL 1 OUTCOMES LEVEL 2 OUTCOMES Program element Outputs (September 2017); Targets (September 2017)* Indicators of Success Data available to report Methods Summary of success Outcomes • 74 schools • 275 teachers • 2,895 Aboriginal and Torres Strait Islander students (Targets of 84 schools, 168 teachers and 2,100 Aboriginal and Torres Strait Islander students) 34 • Improved student results • Improved student engagement • Improved attendance • Students choosing STEM subjects Indigenous STEM Education Project Data available to report Methods • Increased teacher capacity (TPD, quality of resources) I2S2 • 15 clusters Indicators of Pathways to success • Yes • Yes • Yes • No • Program monitoring • Jurisdictional administrative data Benefits for all students include a closing of the gap Particularly beneficial for low performing students • Student engagement through inquiry and Indigenous context • Increased school capacity (high expectations and community engagement) • No • Teacher program monitoring data • No • Case study • Yes LEVEL 1 OUTCOMES PRIME Futures • 10 clusters • 70 schools • 70 principals • Improved student results** LEVEL 2 OUTCOMES • Yes • Surveys • Yes • Jurisdictional administrative data • Improved student engagement** • 315 teacher trainers (Targets of 60 schools, 120 teachers, and 1,500 Aboriginal and Torres Strait Islander students) ASSETS cohorts 2014-15 -28 students (1 summer school) 2015-16 – 98 students (3 summer schools) 2016-17 – 101 students (3 summer schools) • Student choice of STEM subjects yr. 11, 12 and university • Improved student engagement • Yes • Yes • No • Yes • Student surveys (pre and postsummer schools, Yr 11 and 12 and post Year 12). Increased student aspirations to attend university and undertake STEM studies • Parent survey Students choosing more STEM subjects in Year 11 • Improved student results • Students choosing STEM at university -- 2015: 12 -- 2016: 5 -- 2017: 8 • Retention: 20 students in Bachelor of Science (Extended) but 23 including other STEM degrees • Yes • Surveys • No • Teacher reflective journals • Innovative and effective instructional ideas • Sharing summit • Number of students enrolled • Retention • Success rates (completed subjects) Understanding the practices and implementation of the integrated cultural, academic and personal development program • Partial • Processes of individualised academic, personal and cultural support • Partial • Survey data • Case study Initial evidence of ASSETS students transitioning to university Majority engaged in STEM-related degrees. (Target 2016-17: 100 students, 3 summer schools) Bachelor of Science (Extended) • Enrolled: 25 students • Improved teacher capacity in mathematics Initial evidence of improved results and interest in STEM but working off a low base. • 7,090 Aboriginal and Torres Strait Islander students ASSETS Substantial gains in student engagement including for Aboriginal and Torres Strait Islander students • Yes • Yes • Yes • Program monitoring Retention rates are high in the early years of the program • Case study • No • No • Peer network • Indigenous contextualised curriculum (Targets: 2015: 5; 2016: 10; 2017: 15)   35 LEVEL 1 OUTCOMES Science Pathways • 10 schools • Improved student attendance • 45 teachers • Improved student engagement • 61 Indigenous staff/ assistant teachers and • No • No • No • No LEVEL 2 OUTCOMES • Program monitoring Not yet available • Jurisdictional administrative data • Improved student results (Target of 9 schools, 18 teachers, 18 teacher assistants and 225 students) • 15 finalists • Uptake of award categories • Yes • No • Quality of applications • Program monitoring Low initial uptake • Judging panel survey Seven winners across student, STEM professional, teacher and school categories (Target of 200 applications, 26 finalists) • No • Program monitoring • Case study • Aspiration, recognition and role models • Partial • No • Program monitoring • Explore integration with other program elements Local celebrations highly successful • Increasing uptake of ASSETS by I2S2, PRIME Futures and Science Pathways students • Strong representation of program elements in Awards • Yes • Yes • Yes • Program monitoring Low uptake across program elements overall Low uptake with wider CEdO programs • Schools and students in program elements taking up related CSIRO science programs (e.g. Creativity in Science and Technology: CREST) * Except ASSETS and Bachelor of Science (Extended) include all years and Awards 2016 ** Indirect measure (teacher report) only 36 • No • School capacity (cohesive communitybased curriculum, high expectations & partnerships) • 8 winners. Project as a whole (customised STEM pathway) • No • Teacher capacity through resources and teacher professional development • Teachers teach more science • 547 students, Indigenous STEM Awards (2016 data) • 28 entries • Student engagement through on-country activities and associated integrated curriculum Indigenous STEM Education Project Mechanisms and benefits of customised STEM pathway • No Review case studies for evidence of utilisation of additional program elements 4 Appendix D: PRIME Futures principals’ responses Responses by school principals to questions regarding the implementation and impact of the PRIME Futures Program* FOCUS How often were YDM methods used in … To what extent were these strategies effective when implementing YDM? OPTIONS T1 T2 CHANGE K-Year 3 3.0 3.5 + 0.5 Years 4-6 2.9 3.6 + 0.7 Years 7-9 3.5 3.3 - 0.2 Training teachers to support other teachers 3.3 3.9 + 0.6 Providing PD through time at staff meetings/pupil free days 3.0 3.2 + 0.2 Providing extra time for planning/ modelling of lessons 2.8 2.9 + 0.1 Providing peer support through classroom help and/or informal discussions 3.3 3.5 + 0.2 Providing teacher aides 2.1 2.6 + 0.5 3.5 3.7 + 0.2 The time required for planning. 3.2 3.4 + 0.2 The time required for implementation. 2.8 3.4 + 0.6 The loss/transfer of trained staff. 2.6 3.3 + 0.7 The lack of resources. 2.4 2.8 + 0.4 Conflicting priorities. 2.7 3.6 + 0.9 Isolation. 2.1 2.4 + 0.3 The cost of replacing teachers attending professional development. 3.1 3.2 + 0.1 The preparation of materials. 2.9 3.3 + 0.4 Enhanced their confidence in teaching mathematics. 3.8 3.8 Enhanced their mathematical knowledge. 3.5 3.6 Enhanced their pedagogical skills. 3.9 3.9 Enhanced their Indigenous knowledge. 3.5 3.7 + 0.2 Enhanced their expectations of students. 3.8 4.0 + 0.2 … parents and caregivers? 1.7 1.9 + 0.2 … the wider community? 1.7 1.9 + 0.2 … the Indigenous community? 1.8 2.3 + 0.5 Has PRIME Futures influenced your planning? How has each of the following factors hindered the effective implementation of YDM? How has the PRIME Futures improved the capacity of teachers of mathematics in your school with regard to … To what extent has the PRIME Futures involved… +0.1 37 FOCUS What has the school done to involve the Indigenous community and their perspectives? (Note: percentage of principals who reported that they would use this practice) How has PRIME Futures influenced the support of the local Indigenous community for the school’s activities? OPTIONS T1 T2 CHANGE Perspectives embedded in all learning areas. 80% 67% - 13% Used contextual resources in teaching and learning. 70% 67% - 3% Generated awareness of and embraced the identity of Indigenous students as well as students from minority groups. 85% 86% + 1% Explored Indigenous identity and culture. 70% 76% + 6% Created awareness of Indigenous history, culture and knowledge. 80% 76% - 4% Created a familiarity with and recognition of the local Indigenous history. 65% 57% - 8% Used the expertise and knowledge of Indigenous people in classroom/school. 65% 76% + 11% Invited the elders and traditional owners to a “Welcome to Country”. 80% 62% - 18% Attended Indigenous events/ forums in the local community. 90% 86% - 4% Contacted or visited local Indigenous organisations. 65% 71% + 6% Other. 5% 14% + 9% Increased school attendance 2.1 2.8 + 0.7 Support for the school’s mathematics program 2.6 3.2 + 0.6 Support for teaching Aboriginal and Torres Strait Islander knowledge to students 2.4 3.1 + 0.7 other 1.0 2.4 +1.4 Note: * n = 20 and 21 at Time 1 and Time 2, respectively. Values are mean responses, based on a 5-point scale, ranging from 1 = not at all, 2 = very little, 3 = somewhat, 4 = moderately, 5 = extensively, except for Question 7. Question 7 states the percentage of principals who reported that the school would use this practice. Questions and options are paraphrased from the original instrument. 38 Indigenous STEM Education Project 5 Appendix E: PRIME Futures teachers’ responses Responses by teachers to questions regarding the implementation and impact of the PRIME Futures program FOCUS How useful were different components of the PRIME Futures program?1 Please rate the content of the books that you have used since last survey.2 How have you applied the YDM approach in your mathematics classroom? 3 What obstacles have you encountered in using the YDM approach in your classroom? 3 OPTIONS T1 T2 CHANGE Workshops 3.5 3.4 - 0.1 School visits 3.0 2.9 - 0.1 YDM books 3.2 3.0 - 0.2 Online materials 3.1 3.0 - 0.1 Telephone/emails 2.7 2.7 - Teacher reflective journal 2.5 2.1 - 0.4 Usefulness 4.0 4.1 + 0.1 Relevance 3.9 4.1 + 0.2 Mathematical content 4.1 4.2 + 0.1 Mathematical pedagogy 4.0 4.1 + 0.1 Not tried 7% 3% - 4% One or more activities. 77% 74% - 3% Indigenous contexts 26% 26% 0 Pre/post tests 26% 34% + 8% Reduced use of textbooks/worksheets 40% 46% + 6% Used YDM lesson plan developed by someone else 22% 40% + 18% Used own YDM lesson plans 49% 59% + 10% Changed to RAMR in most/all lessons 18% 13% - 5% Other 8% 7% - 1% Little support from HoD/HoC/Principal 7% 18% + 11% Little support from colleagues 11% 7% -4% YDM approach requires a lot of preparation 49% 51% + 2% School’s mathematics program not suited to YDM methods 11% 15% + 4% I lack information about the local Indigenous culture and community 11% 22% + 11% I lack suitable classroom resources 28% 31% + 3% Other 25% 24% - 1% No obstacles, everything has gone well 25% 22% - 3% 39 FOCUS What student outcomes have you observed when using the YDM approach? 3 Thinking about your recent mathematics lessons, please give your opinion about the extent of your students’ engagement. 4 How have different groups in your class(es) increased their engagement in mathematics? 5 OPTIONS T1 T2 CHANGE Increased student engagement 81% 90% + 9% Improved learning/understanding 63% 70% + 7% Better test results 19% 24% + 5% More interest in STEM subjects/ pathways/careers 8% 13% + 5% No perceived change in outcomes 8% 7% - 1% Negative outcomes 1% 4% + 3% Students demonstrate behavioural engagement (i.e. willing to have a go/ positive behaviour/enthusiasm) 3.9 3.9 - Students demonstrate emotional engagement (i.e. confidence & selfesteem/persistence/positive attitude) 3.7 3.7 - Students demonstrate social engagement (i.e. willing to teach and learn from each other) 4.1 4.2 + 0.1 Students demonstrate cognitive engagement (i.e. students extend & challenge themselves/empowered with strategies/ verbalise mathematical thoughts) 3.7 3.7 - Indigenous students 3.8 3.4 - 0.4 Boys 3.8 3.6 - 0.2 Girls 3.7 3.6 - 0.1 Students in the upper ability range 3.7 3.6 - 0.1 Students in the lower ability range 3.7 3.7 - Note: n=73 at Time 1; n=68 at Time 2. 1Values are Mean responses, based on a 4-point scale, ranging from 1 = not useful, 2 = somewhat useful, 3 = generally useful, 4 = very useful. 2Values are Mean responses, based on a 5-point scale, ranging from 1 = poor, 2 = in need of attention, 3 = satisfactory, 4 = good, 5 = excellent. 3Teachers’ responses expressed as a percentage. Values are rounded. 4Teachers responded to a 5-point scale, ranging from 1 = strongly disagree, 2 = disagree, 3 = undecided, 4 = agree, 5 = strongly agree. Values are mean for items measuring the same type of engagement. 5Teachers responded to a 5-point scale, ranging from 1 = not at all, 2 = very little, 3 = somewhat, 4 = moderately, 5 = extensively. Questions and options are paraphrased from the original instrument. 40 Indigenous STEM Education Project 6 Appendix F: student responses to ASSETS survey questions at time 1 and time 2 (2016-17) SECTION Knowledge STEM Your identity About your future PRE (T1) MEAN POST (T2) MEAN DIFFERENCE PAIRED SAMPLE T-VALUE EFFECT SIZE (COHEN’S D) I feel a strong connection between science and my Aboriginal and Torres Strait Islander culture 3.61 4.09 +0.48 -5.26*** 0.62 I think STEM is important for my community 4.30 4.52 +0.21 -2.58* 0.31 I have a good understanding of STEM careers 3.77 4.50 +0.73 -7.09*** 0.76 I am proud of my Aboriginal and/ or Torres Strait Islander identity 4.68 4.77 +0.09 -1.62 0.19 I am aware of practices and beliefs that are relevant to my culture 3.71 4.03 +0.32 -2.64* 0.36 I am involved with practices and beliefs that are relevant to my culture 3.18 3.45 +0.27 -2.18* 0.28 I know where to learn more about my culture 3.59 4.09 +0.50 4.44*** -0.54 I am comfortable discussing my Aboriginal and/or Torres Strait Islander culture with others) 4.36 4.55 +0.18 -2.11* 0.27 In general I feel safe expressing my cultural identity at school 4.38 4.45 +0.08 -0.80 0.10 I have the potential to be a role model/ mentor for young Aboriginal and Torres Strait Islander people(s) 4.27 4.42 +0.15 -1.55 0.19 I am interested in going to university in the future 4.67 4.73 +0.06 -1.27 0.11 I am interested in going to TAFE in the future 2.76 2.76 No difference 0.0 0.00 I would like to work full time rather than study after leaving school 2.48 2.30 -0.18 1.43 0.17 I am unsure about what I want to do in the future 2.70 2.38 -0.32 2.00* 0.26 I intend to study a STEM field at university 3.98 4.39 +0.41 -3.58*** 0.40 I intend to have a career in STEM 3.79 4.35 +0.56 -5.08*** 0.59 I know what a prerequisite subject is 3.97 4.53 +0.56 -5.18*** 0.58 I know what the pre requisite subjects are for what I want to study at university 3.64 4.24 +0.61 -4.75*** 0.53 QUESTION 41 SECTION About your future QUESTION PRE (T1) MEAN POST (T2) MEAN DIFFERENCE PAIRED SAMPLE T-VALUE EFFECT SIZE (COHEN’S D) I know how to apply to university 3.36 3.88 +0.52 -4.27*** 0.50 My family or extended family can assist me with information about university studies 4.03 4.23 +0.20 -1.75 0.20 I know where/how to find information about a career that interests me 4.33 4.44 +0.11 -1.12 0.15 I am capable of being successful at university 4.38 4.42 +0.05 -0.52 0.07 I think I can afford to go to university if I choose to do so 3.50 3.58 +0.08 -0.65 0.07 Choosing a STEM career is more difficult than most other career options 3.17 3.08 -0.09 0.72 0.11 Note: n = 66 for all questions. * p < .05 ** p < .01 *** p < .001. Responses were on a 5-point Likert scale: 1 = strongly disagree; 2 = disagree; 3 = neither disagree nor agree; 4 = agree; 5 = strongly agree. 42 Indigenous STEM Education Project 7 Appendix G: student responses to PISA self efficacy questions at time 1 and time 2 (2016-17) PERCENTAGE OF STUDENT RESPONSES COULD DO EASILY (PER CENT) COULD DO WITH SOME EFFORT (PER CENT) WOULD STRUGGLE ON MY OWN (PER CENT) COULDN’T DO IT (PER CENT) Pre Post Pre Post Pre Post Pre Post (T1) (T2) (T1) (T2) (T1) (T2) (T1) (T2) Recognise the science question that underlies a newspaper report on a health issue. (n = 60) 31.7 55.0 60.0 38.3 8.3 6.7 0.0 0.0 Explain why earthquakes occur more frequently in some areas than in others. 68.9 62.3 23.0 34.4 8.2 1.6 0.0 1.6 Describe the role of antibiotics in the treatment of disease. 34.4 41.0 49.2 44.3 14.8 13.1 1.6 1.6 Identify the science question associated with the disposal of garbage. 19.7 34.4 45.9 50.8 32.8 9.8 1.6 4.9 Predict how changes to an environment will affect the survival of certain species. 63.9 63.9 32.8 31.1 3.3 4.9 0.0 0.0 Discuss how new evidence can lead you to change your understanding about the possibility of life on Mars. 32.8 42.6 39.3 41.0 24.6 14.8 3.3 1.6 Interpret the scientific information provided on the labelling of food items. 39.3 39.3 44.3 49.2 14.8 8.2 31.1 24.6 Identify the better of two explanations for the formation of acid rain. 19.7 24.6 44.3 47.5 31.1 24.6 4.9 3.3 TOTAL 38.8 45.4 42.3 42.1 17.2 10.5 1.6 2.1 Question Note: n = 61 unless otherwise indicated. 43 CONTACT US FOR FURTHER INFORMATION t 1300 363 400 +61 3 9545 2176 e csiroenquiries@csiro.au w www.csiro.au Christopher Banks Monitoring and Evaluation Program Manager t +61 7 3833 5999 e christopher.banks@csiro.au e indigenous-stem-evaluation@csiro.au w www.csiro.au/indigenous-education WE DO THE EXTRAORDINARY EVERY DAY We innovate for tomorrow and help improve today – for our customers, all Australians and the world. Our innovations contribute billions of dollars to the Australian economy every year. As the largest patent holder in the nation, our vast wealth of intellectual property has led to more than 150 spin‑off companies. With more than 5,000 experts and a burning desire to get things done, we are Australia’s catalyst for innovation. Therese Postma Project Director Indigenous STEM Education Project t +61 2 6276 6589 e therese.postma@csiro.au w www.csiro.au/indigenous-education Indigenous STEM Education Project w www.csiro.au/indigenous-education WE IMAGINE WE COLLABORATE WE INNOVATE B&M | 18-00425