Indigenous STEM Education Project Third Evaluation Report September 2014 – September 2018 Acknowledgements Acknowledgement of Country Aboriginal and Torres Strait Islander peoples have longstanding scientific-knowledge traditions. These traditions have developed knowledge about the world through: • observation, using all the senses; • prediction and hypothesis; • testing (trial and error); and • making generalisations within specific contexts. These scientific methods have been practised and transmitted from one generation to the next and contribute to particular ways of knowing the world that are unique as well as complementary to Western scientific knowledge. A deep respect for these Aboriginal and Torres Strait Islander cultural practices and knowledge underpin the philosophy and practice of the Indigenous STEM Education Project. Recognition of traditional contexts for technologies and concepts and their application in the past, present, and future—including supporting modern STEM career pathways for Aboriginal and Torres Strait Islander students—reaffirm the ingenuity and creativity of Aboriginal and Torres Strait Islander peoples’ knowledge systems. The Indigenous STEM Education Project team acknowledges the Traditional Owners of the lands on which this Project operates and their vibrant living cultures and knowledge systems. We pay our respects to Elders past and present, and we thank all community members who are providing the leadership to ensure meaningful and effective engagement with Aboriginal and Torres Strait Islander communities for the six distinct but complementary STEM education programs that make up this Project. CSIRO acknowledges that Aboriginal and Torres Strait Islander peoples make significant contributions to Australia in cultural, economic and scientific domains; for example, incorporating Indigenous knowledge of ecological and social systems is vital to the achievement of sustainable development. Other acknowledgements CSIRO wishes to acknowledge the invaluable contribution of Aboriginal and Torres Strait Islander scientists, educators and program leaders—without their knowledge and leadership the development and implementation of the Indigenous STEM Education Project would not have been possible. In addition, CSIRO acknowledges the advice and guidance of the case study reference group for this research. Their wisdom in relation to respectfully engaging Aboriginal and Torres Strait Islander students in the research was enormously helpful and is appreciated. The authors would like to thank the external peer reviewer and the CSIRO Planning, Performance and Impact unit for their peer review of this report. Their valuable suggestions also made a significant contribution. We also thank the program leaders for their constructive feedback. Finally, CSIRO acknowledges the governance of the Indigenous STEM Education Project Steering Committee and the contributions of former and current members of the program teams and evaluation team who led or supported the program monitoring and evaluation methodologies, data collection, and analysis. In particular, the contributions current and former team members Jessica Fidler, Caja Gilbert, Megan Ladbrook, Celia McNeilly, and Michael Tynan are gratefully acknowledged. Finally, the contributions of Zane Ma Rhea and Shane Phillipson from EEGL, who developed the recommendations in the Second Evaluation Report, are acknowledged. Contents Executive Summary......................................................................................................................................................................1 Introduction....................................................................................................................................................................................4 Inquiry for Indigenous Science Students (I2S2) program......................................................................................6 PRIME Futures...............................................................................................................................................................................11 Aboriginal Summer School for Excellence in Technology and Science (ASSETS)...................................16 Bachelor of Science (Extended) Program.....................................................................................................................19 Science Pathways for Indigenous Communities........................................................................................................23 Indigenous STEM Awards......................................................................................................................................................25 Sustainability................................................................................................................................................................................26 Discussion......................................................................................................................................................................................29 References....................................................................................................................................................................................34 APPENDIX A................................................................................................................................................................................37 APPENDIX B.................................................................................................................................................................................42 APPENDIX C...............................................................................................................................................................................43 Figures Figure 1 IS2S Aboriginal and Torres Strait Islander student achievement levels (Time and Time 2).....................................................................................................................................................................10 Figure 2 PRIME Futures teacher perceptions of student outcomes using the YuMi Deadly Maths approach...............................................................................................................................................13 Figure 3 PRIME Futures teacher perceptions of student engagement in recent mathematics lessons..................................................................................................................................................14 Figure 4 Mean difference in NAPLAN numeracy scores: 2009 to 2016........................................................15 Tables Table 1 Teacher assessments of attendance, engagement and academic achievement of Aboriginal and Torres Strait Islander students participating in the I2S2 program (2017)......................8 Table 2 Teacher assessments of attendance, engagement and academic achievement of non-Indigenous students participating in the I2S2 program (2017).................................................................9 Table 3 Summer school locations, applications and attendance (2014 to 2017)...............................................................................................................................................................................16 Table 4 Case Study participants and total program participants, by stakeholder group...................20 Table 5. Bachelor of Science (Extended) retention rates......................................................................................21 Table 6 Number of teachers, staff and students engaged in Science Pathways for Indigenous Communities................................................................................................................................................................................24 ii Indigenous STEM Education Project List of Acronyms ABC Australian Broadcasting Corporation ACARA Australian Curriculum, Assessment and Reporting Authority AIEO Aboriginal and Islander Education Officer ASSETS Aboriginal Summer School for Excellence in Technology and Science – one of the Project’s six programs CEdO CSIRO Education and Outreach CSIRO Commonwealth Scientific and Industrial Research Organisation EEGL Education Experts Group Limited – consultancy that independently reviewed the Second Evaluation report ICSEA Index of Community Socio-Educational Advantage: a scale which allows for fair and reasonable comparisons among schools with similar students I2S2 Inquiry for Indigenous Science Students – one of the Project’s six programs MSS Mean scale score NAPLAN National Assessment Program – Literacy and Numeracy NITV National Indigenous Television PISA Program for International Student Assessment PRIME Purposeful Rich Indigenous Mathematics Education QUT Queensland University of Technology RAMR Reality-Abstraction-Mathematics-Reflection SBS Special Broadcasting Service STEM Science, Technology, Engineering and Mathematics YDC YuMi Deadly Centre: A research centre at QUT that delivers the PRIME Futures program YDM YuMi Deadly Maths: a mathematics pedagogical framework Across all 8 states and territories 21,116 Aboriginal and/or Torres Strait Islander student contacts 793 teachers involved 159 schools participating I have had a really good response from some lower-level students and their confidence to get up there and have a go at something different has really increased which is fantastic! 1Department of Education (2015)’s Aboriginal Cultural Standard Framework outlines a continuum with four stages: Cultural awareness (emerging), Cultural understanding (developing), Cultural competence (capable), and Cultural Responsiveness (proficient). The Indigenous STEM Education Project and individual program Impact Pathways refer to ‘cultural competence’; however, in terms of the four stages, the Project is aiming for Cultural Responsiveness, the highest level. I believe ASSETS has changed the way I view myself. I have never felt so comfortable in expressing who I truly was until I came on this camp. The camp was so supportive in my goals. This was the first time someone had ever told me to just go straight to medicine. Everyone has always told me to aim lower, do an undergraduate degree. So being told that you are worth it and you can achieve anything you put your mind to, was the best experience I have had.” (ASSETS participant) 2PRIME Futures primarily draws on teacher and principal feedback but does compile NAPLAN data from PRIME Futures and comparison schools. 3As previously mentioned, ‘culturally compentent’ was included in the original Impact Pathway. 4 For all findings comparing 2016 and 2017 results, some of the same students will be in both samples (for example, some Year 8 students in 2016 will be in Year 9 in 2017); however, each sample was considered separately as there were a relatively small number of students whose data could be analysed at an individual level across years. 5Across Australia, in government and non-government schools, Aboriginal and Torres Strait Islander students in Years 1 to 10 had attendance rates of: 84 per cent in 2014; 84 per cent in 2015; 83 per cent in 2016; 83 per cent in 2017; and 82 per cent in 2018 (Productivity Commission, 2019). However, attendance rates differ by remotness. In 2018, across all schools, attendance rates for Aboriginal and Torres Strait Islander students were: 85 per cent in Major Cities; 86 per cent in Inner Regional; 83 per cent in Outer Regional; 76 per cent in Remote; and 63 per cent in Very Remote (Productivity Commission, 2019). Attendance rates are the number of actual full-time equivalent student-days attended by full-time students in Years 1 to 10 as a percentage of the total number of possible student- days attended over the period. 6The number of hours spent on inquiries in a term can range from a minimum of approximately 10 hours to a maximum of approximately 60 hours. 7With currently available data, it is not possible to make evidence-based conclusions on the reasons for increased overall student engagement among low-achieving students compared to the previous year. However, the forthcoming I2S2 Evaluation Case Study Report will have access to additional data and will provide more explanatory commentary. % % % % % % % % % % % % % % % % % % % % % EDCBAPer cent of students achieving at each levelAchievement levelABORIGINAL AND TORRES STRAIT ISLANDER STUDENT ACHIEVEMENT Time Time  8 Achievement by Aboriginal and Torres Strait Islander students across Australia has been improving in some STEM areas but has been more variable in others. For example, the proportion of students achieving at or above the national minimum standard in numeracy has increased across all measured year levels from 2008 to 2017: Year 3 was 78.6 per cent to 82.2%; Year 5 was 69.2 per cent to 80.2 per cent; Year 7 was 78.6 per cent to 79.9 per cent; and Year 9 was 72.5 per cent to 84.0 per cent (Productivity Commission, 2019). In science, results have been variable. The proportion of Year 6 Aboriginal and Torres Strait Islander students at or above proficient standard in science achievement performance was: 25.5 per cent (2006), 19.6 per cent (2009), 20.1 per cent (2012) and 23.4 per cent (2015) (Productvity Commission, 2019). In Information and Communication Technology, Aboriginal and Torres Strait Islander student achievement is also variable. The proportion of students attaining the proficient standard was: Year 6: 30 per cent (2005), 24 per cent (2008), 31 per cent (2011), 22 per cent (2014), and 24 per cent (2017); and Year 10: 35 per cent (2005), 32 per cent (2008), 36 per cent (2011), 20 per cent (2014), and 24 per cent (2017) (Productivity Commission, 2019). These data are estimates only and have relatively large confidence intervals; see Productivity Commission (2019) for more details. 9 The YuMi Deadly Centre is closing at the end of 2019. 10Teachers are asked to make assessments of academic achievement about all relevant students that they teach the YDM to (i.e., not individual classes or students). 11Yumi Deadly Centre assessed that it was not feasible to obtain the jurisdictional approvals required to obtain class-level administrative data. 121 = not at all; 2 = very little; 3 = somewhat; 4 = moderately; 5 = extensively. Students have apositive attitudetowards learningmathematics Students are  willing  to ‘have a go’ Students arechallenge themselves Students areempowered with avariety of strategies Students teach  and learn from  each otherStudents aregenerally willing  to persist with  challenging tasks Students  are able to  verbalise  their thoughtsand strategies  mathematically Students showenthusiasm in  mathematics  lessons Students’ behaviourin class is positive Mean RatingTHINKING ABOUT YOUR RECENT MATHEMATICS LESSONS, PLEASE GIVE YOUR OPINION ABOUT THE FOLLOWINGSTATEMENTS CONCERNING THE EXTENT OF YOUR STUDENTS' ENGAGEMENT IN THOSE LESSONS 13 Secondary school results have not been included in this report because data from only six schools and from 2015-2017 was available and/or relevant. 14The response rates for web-based surveys among students are commonly below 20 per cent (Van Mol, 2017). 15 Including Elders, Traditional Owners, and Ranger and other groups. 16Place-based education emphasises the significance of local people and Elders, local knowledge, and local places as central to teaching and learning, and often includes ‘in the field’ experiences and curricula that is responsive to the interests and aspirations of the community (Green, 2016; McInerney et al., 2011). 17Fully implementing and actively immersed in YDM refers to schools that have implemented YDM pedagogy across the entire school, which is contrasted with YDM-trained schools, in which nominated teachers have completed a minimum of a one-year YDM training program. Impact Pathway for Indigenous CSIRO, BHPF, QUT, UoM ISEP Team, Project partners Technical Experts • BHP Foundation funding $28m/5 years • 30-year CSIRO BHPF relationship in science education • Indigenous leadership • Relationships with Indigenous communities • Experienced Indigenous and non-Indigenous staff • CSIRO experience in science inquiry education – esp. CREST, Land and Learning Program • CSIRO national infrastructure and university partnerships • Partner/Stakeholder expertise (e.g. YuMiDeadly Centre, Tangentyere Land and Learning, UoM Bachelor of Science & Arts extended INPUTS What we invest ACTIVITIES What we do • Innovative curriculum, pedagogy and TPD Development of innovative, place-based, high-expectations Indigenous contextualised curriculum, pedagogy, support resources for schools/universities and associated TPD training • High expectation extra-curricular opportunities and support Development of high-expectations extra-curricular opportunities including summer schools, work placements, awardsand leadership programs with personalised support • Local and strategic engagement Student recruitment and engagement of key stakeholders (esp. schools, universities,CSIRO sites, Aboriginal organisations, Elders and patrons to support the delivery and sustainability of the above) • Management, monitoring and evaluation Deployment of project management, monitoring and evaluation methodologies to support delivery and sustainability of the above Assumptions • Indigenous leadership is essential • Importance of high expectations, culture and personalised support • Importance of working at the cultural interface of two-way science • Importance of Indigenous curriculum contexts and building teacher and school capacity • Importance of building strong relationships with community • Rigorous evaluation is required to demonstrate program effectiveness External factors • National/Global - Availability of STEM jobs, global and local economy, political environment • Jurisdictions/School – culture of low expectations, teacher quality, relationship with Indigenous community • Student – level of family support and understanding of tertiary education context • CSIRO - Staff recruitment and training essential (need external expertise) Participation OUTPUTS Our deliverables Teacher completion of high expectation Indigenous context and pedagogy focused TPD courses and on the job training including train the trainer Innovative, place based, Indigenous contextualised and/ or led STEM curriculum (inquiry based) delivered in schools and university and documented in school plans OUTCOMES The uptake, adoption or consumption of or work Schools/students engaged in extra- curricular supports and alternate pathways e.g. summer schools, awards, leadership and support programs, BScExt Partnerships with schools, jurisdictions, universities, mentors, and other key stakeholders Project Steering Committee (governance), PM tools and databases, skilled staff, M&E frameworks, methods, data, reports and publications IMPACTS Benefits to economy, environment and society Increased student engagement, attendance, results and recognition Increased student aspiration, sense of value, cultural identity and school belonging Increased parental, family and community, engagement and recognition of role models Increased teacher capacity in: inquiry; place based, hands on curriculum development; and delivery in an Indigenous context/ Indigenous led More Indigenous (and non-Indigenous) students pursuing STEM education pathways, enrolment in STEM years 10-12 and university, STEM careers and leadership Schools are culturally competent in delivering two-way science in partnership with Elders, families and communities and flow on benefits to broader curriculum/ teaching Best practice in high expectation science inquiry and maths education programs and TPD identified Jurisdictions, CSIRO, universities and partners scaling up Indigenous knowledge and culture valued: complementarity to western science and maths demonstrated Greater under-standing and care of environment Social cohesion/ reconciliation More, higher quality and greater workforce diversity of STEM professionals Schools, students and families increasing high expectations focus contribute to new cultural norm of Indigenous students attending university and having high STEM engagement Increased innovation and workplace productivity M&E input to PE CQI processes