STEM Links Insights: 2023 Generation STEM Kate Cherry, Impact and Evaluation, CSIRO Education and Outreach January 2024 Generation STEM Links 2023 Snapshot of Achievements In 2023, 61 new placements were commenced. A placement is completed when a student undertakes 200 hours of paid work over a maximum period of 6 months. Total placements completed since program establishment in 2022 is 55, with 40 of these completed in 2023. The Generation STEM Links program secured 100 new placement requests in 2023, achieving 125% of the target Being exposed to the various challenges has helped improve my skills vastly. I had a chance to apply so much of what I've learned through formal and self- education. I was also given the time and agency to learn what I thought would be useful for the role, which I greatly value as it gives me more confidence tackling further challenges. Student intern 100% of supervisors surveyed had their expectations met and 78% of these supervisors stated the internship exceeded or greatly exceeded their expectations 100 per cent of students surveyed were satisfied with the interview and matching process 81% of students surveyed had their expectations met or exceeded by their internship. 100% of supervisors surveyed would recommend Generation STEM Links to other businesses. The candidates we were offered were all good but the candidate we have chosen is an exceptional fit for our business. I doubt we would have been able to achieve this without the aid of the CSIRO. Industry partner 67% of industry partners that have completed their first placement have requested additional placements 83% of students are offered ongoing employment in their place of internship 92% of student interns are from underrepresented groups in STEM Megaphone1 outline Ribbon outline Cycle with people outline Handshake outline Generation STEM Links 2023 Insights Snapshot of Learnings What’s working in the Generation STEM Links program and how this contributes towards emerging program outcomes. When you do a particular task or assessment [at university], they give you feedback and they mark it and then you move on, but in the workplace, you don't move on until its 100% done. Being in an environment like that helps you improve as you go along. Yeah, it forces you to be better. Student intern What’s Working A high quality and targeted recruitment, matching and onboarding phase is valued by students and industry partners and sets the placement up for success. A positive workplace culture enables a positive on-the-job learning experience, supporting improvement in STEM skills and general workplace skills. Experiencing a STEM job and building professional relationships can solidify student connection with the STEM sector and build a stronger understanding of their STEM areas of interest. Generation STEM Links’ sector collaboration activities are supporting diverse stakeholders to collaborate on relevant local issues and design actionable solutions. Emerging Outcomes Industry partners are building their organisational capability and capacity. Industry partners are returning for multiple student interns, and some are streamlining their recruitment and growing their business in the process. Eighty-three per cent of students are offered ongoing employment with their host workplace. The Generation STEM Links program is increasing access for underrepresented groups to engage in paid STEM internships. Chevron arrows with solid fill Chevron arrows with solid fill Chevron arrows with solid fill Chevron arrows with solid fill Chevron arrows with solid fill Chevron arrows with solid fill Chevron arrows with solid fill Chevron arrows with solid fill Acknowledgements The CSIRO Impact & Evaluation team acknowledges the Traditional Owners of the lands with whom this project is collaborating their vibrant living cultures and knowledge systems and acknowledge the Countries on which the evaluation work took place. We pay our respects to Elders past and present and thank all community members who provide the leadership to ensure meaningful and effective engagement with Aboriginal and Torres Strait Islander communities for the Deadly in Generation STEM program. We specifically acknowledge the Traditional Owners of the lands on which the program operated and the evaluation work was conducted, specifically the Wiradjuri people and the Gadigal People of the Eora Nation. CSIRO wishes to acknowledge the significant knowledge and leadership of Aboriginal and/or Torres Strait Islander scientists, educators, and program team members that have made the development and implementation of the Deadly in Generation STEM program possible. The author would like to thank the two peer reviewers, and the former and current members of the evaluation team who supported the program monitoring and evaluation methodologies, data collection, and analyses. Finally, and most importantly, the students, teachers, and parents who helped organise and took part in the evaluation research are gratefully acknowledged. The time that was given to the evaluation team, and the knowledge that was shared, made this insights report possible. Introduction As part of the ongoing monitoring and evaluation of Generation STEM, this summary report was developed by the CSIRO Education and Outreach Impact and Evaluation Team and aims to provide a 2023 evidence snapshot of implementation learnings and emerging student and industry outcomes of Generation STEM Links. Findings in this report are based on evaluation data collected on Generation STEM Links between January to 11 December 2023, and calendar year operational data, including: • survey data gathered at the end of 2023, representing the views and experiences of 17 students and 17 supervisors. • survey data gathered between November 2022 and November 2023, representing the views of 25 sector workshop participants from state government, industry, higher education, VET, and research and public sectors. • interview data gathered throughout 2023, with 4 students and 7 supervisors or executive staff from 5 industry partners. • Program activity and operational data from 2022 and 2023. In this report, the findings are presented in themes demonstrating how program design and implementation elements contribute to benefits for program participants, and the extent to which the outcomes for both groups are closely linked. This snapshot of 2023 learnings provides additional insights to the Generation STEM evaluation report 2019-20211. More detailed findings about what works, in what circumstances, and for whom, will form part of the final evaluation report along with a full analysis of the extent to which the program has achieved its expected outcomes. 1 Banks C, O’Brien M, and Miller K (2023) Generation STEM evaluation report 2019–22: Generation STEM. Canberra, Australia: CSIRO What is the Generation STEM Links program? Generation STEM Links is a paid internship program in NSW that pairs industry organisations with students in their later stage of study in STEM tertiary-level qualifications. The internship placements provide opportunities for students and industry organisations to connect, learn and innovate. The program is co- delivered by two CSIRO teams, CSIRO Education and Outreach and SME Connect, combining an education focus with an industry and enterprise focus respectively. The internship structure is flexible and runs for a minimum of 200 hours; 5-6 weeks full time to a maximum of 25 weeks part time. As part of the internship, students are employed by the business to participate in real-world projects and are paid a minimum of $25ph + super. Industry partners receive a $2,500 grant for each student after completion of the internship as a contribution to cover the student’s wage. The program design prioritises a significant focus on the recruitment and matching of industry partners and student interns to achieve a successful placement. Ongoing monitoring and support of each placement is provided. The program also delivers separate collaborative workshop activities involving industry, the tertiary sector, and related stakeholders in targeted regions to support local solutions aimed at strengthening the STEM- qualified workforce. Key program insights Generation STEM Links program activity in 2023 In 2023, the program exceeded its target of attracting 80 new industry placement requests by 25 per cent, receiving a total of 100 new placement requests. During the 2023 calendar year, 61 placements were commenced. In total, 55 placements have been completed since the program’s early implementation phase in 2022, with 40 of these finalised in 2023. In 2023, CSIRO published the Generation STEM evaluation report 2019 – 2022, assessing the extent to which Generation STEM Links had met its intended outcomes. The report determined that the program was meeting its student skill, career pathway and commitment to STEM outcomes, however insufficient evidence was available to reliably measure other program outcomes. While this insights report is not a comprehensive assessment against program outcomes, evidence from this report indicates positive advancement towards program outcomes in 2023. Emerging learnings for students and industry partners in 2023 A comprehensive and targeted recruitment, matching and onboarding phase is valued by participants and sets students and employers up for success. The initial stage of Generation STEM Links includes a significant amount of work undertaken by program staff to engage with high quality industry partners and match their needs and workplace culture with prospective STEM student interns. This recruitment and matching process is often supplemented with targeted advice and support to both the industry partner and the student to facilitate a smooth onboarding and transition into the internship and the offer of ongoing support as needed. 100 per cent of students surveyed (n=18) reported a high level of satisfaction with the interview and matching process. In their interviews, multiple industry partners spoke about the value of this support to efficiently connect with a suitable student intern. Several industry partners called it “seamless” and another commented “…I just felt like everything was done for us. We didn’t have to do much, we just had to focus on the ground things like safety induction…”. Several students and industry partners commented that they would value additional early support and more regular check-ins during their placement. This was particularly relevant where barriers to a successful placement emerged. …because everyone gets screened…it takes the HR component out of my day and then I can just get on with what I'm doing and get presented [with] candidates who might be suited for the role. – Industry partner Several other industry partners described this initial recruitment and onboarding stage as a useful learning process to better understand how their work can be undertaken by interns and how to best support interns in the workplace. For example, one employer reported that by working with the Generation STEM Links program, “we actually learn how to specifically list the job advertisement that really is matching what we’re offering”. Another industry partner acknowledged that the additional work up front to support the internship set them up for longer term success in the workplace. A positive workplacecultureenablesa positiveon-the-joblearning experience, supporting improvementin both STEM skillsand general workplaceskills. I have been lucky in my placementto havegreatmentorsthat have been able to find where Ican be ofbest valueto the companywhile alsogrowing as an engineer and picking upnewskills.I am encouraged to help the engineerswith maintenance and upgrades and feel that Iam avalued member oftheteam.Student intern When students reporteda positive workplace experience or that their workplacemet orexceededtheirexpectations(81 per cent),theycommonly cited the following reasons:great mentors, feelingvalued bytheir team and seeing thevaluethey bring tothe company in tangibleways, theopportunity to learn skillsnot covered in universityand topractice theapplication ofacademic theory to improve their skills. The top threemostvaluedinternship interactions forstudents surveyed(n=16) were ‘receiving constructive feedbackabouttheir performance’, ratedmoderatelyor highlyvalued by 100per centof students.This was I learnt a lot about the closely followed by‘participating in onsite training’(91 percent) and industry, how to manage my work and keep others‘informal mentoringfrom their supervisor’(85 per cent). updated on my progress, Whenlearningabout ‘what works’fromthe fewstudentsthat reporteda how to switch priorities lesspositive experience in theworkplace,it is important to ensure the between tasks when programcontinues tomonitorand supportthe followingindustry partner needed and how to responsibilities: research topics I wasunfamiliar with in order to •appropriate workplacesupervisionand direction for the student, effectively complete my especially atthe beginningof the internship; work.Student intern •up-to-date training,safety andstandardoperating procedures intheworkplace, and •enough work forthe student to undertaketo complete regular shifts. Figure 1 below showsthe significant numberof positive student responses,agreeingthattheir internshipprovided abeneficialexperience. Figure1Student feedback ontheir internship experience Studentfeedbackon their internship experience (n=16) The work I did contributed to the wider teamI was given work opportunities that were relevant to my fieldof studyI felt I was given appropriate levels of responsibilityI was able to apply my theoretical knowledge in a work-basedsettingI was given meaningful work, tasks and activities I had adequate access and support from my supervisor 024681012141618 Strongly disagreeDisagreeAgreeStronglyUnsure / agreeHard to say Insights: 2023| 7 The majority of students (n=14) providing a post internship assessment ofany changes in theirconfidence, knowledge,and interest inSTEM indicatedimprovementsacross all measures, except forthelevel of confidence in their tertiarystudies, wherethemajority of students surveyed indicated no change inthisarea.In the area of professional working relationships, a similar number of students reported animprovement and no change.Figure 2 below shows the proportions of studentsthat reported changesacrosseach measure. 0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% I feel confidentabout mytertiary studiesI am confidentabout mytechnical STEMskillsI know about mySTEM area ofinterestI am confidentabout my corework-readinessskillsI know aboutthe reality ofworking in STEMI haveprofessionalworkingrelationshipsI want to pursueworking in STEMProportion of students thatreported changeImprovedNo changeDecrease ExperiencingaSTEM job and building professional relationships can solidify studentconnection with theSTEM sector and builda stronger understanding of their STEM areas of interest. Across all measures,thelargestpercent change inmean scores before and after the internships occurredin studentunderstandingof the reality of working in STEM(38per centchange in means) followed closelybyan improvement in their professional working relationships (33 percentchangein means) and anincreasedknowledge about their STEM area of interest (30 per centchangein means).Table 1below showsthat in every category,the percentage of students that ratedthemselves highly increasedsignificantly bythe endof their internship.This data indicates agenerouspositiveshift inthe depth of awareness andconfidenceamongstthe studentsandthat overall,the internships are contributing towards studentconfidence andawareness of theirSTEM pathways. Through theirsurvey responses and interviews,manystudents reported on the benefitsthe internship hadfor their confidence andconnectionwith the STEM sector: Igreatly improved mytechnical and problem-solving skills to complement with my studywhich was exactly what I was hoping to achieve from the placement. I also learned a lot more about how the business ran working alongside the chief manager and eventually became part ofthe business. The work was also highlyflexible,and I was very thankful for the opportunitytogrow and developthrough the pastmonths. It's shown mewhat kind ofenvironments I’d like to look for. It hasgiven me experiencein the field and so I have a better understanding of what to expect. It’s made me feel more comfortable in what field I want to go into as well as feeling inspired to look into jobs I didn’t know existed till my work placement. Table 1 Proportion of students with a high positive rating before and after the placement Area of change Percentage rated highly before (4 or 5) Percentage rated highly after (4 or 5) Percentage point difference I feel confident about my tertiary studies 46.67% 60.00% 13.33 I want to pursue working in STEM 40.00% 73.33% 33.33 I am confident about my technical STEM skills 33.34% 66.67% 33.33 I am confident about my core work- readiness skills (e.g., communication skills, working in a team, planning etc.) 26.66% 66.66% 40.00 I know about my STEM area of interest 20.00% 73.34% 53.34 I have professional working relationships 66.67% 80.00% 13.33 I know about the reality of working in STEM 21.43% 46.67% 25.24 The industry partner experience in 2023 Industry partners are building their workplace capability and capacity2 I can see a big difference with these younger students is their energy and their drive and their eagerness to learn and what that does, it rubs off onto our other staff members. So, they’re seeing younger people coming up through the ranks who have high energy, it makes those other people within the company go well. I've got to really step up to this level as well. Industry partner 2 In the context of this report, workplace capability refers to a team’s knowledge and skills required for work related activities and workplace capacity refers to an organisation’s ability to absorb change through elements such as adequate time, energy, resourcing, and a supportive work culture. Overall, industry partners’ feedback was positive, with a 42.5 per cent supervisor response rate, although not all questions were answered by each respondent (n=17). All 9 supervisors who responded to the question, agreed that it was easy (n=5) or very easy (n=4) to become an industry supervisor. Some supervisors had previous experience with students, with 7 out of 17 supervisors surveyed reporting regularly supervision of early career students prior to their engagement in the Generation STEM Links program. All supervisors had their expectations met with the internships, and 78 per cent of supervisor respondents stated the internship exceeded or greatly exceeded their expectations (7/9). All 12 industry partners who responded were very (67 per cent) or extremely satisfied (33 per cent) with the performance of their intern. In addition, all supervisor respondents agreed their confidence to supervise student interns increased moderately (n=4) or significantly (n=5) after their first intern. Through their interviews, industry supervisors identified a range of improvements to their workplace capability and capacity, including, positive leadership opportunities for senior staff; team cohesion demonstrated through staff enthusiasm and positive attitudes around the internship; workplace agility to respond to customer needs through new and transferred skills; and opportunities to innovate, with students being an additional resource and providing new perspectives in problem solving and critical thinking. Industry supervisors highlighted some of these benefits in the following ways: We're creating mini ecosystems and learning off each other… so we invest in the program, we invest in [the student], but also [the student] is investing in sharing skills with the staff, so it kind of has a double effect. It's good to get them [students] in because we're learning, we're learning from them constantly and consistently, which is super important for a small business. Several industry partners and student interns agreed that a placement block works best for student integration into the workplace. A condensed block of time maximises student hours in the workplace over a shorter period and supports student training and learning and integration into the workplace, fast tracking this at the beginning of the internship. Industry partners are returning for multiple student interns, and some are streamlining their recruitment and growing their business in the process. A clear indicator of the positive experience for industry partners and their students is the finding that 67 per cent of industry partners that have completed their first placement have requested additional placements. Several supervisors spoke favourably about what they learned when engaging their first student intern, which resulted in follow-on interns being easier to recruit, onboard, and support. Several industry partners discussed improvements in “how we get them set up and the process of training” as well as improvements to their training documentation. One industry partner commented, “the second internship was much easier. We pretty much had the same advertisement and position description, and it was just a quick conversation with [the Generation STEM Links team]. You don't have to go through...’these are the kind of interns we're looking for’...that's already established”. Several industry partners reported that having a student intern in their workplace promoted the growth of their business, and this was partly attributed to the wage subsidy offered to industry partners. Several industry partners reported that the student interns allowed them to fill workforce gaps created by a “lack of time available and resources for us to manage and execute our projects”, while another industry partner was looking to the interns to “develop a specific product development they were planning to launch into the market”. Eighty-three per cent of students are offered ongoing employment by the host industry partner. Every industry partner interviewed saw the internship process as an excellent alternative to recruitment processes, explaining that it allowed them to comprehensively assess if a student would fit with their workplace before employing them. Several industry supervisors commented: Dealings with [the Generation STEM Links team] were informative and timely. The quality of applicants was very good. Access to the grant makes it possible for us to offer more opportunities than we would otherwise. We were looking for the right person to offer full time employment to. [The] likelihood of identifying a suitable candidate was greater with this process than conducting the process ourselves, our candidate has been offered a full-time position and has accepted. Both interns were hard working, and their contribution included resolving some long-standing issues. The work experience student was enthusiastic, smart and innovative. CSIRO gives us an opportunity to tap into some of that talent. And they don’t all stay, but certainly those people that we could see have potential, the enthusiasm, the skillset, then we offer potentially some placement whether it’s casual or full-time. And hope that they’ll be retainable over the years. They’ve had an impact on our business from a growth perspective. We attribute having new people on board to help us. So small processes that we could outsource but having them on board helps us to speed things up. So much so that in many areas, they are taking on some of the roles that some of our senior people have been doing, and I dare say do better in some cases. Industry partner For these students, employment offers from their STEM Links Industry partner can be a transition to casual and flexible work in a sector relevant to their field of study, while they are still studying or as they transition out of education. Some students highlighted this as a significant advantage and referenced the difficulties some of their student peers faced in securing employment in their field of study following graduation without meaningful industry experience. Some placement matches were not successful, with 8 students (14 per cent) finishing their internship early, sometimes due to a reported lack of fit between the industry partner and student. Feedback from some industry partners and students in these instances reinforce that increased investment in the recruitment, matching and onboarding phase of the program and an ongoing support plan might mitigate some of these situations, including facilitating explicit conversations between key program participants to clarify expectations and responsibilities and for identifying potential communication barriers that might play out in the workplace. Sector level outcomes Student diversity in STEM The Generation STEM Links program is increasing access for underrepresented groups to engage in paid STEM internships. In 2023, 92 per cent of students placed in STEM Links internships were from groups that are underrepresented in STEM. Fifty-four per cent of student interns include underrepresented cohorts of female students, Aboriginal and/or Torres Strait Islander students, and students who are culturally and linguistically diverse. The remaining 38 per cent of students are from locations with low levels of socio- economic advantage (46 per cent). One industry supervisor discussed the value of the Generation STEM Links program in contributing to their objective of increasing the number of women in their workplace, noting that, “traditionally that group was all male, but we hired two females in the group and then we also now have three female interns in the group, and that's changed the dynamic as well”. The program offers students paid internships which are highly valued by students for the status it affords interns within the workplace, as well as the mitigation of financial pressures. One student related, “I sat for interviews and looked into opportunities that were unpaid internships and I didn't feel like they reflect the kind of workplace you want to be working in”. Several students commented on the value of the wage, including one student who stated, “You don’t have to take on another part-time job at the same time as doing your internship”. Another student summed up by saying, “you can actually just focus on your internship”. One industry supervisor also acknowledged the value of the wage in reducing pressure on students, “it just makes the students not worry. They might have a part-time job at a restaurant or something and they can suspend that during that time, and they don’t have to worry about anything. They’re getting paid”. So the money was make or break for me really because I couldn't have done it otherwise. I know if I lived there, it might have been different. Like maybe I could have done it if I moved. And there's not not many opportunities like that where I am currently, so I feel like I have to travel if I want to do things like that. Student intern Local partnership outcomes Generation STEM Links sector collaboration activities are supporting key stakeholders to collaborate on relevant STEM workforce issues and actionable solutions. A component of the Generation STEM Links program is the delivery of workshops that build on existing sector needs or strengths to increase cross-sector engagement between industry, tertiary, and other related stakeholders around Generation STEM Links objectives. Between November 2022 and November 2023, the Generation STEM Links program facilitated 4 workshops and roundtable events across the Hunter/Newcastle and Orana/Dubbo regions. 95 per cent of workshop survey respondents reported that the session they attended met (41 per cent) or exceeded / greatly exceeded (54 per cent) their expectations. Figure 3 below shows the most valued elements of the collaboration activities across the 12 months, with networking and the opportunity to collaborate across sectors the most valued. Approximately a third of workshop participants surveyed volunteered that they valued the opportunity to collaborate with a diverse range of professionals. One participant commented, “the participants were all well informed and able to contribute effectively to discussion, ideas and proposed solutions”. A third of participants also described the events as useful opportunities to tackle relevant issues and identify solutions they perceived as realistic and actionable. One participant wrote: “It provided real life problems and allowed me to work with other people from different industries to come up with solutions”. With the support of the Generation STEM Links program, these collaboration activities have resulted in a range of tangible and developing deliverables, driven by cross-sector need and resources, including: • Toolkit for industry helping them to provide high quality internships. • Toolkit for students to support them before and during internships. • University of Newcastle’s’ Innovation Festival recognising the value of social connections and coordinated efforts between academia, industry, and government. • Student mentorship program in the Orana region. For each deliverable the Generation STEM Links role is determined in collaboration with other stakeholders. For example, Generation STEM Links is actively leading the development of the industry and student toolkits, whereas the University of Newcastle is leading the Innovation Festival event. For the student mentorships program in Orana, the role of Generation STEM Links is to provide support to local education institutions in the region to lead this initiative. This approach builds on community strengths and resources, enabling a more community owned and driven outcome. Figure2Participant feedback on the collaborationactivities. Most valued elements of thecollaboration activities(n=33) Developing avenues for cross sector collaboration Networking with other sector representatives Discussing challengesfaced by your sector Learning how your sector can contribute to the STEMpipelinLearning about programs and resouces available to yoursector 0%10%20%30%40%50%60%70%80%90% Percentage of participants who rated the elementsveryorextremlyvaluable Conclusion Overall,participant feedback for theGenerationSTEM Links program providesa significant level of positive feedbackand useful insightsto support continuous improvement. In 2023, programthroughput hasincreased,and evaluation evidenceindicatesthat in general,program qualityhasbeensustainedover the yearto support positive participant experiences. TheGeneration STEM links program is afully facilitatedprogramfocusing ontailoredsupport tostrengthenpartnershipsbetweeneducation, industryand alliedsectors. This program designrelies oneffectivecollaboration withstakeholdersand in 2024, the programcontinuesto focuson better understanding how to engage effectivelywiththe Vocational Education andTraining(VET) sector,underrepresented students in STEMand a broader rangeof STEM industries,outsidethe commonlysoughtengineering internship.In 2024,the abilityof the programto sustain itscurrent levelof good practice and tailored supportto student and industry placementswhile supporting a high numberof program participantswill ensure progress towardsplanned outcomes andcontinued impact. Questions thatneeded to be answered came out of the initial industry roundtable...The flow on from that wasideal.Workshop participant I feel like we have more firm direction andcould build an action plan to move forward.Workshop participant Insights: 2023| 13