Making Student Learning Outcomes (SLO) Assessment Meaningful and Actionable

This briefing doc reviews key takeaways and advice from Friday SLO Talks, hosted by California Outcomes Assessment Coordinators’ Hub (COACHES). It explores tools, training, and frameworks to enhance SLO assessment and provides guidance for those new to the process.

Main Themes

• Moving beyond compliance to focus on student learning. SLO assessment should be viewed as a tool to enhance student learning and skill acquisition rather than a bureaucratic box to tick.
• Collaboration and leveraging resources. Connecting with colleagues, utilizing available technology, and engaging with professional development opportunities are crucial to successful SLO implementation.
• Starting small and celebrating progress. Beginning with manageable projects and recognizing successes can build momentum and foster a culture of continuous improvement.

Key Ideas and Facts

• COACHES provides a wealth of resources:
• Curated website with resources for coordinators and faculty
• AI-driven tools for communication including podcasts, YouTube channels, and social media
• Recorded Friday SLO Talks sessions
• Annual SLO Symposium attracting over 1,000 attendees nationwide
• Beyond COACHES, valuable tools and frameworks exist:
• Bloom’s Taxonomy: Helps design measurable learning outcomes progressing from basic to higher-order thinking skills.
• AAC&U's VALUE Rubrics: Offers detailed criteria for assessing essential learning outcomes.
• Learning Management Systems (LMS): Built-in tools for data collection, analysis, and reporting.
• National Institute for Learning Outcomes Assessment (NILOA): Provides research, tools, and best practices for assessment improvement.
• Advice for newcomers to the SLO journey:
• Start Small: Begin with a single course or program and focus on defining clear, measurable outcomes.
• Collaborate: Engage with colleagues within your institution and through networks like COACHES.
• Leverage Technology: Use AI and LMS tools to simplify data collection and analysis.
• Focus on Learning: Approach SLOs as a way to enhance student learning.
• Seek Professional Development: Attend workshops, webinars, and conferences.
• Celebrate Progress: Recognize and share successes to build momentum.

Quotes

• "Focus on Learning, Not Compliance: Approach SLOs as a way to enhance student learning, not as a bureaucratic requirement."
• "Collaboration can reduce the feeling of isolation that often comes with SLO work."
• "SLO assessment is a continuous journey."

Conclusion

SLO assessment is an ongoing process that requires commitment, collaboration, and the utilization of available resources. By focusing on student learning and embracing a culture of continuous improvement, educators can ensure that SLOs serve their intended purpose: to equip students with the skills and knowledge they need to succeed.

Harnessing Generative AI for Enhanced Learning

Source: Transcript of Dr. Curt Bonk's presentation. Friday SLO Talk: "How faculty can harness generative AI for enhanced learning. Part one", hosted by the Coaches Community College network.

Main Themes:

• The educational landscape is rapidly changing: AI is the hot topic in education today, similar to MOOCs a decade ago.
• Generative AI presents both opportunities and challenges: While students are already utilizing AI for learning, educators are navigating awareness, resistance, and implementation stages.
• AI can enhance learning by fostering deeper engagement and critical thinking: Dr. Bonk emphasizes going beyond knowledge acquisition towards higher-order thinking skills.
• Two frameworks for incorporating AI in pedagogy: Dr. Bonk introduces his "Tech Variety" model and the "R2D2" (Read, Reflect, Display, Do) model.

Important Ideas/Facts:

• Student adoption of AI is growing: Studies show significant increases in student use of AI for learning and its perceived impact on grades.
• AI skills are in high demand: Graduates with AI experience on their resumes are securing better job opportunities and salaries.
• Numerous resources are available for educators: Dr. Bonk highlights several books, articles, and websites offering practical examples and guidance on AI pedagogy.
• Ethical considerations surrounding AI in education must be addressed: Discussions on bias in AI-generated content and the implications of students training AI are crucial.

Key Quotes:

• "Data is emerging that indicates students may be more likely to be considered for high paying jobs if they have experience with AI in their job applications." - Ray Schroeder, University of Illinois, Springfield.
• "My job is not to police everybody in here, nor I really don't to rank them, but to find ways to inspire students to get them to join me in the learning process." - Professor at Middlebury College.
• "By leveraging AI, we're able to author not only comprehensive case studies, but generate relevant dialogue feedback with branches that take the learner on a rich journey." - Instructional Technologist, Open RN Project.
• "I cannot do this alone. I need all your help." - Dr. Curt Bonk

Models for AI Integration:

• Tech Variety Model: Based on 10 principles of motivation, it encourages educators to incorporate AI tools in diverse ways to enhance learner engagement.
• R2D2 Model: Focuses on leveraging AI to facilitate reading comprehension, critical reflection, visual representation, and hands-on activities.

Call to Action:

Dr. Bonk urges educators to embrace AI, explore available resources, and collaborate to develop innovative pedagogical approaches that harness the potential of AI for enhanced learning experiences. He emphasizes that individual efforts are not enough and collaborative action is needed to navigate this transformative period in education.

Main Themes:

1. The Higher Education Social Contract
   • Universities must certify students' knowledge and abilities to meet societal expectations.
2. Generative AI (Gen AI) and Academic Integrity
   • Gen AI introduces challenges and opportunities for education, affecting teaching and learning integrity.
3. Shifting from Performance to Mastery
   • Transitioning from performance-based assessments to mastery-oriented learning can reduce cheating and align with the realities of Gen AI.

Most Important Ideas/Facts:

1. The Social Contract and Gen AI
   • Higher education's promise to certify students' abilities is threatened by Gen AI misuse.
   • Gen AI enables students to misrepresent their knowledge and skills, undermining trust.
2. Understanding Student Behavior and Gen AI
   • Students use Gen AI primarily to improve grades, showing extrinsic motivation.
   • Concerns include learning quality, job readiness, and fairness in academic competition.
3. Crafting a Gen AI Policy
   • Clear guidelines on Gen AI use are essential, with options including banning, conditional use, or full integration.
   • Secure assessments are critical to preserving academic integrity, despite ethical concerns around surveillance.
4. Shifting from Performance to Mastery
   • Focus on mastery by:
     • Updating learning outcomes for Gen AI contexts.
     • Personalizing and real-world-connected assessments.
     • Prioritizing learning processes and scaffolding.
     • Using active learning and transparent writing strategies.
     • Adjusting grading rubrics to reflect mastery.
5. AI Detection Tools
   • Use AI detection tools responsibly, following institutional guidelines, and avoid sole reliance on them for identifying misconduct.

Key Takeaways:

1. Gen AI prompts the need for revised teaching and assessment practices in higher education.
2. Open communication, clear policies, and mastery-oriented learning are crucial to maintaining academic integrity in the age of AI.

Driving Change and Co-Creation in Higher Education

Friday SLO Talk: Janet De Wilde, Emily Salines, and Stephanie Fuller of Queen Mary University of London. The presentation explores the challenges and opportunities of driving change in large, legacy institutions, particularly in the context of evolving student needs, economic pressures, and technological advancements.

• Co-creation as a Mechanism for Change: The speakers advocate for co-creation as a collaborative approach to implementing change, involving students, staff, and various stakeholders in the process.
• Leadership, Scholarship, and Recognition: The presentation highlights the importance of effective leadership, scholarship, and recognition systems to support change initiatives and empower individuals within the institution.

Driving Forces for Change:

• New Regulations and Socioeconomic Pressures: Universities face increasingly specific and challenging regulations in the UK, coupled with socioeconomic pressures impacting the sector, institutions, and students alike.
• Diverse Student Needs and Employability: The need to address the diverse requirements of growing student cohorts and the emphasis on employability drive efforts to meet skills gaps and prepare students for the workforce.

Challenges to Change:

• Legacy Systems and Resistance: "People like and feel reassured and comfortable. And and that's our challenge... we need a shared understanding of the need for change."
• Inconsistent Understanding and Practice: "We had inconsistency of practice, and we had inconsistency of educational change. So that was our challenge is, you know, it's very spread out. And it's very different understandings."

Strategies for Implementing Change:

• Alignment and Career Pathways: Establishing alignment between the strategy, institutional frameworks, and career pathways was crucial for effective implementation.
• Role-Based Leadership Programs: Specific educational leadership programs focused on role-based leadership, such as empowering module organizers to lead effectively.
• Focus on Scholarship and Evidence-Informed Practice: Significant investment in leadership and training helped staff develop evidence-based approaches to evaluating and implementing changes in their practices.
• Utilizing Co-Creation: Co-creation was embraced as a key mechanism to drive change and ensure widespread engagement and collaboration.

Co-Creation in Practice:

Shared Understanding and Distributed Leadership: The 2030 strategy faced challenges because many assumed it was solely the university's responsibility. To succeed, individuals needed to understand their accountability and feel empowered to act with agency.

Importance of Trust and Recognition: Trust develops through effective frameworks, rewards, and recognition, which foster a sense of reliability and collaboration.

Student Involvement and Recognition: Students played a key role in co-creating the University's graduate attributes, leading to the creation of the Seed Award to honor student-enhanced engagement.

Key Quotes:

• "Strategy can drive change. But the whole thing needs scaffolding so that Staff can engage. We can't just have a strategy."
• "Co-creation at Queen Mary is central to our strategy. We will deliver outstanding inclusive world class education co-created with our diverse student body."

Exploring Assessment Choice to Enhance Practice and Inclusivity in Higher Education

Friday SLO Talks: Stephanie Marshall, Janet De Wilde, Emily Salines, and Stephanie Fuller of Queen Mary University of London's Queen Mary Academy.

Main Themes:

• Inclusive Assessment: Queen Mary University prioritizes inclusive assessment practices, recognizing that assessment drives learning and significantly impacts student outcomes and well-being. The presentation explores how offering assessment choices can enhance inclusivity and support students from diverse backgrounds.
• Assessment Choice as a Driver for Change: The presentation highlights the benefits of assessment choice, such as promoting self-regulation, self-efficacy, engagement, and performance, aligning with principles of Universal Design for Learning. It showcases a small-scale research project exploring the impact of offering choice in a postgraduate academic practice program.
• Addressing Barriers to Change: The presentation acknowledges challenges in implementing assessment changes within a large institution, including potential resistance to moving away from traditional approaches, concerns about workload, and navigating regulatory frameworks.
• The Role of Educational Development: The presentation emphasizes the role of educational development units like the Queen Mary Academy in driving assessment change. It positions educational development programs as catalysts for fostering a community of innovative educators and facilitating a shift toward more inclusive practices.
• Burkana Institute's Two-Loop Change Model: The presentation draws inspiration from the Burkana Institute's two-loop change model, highlighting the importance of identifying and supporting pioneers of change, convening networks of innovators, and strategically phasing out outdated practices while respecting their value.

Key Ideas and Facts:

• Queen Mary University prides itself on being an inclusive Russell Group University, committed to transforming lives and opening doors of opportunity for its diverse student body.
• "You can survive bad teaching, but you can't survive bad assessment." This quote highlights the lasting negative impact of poor assessment practices on student learning and outcomes.
• A national student survey in the UK consistently reveals student dissatisfaction with assessment and feedback, underscoring the need for reform.
• "Rethinking assessment really means rethinking long-held beliefs, and that can be difficult and painful at times." This quote acknowledges the emotional dimension of challenging ingrained assessment practices.
• A research project on introducing assessment choice in a postgraduate program showed that most participants found the experience positive, inclusive, and less stressful. 90% of respondents believed that offering assessment choice would be beneficial for their own students. 60% were planning to implement it in their practice.
• The Burkana Institute’s model encourages identifying "pioneers" of change, "convening" them into communities of practice, and utilizing "storytellers" to disseminate innovative approaches.

Quotes:

• "Assessment drives students' learning."
• "[Assessment is] a key element in supporting social justice in education."
• Having the choice was surprisingly empowering."

Learning Principles and Motivation

Source: Excerpts from a Friday SLO Talk webinar featuring Dr. Marie Norman, co-author of How Learning Works: Eight Research-Based Principles for Smart Teaching.

Main Themes:

1. Teaching vs. Learning: Content expertise alone doesn’t ensure effective teaching. For instance, Isaac Newton, despite his brilliance, struggled to convey his knowledge through lectures."If you know a lot about something... you are then automatically qualified to teach. But... teaching has its own very specific set of skills and knowledge."
2. Importance of Prior Knowledge: Students bring varied knowledge, beliefs, and misconceptions to the classroom. Effective learning relies on activating, building upon, and addressing existing knowledge gaps and inaccuracies."Prior knowledge... is the single most important component in new learning." Challenges include:
   • Inactive knowledge: Difficulty recognizing or applying prior knowledge.
   • Gaps in knowledge: Missing foundational information.
   • Inappropriate prior knowledge: Misapplying correct knowledge.
   • Inaccurate prior knowledge: Holding misconceptions.

   Dr. Norman’s card test illustrates this: students struggled with an abstract problem but succeeded when it was framed in a familiar context (drinking age rules).

3. Expert Blind Spot: Experienced instructors may overestimate students' understanding. For example, in a research methods course, a professor assumed students could apply statistical tests, overlooking the actual complexity for novice learners."It simply requires that students take a data set, select and apply... things which are actually quite complex."
4. Motivation as a Key Driver of Learning: Motivation fuels student effort, with two crucial aspects:
   • Value: Students’ perception of relevance.
   • Expectancy: Belief in their ability to succeed and in the fairness of the environment.

   "Students' motivation generates, directs, and sustains what they do to learn." Strategies to foster motivation include connecting content to student interests, promoting competence and autonomy, and providing clear expectations and feedback.

Recommended Resources:

• How Learning Works by Ambrose et al.
• Make It Stick: The Science of Successful Learning by Brown, Roediger, & McDaniel
• Teach Like a Champion by Lemov

Conclusion: Dr. Norman underscores the value of understanding and applying learning principles. Addressing issues such as prior knowledge and motivation allows instructors to create more engaging and effective learning experiences.

This podcast explores the pressing need for a transformation in higher education assessment practices, particularly given the impact of generative AI. Professor Elkington argues that traditional, high-stakes assessments fail to prepare students adequately for modern careers, advocating instead for a learning-centered approach that values process, feedback, and evaluative skills.

Key Themes and Insights:

• Assessment’s Evolving Role: Current assessments often impose high stakes with limited relevance to future career demands. The narrow range of formats limits student expression. Generative AI challenges traditional assessments, prompting a reassessment of academic integrity and a renewed focus on distinctly human skills.“The high-risk nature of many of our established assessment practices. We’re weighting assessments at 100, so students have a single opportunity to demonstrate their learning. That’s high-risk and high-stress.”
• Three Purposes of Assessment:
  • Assessment of Learning: Summative, focused on final grades.
  • Assessment for Learning: Formative, guiding through feedback.
  • Assessment as Learning: Engaging students with real-world issues and metacognitive skill-building.

  “Assessment as learning focuses on authentic problems, issues, challenges, and projects.”

• Principles for Effective Assessment:
  • Alignment with Learning Outcomes: Ensuring assessments reflect key skills.
  • Engaging Tasks: Encouraging active, relevant learning.
  • Timely Feedback: Providing actionable insights for immediate and future improvement.
  • Building Evaluative Expertise: Guiding students to recognize quality work.

  “When it comes to assessment, you get what you model for.”

• Flexible Assessment at Teesside University: Teesside emphasizes inclusivity and transparency in its assessment design, offering students choice and promoting autonomy.“Flexible assessment is inclusive, learning-focused, and transparent.”
• Broadening the Assessment Lens: Recognizing students’ entire learning journey fosters coherence. Elkington calls for “assessment practice with an S” (processes) alongside “assessment practice with a C” (outcomes) to enable a more holistic and progressive approach.“We need to embed assessment practice with an S [processes] and assessment practice with a C [outcomes].”
• Assessment in an AI-Driven Era: Embracing flexible, multimodal assessment formats is crucial. Offering diverse submission options and setting negotiated criteria encourages authentic, skill-based learning for graduates in an AI-influenced world.
• Practical Strategies for Modern Assessment:
  • Employer Engagement: Align assessments with authentic, industry-related tasks.
  • Student Agency: Promote self-regulation and ownership.
  • Collaboration: Prepare students for teamwork with collaborative projects.
  • Adaptability: Shift towards flexible, personalized assessments.

Analogies:

• Swiss Cheese Paradox: Layering multiple assessment types to minimize learning gaps.
• Mass-Produced vs. Crafted Chairs: Choosing between standardization and personalized, creative skill demonstration.

In this episode, Dr. Gianina Baker, Associate Director of the Office of Community College Research and Leadership and Acting Director of the National Institute for Learning Outcomes Assessment (NILOA), explores how assessment practices in higher education can shift from a compliance-driven model to one focused on student empowerment and continuous improvement.

Key themes include:

• Evolving Assessment Practices: Baker outlines how assessment began with an accreditation-driven approach but has increasingly moved towards a model that values student-centered and improvement-focused assessment.
• Transparency in Communication: The NILOA Transparency Framework is highlighted as a guide for institutions to effectively communicate assessment practices and learning outcomes to both internal and external stakeholders. Transparency goes beyond availability—it requires clarity and context.
• Using Data for Improvement: Emphasizing action over mere data collection, Baker calls for using assessment evidence to inform decisions, support improvement initiatives, and communicate the value of student learning.
• Equity in Assessment: Baker stresses the importance of analyzing data by student demographics to address disparities and support all learners effectively.

Notable points include:

• The NILOA Transparency Framework: Comprised of six components, this framework supports institutions in making learning evidence clear, accessible, and actionable.
• Student-Centered Assessment: Moving beyond accreditation, the presentation emphasizes putting students at the heart of assessment through agency, feedback loops, and a holistic view of learning experiences.
• Quotes and Insights: Quotes from thought leaders like George Kuh reinforce the importance of patience and practice in assessment, with Baker noting that institutions must actively use assessment data to earn distinctions like the “Excellence in Assessment” designation.

Takeaways:

1. Apply the NILOA Transparency Framework to your institution’s assessment practices.
2. Enhance communication around assessment data, ensuring it’s accessible to diverse audiences.
3. Prioritize using assessment data to drive improvement and demonstrate educational value.
4. Involve students as active participants, emphasizing their feedback and impact on assessment outcomes.

This presentation provides a powerful call for institutions to rethink assessment as a tool for empowerment, transparency, and true educational impact.

In this episode, Kristen DiCerbo, Chief Learning Officer at Khan Academy, provides a deep dive into Khan Academy’s research-backed approach to improving student learning outcomes. From understanding the importance of usage in K-12 learning to unveiling the potential of AI-powered tools, DiCerbo shares insights on the methodologies driving Khan Academy’s commitment to educational efficacy and equity.

Key themes include:

• Efficacy in Action: DiCerbo details how Khan Academy measures impact, emphasizing that even 30 minutes of weekly engagement can lead to significant learning gains. She explains that these benefits span across demographics, including gender, ethnicity, and socioeconomic status, though there are opportunities to enhance support for English language learners.
• Mastery Learning: Khan Academy champions deep learning over surface-level coverage, using the “Skills to Proficient” metric to track skill mastery, which closely correlates with external assessments. DiCerbo argues for focusing on proficiency in fewer skills rather than broad, superficial coverage to reinforce long-term retention.
• AI-Powered Learning with Conmigo: Khan Academy’s AI tutor, Conmigo, helps students in math, science, and writing by guiding them through problem-solving steps without giving direct answers, fostering active learning and “productive struggle.” Available in multiple languages, Conmigo also supports English language learners and includes transparency features for teachers to track student-AI interactions.
• Supporting Student Agency and Ethics in AI: DiCerbo discusses how AI tools can empower students to become active learners. She also emphasizes the ethical considerations of AI in education and advocates for community-driven conversations around AI literacy to responsibly integrate these technologies in learning environments.

Looking ahead, Khan Academy is refining Conmigo’s design to better balance productive struggle with student frustration, ensuring learning remains both challenging and supportive. This episode highlights Khan Academy’s forward-thinking use of AI and its dedication to creating data-informed, ethical solutions for students and teachers. Join us to explore how technology is reshaping education and the future of learning.

In this episode, Christina Supe examines how AI is reshaping writing, literacy, and authorship. She opens by highlighting literacy’s longstanding link to power and social mobility, emphasizing that AI tools might disrupt this by enabling “cognitive outsourcing.” This trend could risk intellectual stagnation and perpetuate existing power imbalances by reducing opportunities for critical thinking.

The discussion challenges traditional definitions of a "writer," exploring AI as an agent that creates text and forces us to rethink what qualifies as writing and authorship. Supe raises ethical concerns, particularly about when AI assistance is appropriate. She emphasizes the need for clear distinctions between tasks requiring authentic human input and those that may reasonably involve AI. Copyright laws are evolving to address AI’s role in content creation, with recent changes from the US Copyright Office now allowing copyright for works with minimal AI input, signaling the need for defined policies.

Key quotes from Supe underline literacy as a means to access power, warning against the dangers of “outsourcing” cognitive skills to machines. She discusses the value of "reasonable reluctance" toward AI, as voiced by educators, who fear AI could undermine critical thinking and erode essential skills in close reading and analysis. AI, she argues, poses risks as a "shortcut" that may diminish learning and creativity if used inappropriately.

For educational institutions, the discussion underscores the need to:

• Establish Clear AI Policies: Clearly outline when and how AI can be ethically used in student work to guard against plagiarism and intellectual shortcuts.
• Foster Critical Thinking: Encourage original thought and close engagement with materials to help students retain control over their intellectual growth.
• Educate on AI’s Role: Help students understand AI’s limitations and ethical considerations in writing.
• Reevaluate Assessments: Develop assessments that prioritize originality and deeper engagement, ensuring AI use doesn’t overshadow learning objectives.

Through Supe’s insights, this episode encourages proactive strategies to balance AI’s potential with a commitment to authentic learning and intellectual development.