Friday SLO Talks

Podcast Episode: Assessment of Student Learning – The Case for Observable Behavior with Dr. Jarek Janio

In this episode of Friday SLO Talks, organized by the California Outcomes Assessment Hub (COACHES), Dr. Jarek Janio from Santa Ana College led a compelling discussion on the role of observable behavior in assessing student learning. The session challenged traditional metrics of success such as grades, graduation rates, and course completion by questioning whether they truly reflect learning or simply institutional priorities.

The Gap Between Institutional and Student Success Metrics

Dr. Janio identified a disconnect between what students seek practical skills and competencies and how institutions evaluate success. He argued that while colleges track graduation and employment rates, these indicators do not necessarily capture learning outcomes. He invited participants to consider alternative assessment strategies that focus on what students can do rather than just what they remember or complete.

The Role of Learning Theories in Assessment

The discussion transitioned to educational theories, including behaviorism, constructivism, and transformative learning. Dr. Janio highlighted that these frameworks influence how faculty teach and assess learning, yet they often fail to translate into meaningful, skill-based assessment practices. He proposed flipping Bloom’s Taxonomy on its head starting with application and problem-solving before introducing memorization so that students first engage with real-world challenges and later refine their foundational knowledge in context.

Observable Behavior as the Key to Effective Assessment

Dr. Janio argued that the only valid measure of learning is observable behavior. He criticized traditional assessments like standardized tests and essays, noting that they often fail to capture real competencies. Instead, he advocated for:

• Performance-Based Assessments: Evaluating students based on their ability to demonstrate skills in real-world scenarios.
• Portfolios and Projects: Allowing students to compile evidence of their learning over time.
• Self-Assessment and Peer Feedback: Encouraging students to reflect on their own progress and engage in meaningful discussions about their learning.
• AI-Assisted Assessment: Leveraging generative AI tools to analyze student responses, create adaptive learning paths, and enhance formative assessment strategies.

The Role of AI in Assessment

Dr. Janio explored how artificial intelligence can support assessment by automating rubric development, generating feedback, and helping faculty analyze student progress over time. However, he cautioned against over-reliance on AI-generated responses, emphasizing that faculty must take responsibility for interpreting and applying AI insights in a meaningful way.

Redefining Student Engagement and Success

A significant portion of the discussion focused on student engagement. Dr. Janio questioned whether traditional engagement metrics such as class participation, assignment completion, and time spent in tutoring centers actually indicate learning. He proposed that engagement should instead be measured by students’ ability to articulate and apply what they have learned in meaningful ways.

The episode concluded with a call to action: Educators must rethink assessment practices to ensure they capture real learning, rather than relying on outdated proxies like grades and course completion. By shifting toward skill- and competency-based assessment, faculty can provide students with a more meaningful and equitable learning experience.

Here’s a more detailed description of the episode:

Podcast Episode: GenAI – Your Supercharged Assessment Assistant with Dr. Gavin Henning

In this episode of Friday SLO Talks, organized by the California Outcomes Assessment Hub (COACHES), Dr. Gavin Henning, professor of higher education at New England College, provided an in-depth look at how Generative AI (GenAI) is revolutionizing assessment in higher education. With over 25 years of experience in student learning assessment and institutional research, Dr. Henning explored the practical applications of AI in streamlining assessment processes, reducing faculty workload, and improving the quality of learning outcomes.

Moderated by Dr. Jarek Janio from Santa Ana College and Enrique Jauregui from Fresno City College, the session opened with a discussion on the rapid evolution of AI, highlighting how tools such as ChatGPT, NotebookLM, and Gamma are reshaping assessment strategies. Dr. Henning acknowledged the common anxieties faculty have about AI—particularly concerns about academic integrity and student cheating—but emphasized the immense potential of AI to enhance, rather than undermine, meaningful assessment.

Throughout the presentation, Dr. Henning demonstrated multiple AI-driven applications for assessment, including:

• Developing and Refining Learning Outcomes – AI can generate student learning outcomes based on course descriptions, revise them for clarity, and align them with different learning taxonomies such as Bloom’s Taxonomy or Fink’s Significant Learning.
• Rubric Creation and Evaluation – AI tools can draft assessment rubrics, refine them based on institutional needs, and help faculty evaluate the quality of their rubrics to ensure clear, measurable learning criteria.
• Survey and Interview Protocols – AI can assist in designing assessment instruments, such as survey questions and focus group protocols, reducing the time needed to create structured assessment tools.
• Qualitative and Quantitative Data Analysis – While Dr. Henning cautioned against relying too heavily on AI for statistical analyses, he demonstrated how AI can identify themes in qualitative data, synthesize feedback, and even generate reports summarizing key findings.
• Program Review and Accreditation Support – AI can streamline the program review process by helping faculty organize assessment data, generate accreditation reports, and compare institutional assessment strategies.

Dr. Henning also showcased how AI-generated reports and summaries can be transformed into interactive learning materials, including AI-generated podcasts. He demonstrated Google’s NotebookLM, a tool that allows users to create AI-powered podcasts from assessment reports, complete with interactive question-and-answer capabilities. This innovative application of AI, he suggested, could make assessment findings more engaging and accessible to broader audiences.

The discussion also touched on the ethical implications of AI in assessment, including concerns about bias, data privacy, and the environmental cost of AI technology. Dr. Henning encouraged faculty to approach AI as a tool for efficiency and equity while remaining vigilant about its limitations. He also highlighted the importance of maintaining faculty agency in assessment design, ensuring that AI complements, rather than replaces, human expertise.

Dr. Henning’s message was clear: AI is moving at the speed of the Road Runner, and while it presents challenges, it also offers unprecedented opportunities to improve assessment, streamline faculty workload, and enhance student learning outcomes.

1. The Evolving Role of Assessment Professionals: As AI automates routine tasks, assessment professionals must focus on higher-order skills like identifying key data, designing effective data collection strategies, and interpreting AI-generated insights. "These tools are going to allow us to collect a lot more data about students… giving us deeper insights into what’s working and what’s not in our programs."

2. Agent-Based AI Models: The transition from dialogue-based tools (e.g., ChatGPT) to autonomous agent-based models introduces new considerations for task delegation and ethical use. "When we have models that we can delegate tasks to, it becomes critical to be thoughtful about what tasks are delegated."

3. Thoughtful AI Adoption: Implementing AI in education requires careful consideration of student needs, data privacy, and equity to ensure ethical and effective use.

Most Important Ideas

• Automation of Routine Tasks: AI streamlines assessment, freeing professionals for strategic roles. For example, Walton’s tool analyzes student artifacts automatically using pre-defined rubrics.
• Prompt Quality Matters: The "garbage in, garbage out" principle applies. Poorly crafted prompts lead to suboptimal AI outputs. "If you give it a dumb prompt, you’ll access the dumb part of that model’s brain."
• Equity Concerns: Wealthier institutions investing heavily in AI may widen gaps between privileged and underserved students. "This is the single biggest equity topic…wealthier schools are throwing money at AI for their students."
• Human Connection is Essential: AI should complement, not replace, human tutors, allowing them to focus on personalized support and emotional needs.
• Communicating AI Benefits: Institutions must clearly convey the advantages of AI-powered tools to students and parents. "How are we going to package this…in a way that helps students understand the benefit?"

Key Quotes

• Devan Walton: "We’re shifting to agent-based tools…thoughtfulness about delegated tasks is crucial."
• Peter Shea: "Wealthier schools are already investing heavily in AI, creating an equity gap."
• Anne Converse Willkomm: "We need to help students understand AI’s benefits."
• Ruth Slotnick: "This is a once-in-a-generation technology…bigger than the Internet."

Call to Action

1. Engage with AI: Assessment professionals should develop a deep understanding of AI’s capabilities and limitations.
2. Create Ethical Frameworks: Institutions must establish guidelines for responsible AI use in education.
3. Foster Collaboration: Open dialogue between educators, students, and technology experts is essential to ensure AI serves all stakeholders effectively.

AI offers transformative potential for higher education assessment, but ethical, equitable, and strategic adoption is critical to harness its benefits fully.

AI and Instructional Design in Higher Education

1. A New Pedagogical Model: AI offers an opportunity to transition from traditional teaching methods to "generativism," a novel approach combining evidence-based learning principles with AI technology. This shift is essential for maximizing AI's potential. "We are largely trying to use AI to drag along an older pedagogical model... rather than build an entirely new model using the unique affordances of AI."

2. Institutional AI Adoption: Colleges and universities vary in their approach to AI adoption, characterized as:

• Ostriches: Ignoring AI.
• Crows: Slowly exploring policies.
• Falcons: Actively experimenting with AI in teaching and learning. "Falcons tend to be those schools... exploring AI for tutoring chatbots, experiential learning, and simulations."

3. Transformative Instructional Design: AI revolutionizes instructional design by enhancing productivity, creating interactive learning tools, and enabling performance-based assessment. "The ability to create learning tools that observe students while they perform tasks is one of the most exciting possibilities."

4. Equity and Accessibility: AI bridges equity gaps by providing consistent, high-quality assessment and feedback. It also supports continuous learning through accessible ecosystems beyond the classroom. "A well-designed AI, guided by a rubric, can generate high-quality feedback quickly. For me as a writing instructor, it's revolutionized student support."

Most Important Ideas

• Generativism: A pedagogical approach leveraging AI's unique capabilities to reimagine teaching and learning.
• Instructional Design Productivity: AI streamlines material creation, like simulations, freeing instructional designers for strategic roles.
• Performance-Based Assessment: AI enables real-time tracking and assessment of student performance, improving personalization and outcomes.
• AI-Powered Ecosystems: AI fosters engaging, continuous learning environments beyond the classroom, enhancing knowledge retention.

Key Takeaways

• Institutions must embrace new models like generativism to harness AI's transformative potential.
• AI enhances instructional design efficiency, personalizes learning, and bridges equity gaps in feedback and assessment.
• Developing AI-powered ecosystems is vital for lifelong learning and extending support beyond traditional classrooms.

Recommendations

1. Adopt Generativism: Incorporate generativism principles into instructional design.
2. Invest in AI Tools: Focus on tools that create interactive and engaging materials.
3. Implement Performance-Based Assessment: Integrate real-time AI-driven feedback into learning experiences.
4. Collaborate: Build partnerships among instructional designers, faculty, and technology experts for effective AI solutions.
5. Professional Development: Stay informed about evolving AI trends in education.

AI offers transformative potential for higher education, requiring institutions to rethink pedagogy, embrace innovation, and prioritize accessibility and equity.

Cautionary Points and Considerations Around the Adoption and Use of AI in Higher Education

1. Accessibility

Availability & Communication: Institutions must clarify which AI tools are accessible to students, faculty, and staff, balancing enterprise versus free versions and effectively communicating available resources.

Cost & Equity: Free versions often have security risks and limited features, while enterprise versions raise concerns about equitable access for all students.

Faculty Attitudes: Faculty views on AI range from banning its use to fully embracing it. Those embracing AI must educate students on responsible and ethical usage.

2. Dependence on Generative AI

Time Investment & Efficiency: Crafting prompts for AI can be time-consuming. Institutions must find a balance between exploration and efficiency to avoid wasted effort.

Over-Reliance & Cognitive Skills: Overuse of AI may harm critical thinking, creativity, and ethical writing. A 2024 Smart Learning Environments study noted AI can boost writing proficiency but risks originality and ethical standards.

3. Environmental Costs

Electronic Waste & Resource Use: AI data centers generate significant e-waste and consume large amounts of raw materials.

Energy & Power Demands: These centers require high energy consumption, mostly from fossil fuels, contributing to greenhouse gas emissions. For example, "A ChatGPT request uses 10 times the electricity of a Google search."

Water Usage: Cooling and constructing AI systems require substantial water resources, posing environmental concerns.

4. Misinformation & Disinformation

Bias: Algorithmic and user biases can produce misleading information. Confirmation bias, where users shape prompts to affirm beliefs, is particularly problematic.

Inaccurate Information: AI-generated "hallucinations" (false outputs) pose a significant challenge.

5. Privacy & Consent Issues

Cyber Threats: AI facilitates cyber risks like malware, deep fakes, phishing, and impersonation.

Non-Consensual Data Use: Many AI systems lack clear mechanisms for opting out of data collection, raising ethical concerns about personal data use.

Input Privacy: The security of data submitted to AI remains a major issue.

Looking Ahead

Enterprise AI & Critical Thinking: The growing adoption of enterprise AI may hinder long-term critical thinking, echoing social media's effects on judgment.

Environmental Impact & Regulation: While AI offers environmental solutions, stricter policies are needed to mitigate its negative impacts. Promising initiatives from companies like Google and Microsoft to pair data centers with renewable energy provide hope for sustainability.

Misinformation & Disinformation: Addressing these issues requires greater regulation at all levels.

Job Displacement: Planning is essential to address AI-driven job disruptions.

Conclusion

AI offers significant potential for higher education, but addressing its risks is essential. Institutions must prioritize awareness, education, and proactive research to ensure ethical and responsible AI integration within academia.

AI in Higher Education: Policy, Impact, and Ethics

Drawing from her experience as a faculty member and policy developer at Drexel University, Willkomm identifies critical focus areas as institutions adapt to the fast-evolving landscape of AI.

I. Policy Considerations:

Willkomm highlights the urgency of developing AI policies that address the needs of all stakeholders: faculty, students, and staff.

Faculty: Institutions must ensure faculty understand AI's capabilities, limitations, and ethical implications. Providing clear guidelines on responsible AI use in classrooms is vital for both educators and students. "We created documents to help faculty understand AI—what it truly is and how it can be used. Some are familiar with it; others avoid it altogether."

Students: Willkomm warns against over-reliance on AI detection tools due to their limitations and biases. Instead, she advocates for an educational approach that promotes ethical and responsible AI use over punitive measures. "We need to focus on education rather than punishment when dealing with students and AI."

Staff: Often overlooked in AI policy discussions, staff need clear guidelines on AI usage, especially for handling sensitive data. "Staff are frequently excluded from these conversations, which is problematic."

II. Potential Impacts:

Willkomm underscores the consequences of unclear or inconsistent AI practices.

Faculty: A lack of guidance leads to confusion and inequity in how AI is applied, creating disparities for students. "Inconsistent application of AI tools by faculty promotes inequity."

Staff: Ambiguity around AI usage can result in misuse, including unintentional breaches of data privacy. "Without clear boundaries, even unintentional misuse of AI could lead to serious issues."

Students: Navigating inconsistent AI practices across courses hinders students' learning experiences. "Students face challenges when AI usage varies from one class to another."

III. Ethical Considerations:

Willkomm strongly advocates embedding ethics into all aspects of AI integration.

Faculty: Educators should discuss AI ethics with students, model responsible use, and treat misuse as teachable moments. "We must talk to students about ethical AI usage and model it ourselves."

Staff: Establishing ethical guidelines for staff is essential, especially for work involving sensitive data. "Universities need clear rules on when staff can ethically use AI for tasks like assessment or reporting."

Students: Teaching students about AI ethics now prepares them for responsible use in their careers. "If we don't address AI ethics consistently, what are we truly teaching them for their future in industry?"

Willkomm's insights emphasize the importance of policies, equity, and ethics as institutions incorporate AI into education and operations.

The Impact of AI on Assessment Professionals

Source: Excerpts from "Bobbijo Grillo Pinnelli.txt"(transcript of the presentation)

Main Themes:

• The rapid evolution of AI and its impact on the roles of assessment professionals.
• The need for assessment professionals to adapt and acquire new skills to remain relevant in the age of AI.
• A predicted shift from traditional assessment roles towards more strategic, coaching, and leadership positions.

Key Ideas and Facts:

1. The Changing Landscape:

• The speaker, Bobbijo Pinnelli, reflects on the initial anxiety and uncertainty surrounding AI's impact on assessment roles, similar to the feelings experienced during the rise of instructional design tools.
• Quote: "I couldn't keep up with AI. And what I know now is that we'll never be able to keep up with the pace of AI, and that's okay?"
• It's acknowledged that AI is rapidly evolving and will continue to do so, requiring constant adaptation from professionals.

2. AI's Efficiency and Potential Obsolescence:

• Pinnelli acknowledges the potential for AI to replace some assessment functions, particularly those heavily reliant on processing and basic tasks.
• Quote: "Will some of the roles in the assessment become completely obsolete due to the efficiency the AI offers."
• However, roles involving leadership, creativity, coaching, and mentoring are deemed essential and likely to remain in demand.

3. The Need for Skill Development and Adaptation:

• Assessment professionals are urged to identify their current skill sets and areas where AI can be leveraged.
• Quote: "I need to identify where I stand right like, what do I already know how to do around AI."
• Acquiring new skills, applying them in practice, and collaborating with others are crucial steps in remaining relevant.

4. A Shift in Roles and Focus:

• Pinnelli predicts a significant shift from traditional assessment roles to more strategic positions, such as "experienced architects," coaches, and strategists.
• Quote: "We are going to see an entirely new position in the field. For some...some will develop the skills and capacities needed for that experience architect role, but not everybody will get there."
• This shift necessitates a greater emphasis on interpersonal skills and human oversight in AI-driven assessment processes.

5. Call to Action and Career Development:

• Assessment professionals are urged to engage in self-reflection and analyze their current roles and responsibilities.
• Quote: "Think about in your current position...how much time? What is your responsibility? What percentage of responsibility is spent processing, what percentage is reporting, what percentage is consuming data, creating leading others, and really be very honest, and sit and look at that."
• This introspection helps identify areas where skills development is needed to stay ahead of the curve and ensure continued relevance in the evolving field of assessment.

Overall, the excerpt highlights the critical need for assessment professionals to proactively embrace the opportunities and challenges presented by AI. By focusing on developing new skills, adapting to changing roles, and prioritizing human oversight, professionals can remain relevant and effectively support student learning in the age of AI.

AI in Higher Education Assessment

Main Themes:

• Urgency of AI adoption in assessment: Assessment professionals need to proactively engage with AI tools and their implications for student learning despite feelings of overload or a wait-and-see approach.
• Leadership role of assessment professionals: Professionals must lead AI adoption in their institutions, building policies and practices, and collaborating within and across institutions.
• Human-AI partnership: While AI offers opportunities to enhance assessment practices, human oversight remains crucial for ethical considerations and ensuring the validity of insights.
• Shifting roles in assessment: The traditional role of the assessment director is predicted to evolve into an “experience architect,” focusing on personalized learning and leveraging AI to predict student success.

Key Facts and Ideas:

• The Java Jams initiative, a year-long community-driven effort, focused on exploring the intersection of AI and higher education assessment.
• A survey of 264 assessment professionals revealed:
• 50% believe AI will play a major or moderate role in assessment work.
• 70% are not currently using AI in assessment.
• 55% feel prepared for AI integration despite limited practical experience.
• Two main reasons for hesitation: technology overload and a wait-and-see attitude for AI to mature.
• Successful AI integration in assessment requires:
• Experimentation with AI tools to understand their capabilities and limitations.
• Collaborative efforts across departments and institutions to share knowledge and best practices.
• Proactive leadership in developing policies and guidelines for ethical and effective AI use.
• The future of assessment points towards a more personalized learning experience, with AI playing a key role in predicting student success and shaping learning pathways.

Important Quotes:

• On the need for leadership: "If you're an assessment professional in AI, you got to get out there and lead. If you have no idea what you're talking about in AI... you gotta experiment with the tools." - Ruth Slotnick
• On human involvement: "The human always stays in the loop... for insight and oversight, for validity, and just ethically." - Ruth Slotnick
• On the future of assessment: "We will be shapeshifting from that traditional assessment director to something else... moving more to what I call the experience architect." - Ruth Slotnick
• On hesitancy to adopt AI: "One person said that they have, since the pandemic, technology overload... and the second one is, 'I'm not concerned at all... I just want others to fine tune it.'" - Ruth Slotnick

Call to Action:

The Java Jams Grand Finale serves as a call to action for assessment professionals to actively engage with AI and lead its integration into higher education assessment practices. By embracing experimentation, collaboration, and a forward-thinking approach, professionals can harness the power of AI to enhance student learning and shape the future of assessment.

Demystifying Student Learning Outcomes (SLOs) in Higher Education

Main Themes:

• Defining and differentiating Student Learning Outcomes (SLOs) from other outcome measures in higher education.
• Understanding the relationship between SLOs and student achievement.
• Exploring the interconnected levels of learning outcomes: Institutional (ILOs), Program (PLOs), and Student (SLOs).
• Addressing the challenge of aligning outcomes across these levels to ensure meaningful learning and achievement.

Key Ideas and Facts:

1. What are SLOs?

SLOs are "observable and measurable demonstrations of what students can do as a result of instruction." They represent the specific skills, knowledge, abilities, and attitudes students should acquire during a course, program, or activity.

SLOs vs. Student Achievement:

While both relate to student progress, they differ in focus:

• SLOs emphasize the learning process and application of acquired skills. Examples include writing a persuasive essay, solving practical math problems, or demonstrating software proficiency.
• Student achievement focuses on measurable milestones like grades, test scores, and completion rates. These are aggregated at an institutional level and don't always reflect specific skills learned.

Relationship: SLOs are the building blocks of student achievement. Aligning instruction with well-defined SLOs ensures that achievement reflects genuine learning.

2. Levels of Learning Outcomes:

• ILOs (Institutional Learning Outcomes): Broadest level, aligning with the institution's mission and vision. They represent overarching competencies all students should develop, like critical thinking, communication, and problem-solving.
• PLOs (Program Learning Outcomes): Describe what students should achieve by completing a program. They often align with external guidelines and standards relevant to the discipline.
• SLOs (Student Learning Outcomes): Most specific level, tied to individual courses. They specify the knowledge, skills, and competencies students should demonstrate after completing the course.

Alignment: These levels are interconnected, creating a cohesive learning experience:

• SLOs (course-level) build foundational skills.
• PLOs (program-level) apply and synthesize these skills, leading to program-specific competencies.
• ILOs (institutional-level) translate these competencies into broader skills aligned with the institution's goals.

Key Distinction:

• All student learning outcomes are assessed at the course level through observable behavior, not just through surveys or evaluations.

Challenges:

• The term "SLOs" is not widely recognized outside academia, leading to communication challenges.
• The term "outcomes" is often used ambiguously in institutional accountability, encompassing both learning and achievement, which can obscure the focus on genuine learning.

Centering Learning and Serving Students

Main Themes:

• Student-centric approach: The current higher education system often prioritizes administrative processes and metrics over student needs and learning outcomes.
• Accountability through SLO assessment: Student Learning Outcomes (SLO) assessment is crucial for ensuring students gain real-world skills and knowledge, yet it remains underfunded and underutilized.
• Empowering faculty: Faculty are key drivers of student success but require adequate resources and institutional support to focus on learning.
• Redefining success: Institutions need to shift their focus from attendance and completion to actual learning outcomes as the primary measure of success.

Key Ideas & Facts:

• The current system can be a barrier for non-traditional students, with complicated processes, long wait times, and financial hurdles.

"Imagine this scenario: a student loses a job right before the holidays and turns to a community college for help, only to discover they can’t meet with a counselor until mid-January."

• Institutions often measure attendance, persistence, and completion, but these metrics don't necessarily reflect the quality of education or student learning.

"Yet, while institutions measure attendance, persistence, and completion, they rarely measure what truly matters: learning."

• SLO assessment is crucial for accountability and ensuring students develop valuable skills, yet it lacks sufficient funding.

"If we want to transform this system, funding for SLO assessment is non-negotiable."

• Faculty play a critical role in student learning but face constraints due to limited resources and institutional support.

"Faculty must be empowered—not just through funding but through institutional commitment—to make learning the central focus of everything we do."

• Institutions need to prioritize learning outcomes as the primary measure of success, reflecting the true purpose of higher education.

"Students don’t attend college to sit in a classroom; they come to acquire skills, gain knowledge, and change their lives."

Call to Action:

The briefing document advocates for a paradigm shift in higher education, calling for:

• Increased funding for SLO assessment to ensure accountability and focus on learning outcomes.
• Stronger institutional support for faculty, providing them with the resources and autonomy to prioritize student learning.
• A fundamental shift in how success is measured, moving beyond attendance and completion to focus on demonstrable learning outcomes.

By implementing these changes, higher education can rebuild trust and ensure that every student's journey leads to meaningful success. The focus must be on serving the students and empowering them with the knowledge and skills they need to thrive.