In this podcast, Ryan Smith, Director of University Assessment at Illinois State University, discusses how data storytelling transforms overwhelming data in higher education into actionable insights that inspire meaningful change. He addresses the challenge of institutions collecting vast amounts of data but lacking the capacity to utilize it effectively. Ryan advocates shifting from viewing data as purely objective to embracing the storyteller's role in crafting impactful narratives.

Main Themes:

• Data Overload vs. Utilization: Institutions are inundated with data they struggle to use effectively. Data storytelling offers a solution by turning complex datasets into clear, compelling narratives.
• The Role of the Storyteller: Acknowledging the storyteller's influence is essential for impactful assessment, moving beyond a strictly positivistic approach.
• Building Trust and Understanding Stakeholders: Successful data storytelling requires understanding diverse stakeholder needs, engaging non-technical audiences, and building trust through transparency and empathy.

Key Points:

• Data Storytelling Process:
  • Identify the Problem: Focus on the specific issue the data story addresses. ("Ask yourself what is the point...and if you can't answer that, don't do it." – Ryan Smith)
  • Know Your Audience: Understand their background and data perception. ("Prioritize the top 3 and don't make them up." – Ann Emory)
  • Define the Purpose: Decide if the visualization is for exploration, explanation, or both.
  • Craft the Narrative: Use storytelling elements like setting, characters, conflict, and resolution.
• Types of Visualizations:
  • Exploratory: Reader-driven dashboards for interactive insight discovery.
  • Explanatory: Author-driven visualizations conveying specific narratives.
• Key Considerations:
  • Build Trust: Be transparent about data limitations and engage stakeholders. ("People know us...they know what we're doing. So it's important to be confident and tell that story...and of course, to be genuine, too." – Ryan Smith)
  • Emphasize Utility: Focus on stories highlighting areas for improvement. ("If you're not using assessment, why are you doing it?" – Ryan Smith)
  • Address Challenges: Recognize different stakeholder interpretations and use storytelling to bridge perspectives.

Key Quotes:

• "It's seeing something that doesn't exist already. It's taking a giant data set...and then making it into a decision and making it into a story." – Ryan Smith
• "The amount of data we have far exceeds our capacity to use it." – Ryan Smith
• "When you know information that can result in more effective student learning, and you don't do anything with it...shame on us!" – Daphne Bernard

Actionable Takeaways:

• Initiate Stakeholder Conversations: Understand data needs and preferred storytelling methods.
• Start Small: Focus on a single, impactful data story addressing a key priority.
• Experiment with Visualizations: Use tools like Power BI and Tableau.
• Continuous Learning: Develop skills through resources and communities.
• Advocate for Cultural Shift: Move from data collection to data utilization via storytelling.

In this podcast, Dr. Daphne Bernard and Glenn Phillips explain how Howard University enhanced its assessment process by incorporating storytelling to improve student learning outcomes (SLOs) and competency attainment.

Context:

Howard University, a historically Black institution with a unique mission, faced an accreditation review. Traditional, compliance-focused assessment methods didn't fully capture its rich educational experiences. The COVID-19 pandemic added further challenges.

Integrating Storytelling:

They adopted Banks-Wallace's framework:

• Story: Providing meaningful context to events.
• Storying: Weaving events into coherent narratives.
• Storytelling: Engaging stakeholders by sharing these narratives.

Implementation:

• Faculty Engagement: Faculty shared stories of curricular improvements and student successes.
• Student Involvement: An essay contest invited students to reflect on their learning experiences.
• Pandemic Adaptation: Documented remote learning challenges and adaptations.

Outcomes:

• Enhanced Understanding: Combined quantitative data with personal narratives for a fuller picture of student learning.
• Increased Engagement: Personal stories fostered deeper investment from faculty and students.
• Mission Alignment: The approach resonated with Howard's cultural heritage and mission.
• Continuous Improvement: Stories informed enhancements in programs and services.

Key Insights:

• Storytelling uncovers deeper insights into student learning.
• Engaging stakeholders as storytellers fosters ownership.
• Tailoring assessment to the institution's context enhances effectiveness.
• Balancing narratives with data leads to comprehensive understanding.

Conclusion:

By integrating storytelling into assessment, Howard University enriched its evaluation of student outcomes and aligned the process with its mission. This approach offers valuable insights for other institutions seeking to enhance their assessment practices in meaningful and culturally responsive ways.

"Leveling the Learning Landscape: An Outcomes-Driven Approach to Equity-Focused Curriculum Change"

In this podcast, Stanford University's Center for Teaching and Learning discusses their "Leveling the Learning Landscape" project—a five-year initiative aimed at transforming the undergraduate curriculum to enhance equity, accessibility, and student learning outcomes.

Key Themes:

1. SMART Goals for Curriculum Transformation:
   • Utilizing Specific, Measurable, Attainable, Relevant, Time-bound goals to ensure clear and effective curriculum changes that enhance skills and competencies.
2. Equity-Focused Curriculum:
   • Addressing challenges of equity and belonging by redesigning courses, creating new learning pathways, and fostering inclusive learning environments.
3. Collaborative Process:
   • Engaging faculty, staff, and students in interdisciplinary teams.
   • Teams participate in the Curriculum Transformation Institute (CTI) to develop proposals using evidence-based practices and receive peer feedback.
4. Continuous Assessment and Evaluation:
   • Implementing strategies like placement exams, student feedback, and data analysis to measure the impact of curriculum changes and tailor them to student needs.

Examples of Initiatives:

• Center for Comparative Studies in Race and Ethnicity (CCSRE):
  • Developing a new introductory curriculum that emphasizes student feedback and fosters a sense of belonging.
• Statistics Team:
  • Creating new courses with better pacing and providing a free online textbook to enhance accessibility.
• Physics Team:
  • Using placement diagnostic exams to assess prior knowledge and promote equity by guiding students to appropriate courses.
• Inclusive Field Education Team:
  • Enhancing inclusive practices through class observations and providing feedback to instructors.

Challenges and Solutions:

• Faculty Workload and Resources:
  • Addressed by offering seed grants, potential course releases, and support from data coaches to sustain faculty engagement.
• Navigating Institutional Processes:
  • The CTI assists teams in understanding and navigating complex approval procedures for curricular changes.
• Data Access and Privacy:
  • Balancing the need for data to understand student experiences with privacy concerns through collaboration with institutional research and adherence to policies.

Key Insights:

• Effectiveness of SMART Goals:
  • Clear and measurable objectives lead to more effective curriculum changes and improved competency attainment.
• Student Partnerships Enhance Equity:
  • Involving students in the assessment process aligns the curriculum with their needs and promotes a sense of belonging.
• Continuous Assessment Drives Improvement:
  • Ongoing evaluation allows for curriculum adjustments that better meet student needs.

Utilizing Course Outcomes for Continuous Improvement

Source: Excerpts from a presentation by Dr. Will Miller, Associate VP for Continuous Improvement at Embry-Riddle Aeronautical University, on leveraging course-level outcomes for student success.

Main Themes:

The Unit Disconnect: Miller addresses the disconnect between program-level and faculty-level assessment views, resulting in fragmented learning for students. Misaligned courses and inconsistent objectives hinder students from synthesizing knowledge effectively.

Quote: "The unit disconnect concerns me... It leads to fragmented learning."

Holistic Assessment: Advocates for assessing the entire student experience, including academics and co-curriculars. Emphasizes aligning objectives and recognizing experiential learning.

Quote: "We want assessment at Riddle to be holistic... look at the entire student experience."

Individual Student Tracking: Highlights the need for tracking student learning at the course level to enable personalized learning, targeted remediation, and proactive support.

Quote: "We need faculty on board with tracking individual student learning outcome achievement."

Data-Driven Decision Making: Emphasizes using data to improve areas like curriculum mapping, prerequisites, instructor effectiveness, and course scheduling. Outcome data helps identify gaps and redundancies.

Quote: "It should lead to better decisions."

Curriculum Mapping: Encourages a dynamic, data-driven approach to curriculum mapping to gain real-time insights into student learning.

Quote: "We're moving towards a much more dynamic, data-driven process."

Importance of Culture and Collaboration: Stresses the need for institutional commitment, data literacy, and collaboration among faculty, advisors, and administrators.

Quote: "We need the data literacy. We need collaboration."

Key Ideas/Facts:

• Granular Data Matters: Analyzing individual outcomes reveals important patterns and correlations.
• Predicting Success: Understanding crucial outcomes can help better prepare students for future courses.
• Data Improves Teaching: Outcome data identifies effective teaching practices and supports personalized learning.
• Ethical Use of Data: Emphasizes privacy, transparency, and avoiding punitive use of data.
• Change Requires Commitment: A data-driven approach needs institutional buy-in and faculty development.

Selected Quotes:

• On the Disconnect Between Grades and Learning: "Success isn't measured by hoops or clock hours. We have to really pick apart these subjects."
• On Data to Inform Decisions: "I love giving advisors access to data because parents and students respond when shown the trends."
• On Transparency: "Please tell students the course outcomes. If it helps them take it seriously, it's a win for everyone."

Overall: Miller advocates for shifting from a grade-centric approach to a holistic, data-driven model prioritizing individual learning outcomes.

Key Takeaways

1. Experiential Learning (EL) as an Ecosystem:

• EL is a dynamic ecosystem involving students, faculty, administrators, employers, and community members. Each stakeholder has unique perspectives and expectations, creating a complex environment where definitions and approaches to EL vary.

2. The Multifaceted Nature of EL:

• EL encompasses diverse practices, from internships to cross-disciplinary projects. Given this diversity, acknowledging the varied approaches and epistemologies of EL across disciplines and institutional levels is essential for developing inclusive and effective assessment methods.

3. Assessing the Full Cycle of EL:

• Effective EL assessment requires going beyond traditional grading to evaluate the full experiential learning cycle, including engagement, reflection, skill development, and application.

4. Data as a Powerful Tool:

• Thoughtful data collection and analysis are critical for understanding EL’s impact on student learning outcomes and institutional effectiveness.

Detailed Review

Understanding the EL Ecosystem:

• The EL ecosystem is complex, involving various stakeholders with distinct perspectives on EL. Heinrich emphasizes the importance of recognizing these differences for effective communication and assessment, as different stakeholders may interpret "experiential learning" in unique ways.
• Institutions are increasingly expected to demonstrate the value of their programs to students, families, employers, and legislators. EL is seen as a driver of student success and institutional accountability, aligning with the goals of higher education.

Exploring EL Pedagogy:

• EL is grounded in theories like Kolb’s Experiential Learning Cycle, which emphasizes four stages: concrete experience, reflective observation, abstract conceptualization, and active experimentation.
• EL engages multiple parts of the brain, enhancing retention and learning. This approach aligns with brain-based design principles, which support active and reflective learning.
• The session includes examples from JMU X-Labs, an innovative, cross-disciplinary learning environment, where students’ reflections provide insights into their learning experiences, offering valuable qualitative data for assessment.

Creating a Data Model for EL:

• The presenters introduce a data model linking Kolb’s cycle stages to specific student outcomes:
  • Concrete Experience: Fosters student engagement, belonging, and retention.
  • Reflective Observation: Develops self-awareness, empathy, and holistic growth.
  • Abstract Conceptualization: Leads to knowledge acquisition and academic success.
  • Active Experimentation: Builds career readiness and applied skills.

Addressing Practical Challenges in EL Assessment:

• The session addresses the challenges of managing extensive data generated by EL assessments. Heinrich notes the importance of aligning EL assessment with disciplinary standards and institutional priorities. Effective EL assessment requires collaboration and communication across departments. Heinrich encourages institutions to consider their “ecosystem” and find new stakeholders who can benefit from EL data.

Call to Action

• Embrace the complexity of the EL ecosystem by considering stakeholder perspectives.
• Develop comprehensive assessment strategies to evaluate EL practices.

Most Important Ideas:

Questioning the Credit Hour: Dr. Blondin argues that while the credit hour is convenient for administrative purposes, it fails to measure genuine learning. She urges institutions to explore alternative methods that better reflect student competencies. "I've been associated with that battle for over 30 years… The credit hour drives financial aid, it drives transfer… but are we truly seeing results? Are we bearing fruit?"

Prior Learning Assessment: Recognizing and valuing the knowledge students bring from previous experiences—whether through formal education, work, or life—is essential for equity and student success. "I urge you to help develop and expand policies that acknowledge prior learning. I'm constantly amazed by what our students already know."

Stakeholder Involvement: Faculty, students, and employers should collaborate in defining and assessing student learning outcomes, ensuring relevance to the workforce and gaining buy-in from all involved. "What if this advisory group helped develop outcomes? Or, dare I say, students helped shape the outcomes for a program or course."

Seamless Transition Between Credit and Non-Credit: Dr. Blondin advocates for removing barriers between credit and non-credit programs, emphasizing that learning occurs in multiple forms and settings. "I envision a permeable boundary between credit and non-credit, allowing everything to flow together seamlessly. That’s my dream, and I have a year and a half to make it happen."

Flexibility in Learning: Acknowledging that students learn at different paces and in varied ways, institutions must offer flexibility in how, when, and where learning occurs. This includes embracing online learning, competency-based education, and other innovative approaches. "Flexibility in where, when, and how learning happens is a mindset shift for many—administrators, faculty, and staff alike."

Data-Driven Decision Making: Analyzing data from student learning assessments can reveal areas for improvement, inform program development, and guide resource allocation to support student success. "We've held data summits to share and learn, and we use this data to inform our budget. For instance, if students aren’t achieving outcomes in developmental math, we ask, ‘What can we do to help?’"

Key Quotes:

"It's not just our students who are stakeholders… some employers have lost confidence in our ability and are deciding to train in-house instead."

"No one ever, on the first day of any job, handed me a blue book."

"I've been in a lifelong battle with the credit hour and the Carnegie unit..."

Conclusion:

This discussion emphasizes the need for a paradigm shift in higher education assessment, moving from outdated, rigid models to more flexible, learner-centered approaches. By prioritizing student intent, accommodating diverse learning styles, and collaborating with key stakeholders, institutions can create a more equitable and effective learning environment that better prepares students for success in both the workforce and life.