How is ACT using NLP and GPT2 for Sphinx and other automatic content generation?

In this episode of the Navigator podcast, we discuss Natural Language Processing with Yuchi Huang, a Senior Manager in the Artificial Intelligence and Machine Learning team at ACTNext. Yuchi takes us through some of the pitfalls and practices of using machine learning for text content analysis and generation. 

A transcript is available at:  https://actnext.org/research-and-projects/podcast-ep-11-nlp-gpt2-sphinx/

The views and opinions expressed in this podcast are those of the authors only and do not necessarily reflect the official policy or position of ACT, Inc.

How is ACTNext using AI and ML to transform the learning, measurement, and navigation landscape?

In this episode of the Navigator podcast, we discuss some of the artificial intelligence (AI) timeline, past, present and future, with Saad Khan , director of AI and machine learning for ACTNext. We discuss "black box" perceptions of AI research and we'll learn a little about how the AI/ML team is using deep learning to create new tools for measurement and learning.

A transcript is available at:  https://actnext.org/research-and-projects/podcast-ep10-aiml-saad-khan/ ‎

This episode is brought to you by Sphinx! Are you looking for automated passage generation? Sphinx encodes text data to automate summarization, sentence recommendation, paraphrasing and more. Sphinx, the natural language processor for creating English passages quickly. Find out more about Sphinx at ACTNext.org.

The views and opinions expressed in this podcast are those of the authors only and do not necessarily reflect the official policy or position of ACT, Inc.

How do babies learn to crawl, walk, and run? In this podcast, we talk about learning with Dr. Vanessa Simmering. She's a Senior Research Scientist in Learning Solutions and studies learning and development.

Last year, she wrote about how babies learn to walk. In "Lessons from Learning to Walk," Simmering shared three ideas about the foundations of learning and how they relate to "higher-order" skills like social interactions and language development.

For this podcast, we also discuss the Stroop Color and Word Test (try it here), crystallized versus fluid intelligence, and the Dynamic Systems Theory of learning.

Joining us on the podcast are Gunter Maris (listen to his interview on The Wiring of Intelligence), Michael Yudelson, Kristin Stoeffler, and Saad Khan. They share their views on learning and intelligence.

The views and opinions expressed in this podcast are those of the authors only and do not necessarily reflect the official policy or position of ACT, Inc.

Visit https://actnext.org/research-and-projects/podcast-ep-9-what-is-learning/ to read a transcript of the show.

In this interview, Gunter Maris explains the theory behind The Wiring of Intelligence, a network model of learning. He also quotes Leo Tolstoy and references Sherlock Holmes.

Maris describes the actions of a “crazy electrician” who wires the connections in our brains. The network theory of intelligence developed by Maris (with colleagues Alexander Savi, Maarten Marsman, and Han van der Maas) is based on the idea of a house with interconnected electrical sockets in every room that represent synaptic knowledge. Language, speaking, and reading have tight connections in the brain while mathematics skills are grouped together in another room.

For example, in the mind of a statistician, math connections can be tight and dense. Maris’ primarily Dutch-speaking brain has many “wired connections” pertaining to English terminology around psychometrics but probably fewer that cover cooking or gardening.

Connections from disparate rooms build measurable broad intelligence. The model helps explain the positive manifold theory (also called the g factor or general intelligence factor) first developed by Charles Spearman in the early 1900s.

Near the end of the interview, Maris explains how the simple model of networked intelligence is being overlaid with ACT’s Holistic Framework.

The views and opinions expressed in this podcast are those of the authors only and do not necessarily reflect the official policy or position of ACT, Inc.

Read the transcript: https://actnext.org/research-and-projects/podcast-ep8-wiring-of-intelligence/ 

A lot of research goes into ACT tests. Every question begins with psychometric grounding. Items are field tested and answers are checked for validity. Assessments must measure ability and not reward random guesswork on the part of the test-taker. In addition to accuracy, test items are also extensively reviewed for fairness. Subject matter experts (SMEs) develop questions in math, reading and comprehension, science, and graphic literacy.

But that can take time, so test developers have created some ways to streamline the assessment pipeline.

In this episode of ACTNext Navigator podcast, we discuss the history of automated item generation (AIG) at ACT with Rick Meisner. He's been with ACT for over 30 years and developed some of the first AIG content for math using BASIC programming language. Meisner also holds several patents related to AIG and automated scoring.

Later in the show, we'll hear from Ian MacMillan and Brad Bolender. They've developed AIG software for WorkKeys graphic literacy assessment (AIGL) and passage organizing and extraction (POE) respectively.

Each presented a poster at the 2019 Education Technology and Computational Psychometrics Symposium research poster and tech demo reception on October 9, 2019.

The views and opinions expressed in this podcast are those of the authors only and do not necessarily reflect the official policy or position of ACT, Inc.

Read a transcript of the show at  https://actnext.org/research-and-projects/podcast-ep7-aig-aigl-poe/

In this episode of ACTNext Navigator podcast, we’ll go under the hood of ACT's automated essay scoring engine, CRASE+.

Our guests are Erin Yao and Scott Wood. They’ve been working for many years on CRASE+, a product acquired in 2014 when ACT purchased Pacific Metrics. CRASE+ is a writing assessment tool that begins with human graders to develop a rubric. Data from human graders is used to train the automatic grading on a large scale.

We discuss CRASE+, automated scoring, the challenges of automated scoring using natural language processing, and how to address the biases in human graders.

Scott has a book chapter coming out in February 2020, "Public perception and communication around automated essay scoring," in the Handbook of Automated Scoring: Theory into Practice.

Read a transcript of the show at:  https://actnext.org/research-and-projects/navigator-podcast-ep-6-crase-plus/ 

The views and opinions expressed in this podcast are those of the authors only and do not necessarily reflect the official policy or position of ACT, Inc.

In this episode, guest Jay Thomas, Senior Assessment Designer at ACT, discusses how eye-tracking can go beyond psychometrics to evaluate and validate assessment and testing.

There are several parts to eye-tracking. The first is saccades, the rapid eye movements made during reading that don't always follow left-to-right. It's estimated that humans make over 100,000 saccades daily, including during rapid eye movement (REM) sleep cycles when we're dreaming.

Other obvious parts are fixations, which are fixed moments when the eyes are concentrated on a particular location, and blinking.

An important piece of eye-tracking that has been around for decades but has only recently become easier to study is pupillometry, the measurement of pupil dilation. Dilation can be interpreted to measure thinking and cognitive effort, in particular the size changes and acceleration of dilation. Pupil dilation is also an autonomic bodily response. Unlike breathing or heart rates, which can be modified to trick lie detectors you cannot fake or consciously control dilating your pupils.

Thomas says that eye-tracking goes beyond psychometrics to validate tests and give insight into cognitive thought processes.

With Langenfeld, Zhu, and Morris, he created a formula to measure Total Cognitive Effort (TCE) for test items. Jay walks us through the TCE formula in the podcast.

Mr. Thomas was a science teacher for 19 years and also worked for Kaplan Test Prep before coming to ACT.

In this episode, guests Yigal Rosen, Kristin Stoeffler, and Laurel Ozersky talk about the cross-cutting capabilities (CCCs) of creative and computational thinking. CCCs are part of ACT's Holistic Framework for college and career readiness.

Yigal and his group are leading some of the research and development in support of ACT’s transition to a learning company.

Kristin is a creative thinking specialist. She’s working on the 2021 PISA assessment of creative thinking and she’s also developing ways to foster students’ creative thinking skills for EDU2050. She is also an artist.

Laurel is a former math and science teacher. Her work on EDU2050 is about how to develop computational thinking for students. As she’ll describe, computational thinking is not just for computer programmers, but can be used in historical research for example or solving societal problems like food waste.

In 2021, ACT’s creative thinking assessment, the first of its kind will be a part of the international test called PISA. Students from about 70 countries will take PISA which assesses 15 year old students in mathematics, science, and reading and a fourth, innovative domain that changes with each testing cycle. The ACT-developed creative thinking assessment will be the fourth element in the 2021 testing cycle. 

In addition to PISA, Yigal's Learning Solutions team is developing its own course work, called EDU2050.

You can follow Yigal and Kristin on Twitter.

A full transcript is available at: https://actnext.org/research-and-projects/navigator-podcast-ep-4-ccc/

We go in-depth on the CPSx game study that measures collaborative problem-solving. In this episode, Saad Kahn and Dave Edwards discuss the game, "Crisis in Space," they use to gather data from players. The data comes from natural language processing, eye-tracking, and other records.

Saad introduced Dave's presentation at an ACT Power Hour lunch.

This podcast includes an audio clip from KCRG TV's news story about the CPSx study in Iowa City and Cedar Rapids schools. The clip is used with their permission. Thank you!

In this episode of ACTNext Navigator, we discuss the intra-ACT collaboration between ACTNext and the ACT Center for Equity in Learning (CEL).

Lew Montgomery is the Director for the CEL and he arranged 100 students to participate in the Collaborative Problem-Solving game, "Crisis in Space," in May 2019. Montgomery lined up students from middle and high schools in Iowa City, Coralville, North Liberty and Cedar Rapids for researchers Pravin Chopade and Dave Edwards.

You can learn more about the game here. The next Navigator podcast, Episode 3, will examine the game design and research that will come out of the study.

You can follow Lew, Pravin and Dave on Twitter.