Brain Inspired

Sanjeev and I discuss some of the progress toward understanding how deep learning works, specially under previous assumptions it wouldn't or shouldn't work as well as it does. Deep learning theory poses a challenge for mathematics, because its methods aren't rooted in mathematical theory and therefore are a "black box" for math to open. We discuss how Sanjeev thinks optimization, the common framework for thinking of how deep nets learn, is the wrong approach. Instead, a promising alternative focuses on the learning trajectories that occur as a result of different learning algorithms. We discuss two examples of his research to illustrate this: creating deep nets with infinitely large layers (and the networks still find solutions among the infinite possible solutions!), and massively increasing the learning rate during training (the opposite of accepted wisdom, and yet, again, the network finds solutions!). We also discuss his past focus on computational complexity and how he doesn't share the current neuroscience optimism comparing brains to deep nets.

• Sanjeev's website.
• His Research group website.
• His blog: Off The Convex Path.
• Papers we discuss
  • On Exact Computation with an Infinitely Wide Neural Net.
  • An Exponential Learning Rate Schedule for Deep Learning
• Related
  • The episode with Andrew Saxe covers related deep learning theory in episode 52.
  • Omri Barak discusses the importance of learning trajectories to understand RNNs in episode 97.
  • Sanjeev mentions Christos Papadimitriou.

Timestamps

What is creativity? How do we measure it? How do our brains implement it, and how might AI?Those are some of the questions John, David, and I discuss. The neuroscience of creativity is young, in its "wild west" days still. We talk about a few creativity studies they've performed that distinguish different creative processes with respect to different levels of expertise (in this case, in jazz improvisation), and the underlying brain circuits and activity, including using transcranial direct current stimulation to alter the creative process. Related to creativity, we also discuss the phenomenon and neuroscience of insight (the topic of John's book, The Eureka Factor), unconscious automatic type 1 processes versus conscious deliberate type 2 processes, states of flow, creative process versus creative products, and a lot more.

• John Kounios.
• Secret Chord Laboratories (David's company).
• Twitter: @JohnKounios; @NeuroBassDave.
• John's book (with Mark Beeman) on insight and creativity.
  • The Eureka Factor: Aha Moments, Creative Insight, and the Brain.
• The papers we discuss or mention:
  • All You Need to Do Is Ask? The Exhortation to Be Creative Improves Creative Performance More for Nonexpert Than Expert Jazz Musicians
  • Anodal tDCS to Right Dorsolateral Prefrontal Cortex Facilitates Performance for Novice Jazz Improvisers but Hinders Experts
  • Dual-process contributions to creativity in jazz improvisations: An SPM-EEG study.

Timestamps

Randal, Ken, and I discuss a host of topics around the future goal of uploading our minds into non-brain systems, to continue our mental lives and expand our range of experiences. The basic requirement for such a subtrate-independent mind is to implement whole brain emulation. We discuss two basic approaches to whole brain emulation. The "scan and copy" approach proposes we somehow scan the entire structure of our brains (at whatever scale is necessary) and store that scan until some future date when we have figured out how to us that information to build a substrate that can house your mind. The "gradual replacement" approach proposes we slowly replace parts of the brain with functioning alternative machines, eventually replacing the entire brain with non-biological material and yet retaining a functioning mind.

• Randal A Koene
  • Twitter: @randalkoene
  • Carboncopies Foundation.
  • Randal's website.
• Ken Hayworth
  • Twitter: @KennethHayworth
  • Brain Preservation Foundation.
    • Youtube videos.

Timestamps

Mark and I discuss his book, The Spike: An Epic Journey Through the Brain in 2.1 Seconds. It chronicles how a series of action potentials fire through the brain in a couple seconds of someone's life. Starting with light hitting the retina as a person looks at a cookie, Mark describes how that light gets translated into spikes,  how those spikes get processed in our visual system and eventually transform into motor commands to grab that cookie. Along the way, he describes some of the big ideas throughout the history of studying brains (like the mechanisms to explain how neurons seem to fire so randomly), the big mysteries we currently face (like why do so many neurons do so little?), and some of the main theories to explain those mysteries (we're prediction machines!). A fun read and discussion. This is Mark's second time on the podcast - he was on episode 4 in the early days, talking more in depth about some of the work we discuss in this episode!

• The Humphries Lab.
• Twitter: @markdhumphries
• Book: The Spike: An Epic Journey Through the Brain in 2.1 Seconds.
• Related papers
  • A spiral attractor network drives rhythmic locomotion.

Timestamps:

0:00 - Intro

Steve and I discuss his book, How to Motivate Your Students to Love Learning, which is both a memoir and a guide for teachers and students to optimize the learning experience for intrinsic motivation. Steve taught neuroscience and engineering courses while running his own lab studying the activity of live cultured neural populations (which we discuss at length in his previous episode). He relentlessly tested and tweaked his teaching methods, including constant feedback from the students, to optimize their learning experiences. He settled on real-world, project-based learning approaches, like writing wikipedia articles and helping groups of students design and carry out their own experiments. We discuss that, plus the science behind learning, principles important for motivating students and maintaining that motivation, and many of the other valuable insights he shares in the book.

• Potter Lab.
• Twitter: @stevempotter.
• The Book: How to Motivate Your Students to Love Learning.
• The glial cell activity movie.

0:00 - Intro

We made it to the last bit of our 100th episode celebration. These have been super fun for me, and I hope you've enjoyed the collections as well. If you're wondering where the missing 5th part is, I reserved it exclusively for Brain Inspired's magnificent Patreon supporters (thanks guys!!!!). The final question I sent to previous guests:

Do we already have the right vocabulary and concepts to explain how brains and minds are related? Why or why not?

Timestamps:

0:00 - Intro

In the 4th installment of our 100th episode celebration, previous guests responded to the question:

What ideas, assumptions, or terms do you think is holding back neuroscience/AI, and why?

As usual, the responses are varied and wonderful!

Timestamps:

0:00 - Intro

Part 3 in our 100th episode celebration. Previous guests answered the question:

Given the continual surprising progress in AI powered by scaling up parameters and using more compute, while using fairly generic architectures (eg. GPT-3):

Do you think the current trend of scaling compute can lead to human level AGI? If not, what's missing?

It likely won't surprise you that the vast majority answer "No." It also likely won't surprise you, there is differing opinion on what's missing.

Timestamps:

0:00 - Intro

In this 2nd special 100th episode installment, many previous guests answer the question: What is currently the most important disagreement or challenge in neuroscience and/or AI, and what do you think the right answer or direction is? The variety of answers is itself revealing, and highlights how many interesting problems there are to work on.

Timestamps:

0:00 - Intro

Brain Inspired turns 100 (episodes) today! To celebrate, my patreon supporters helped me create a list of questions to ask my previous guests, many of whom contributed by answering any or all of the questions. I've collected all their responses into separate little episodes, one for each question. Starting with a light-hearted (but quite valuable) one, this episode has responses to the question, "In the last five years, what new belief, behavior, or habit has most improved your career or well being?" See below for links to each previous guest. And away we go...

Timestamps:

0:00 - Intro