Sign up to save your podcasts
Or
Share Mamba 3
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Mamba 3
In this episode:
• Welcome and the Mamba Lineage: Professor Norris and Linda introduce Mamba-3, discussing the shift towards inference-time efficiency and the need for sub-quadratic models.
• Exponential-Trapezoidal Discretization: Linda explains how Mamba-3 upgrades to a second-order trapezoidal rule, creating an implicit convolution that removes the need for explicit causal convolution layers.
• Complex-Valued States and the RoPE Trick: The hosts discuss the limitations of real-valued SSMs in state tracking and how Mamba-3 uses a data-dependent RoPE trick to efficiently implement complex-valued rotational dynamics.
• MIMO and Hardware Arithmetic Intensity: Linda details how Mamba-3 uses a Multi-Input, Multi-Output formulation to increase arithmetic intensity, overlaying free compute on top of memory bottlenecks during decoding.
• Performance Results and Wrap Up: Professor Norris is convinced by the empirical results, noting how Mamba-3 advances the Pareto frontier by matching Mamba-2's performance with half the latency.
• Welcome and the Mamba Lineage: Professor Norris and Linda introduce Mamba-3, discussing the shift towards inference-time efficiency and the need for sub-quadratic models.
• Exponential-Trapezoidal Discretization: Linda explains how Mamba-3 upgrades to a second-order trapezoidal rule, creating an implicit convolution that removes the need for explicit causal convolution layers.
• Complex-Valued States and the RoPE Trick: The hosts discuss the limitations of real-valued SSMs in state tracking and how Mamba-3 uses a data-dependent RoPE trick to efficiently implement complex-valued rotational dynamics.
• MIMO and Hardware Arithmetic Intensity: Linda details how Mamba-3 uses a Multi-Input, Multi-Output formulation to increase arithmetic intensity, overlaying free compute on top of memory bottlenecks during decoding.
• Performance Results and Wrap Up: Professor Norris is convinced by the empirical results, noting how Mamba-3 advances the Pareto frontier by matching Mamba-2's performance with half the latency.
View all episodes
By Mechanical DirkIn this episode:
• Welcome and the Mamba Lineage: Professor Norris and Linda introduce Mamba-3, discussing the shift towards inference-time efficiency and the need for sub-quadratic models.
• Exponential-Trapezoidal Discretization: Linda explains how Mamba-3 upgrades to a second-order trapezoidal rule, creating an implicit convolution that removes the need for explicit causal convolution layers.
• Complex-Valued States and the RoPE Trick: The hosts discuss the limitations of real-valued SSMs in state tracking and how Mamba-3 uses a data-dependent RoPE trick to efficiently implement complex-valued rotational dynamics.
• MIMO and Hardware Arithmetic Intensity: Linda details how Mamba-3 uses a Multi-Input, Multi-Output formulation to increase arithmetic intensity, overlaying free compute on top of memory bottlenecks during decoding.
• Performance Results and Wrap Up: Professor Norris is convinced by the empirical results, noting how Mamba-3 advances the Pareto frontier by matching Mamba-2's performance with half the latency.
• Welcome and the Mamba Lineage: Professor Norris and Linda introduce Mamba-3, discussing the shift towards inference-time efficiency and the need for sub-quadratic models.
• Exponential-Trapezoidal Discretization: Linda explains how Mamba-3 upgrades to a second-order trapezoidal rule, creating an implicit convolution that removes the need for explicit causal convolution layers.
• Complex-Valued States and the RoPE Trick: The hosts discuss the limitations of real-valued SSMs in state tracking and how Mamba-3 uses a data-dependent RoPE trick to efficiently implement complex-valued rotational dynamics.
• MIMO and Hardware Arithmetic Intensity: Linda details how Mamba-3 uses a Multi-Input, Multi-Output formulation to increase arithmetic intensity, overlaying free compute on top of memory bottlenecks during decoding.
• Performance Results and Wrap Up: Professor Norris is convinced by the empirical results, noting how Mamba-3 advances the Pareto frontier by matching Mamba-2's performance with half the latency.