With the last episode in 2024, I dare to release an solo episode, summarizing my christmas research on the topics of

- Small Language models

- Agentic Systems

- Advanced Reasoning / Test time compute paradigm

I hope you find it interesting and useful!

All the best for 2025!

## AAIP Community

Join our discord server and ask guest directly or discuss related topics with the community.

https://discord.gg/5Pj446VKNU

## TOC

00:00:05 Intro

00:01:52 Part 1 - Small Language Models

00:20:16 Part 2 - Agentic Systems

00:36:16 Part 3 - Advanced Reasoning

00:58:08 Outro

## References

- Testing Qwen2.5 - https://huggingface.co/spaces/Qwen/Qwen2.5

- Qwen2.5 Technical report - https://arxiv.org/pdf/2412.15115

- Agents: https://www.superannotate.com/blog/llm-agents

- Scaling Test-time compute: https://arxiv.org/html/2408.03314v1

- Test time compute: https://huggingface.co/spaces/HuggingFaceH4/blogpost-scaling-test-time-compute

- O3 achieving 88% on ARC-AGI https://arcprize.org/blog/oai-o3-pub-breakthrough

- https://arxiv.org/html/2409.01374v1 - Human performance on ARC-AGI 76%

## Summary

Today we have as a guest Alexander Zehetmaier, co-founder of SunRise AI Solutions.

Alex will explain how SunRise AI is partnering with companies to navigate this challenging space, by providing their guidance, knowledge and network of experts to help companies apply AI successfully.

Alex will talk in detail about one of their Partners, Mein Dienstplan that is developing an Graph Neural Network based Solution that is generating complex work time tables. Scheduling a Timetable for a large number of employees and shifts is not an easy task, specially if one has to satisfy hard constraints like labor laws, and soft constraints like employee preferences.

Alex will explain in detail how they have developed a hybrid solution to use Graph Neural Network to create candidates that are validated and improved through heuristic based methods.

## AAIP Community

Join our discord server and ask guest directly or discuss related topics with the community.

https://discord.gg/5Pj446VKNU

## TOC

00:00:00 Beginning

00:02:23 Guest Introduction

00:04:19 SunRise AI Solutions

00:7:45 Mein Dienstplan

00:19:52 Building timetables with genAI

00:39:36 How SunRise AI can help startups

## References

Alexander Zehetmaier: https://www.linkedin.com/in/alexanderzehetmaier/

SunRise AI Solutions: https://www.sunriseai.solutions/

MeinDienstplan: https://www.meindienstplan.at/

As you surely know, OpenAI is not very open about how their systems works or how they build them. More importantly for most uses and business, OpenAI is agnostic about how users apply their services and how to make most out of the models multi-step "reasoning" capabilities .

As a stark contrast to OpenAI, today I am talking to Marius Dinu, the CEO and co-founder of the austrian startup extensity.ai. Extensity.ai as a company follows an open core model, building an open source framework which is the foundation for AI Agent systems that perform multi-step reasoning and problem solving, while generating revenue by providing enterprise support and custom implementation's.

Marius will explain how their Neuro-Symbolic AI Framework is combining the strengths of symbolic reasoning, like problem decomposition, explainability, correctness and efficiency with an LLM's understanding of natural language and their capability to operate on unstructured text following instructions.

We will discuss how their framework can be used to build complex multi-step reasoning workflows and how the framework works like an orchestrator and reasoning engine that applies LLM's as semantic parsers that at different decision points decide what tools or sub-systems to apply and use next. As well how in their research, they focus on ways to measure the quality and correctness of individual workflow steps in order to optimize workflow end-to-end and build a reliable, explainable and efficient problem solving system.

I hope you find this episode useful and interesting.

## AAIP Community

Join our discord server and ask guest directly or discuss related topics with the community.

https://discord.gg/5Pj446VKNU

## TOC

00:00:00 Beginning

00:03:31 Guest Introduction

00:08:32 Extensity.ai

00:17:38 Building a multi-step reasoning framework

00:26:05 Generic Problem Solver

00:48:41 How to ensure the quality of results?

01:04:47 Compare with OpenAI Strawberry

### References

Marius Dinu - https://www.linkedin.com/in/mariusconstantindinu/

https://www.extensity.ai/

Extensity.ai - https://www.extensity.ai/

Extensity.ai YT - https://www.youtube.com/@extensityAI

SymbolicAI Paper: https://arxiv.org/abs/2402.00854

Today on the podcast I have to pleasure to talk to Jules Salzinger, Computer Vision Researcher at the Vision & Automation Center of the AIT, the Austrian Institute of Technology.

Jules will share with us, his newest research on applying computer vision systems that analyze drone videos to perform remote plant phenotyping. This makes it possible to analyze plants growth, but as well how certain plant decease spreads within a field.

We will discuss how the diversity im biology and agriculture makes it challenging to build AI systems that generalize between plants, locations and time.

Jules will explain how in their latest research, they focus on performing experiments that provide insights on how to build effective AI systems for agriculture and how to apply them. All of this with the goal to build scalable AI system and to make their application not only possible but efficient and useful.

## TOC

00:00:00 Beginning

00:03:02 Guest Introduction

00:15:04 Supporting Agriculture with AI

00:22:56 Scalable Plant Phenotyping

00:37:33 Paper: TriNet

00:70:10 Major findings

### References

- Jules Salzinger: https://www.linkedin.com/in/jules-salzinger/

- VAC: https://www.ait.ac.at/en/about-the-ait/center/center-for-vision-automation-control

- https://www.ait.ac.at/en/about-the-ait/center/center-for-vision-automation-control

- AI in Agriculture: https://intellias.com/artificial-intelligence-in-agriculture/

- TriNet: Exploring More Affordable and Generalisable Remote Phenotyping with Explainable Deep Models: https://www.mdpi.com/2504-446X/8/8/407

## Summary

Today on the show I am talking to Proofreads CTO Alexandre Paris. Alex explains in great detail how they analyze digital books drafts to identify mistakes and instances within the document that dont follow guidelines and preferences of the user.

Alex is explaining how they fine-tune LLMs like, Mistrals 7B to achieve efficient resource usage and provide customizations and serve multiple uses cases with a single base model and multiple lora adapters.

We talk about the challenges and capabilities of fine-tuning, how to do it, when to apply it and when for example prompt engineering of an foundation model is the better choice.

I think this episode is very interesting for listeners that are using LLMs in a specific domain. It shows how fine-tuning a base model on selected high quality corpus can be used to build solutions outperform generic offerings by OpenAI or Google.

## AAIP Community

Join our discord server and ask guest directly or discuss related topics with the community.

https://discord.gg/5Pj446VKNU

## TOC

00:00:00 Beginning

00:02:46 Guest Introduction

00:06:12 Proofcheck Intro

00:11:43 Document Processing Pipeline

00:26:46 Customization Options

00:29:49 LLM fine-tuning

00:42:08 Prompt-engineering vs. fine-tuning

### References

https://www.proofcheck.io/

Alexandre Paris - https://www.linkedin.com/in/alexandre-paris-92446b22/

Today I am talking to Philip Winter, researcher at the Medical Imaging group of the VRVis, a research center for virtual realities and visualizations.

Philip will explain the benefits and challenges in continual learning and will present his recent paper "PARMESAN: Parameter-Free Memory Search and Transduction for Dense Prediction Tasks". Where he and his colleagues have developed a system that uses a frozen hierarchical feature extractor to build a memory database out of the labeled training data. During inference the system identified training examples similar to the test data and prediction is performed through a combination of parameter-free correspondence matching and message passing based on the closes training datapoints.

I hope you enjoy this episode and will find it useful.

## AAIP Community

Join our discord server and ask guest directly or discuss related topics with the community.

https://discord.gg/5Pj446VKNU

## TOC

00:00:00 Beginning

00:03:04 Guest Introduction

00:06:50 What is continual learning?

00:15:38 Catastrophic forgetting

00:27:36 Paper: Parmesan

00:40:14 Composing Ensembles

00:46:12 How to build memory over time

00:55:37 Limitations of Parmesan

### References

Philip Winter - https://www.linkedin.com/in/philip-m-winter-msc-b15679129/

VRVIS - https://www.vrvis.at/

PARMESAN: Parameter-Free Memory Search and Transduction for Dense Prediction Tasks - https://arxiv.org/abs/2403.11743

Continual Learning Survey: https://arxiv.org/pdf/1909.08383

## Summary

AI is currently dominated by Deep Learning and Large Language Models, but there is other very interesting research that has the potential to have great impact on our lives in the future; one of them being Quantum Machine Learning (QML)

Today on the podcast, I am talking to Christa Zoufal, Quantum Machine Learning Researcher at IBM.

Christa will explain how Quantum Computing (QC) and Quantum Machine Learning relates to classical computing (CC).

We will discuss Qbit's, the fundamental information storage and processing unit of quantum computers that in comparison with bits in a classical computers, not only store a single state of zero or one, but a superposition of the two.

Christa will explain how Quantum algorithms are created by building quantum circuits that operate on those qbits. We will talk about the challenges when operating on those physical qbits that need to be kept at very low temperatures in order to operate without noise perturbing their states, and why one combines multiple physical qbits to a single logical qbit in order to perform error correction.

How current quantum circuits are limited in the number of logical qbits they can contain and the number of operations; in the sense of the size of an algorithm, they can perform.

Why these properties make QC is the wrong choice for BigData applications, but QM could in the future be used for applications where very little data is available and quantum algorithms exist that outperform their known classical counterparts.

## AAIP Community

Join our discord server and ask guest directly or discuss related topics with the community.

https://discord.gg/5Pj446VKNU

## TOC

00:00:00 Beginning

00:03:21 Guest Introduction

00:07:08 Quantum Computer (QC) & Quantum Machine Learning (QML)

00:14:50 Difference between classical computing and QC

00:24:59 What kind of programs one can run on a QC

00:38:43 Examples of Quantum Machine Learning

00:49:25 Current applications of QC

## References

Christa Zoufal: https://www.linkedin.com/in/christa-zoufal/

IBM Quantum Learning: https://learning.quantum.ibm.com/course/basics-of-quantum-information

Quantum machine learning course: https://github.com/Qiskit/textbook/tree/main/notebooks/quantum-machine-learning

Scott Aaronson @ Lex Podcast https://www.youtube.com/watch?v=nK9pzRevsHQ

State of the Art Bottlenecks: https://arxiv.org/abs/2312.09121

Hello and welcome back to the AAIP

This is the second part of my interview with Eldar Kurtic and his research on how to optimiz inference of deep neural networks.

In the first part of the interview, we focused on sparsity and how high unstructured sparsity can be achieved without loosing model accuracy on CPU's and in part on GPU's.

In this second part of the interview, we are going to focus on quantization. Quantization tries to reduce model size by finding ways to represent the model in numeric representations with less precision while retaining model performance. This means that a model that for example has been trained in a standard 32bit floating point representation is during post training quantization converted to a representation that is only using 8 bits. Reducing the model size to one forth.

We will discuss how current quantization method can be applied to quantize model weights down to 4 bits while retaining most of the models performance and why doing so with the models activation is much more tricky.

Eldar will explain how current GPU architectures, create two different type of bottlenecks. Memory bound and compute bound scenarios. Where in the case of memory bound situations, the model size causes most of the inference time to be spend in transferring model weights. Exactly in these situations, quantization has its biggest impact and reducing the models size can accelerate inference.

Enjoy.

## AAIP Community

Join our discord server and ask guest directly or discuss related topics with the community.

https://discord.gg/5Pj446VKNU

### References

Eldar Kurtic: https://www.linkedin.com/in/eldar-kurti%C4%87-77963b160/

Neural Magic: https://neuralmagic.com/

IST Austria Alistarh Group: https://ist.ac.at/en/research/alistarh-group/

Hello and welcome back to the AAIP

If you are an active Machine Learning engineer or are simply interested in Large Language models, I am sure you have seen the discussions around quantized models and all kind of new frameworks that have appeared recently and achieve astonishing inference performance of LLM's on consumer devices.

If you are curious how modern Large Language Models with their billions of parameters can run on a simple laptop or even an embedded device, than this episode is for you.

Today I am talking to Eldar Kurtic, researcher in the Alistarh group at the IST in lower Austrian and senior research engineer at the American startup Neural Magic.

Eldar's research focuses on optimizing Inference of Deep Neural Networks. On the show he is going to explain in depth show sparsity and quantization works, and how they can be applied to accelerate inference of big models, like LLM's on devices with limited resources.

Because of the length of the interview, I decided to split it into two parts.

This one, the first part, is going to focus on sparsity to reduce model size and enable faster inference by reducing the amount of memory and compute that is needed to store and run models.

The second part is going to focus on quantization as a mean to find representations of models with lower numeric precision that require less memory to store and process, while retaining accuracy.

In this first part about sparsity, Eldar will explain fundamental concepts like structured and unstructured sparsity. How and why they work and how currently we can achieve performant inference of unstructured sparsity only on CPU's and far less on GPU's.

We will discuss how to achieve crazy numbers of up to 95% unstructured sparsity while retaining the accuracy of models, but why it is difficult to leverage this under quoutes, reduction in model size, to actually accelerate model inference.

Enjoy.

## AAIP Community

Join our discord server and ask guest directly or discuss related topics with the community.

https://discord.gg/5Pj446VKNU

### References

Eldar Kurtic: https://www.linkedin.com/in/eldar-kurti%C4%87-77963b160/

Neural Magic: https://neuralmagic.com/

IST Austria Alistarh Group: https://ist.ac.at/en/research/alistarh-group/

## Summary

Today on the show I am talking to Veronika Vishnevskaia. Solution Architect at ONTEC where she specialises in building RAG based Question-Answering systems.

Veronika will provide a deep dive into all relevant steps to build a Question-Answering system. Starting from data extraction and transformation, followed by text embedding, chunking and hybrid retrieval to strategies and last but not least methods to mitigate hallucinations of LLMs during the answer creation.

## AAIP Community

Join our discord server and ask guest directly or discuss related topics with the community.

https://discord.gg/5Pj446VKNU

## TOC

00:00:00 Beginning

00:03:33 Guest Introduction

00:08:51 Building Q/A Systems for businesses

00:16:27 RAG: Data extraction & pre-processing

00:26:08 RAG: Chunking & Embedding

00:36:13 RAG: Information Retrieval

00:48:59 Hallucinations

01:02:21 Future RAG systems

## Sponsors

- Quantics: Supply Chain Planning for the new normal - the never normal - https://quantics.io/

- Belichberg GmbH: Software that Saves the Planet: The Future of Energy Begins Here - https://belichberg.com/

### References

Veronika Vishnevskaia - https://www.linkedin.com/in/veronika-vishnevskaia/

Ontec - www.ontec.at

Review Hallucination Mitigation Techniques: https://arxiv.org/pdf/2401.01313.pdf

Aleph-Alpha: https://aleph-alpha.com/de/technologie/