The Lindahl Letter

You might well be aware that multimodal machine learning (MMML) is a slice of the machine learning universe. Even typing the title of this post was challenging. I really wanted to type multi model vs. modal with an “E” before the last “L” instead of an “A” like the actual wording requires in this case. The definition of multimodal is really direct and does not include a ton of mystery. The word is used to describe something with more than one mode. You might end up quickly running down the path of multimodal deep learning to help describe it. As a person is capable of taking in the world with multiple senses and converting those signal paths into one stream for analysis. A multimodal deep learning network could be built to evaluate multi types of inputs. That really becomes a lot more complicated than it sounds within the modeling space. Our current class of models does not demonstrate the practical skill a person would at differentiating senses and understanding them in real time [1].

Within the machine learning space the first generation of models were really focused on achieving very specific tasks. They received highly defined training on very specific data problems with very curated training datasets. At some point, machine learning models or builds will need to be able to take on more than one type of tasking. I think that is a really fascinating part of machine learning to study. Expanding the capabilities and ultimately what is possible changes future trajectories. I ended up reading a paper titled, “Recent Advances and Trends in Multimodal Deep Learning: A Review,” from 2021 [2]. It was pretty much the exact paper I was looking to read to really dig into the topic under consideration today. That paper really focused on video and language examples which really put things in context. 

One of the things that I realized during the course of my research on this topic was that a treasure trove of recorded lectures exist on YouTube. A lot of them are related to computer science, machine learning, and artificial intelligence. That is a really good thing if you were trying to put together a syllabus geared toward providing an introduction to machine learning. The video that I spent the most time watching this time around was from Victoria Dean’s, “MIT 6.S191 Lecture 5 Multimodal Deep Learning,” lecture from 2017 [3]. Somebody who was willing to put in the work to curate the content could easily pull together all the lectures necessary from a multitude of different sources. 

After reviewing the current trends in multimodal deep learning my interests shifted to one particular topic related to automated ICD coding. A quick Google Scholar search for "automated icd coding" in quotes and multimodal machine learning produced a ton of interesting results [4]. My search returned 94 results which was pretty surprising given the targeted nature of the terms used. Some of the articles were related to feature extraction and others were really keyed in on trying to get to the point of working with an ICD code or completing the action of coding content. One of the results that dug into automated ICD coding caught my attention was titled, “A Deep Learning Framework for Automated ICD-10 Coding” [5]. Ultimately the automation would help physicians be more productive and accurate in working toward a diagnosis. That seems like a noble effort to assist physicians and help patients.

Links and thoughts:

“MIT 6.S191 Lecture 5 Multimodal Deep Learning”

“Steam Deck: What I Didn't Say In My Review - WAN Show February 25, 2022”

Top 5 Tweets of the week:

Footnotes:

[1] https://towardsdatascience.com/multimodal-deep-learning-ce7d1d994f4

[2] https://arxiv.org/pdf/2105.11087.pdf 

[3]

[4] https://scholar.google.com/scholar?hl=en&as_sdt=0,6&as_vis=1&qsp=6&q=%22automated+icd+coding%22+multimodal+machine+learning&qst=br  

[5] https://books.google.com/books?id=81A2EAAAQBAJ&lpg=PA347&ots=2k6gK3lL_d&dq=%22automated%20icd%20coding%22%20multimodal%20machine%20learning&lr&pg=PA347#v=onepage&q=%22automated%20icd%20coding%22%20multimodal%20machine%20learning&f=false 

What’s next for The Lindahl Letter?

* Week 60: General artificial intelligence

* Week 61: AI network platforms

* Week 62: Touching the singularity

* Week 63: Sentiment and consensus analysis

* Week 64: Language models revisited

I’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed reading this content, then please take a moment and share it with a friend.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

Let me start with a quick update on something Substack related. Throughout the last month I attended an invitation-only weekly forum hosted by Substack called Substack Go where writers were brought together to help encourage each other. For me writing on Substack is generally a solitary activity that happens mostly on Saturday morning. That effort is followed by a Sunday morning of editing and expansion. Finally the rest of the week is spent tinkering to get to a final post that goes out on Friday. A lot of the content being presented during Substack Go is about how to craft and write a newsletter which is interesting to hear about. This particular series has reached 58 weeks and the weekly rhythm of writing and publishing seems to be working. Even my new podcast variation of this series has been working. 

The two topics that I have spent the most time writing about within the machine learning space are strategy (ROI/KPIs/budgeting) and training efforts like bringing somebody or a team up to speed. Training is wholesale an element that should be a part of any organizational ML strategy. This post certainly could start by harkening back to Week 4 that appeared on February 19, 2021. During that week I tackled the topic, “Have an ML strategy… revisited.” Contained within that analysis were two questions targeting “What exactly is an ML strategy?” and “Do you even need an ML strategy?” The answer to that pivotal question is still of course an organization should have a machine learning strategy. Beginning with the end in mind and having that direction tied to budget level KPI is really a minimum standard at this point. 

Within this analysis the question really is if teaching or training machine learning skills is the right path to take. Inherent within that analysis has to be a question about the reason for wanting to learn the machine learning skills or acquire that specific knowledge. Bringing team members up to speed is certainly a reason to champion the training process for introducing machine learning skills. On the other side of that consideration is a scenario where learning about the foundations and building up knowledge inspired the need to pick up machine learning skills. In that case the collecting of knowledge simply compels more collecting. Continuing this general line of thought about training; I am going to break up the next two sections into concentrated coverage of individual enrichment followed by coverage of team building. 

Individual enrichment as a reason for advancing machine learning skills covers a significant, but not overwhelming number of people that comprise this use case universe. Students and lifelong learners alike see the world of machine learning and either have a fear of missing out or are wondering what the hype is all about. The hype in the marketplace is all about a new technology that can be applied to business use cases. People look at potential and are willing to make a leap to what it could possibly mean for them or what it could possibly do for the organization. Within the machine learning space going from model to production is a journey. Even the process of keeping a model well tuned and running in production is a consideration that means that the journey never really ends. 

Team building or more to the point the individual practitioners within teams or solo efforts that advance machine learning within organization certainly comprise the overwhelming bulk of the people contained in this use case universe. People at all sorts of different skill levels are working and learning within the machine learning space. Very few of them have the knowledge, skills, and abilities to create a machine learning system like TensorFlow or PyTorch. The preponderance of work in the machine learning space is done on the shoulders of giants based on using frameworks and systems that were already built. However, we are starting to see a lot of the newest innovations coming out of organizations like Hugging Face and EleutherAI. The developers with the skills to produce those foundational platforms and tooling that people are going to be using in the machine learning space will continue to be a tier above the much larger pool of people using the products. That asymmetrical dynamic is unlikely to change. 

To end this post, I thought I would provide you with a short update on my audio editing strategy. Within the Audacity software I’m using three different effects to clean up my podcast audio. First, a noise reduction effect which I have used for a long time in the product is kicked off and executed. The noise gate effect would probably catch it all, but I still run it first. Second, a loudness normalization effect is run which is key for any podcast as it makes sure nothing shocking in terms of a volume spike happens to the listener. Third, I have started using the built in noise gate effect function to trim out background noises and any breaths that might remain. Between those three effects in Audacity my podcast audio is being edited for your listening pleasure.

Links and thoughts:

From Machine Learning Street Talk, “#063 - Prof. YOSHUA BENGIO - GFlowNets, Consciousness & Causality”

“Valve's Making Everyone Else Look Bad - WAN Show February 18, 2022”

“AI Show Live | Ep 52 | Analyze unstructured docs and more with Azure Form Recognizer”

Top 5 Tweets of the week:

Footnotes: None.

What’s next for The Lindahl Letter?

* Week 59: Multimodal machine learning revisited

* Week 60: General artificial intelligence

* Week 61: AI network platforms

* Week 62: Touching the singularity

* Week 63: Sentiment and consensus analysis

I’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed reading this content, then please take a moment and share it with a friend.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

This is the third week of including audio integration as a part of the weekly Substack post for The Lindahl Letter. Recording a reading of the first section of the post that contains the main content is easy enough to achieve using Audacity with a basic use of both the noise reduction and loudness normalization effects. Between those two effects it will keep a noise floor in place to keep anything from being too loud during the recording. For those of you who are curious to complete the recordings I’m using a Blue Microphones Yeti X professional microphone on my desk in my office. It has proven to be a little more dynamic than the Samson microphone that normally accompanies my audio recordings. 

My very high level introduction to the creation of a machine learning syllabus would center on 8 core topics:

* Bayesian optimization

* Deep learning

* Classic ML algorithms

* Classic neural networks

* Neuroscience

* Reinforcement learning

* Graph neural networks

* Ethics (fairness, bias, privacy)

I would probably break out the systems and tools part of the conversation to a separate course or keep that content included as supplemental reading. Getting to the point of being able to work toward an 8 topic introduction to machine learning involved really digging into a few other machine learning course introductions. I’m going to share links to six of them with you today, but that should not be considered an exhaustive list. It is just a list of content that caught and held my attention enough that I felt curating it in a list would be helpful. 

* Stanford CS229: Machine Learning [1]

* MIT Opencourseware: Introduction to Machine Learning [2]

* NYU Introduction to Machine learning [3]

* Tufts CS COMP 134 Intro ML [4]

* Berkeley CS 189/289A Introduction to Machine Learning [5]

* Washington: CSE/STAT 416, Summer 2020: Introduction to Machine Learning [6]

Taking a step back and thinking about the most direct solution moving forward on this path I would probably just assign a class or at least highly recommend the students to read a book called, “Artificial Intelligence: A Modern Approach,” by Stuart Russell and Peter Norvig [7]. The current version appears to be the 4th edition [8]. My bookshelf has a copy of the 3rd edition and that is the one that I picked up and started reading a few years ago. Oddly enough during the course of reading about the 4th edition the Pearson website suggested that I read, “Quantum Computing Fundamentals,” [9]. That would take this dialogue into another direction. At some point, that pivot might happen for a few weeks. I did check eBay to see if anybody was selling a used copy of that book on quantum computing. Nobody has let go of that tome of quantum goodness just yet. At some point, I’m sure one will show up for resale or my research interests will veer that direction enough to justify the price.

Getting back to the main point of talking about that book by Russell and Norvig. Machine learning elements always feel like a foundational build up to work with more advanced topics in artificial intelligence. Machine learning models that are able to handle a multitude of problems are potentially becoming more likely. Week 79 to 86 of this newsletter will be devoted to creating outlines or a first draft of what the 8 core topics in my proposed syllabus would need to contain. I’m going to try to intermix a lot of artificial intelligence related observations into the machine learning focused content for each of those Substack posts. Bringing the two content sets together in a very intentional way should provide a solid written outcome.

Links and thoughts:

“Newegg... More Like Rotten Egg! - WAN Show February 11, 2022”

“AI Show Live - Episode 51 - NVIDIA DeepStream development with Microsoft Azure”

 “HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learning (w/ Author)”

 “[ML News] Uber: Deep Learning for ETA | MuZero Video Compression | Block-NeRF | EfficientNet-X”

 Top 5 Tweets of the week:

 

Footnotes:

[1] Stanford University. (n.d.). Syllabus and Course Schedule. CS229. Retrieved February 13, 2022, from https://cs229.stanford.edu/syllabus.html 

[2] Massachusetts Institute of Technology. (n.d.). Introduction to Machine Learning. MIT OPENCOURSEWARE. Retrieved February 13, 2022, from https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-036-introduction-to-machine-learning-fall-2020/  

[3] Mehryar Mohri. (n.d.). Introduction to Machine Learning. Retrieved February 13, 2022, from https://cs.nyu.edu/~mohri/mlu11/ 

[4] Hughes, M. (n.d.). Introduction to Machine Learning. Syllabus. Retrieved February 13, 2022, from https://www.cs.tufts.edu/comp/135/2020f/index.html 

[5] Shewchuk, J. (n.d.). CS 189/289A Introduction to Machine Learning. Retrieved February 13, 2022, from https://people.eecs.berkeley.edu/~jrs/189/  

[6] Swamy, V. (n.d.). CSE/STAT 416, Summer 2020: Introduction to Machine Learning. Retrieved February 13, 2022, from https://courses.cs.washington.edu/courses/cse416/20su/ 

[7] Russell, S., & Norvig, P. (n.d.). Artificial Intelligence: A Modern Approach, 4th US ed. Retrieved February 13, 2022, from http://aima.cs.berkeley.edu/

[8] Link to buy the book: https://www.pearson.com/store/p/artificial-intelligence-a-modern-approach/P100000291856/9780137505135 

[9] Link to buy the book: https://www.pearson.com/store/p/quantum-computing-fundamentals/P100002994167/9780137460328#

What’s next for The Lindahl Letter?

* Week 58: Teaching or training machine learning skills

* Week 59: Multimodal machine learning revisited

* Week 60: General artificial intelligence

* Week 61: AI network platforms

* Week 62: Touching the singularity

I’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed reading this content, then please take a moment and share it with a friend.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

This topic has been added to my list of academic papers that I need to one day complete. Beyond that scholarly realization the core of this topic really has me conflicted about how to approach consideration about what a national AI strategy really entails. I’m fundamentally questioning what the trajectory of the strategic action would be for the different players involved. Within that conflict I have to wonder if the corporate players within a nation taking action would be inclusive of the strategy or only the official government plan should be included. Those sets of actors do not have to work together. It is an asymmetrical dynamic where one group needs the other a lot more to implement a strategy. To that end I set out to find government documents on the subject. Finding a national AI strategy document in English for a variety of different nations is actually harder to achieve from basic searching than I expected at the start of this journey. It might be an opportunity for an industrious or benevolent researcher to index all of the current plans and build a dataset for comparative analysis. 

Getting to the National AI Strategy for the United Kingdom online is really easy [1]. The PDF of that content comes in at a slim 35 pages. It is well written and easy to read. Clearly a 10 year plan exists to make sure Britain is competitive in the AI space. The only really big problem with that is that several of the key corporations in the artificial intelligence space are located several thousand miles away. Getting to the Artificial Intelligence Strategy document from the United States Department of Health and Human Services was fairly easy to achieve. It was published in January of 2021 and is only 8 pages in PDF format [2]. Reading both of those reports was pretty easy and only took a few minutes. From that point forward I started to wonder about the possibility of reading some comparative reports that might have been recently widely published. One of those reports that caught my attention was from the Brookings Institute titled, “Winners and losers in the fulfillment of national artificial intelligence aspirations,” [3]. That document let me know that 44 countries have entered the race by creating a national AI strategic plan. I immediately wanted to get the list and hoped that they shared links to this potential treasure trove. The Brookings Institute had two previous reports one about differing views and the other focusing on the analysis of 34 plans [4][5]. 

Yes, my primary focus of this inquiry so far was to dig into the plans of the United States and the United Kingdom and then reading three different reports from the Brookings Institute. I have no real preference for their reporting or research, but in this instance it was good quality reading material on the exact topic I wanted to take into consideration. The researchers did take the time to prepare a figure that shows the leaders in national AI strategy implementation which included India, Germany, China, South Korea, United Kingdom, and Canada. The United States actually fell distinctly below the line and was in an entirely different quadrant than the leaders called technology prepared. You could go read the Global AI Vibrancy Tool page from Stanford University to see a different perspective on rankings [6]. After reading all that content and starting to wonder why they did not share the actual dataset with links to all these national AI strategies I may have to reach out to some of the researchers to see if they published it somewhere else that I may have just missed during my exploration.

It seemed like a good idea to go and dig into Google Scholar to see if anything stood out comparing national AI strategies [7]. It does look like some of the comparative studies schools in public policy and political science have done some groundwork. None of it really dug in and provided the higher level statistical analysis with good coding on key metrics I was looking for to better understand which strategy and trajectory would be the most successful or is being primarily implemented. It is entirely possible that the free market will end up defining where AI is going to end up flourishing. Regulation could certainly constrain the free market innovation machine enough to diminish a national AI strategy. The other consideration is that the multinational nature of modern corporations means that even the best national AI strategy could be confounded by organizational reach.

Links and thoughts:

“[ML News] DeepMind AlphaCode | OpenAI math prover | Meta battles harmful content with AI”

“Is AI just statistics? | Yann LeCun and Lex Fridman”

From SpaceX, “Starship Update”

Top 5 Tweets of the week:

Footnotes:

[1] National AI Strategy. United Kingdom. (2021, September 22). Retrieved February 12, 2022, from https://www.gov.uk/government/publications/national-ai-strategy or you can go directly to the document here: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/1020402/National_AI_Strategy_-_PDF_version.pdf [2] U.S. Department of Health and Human Services. (2021, January). Artificial Intelligence (AI) Strategy. Retrieved February 12, 2022, from https://www.hhs.gov/sites/default/files/final-hhs-ai-strategy.pdf  [3] Fatima, S., Dawson, G. S., Desouza, K. C., & Denford, J. S. (2021, October 21). Winners and losers in the fulfillment of national artificial intelligence aspirations. Brookings Institute . Retrieved February 12, 2022, from https://www.brookings.edu/blog/techtank/2021/10/21/winners-and-losers-in-the-fulfilment-of-national-artificial-intelligence-aspirations/ [4] Fatima, S., Desouza, K. C., & Dawson, G. S. (2020, June 17). How different countries view artificial intelligence. Brookings Institute . Retrieved February 12, 2022, from https://www.brookings.edu/research/how-different-countries-view-artificial-intelligence/ [5] Fatima, S., Desouza, K. C., Dawson, G. S., & Denford, J. S. (2021, May 13). Analyzing artificial intelligence plans in 34 countries. Brookings Institute. Retrieved February 12, 2022, from https://www.brookings.edu/blog/techtank/2021/05/13/analyzing-artificial-intelligence-plans-in-34-countries/  [6] Global AI Vibrancy Tool: Who’s leading the global AI race? Stanford University. (n.d.). Retrieved February 12, 2022, from https://aiindex.stanford.edu/vibrancy/ [7] national ai strategy comparison. Google Scholar. (n.d.). Retrieved February 12, 2022, from https://scholar.google.com/scholar?hl=en&as_sdt=0%2C6&q=national+ai+strategy+comparison&btnG=  

What’s next for The Lindahl Letter?

* Week 57: How would I compose an ML syllabus?

* Week 58: Teaching or training machine learning skills

* Week 59: Multimodal machine learning revisited

* Week 60: General artificial intelligence

* Week 61: AI network platforms

I’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed reading this content, then please take a moment and share it with a friend.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

Considering this topic really involved taking things in two different directions. First, I started to really try to understand the world and landscape of machine learning patents. Second, it became abundantly clear that acquisitions have fueled a lot of change in the machine learning space. A history of machine learning acquisitions could be a topic one day. That will probably happen in week 78. Some of that acquisition has been about bringing talent to internal teams and some of it has been about patent acquisition [1]. After those two considerations bubbled up to the forefront of my thoughts I also started to wonder about patenting ML models compared to patenting the underlying technology. You can very quickly go out and find out that last year the United States Patent and Trademark Office’s (USPTO) released a new artificial intelligence patent dataset [2].

That gigabyte sized dataset download included 13.2 million patents and pre-grants that you can study on the topic of artificial intelligence [3]. The actual download speed to get that file was exceedingly slow and even with my gigabit internet connection took over 20 minutes. While I was waiting to review the actual data I tried to read the linked journal article about the dataset from Giczy, A.V., Pairolero, N.A. & Toole, A.A. Identifying artificial intelligence (AI) invention: a novel AI patent dataset. The Journal of Technology Transfer (2021) [4]. That article is behind a Springer paywall and costs $39.95 which is a little wild for access to a full PDF of an article. However, it was much easier and surprisingly informative to just watch a narrated PowerPoint presentation from Nicholas A. Pairolero from June 25, 2021 that was presented at the AI and Patents Workshop at ICAIL 2021 [5]. 

Let’s pivot back to understanding the landscape of machine learning patents. You will very quickly find out that the team over at Google is proudly displaying a patent from December 1, 2005 about machine learning systems and methods [6]. The patent in PDF format is 17 pages long and a pretty easy read. Pivoting from that patent I started to wonder about who is getting patents now and if that really matters anymore in the machine learning space [7]. I learned during that research that it might be fun to work with a website called Patent Guru (patentguru.com) to visualize frequency [8]. I pretty quickly hit the limits of what free access to that website using the analysis features would provide and ended up circling back to the dataset I downloaded. During the course of getting that dataset loaded up I started to consider the idea of patenting AI algorithms and even the future consideration of if an AI built for drug discovery could own the patent or resulting intellectual property from that discovery.

Links and thoughts:

“Identifying Artificial Intelligence Invention: A Novel AI Patent Dataset (AI & Patents 2021)”

Data Science Grandmaster Series from NVIDIA

https://www.youtube.com/playlist?list=PL5B692fm6--uXbxtmPJz5nu3Xmc1JUm3F 

“tinyML Talks: Energy-Efficiency and Security for TinyML and EdgeAI: A Cross-Layer Approach”

“GPT-NeoX-20B - Open-Source huge language model by EleutherAI (Interview w/ co-founder Connor Leahy)”

“Getting Started with Sentiment Analysis using Python”

https://huggingface.co/blog/sentiment-analysis-python 

“They Almost Got Away With It! - WAN Show February 04, 2022”

Top 5 Tweets of the week:

Footnotes:

[1] Sawers, P. (n.d.). 13 acquisitions highlight Big Tech’s AI talent grab in 2020. Venture Beat. Retrieved February 6, 2022, from https://venturebeat.com/2020/12/25/13-acquisitions-highlight-big-techs-ai-talent-grab-in-2020/ 

[2] USPTO releases new Artificial Intelligence Patent Dataset. USPTO. (n.d.). Retrieved February 6, 2022, from https://www.uspto.gov/about-us/news-updates/uspto-releases-new-artificial-intelligence-patent-dataset 

[3] Artificial Intelligence Patent Dataset. USPTO. (n.d.). Retrieved February 6, 2022, from https://www.uspto.gov/ip-policy/economic-research/research-datasets/artificial-intelligence-patent-dataset 

[4] Giczy, A.V., Pairolero, N.A. & Toole, A.A. Identifying artificial intelligence (AI) invention: a novel AI patent dataset. J Technol Transf (2021). https://doi.org/10.1007/s10961-021-09900-2

[5]

[6] https://patents.google.com/patent/US20050267850A1/en 

[7] Obee, E. (2021, August 18). Artificial Intelligence Patent 101. Towards Data Science. Retrieved February 6, 2022, from https://towardsdatascience.com/artificial-intelligence-patent-101-3eebf93f5297 

[8] "machine learning". PatentGuru. (n.d.). Retrieved February 6, 2022, from https://www.patentguru.com/analysis?area=US&q=%22machine+learning%22

What’s next for The Lindahl Letter?

* Week 56: Comparative analysis of national AI strategies

* Week 57: How would I compose an ML syllabus?

* Week 58: Teaching or training machine learning skills

* Week 59: Multimodal machine learning revisited

* Week 60: General artificial intelligence

I’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed reading this content, then please take a moment and share it with a friend.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com