The Lindahl Letter

Perhaps you were looking for a bit more of a deep dive about deep generative models than will be contained in this relatively short missive. You could go check out Stanford University’s CS236 from Fall 2021 course [1]. That would help you begin to figure out just how unsupervised learning could be used to figure out the data distribution well enough to generate predicted other data elements. The content is broken up into 5 sections and you could contribute to the class GitHub if you wanted to provide feedback or improvement suggestions. Alternatively, you could learn more about this one from Prakash Pandey over at Towards Data Science from back in 2018 [2]. It's a faster read and pretty easy to digest compared to taking on a college level course. You could go the academic paper route for an introduction and dig into a work from Ruthotto & Haber from 2021:

Ruthotto, L., & Haber, E. (2021). An introduction to deep generative modeling. GAMM‐Mitteilungen, 44(2), e202100008. https://arxiv.org/pdf/2103.05180.pdf 

My initial run at digging into deep generative models opened the door to a bunch of different topics within the space. Right at the start I ran into semi-supervised learning, graphs, urban mobility, and molecular science. You can imagine the urban mobility one made me a little curious what people were doing to model that with deep generation. Apparently, Google Scholar had enough data to offer up three paths including migration, mobility, and morphology. To get started I dug in with a quick search on "Deep generative models" with urban mobility [3]. None of the articles within this search space have a lot of references. It might be a relatively small area of academic inquiry at the moment. You could read most of the relevant academic content related to deep generative models in an afternoon. Trying to dig into using them for some type of use case will take a bit more effort in terms of setup, technology, and selection of that use case. 

Here are 3 papers that showed up with the urban mobility search:

Eigenschink, P., Vamosi, S., Vamosi, R., Sun, C., Reutterer, T., & Kalcher, K. (2021). Deep Generative Models for Synthetic Data. ACM Computing Surveys. https://epub.wu.ac.at/8394/1/Deep_Generative_Models_for_Sequential_Data__WU_ePub_.pdf

Anda, C., & Ordonez Medina, S. A. (2019). Privacy-by-design generative models of urban mobility. Arbeitsberichte Verkehrs-und Raumplanung, 1454. https://www.research-collection.ethz.ch/bitstream/handle/20.500.11850/357034/3/ab1454.pdf

Johnsen, M., Brandt, O., Garrido, S., & Pereira, F. (2022). Population synthesis for urban resident modeling using deep generative models. Neural Computing and Applications, 34(6), 4677-4692. https://arxiv.org/ftp/arxiv/papers/2011/2011.06851.pdf 

3 decently cited academic papers:

Salakhutdinov, R. (2015). Learning deep generative models. Annual Review of Statistics and Its Application, 2, 361-385. https://www.utstat.toronto.edu/~rsalakhu/papers/Russ_thesis.pdf

Kingma, D. P., Mohamed, S., Jimenez Rezende, D., & Welling, M. (2014). Semi-supervised learning with deep generative models. Advances in neural information processing systems, 27. https://proceedings.neurips.cc/paper/2014/file/d523773c6b194f37b938d340d5d02232-Paper.pdf

Maaløe, L., Sønderby, C. K., Sønderby, S. K., & Winther, O. (2016, June). Auxiliary deep generative models. In International conference on machine learning (pp. 1445-1453). PMLR. http://proceedings.mlr.press/v48/maaloe16.pdf 

Links and thoughts:

“Teaching MLOps at scale with GitHub - Universe 2022”

“#83 Dr. ANDREW LAMPINEN (Deepmind) - Natural Language, Symbols and Grounding [NEURIPS2022 UNPLUGGED]”

Top 5 Tweets of the week:

Footnotes:

[1] https://deepgenerativemodels.github.io/ 

[2] https://towardsdatascience.com/deep-generative-models-25ab2821afd3 

[3] https://scholar.google.com/scholar?hl=en&as_sdt=0%2C6&q=%22Deep+generative+models%22+urban+mobility&btnG= 

What’s next for The Lindahl Letter?

* Week 100: Overcrowding and ML

* Week 101: Back to the ROI for ML 

* Week 102: ML pracademics

* Week 103: Rethinking the future of ML

* Week 104: That 2nd year of posting recap

I’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. New editions arrive every Friday. Thank you and enjoy the week ahead.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

We are going to break out of the planned programming and go where everybody else involved in the machine learning space is going to go this week. Something new arrived this week and has captured the attention of the public mind. Seriously, I thought stable diffusion would be the big thing for 2022, but something else arrived in the intellectual space that might be more influential in the long run. The contents of this post were written right after the release, and I have been tinkering with that content throughout the week. 

Staying current and staying well-grounded in the field of machine learning has been increasingly difficult. Those two things are ultimately very hard to achieve at the same time. This is an area with a great deal of breath and depth. I say that after writing 98 consecutively published weekly installments of a Substack machine learning newsletter. The main example this week would be of the new interactive session based chat framework (bot) that OpenAI released this week. Like most of the people actively curious about how bot’s have improved these days, I went out to https://chat.openai.com/chat which the team over at OpenAI blogged about here https://openai.com/blog/chatgpt/. All you need to do is create an account and you can sign in and chat with the bot. Each new session is tabula rasa as a reset to the model without the additional layer of your previous interactions. This is the interesting part of the equation as building a model and then having the model keep context within a conversation or a series of conversations. That ability to keep conversational context across multiple conversations is not a part of the current deployment. 

The research preview they are sharing right now does not have access to the internet. It was also trained on data from about a year ago. Given the size of the language model they are invoking I would think it has a very large knowledge graph included , but that does not really appear to be the case. I gave it the following series of prompts to see what would happen in terms of how it generates content. The answers are in screenshot to help identify that the content was not created by me as a part of my normal writing output.

My prompt: “write 10 words about machine learning”

My prompt: “write 25 words about machine learning”

My prompt: “write 50 words about machine learning”

My prompt: “write 100 words about machine learning”

The last prompt I used was to ask it to “write about paper about machine learning with citations” which caused it to spit out 5 paragraphs that were pretty good.

This topic is way too early for academic articles [1]. A lot of news articles have been written about OpenAI’s chatbot they recently shared called ChatGPT. Here are 4 of them that came out this week:

“OpenAI’s new chatbot can explain code and write sitcom scripts but is still easily tricked”

https://www.theverge.com/23488017/openai-chatbot-chatgpt-ai-examples-web-demo

“OpenAI’s new ChatGPT is scary-good, crazy-fun, and—so far—not particularly evil.” https://slate.com/technology/2022/12/chatgpt-openai-artificial-intelligence-chatbot-whoa.html

“OpenAI invites everyone to test new AI-powered chatbot—with amusing results”

https://arstechnica.com/information-technology/2022/12/openai-invites-everyone-to-test-new-ai-powered-chatbot-with-amusing-results/

“OpenAI’s ChatGPT shows why implementation is key with generative AI”

https://techcrunch.com/2022/12/02/openais-chatgpt-shows-why-implementation-is-key-with-generative-ai/

Links and thoughts:

“E106: SBF's media strategy, FTX culpability, ChatGPT, SaaS slowdown & more”

“Why Do I Keep Getting Called Out - WAN Show December 2, 2022”

Top 6 “ChatGPT” Tweets of the week:

Footnotes:

[1] https://scholar.google.com/scholar?hl=en&as_sdt=0%2C6&q=chatgpt&btnG=

What’s next for The Lindahl Letter?

* Week 99: Deep generative models

* Week 100: Overcrowding and ML

* Week 101: Back to the ROI for ML

* Week 102: ML pracademics

* Week 103: Rethinking the future of ML

* Week 104: That 2nd year of posting recap

I’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. New editions arrive every Friday. Thank you and enjoy the week ahead.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

Recently, I started spending a bit more time writing about quantum machine learning and quantum computing in general. One of the things I became curious about was related to a thread of thoughts about how they code something for a quantum computer. One of the first things that I came across while trying to learn more about how people were coding with Twist was an article in IEEE Spectrum called, “Meet Twist: MIT’s Quantum Programming Language: Keeping tabs on data entanglement keeps reins on buggy quantum code” [1]. This article referenced out to CSAIL or the Computer Science and Artificial Intelligence Laboratory which happens to be located at the Massachusetts Institute for Technology. You can find their delightful website right here: https://www.csail.mit.edu/ and it does pretty easily direct you toward Twist with a few searches [2].

Within that website you will find that the researchers Charles Yuan, Christopher McNally, and Michael Carbin shared a paper at POPL 2022 [3].

Proceedings of the ACM on Programming LanguagesVolume 6Issue POPL January 2022 Article No.: 30pp 1–32 https://doi.org/10.1145/3498691

Or you could go out to the paper on arXiv and download the PDF.

Yuan, C., McNally, C., & Carbin, M. (2022). Twist: sound reasoning for purity and entanglement in Quantum programs. Proceedings of the ACM on Programming Languages, 6(POPL), 1-32. https://arxiv.org/abs/2205.02287

You might be curious how many papers cite that paper and the answer at this very moment from Google Scholar happens to be 6 [4]. None of this content seems to be highly citated at this point in terms of a network of academic coverage.

However, if you were wondering about an MIT course you could take related to this, then you are in luck. For only $2,249.00 you could take the 4-week course that starts on January 23, 2023: https://learn-xpro.mit.edu/quantum-computing.

At this point in the process, I started looking around for coding examples or notebooks with something to try to absorb. You are going to end up with the paper reading pages 46-53 of that paper linked above from arXiv.

I took a look at this artifact up on GitHub here: https://github.com/psg-mit/twist-popl22

Over the next year I’ll be looking for more practical coding examples or maybe a tutorial that really explains how to use the Twist quantum programing language. I could not find an emulator or a code dojo to test out things either. That is problematic given that I’m probably not going to pay for time on a quantum computer to learn how to write Twist code or more to the point you are paying to engage in the activity of coding.

Links and thoughts:

“A Hard Fork in the Road: FTX’s Unraveling and Elon’s Loyalty Oath”

“#81 JULIAN TOGELIUS, Prof. KEN STANLEY - AGI, Games, Diversity & Creativity [UNPLUGGED]”

“Galactica: A Large Language Model for Science (Drama & Paper Review)”

“A Sports Card Documentary IN THEATERS?! 👀 (Behind The Card)”

Top 5 Tweets of the week:

Footnotes:

[1] https://spectrum.ieee.org/quantum-programming-language-twist[2] https://www.csail.mit.edu/news/language-quantum-computing[3] https://popl22.sigplan.org/details/POPL-2022-popl-research-papers/30/Twist-Sound-Reasoning-for-Purity-and-Entanglement-in-Quantum-Programs[4] https://scholar.google.com/scholar?cites=6310617750987639534&as_sdt=4005&sciodt=0,6&hl=en

What’s next for The Lindahl Letter?

* Week 98: Deep generative models

* Week 99: Overcrowding and ML

* Week 100: Back to the ROI for ML

* Week 101: Revisiting my MLOps paper

* Week 102: ML pracademics

* Week 103: Rethinking the future of ML

* Week 104: That 2nd year of posting recap

I’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. New editions arrive every Friday. Thank you and enjoy the week ahead.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

Within the broader generative AI space, the part I tend to focus on is related to the written word. Right now, all of the visual generation parts of generative AI in terms of images and videos are wholesale living in the public mind [1]. Creative people are generating thumbnails and playing with all sorts of plausible image generation technology. A few teams are rapidly working on how to make video from that same type of generative AI and that is going to be interesting. We are probably going to see generative AI shows that people create very soon. I have previously written (Week 78) that I think all public trust in imagines is going to erode and that we are going to hit a zero-trust wall when it comes to being able to believe what we see [2]. This missive will be about the future of where large language models are going and a bit of a reflection on what has happened in the last couple of years.

Back on October 26, 2021, the folks over on Hugging Face shared out a post called, “Large Language Models: A New Moore's Law? [3]. This post starts out with a very familiar graphic of the models in terms of billions of parameters over time. This is a relatively recent phenomenon with a start during 2018 and massive acceleration after 2020. You may well have heard about the Megatron-Turing natural language generation model (MT-NLG) [4][5][6]. You can imagine that people were thinking they should make larger models. After all what is better than a billion-parameter model? It obviously has to be a trillion-parameter model. I would argue that the reality of having unique parameters within that large of a search space is probably something that is being disregarded at this point, but that has not stopped the march for more and more parameters. You might be thinking that nobody has really done that type of effort in practice.

The M6 model happens to be 10 trillion parameters [7]. Yeah, a 10 trillion parameter model exists. That is mind bogglingly large.

One of the longest papers with the most authors was published by a lot of people from Stanford was “On the opportunities and risks of foundation models” [8]. You probably could have guessed that I was going to throw the title of that paper into Google Scholar to see what other people were adding with that citation to the scholarly world aka the academy [9]. That query offered up about 565 results to consider.

Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., ... & Liang, P. (2021). On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258. https://arxiv.org/pdf/2108.07258.pdf

This link will take you right to the 412 papers that Google Scholar believes have a direct citation to that massive 212 page paper [10].

One of the real concerns when that paper got published was that it covered so much ground and had so many coauthors that it would very quickly become an anchor citation that was heavily cited. Some people were worried it would just become a stock or default citation for people in literature reviews. I’m pretty sure that the number of scholars that came together on that work will pretty much guarantee that it gets cited a ton going forward. The other element that will help with that is that the paper is highly readable for people wanting to learn and understand large language models. Together those two elements of it being useful to read and known by a large number of scholars from the start pretty much guarantee that people will hear about it for years to come.

Links and thoughts:

“[ML News] Multiplayer Stable Diffusion | OpenAI needs more funding | Text-to-Video models incoming”

“We've Made Some Big Mistakes - WAN Show November 18, 2022”

Top 5 Tweets of the week:

Footnotes:

[1] https://venturebeat.com/ai/how-2022-became-the-year-of-generative-ai/

[2]

[3] https://huggingface.co/blog/large-language-models

[4] https://developer.nvidia.com/megatron-turing-natural-language-generation

[5] https://arxiv.org/abs/1909.08053

[6] https://developer.nvidia.com/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/

[7] https://towardsdatascience.com/meet-m6-10-trillion-parameters-at-1-gpt-3s-energy-cost-997092cbe5e8

[8] https://arxiv.org/pdf/2108.07258.pdf

[9] https://scholar.google.com/scholar?hl=en&as_sdt=0%2C6&q=%22On+the+Opportunities+and+Risks+of+Foundation+Models%22&btnG=

[10] https://scholar.google.com/scholar?cites=9595110325981705564&as_sdt=4005&sciodt=0,6&hl=en

What’s next for The Lindahl Letter?

* Week 97: MIT’s Twist Quantum programming language

* Week 98: Deep generative models

* Week 99: Overcrowding and ML

* Week 100: Back to the ROI for ML

* Week 101: Revisiting my MLOps paper

* Week 102: ML pracademics

* Week 103: Rethinking the future of ML

* Week 104: That 2nd year of posting recap

I’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. New editions arrive every Friday. Thank you and enjoy the week ahead.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

We are living on the edge of meeting the weekly Friday publishing deadline at this point. As we quickly approach the 104th post and the major two-year milestone I’m still working on the same Saturday and Sunday schedule of early morning writing. I’m just having to be extra mindful of making sure I don’t get sidetracked into working on other things. This week’s topic could be an entire book full of insight. People are certainly going to fill the shelves with quantum machine learning books in the coming years. It is certainly starting to turn into whole conferences and other gatherings of people interested in telling people all about and sharing stories of quantum machine learning.

Nobody is really bringing a quantum computer to any of these events. These are not something that is going to fit in your car and be ready to head out to a conference event. Seriously, to the best of my knowledge no laptop or portable quantum computer exists at this time and if it did nobody is using it for machine learning. They would probably be taking it to conferences or other gathering to show people how delightfully wonderful it is to be able to carry around such a power computing device. I imagine that it will be like Steve Wozniak showing up and assembling an early homebrew computer club kit. Those were moments of endless possibility, delight, and wonder. Bringing back the chance at some type of epic moment like that is certainly something that could very well happen within this space. My money is on the team over at IBM making that happen at some point.

Let’s rewind the coverage here for a moment and reflect on two of my previous Substack posts:

Week 42: Time crystals and machine learning (this one was epic)

Week 77: Is quantum machine learning gaining momentum?

You may have forgotten from the first paragraph that this is week 95 of The Lindahl Letter and quantum computing has received 3 different weeks of coverage. You can tell from that degree of focus that I believe it is a topic that will eventually change the nature of compute.  

All right let’s jump right into the best academic articles about quantum machine learning.

Biamonte, J., Wittek, P., Pancotti, N., Rebentrost, P., Wiebe, N., & Lloyd, S. (2017). Quantum machine learning. Nature, 549(7671), 195-202. https://arxiv.org/pdf/1611.09347.pdf

Schuld, M., Sinayskiy, I., & Petruccione, F. (2015). An introduction to quantum machine learning. Contemporary Physics, 56(2), 172-185. https://arxiv.org/pdf/1409.3097.pdf

I did enjoy this 35-minute YouTube video about quantum computing from TechTechPotato which is hosted by Dr. Ian Cutress, “Quantum Computing: Now Widely Available!”

Being able to see some of these videos with the crew from IBM make me feel better about this technology actually existing. This would be much easier to accept as science fiction. Within that video the part of the coverage that caught my attention the most was that really it was academics and some startups that were coding for practical use cases to do something with quantum computing. For the most part, the type of machine learning efforts that would be easily transition over into this type of compute are not called out super clearly.

One of the things I’m considering for next year is maybe coding something simple up and running it on one of these IBM systems. To really get a handle on what is happening within this quantum computing system it feels like only some hands work will close the gap for me here.

Links and thoughts:

“The Uses of IBM's Next Generation 433 Qubit Chip”

“#036 - Max Welling: Quantum, Manifolds & Symmetries in ML”

This is a podcast from the folks who are in the room with Twitter… “E104: FTX collapse with Coinbase CEO Brian Armstrong + election results, macro update & more”

Top 5 Tweets of the week:

Footnotes:

None.

What’s next for The Lindahl Letter?

* Week 96: Where are large language models going?

* Week 97: MIT’s Twist Quantum programming language

* Week 98: Deep generative models

* Week 99: Overcrowding and ML

* Week 100: Back to the ROI for ML

I’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. New editions arrive every Friday. Thank you and enjoy the week ahead.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

I must have missed it when Samuel K. Moore covered this topic back on February 24, 2022 in an article titled, “RISC-V AI Chips Will Be Everywhere Esperanto Technology’s chip heralds new era in open-source architecture; Intel set to cash in,” [1]. I’m going to admit that I did fix the spelling on the word technology when I cut and pasted that title over to this Substack post. Fast forward just a couple of months and in April an article in Forbes talked about people actually sampling the 1,000 core hardware [2]. Within the broader context of things 1,000 cores is a lot of computing cores for a CPU chip. This computer for example is running an Intel i7 series chip and has 6 cores. You can see the difference in the number of cores is very large.

You can pretty easily get to the website for Esperanto Technologies

https://www.esperanto.ai/

and it has some information on it. They clearly believe that RISC-V is the future of computing innovation. For those of you who might be chuckling that “RISC architecture is going to change everything” yes that line from the 1995 feature film Hackers certainly continues to resonate with people. The reduced instruction set computer architecture certainly has had a strong run. I’m pretty sure that the RISC-V design is about 12 years old (2010) and it really is an open standard instruction set architecture.

You can read about at the Berkeley Architecture Research site here: https://bar.eecs.berkeley.edu/projects/riscv.html  

Some of you may have already guessed that I was going to search Google Scholar for anything on this topic that might be interesting [3]. You can pretty quickly dig into the 18 results that came back for that search. Here are a sampling of those results:

Imran, H. A., Mujahid, U., Wazir, S., Latif, U., & Mehmood, K. (2020). Embedded development boards for edge-AI: A comprehensive report. arXiv preprint arXiv:2009.00803. https://arxiv.org/ftp/arxiv/papers/2009/2009.00803.pdf

Reuther, A., Michaleas, P., Jones, M., Gadepally, V., Samsi, S., & Kepner, J. (2022). AI and ML Accelerator Survey and Trends. arXiv preprint arXiv:2210.04055. https://arxiv.org/pdf/2210.04055.pdf

Dokic, K., Mikolcevic, H., & Radisic, B. (2021). Inference speed comparison using convolutions in neural networks on various SoC hardware platforms using MicroPython. In RTA-CSIT (pp. 67-73). http://ceur-ws.org/Vol-2872/paper10.pdf

Overall, I read this article in Interesting Engineering that talked about RISC-V having shipped more than 10 billion cores already [4]. The only reference to RISC-V AI in that article does reference Esperanto and its in the last paragraph. I was really hoping to find more content about these AI hardware chips. My guess here is that some other phrase is being used to describe the technology. I had spent some time looking around at searches related to, “Esperanto Technologies competitors.” That did not really yield anything major to share here. This is one I will need to circle back to at some point to see how AI specific hardware is changing. My efforts going down that road ended up looking at the IBM, “AI Hardware Center” [5]. I feel like a lot more companies are working in this space and more coverage could help highlight that at some later point.

Links and thoughts:

“Why Signal won’t compromise on encryption, with president Meredith Whittaker”

“Stanford Seminar - ML Explainability Part 1 I Overview and Motivation for Explainability”

Top 6 Tweets of the week:

Footnotes:

[1] https://spectrum.ieee.org/risc-v-ai

[2] https://www.forbes.com/sites/karlfreund/2022/04/20/risc-v-startup-esperanto-technologies-samples-first-ai-silicon/?sh=37506f9c773d

[3] https://scholar.google.com/scholar?hl=en&as_sdt=0%2C6&q=%22risc-v+ai%22&btnG=

[4] https://interestingengineering.com/innovation/rise-of-risc-v-the-computer-chip

[5] https://research.ibm.com/collaborate/ai-hardware-center

What’s next for The Lindahl Letter?

* Week 95: Quantum machine learning

* Week 96: Generative AI: Where are large language models going?

* Week 97: MIT’s Twist Quantum programming language

* Week 98: Deep generative models

* Week 99: Overcrowding and ML

I’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. New editions arrive every Friday. Thank you and enjoy the week ahead.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

We are going to get to the 104th Substack post before you know it here for The Lindahl Letter publication. Things are moving along, and we are on the very tail end of that journey. Don’t panic about this post not being full of links. (Spoiler alert) You will have plenty of perspectives to read this week that are linked to for your reading pleasure.

This is one of the topics that deserves a lot of attention. I circle back to asking people to always consider ethics within the context of both ML and AI. One of the considerations within that argument would be to really try to understand the outcomes of what the technology is being used to achieve or the negative externalities that would be possible from it’s use. You have heard me say it before and you will certainly hear it again, “Just because you can do a thing, does not mean you should.” It’s a real consideration within the AI/ML space. A lot of the things that can be done with both ML and AI are unconscionable and should be avoided. That is why ethics should be a core part of the AI/ML journey without question. Full stop.

Beyond that consideration I tried to gather up a bunch of papers critical of ML in general. It was actually much harder in practice to find written criticism of ML than I expected in published article forms. You are certainly welcome to out and subscribe to the Substack of Gary Marcus who publishes, “The Road to AI We Can Trust,” [1]. A good scroll across those posts will give you a real sense of criticism and questions about the ML space. It has some really solid engagement as well from people who care enough to deeply question things. I wholesale consider that to be a healthy part of the process.

Marcus, G. (2018). Deep learning: A critical appraisal. arXiv preprint arXiv:1801.00631. https://arxiv.org/ftp/arxiv/papers/1801/1801.00631.pdf

Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., & Sutskever, I. (2021). Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12), 124003. https://arxiv.org/pdf/1912.02292.pdf

Lake, B., & Baroni, M. (2018). Still not systematic after all these years: On the compositional skills of sequence-to-sequence recurrent networks. https://openreview.net/pdf?id=H18WqugAb

Mitchell, M. (2021). Why AI is harder than we think. arXiv preprint arXiv:2104.12871. https://arxiv.org/pdf/2104.12871.pdf

Biderman, S., & Scheirer, W. J. (2020). Pitfalls in machine learning research: Reexamining the development cycle. http://proceedings.mlr.press/v137/biderman20a/biderman20a.pdf

Henderson, P., & Brunskill, E. (2018). Distilling information from a flood: A possibility for the use of meta-analysis and systematic review in machine learning research. arXiv preprint arXiv:1812.01074. https://arxiv.org/pdf/1812.01074.pdf

Vinny, P. W., Garg, R., Padma Srivastava, M. V., Lal, V., & Vishnu, V. Y. (2021). Critical Appraisal of a Machine Learning Paper: A Guide for the Neurologist. Annals of Indian Academy of Neurology, 24(4), 481–489. https://doi.org/10.4103/aian.AIAN_1120_20 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8513942/

You can get to a bit more content outside of scholarly articles when it comes to finding critics of machine learning. I’m going to share a handful of links to different things that I found interesting along the way to researching this Substack post.

“5 myths about learning and innateness” 

https://open.substack.com/pub/garymarcus/p/5-myths-about-learning-and-innateness?r=8oh0m&utm_campaign=post&utm_medium=web

“The Limitations of Machine Learning”

https://towardsdatascience.com/the-limitations-of-machine-learning-a00e0c3040c6

“When Machine Learning Goes Off the Rails”

https://hbr.org/2021/01/when-machine-learning-goes-off-the-rails

“The way we train AI is fundamentally flawed”

https://www.technologyreview.com/2020/11/18/1012234/training-machine-learning-broken-real-world-heath-nlp-computer-vision/

“Why deep-learning AIs are so easy to fool”

https://www.nature.com/articles/d41586-019-03013-5

“How a Pioneer of Machine Learning Became One of Its Sharpest Critics”

https://www.theatlantic.com/technology/archive/2018/05/machine-learning-is-stuck-on-asking-why/560675/

“AI researchers allege that machine learning is alchemy”

https://www.science.org/content/article/ai-researchers-allege-machine-learning-alchemy

Links and thoughts:

“Generative AI is Here. Who Should Control It?”

“Twitter is now an Elon Musk company”

“Apple's new App Store tax, Microsoft Surface reviews, and Meta's earnings”

“Emergency Pod: Elon Musk Owns Twitter”

Top 5 Tweets of the week:

Footnotes:

[1] Gary Marcus’s Substack “The Road to AI We Can Trust”

What’s next for The Lindahl Letter?

* Week 94: AI hardware (RISC-V AI Chips)

* Week 95: Quantum machine learning

* Week 96: Generative AI: Where are large language models going?

* Week 97: MIT’s Twist Quantum programming language

* Week 98: Deep generative models

I’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. New editions arrive every Friday. Thank you and enjoy the week ahead.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

It seems like having a national artificial intelligence initiative is popular these days. Back on February 18, 2022, I shared my week 56 Substack post, “Comparative analysis of national AI strategies.” That missive continues to get a good bit of traffic so I thought now would be a good time to go ahead and revisit national AI strategies, advisory committees, institutes, legislation, and the myriad of research institutes or think tanks that are jumping into this area of consideration. This is an area that I think could be a good place for some solid academic contributions. Instead of digging into all of those areas my attention really got focused on one advisory committee. That will become clear here in the next couple of sections of content.

This week I have considered shifting The Lindahl Letter over to being an AI strategy advisory committee after spending a bunch of time reading about them this week. I’m not going to do that as it would limit my creative output to just one area and that sounds intellectually exhausting. One of them you can read about would be the National AI Advisory Committee (NAIAC) [1]. The next committee meeting was about to happen before writing this post. I had plans to listen live and I was totally signed up for everything [2]. Go forward I’m fully registered and signed up for alerts from the NAIAC. I would be happy to provide them guidance on effective national AI strategies from a comparative perspective, but that has not happened so far. This topic is an interesting space to consider at length. We are seeing a huge amount of academic work and companies like Hugging Face democratizing AI through community. Consider for a moment just how fast stable diffusion showed up and then was actively built into things and deployed. We are seeing massive changes within the ML/AI space and the deployment cycle is super-fast based on how interconnected the community happens to be worldwide. That has huge ramifications for any advisory committee considering the national level of AI strategy. Adapting to the rate of change and decentralized nature of things requires a different type of national AI strategy. I’ll be listening to the NAIAC in October to see how things are going. You can find the sessions on YouTube by searching for “NAIAC” pretty easily.

“National Artificial Intelligence Advisory Committee (NAIAC) Meeting”

“National Artificial Intelligence Advisory Committee (NAIAC) Field Hearing”

You could read the meeting minutes from May 4, 2022.

https://www.ai.gov/wp-content/uploads/2022/07/NAIAC-Minutes-05042022.pdf

I went out to Google Scholar and took a look to see if anybody had published or shared anything with this advisory committee referenced [3]. Nothing really came up except the above-mentioned meeting minutes from May 4, 2022. Nothing really showed up during a search of arXiv either [4]. It’s possible in about 6 months more content will show up reacting to the hours of meetings that are linked above. Right now, we appear to be a little bit ahead of things in terms of reactions to the work being done by this advisory committee. I’m going to keep an eye out for more content related to NAIAC. It’s possible sometime next year it will be the right time to dig back into this one.

Links and thoughts:

“5 Practical Machine Learning Lessons You’re NOT Taught in School”

“This Has Never Happened Before - WAN Show October 14, 2022”

“Stanford CS330: Deep Multi-Task & Meta Learning I 2021 I Lecture 4”

“Confusing new Apple products, Netflix password sharing, and NFT cults”

Top 5 Tweets of the week:

Footnotes:

[1] https://www.ai.gov/naiac/

[2] https://events.nist.gov/profile/form/index.cfm?PKformID=0x17861abcd

[3] https://scholar.google.com/scholar?hl=en&as_sdt=0%2C6&q=National+Artificial+Intelligence+Advisory+Committee&btnG=

[4]

https://search.arxiv.org/?in=&query=%22National%20Artificial%20Intelligence%20Advisory%20Committee%22

What’s next for The Lindahl Letter?

* Week 93: Papers critical of ML

* Week 94: AI hardware (RISC-V AI Chips)

* Week 95: Quantum machine learning

* Week 96: Where are large language models going?

* Week 97: MIT’s Twist Quantum programming language

I’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. New editions arrive every Friday. Thank you and enjoy the week ahead.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

For those of you that keep track of these types of things we are now in real time based on my publishing schedule. Over the course of the next few weeks no backlog exists as we make the run to 104 consecutive Substack posts spanning 2 years of content creation on this platform. It’s week 91 right now in the publishing schedule which means that only 13 blocks of super exciting writing about machine learning stand between you and the completion of that penultimate tasking. To that end I’ll be working without a sizeable backlog that would prevent procrastination or a loss of focus from breaking the streak. At this very moment, I’m probably writing about that consideration to help refocus my efforts on completing this last stretch. You may recall that after the 2-year mark I’m going to mix things up a bit and switch focus from machine learning to artificial intelligence in general. The format might change a bit as well, but you will have to stay tuned to see what ends up showing up every Friday.

My physical notebook where I write things down by hand with a Parker Sonnet fountain pen includes a few different sketches related to building a universal request handler. A lot of the voice assistants receive queries that need to be given to some type of ML model to solve. Making a decision about which model to apply and then how to manage and sort those results back into the general knowledge graph is an interesting problem to solve. Generally, one solution is to just fire off the request to a series of API channels and the one that reports back a probable answer in the shortest time is what gets served up by the voice assistant. All of that happens so very quickly that you don’t really notice a long delay. We as people handle super complex reasoning tasks and work on things with an extreme depth without even questioning the solution selection process.

Perhaps as a subset of that general selection of what model to use when questioning something else props up from researchers and practitioners. One of the things people who are involved with work in the ML models space ask from time to time is about why you cannot just combine all the ML models together and make a super model. One of the ways people are working to bring models together involves the ensemble method for machine learning models. This methodology involves making a few models and then combining them to improve results. This is not a method to just stack random models and try to make it work. The ensemble method is a technique that is based on working with the same dataset and maybe combining for example a bunch of favorable random forests or some other set of similar models to form an ensemble. From what I have been able to tell from reading articles in this space its not a super solution to just bring all machine learning models together in one unified model theory.

Dietterich, T. G. (2000, June). Ensemble methods in machine learning. In International workshop on multiple classifier systems (pp. 1-15). Springer, Berlin, Heidelberg. https://web.engr.oregonstate.edu/~tgd/publications/mcs-ensembles.pdf

Dietterich, T. G. (2002). Ensemble learning. The handbook of brain theory and neural networks, 2(1), 110-125. https://courses.cs.washington.edu/courses/cse446/12wi/tgd-ensembles.pdf

This one is out and in use in the wild. You can actually utilize ensemble ML models from some of the systems like scikit-learn [1]. You can also pretty quickly implement ensemble models with the “The Functional API” as a part of TensorFlow core [2]. You can pretty quickly get up to speed and use this one in notebooks or other places.

Links and thoughts:

Lex Fridman Podcast “#324 – Daniel Negreanu: Poker”

Lex Fridman Podcast “#315 – Magnus Carlsen: Greatest Chess Player of All Time”

“Microsoft's Surface event, Pixel 7 and Pixel Watch reviews, and Meta Connect 2022”

“Mark Zuckerberg on the Quest Pro, future of the metaverse, and more”

Top 5 Tweets of the week:

Footnotes:

[1] https://scikit-learn.org/stable/modules/ensemble.html[2] https://www.tensorflow.org/guide/keras/functional

What’s next for The Lindahl Letter?

●     Week 92: National AI strategies revisited

●     Week 93: Papers critical of ML

●     Week 94: AI hardware (RISC-V AI Chips)

●     Week 95: Quantum machine learning

●     Week 96: Where are large language models going?

I’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. New editions arrive every Friday. Thank you and enjoy the week ahead.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

The post this week is going to be on the shorter side of things. I think that is in part due to the very straightforward nature of the topic under consideration. It really could have just been a link to a single book on the subject with a polite note that reading it would help you understand pretty much everything you need to know. To that end, it looks like the book on probabilistic machine learning from Kevin Patrick Murphy has been downloaded 168 thousand times [1]. That is pretty darn good for something in the machine learning space where the inflection point is generally over or under around 10,000 points of interest on a topic. It appears that Kevin really surpassed that ceiling by a ton of downloads. The book is very easy to get to and the search engines really seem to algorithmically love it as well. Given that this topic has a lot of refences to Bayesian decision theory you probably could predict that it would get my full attention. The topics that generally grounds all of my efforts in the machine learning space is that background in statistics and my enjoyment of working with Bayesian pooling. Let’s begin to breakdown the idea of probabilistic machine leaning involves understanding two general steps. First, you must accept that you want to explain observed data with your machine learning models. Second, those explanations are going to need to come from inferring plausible models to aid you in that explanation. Together those two steps help you begin to evaluate data in a probabilistic way which means that you are aided by the power of statistical probability grounding you to a rational approach. To me this sort of spells out an approach that is not based on randomness or anything particularly chaotic.

Murphy, K. P. (2012). Machine learning: a probabilistic perspective. MIT press. https://research.google/pubs/pub38136.pdf

Probabilistic machine learning papers

Ghahramani, Z. (2015). Probabilistic machine learning and artificial intelligence. Nature, 521(7553), 452-459. https://www.repository.cam.ac.uk/bitstream/handle/1810/248538/Ghahramani%25202015%2520Nature.pdf?sequence=1

Rain, C. (2013). Sentiment analysis in amazon reviews using probabilistic machine learning. Swarthmore College. https://www.sccs.swarthmore.edu/users/15/crain1/files/NLP_Final_Project.pdf

Probabilistic deep learning papers

Nie, S., Zheng, M., & Ji, Q. (2018). The deep regression bayesian network and its applications: Probabilistic deep learning for computer vision. IEEE Signal Processing Magazine, 35(1), 101-111. https://sites.ecse.rpi.edu/~cvrl/Publication/pdf/Nie2018.pdf

Peharz, R., Vergari, A., Stelzner, K., Molina, A., Shao, X., Trapp, M., ... & Ghahramani, Z. (2020, August). Random sum-product networks: A simple and effective approach to probabilistic deep learning. In Uncertainty in Artificial Intelligence (pp. 334-344). PMLR. http://proceedings.mlr.press/v115/peharz20a/peharz20a.pdf

Andersson, T. R., Hosking, J. S., Pérez-Ortiz, M., Paige, B., Elliott, A., Russell, C., ... & Shuckburgh, E. (2021). Seasonal Arctic sea ice forecasting with probabilistic deep learning. Nature communications, 12(1), 1-12. https://www.nature.com/articles/s41467-021-25257-4?tpcc=nleyeonai

Links and thoughts:

“How Arm conquered the chip market without making a single chip, with CEO Rene Haas”

Top 5 Tweets of the week:

Footnotes:

[1] https://probml.github.io/pml-book/book1.html

What’s next for The Lindahl Letter?

* Week 91: What are ensemble ML models?

* Week 92: National AI strategies revisited

* Week 93: Papers critical of ML

* Week 94: AI hardware (RISC-V AI Chips)

* Week 95: Quantum machine learning

I’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. New editions arrive every Friday. Thank you and enjoy the week ahead.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com