The Lindahl Letter

Thank you for tuning in to this audio only podcast presentation. This is week 166 of The Lindahl Letter publication. A new edition arrives every Friday. This week the topic under consideration for The Lindahl Letter is, “Those recent Nobel Prizes.”

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

For those of you listening to the audio version this week you may have noticed that we moved from the natural vocal capture to the narration vocal capture. I’m still recording the podcast audio for this adventure using a Blue Microphones Yeti X and my MacBook Air with the included GarageBand software. To further enhance your listening experience I’m still working to dial in the best possible recording technique to deliver superior podcast audio. 

Last week we really dug deep into the major corporate players that released quantum computing programing languages for general use. This week unfortunately needs to start with an epic spoiler alert. Please know that you should be aware that it does not appear quantum computing is practical at this time or really very scalable. I have read a fair number of jokes throughout the last week about neither the blockchain or quantum computer being scalable. With that spoiler delivered upfront this week it is time to move from the breadth of coverage to the depth of understanding related to what people are actually doing with practical use cases within the quantum computing space. One of the places I went to search around and learn a little bit more about use cases was the NASA Quantum Artificial Intelligence Laboratory [1]. The good folks over at NASA shared a paper in June of 2024 that was about “Assessing and Advancing the Potential of Quantum Computing: A NASA Case Study” [2]. For those of you wanting to read more quantum computing papers this is pretty easy to read and digest. It’s 27 pages and is very well cited throughout. 

You can join the quantum computing reddit community and it seems to be pretty active with academic discussion amongst the 50k+ members [3]. Beyond that reddit community the next place I ended up spending some time looking around was the Quantum Open Source Foundation which had a lot of content in terms of links [4]. They have a lot of curated links on the GitHub page that they maintain [5]. You could spend hours and hours of time just clicking around and looking at all of those projects. Eventually I ran into another GitHub repository called Awesome Quantum Computing that is another collection of curated links [6]. These collections of links will send you all over the place to see some interesting projects people are developing. 

Somehow during the hunt for the best projects using quantum computing to accomplish things I ended up back looking at the IBM Quantum Learning pages to see what things they were encouraging people to code as they learn to program [7]. The Azure Quantum team had a whole section devoted to trying to explain what solutions they are offering [8]. A lot of that seems to be focused on physics, chemistry, and ultimately material discovery and other applied applications to understand some type of complex interaction. Modeling really complex things seems to be a core use case that quantum computing has centered on based on the available evidence. I really think at this point I’m going to invest some time into completing a couple of these courses to get more hands on in the quantum computing space. I’ll share one last note about the fastest quantum computer that now has 1,180 qubits from Atom Computing [9]. It was a sizable leap from the IBM’s Osprey that was capable of 433 qubits. Hopefully we will see a bunch of fastest quantum computer records broken in the coming years. That will be a good sign that things have forward momentum in the quantum computing space.

Things to consider this week:

* You might want to read the 15,000 word October 2024 essay from Anthropic CEO Dario Amodei called “Machines of loving grace.” https://darioamodei.com/machines-of-loving-grace 

* The Anthropic team also released an updated responsible scaling policy that is a related read with the Dario Amodei essay https://www.anthropic.com/news/announcing-our-updated-responsible-scaling-policy 

* I enjoyed listening to Yannic Kilcher talk about and question a paper this week “GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models” https://arxiv.org/abs/2410.05229

Footnotes:

[1] https://www.nasa.gov/intelligent-systems-division/discovery-and-systems-health/nasa-quail/ 

[2] https://arxiv.org/abs/2406.15601 

[3] https://www.reddit.com/r/QuantumComputing/ 

[4] https://qosf.org/project_list/ 

[5] https://github.com/qosf/awesome-quantum-software 

[6] https://github.com/desireevl/awesome-quantum-computing 

[7]

https://learning.quantum.ibm.com/

 

[8] https://quantum.microsoft.com/en-us/solutions/azure-quantum-solutions 

[9] https://www.newscientist.com/article/2399246-record-breaking-quantum-computer-has-more-than-1000-qubits/ 

What’s next for The Lindahl Letter? At some point this series will move back to being planned out 5 weeks ahead of publication. 

If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. Make sure to stay curious, stay informed, and enjoy the week ahead!

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

Over the last week I started digging into quantum computer programming to see what is currently going on and what people are doing in that space [1][2]. General availability of actual quantum computing systems is the major barrier to using one to write some code [3]. You can buy time on a quantum computer or you can use a simulator. The simulators are sort of weird in general as the computing power you are using to simulate the quantum computer has a very small fractional power equivalent in terms of computing power. The biggest thing to really consider in terms of understanding where we are in terms of quantum computing is not cost or availability, but instead its a true question about the computing method itself related to having fully fault-tolerant quantum computing. Not only do you have to have methods for error correction within your quantum computing setup, but also you need a design that has proper fault-tolerant gates to avoid introducing increasing error levels within your computing. You are probably wondering if I went out to Google Scholar to find papers about fault-tolerance in quantum computing [4]. Of course that was where I went to look for papers. Here are 3 academic papers you can read with over 300 citations to consider:

Steane, A. M. (1999). Efficient fault-tolerant quantum computing. Nature, 399(6732), 124-126. https://arxiv.org/pdf/quant-ph/9809054 

Chow, J. M., Gambetta, J. M., Magesan, E., Abraham, D. W., Cross, A. W., Johnson, B. R., ... & Steffen, M. (2014). Implementing a strand of a scalable fault-tolerant quantum computing fabric. Nature communications, 5(1), 4015. https://www.nature.com/articles/ncomms5015 

Preskill, J. (1998). Fault-tolerant quantum computation. In Introduction to quantum computation and information (pp. 213-269). https://arxiv.org/pdf/quant-ph/9712048 

Let’s set aside those questions about making quantum computing sustainable and consider the code part of the equation. Major programming languages for quantum computing do exist from the players you would expect. They tend to have a lot of documentation and GitHub profiles to share the code. You could spend whole days looking at some of the languages made by major players. Here are 3 examples of major players making quantum computing language contributions. First, Qiskit was introduced as a quantum computing programming language by IBM teams back in 2017 [5][6]. Second, teams at Microsoft introduced Q# back in 2017 [7][8]. Third, Cirq is from Google AI and was introduced in 2018 [9][10]. You can start to dig into those code bases and I could find a lot of content related to the languages and a lot of hype about quantum computing. 

You can see that a lot of evidence exists related to quantum programming languages. The next logical questions would be how do they execute that code in practice and maybe what exactly are they doing with this quantum code. The teams at IBM have been pretty good about sharing plans to build faster and faster quantum computers [11]. You could watch the hype film about the IBM Quantum System Two that they shared to YouTube back on December 4, 2023. It’s always interesting to look at quantum computer builds; they are not the sort of thing that is going to sit under my desk or on my desk at this point within the technology curve. 

After all that foundational digging into the current state of quantum computing my thoughts are still conflicted about it. I know some questions exist about exactly how to deploy large scale efforts that work without propagating errors and the complexity of fault-tolerant gates. Probably the next step in my research process will be to find some examples of solid quantum computing code being used or actually deployed. That is maybe the best way to get a sense of what is being done within the quantum computing space. That research note about practical quantum computing and getting things done will probably shed some light on where things are trending. Beyond people setting records for the fastest machine or sharing hype videos the real deeper question here is what use cases are going to end up defining the technology. 

Footnotes:

[1] https://towardsdatascience.com/an-introduction-to-quantum-computers-and-quantum-coding-e5954f5a0415 

[2] https://www.mckinsey.com/featured-insights/mckinsey-explainers/what-is-quantum-computing

[3] https://github.com/qosf/awesome-quantum-software 

[4] A Google Scholar search for “fully fault-tolerant quantum computing” https://scholar.google.com/scholar?q=fully+fault-tolerant+quantum+computing&hl=en&as_sdt=0&as_vis=1&oi=scholart 

[5] https://www.ibm.com/quantum/qiskit 

[6] https://github.com/qiskit 

[7] https://learn.microsoft.com/en-us/azure/quantum/qsharp-overview 

[8] https://github.com/microsoft/qsharp 

[9] https://quantumai.google/cirq 

[10] https://github.com/quantumlib/Cirq 

[11] https://www.fastcompany.com/90992708/ibm-quantum-system-two 

What’s next for The Lindahl Letter? At some point this series will move back to being planned out 5 weeks ahead of publication. That point has not arrived just yet so you can expect a more fluid selection of topic coverage. 

If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. Make sure to stay curious, stay informed, and enjoy the week ahead! 

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

We are facing a new reality where the tipping point has been passed and a majority of content being generated online within the internet is now synthetic. Organic content generation simply cannot keep up with the flood of synthetic content. Those bot farms creating content never sleep. They just churn out content and pretend that it remains evergreen. My consideration on this topic started back on Wednesday, July 31, 2024, when I was invited to access SearchGPT from OpenAI [1]. Using that platform made me think a lot about how we access information and the ways that will change going forward. People are now getting summaries and completing searches that go beyond Googling something. You are probably well aware by now that I’m deeply concerned about how facts and knowledge are going to be stored and curated going forward. 

Going forward whoever owns the stores of facts or knowledge will effectively own history and how it is presented which is truly a watershed change in our shared understanding of the world. Individual voices and publications will be overshadowed by these collections. Owning the datastores that provide definitive facts or knowledge will be the cornerstone of whatever emerges going forward and should not be underestimated in terms of future value. I don’t think the ownership of facts will become commoditized and open sourced. I really do think it will be privatized and tightly controlled. Somebody who wanted to pivot our understanding on a particular point of inquiry could just start serving up that alternative perspective. Instead of people funding think tanks to ultimately change the messaging the next step will be funding content farms to just flood the message delivery. Keep in mind that people generally are not really reading books anymore [2]. That means that reasoning during the course of interpreting information may be a diminishing skillset.

Organically written original content exists online. Synthetically generated content has been on the rise. A lot of bots are scraping content for model training and trying to figure out what is organic and what is synthetic has become increasingly difficult. One of the hallmarks of my writing efforts has been the originality or maybe novelty of my research efforts. Within the broader context of the academy of academic thought, original contributions are what build that content and strengthen it overall. Diluting, derivative, and otherwise mediocre publications just flood the overall academic community. It’s perfectly fine to write a publication and decide it was not a significant contribution. Instead of maybe holding back those lesser works they are now freely shared in online archives and unfortunately a new generation of journals. The increase in AI related publications has been astonishing from 2010 to 2022 the number of publications nearly tripled [3]. 

Now we have a mix of the poorly written articles mixing with the synthetically generated to create a truly problematic future of consuming content. I’m considering web traffic at the moment, but the overall storage of facts and knowledge is certainly in scope. We reached the tipping point around 2016 where more traffic is mobile traffic than from a desktop browser [4]. OpenAI has now launched SearchGPT and beyond the dichotomy between mobile and desktop traffic we are about to see the rise of LLM interpreted results where people may never actually leave the landing page or interface of the search engine. It’s possible that dichotomy will fade away and the majority of traffic will be from bots scraping things to share within the newly powered search interfaces. People may very well interact with the grand volume of online information from applications using APIs to respond that are completely disconnected from what was the open internet people surfed and experienced. From reading the thoughts of a single writer to interpreting the output of the largest language models ever created. Things are changing at an incredibly rapid pace.

Now it’s time for a brief editorial note. Please note that my writing output over the last few years became over indexed on artificial intelligence and machine learning. Going down that rabbit hole was good at first and it was an effort truly focused on depth and breadth within the subject. Unfortunately, my focus lingered and instead of writing research notes about technological innovation, civil society, and the intersection of technology and modernity that pesky over indexing occurred. Now thanks to a moment of reflective practitioning I’m breaking out of that pattern and returning to what I consider a better balance of writing topics. Thank you for being along for that journey and the upcoming course correction.

Footnotes:

[1] https://chatgpt.com/search  

[2] https://www.pewresearch.org/short-reads/2021/09/21/who-doesnt-read-books-in-america/  

[3] https://arxiv.org/abs/2405.19522 or https://aiindex.stanford.edu/wp-content/uploads/2024/04/HAI_AI-Index-Report-2024_Chapter1.pdf 

[4] “>50% of web traffic comes from mobile.” Google Analytics Data, U.S., Q1 2016. https://www.thinkwithgoogle.com/marketing-strategies/app-and-mobile/mobile-web-traffic-statistics/ 

What’s next for The Lindahl Letter? I’m just going to sit down and write weekly or more likely bi-weekly so the topic won’t be planned out 5 weeks ahead of publication.

If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. Stay curious, stay informed, and enjoy the week ahead!

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

In the digital age, the methods we use to organize and comprehend information are continually evolving. Two significant approaches stand out: indexing facts and graphing knowledge. Both play essential roles in how we structure, retrieve, and understand data, but they serve distinct purposes and offer different advantages.

Let’s start out by looking a little deeper into indexing facts. Indexing is a traditional and straightforward method. It involves categorizing and listing information in a manner that allows for easy retrieval. Think of it as a library catalog, where every book has a unique identifier and is placed in a specific location based on its subject. This system is incredibly efficient for finding discrete pieces of information quickly. For example, a keyword search in a database relies heavily on indexing.

Indexes are foundational to databases and search engines. They allow us to locate specific data points without having to sift through every piece of information manually. This method is highly effective for tasks that require precision and speed. However, indexing has its limitations. It often lacks context and relational understanding between different pieces of data. An index can tell you where something is but not necessarily how it connects to other information.

Graphing knowledge, on the other hand, is about mapping relationships between data points. This approach is exemplified by knowledge graphs, which visually represent the connections between different concepts. A knowledge graph is more than a mere collection of facts; it is an interconnected web that shows how different pieces of information relate to one another.

In a knowledge graph, nodes represent entities (such as people, places, or concepts), and edges represent the relationships between these entities. This structure allows for a more holistic understanding of information. For instance, a knowledge graph can illustrate how historical events are connected, how scientific concepts overlap, or how social networks operate.

The advantages of graphing knowledge are manifold. It provides context, reveals patterns, and helps in discovering new insights that might not be apparent through traditional indexing. Knowledge graphs are particularly useful in fields that require deep understanding and analysis, such as artificial intelligence, semantic web technologies, and complex decision-making processes.

The intersection of indexing facts and graphing knowledge represents the future of information management. By combining the precision of indexing with the relational depth of knowledge graphs, we can create systems that are both efficient and insightful. This hybrid approach can enhance our ability to process and understand vast amounts of data, making it possible to derive meaningful insights quickly.

For example, search engines are evolving to incorporate elements of both indexing and graphing. Google's Knowledge Graph is a prime example, aiming to understand the context behind search queries to provide more relevant results. This system not only indexes web pages but also understands the relationships between different pieces of information, delivering a more nuanced response to users' queries.

Indexing facts and graphing knowledge are not mutually exclusive; they are complementary methods that, when combined, can significantly enhance our understanding and management of information. By leveraging the strengths of both approaches, we can build more robust systems that not only store and retrieve data efficiently but also provide deeper insights and understanding. As we continue to advance in the digital age, the fusion of these methods will undoubtedly play a pivotal role in shaping the future of information technology and knowledge management.

What’s next for The Lindahl Letter? 

* Week 163: Self-Supervised Learning

* Week 164: Graph-Based Feature Engineering

* Week 165: Federated Feature Engineering

* Week 166: Explainable Feature Engineering

* Week 167: Adaptive Feature Engineering

If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. Stay curious, stay informed, and enjoy the week ahead!

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

You never really have to worry about how you store knowledge. All that knowledge just gets accumulated day by day and how it is stored, structured, and even retrieved happens without any intervention. It’s just something that you do autonomously. That is very different when you have to manage all that data and store it. Navigating the intricate maze of data science, structuring really large knowledge graphs presents itself as both an art and a science. This endeavor, crucial for deepening our understanding and utilization of complex datasets, entails a series of pivotal steps and considerations. Today, we'll explore the foundational principles and practical strategies for effectively structuring these expansive networks of interconnected information, while drawing on the organizational wisdom of traditional knowledge structures like the Dewey Decimal System (DDS).

Understanding the basics seems to take more and more time these days. Knowledge graphs are more than mere data structures; they represent information through a web of entities and their interrelations. They offer a robust framework for integrating data from a multitude of sources, enhancing our ability to derive richer insights and make more informed decisions. As the size of the knowledge graph expands, so too does the complexity of structuring it effectively.

Key Components of Knowledge Graphs

1. Nodes and Edges: At the heart of any knowledge graph are nodes (entities) and edges (relationships). Nodes can embody concepts, objects, or events, while edges illustrate the interconnections among these nodes. The quality and comprehensiveness of your knowledge graph hinge on the precise definition and linking of these elements. You deal with these types of relationships every day without even an afterthought. The types of things you manage passively are more complex when they have to be handled in a planful way. 

2. Ontology: This serves as the schema or structural framework that delineates the types of entities and relationships within the graph. A well-crafted ontology ensures consistency and coherence, enabling more effective querying and analysis. I’m actually a fan of declaring things as a fact or not a fact and then storing those facts in buckets that are easy to retrieve. 

3. Data Ingestion and Integration: Large knowledge graphs often amalgamate data from various sources. Efficiently integrating this data while preserving its integrity and relevance is a critical challenge, involving data cleaning, normalization, and transformation.

Knowledge Structures and the Dewey Decimal System

Drawing inspiration from the Dewey Decimal System can provide valuable insights into structuring knowledge graphs. The DDS organizes information into a hierarchical, decimal-based classification system, which can serve as a model for categorizing and indexing data within a knowledge graph.

1. Hierarchical Classification: Like the DDS, hierarchical classification in a knowledge graph helps organize information into broad categories and narrower subcategories. This ensures that related entities are grouped together, facilitating easier navigation and retrieval.

2. Decimal Notation: Utilizing a decimal notation system to categorize entities and relationships can add a layer of precision and order to a knowledge graph. Each node and edge can be assigned a unique identifier, akin to how books are classified in libraries.

3. Subject Headings: Implementing subject headings, similar to those in the DDS, can aid in tagging and describing nodes with relevant keywords. This enhances the searchability and contextual understanding of the graph.

Strategies for Structuring Large Knowledge Graphs

1. Scalability: Ensure your infrastructure can handle the increasing volume of data. This often involves distributed computing and storage solutions, such as cloud-based platforms that can scale horizontally.

2. Data Modeling: Design your data model with future growth in mind. Anticipate new types of entities and relationships, ensuring that the graph can evolve without significant restructuring.

3. Indexing and Partitioning: Use indexing to speed up queries and improve performance. Partitioning the graph into manageable sub-graphs can also enhance efficiency, especially when dealing with very large datasets.

4. Query Optimization: Develop efficient query strategies to handle complex searches. This might involve using specialized query languages like SPARQL or leveraging graph database technologies that support high-performance querying.

5. Visualization and Interaction: For large knowledge graphs, visualization tools are invaluable. They help in understanding the structure and relationships within the graph, making it easier to navigate and extract insights.

Tools and Technologies

Several technologies and tools are pivotal in constructing and managing large knowledge graphs:

1. Graph Databases: Neo4j, Amazon Neptune, and ArangoDB are designed to handle large-scale knowledge graphs, offering robust querying capabilities and efficient management of complex relationships.

2. Data Integration Tools: Tools like Apache NiFi and Talend facilitate seamless ingestion and integration of data from various sources into your knowledge graph.

3. Ontology Management: Protégé and TopBraid Composer are popular tools for creating and managing ontologies, ensuring your knowledge graph has a solid structural foundation [1].

Conclusion

Structuring really large knowledge graphs is a multifaceted challenge that requires careful planning, robust technology, and a strategic approach to data integration and management. By focusing on scalability, data modeling, and efficient querying, and drawing inspiration from traditional knowledge structures like the Dewey Decimal System, we can harness the full potential of knowledge graphs to drive innovation and insight across various fields. As we continue to refine these processes, the power and utility of knowledge graphs will only grow, unlocking new opportunities for understanding and leveraging the vast sea of data that defines our world.

In the spirit of continuous improvement and adaptation, our journey with knowledge graphs reflects the age-old pursuit of knowledge itself—ever-evolving, always striving for greater understanding.

Footnotes:

[1] https://protege.stanford.edu/

What’s next for The Lindahl Letter? 

* Week 162: Indexing facts vs. graphing knowledge

* Week 163: Self-Supervised Learning

* Week 164: Graph-Based Feature Engineering

* Week 165: Federated Feature Engineering

* Week 166: Explainable Feature Engineering

If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. Stay curious, stay informed, and enjoy the week ahead!

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

Over the years the team over at Google has made a really big knowledge graph that you can access via an API and that they use as an informational backbone [1]. In some ways it is the best of what the old web had to offer. They note that it is a database of billions of facts. We are starting to see the creation of just a ton of middling, mediocre, or otherwise terribly written content online [2][3]. Now imagine you had built a knowledge graph of billions of facts. You can’t stop updating that large of a knowledge graph. It would grow stale so quickly with how fast the intersection of technology and modernity is occurring. Let me say that another way you now face a situation where a great flooding of bad content is going to overwhelm your knowledge graph. Yeah a tsunami of imagined information and otherwise hallucinated content is going to destabilize the integration of that knowledge graph. Even the notion that it would be built on facts begins to fade away as a sea of LLMs spit out confusion in the form of very confidently written fabrication.

I’m now going to dig into the world of thought related to combining or using in concert LLMs and knowledge graphs. Probably the most interesting breakdown for the future of knowledge graphs will be proprietary locked in ones vs. the decentralized knowledge graphs that could even be powered by a blockchain [4]. We are going to see a huge battle between decentralized knowledge graphs that maybe use even a federated approach to stay fresh and the near monolithic large knowledge graphs that individual corporations are trying to keep perpetually fresh. One paper dealing with the combination of LLMs and knowledge graphs would be, “Large Language Models and Knowledge Graphs: Opportunities and Challenges,” that was published back in 2023 [5]. Another paper from 2023 would be, “Connecting AI: Merging Large Language Models and Knowledge Graph,” that is generally covering the same conceptual landscape [6].

Some people are starting to make arguments that maybe the internet is really starting to break. The internet, once hailed as a beacon of boundless opportunity, now finds itself at a precarious crossroads clouded by mounting concerns. As behemoth tech entities tighten their grip, questions of control and influence darken the digital horizon. Privacy breaches and the insidious spread of misinformation cast a long shadow over its once-promising landscape. Algorithms meant to connect have inadvertently fueled division, while the exploitation of personal data raises profound ethical quandaries. Meanwhile, the internet's infrastructure strains under the weight of cyber threats and a persistent digital divide, where access remains unequal and opportunities unevenly distributed. Yet amidst these challenges, a sense of loss pervades—the fading promise of an open, inclusive digital future. Navigating this uncertain terrain demands a collective effort to reclaim the internet's original ideals of empowerment and connectivity, ensuring it remains a force for good amid mounting challenges.

Footnotes:

[1] https://support.google.com/knowledgepanel/answer/9787176?hl=en 

[2] https://www.niemanlab.org/2022/12/the-ai-content-flood/ 

[3] https://www.thealgorithmicbridge.com/p/how-the-great-ai-flood-could-kill

[4] https://medicpro.london/decentralised-knowledge-graphs/ 

[5] https://arxiv.org/pdf/2308.06374 

[6] https://www.computer.org/csdl/magazine/co/2023/11/10286238/1RimWA0RzFK 

What’s next for The Lindahl Letter? 

* Week 161: Structuring really large knowledge graphs

* Week 162: Indexing facts vs. graphing knowledge

* Week 163: Self-Supervised Learning

* Week 164: Graph-Based Feature Engineering

* Week 165: Federated Feature Engineering

If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. Stay curious, stay informed, and enjoy the week ahead!

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

Things are starting to align within this renewed writing project as my content creation process gets back into some semblance of a proper routine. We are getting pretty close to a place based on the current state of technology where my weekly podcast audio could be produced using a model based on my voice in a matter of seconds. That is not really something that I am considering. I have recorded the last two podcast episodes using my newly acquired MacBook Air using the freely supplied GarageBand software instead of using Audacity on my Windows powered desktop computer. I’m still using the Yeti X microphone and a Marantz Sound Shield Live professional vocal reflection filter, but the operating system and software being used for recording the audio is very different. For scientific purposes, you are welcome to go back and listen to a few of the previous recordings and then check out any episode from 157 forward to see if the audio quality is different. I think the overall quality of the recording is higher with the new setup. 

We are going to jump into the deep end of featurization for machine learning this week. To achieve that effort in practice a series of potential next level featurization techniques will be evaluated. Yes – you guessed it, a new series is forming. Within 7 upcoming editions of this Substack newsletter, including upcoming weeks 163 to 169, I’m going to try to pull together some solid coverage and include some academic articles to read related to these topics. You know that I strive to find the best open research papers to share. Things that reside behind a paywall where practitioners and pracademics cannot easily read them I tend to exclude from these missives. That is a choice that is being made on purpose to favor open research. I’ll be really digging into each of these topics in more detail during some future missives. On a side note, It’s about time to refresh my open source intro to machine learning syllabus as well [1].     

Here are some concepts that to me are highly promising strategies in feature engineering that represent good places to focus understanding as we move toward the future of the field:

* Self-Supervised Learning: Leveraging large amounts of unlabeled data to automatically learn feature representations.

* Graph-Based Feature Engineering: Utilizing graph neural networks to capture relationships and dependencies in graph-structured data.

* Federated Feature Engineering: Creating features in a decentralized manner to enhance privacy and security by keeping data distributed.

* Explainable Feature Engineering: Developing features that improve model interpretability and explainability.

* Adaptive Feature Engineering: Using dynamic techniques that evolve features based on real-time data and model feedback.

* Synthetic Data Generation: Generating synthetic datasets to create new features and augment training data.

* Transfer Learning for Features: Reusing feature representations learned from one domain or task to another, reducing the need for extensive feature engineering in new tasks.

These strategies power feature engineering by providing more advanced, adaptive, and interpretable features for cutting-edge machine learning models. Feature engineering is crucial to machine learning for several reasons:

* Improves Model Performance and Efficiency: Well-engineered features enhance the predictive power and efficiency of machine learning models, leading to better accuracy and faster convergence during training.

* Simplifies Complexity and Enhances Interpretability: Effective feature engineering simplifies the problem space, making models easier to understand and interpret, thereby increasing stakeholder trust in the model's predictions.

* Incorporates Domain Knowledge and Handles Diverse Data: Integrating domain-specific knowledge and transforming diverse data types into a consistent format ensures models can process information effectively and produce relevant results.

* Addresses Data Quality and Robustness: Feature engineering helps clean and normalize data, handle missing values and outliers, and improves the model's robustness to changes in data distribution and external conditions.

Now that the foundation has been set for considering featurization within the machine learning space you can sit back and relax as these topics receive even more evaluation within future editions of this newsletter. 

Footnotes:

[1] https://www.researchgate.net/publication/362679091_An_independent_study_based_introduction_to_machine_learning_syllabus_for_2022 

What’s next for The Lindahl Letter? 

* Week 160: Increasingly problematic knowledge graph updates

* Week 161: Structuring really large knowledge graphs

* Week 162: Indexing facts vs. graphing knowledge

* Week 163: Self-Supervised Learning

* Week 164: Graph-Based Feature Engineering

If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. Stay curious, stay informed, and enjoy the week ahead!

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

Longtime readers of my work know that within my normative bias I tend to break things down into form, function, assumptions, and structure (FFAS). Instead of taking that path with this Substack based letter format, each week my commentaries are going to drift into a more ongoing narrative about the patterns, traditions, and concerns that rise to the forefront of my thoughts. All right, now is the time, let’s jump into that narrative at the deepest end of the things being considered. Maybe the theme of today is about thinking globally and building action locally. That trope remains popular and will continue to be popular moving forward. Zooming out to the global view of things a number of ongoing narratives abound these days with a sea of digital content being created. A great flood of information has been intensifying. I would actually begin to build up an argument that even the best collections of knowledge are going to start breaking down as the flood intensifies. We are now seeing the script get flipped and things are going from macro flooding to incredibly local models that are unique to individual computer operating systems. We are probably going to have to see some sort of defense against actors trying to federate all the local models into a larger system of trading.

Apple executives are reasoned and measured in the deployment of products. Like many of you, I was seriously curious to see what they would do at Apple as AI hype reached a crescendo. Earnings calls and forecasts seem to be triangulated on what AI will do for a company. Apparently, Apple Intelligence is going to do a lot of things [1]. It’s going to do so many things within the Apple ecosystem that endless hours have speculated about it. Google had the opportunity to really bring all the data within their ecosystem together in a very local way, but for some reason they just did not deliver on that potential. We are seeing Microsoft teams bringing forward a feature called recall which uses continuous screenshots with a local model. We are also starting to see the arrival of Copilot machines [2]. That means both Apple and Microsoft are going to provide very local personalized AI experiences. It’s unlikely that Apple executives will try to capture local user data and use it for federated LLM training. However, our friends at Microsoft will probably call this anonymized continuous learning a feature that enhances the model. 

Over the last couple of years I have had numerous conversations with people about understanding ROI related to technology projects. During those chats I try to explain that AI or ML is not really the product of the future for most companies. Telling people that their company is probably not the one that will become infinitely rich off of AI is always dicey. I think the underlying technology, models, and methods will become commoditized. Whatever company emerges at the top may have a brief advantage, but it will fade quickly as no moat exists for a repeatable idea. Generally speaking, most companies will end up using AI/ML to augment, automate, or add features to products they already have or are considering building. We are starting to see major players like Apple, Google, and others explaining that AI will power features or delivery in core products. It’s happening now in terms of announcements and we are now waiting for all the future AI features to launch. Maybe at some point along the way during the true intersection of technology and modernity we will see an AGI event or something on that level. 

All the future AI features are about to get a lot more coverage as people realize just how much has been spent to get to where we are right now. Nvidia as a company had a stock split and has the largest market cap after recently beating out both Apple and Microsoft. A mind boggling amount of money has been spent training models in both the cloud and in terms of hardware. We are starting to see some more specialized hardware showing up to the party. For some companies the GPU was the coin of the realm. Over at Hugging Face you can take a look at the open LLM leaderboard to see what model is currently king of the hill [3]. You will see from that sightseeing tour of the Hugging Face leaderboards that a lot of different LLM models exist right now and a lot of them are bunched up in terms of the rankings. What is interesting about that is that you can download models. Some of them even run locally and are pretty accessible in terms of deployment and usability.  

Footnotes:

[1] https://www.apple.com/apple-intelligence/ 

[2] https://blogs.microsoft.com/blog/2024/05/20/introducing-copilot-pcs/ 

[3] https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard 

What’s next for The Lindahl Letter? 

* Week 159: The next level of featurization

* Week 160: Increasingly problematic knowledge graph updates

* Week 161: Structuring really large knowledge graphs

* Week 162: Indexing facts vs. graphing knowledge

If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. Stay curious, stay informed, and enjoy the week ahead!

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

It’s time to get ruthless about actually managing my epic writing backlog. You can rewind to a previous Substack newsletter with a search or for you subscribers you can check your previous email for the epic 2024 predictions post on January 19, 2024. That was according to my records the post for week 156. Instead of starting up a season two of the podcast I’m just going to lean into the signal of things and publish this missive as week 157 of the Lindahl Letter. For those of you who are new to this ongoing chautauqua of learning and consideration, welcome to the journey. Let’s refocus on working to fix that problematic backlog. Right now it is a Google Doc stored backlog with 147 line items or topics (years worth) that were cataloged for future coverage. At this point in that writing journey, I’m not entirely sure that during that backlog acceptance process a degree of good judgment was used. A lot of things piled up and were not ruthlessly screened for quality or the best possible adventure. 

You can’t stop the signal. We as a society have opened the door to a never ending, always growing, or perpetually flooding stream of content. Even experts in their respective fields of study are facing more content being created than can be consumed. At that point, even the experts are having to gate, limit, or constrain the universe of possible material to consider. We may have hit that weird tipping point where no matter what the amount of content that exists it is greater than what can be consumed. Not only are we at the edge of technology intersecting with modernity, but also we have crossed the maximum of human consumable knowledge. No matter what as we go forward even the best specialized experts will have a limited view of the possible universe of knowledge. You have to pick the best possible window of understanding. That means that really only the researchers at the edge of what is possible will be able to define what’s next, but only for a certain window of time which will quickly be reframed by new windows.

One way to look at the flooding of academic articles is to evaluate how reviewers (functionally the gatekeepers) are being impacted by the flooding of content [1][2]. Maybe just maybe the review system itself will break down and something else will need to be created. I actually favor a system where each university willing to do the work as a department would be the home of a journal and the system is generally more open for people to be able to read academic research. I think this will focus and push clear research trajectories forward. That system might just help push things along toward a system where the answer is to conduct more research and publish it. All that research will beget more writing at the edge of knowledge. Questions will be answered. New questions will appear and that cycle will continue going forward. A lot of academic articles should be sorted by contribution level to the academy. That might help researchers limit the universe of articles that need to be reviewed during any given research project. 

Footnotes:

[1] Hanson, M. A., Barreiro, P. G., Crosetto, P., & Brockington, D. (2023). The strain on scientific publishing. arXiv preprint arXiv:2309.15884. https://arxiv.org/abs/2309.15884 

[2] Thelwall, M., & Sud, P. (2022). Scopus 1900–2020: Growth in articles, abstracts, countries, fields, and journals. Quantitative Science Studies, 3(1), 37-50. https://direct.mit.edu/qss/article/3/1/37/109076/Scopus-1900-2020-Growth-in-articles-abstracts 

What’s next for The Lindahl Letter? 

* Week 158: All the future AI features

* Week 159: The next level of featurization

* Week 160: Increasingly problematic knowledge graph updates

* Week 161: Structuring really large knowledge graphs

* Week 162: Indexing facts vs. graphing knowledge

If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. Stay curious, stay informed, and enjoy the week ahead!

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com