We all thought you would be able to easily ask the house to turn on all the lights or command your television by voice to do things by walking into the room. Some of us thought the entire wall would be a television screen by now and that has not happened either. However, some of the new 100 inch TVs on the market are really large. Enabling automated actions is what is happening within the latest development kits related to the companies making and contributing to LLMs. We have seen Google teams introduce low stakes automated actions like making dinner reservations or screening calls. Those types of training activities help them build and reinforce automations without really doing anything particularly risky. It’s all about the ecosystems and when Google teams start to really deeply allow an assistant based agent to do things that are deeply integrated at that point things are going to rapidly change. That is when your agent will be empowered to the point of being able to really automate some things that will be impactful. Sam Altman of OpenAI has said that 2025 will be the year that we will see agents working effectively [1]. I’m guessing that Sam has spent some time thinking deeply about what these agents are going to be capable of doing. 

We will probably start to see Google calendar automations where meetings with unfulfilled action items automatically get scheduled or task follow ups by chat can come from the agent. This type of recursive review of things that happened where a transcript is recorded and checked against a project plan or calendar is certainly on the roadmap. It’s going to be about bringing the next set of low stakes actions to the business world and calling it revolutionary. A lot of hype is going to occur. Sure systems with robotic process automation or coded workflows have been able to automate things for people willing to invest in those automations. With the advent of agents that are able to schedule automated actions it changes the barrier to entry by fundamentally lowering it. People are probably going to be more willing to trust one of the known major brands with this technology considering that most smartphones have banking information saved and are logged into a myriad of other consequential accounts. Having practical limits on what agents are able to enable in terms of automation remains probably the most important process enablement gate to be considered. Apparently the teams at Google don’t expect to deploy any useful agents until 2025 at the earliest [2].

The push toward true agent-based automation is an ongoing journey. While current tools may seem like small, incremental advances—like handling calendar follow-ups or screening calls—they represent foundational steps toward a more integrated, intuitive digital ecosystem. As AI agents begin to bridge the gap between simple command-driven functions and context-aware actions, we’re stepping into an era where automation isn't just a convenience but a fundamental part of daily life. This gradual transformation will bring more impactful applications, positioning agents not as isolated tools but as active partners in our productivity.

Footnotes:

[1] https://www.tomsguide.com/ai/chatgpt/the-agents-are-coming-openai-confirms-ai-will-work-without-humans-in-2025 

[2] https://techcrunch.com/2024/10/29/google-says-its-next-gen-ai-agents-wont-launch-until-2025-at-the-earliest/ 

If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. Make sure to stay curious, stay informed, and enjoy the week ahead! 

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

This post is all Dr. Flint Dibble’s fault. That fault is wholesale based on all the #realarchaeology posts that have been flying around the internet [1]. If you missed all that online content about archaeology, then consider diving into all that informationally rich real archeology content. This week Flint got me thinking about how artificial intelligence and machine learning fit within the broader academic domain of archaeology. Technology is always approaching the intersection with modernity. Our technology now is ultimately becoming very different based on what the fields of AI, ML, and robotics are able to accomplish. I should have thrown quantum computing on that list, but I’m still a little skeptical about scalability.  

A lot of modern archaeology on television is about the discovery part of the process. Finding things like buried treasure, missing cities, or maybe a significant shipwreck yields pure excitement. Just this week I watched an episode of the ongoing television show Expedition Unknown with Josh Gates digging at Petra [2]. You may be aware that I’m generally interested in all things Indiana Jones related and this adventure certainly was. Josh Gates joined Dr. Pearce Paul Creasman onsite for the discovery and the American Center of Research is the group facilitating the actual archaeology.  

Generally speaking, finding new things is hard, but the process of trying to understand them is where the work of archaeology happens. Applying some type of scientific rigor to the process of figuring things out brings forward quality and makes the process definable and repeatable. We have done a lot of exploring and studying the world with satellites every day. A lot of laboratory, office, or digging work happens that is more hands-on and is about the academic parts of archaeology. That is where I was curious about how both AI and ML fit into the actual practice of archaeology. I wondered what people are doing with advanced technology. I could easily imagine people trying a machine learning model to evaluate satellite images to try to find structures in a jungle or desert. You could use a machine learning model to match images of text fragments or match a partial text to other larger texts. 

I’m going to share my top 10 thoughts about how AI or ML could be impactful within the field of archeology. 

* Automated Site Detection: Using AI to analyze satellite images and locate hidden archaeological sites

* Predictive Modeling: Guided site discovery by predicting likely artifact locations from geological and historical data

* Excavation Data Analysis: Speeding up artifact categorization and soil dating during digs

* 3D Reconstruction: Rebuilding artifacts or sites digitally to visualize original structures

* Text Decipherment: Using AI to decode ancient texts and connect languages or symbols

* Remote Sensing Interpretation: Processing LiDAR and radar data to reveal hidden structures

* Artifact Classification: Identifying and classifying artifacts using computer vision

* Preservation Monitoring: Predicting and preventing environmental damage to sites

* Cross-dataset Analysis: Finding patterns across separate data sources for deeper historical connections

* Virtual Archaeology and Immersive Experiences: AI can create virtual reality (VR) and augmented reality (AR) experiences, allowing researchers and the public to explore reconstructed sites and artifacts interactively

Beyond considering that list, you know I went out to Google Scholar and started to look for highly cited papers within this space [3]. I pulled together 5 papers you can read about AI and ML within the academic space of archaeology. None of these papers have very high citation numbers so they are not widely read like those papers I recently shared from Dr. Geoffrey Hinton that bridged 100,000 citations. These papers were in the sub 100 citation range and that does mean people are reading them, but not at the same prolific rates of core AI or ML papers. 

Bickler, S. H. (2021). Machine learning arrives in archaeology. Advances in Archaeological Practice, 9(2), 186-191. https://www.researchgate.net/profile/Simon-Bickler/publication/351713328_Machine_Learning_Arrives_in_Archaeology/links/60a62e36a6fdcc731d3ea200/Machine-Learning-Arrives-in-Archaeology.pdf 

Barceló, J. A. (2007). Automatic archaeology: Bridging the gap between virtual reality, artificial intelligence, and archaeology. https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=9f03a88221022b93d07f271037776a2c01099fce 

Mantovan, L., & Nanni, L. (2020). The computerization of archaeology: Survey on artificial intelligence techniques. SN Computer Science, 1(5), 267. https://arxiv.org/pdf/2005.02863 

Casini, L., Marchetti, N., Montanucci, A., Orrù, V., & Roccetti, M. (2023). A human–AI collaboration workflow for archaeological sites detection. Scientific Reports, 13(1), 8699. https://www.nature.com/articles/s41598-023-36015-5 

Argyrou, A., & Agapiou, A. (2022). A review of artificial intelligence and remote sensing for archaeological research. Remote Sensing, 14(23), 6000. https://www.mdpi.com/2072-4292/14/23/6000 

The broader implications of how AI is reshaping archaeology’s future are currently unfolding. What we’re seeing now is that technology is actively transforming archaeology. As AI helps detect patterns and analyze data in ways humans simply can’t, it’s redefining the possibilities within this field. AI isn’t here to replace the archaeologist’s trowel but rather to enhance their insights. It’s an assistive tool, an enabler, expanding the reach and depth of what we can know about our past. And that’s exactly why this conversation about AI in archaeology matters—it’s the next step in understanding not just what we’ve found but, ultimately, what it means.

Footnotes:

[1] https://real-archaeology.com/

[2] https://www.cnn.com/2024/10/12/science/petra-tomb-indiana-jones-discovery/index.html 

[3] https://scholar.google.com/scholar?hl=en&as_sdt=0%2C6&q=artificial+intelligence+archeology&btnG= 

What’s next for The Lindahl Letter? New editions arrive every Friday. If you are still listening at this point and enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. Make sure to stay curious, stay informed, and enjoy the week ahead! 

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

Thank you for tuning in to this audio only podcast presentation. This is week 166 of The Lindahl Letter publication. A new edition arrives every Friday. This week the topic under consideration for The Lindahl Letter is, “Those recent Nobel Prizes.”

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

For those of you listening to the audio version this week you may have noticed that we moved from the natural vocal capture to the narration vocal capture. I’m still recording the podcast audio for this adventure using a Blue Microphones Yeti X and my MacBook Air with the included GarageBand software. To further enhance your listening experience I’m still working to dial in the best possible recording technique to deliver superior podcast audio. 

Last week we really dug deep into the major corporate players that released quantum computing programing languages for general use. This week unfortunately needs to start with an epic spoiler alert. Please know that you should be aware that it does not appear quantum computing is practical at this time or really very scalable. I have read a fair number of jokes throughout the last week about neither the blockchain or quantum computer being scalable. With that spoiler delivered upfront this week it is time to move from the breadth of coverage to the depth of understanding related to what people are actually doing with practical use cases within the quantum computing space. One of the places I went to search around and learn a little bit more about use cases was the NASA Quantum Artificial Intelligence Laboratory [1]. The good folks over at NASA shared a paper in June of 2024 that was about “Assessing and Advancing the Potential of Quantum Computing: A NASA Case Study” [2]. For those of you wanting to read more quantum computing papers this is pretty easy to read and digest. It’s 27 pages and is very well cited throughout. 

You can join the quantum computing reddit community and it seems to be pretty active with academic discussion amongst the 50k+ members [3]. Beyond that reddit community the next place I ended up spending some time looking around was the Quantum Open Source Foundation which had a lot of content in terms of links [4]. They have a lot of curated links on the GitHub page that they maintain [5]. You could spend hours and hours of time just clicking around and looking at all of those projects. Eventually I ran into another GitHub repository called Awesome Quantum Computing that is another collection of curated links [6]. These collections of links will send you all over the place to see some interesting projects people are developing. 

Somehow during the hunt for the best projects using quantum computing to accomplish things I ended up back looking at the IBM Quantum Learning pages to see what things they were encouraging people to code as they learn to program [7]. The Azure Quantum team had a whole section devoted to trying to explain what solutions they are offering [8]. A lot of that seems to be focused on physics, chemistry, and ultimately material discovery and other applied applications to understand some type of complex interaction. Modeling really complex things seems to be a core use case that quantum computing has centered on based on the available evidence. I really think at this point I’m going to invest some time into completing a couple of these courses to get more hands on in the quantum computing space. I’ll share one last note about the fastest quantum computer that now has 1,180 qubits from Atom Computing [9]. It was a sizable leap from the IBM’s Osprey that was capable of 433 qubits. Hopefully we will see a bunch of fastest quantum computer records broken in the coming years. That will be a good sign that things have forward momentum in the quantum computing space.

Things to consider this week:

* You might want to read the 15,000 word October 2024 essay from Anthropic CEO Dario Amodei called “Machines of loving grace.” https://darioamodei.com/machines-of-loving-grace 

* The Anthropic team also released an updated responsible scaling policy that is a related read with the Dario Amodei essay https://www.anthropic.com/news/announcing-our-updated-responsible-scaling-policy 

* I enjoyed listening to Yannic Kilcher talk about and question a paper this week “GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models” https://arxiv.org/abs/2410.05229

Footnotes:

[1] https://www.nasa.gov/intelligent-systems-division/discovery-and-systems-health/nasa-quail/ 

[2] https://arxiv.org/abs/2406.15601 

[3] https://www.reddit.com/r/QuantumComputing/ 

[4] https://qosf.org/project_list/ 

[5] https://github.com/qosf/awesome-quantum-software 

[6] https://github.com/desireevl/awesome-quantum-computing 

[7]

https://learning.quantum.ibm.com/

 

[8] https://quantum.microsoft.com/en-us/solutions/azure-quantum-solutions 

[9] https://www.newscientist.com/article/2399246-record-breaking-quantum-computer-has-more-than-1000-qubits/ 

What’s next for The Lindahl Letter? At some point this series will move back to being planned out 5 weeks ahead of publication. 

If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. Make sure to stay curious, stay informed, and enjoy the week ahead!

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

Over the last week I started digging into quantum computer programming to see what is currently going on and what people are doing in that space [1][2]. General availability of actual quantum computing systems is the major barrier to using one to write some code [3]. You can buy time on a quantum computer or you can use a simulator. The simulators are sort of weird in general as the computing power you are using to simulate the quantum computer has a very small fractional power equivalent in terms of computing power. The biggest thing to really consider in terms of understanding where we are in terms of quantum computing is not cost or availability, but instead its a true question about the computing method itself related to having fully fault-tolerant quantum computing. Not only do you have to have methods for error correction within your quantum computing setup, but also you need a design that has proper fault-tolerant gates to avoid introducing increasing error levels within your computing. You are probably wondering if I went out to Google Scholar to find papers about fault-tolerance in quantum computing [4]. Of course that was where I went to look for papers. Here are 3 academic papers you can read with over 300 citations to consider:

Steane, A. M. (1999). Efficient fault-tolerant quantum computing. Nature, 399(6732), 124-126. https://arxiv.org/pdf/quant-ph/9809054 

Chow, J. M., Gambetta, J. M., Magesan, E., Abraham, D. W., Cross, A. W., Johnson, B. R., ... & Steffen, M. (2014). Implementing a strand of a scalable fault-tolerant quantum computing fabric. Nature communications, 5(1), 4015. https://www.nature.com/articles/ncomms5015 

Preskill, J. (1998). Fault-tolerant quantum computation. In Introduction to quantum computation and information (pp. 213-269). https://arxiv.org/pdf/quant-ph/9712048 

Let’s set aside those questions about making quantum computing sustainable and consider the code part of the equation. Major programming languages for quantum computing do exist from the players you would expect. They tend to have a lot of documentation and GitHub profiles to share the code. You could spend whole days looking at some of the languages made by major players. Here are 3 examples of major players making quantum computing language contributions. First, Qiskit was introduced as a quantum computing programming language by IBM teams back in 2017 [5][6]. Second, teams at Microsoft introduced Q# back in 2017 [7][8]. Third, Cirq is from Google AI and was introduced in 2018 [9][10]. You can start to dig into those code bases and I could find a lot of content related to the languages and a lot of hype about quantum computing. 

You can see that a lot of evidence exists related to quantum programming languages. The next logical questions would be how do they execute that code in practice and maybe what exactly are they doing with this quantum code. The teams at IBM have been pretty good about sharing plans to build faster and faster quantum computers [11]. You could watch the hype film about the IBM Quantum System Two that they shared to YouTube back on December 4, 2023. It’s always interesting to look at quantum computer builds; they are not the sort of thing that is going to sit under my desk or on my desk at this point within the technology curve. 

After all that foundational digging into the current state of quantum computing my thoughts are still conflicted about it. I know some questions exist about exactly how to deploy large scale efforts that work without propagating errors and the complexity of fault-tolerant gates. Probably the next step in my research process will be to find some examples of solid quantum computing code being used or actually deployed. That is maybe the best way to get a sense of what is being done within the quantum computing space. That research note about practical quantum computing and getting things done will probably shed some light on where things are trending. Beyond people setting records for the fastest machine or sharing hype videos the real deeper question here is what use cases are going to end up defining the technology. 

Footnotes:

[1] https://towardsdatascience.com/an-introduction-to-quantum-computers-and-quantum-coding-e5954f5a0415 

[2] https://www.mckinsey.com/featured-insights/mckinsey-explainers/what-is-quantum-computing

[3] https://github.com/qosf/awesome-quantum-software 

[4] A Google Scholar search for “fully fault-tolerant quantum computing” https://scholar.google.com/scholar?q=fully+fault-tolerant+quantum+computing&hl=en&as_sdt=0&as_vis=1&oi=scholart 

[5] https://www.ibm.com/quantum/qiskit 

[6] https://github.com/qiskit 

[7] https://learn.microsoft.com/en-us/azure/quantum/qsharp-overview 

[8] https://github.com/microsoft/qsharp 

[9] https://quantumai.google/cirq 

[10] https://github.com/quantumlib/Cirq 

[11] https://www.fastcompany.com/90992708/ibm-quantum-system-two 

What’s next for The Lindahl Letter? At some point this series will move back to being planned out 5 weeks ahead of publication. That point has not arrived just yet so you can expect a more fluid selection of topic coverage. 

If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. Make sure to stay curious, stay informed, and enjoy the week ahead! 

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

We are facing a new reality where the tipping point has been passed and a majority of content being generated online within the internet is now synthetic. Organic content generation simply cannot keep up with the flood of synthetic content. Those bot farms creating content never sleep. They just churn out content and pretend that it remains evergreen. My consideration on this topic started back on Wednesday, July 31, 2024, when I was invited to access SearchGPT from OpenAI [1]. Using that platform made me think a lot about how we access information and the ways that will change going forward. People are now getting summaries and completing searches that go beyond Googling something. You are probably well aware by now that I’m deeply concerned about how facts and knowledge are going to be stored and curated going forward. 

Going forward whoever owns the stores of facts or knowledge will effectively own history and how it is presented which is truly a watershed change in our shared understanding of the world. Individual voices and publications will be overshadowed by these collections. Owning the datastores that provide definitive facts or knowledge will be the cornerstone of whatever emerges going forward and should not be underestimated in terms of future value. I don’t think the ownership of facts will become commoditized and open sourced. I really do think it will be privatized and tightly controlled. Somebody who wanted to pivot our understanding on a particular point of inquiry could just start serving up that alternative perspective. Instead of people funding think tanks to ultimately change the messaging the next step will be funding content farms to just flood the message delivery. Keep in mind that people generally are not really reading books anymore [2]. That means that reasoning during the course of interpreting information may be a diminishing skillset.

Organically written original content exists online. Synthetically generated content has been on the rise. A lot of bots are scraping content for model training and trying to figure out what is organic and what is synthetic has become increasingly difficult. One of the hallmarks of my writing efforts has been the originality or maybe novelty of my research efforts. Within the broader context of the academy of academic thought, original contributions are what build that content and strengthen it overall. Diluting, derivative, and otherwise mediocre publications just flood the overall academic community. It’s perfectly fine to write a publication and decide it was not a significant contribution. Instead of maybe holding back those lesser works they are now freely shared in online archives and unfortunately a new generation of journals. The increase in AI related publications has been astonishing from 2010 to 2022 the number of publications nearly tripled [3]. 

Now we have a mix of the poorly written articles mixing with the synthetically generated to create a truly problematic future of consuming content. I’m considering web traffic at the moment, but the overall storage of facts and knowledge is certainly in scope. We reached the tipping point around 2016 where more traffic is mobile traffic than from a desktop browser [4]. OpenAI has now launched SearchGPT and beyond the dichotomy between mobile and desktop traffic we are about to see the rise of LLM interpreted results where people may never actually leave the landing page or interface of the search engine. It’s possible that dichotomy will fade away and the majority of traffic will be from bots scraping things to share within the newly powered search interfaces. People may very well interact with the grand volume of online information from applications using APIs to respond that are completely disconnected from what was the open internet people surfed and experienced. From reading the thoughts of a single writer to interpreting the output of the largest language models ever created. Things are changing at an incredibly rapid pace.

Now it’s time for a brief editorial note. Please note that my writing output over the last few years became over indexed on artificial intelligence and machine learning. Going down that rabbit hole was good at first and it was an effort truly focused on depth and breadth within the subject. Unfortunately, my focus lingered and instead of writing research notes about technological innovation, civil society, and the intersection of technology and modernity that pesky over indexing occurred. Now thanks to a moment of reflective practitioning I’m breaking out of that pattern and returning to what I consider a better balance of writing topics. Thank you for being along for that journey and the upcoming course correction.

Footnotes:

[1] https://chatgpt.com/search  

[2] https://www.pewresearch.org/short-reads/2021/09/21/who-doesnt-read-books-in-america/  

[3] https://arxiv.org/abs/2405.19522 or https://aiindex.stanford.edu/wp-content/uploads/2024/04/HAI_AI-Index-Report-2024_Chapter1.pdf 

[4] “>50% of web traffic comes from mobile.” Google Analytics Data, U.S., Q1 2016. https://www.thinkwithgoogle.com/marketing-strategies/app-and-mobile/mobile-web-traffic-statistics/ 

What’s next for The Lindahl Letter? I’m just going to sit down and write weekly or more likely bi-weekly so the topic won’t be planned out 5 weeks ahead of publication.

If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. Stay curious, stay informed, and enjoy the week ahead!

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

In the digital age, the methods we use to organize and comprehend information are continually evolving. Two significant approaches stand out: indexing facts and graphing knowledge. Both play essential roles in how we structure, retrieve, and understand data, but they serve distinct purposes and offer different advantages.

Let’s start out by looking a little deeper into indexing facts. Indexing is a traditional and straightforward method. It involves categorizing and listing information in a manner that allows for easy retrieval. Think of it as a library catalog, where every book has a unique identifier and is placed in a specific location based on its subject. This system is incredibly efficient for finding discrete pieces of information quickly. For example, a keyword search in a database relies heavily on indexing.

Indexes are foundational to databases and search engines. They allow us to locate specific data points without having to sift through every piece of information manually. This method is highly effective for tasks that require precision and speed. However, indexing has its limitations. It often lacks context and relational understanding between different pieces of data. An index can tell you where something is but not necessarily how it connects to other information.

Graphing knowledge, on the other hand, is about mapping relationships between data points. This approach is exemplified by knowledge graphs, which visually represent the connections between different concepts. A knowledge graph is more than a mere collection of facts; it is an interconnected web that shows how different pieces of information relate to one another.

In a knowledge graph, nodes represent entities (such as people, places, or concepts), and edges represent the relationships between these entities. This structure allows for a more holistic understanding of information. For instance, a knowledge graph can illustrate how historical events are connected, how scientific concepts overlap, or how social networks operate.

The advantages of graphing knowledge are manifold. It provides context, reveals patterns, and helps in discovering new insights that might not be apparent through traditional indexing. Knowledge graphs are particularly useful in fields that require deep understanding and analysis, such as artificial intelligence, semantic web technologies, and complex decision-making processes.

The intersection of indexing facts and graphing knowledge represents the future of information management. By combining the precision of indexing with the relational depth of knowledge graphs, we can create systems that are both efficient and insightful. This hybrid approach can enhance our ability to process and understand vast amounts of data, making it possible to derive meaningful insights quickly.

For example, search engines are evolving to incorporate elements of both indexing and graphing. Google's Knowledge Graph is a prime example, aiming to understand the context behind search queries to provide more relevant results. This system not only indexes web pages but also understands the relationships between different pieces of information, delivering a more nuanced response to users' queries.

Indexing facts and graphing knowledge are not mutually exclusive; they are complementary methods that, when combined, can significantly enhance our understanding and management of information. By leveraging the strengths of both approaches, we can build more robust systems that not only store and retrieve data efficiently but also provide deeper insights and understanding. As we continue to advance in the digital age, the fusion of these methods will undoubtedly play a pivotal role in shaping the future of information technology and knowledge management.

What’s next for The Lindahl Letter? 

* Week 163: Self-Supervised Learning

* Week 164: Graph-Based Feature Engineering

* Week 165: Federated Feature Engineering

* Week 166: Explainable Feature Engineering

* Week 167: Adaptive Feature Engineering

If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. Stay curious, stay informed, and enjoy the week ahead!

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

You never really have to worry about how you store knowledge. All that knowledge just gets accumulated day by day and how it is stored, structured, and even retrieved happens without any intervention. It’s just something that you do autonomously. That is very different when you have to manage all that data and store it. Navigating the intricate maze of data science, structuring really large knowledge graphs presents itself as both an art and a science. This endeavor, crucial for deepening our understanding and utilization of complex datasets, entails a series of pivotal steps and considerations. Today, we'll explore the foundational principles and practical strategies for effectively structuring these expansive networks of interconnected information, while drawing on the organizational wisdom of traditional knowledge structures like the Dewey Decimal System (DDS).

Understanding the basics seems to take more and more time these days. Knowledge graphs are more than mere data structures; they represent information through a web of entities and their interrelations. They offer a robust framework for integrating data from a multitude of sources, enhancing our ability to derive richer insights and make more informed decisions. As the size of the knowledge graph expands, so too does the complexity of structuring it effectively.

Key Components of Knowledge Graphs

1. Nodes and Edges: At the heart of any knowledge graph are nodes (entities) and edges (relationships). Nodes can embody concepts, objects, or events, while edges illustrate the interconnections among these nodes. The quality and comprehensiveness of your knowledge graph hinge on the precise definition and linking of these elements. You deal with these types of relationships every day without even an afterthought. The types of things you manage passively are more complex when they have to be handled in a planful way. 

2. Ontology: This serves as the schema or structural framework that delineates the types of entities and relationships within the graph. A well-crafted ontology ensures consistency and coherence, enabling more effective querying and analysis. I’m actually a fan of declaring things as a fact or not a fact and then storing those facts in buckets that are easy to retrieve. 

3. Data Ingestion and Integration: Large knowledge graphs often amalgamate data from various sources. Efficiently integrating this data while preserving its integrity and relevance is a critical challenge, involving data cleaning, normalization, and transformation.

Knowledge Structures and the Dewey Decimal System

Drawing inspiration from the Dewey Decimal System can provide valuable insights into structuring knowledge graphs. The DDS organizes information into a hierarchical, decimal-based classification system, which can serve as a model for categorizing and indexing data within a knowledge graph.

1. Hierarchical Classification: Like the DDS, hierarchical classification in a knowledge graph helps organize information into broad categories and narrower subcategories. This ensures that related entities are grouped together, facilitating easier navigation and retrieval.

2. Decimal Notation: Utilizing a decimal notation system to categorize entities and relationships can add a layer of precision and order to a knowledge graph. Each node and edge can be assigned a unique identifier, akin to how books are classified in libraries.

3. Subject Headings: Implementing subject headings, similar to those in the DDS, can aid in tagging and describing nodes with relevant keywords. This enhances the searchability and contextual understanding of the graph.

Strategies for Structuring Large Knowledge Graphs

1. Scalability: Ensure your infrastructure can handle the increasing volume of data. This often involves distributed computing and storage solutions, such as cloud-based platforms that can scale horizontally.

2. Data Modeling: Design your data model with future growth in mind. Anticipate new types of entities and relationships, ensuring that the graph can evolve without significant restructuring.

3. Indexing and Partitioning: Use indexing to speed up queries and improve performance. Partitioning the graph into manageable sub-graphs can also enhance efficiency, especially when dealing with very large datasets.

4. Query Optimization: Develop efficient query strategies to handle complex searches. This might involve using specialized query languages like SPARQL or leveraging graph database technologies that support high-performance querying.

5. Visualization and Interaction: For large knowledge graphs, visualization tools are invaluable. They help in understanding the structure and relationships within the graph, making it easier to navigate and extract insights.

Tools and Technologies

Several technologies and tools are pivotal in constructing and managing large knowledge graphs:

1. Graph Databases: Neo4j, Amazon Neptune, and ArangoDB are designed to handle large-scale knowledge graphs, offering robust querying capabilities and efficient management of complex relationships.

2. Data Integration Tools: Tools like Apache NiFi and Talend facilitate seamless ingestion and integration of data from various sources into your knowledge graph.

3. Ontology Management: Protégé and TopBraid Composer are popular tools for creating and managing ontologies, ensuring your knowledge graph has a solid structural foundation [1].

Conclusion

Structuring really large knowledge graphs is a multifaceted challenge that requires careful planning, robust technology, and a strategic approach to data integration and management. By focusing on scalability, data modeling, and efficient querying, and drawing inspiration from traditional knowledge structures like the Dewey Decimal System, we can harness the full potential of knowledge graphs to drive innovation and insight across various fields. As we continue to refine these processes, the power and utility of knowledge graphs will only grow, unlocking new opportunities for understanding and leveraging the vast sea of data that defines our world.

In the spirit of continuous improvement and adaptation, our journey with knowledge graphs reflects the age-old pursuit of knowledge itself—ever-evolving, always striving for greater understanding.

Footnotes:

[1] https://protege.stanford.edu/

What’s next for The Lindahl Letter? 

* Week 162: Indexing facts vs. graphing knowledge

* Week 163: Self-Supervised Learning

* Week 164: Graph-Based Feature Engineering

* Week 165: Federated Feature Engineering

* Week 166: Explainable Feature Engineering

If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. Stay curious, stay informed, and enjoy the week ahead!

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

Over the years the team over at Google has made a really big knowledge graph that you can access via an API and that they use as an informational backbone [1]. In some ways it is the best of what the old web had to offer. They note that it is a database of billions of facts. We are starting to see the creation of just a ton of middling, mediocre, or otherwise terribly written content online [2][3]. Now imagine you had built a knowledge graph of billions of facts. You can’t stop updating that large of a knowledge graph. It would grow stale so quickly with how fast the intersection of technology and modernity is occurring. Let me say that another way you now face a situation where a great flooding of bad content is going to overwhelm your knowledge graph. Yeah a tsunami of imagined information and otherwise hallucinated content is going to destabilize the integration of that knowledge graph. Even the notion that it would be built on facts begins to fade away as a sea of LLMs spit out confusion in the form of very confidently written fabrication.

I’m now going to dig into the world of thought related to combining or using in concert LLMs and knowledge graphs. Probably the most interesting breakdown for the future of knowledge graphs will be proprietary locked in ones vs. the decentralized knowledge graphs that could even be powered by a blockchain [4]. We are going to see a huge battle between decentralized knowledge graphs that maybe use even a federated approach to stay fresh and the near monolithic large knowledge graphs that individual corporations are trying to keep perpetually fresh. One paper dealing with the combination of LLMs and knowledge graphs would be, “Large Language Models and Knowledge Graphs: Opportunities and Challenges,” that was published back in 2023 [5]. Another paper from 2023 would be, “Connecting AI: Merging Large Language Models and Knowledge Graph,” that is generally covering the same conceptual landscape [6].

Some people are starting to make arguments that maybe the internet is really starting to break. The internet, once hailed as a beacon of boundless opportunity, now finds itself at a precarious crossroads clouded by mounting concerns. As behemoth tech entities tighten their grip, questions of control and influence darken the digital horizon. Privacy breaches and the insidious spread of misinformation cast a long shadow over its once-promising landscape. Algorithms meant to connect have inadvertently fueled division, while the exploitation of personal data raises profound ethical quandaries. Meanwhile, the internet's infrastructure strains under the weight of cyber threats and a persistent digital divide, where access remains unequal and opportunities unevenly distributed. Yet amidst these challenges, a sense of loss pervades—the fading promise of an open, inclusive digital future. Navigating this uncertain terrain demands a collective effort to reclaim the internet's original ideals of empowerment and connectivity, ensuring it remains a force for good amid mounting challenges.

Footnotes:

[1] https://support.google.com/knowledgepanel/answer/9787176?hl=en 

[2] https://www.niemanlab.org/2022/12/the-ai-content-flood/ 

[3] https://www.thealgorithmicbridge.com/p/how-the-great-ai-flood-could-kill

[4] https://medicpro.london/decentralised-knowledge-graphs/ 

[5] https://arxiv.org/pdf/2308.06374 

[6] https://www.computer.org/csdl/magazine/co/2023/11/10286238/1RimWA0RzFK 

What’s next for The Lindahl Letter? 

* Week 161: Structuring really large knowledge graphs

* Week 162: Indexing facts vs. graphing knowledge

* Week 163: Self-Supervised Learning

* Week 164: Graph-Based Feature Engineering

* Week 165: Federated Feature Engineering

If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. Stay curious, stay informed, and enjoy the week ahead!

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com

Things are starting to align within this renewed writing project as my content creation process gets back into some semblance of a proper routine. We are getting pretty close to a place based on the current state of technology where my weekly podcast audio could be produced using a model based on my voice in a matter of seconds. That is not really something that I am considering. I have recorded the last two podcast episodes using my newly acquired MacBook Air using the freely supplied GarageBand software instead of using Audacity on my Windows powered desktop computer. I’m still using the Yeti X microphone and a Marantz Sound Shield Live professional vocal reflection filter, but the operating system and software being used for recording the audio is very different. For scientific purposes, you are welcome to go back and listen to a few of the previous recordings and then check out any episode from 157 forward to see if the audio quality is different. I think the overall quality of the recording is higher with the new setup. 

We are going to jump into the deep end of featurization for machine learning this week. To achieve that effort in practice a series of potential next level featurization techniques will be evaluated. Yes – you guessed it, a new series is forming. Within 7 upcoming editions of this Substack newsletter, including upcoming weeks 163 to 169, I’m going to try to pull together some solid coverage and include some academic articles to read related to these topics. You know that I strive to find the best open research papers to share. Things that reside behind a paywall where practitioners and pracademics cannot easily read them I tend to exclude from these missives. That is a choice that is being made on purpose to favor open research. I’ll be really digging into each of these topics in more detail during some future missives. On a side note, It’s about time to refresh my open source intro to machine learning syllabus as well [1].     

Here are some concepts that to me are highly promising strategies in feature engineering that represent good places to focus understanding as we move toward the future of the field:

* Self-Supervised Learning: Leveraging large amounts of unlabeled data to automatically learn feature representations.

* Graph-Based Feature Engineering: Utilizing graph neural networks to capture relationships and dependencies in graph-structured data.

* Federated Feature Engineering: Creating features in a decentralized manner to enhance privacy and security by keeping data distributed.

* Explainable Feature Engineering: Developing features that improve model interpretability and explainability.

* Adaptive Feature Engineering: Using dynamic techniques that evolve features based on real-time data and model feedback.

* Synthetic Data Generation: Generating synthetic datasets to create new features and augment training data.

* Transfer Learning for Features: Reusing feature representations learned from one domain or task to another, reducing the need for extensive feature engineering in new tasks.

These strategies power feature engineering by providing more advanced, adaptive, and interpretable features for cutting-edge machine learning models. Feature engineering is crucial to machine learning for several reasons:

* Improves Model Performance and Efficiency: Well-engineered features enhance the predictive power and efficiency of machine learning models, leading to better accuracy and faster convergence during training.

* Simplifies Complexity and Enhances Interpretability: Effective feature engineering simplifies the problem space, making models easier to understand and interpret, thereby increasing stakeholder trust in the model's predictions.

* Incorporates Domain Knowledge and Handles Diverse Data: Integrating domain-specific knowledge and transforming diverse data types into a consistent format ensures models can process information effectively and produce relevant results.

* Addresses Data Quality and Robustness: Feature engineering helps clean and normalize data, handle missing values and outliers, and improves the model's robustness to changes in data distribution and external conditions.

Now that the foundation has been set for considering featurization within the machine learning space you can sit back and relax as these topics receive even more evaluation within future editions of this newsletter. 

Footnotes:

[1] https://www.researchgate.net/publication/362679091_An_independent_study_based_introduction_to_machine_learning_syllabus_for_2022 

What’s next for The Lindahl Letter? 

* Week 160: Increasingly problematic knowledge graph updates

* Week 161: Structuring really large knowledge graphs

* Week 162: Indexing facts vs. graphing knowledge

* Week 163: Self-Supervised Learning

* Week 164: Graph-Based Feature Engineering

If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. Stay curious, stay informed, and enjoy the week ahead!

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.nelsx.com