MetaDAMA - Data Management in the Nordics

«Data Mesh promises so much, so of course everyone is talking about it.»

I had the pleasure to chat with Karin Håkansson about Data Governance in a Mesh. Karin has worked with Data Governance and Data Mesh and is really active in the Data Mesh Community, speaking at podcasts and moderating the LinkedIn Group «Data Governance - Data Mesh»

Here are my key takeaways from our conversation:

Data in Retail

• The culture in retail is about innovation, experimentation, new products. So governance has to adapt to this environment in order to be successful. 
• If retail would do, what we do in data a fashion retailer would sell yarn instead of t-shirts.
• Retail knows what the customer wants, before the customer wants it. What would happen if we in data think like retailers?
• Its more about understanding the business better, then making the business data literate.

Data Governance

• Data Governance best practices in the DMBOK is still relevant, also in a Data Mesh setting.
• Data Governance has been on a journey from compliance driven to business value driven.
• Centralized Data governance creates a bottleneck. Decentralized Governance creates silos. So federated Data Governance is the middle ground.
• Create incentives to create trust.
• If you utilize your platform correctly, you can have high expectations towards computational governance.

Data Mesh

• Data Mesh comes with a cost - you need to invest in Data Mesh.
• But more than anything Data Mesh implementation is an enormous change effort.
• If you don’t know why Data Mesh, you will implement something else
• Implement Data Mesh in an agile way: «start small, fail fast and iterate»
• To start with Data Mesh, work with a business team that is eager to get started and sees the benefits - «You have to have business onboard, otherwise its not going to work.»
• Always check if you get the value that you expected.
• When you do it, make sure you get Governance, business and tech teams to work together and a re aligned on the why.
• Make sure two upskill for Data Mesh - its is fundamentally different: talk about it, have debates, book clubs, ++
• The 4 elements of data mesh: Can you implement those in a sequence or should you look at them as a unite to implement within a limited scope? 
• Start finding ways for people to work together (eg. Common goal, and environment where it is ok to share).
• A good first step is to find an example of data with a certain issue or limitation and talk with the business user about exactly this.
• Data Governance, as much as Data Mesh, is about change management: You need to get close to the business and collaborate actively.
• Your first two steps should be:

1. Start with one business unite, an early adopter.
2. Find their most critical data and talk about actual data.

MDM and Data Mesh

• Are we still hunting for that golden record? How do we work with MDM in a mesh? This is not solved yet.
• You can refer to data, instead of collecting data in a MDM system.
• Maybe the best approach so far are global IDs to track data cross domains, but how you link your data might become the new MDM.
• «You still need to connect the data, but you don’t need to collect the data.» - MDM in a Mesh.

Domains, federation and responsibility

• If you federate responsibility to the domains, they also need the resources and competency to fulfill these responsibilities.
• If the domain data teams are successful in abstracting the complexity, it will become easy to create data products.
• If you scale too fast (faster than your data platform), you might end up having to duplicate teams.

«If we think of Data Mesh as an evolution of data lakes, knowledge graphs are an evolution of Master Data Management.»

Data overload is becoming a real challenge for all types of businesses. With all data that is gathered, both in multiple formats and huge volumes, has created a need for connected, contextualized data. Combined with continuing developments in AI, has resulted in increasing interest in knowledge graphs as a means to generate context-based insights.

I had a fantastic chat with Mozhgan Tavakolifard. Mozhgan describes herself as «incubator and alchemist». She worked with PhD-research on trust and SOME. The techniques Mozhgan used to collect data for her research introduced her to Data Science. On this episode of #MetaDAMA, we talked about Knowledge Graph enabled Data Mesh.

Here are my key takeaways:

Data-driven transformation

• Personal transformation and transformation of businesses are quite similar.
• Very few companies can blame that they are industrializing data and are transformed in a data-driven way.
• For a transformation to be successful, you need to have a holistic view and invest in a practical manner according to your business case.
• Don’t change your business culture to match a data culture, but rather let data be an enabler for the business.

Data Mesh

• Each domain can have a different culture, produce different data products.
• 4 indigents:

1. Give data back to the domains where it is produced
2. Create self-service data infrastructure
3. Federated governance
4. Data as a product

• The centralized data and analytics platform has failed.
• Knowledge Graph can basically be considered as Data Supply Chain for Data Mesh.
• KG can be used to semantically link data products.

Knowledge Graphs

• KG enable a human brain-like approach to derive new knowledge. 
• A KG is quite simply any graph of data that accumulates and conveys knowledge of the real world.
• Every consumer-facing digital brand, such as Google, Amazon, Facebook, Spotify, etc., has invested significantly in building knowledge and the concept of graphs has evolved to underpin everything from critical infrastructure to supply chains and policing. 
• There is a difference between Knowledge graph and graph data store. The semantic layer is what makes data smart.
• KG is when you have a dynamic and rich context around knowledge.
• KG can be used to semantically search for data.
• KG are a very important part of data mesh.
• If you want to start with knowledge graphs: find your business case. What is the purpose for you?

«RIP Semantic Web! The Semantic Web is dead.»

• Maybe part of the problem was that is was academy focused, more than practical, industry-focused.
• If we manage to implement Data Mesh on a society level, we might have taken a big step realizing some parts of the semantic web.

Trust

• Trust and explainability is based on context. Knowledge graphs can provide context and connect between them, this can ultimately generate trust.

«How can you use AI algorithms to make your city more sustainable?»

How can we use AI to work more sustainable and optimize our operations for less pollution and more efficiency? I talked with Umair, Head of Data Science, Data Warehouse and Artificial Intelligence at Ruter AS, the public transport authority in the Oslo region, the Norwegian capital. Umair is also a associate professor at OsloMet, teaching AI to bachelor level student, as well as founder and CTO of Bineric Crowdsourcing and founder of the volunteer organization Offentlig AI.

Public Transportation;:

• «Public transportation is complex, because data is not coming in from many different internal and external sources.
• «A bus can have minimum 20 sensors.» All theses sensors are sending realtime data.
• A huge amount of data is collected through external sources.
• Ruter is going away from centralized Data Management teams to more of a mesh approach.
• Data Mesh will give you a more complex data function, with need for more people and more organization and coordination.
• One team cannot have the full ownership for the entire data-driven prerogative of a company.
• 2 factors that helped Ruter succeed with Data and AI:

1. E2E responsibility for the whole AI algorithm
2. Create in Production and don’t overdue PoCs

Sustainability & AI

1. Capacity prediction

• Predict capacity 3 days in advance, but possible up to 1 month in advance
• This gives passages the possibility to plan their trips better
• An operations team that sees a pick in traffic realtime, send additional busses to ensure enough capacity
• Through the prediction algorithm, fleet capacity can be reduced and it is easy to plan for balancing the load beforehand

1. Fleet management

• The long term vision for Ruter is to work more with Order services. In the future you shouldn’t have to walk to a bus stop, but can order a bus to your home.
• The existing solution is for seniors (67+) and is tested in the Viken area
• But how can you ensure that the busses are close to an eventual future order?
• Ruter is training an algorithm to predict where orders might come from to ensure a bus is parked close by. This results in less driving and less emissions.
• To train the algorithm Ruter uses mainly historical information, but combined with e.g. weather information

1. Analysis of customer feedback

• Sentiment analysis to see how happy/unsatisfied a customer is

Explainable AI

• «AI is just statistics on steroids!»
• Its hard to explain how a probability output is achieved
• That makes an AI algorithm a black box
• Developers create a set of tools and applications which can give insight on different factors how a decision was achieved by an AI algorithm

Quantum Computing

• Traditional computing is expensive and needs more time and resources to reach a specific output
• Volumes of data are constantly increasing
• This was a research project together with OsloMet
• Quantum computing was cheaper to work with then traditional computing
• Quantum Computers are more sustainable and energy efficient

🗓️ 1 Year
🎙️ 2 Podcasts
💡7 Lessons
👨‍💻 50 Data Leaders

Together with Loris Marini, host of the great data podcast Discovering Data, I had a chat live for a retrospect on a fantastic podcast year 2022. Combined, we have had the pleasure to talk to over 50 Data Leaders in 2022 and shared their message with our listeners.

Here are our 7 lessons for 2023:

1. Lead: Data is a shared asset with shared benefits. Data quality is a shared responsibility.

2. Network: Tips to build stronger relationships with your stakeholders.

3. Sell: Map the compelling business need behind your data management initiative.

4. Be memorable: A simple framework to increase your storytelling 10x

5. Support: Enable the business to take lead. Provide guidance.

6. Observe: Don't jump to conclusions, but observe the demographics, ways of working, environment.

7. Have fun: provide a framework to ensure collaborative readiness, physiological safety, and encourage failure and try again.

«Its not just about the use of data, but the use of data in a cross-functional setting.»

What a fantastic conversation with Nina Walberg. Nina has been with Oda since 2019 and has a background in Optimization and SCM from NTNU.

Oda strive  to create a society where people have more space for life. Make life as hassle-free as possible. And to achievetis with help from data, Oda has created its 6 principles for how they create value with data.

Here are my key takeaways:

Business model and use of data

• Odas business model allows for a better and more cautious way of thinking sustainability. The quantum of products can be tailored to the actual need.
• Climate recipe: Oda provides data on the climate footprint to its customers, when ordering products.
• Use data to provide not just inspiration to your customers, but also help them to create a complete basket of groceries for the week, to avoid any additional trips to the super market.
• The use of data needs to be combined with all functions, also business functions. All should be part of the development process

6 Principles

• 6 principles recently updated, but originally created by the entire team in 2019.
• Oda believes in autonomous teams and that trust and responsibility is given to these team.

1. Domain knowledge and discipline expertise
   • 70% of Data and Insight people are embedded in cross-functional teams.
   • Data is connected across the company and across domains. So you can not work exclusively in an embedded model. You need some central functionality.
   • Data Maturity differs between domains. So really the embedded model depends on the circumstances and how Oda applies that.
   • Data Mesh has been an inspiration.
2. Distributed data ownership, shared data governance
   • Processes and parts of the product for Oda are developed by the domain teams. To federate the ownership is a natural move.
   • Domain boundaries need to be explicit. «Every model that we have in dbt is tagged with a team.»
   • Ownership of certain products is harder or sometimes not right to distribute. It can either be core products with no natural domain team to own, and that many are dependent on, or the core data platform itself.
   • Customer data first, but without proper product data you can’t live up to that
3. Data as a product
   • Make sure you don’t just deliver a product, but that it meets the customer need, not just the customer demand.
   • Consistency matters, structure matters, naming matters when it comes to data products.
4. Enablement over handovers
   • Enabling others to do what they should be able to do themselves.
   • Oda has established a segmentation model for five levels of self-service with different expectations to different user groups.
   • Self-service needs to be tailored to different roles, maturity, and needs of the internal users.
   • Data University, Data Hours and many other initiatives help to create a learning culture and improve data literacy.
5. Impact through exploration and experimentation
   • It is important to test and see how a solution actually provides value to the expectations.
   • This provides insights and information you can act on.
6. Proactive attitude towards privacy and data ethics
   • Data ethics needs to be incorporated and can’t be an afterthought.
   • Company values can and should be directly linked to the work with ethics.

This episode was recorded in September 2022. Click here if you what to know more.

«If you do guidance correctly, people will follow it. People want to do the correct thing. Nobody wants to do things wrong.»

From fisherman on Island to Data MVP in Copenhagen! Ásgeir Gunnarsson has been through a fantastic journey and we had a great conversation around PowerBI Governance. Asgeir has his own blog about the topic.

Here are my key takeaways:

• Think of platforms at a translation: You are own a journey and platform is where you stand on, from where you push off, start your journey.
• A lot of organizations are only on maturity level 1 when it comes to Power BI, even though they think they range higher, or even if there are different levels of maturity in different parts of the organization.
• Buzzwords help shape opinions, and to have discussions. With good help, these opinions can be shaped into actions.
• If you have the possibility to do a green field approach to data mesh - consider it: It will not get easier than this.
• The parts of Data Governance, that can be solved with a focus on technology, and process are much more mature and easier to handle than the softer parts, that are concerned with people.
• It is easier to lock out intentional mistakes than unintentional.
• Governance is changing focus, from being compliance driven to providing a framework of how we can get more value from our data, it’s about making people productive.
• By making people more productive and efficient, you can pay off the cost of governance really fast.
• If you use data more proactively, you are moving the power from people to the machine. But there needs to be a balance, its not either or.

PoweBI

• In PowerBI implementations we should have talked Governance from the start.
• PowerBI enables people in your organization to use data on their own. There is always value ion that.
• To use self-service tools correctly, you need to either have people formally trained or/and have guidelines in place BEFORE they start doing things.
• Not everyone in your organization will become a PowerBI expert or is a data person. Don’t expect everyone to go there.
• Governance is about giving people a framework and a good chance to do things correctly from the beginning.
• Most low hanging fruit: Build a report inventory!
• Try to keep the usage of PowerBI to the M365 environment and meet people where they are.
• Know your requirements before you start using tools.

The 5 pillars cover all of what your governance strategy implementation should cover:

• The people pillar is about having the right roles in place and train people to perform in these roles.
• Because PowerBI is a self-service tool the administrator role is often allocated randomly.
• The processes and framework pillar is about what proper documents in place to make it work so users can use Power BI correctly and be compliant. This covers administrative documents as well as end user documents.
• The training and support pillar is about making sure everyone that uses Power BI has gotten the required training. This is also about deciding what kind of support mechanisms you need to have in place.
• The monitoring pillar is about setting up monitoring of Power BI. Usually, it involves extracting data from the Power BI activity log as well as the Power BI REST APIs for information about existing artifacts in your Power BI tenant. 
• The settings and external tools pillar is about making sure Power BI settings are correctly sat as well as how to use other approved tools to support Power BI (extensions to Power BI).

«If you think about how we can work effectively together, you need to look at how you can effectuate your delivery teams. »

I had the pleasure of chatting with Trond Sogn-Lunden from Veidekke, one of the biggest construction contractors in the Nordics. The construction industry is characterized by Merger and Acquisition. This is an interesting setting and reflects the importance to understand the organizations culture when looking at ways of working. The other important element for Veidekkes culture and organization is project-orientation. Some projects can last for years and are located away from the corporate office.

Here are my key takeaways:

The Project setting

• "A project is the smallest unit for profit and loss. If the project is a success, the business succeeds. »
• Even if some projects are more digital than others, even frontrunners need to use the same base systems and be consistent in how problems are tackled.
• There is a balance to strike between project autonomy and application landscape consolidation.
• In the same way there is an important distinction between what is best for each project vs. what is best for the company as a whole. For the insight factory, this is a constant push and pull.

The Insight Factory

• The Insight factory brings a factory mindset to data and ensures that data products are delivered.
• Capacity to deliver is more important than the technology stack.
• Developers (solution responsibles) translate business requirement to solutions, close to the business.
• A developer needs to be a combination of «superman and Jesus».
• A coordinating role, ala business analyst, can help to ensure a unified application portfolio.
• This role can help to access relevant raw data for the insight factory to be used for analysis.
• The Insight factory is providing data sets to the business and help to enable the business to do their data visualization on top of it.
• This helps to build data competency in the business, secures ownership, and added capacity.

Product Management

• Product Management is an important concept on how deliveries are organized.
• Product focus and pipeline structure enables Veidekke to have different change rates for different products and manage them as units and can deliver on demand.
• Every product can be connected to a domain, which is what insight is organized around.
• You need allies in the business to get the reach you need as a central insight factory.
• Business users should be part of your data product groups.
• You can set up inspirational events, to get traction and understanding for your insight & analytics work.

«Think of data availability as online vs. offline.»

What much of the discussions around data products, data catalogs, self-service boil down to is data discoverability, observability and availability.

I talked to Ivan Karlovic, Director of Data Analytics and Master Data at Norwegian about these topics and gained some fantastic insights. Ivan always loved analytics and using data to improve the business and started his dat journey with a course in data mining and with «Pure curiosity on how we can use data!»

Here are my key takeaways:

The Airline sector

• Airline industry is reliant on partner «A Flight is just a subset of an end-2-end journey»
• Data privacy and ethics are important topics for Norwegian and are faced with a systematic approach with an aim for automation.
• Norwegian is building a cloud based analytical platform to ensure a greater visibility of the data analytics setup.
• The first improvement should be on data discoverability, closely connected to data observability.
• «A Data Catalog will raise awareness of what we have of data assets.»
• There is a clear goal to ensure an automated observation of all data assets in Real time.

• A central team needs to be able to deliver cross-domain use cases, also across domains with different data maturity.
• «With this crisis-domain approach we are putting away some of the legacy discussions.» We can engage with each domain.
• It is ok to have specific crawlers on local data, but you need to synchronize it into the central data catalog.
• The organization needs to have a way to stay aware of everything that is produced.

Data Availability

• Except for sensitive data, everyone in a domain should be able to see all domain data. Outside the domain, people should be aware of what kind of data each domain maintains.
• «If you work with analytics or machine learning you always what to talk to the domain people, because you can easily misinterpret if you don’t have that domain experience.»
• Domain data products that are domain spesific without a use case outside the domain, do not have to adhere to central strandards. But if they can have a use cases outside the domain, they need to be fed into the central data catalog.
• Communication and understanding intentions from data producers to data users is really important. You have to continuously work with understanding.
• There is lost out business potential in not having data discoverable, no matter the quality.
• Most effort is wasted in rework of data products that where just not discoverable.

Self-Service

• «When it comes to self-service we need to set up technology in a way that the end-user does not have to think about the data, only the problem to solve»
• Even if only 70% of use cases can be solved by self service, we need to strive for 100% to ensure that we offload the expert data analytics team as much as possible to work on the tough cases.
• «Data Catalog: You can buy a monster that gives you 95% of things you don’t need, or cutting edge super-niche start ups. But you have some interesting players somewhere in the midle.»
• «Can we do data engineering on a meta level, without seeing the underlying data? Eg. For PII?»

Documentation

• How can you retain knowledge in a distributed architecture?

1. Ensure domain knowledge is fostered in the domains. Build a documentation repository
2. Infrastructure as code. Technical knowledge supplied with context

• Domain knowledge is the most tricky and most difficult to replace
• Great documentation can be both motivating and time saving. Motivating to reach a higher standard and time saving for problem finding, onboarding, etc.

"The data lifecycles collides with the system lifecycles. It’s a classic."

Let’s talk about the paradoxes of Data: Data Lifecyle, Search and Data Catalog!

 What a fantastic chat Ole Olesen-Bagneux and I had! Ole is writing his O’Reilly book Enterprise Data Catalog, has newsletter Symphony of Search. He brings in a new perspective from Library and Information Science and is a great advocate for transforming the way we think around data and search.

 Ole has worked as a specialist, as a leader and as an architect, and has an academical background as PhD in Information Science from University of Copenhagen.

 Here are my key takeaways:

• Data Lifecycle was first mentioned as the POSMAD lifecycle
  •  Plan - Plan for creation 
  • Obtain - Acquire data 
  • Store - store it in a system 
  • Share - expose it and make it accessible 
  • Maintain - curate data, keep it accurate 
  • Apply - Use the data 
  • Dispose - Archive or delete
• Store, share and apply is where the business value is derived
• The points where you get value from data are normally not the same, we use to manage data.
• The work e.g. national archives do, in cataloguing, and readying data for research is done at the very last stage of the lifecycle. But the value resides much earlier in the lifecycle.
• Data-driven innovation, data-drive culture… What these terms actually mean is that we need to get better at utilizing the value insight data.
• Intangible assets hold the highest value - data is the key to value creation.
• One of the potentials of a data catalog is to push the high-level DM activities to earlier stages of the lifecycle.
• Catalogs are pushing inventory activities from the dispose phase to the store and share phase of the lifecycle.
• There is a huge difference in the perspective of an IT system lifecycle and data lifecycle.
• Data always resides in a system, and that system has its own lifecycle. These lifecycles do not match.
• If you do not maintain data in your systems, any potential data migration becomes exponentially more difficult. What do we migrate, what do we keep, what do we delete?
• The solution can be a Data Catalog and/or metadata repository with retention policies for data.
• The distinction between searching in and searching for data has become really important due to the rise of data science.
• When you search for data, you are looking at data sources with potential value to search in.
• Metadata is key in searching for data - that means we have to manage the metadata lifecycle as well.
• A data Catalog is basically just a search engine.
• Data Catalogs rely more and more on the same technology components as search engines for the web, e.g. knowledge graphs.
• The key capability of data catalogs is a metadata overview over the data in your company.
• Data catalogs have an untouched potential to ensure data lifecycle management

"Data is mainly used to create value for customers, both inside the company and outside!"

 Customer centric is one of the great mantras in data at the time. I wanted to get to the bottom of what Customer Experience actually means. So, whom better to as then Leif Eric Fredheim, Customer Insights Manager at Elkjøp and one of the top 100 Data, Analytics and AI professionals in the Nordics?

 We talked about the retail data quest, what we can learn from retain in other sectors and naturally the value of customer experience and insight.

  Here are my key takeaways:

• Data has to be fast and easily accessible for the business, so they can use and consume data when they need it to make better decisions. This happens on a self-service basis.
• Make people aware how they can use, combine, and analyze data.
• Analysts and Scientists must be placed at the different business domains.
• Elkjøp is organizing towards a product-oriented organization.
• The data governance structure is organized towards freedom under responsibility. There are rules in place, but the creativity should not be hindered.
• Business critical data products are created and maintained centrally.
•  Data privacy is important, especially when it comes to customer data.
• Privacy is an important element of good Customer experience.
• Are data savvy front runners setting the requirements of how we work with data?
• There are data science & analysis teams embedded in data savvy departments.
• Analysis should happen embedded in the business and involve the central data team in necessary.
• Lasting customer relations and a strong focus on customer experience across all channels are learnings from retail towards other industries.
• Omnichannel players need to find ways to connect the physical customer experience with the digital.
• Retail can learn from other sectors like banking and finance, that there is a need for explainable models - how to be transparent with your customers.
• How do we manage the gap between explain ability and performance?
• CX is about listening to the customer and use tools and data to better their experience. 
• Digital stores make it easier to use data for better customer experience then in physical stores
• Quantitative analysis is about CX without directly involving the costumer.
• Qualitative is the opposite: the customer is actively involved to better his/her experience.
• Only 30-40% of CX is focus on the actual buy, the rest is moved before the point of decision and after a purchase.
• Correct and updated product information is vital before the purchase. You can support the decision process and idea phase through helpful articles and inspirational content at the right time at the right place.
• Natural Language Processing (NLP) for review data is a vital part to ensure good customer experience.
• Many have a way to go when it comes to self-service CX. 
• You need to instrumentalize all your customer touchpoints to gain a reliable variant data foundation to ensure CX success.