The Clearly Podcast

Send us Fan Mail

Andy Clark, Tom Gough and Shailan Chudasama return to a topic they first covered a few years ago, and the answer has changed a lot. Tom explains the basics: a data lake is large-scale, file-based storage, and in Fabric everything sits in OneLake anyway. The real difference between a Lakehouse and a Warehouse is how you work with the data. A Warehouse behaves like a traditional SQL Server and is driven by T-SQL. A Lakehouse is built around Python and PySpark and handles unstructured data much better.

Shailan describes how client conversations usually go. They start with goals, data sources, data types, refresh frequency and sensitivity. More clients now want to bring in media, telemetry and other semi-structured data alongside their ERP and finance systems, which pushes them toward a Lakehouse. The team agrees the gap has narrowed sharply. Lakehouse performance has caught up, Spark SQL is close enough to T-SQL to make the switch easy, and the Warehouse can now be read from Python. The clearest remaining reason to choose a Warehouse is a lift-and-shift migration of an existing on-prem SQL Server warehouse, or a hard requirement to write data with T-SQL.

The hosts also cover:

• Future-proofing and recruitment, given Python’s rise as the default entry language.
• Hybrid medallion designs, such as a Silver Lakehouse with a Gold Warehouse, as a way to transition gradually.
• Their prediction that the two will merge into a single analytics platform within a couple of years.
• Copilot now being available across Fabric SKUs from F2 upward, with a reminder that it’s billed separately from capacity.

Key takeaway: Don’t overthink it. Because everything lives in OneLake, neither choice is a bad one, and switching later isn’t hard. Pick based on your team’s existing skills, whether that’s T-SQL or Python. Or, as Tom puts it, Lakehouse vs Warehouse is the rosé wine of Fabric.

Send us Fan Mail

Andy Clark, Tom Gough and Shailan Chudasama (both nursing colds) take on one of the most hyped topics in the industry: how the data and BI profession is changing, and how much of the AI noise is real.

Shailan starts by defining what they do today. BI delivery is still the core, but the role now spans architecture, stakeholder management and project delivery, with senior executives closely involved in the reporting. Clients increasingly want AI built in from the start, along with help training their own staff to use it well. Tom says the role is more complex at the senior level and more specialized at the junior level. Work that one person once did end to end is now split across business analysis, data engineering and modeling. His advice to newcomers is to specialize first, and data engineering has the biggest skills shortage, though also the steepest learning curve.

On AI, the hosts push back on inflated productivity claims. Tom warns that if people stop writing code, they lose the ability to review what AI produces, and unlike a human developer or vendor, there’s no redress when an AI gets it wrong. Andy argues that the skills now in demand are the ones the industry has always neglected: documentation and testing. Detailed documentation of the business problem gives AI something reliable to work from. Tom adds that clients are good at describing what they want but not what they need, and AI tends to agree rather than challenge, which keeps human judgment essential.

They also discuss a looming pipeline problem. If AI absorbs junior work, where will future seniors come from? And relying entirely on AI providers leaves firms exposed when heavily subsidized token prices go up.

Final thoughts:

• Tom: AI will be a big part of the future, but you still need people with senior skills who can describe the problem and direct the AI.
• Shailan: we’re now business consultants more than ever.
• Andy: AI projects are harder than people expect, so plan accordingly.

Send us Fan Mail

After 834 days away, Andy Clark, Tom Gough and Shailan Chudasama are back. They open with a quick stats quiz:

• It’s been 2,181 days since the podcast first launched.
• The show has had 3,421 downloads during the break alone.
• Their book, Financial Modelling and Reporting with Microsoft Power BI, came out 151 days ago and is holding a 4.8/5 rating on Amazon.

They then look at what’s changed. Fabric has gone from a hard sell (“why not just buy the Azure pieces?”) to a mature platform where the financial case is easy to make, even for smaller teams. Tom’s work has shifted toward Python notebooks for metadata-driven loading and handling schema drift. Shailan notes that clients now arrive asking for a medallion Lakehouse by name, rather than needing to be talked out of a classic data warehouse. They put this down to Microsoft’s push and to people discovering Lakehouses when they spin up Fabric trials.

On Copilot, perceptions are improving, and its availability across all F SKUs makes it an easy box to tick in procurement. The move to consumption-based billing for AI raises real cost concerns, though. Tom suspects that once AI costs show up as a clear line on the P&L, some of the expected savings will evaporate. Andy questions how much prompt-based querying of semantic models is actually happening, and Tom argues a well-designed model shouldn’t need it. All three warn that AI-generated reporting could worsen report sprawl and conflicting numbers. As Tom puts it, a man with two watches is never quite sure what time it is.

The episode closes with reflections on writing the book: a year of enjoyable writing followed by an exhausting editing and proofing process, the difficulty of writing for an unseen global audience, and a new appreciation for why tech books cost what they do.

Send us Fan Mail

Welcome to the Clearly Podcast. The podcast is taking a break for a few weeks due to another project, and upon return, there will be exciting news. Today's discussion revisits what it takes to be a Power BI consultant, focusing on the necessary skills and qualities.

Experience with Power BI is valuable, but not mandatory for juniors. Key qualities include consulting skills, analytics, business experience, understanding data, and articulating it clearly. Being able to work with senior people to address business processes is also essential.

For those fresh out of college or university, it's important to have a grasp of business fundamentals and good Q&A skills. Graduates often have skills in algorithms and data science. Conversing with business users to understand their challenges and developing solutions is crucial.

The role of a Power BI consultant now often includes elements of data engineering, data warehousing, or machine learning. It’s important to have a broader knowledge of the data landscape. Employers should seek a mix of skills rather than expertise in every area.

For someone starting, understanding data engineering and data modeling is key. Learning to build models that others can use and gaining experience in visualizing data is important. Download Power BI, use public datasets, and practice.

Knowing how to find information about connectors for other applications is crucial. Being able to get back to clients with accurate information is essential.

During interviews, honesty about skills is important. People often claim more experience than they have. Private projects show initiative and learning and should not be dismissed.

There is high demand for roles, especially for those who understand the entire Microsoft stack. Consultants should keep their knowledge up to date through communities, podcasts, and continual learning.

For aspiring consultants, working on side projects and learning new skills is beneficial. Employers appreciate self-starters who take initiative.

The podcast will return with exciting news after the break. Thank you for listening and stay tuned for more insights.

You can download Power BI Desktop from here.

If you already use Power BI, or are considering it, we strongly recommend you join your local Power BI user group here.

To find out more about our services and the help we can offer, contact us at one of the websites below:
UK and Europe: https://www.clearlycloudy.co.uk/
North America: https://clearlysolutions.net/

Send us Fan Mail

The podcast discusses the current state of the IT consulting job market, particularly focusing on skills and opportunities in 2024. The conversation highlights the end of the post-COVID job market boom, noting that recruitment has slowed, with hiring managers delaying decisions and tighter budgets. However, there are signs of recovery with an increase in RFPs.

The job market trend is similar in both the UK and the US, though the US is slightly better off, with contractors getting renewed more frequently. During the post-COVID boom, even mediocre candidates demanded high salaries, which is no longer sustainable. Now, candidates need to upskill and be proactive.

Skills in high demand include pre-sales capabilities, combining product knowledge with the ability to confidently present to clients. Soft skills are emphasized as being equally important as technical skills. Companies are increasingly requiring employees to be in the office, at least part-time, to enhance learning and integration, especially for new employees. The hybrid work model is becoming more common, balancing office and remote work.

Job seekers are advised to be proactive by updating their CVs, tailoring them for each job, and engaging directly with employers and recruiters. Side projects and continuous learning can help candidates stand out.

There is growing demand for AI skills, but AI will not replace human roles entirely. It enhances human capabilities, and the best candidates will continue to improve and adapt. The importance of validating and understanding AI outputs, especially in critical fields, is highlighted.

The podcast concludes with a suggestion to make this job market update a regular segment to keep the audience informed.

Send us Fan Mail

Welcome to the Clearly Podcast. After some light-hearted banter about annual leave and running shoes, the discussion shifts to the main topic: Azure vs. Fabric.

The conversation covers the components of Azure and Fabric, focusing on Power BI, Azure Data Factory (ADF), OneLake, and Synapse Analytics. Fabric offers better integration and cost control for users who need multiple components. However, for those using a single component like ADF, the benefits of migrating to Fabric are less clear.

From a customer perspective, streamlining licencing and management is a significant advantage of Fabric. Azure requires separate licences for each component, while Fabric consolidates this into a single licence. This can simplify administration and reduce costs, especially for future expansions.

For on-premise customers considering a move to the cloud, the best approach might be a gradual shift. First, migrate to Azure components, then transition to Fabric. This incremental strategy minimizes service interruptions and spreads out the rearchitecture workload.

There are also scenarios where sticking with Azure might be more practical. If an organization primarily uses components like ADF for non-analytical purposes, the complexity of managing Fabric could outweigh the benefits. The consensus is that while Fabric offers significant advantages for data analytics, Azure components will continue to cater to other use cases, and there is no immediate push for all customers to move to Fabric.

In summary, customers should evaluate their specific needs, considering both immediate and future use cases, to decide between Azure and Fabric. The transition should be planned carefully, leveraging Azure as a stepping stone when moving from on-premise environments.

Send us Fan Mail

Welcome to the Clearly Podcast, where today's discussion focuses on Power Apps, particularly canvas apps, model-driven apps, and Power Pages, and how to choose the right one.

Power Pages are ideal for external, infrequent users due to their pricing model and ability to handle large user bases. In contrast, canvas and model-driven apps are more suited for internal use. Canvas apps offer high customization and a user-friendly mobile interface, making them great for tasks like inventory logging. However, they require careful management to avoid becoming overly complex. Model-driven apps, which are form-based and linked to Microsoft CRM, are faster to set up and better for structured, transactional tasks but are limited to using Dataverse.

Client discussions typically start by identifying the project’s goals, budget, and timeline, prioritizing essential features, and planning additional phases. Managing expectations is crucial to prevent overcomplicating the project.

When choosing data sources for canvas apps, Dataverse is preferred, but budget constraints might lead to using SharePoint lists, which must be managed carefully to avoid data issues. Deciding on data storage upfront is essential to avoid complications later.

Power Pages are low-code environments suitable for creating external portals without needing individual licenses, making them ideal for customer portals. They support both authenticated and unauthenticated users and are priced based on usage.

Clients often start using Power Apps by transitioning from cumbersome Excel spreadsheets to improve data management and security. Demonstrating practical examples helps clients understand the benefits.

Long-term care and maintenance of Power Apps involve planning for support, ensuring sufficient storage, and considering future improvements. A support plan can be more cost-effective than hiring full-time staff and ensures ongoing maintenance.

Power Apps are an investment requiring ongoing care and development. Thank you for joining the discussion, and we look forward to the next episode.

Send us Fan Mail

In this episode of the Clearly Podcast, the hosts discuss data quality dashboards. These dashboards monitor data entry to flag errors and ensure data accuracy, especially in systems with significant human input. Key points include:

• Definition and Use: Data quality dashboards highlight data entry errors, allowing for prompt correction. Errors disappear from the report once fixed, providing satisfaction to users.
• Routine Recommendation: These dashboards are essential, particularly where user input is high.
• Risks of Poor Data Quality: Inaccurate data can lead to flawed reports and decisions. Regular monitoring helps maintain data integrity.
• Application Redesign: Dashboards can reveal systematic issues, sometimes leading to application redesigns. Simplifying the interface or automating processes can prevent errors.
• Building the Dashboard: Start by identifying key issues through stakeholder discussions. Focus on major errors first, like dates and large transactions. Decide whether to automate error resolution or require manual approval.
• Senior Management Buy-In: Effective communication ensures the dashboard is seen as a tool for improvement, not punishment. This promotes user engagement and data quality.
• Continuous Improvement: If errors persist, it may indicate a deeper application issue requiring broader solutions.

The hosts emphasize the importance of data quality dashboards in maintaining accurate data, which is crucial for reliable reporting and decision-making.

Send us Fan Mail

Hosts: Andy Clark, Tom Gough

Guest: Greg Bounds 

Topic: Dealing With a Mess of Data Systems and Products

Key Points:

1. Introduction:

·      Andy and Tom introduce Greg Bounds, who works as a data architect for a national nonprofit and also consults on data projects.

2. Defining the Problem:

·      The discussion focuses on inheriting a complex data environment, termed a "dog's breakfast," with a mix of applications, systems, and databases.

·      Tom emphasizes the challenges of dealing with multiple cloud providers and on-premises systems.

3. Scoping the problem:

·      Start by talking to various stakeholders to understand the importance of different systems.

·      Auditing and documenting the systems should follow naturally from these conversations.

4. Managing Stakeholder Priorities:

·      Understand the challenge of balancing different departments' priorities and stresses the importance of understanding the entire data flow.

·      Show the benefits of changes through small, manageable projects.

5. Technical Approach:

·      Look for low-hanging fruit and opportunities to consolidate systems for quick wins.

·      Simplifying the data architecture can de-risk the environment and reduce points of failure.

6. Benefits of Consolidation:

·      Understand there may not be immediate cost savings, reducing the number of platforms simplifies governance and management.

·      It also minimizes the need for specialized knowledge across multiple systems.

7. Planning for the Long Term:

·      The importance of data governance and having a data team that is approachable and open to new solutions.

·      Ensuring the data team has the right skills and capacity is crucial to maintaining a streamlined environment.

8. Key Takeaways:

·      Communication with stakeholders and understanding their needs is essential.

·      Focus on data quality and transparency, even if it means creating "snitch reports" to highlight issues.

·      Look for quick wins to build momentum and show the value of changes.

·      Engage the C-suite to drive adoption and support across the organization.

 You can download Power BI Desktop from here.

If you already use Power BI, or are considering it, we strongly recommend you join your local Power BI user group here.

To find out more about our services and the help we can offer, contact us at one of the websites below:
UK and Europe: https://www.clearlycloudy.co.uk/
North America: https://clearlysolutions.net/

Send us Fan Mail

This week, Andy, Tom, and Shailan, discuss the challenges and considerations of working with technologies outside the Microsoft stack from a consulting perspective.

Key Points:

Encountering Non-Microsoft Technologies:

·      Clients often use various non-Microsoft technologies like Google Analytics, Salesforce, Oracle, etc.

·      Consultants must be prepared to handle data sources and systems outside the Microsoft ecosystem.

Balancing Expertise and External Help:

·      Evaluate whether to learn new technologies internally or bring in external experts.

·      Consider the long-term strategic value of learning a new technology versus the immediate need for expertise.

Cost and Time Considerations: 

·      Training and rework can be costly and time-consuming when working with technologies outside your skill-set.

·      It's often more efficient to hire experienced consultants for niche technologies to avoid mistakes and reduce implementation time.

Deciding When to Say No:

·      Assess if a project aligns with your core expertise and strategic goals.

·      Avoid taking on projects that significantly deviate from your primary skill set to maintain focus and reputation.

Strategic Partnerships:

·      Partnering with vendors can provide necessary training and support.

·      Ensure the vendor has good implementation skills, not just product knowledge.

Pricing and Client Communication:

·      Factor in the cost of external resources when pricing for clients.

·      Have honest conversations with clients about the costs and impacts of using niche technologies.

Opportunity Costs:

·      Taking on projects outside your expertise can lead to missed opportunities in your core area.

·      Focus on projects that align with your business direction and offer better margins.

Project Selection:

·      Deliberately choose projects that help expand your expertise in desired areas, like broader Power Platform work.

·      Avoid projects that don't align with your strategic aims, even if they seem lucrative.

Conclusion:

Navigating non-Microsoft technologies requires a balance between internal learning and external expertise, clear communication with clients, and strategic project selection to maintain focus and deliver quality results.

You can download Power BI Desktop from here.

If you already use Power BI, or are considering it, we strongly recommend you join your local Power BI user group here.

To find out more about our services and the help we can offer, contact us at one of the websites below:
UK and Europe: https://www.clearlycloudy.co.uk/
North America: https://clearlysolutions.net/