Azure Friday

Thomas Alex joins Lara Rubbelke to discuss how Microsoft uses Apache Kafka for HDInsight to power Siphon, a data ingestion service for internal use. Apache Kafka for HDInsight is an enterprise-grade, open-source, streaming ingestion service. Microsoft created Siphon as a highly available and reliable service to ingest massive amounts of data for processing in near real time. Siphon handles ingestion of over a trillion events per day across multiple business-critical scenarios at Microsoft. In this episode, learn how Siphon uses Apache Kafka for HDInsight as its scalable pub/sub message queue.Jump To: [01:50] Demo Start For more information:Quickstart: Create a Kafka on HDInsight clusterTutorial: Use Spark Structured Streaming with Kafka on HDInsightApache Kafka for HDInsight overviewAzure HDInsight pricingSiphon: Streaming data ingestion with Apache KafkaCreate a free account (Azure)Follow @sqlgal Follow @AzureFriday Follow @AzureHDInsight

Anubhav Mishra (Developer Advocate, HashiCorp), joins Scott Hanselman to discuss how to use HashiCorp Terraform to create & manage Kubernetes clusters in Azure using Azure Kubernetes Service (AKS). Mishra further explains the benefits of using Terraform to provision Azure infrastructure and demonstrates how to configure a Kubernetes cluster on AKS.Jump To: [01:52] Demo Start For more information:HashiCorp Terraform overviewGet started with Terraform EnterpriseAzure Kubernetes Service (AKS) with Terraform (GitHub)Azure Kubernetes Service (AKS) overviewDevOps Tools Integration - HashiCorp TerraformTerraform on Azure documentationCreate a free account (Azure)Follow @SHanselman Follow @AzureFriday Follow @anubhavm

Gaurav Malhotra joins Lara Rubbelke to discuss how you can operationalize Jars and Python scripts running on Azure Databricks as an activity step in a Data Factory pipeline.Jump To: [01:55] Demo Start For more information:Transform data by running a Jar activity in Azure Databricks docsTransform data by running a Python activity in Azure Databricks docsAzure Databricks overviewAzure Data Factory overviewCreate a free account (Azure)Follow @sqlgal Follow @AzureFriday

Murali Krishnaprasad joins Lara Rubbelke to discuss Interactive Query (also called Hive LLAP, or Low Latency Analytical Processing, or Live Long and Process), which is an Azure HDInsight cluster type. Interactive Query supports in-memory caching, which makes Hive queries super-fast and interactive. See how to use HDInsight Interactive Query to analyze extremely large datasets (~100TB) in common file formats such as ORC and CSV using common BI/SQL tools including Zeppelin notebooks and VS Code.Jump To: [04:12] Demo Start For more information, see:Use Interactive Query with HDInsight (docs)HDInsight overviewHDInsight pricing detailsCreate a free account (Azure)Follow @sqlgal Follow @AzureFriday Follow @AzureHDInsight

Kylie Liang joins Lara Rubbelke to demo how to use Ansible, an open-source, simple IT automation engine for automating cloud provisioning, configuration management and application deployments on Azure. Kylie also shares the roadmap for Ansible on Azure.Jump To:[00:15]  Demo Start For more information:Ansible in Azure Documentation (docs)Ansible Playbook Samples for Azure (GitHub)Ansible modules (GitHub)Microsoft Azure Guide on Ansible (docs)Ansible on Azure RoadmapAnsible Visual Studio Code Extension on Visual Studio MarketplaceAzure Preview Modules on AnsibleCreate a free account (Azure)Follow @sqlgal Follow @AzureFriday Follow @LiangKylie

Jean-Sébastien Brunner joins Lara Rubbelke to discuss new features that enable you to implement the intelligent cloud, intelligent edge vision for streaming analytics: Stream Analytics running real-time analytics with custom code on IoT Edge. We also discuss substreams, which is a new time management feature for independently processing the timeline of each device (very useful for an IoT scenario), and the recently announced Session Window.Jump To:[01:12] Demo Start For more information:Azure Stream Analytics (product info)Azure Stream Analytics now available on IoT Edge (Azure blog)4 new features now available in Azure Stream Analytics (Azure blog)Announcing real-time Geospatial Analytics in Azure Stream Analytics (Azure blog)Azure Stream Analytics DocumentationCreate a free account (Azure)Follow @sqlgal Follow @AzureFriday Follow @AzureStreaming

Steve Lasker joins Lara Rubbelke to discuss ACR Build, a cloud-native container build solution enabling pre-check, git commit and base image update docker builds for OS and framework patching.  Jump To:[03:18] Demo Start For more information:  Automate OS and framework patching with ACR Build (docs)Demo 42 sample (GitHub)Azure Container Registry (product info)Azure Container Registry (pricing)Azure Container Registry Roadmap (GitHub)Create a free account (Azure)Follow @sqlgal Follow @AzureFriday Follow @stevelasker

Gaurav Malhotra joins Scott Hanselman to discuss Event-driven architecture (EDA), which is a common data integration pattern that involves production, detection, consumption, and reaction to events. Learn how you can do event-based data integration using Azure Data Factory.Jump To:[01:53] Demo Start For more information:Event trigger-based data integration with Azure Data Factory (blog post)Create a trigger that runs a pipeline in response to an event (docs)Azure Data Factory product infoAzure Data Factory pricingCreate a free account (Azure)Follow @SHanselman Follow @AzureFriday

Justin Luk joins Scott Hanselman to show the new Azure Cloud Shell editor. Since its launch, Cloud Shell included a variety of editors (vi, emacs, and nano) for editing files. In collaboration with the Visual Studio Code team and their open-source Monaco project, the same web standards-based editor that powers Visual Studio Code is now integrated directly into Cloud Shell.Jump To:[02:05] Demo StartFor more informationCloud Shell documentationCloud Shell feature pageCloud Shell pricingMonaco project (GitHub repo)Announcing the Azure Cloud Shell editor in collaboration with Visual Studio Code (blog post)Create a free account (Azure)Follow @SHanselman Follow @AzureFriday Follow @whosjluk

Chacko Daniel joins Scott Hanselman to discuss Azure Service Fabric Mesh, which offers the same reliability, mission-critical performance and scale customers get with Service Fabric, but no more overhead of cluster management and patching operations. Service Fabric Mesh supports both Windows and Linux containers allowing you to develop with any programming language and framework of your choice.Jump Tp:[04:35] Demo StartFor more information:Azure Service Fabric Mesh documentationAzure Service Fabric Mesh samplesAzure Service Fabric Mesh pricingCreate a free account (Azure)Follow @SHanselman Follow @AzureFriday Follow @chackod