The Python Podcast.init

By Tobias MaceyTechnology Education

The Python Podcast.init episodes

September 19, 2022Catching Up With Pyre, A Fast Type Checker For Python
Summary
Static typing versus dynamic typing is one of the oldest debates in software development. In recent years a number of dynamic languages have worked toward a middle ground by adding support for type hints. Python’s type annotations have given rise to an ecosystem of tools that use that type information to validate the correctness of programs and help identify potential bugs. At Instagram they created the Pyre project with a focus on speed to allow for scaling to huge Python projects. In this episode Shannon Zhu discusses how it is implemented, how to use it in your development process, and how it compares to other type checkers in the Python ecosystem.
Announcements
Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. And now you can launch a managed MySQL, Postgres, or Mongo database cluster in minutes to keep your critical data safe with automated backups and failover. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Your host as usual is Tobias Macey and today I’m interviewing Shannon Zhu about Pyre, a type checker for Python 3 built from the ground up to support gradual typing and deliver responsive incremental checks
Interview
Introductions
How did you get introduced to Python?
Can you describe what Pyre is and the story behind it?
There have been a number of tools created to support various aspects of typing for Python. How would you describe the various goals that they support and how Pyre fits in that ecosystem?
What are the core goals and notable features of Pyre?
Can you describe how Pyre is implemented?
How have the design and goals of the project changed/evolved since you started working on it?
What are the different ways that Pyre is used in the development workflow for a team or individual?
What are some of the challenges/roadblocks that people run into when adopting type definitions in their Python projects?
How has the evolution of type annotations and overall support for them affected your work on Pyre?
As someone who is working closely with type systems, what are the strongest aspects of Python’s implementation and opportunities for improvement?
What are the most interesting, innovative, or unexpected ways that you have seen Pyre used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Pyre?
When is Pyre the wrong choice?
What do you have planned for the future of Pyre?
Keep In Touch
shannonzhu on GitHub
Picks
Tobias
Lord Of The Rings: The Rings of Power on Amazon Video
Shannon
King’s Dilemma board game
Closing Announcements
Thank you for listening! Don’t forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. The Machine Learning Podcast helps you go from idea to production with machine learning.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
PYre
MyPy
Podcast Episode
PyRight
PyType
MonkeyType
Podcast Episode
Java
C
PEP 484
Flow
Hack
Continuous Integration
OCaml
PEP 675 – Arbitrary literal strings
Gradual Typing
AST == Abstract Syntax Tree
Language Server Protocol
Tensor
Type Arithmetic
PyCon: Securing Code With The Python Type System
PyCon: Type Checked Python In The Real World
PyCon: Łukasz Lange 2022 Keynote
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
52 min
September 13, 2022Standardizing On Python For All Software Projects At Ascend.io
Summary
Every software project is subject to a series of decisions and tradeoffs. One of the first decisions to make is which programming language to use. For companies where their product is software, this is a decision that can have significant impact on their overall success. In this episode Sean Knapp discusses the languages that his team at Ascend use for building a service that powers complex and business critical data workflows. He also explains his motivation to standardize on Python for all layers of their system to improve developer productivity.
Announcements
Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. And now you can launch a managed MySQL, Postgres, or Mongo database cluster in minutes to keep your critical data safe with automated backups and failover. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Your host as usual is Tobias Macey and today I’m interviewing Sean Knapp about his motivations and experiences standardizing on Python for development at Ascend
Interview
Introductions
How did you get introduced to Python?
Can you describe what Ascend is and the story behind it?
How many engineers work at Ascend?
What are their different areas of focus?
What are your policies for selecting which technologies (e.g. languages, frameworks, dev tooling, deployment, etc.) are supported at Ascend?
What does it mean for a technology to be supported?
You recently started standardizing on Python as the default language for development. How has Python been used up to now?
What other languages are in common use at Ascend?
What are some of the challenges/difficulties that motivated you to establish this policy?
What are some of the tradeoffs that you have seen in the adoption of Python in place of your other adopted languages?
How are you managing ongoing maintenance of projects/products that are not written in Python?
What are some of the potential pitfalls/risks that you are guarding against in your investment in Python?
What are the most interesting, innovative, or unexpected ways that you have seen Python used where it was previously a different technology?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on aligning all of your development on a single language?
When is Python the wrong choice?
What do you have planned for the future of engineering practices at Ascend?
Keep In Touch
LinkedIn
@seanknapp on Twitter
Picks
Tobias
Delver Lens app for scanning Magic: The Gathering cards
Sean
Typer
DuckDB
Amp It Up book (affiliate link)
Closing Announcements
Thank you for listening! Don’t forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. The Machine Learning Podcast helps you go from idea to production with machine learning.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
Ascend
Data Engineering Podcast Episode
Perl
Google Sawzall
Technical Debt
Ruby
gRPC
Go Language
Java
PySpark
Apache Arrow
Thrift
SQL
Scala
Snowflake runtime for Python Snowpark
Typer CLI framework
Pydantic
Podcast Episode
Pulumi
Podcast Episode
PyInfra
Podcast Episode
Packer
Plot.ly Dash
DuckDB
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
51 min
September 05, 2022Exploring The Process And Practice Of Building Better Software Through Code Reviews
Summary
Writing code is only one piece of creating good software. Code reviews are an important step in the process of building applications that are maintainable and sustainable. In this episode On Freund shares his thoughts on the myriad purposes that code reviews serve, as well as exploring some of the patterns and anti-patterns that grow up around a seemingly simple process.
Announcements
Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. And now you can launch a managed MySQL, Postgres, or Mongo database cluster in minutes to keep your critical data safe with automated backups and failover. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Your host as usual is Tobias Macey and today I’m interviewing On Freund about the intricacies and importance of code reviews
Interview
Introductions
How did you get introduced to Python?
Can you start by giving us your description of what a code review is?
What is the purpose of the code review?
At face value a code review appears to be a simple task. What are some of the subtleties that become evident with time and experience?
What are some of the ways that code reviews can go wrong?
What are some common anti-patterns that get applied to code reviews?
What are the elements of code review that are useful to automate?
What are some of the risks/bad habits that can result from overdoing automated checks/fixes or over-reliance on those tools in code reviews?
identifying who can/should do a review for a piece of code
how to use code reviews as a teaching tool for new/junior engineers
how to use code reviews for avoiding siloed experience/promoting cross-training
PR templates for capturing relevant context
What are the most interesting, innovative, or unexpected ways that you have seen code reviews used?
What are the most interesting, unexpected, or challenging lessons that you have learned while leading and supporting engineering teams?
What are some resources that you recommend for anyone who wants to learn more about code review strategies and how to use them to scale their teams?
Keep In Touch
LinkedIn
@onfreund on Twitter
Picks
Tobias
The Girl Who Drank The Moon
On
Better Call Saul
Closing Announcements
Thank you for listening! Don’t forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. The Machine Learning Podcast helps you go from idea to production with machine learning.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
Wilco
Code Review
Home Assistant
Podcast Episode
Trunk-based Development
Git Flow
Pair Programming
Feature Flags
Podcast Episode
KPI == Key Performance Indicator
MIT Open Learning Engineering Handbook
PEP Repository
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
58 min
August 28, 2022Ship With Confidence By Automating Quality Assurance
Summary
Quality assurance in the software industry has become a shared responsibility in most organizations. Given the rapid pace of development and delivery it can be challenging to ensure that your application is still working the way it’s supposed to with each release. In this episode Jonathon Wright discusses the role of quality assurance in modern software teams and how automation can help.
Announcements
Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. And now you can launch a managed MySQL, Postgres, or Mongo database cluster in minutes to keep your critical data safe with automated backups and failover. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Your host as usual is Tobias Macey and today I’m interviewing Jonathon Wright about the role of automation in your testing and QA strategies
Interview
Introductions
How did you get introduced to Python?
Can you share your relationship with software testing/QA and automation?
What are the main categories of how companies and software teams address testing and validation of their applications?
What are some of the notable tradeoffs/challenges among those approaches?
With the increased adoption of agile practices and the "shift left" mentality of DevOps, who is responsible for software quality?
What are some of the cases where a discrete QA role or team becomes necessary? (or is it always necessary?)
With testing and validation being a shared responsibility, competing with other priorities, what role does automation play?
What are some of the ways that automation manifests in software quality and testing?
How is automation distinct from software tests and CI/CD?
For teams who are investing in automation for their applications, what are the questions they should be asking to identify what solutions to adopt? (what are the decision points in the build vs. buy equation?)
At what stage(s) of the software lifecycle does automation live?
What is the process for identifying which capabilities and interactions to target during the initial application of automation for QA and validation?
One of the perennial challenges with any software testing, particularly for anything in the UI, is that it is a constantly moving target. What are some of the patterns and techniques, both from a developer and tooling perspective, that increase the robustness of automated validation?
What are the most interesting, innovative, or unexpected ways that you have seen automation used for QA?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on QA and automation?
When is automation the wrong choice?
What are some of the resources that you recommend for anyone who wants to learn more about this topic?
Keep In Touch
LinkedIn
@Jonathon_Wright on Twitter
Website
Picks
Tobias
The Sandman Netflix series and Graphic Novels by Neil Gaimain
Jonathon
House of the Dragon HBO series
Mystic Quest TV series
It’s Always Sunny in Philadelphia
Links
Haskell
Idris
Esperanto
Klingon
Planguage
Lisp Language
TDD == Test Driven Development
BDD == Behavior Driven Development
Gherkin Format
Integration Testing
Chaos Engineering
Gremlin
Chaos Toolkit
Podcast Episode
Requirements Engineering
Keysight
QA Lead Podcast
Cognitive Learning TED Talk
OpenTelemetry
Podcast Episode
Quality Engineering
Selenium
Swagger
XPath
Regular Expression
Test Guild
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
1 hr 10 min
August 14, 2022Remove Roadblocks And Let Your Developers Ship Faster With Self-Serve Infrastructure
Summary
The goal of every software team is to get their code into production without breaking anything. This requires establishing a repeatable process that doesn’t introduce unnecessary roadblocks and friction. In this episode Ronak Rahman discusses the challenges that development teams encounter when trying to build and maintain velocity in their work, the role that access to infrastructure plays in that process, and how to build automation and guardrails for everyone to take part in the delivery process.
Announcements
Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. And now you can launch a managed MySQL, Postgres, or Mongo database cluster in minutes to keep your critical data safe with automated backups and failover. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Your host as usual is Tobias Macey and today I’m interviewing Ronak Rahman about how automating the path to production helps to build and maintain development velocity
Interview
Introductions
How did you get introduced to Python?
Can you describe what Quali is and the story behind it?
What are the problems that you are trying to solve for software teams?
How does Quali help to address those challenges?
What are the bad habits that engineers fall into when they experience friction with getting their code into test and production environments?
How do those habits contribute to negative feedback loops?
What are signs that developers and managers need to watch for that signal the need for investment in developer experience improvements on the path to production?
Can you describe what you have built at Quali and how it is implemented?
How have the design and goals shifted/evolved from when you first started working on it?
What are the positive and negative impacts that you have seen from the evolving set of options for application deployments? (e.g. K8s, containers, VMs, PaaS, FaaS, etc.)
Can you describe how Quali fits into the workflow of software teams?
Once a team has established patterns for deploying their software, what are some of the disruptions to their flow that they should guard against?
What are the most interesting, innovative, or unexpected ways that you have seen Quali used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Quali?
When is Quali the wrong choice?
What do you have planned for the future of Quali?
Keep In Touch
@OfRonak on Twitter
Picks
Tobias
The Terminal List on Amazon
Ronak
Midnight Gospel on Amazon
Closing Announcements
Thank you for listening! Don’t forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. The Machine Learning Podcast helps you go from idea to production with machine learning.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
Quali
Torque
Visual Studio Plugin
Subversion
IaC == Infrastructure as Code
DevOps
Terraform
Pulumi
Podcast Episode
Cloudformation
Flask
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
1 hr 2 min
July 31, 2022The Benefits Of Python And Django For Going From Zero To MVP At Speed
Summary
Every startup begins with an idea, but that won’t get you very far without testing the feasibility of that idea. A common practice is to build a Minimum Viable Product (MVP) that addresses the problem that you are trying to solve and working with early customers as they engage with that MVP. In this episode Tony Pavlovych shares his thoughts on Python’s strengths when building and launching that MVP and some of the potential pitfalls that businesses can run into on that path.
Announcements
Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. And now you can launch a managed MySQL, Postgres, or Mongo database cluster in minutes to keep your critical data safe with automated backups and failover. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Your host as usual is Tobias Macey and today I’m interviewing Tony Pavlovych about Python’s strengths for startups and the steps to building an MVP (minimum viable product)
Interview
Introductions
How did you get introduced to Python?
Can you describe what PLANEKS is and the story behind it?
One of the services that you offer is building an MVP. What are the goals and outcomes associated with an MVP?
What is the process for identifying the product focus and feature scope?
What are some of the common misconceptions about building and launching MVPs that you have dealt with in your work with customers?
What are the common pitfalls that companies encounter when building and validating an MVP?
Can you describe the set of tools and frameworks (e.g. Django, Poetry, cookiecutter, etc.) that you have invested in to reduce the overhead of starting and maintaining velocity on multiple projects?
What are the configurations that are most critical to keep constant across projects to maintain familiarity and sanity for your developers? (e.g. linting rules, build toolchains, etc.)
What are the architectural patterns that you have found most useful to make MVPs flexible for adaptation and extension?
Once the MVP is built and launched, what are the next steps to validate the product and determine priorities?
What benefits do you get from choosing Python as your language for building an MVP/launching a startup?
What are the challenges/risks involved in that choice?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on MVPs for your clients at PLANEKS?
When is an MVP the wrong choice?
What are the developments in the Python and broader software ecosystem that you are most interested in for the work you are doing for your team and clients?
Keep In Touch
LinkedIn
Picks
Tobias
datamodel-code-generator
Tony
Screw It, Let’s Do It by Richard Branson (affiliate link)
Closing Announcements
Thank you for listening! Don’t forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. The Machine Learning Podcast helps you go from idea to production with machine learning.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
PLANEKS
Minimum Viable Product
Django
Cookiecutter
Django Boilerplate
OCR == Optical Character Recognition
Tesseract OCR framework
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
48 min
July 25, 2022Powering The Next Generation Of Application Architectures With Web Assembly And The Fermyon Platform
Summary
Application architectures have been in a constant state of evolution as new infrastructure capabilities are introduced. Virtualization, cloud, containers, mobile, and now web assembly have each introduced new options for how to build and deploy software. Recognizing the transformative potential of web assembly, Matt Butcher and his team at Fermyon are investing in tooling and services to improve the developer experience. In this episode he explains the opportunity that web assembly offers to all language communities, what they are building to power lightweight server-side microservices, and how Python developers can get started building and contributing to this nascent ecosystem.
Announcements
Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. And now you can launch a managed MySQL, Postgres, or Mongo database cluster in minutes to keep your critical data safe with automated backups and failover. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Need to automate your Python code in the cloud? Want to avoid the hassle of setting up and maintaining infrastructure? Shipyard is the premier orchestration platform built to help you quickly launch, monitor, and share python workflows in a matter of minutes with 0 changes to your code. Shipyard provides powerful features like webhooks, error-handling, monitoring, automatic containerization, syncing with Github, and more. Plus, it comes with over 70 open-source, low-code templates to help you quickly build solutions with the tools you already use. Go to dataengineeringpodcast.com/shipyard to get started automating with a free developer plan today!
Your host as usual is Tobias Macey and today I’m interviewing Matt Butcher about Fermyon and the impact of WebAssembly on software architecture and deployment across language boundaries
Interview
Introductions
How did you get introduced to Python?
For anyone who isn’t familiar with WebAssembly can you give your elevator pitch for why it matters?
What is the current state of language support for Python in the WASM ecosystem?
Can you describe what Fermyon is and the story behind it?
What are your goals with Fermyon and what are the products that you are building to support those goals?
There has been a steady progression of technologies aimed at better ways to build, deploy, and manage software (e.g. virtualization, cloud, containers, etc.). What are the problems with the previous options and how does WASM address them?
What are some examples of the types of applications/services that work well in a WASM environment?
Can you describe how you have architected the Fermyon platform?
How did you approach the design of the interfaces and tooling to support developer ergonomics?
How have the design and goals of the platform changed or evolved since you started working on it?
Can you describe what a typical workflow is for an application team that is using Spin/Fermyon to build and deploy a service?
What are some of the architectural patterns that WASM/Fermyon encourage?
What are some of the limitations that WASM imposes on services using it as a runtime? (e.g. system access, threading/multiprocessing, library support, C extensions, etc.)
What are the new and emerging topics and capabilities in the WASM ecosystem that you are keeping track of?
With Spin as the core building block of your platform, how are you approaching governance and sustainability of the open source project?
What are your guiding principles for when a capability belongs in the OSS vs. commercial offerings?
What are the most interesting, innovative, or unexpected ways that you have seen Fermyon used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Fermyon?
When is Fermyon the wrong choice?
What do you have planned for the future of Fermyon?
Keep In Touch
LinkedIn
@technosophos on Twitter
technosophos on GitHub
Picks
Tobias
Thor: Love & Thunder movie
Matt
Remembrance of Earth’s Past trilogy ("Three Body Problem" is the first) by Cixin Liu
Closing Announcements
Thank you for listening! Don’t forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. The Machine Learning Podcast helps you go from idea to production with machine learning.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
Fermyon
Our Python entry for the Wasm Language Matrix
SingleStore’s WASI-Python
Great notes about Wasm support in CPyton
Pyodide for Python in the Browser
SlashDot
Web Assembly (WASM)
Rust
AssemblyScript
Grain WASM language
SingleStore
Data Engineering Podcast Episode
WASI
PyO3
PyOxidizer
RustPython
Drupal
OpenStack
Deis
Helm
RedPanda
Data Engineering Podcast Episode
Envoy Proxy
Fastly
Functions as a Service
CloudEvents
Finicky Whiskers
Fermyon Spin
Nomad
Tree Shaking
Zappa
Chalice
OpenFaaS
CNCF
Bytecode Alliance
Finicky Whiskers Minecraft
Kotlin
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
1 hr 11 min
July 17, 2022Gain A Deeper Understanding Of What Your Code Is Doing And Where It Spends Its Time With VizTracer
Summary
As your code scales beyond a trivial level of complexity and sophistication it becomes difficult or impossible to know everything that it is doing. The flow of logic and data through your software and which parts are taking the most time are impossible to understand without help from your tools. VizTracer is the tool that you will turn to when you need to know all of the execution paths that are being exercised and which of those paths are the most expensive. In this episode Tian Gao explains why he created VizTracer and how you can use it to gain a deeper familiarity with the code that you are responsible for maintaining.
Announcements
Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. And now you can launch a managed MySQL, Postgres, or Mongo database cluster in minutes to keep your critical data safe with automated backups and failover. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Need to automate your Python code in the cloud? Want to avoid the hassle of setting up and maintaining infrastructure? Shipyard is the premier orchestration platform built to help you quickly launch, monitor, and share python workflows in a matter of minutes with 0 changes to your code. Shipyard provides powerful features like webhooks, error-handling, monitoring, automatic containerization, syncing with Github, and more. Plus, it comes with over 70 open-source, low-code templates to help you quickly build solutions with the tools you already use. Go to dataengineeringpodcast.com/shipyard to get started automating with a free developer plan today!
Your host as usual is Tobias Macey and today I’m interviewing Tian Gao about VizTracer, a low-overhead logging/debugging/profiling tool that can trace and visualize your python code execution
Interview
Introductions
How did you get introduced to Python?
Can you describe what VizTracer is and the story behind it?
What are the main goals that you are focused on with VizTracer?
What are some examples of the types of bugs that profiling can help diagnose?
How does profiling work together with other debugging approaches? (e.g. logging, breakpoint debugging, etc.)
There are a number of profiling utilities for Python. What feature or combination of features were missing that motivated you to create VizTracer?
Can you describe how VizTracer is implemented?
How have the design and goals changed since you started working on it?
There are a number of styles of profiling, what was your process for deciding which approach to use?
What are the most complex engineering tasks involved in building a profiling utility?
Can you describe the process of using VizTracer to identify and debug errors and performance issues in a project?
What are the options for using VizTracer in a production environment?
What are the interfaces and extension points that you have built in to allow developers to customize VizTracer?
What are some of the ways that you have used VizTracer while working on VizTracer?
What are the most interesting, innovative, or unexpected ways that you have seen VizTracer used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on VizTracer?
When is VizTracer the wrong choice?
What do you have planned for the future of VizTracer?
Keep In Touch
gaogaotiantian on GitHub
LinkedIn
Picks
Tobias
Travelers show on Netflix
Tian
objprint
Lincoln Lawyer
bilibili – Tian’s coding sessions in Chinese
Closing Announcements
Thank you for listening! Don’t forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. The Machine Learning Podcast helps you go from idea to production with machine learning.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
Viztracer
Python cProfile
Sampling Profiler
Perfetto
Coverage.py
Podcast Episode
Python setxprofile hook
Circular Buffer
Catapult Trace Viewer
py-spy
psutil
gdb
Flame graph
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
49 min
July 10, 2022Stream Processing In Real Time And At Scale In Pure Python With Bytewax
Summary
Analysis of streaming data in real time has long been the domain of big data frameworks, predominantly written in Java. In order to take advantage of those capabilities from Python requires using client libraries that suffer from impedance mis-matches that make the work harder than necessary. Bytewax is a new open source platform for writing stream processing applications in pure Python that don’t have to be translated into foreign idioms. In this episode Bytewax founder Zander Matheson explains how the system works and how to get started with it today.
Announcements
Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. And now you can launch a managed MySQL, Postgres, or Mongo database cluster in minutes to keep your critical data safe with automated backups and failover. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it. Select Star’s data discovery platform solves that out of the box, with a fully automated catalog that includes lineage from where the data originated, all the way to which dashboards rely on it and who is viewing them every day. Just connect it to your dbt, Snowflake, Tableau, Looker, or whatever you’re using and Select Star will set everything up in just a few hours. Go to pythonpodcast.com/selectstar today to double the length of your free trial and get a swag package when you convert to a paid plan.
Need to automate your Python code in the cloud? Want to avoid the hassle of setting up and maintaining infrastructure? Shipyard is the premier orchestration platform built to help you quickly launch, monitor, and share python workflows in a matter of minutes with 0 changes to your code. Shipyard provides powerful features like webhooks, error-handling, monitoring, automatic containerization, syncing with Github, and more. Plus, it comes with over 70 open-source, low-code templates to help you quickly build solutions with the tools you already use. Go to dataengineeringpodcast.com/shipyard to get started automating with a free developer plan today!
Your host as usual is Tobias Macey and today I’m interviewing Zander Matheson about Bytewax, an open source Python framework for building highly scalable dataflows to process ANY data stream.
Interview
Introductions
How did you get introduced to Python?
Can you describe what Bytewax is and the story behind it?
Who are the target users for Bytewax?
What is the problem that you are trying to solve with Bytewax?
What are the alternative systems/architectures that you might replace with Bytewax?
Can you describe how Bytewax is implemented?
What are the benefits of Timely Dataflow as a core building block for a system like Bytewax?
How have the design and goals of the project changed/evolved since you first started working on it?
What are the axes available for scaling Bytewax execution?
How have you approached the design of the Bytewax API to make it accessible to a broader audience?
Can you describe what is involved in building a project with Bytewax?
What are some of the stream processing concepts that engineers are likely to run up against as they are experimenting and designing their code?
What is your motivation for providing the core technology of your business as an open source engine?
How are you approaching the balance of project governance and sustainability with opportunities for commercialization?
What are the most interesting, innovative, or unexpected ways that you have seen Bytewax used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Bytewax?
When is Bytewax the wrong choice?
What do you have planned for the future of Bytewax?
Keep In Touch
Slack
Twitter
LinkedIn
Picks
Tobias
Alta Racks
Zander
Atherton Bikes
Links
Bytewax
GitHub
Flink
Data Engineering Podcast Episode
Spark Streaming
Kafka Connect
Faust
Podcast Episode
Ray
Podcast Episode
Dask
Data Engineering Podcast Episode
Timely Dataflow
PyO3
Materialize
Data Engineering Podcast Episode
HyperLogLog
Python River Library
Shannon Entropy Calculation
The blog post using incremental shannon entropy
NATS
waxctl
Prometheus
Grafana
Streamz
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
43 min
July 03, 2022Tetra: A Full Stack Web Framework That Doesn't Make You Write Everything Twice
Summary
Building a fully functional web application has been growing in complexity along with the growing popularity of javascript UI frameworks such as React, Vue, Angular, etc. Users have grown to expect interactive experiences with dynamic page updates, which leads to duplicated business logic and complex API contracts between the server-side application and the Javascript front-end. To reduce the friction involved in writing and maintaining a full application Sam Willis created Tetra, a framework built on top of Django that embeds the Javascript logic into the Python context where it is used. In this episode he explains his design goals for the project, how it has helped him build applications more rapidly, and how you can start using it to build your own projects today.
Announcements
Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. And now you can launch a managed MySQL, Postgres, or Mongo database cluster in minutes to keep your critical data safe with automated backups and failover. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
So now your modern data stack is set up. How is everyone going to find the data they need, and understand it? Select Star is a data discovery platform that automatically analyzes & documents your data. For every table in Select Star, you can find out where the data originated, which dashboards are built on top of it, who’s using it in the company, and how they’re using it, all the way down to the SQL queries. Best of all, it’s simple to set up, and easy for both engineering and operations teams to use. With Select Star’s data catalog, a single source of truth for your data is built in minutes, even across thousands of datasets. Try it out for free and double the length of your free trial today at pythonpodcast.com/selectstar. You’ll also get a swag package when you continue on a paid plan.
Need to automate your Python code in the cloud? Want to avoid the hassle of setting up and maintaining infrastructure? Shipyard is the premier orchestration platform built to help you quickly launch, monitor, and share python workflows in a matter of minutes with 0 changes to your code. Shipyard provides powerful features like webhooks, error-handling, monitoring, automatic containerization, syncing with Github, and more. Plus, it comes with over 70 open-source, low-code templates to help you quickly build solutions with the tools you already use. Go to dataengineeringpodcast.com/shipyard to get started automating with a free developer plan today!
Your host as usual is Tobias Macey and today I’m interviewing Sam Willis about Tetra, a full stack component framework for your Django applications
Interview
Introductions
How did you get introduced to Python?
Can you describe what Tetra is and the story behind it?
What are the problems that you are aiming to solve with this project?
What are some of the other ways that you have addressed those problems?
What are the shortcomings that you encountered with those solutions?
What was missing in the existing landscape of full-stack application development patterns that prompted you to build a new meta-framework?
What are some of the sources of inspiration (positive and negative) that you looked to while deciding on the component selection and implementation strategy?
Can you describe how Tetra is implemented?
What are the core principles that you are relying on to drive your design of APIs and developer experience?
What is the process for building a full component in Tetra?
What are some of the application design challenges that are introduced by Combining the javascript and Django logic and attributes? (e.g. reusing JS logic/CSS styles across components)
A perennial challenge with combining the syntax across multiple languages in a single file is editor support. How are you thinking about that with Tetra’s implementation?
What is your grand vision for Tetra and how are you working to make it sustainable?
What are the most interesting, innovative, or unexpected ways that you have seen Tetra used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Tetra?
When is Tetra the wrong choice?
What do you have planned for the future of Tetra?
Keep In Touch
@samwillis on Twitter
Website
LinkedIn
samwillis on GitHub
Picks
Tobias
The Machine Learning Podcast
Sam
Slow Horses TV Show
Closing Announcements
Thank you for listening! Don’t forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. The Machine Learning Podcast helps you go from idea to production with machine learning.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
Tetra Framework
Django
PHP
ASP
Alpine.js
HTMX
Ruby
Ruby on Rails
Flutterbox
Vue.js
Laravel Livewire
Python Import Hooks
python-inline-source
Tailwind CSS
PostCSS
Pickle
Fernet
esbuild
Webpack
Rich
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
54 min