The Python Podcast.init

By Tobias MaceyTechnology Education

The Python Podcast.init episodes

March 28, 2022Building A Community And Technology Stack For Scalable Big Data Geoscience At Pangeo
Summary
Science is founded on the collection and analysis of data. For disciplines that rely on data about the earth the ability to simulate and generate that data has been growing faster than the tools for analysis of that data can keep up with. In order to help scale that capacity for everyone working in geosciences the Pangeo project compiled a reference stack that combines powerful tools into an out-of-the-box solution for researchers to be productive in short order. In this episode Ryan Abernathy and Joe Hamman explain what the Pangeo project really is, how they have integrated a combination of XArray, Dask, and Jupyter to power these analytical workflows, and how it has helped to accelerate research on multidimensional geospatial datasets.
Announcements
Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
So now your modern data stack is set up. How is everyone going to find the data they need, and understand it? Select Star is a data discovery platform that automatically analyzes & documents your data. For every table in Select Star, you can find out where the data originated, which dashboards are built on top of it, who’s using it in the company, and how they’re using it, all the way down to the SQL queries. Best of all, it’s simple to set up, and easy for both engineering and operations teams to use. With Select Star’s data catalog, a single source of truth for your data is built in minutes, even across thousands of datasets. Try it out for free and double the length of your free trial today at pythonpodcast.com/selectstar. You’ll also get a swag package when you continue on a paid plan.
Your host as usual is Tobias Macey and today I’m interviewing Ryan Abernathy and Joe Hamman about Pangeo, a community platform for Big Data geoscience
Interview
Introductions
How did you get introduced to Python?
Can you describe what Pangeo is and the story behind it?
What is your role in the project/community and how did you get involved?
What are the goals of the project and community?
What are the areas of effort and how are they organized?
What are the scientific domains that Pangeo is focused on supporting?
What are the primary challenges associated with data management and analysis in these scientific communities?
What are the forms that these data take and how have they been evolving? (e.g. formats/sources)
What are some of the challenges introduced by the widespread adoption of cloud resources and the associated architectural patterns?
Can you describe the technical components that fall under the Pangeo umbrella?
How do they come together to form a functional workflow for geo sciences?
How has the scope of the Pangeo project changed or evolved since it started?
What are the most interesting, innovative, or unexpected ways that you have seen Pangeo used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Pangeo?
When is Pangeo the wrong choice?
What do you have planned for the future of Pangeo?
Keep In Touch
Joe
@HammanHydro on Twitter
Ryan
@rabernat on Twitter
rabernat on GitHub
Website
Picks
Tobias
Mountain Biking
Ryan
Klara And The Sun by Kazuo Ishiguro
Joe
Range by David Epstein
Closing Announcements
Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
Pangeo
Pangeo Forge
CarbonPlan
M2LInES
LEAP
Columbia University
XArray
MIT
MatLab
PHP
Ruby
Java
NumPy
SciPy
Matplotlib
C
Fortran
Perl
Dask
Data Engineering Podcast Episode
Jupyter
IDL
HDF5
Unidata
NetCDF
CF Metadata Conventions
Intake
Podcast Episode
FSSpec
Parquet
Data Engineering Podcast Episode
Zarr
Data Engineering Podcast
Pangeo Forge
Airbyte
Data Engineering Podcast Episode
Fivetran
Data Engineering Podcast Episode
Stitch
TileDB
Data Engineering Podcast Episode
Pythia
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
53 min
March 20, 2022Automating Application Lifecycles For Developer Happiness At Wayfair
Summary
A common piece of advice when starting anything new is to "begin with the end in mind". In order to help the engineers at Wayfair manage the complete lifecycle of their applications Joshua Woodward runs a team that provides tooling and assistance along every step of the journey. In this episode he shares some of the lessons and tactics that they have developed while assisting other engineering teams with starting, deploying, and sunsetting projects. This is an interesting look at the inner workings of large organizations and how they invest in the scaffolding that supports their myriad efforts.
Announcements
Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
So now your modern data stack is set up. How is everyone going to find the data they need, and understand it? Select Star is a data discovery platform that automatically analyzes & documents your data. For every table in Select Star, you can find out where the data originated, which dashboards are built on top of it, who’s using it in the company, and how they’re using it, all the way down to the SQL queries. Best of all, it’s simple to set up, and easy for both engineering and operations teams to use. With Select Star’s data catalog, a single source of truth for your data is built in minutes, even across thousands of datasets. Try it out for free and double the length of your free trial today at pythonpodcast.com/selectstar. You’ll also get a swag package when you continue on a paid plan.
Your host as usual is Tobias Macey and today I’m interviewing Joshua Woodward about how the application lifecycle team at Wayfair uses Python to
Interview
Introductions
Josh Woodward, for the past year have been managing the application lifecycle team at Wayfair. Prior to that, IC on python platforms team. Embed with teams looking to decouple from monolith. See pain points first hand.
How did you get introduced to Python?
High school physics class, TI84 Calculator, friend wrote a program to solve vector problems, I thought it was amazing.
Used TI-Basic to solve specific physics problems for me. (Give fixed inputs, run through equation, get outputs)
Approaching college, thinking about student loans.
Heard about python and decided to give it a shot.
Wrote program to simulate various payback / interest scenarios.
Went to college for ME, switched to SE when I found out my dorm neighbors were using python to draw cool images with python + turtle
Can you describe what the role of the application lifecycle team is and the story behind it?
Story behind it:
Around 2018, in a state where we had deploy congestion, challenging to iterate and ship changes. tech org invested in containerization and decoupling to directly combat this problem. Teams incentiviced to decouple.
While on python platforms, the team had already been experimenting with code templating.
Standard cookiecutter template for flask apps.
Wayfair experimenting with Kubernetes late 2017.
Spent 1 year embedding with 4 different teams to help knowledge transfer re: k8s, containers, application setup, python best practices, testing, linting, etc – through that we got a lot of great feedback on our tooling.
Took senior engineers weeks to get something setup.
Know who to contact, click the right buttons, file the right ticket
Approach: Counted manual steps. Something like 60 distinct / atomic activities that had to be performed to get a "hello world" response from a basic flask app in production.
Focus on reduce manual steps
Released product (Mamba, on theme of snakes)
Initially, supporting one main user story.
User story: "As an engineer, I would like to create a production ready application in 10 minutes so that I can have a reliable and standardized application setup that follows best practices."
grew out of python platforms, created own team with own scope, that was about 1.5 years ago.
What is your team’s scope now?
Team Scope is to facilitate the creation, maintenance, and decommissioning of decoupled applications at Wayfair.
What are the interfaces that your team has to the rest of the organization?
People Interfaces:
We value getting feedback on our work to build strong products.
Make assumptions, Willing to be wrong. Validate assumptions with customers.
Software Interfaces:
for mamba, CLI at first
Backstage (open sourced from spotify)
Lots of Github
What is your method of determining what projects to work on?
(See above). Known pain points. Intuition, Free day fridays. Being comfortable taking risk (using friday time). Vet solution with customers.
How do you measure the impact of your work on the rest of the organization?
We don’t force use of our products. Adoption of tooling.
Number of microservices being spun up.
Number of automated pull requests being created, merged.
DORA metrics throughput (deployment frequency, lead time for changes) and stability (change failure rate, mean time to recovery)
What is the role of Python in your work?
we use it and love it!
existing skillset from incubation phase within python platforms
right tool for the job
lightweight automation
hitting lots of APIs
define lots of user facing specifications (json, yaml)
pydantic has been great for creating descriptive, human and machine specifications.
open source (we rely on it, we also have some presence)
cookiecutter -> columbo
gitpython -> pygitops
Can you tell me more about your application creation solution. Who can use it, and what does it actually do?
Written in python, though it templates out code for any language.
Runs automation to onboard an application to production
git repo, build pipeline, calling out to various APIs to signal a new app is present
Wayfair has a variety of applications (python, java, .net, php, javascript, some go)
Team interested in integrating with our solution will create a github repository containing 1..* cookiecutter template(s)
Provide a specification for what questions to ask users.
Limitation with cookiecutter where the approach to ask questions isn’t dynamic. lack of validation.
Pat Lannigan -> Columbo (open sourced). Python DSL to describe the set of questions to ask users.
python fastapi application will have a completely different set of questions than a java library for example.
You had mentioned that another part of your team scope is to facilitate the maintenance of applications. Can you tell me more about that?
Reduce engineering toil around keeping applications up to date.
Average engineer owns several, dozens of repos
Create automated pull requests:
Versioned dependencies (Renovate)
Propagating platform changes (Gator)
Ex1: python apps use "black" to format code and our python platform team would like to prescribe a line length. Our tooling can be used to declare desired changes. yaml specification -> pr automation at scale.
Ex2: shared library, new version released, breaking interface change. Code instructions for performing AST manipulation and resolving breaking change for people.
Shift from: "We need you to do this", "I am proactively letting you know that something needs to change, and I also made the change for you!"
How do you actually go about creating automated pull requests?
manual steps would involve cloning, checking out feature branch, applying code changes, staging / committing, pushing up branch, creating the PR
gitpython is an existing and extremely powerful tool, but its api is fairly involved and (by design) doesn’t provide the type of high level abstractions that we need.
created pygitops (open sourced), built completely on top of gitpython
high level abstractions for the workflow I described.
coolest / most pythonic part about it is the "feature branch" context manager.
code changes are made in the context of a feature branch
when you intentionally or accidentally leave the context of a feature branch, we want certain things to be true (default / main branch, clean workdir, no unstaged changes)
when writing PR automation, don’t have to worry about this!
Can you describe some of the more technical details about how your change propagation system (Gator) works?
heavily inspired by kubernetes resource model (resources are defined via a declarative specification)
Kubernetes itself ships with resources that implement behaviors of common resources (pods, services, etc)
Gator’s execution model is broken up into two parts:
what repos to act on (Source)
what are the changes that need to be applied. (Output)
Ex: Source to proxy github search. write github search query to get back list of repos
Output to scan a repo for regex pattern at specified paths and replace with some fixed term. Very popular, engineers love find and replace.
What are the most interesting, innovative, or unexpected ways that you have seen mamba / gator used?
resource model of gator supports the idea of we don’t know, what we don’t know
reference k8s, CRDs, resource model.
container execution
log4j identification and remidiation
automate some of the work for identifying vulnerabilities
java platform team was able to use java native tooling in the environment of their choosing to identify vulnerable apps.
What are the most interesting, unexpected, or challenging lessons that you have learned while working on application lifecycle concerns?
What do you have planned for the future of application lifecycle management/developer experience improvements at Wayfair?
Hope to start open sourcing interesting aspects of our change propagation tool (Gator)
As someone who maintains many open source projects, or even at the enterprise level, we think that some of our patterns and approaches can be shared! yaml -> code changes
Keep In Touch
Email
Github
Linkedin
Picks
Tobias
Nocciolata hazelnut spread
Joshua
Cities Skylines Game
Cities Skylines – Cities Planner Plays: Verde Beach
Links
pygitops
columbo
backstage
renovate
DORA metrics
TI-84 Calculator
TI BASIC
Wayfair Python Platforms Team Podcast Episode
Pydantic
Podcast Episode
Helm
PyUp
GitPython
The intro and outro music is from Requiem for a Fish The…
47 min
March 14, 2022Run Your Applications Reliably On Kubernetes Without Losing Sleep With Robusta
Summary
Kubernetes is a framework that aims to simplify the work of running applications in production, but it forces you to adopt new patterns for debugging and resolving issues in your systems. Robusta is aimed at making that a more pleasant experience for developers and operators through pre-built automations, easy debugging, and a simple means of creating your own event-based workflows to find, fix, and alert on errors in production. In this episode Natan Yellin explains how the project got started, how it is architected and tested, and how you can start using it today to keep your Python projects running reliably.
Announcements
Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
So now your modern data stack is set up. How is everyone going to find the data they need, and understand it? Select Star is a data discovery platform that automatically analyzes & documents your data. For every table in Select Star, you can find out where the data originated, which dashboards are built on top of it, who’s using it in the company, and how they’re using it, all the way down to the SQL queries. Best of all, it’s simple to set up, and easy for both engineering and operations teams to use. With Select Star’s data catalog, a single source of truth for your data is built in minutes, even across thousands of datasets. Try it out for free and double the length of your free trial today at pythonpodcast.com/selectstar. You’ll also get a swag package when you continue on a paid plan.
Your host as usual is Tobias Macey and today I’m interviewing Natan Yellin about Robusta,
Interview
Introductions
How did you get introduced to Python?
Can you describe what Robusta is and the story behind it?
What are some of the challenges that teams face when running their systems in Kubernetes?
How does Robusta help address those difficulties?
How does Robusta compare to e.g. Rookout?
What are some of the ways that Robusta is able to provide specific insights for Python applications?
Can you describe how Robusta is implemented?
What are some of the most challenging engineering tasks that you have had to work through while building Robusta?
How have the capabilities and components evolved from when you started working on it?
What is the workflow for integrating Robusta into a Kubernetes environment and a team’s maintenance processes?
What are some examples of the kinds of questions that Robusta can help answer out of the box?
What are some tasks that Robusta facilitates which require manual exploration?
What are the interfaces available for customizing and extending the functionality of Robusta?
What is involved in adding a new automation capability to Robusta?
How have you approached the design of the tool to make it ergonomic and intuitive so that it doesn’t contribute to the stresses of dealing with errors in production?
Given that it is a tool to help resolve problems in production infrastructure, how have you worked to ensure its reliability and resilience?
What is the governance and sustainability model for Robusta?
What are the most interesting, innovative, or unexpected ways that you have seen Robusta used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Robusta?
When is Robusta the wrong choice?
What do you have planned for the future of Robusta?
Keep In Touch
LinkedIn
@aantn on Twitter
aantn on GitHub
Website
Picks
Tobias
Kubernetes: Up And Running (affiliate link)
Natan
Kubernetes for SysAdmins Youtube video by Kelsey Hightower
Learn to delegate
Closing Announcements
Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
Robusta
GHOP
Objective C
Snyk
Heroku
Google AppEngine
OOM Killer
Bin Packing/Knapsack Problem
Prometheus
Kubernetes Pods
PySpy
tracemalloc
Pyrasite
VSCode Debugger
Pydantic
Podcast Episode
Helm – Kubernetes package manager
Why Profiler
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
54 min
March 06, 2022Accelerate The Development And Delivery Of Your Machine Learning Applications Using Ray And Deploy It At Anyscale
Summary
Building a machine learning application is inherently complex. Once it becomes necessary to scale the operation or training of the model, or introduce online re-training the process becomes even more challenging. In order to reduce the operational burden of AI developers Robert Nishihara helped to create the Ray framework that handles the distributed computing aspects of machine learning operations. To support the ongoing development and simplify adoption of Ray he co-founded Anyscale. In this episode he re-joins the show to share how the project, its community, and the ecosystem around it have grown and evolved over the intervening two years. He also explains how the techniques and adoption of machine learning have influenced the direction of the project.
Announcements
Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Your host as usual is Tobias Macey and today I’m interviewing Robert Nishihara about his work at Anyscale and the Ray distributed execution framework
Interview
Introductions
How did you get introduced to Python?
Can you describe what Anyscale is and the story behind it?
How has the Ray project and ecosystem evolved since we last spoke? (2 years ago)
How has the landscape of AI/ML technologies and techniques shifted in that time?
What are the main areas where organizations are trying to apply ML/AI?
What are some of the issues that teams encounter when trying to move from prototype to production with ML/AI applications?
What are the features of Ray that help to mitigate those challenges?
With the introduction of more widely available streaming/real-time technologies the viability of reinforcement learning has increased. What new challenges does that approach introduce?
What are some of the operational complexities associated with managing a deployment of Ray?
What are some of the specialized utilities that you have had to develop to maintain a large and multi-tenant platform for your customers?
What is the governance model around the Ray project and how does the work at Anyscale influence the roadmap?
What are the most interesting, innovative, or unexpected ways that you have seen Anyscale/Ray used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Ray and Anyscale?
When is Anyscale/Ray the wrong choice?
What do you have planned for the future of Anyscale/Ray?
Keep In Touch
robertnishihara on GitHub
@robertnishihara on Twitter
Website
LinkedIn
Picks
Tobias
The Edge Chronicles: Beyond The Deepwoods
Robert
Production RL Summit
Project Hail Mary by Andy Weir
Closing Announcements
Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
Ray
Podcast Episode
Anyscale
UC Berkeley
Matlab
Deep Learning
Pandas
NumPy
Horovod
Podcast Episode
XGBoost
Modin
Podcast Episode
Dask
Ray Datasets
Reinforcement Learning
Production Reinforcement Learning Summit
AlphaGo
Databricks
Snowflake
Data Engineering Podcast Episode
TPU == Tensor Processing Unit
Weights and Biases
MLFlow
RLLib
Ray Serve
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
46 min
February 28, 2022See The Structure Of Your Software At A Glance With Call Graphs From Code2Flow
Summary
As software projects grow and change it can become difficult to keep track of all of the logical flows. By visualizing the interconnections of function definitions, classes, and their invocations you can speed up the time to comprehension for newcomers to a project, or help yourself remember what you worked on last month. In this episode Scott Rogowski shares his work on Code2Flow as a way to generate a call graph of your programs. He explains how it got started, how it works, and how you can start using it to understand your Python, Ruby, and PHP projects.
Announcements
Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Subsurface Live is the cloud data lake conference, a virtual conference where data engineers, data scientists, data architects, and data analysts can gather and hear about cloud data lakes and the data ecosystem. Subsurface Live Winter 2022 includes keynote talks from Bill Inmon, the father of the data warehouse, Author of Deep Work Cal Newport, and several more from companies such as Dremio, AWS, dbt, and more. Subsurface will also have many breakout sessions featuring Pandas creator Wes McKinney, Apache Superset & Airflow creator Maxime Beauchemin, and engineers from Apple, Uber, Adobe, Bloomberg, and more. Meet other data professionals and learn about the data technologies and practices helping companies meet their current and future data needs. Register today at pythonpodcast.com/subsurface
Your host as usual is Tobias Macey and today I’m interviewing Scott Rogowski about Code2Flow, a utility for generating "pretty good" call graphs for dynamic languages
Interview
Introductions
How did you get introduced to Python?
Can you describe what Code2Flow is and the story behind it?
What are some of the ways that a program’s call graph might be used?
How does the visual representation generated by Code2Flow help with exploring the structure of a project?
What are some of the alternative approaches/tools that might be used to gain similar insights?
What do you see as the overlap in utility between Code2Flow and e.g. SourceGraph?
Can you describe how the Code2Flow project is implemented?
How have the design and goals of the project changed since you first began working on it?
Given that Code2Flow is implemented in Python, how have you managed the parsing/processing of the other languages that you support?
Visualizing a complex program can quickly become very messy. How have you approached the layout of the output to enhance comprehension?
What are some of the situations where Code2Flow will be unable to provide a full picture of a program’s call graph?
What are some of the pieces of information that are unavailable due to the static analysis approach that you have taken?
Can you describe the process of applying Code2Flow to a project?
Once the structure is on display, what are some next steps that an individual or team might take to analyze and act on the information?
Given the static nature of the output, how might Code2Flow be incorporated in a CI/CD system to provide insight into the evolution of a projects structure?
What are the most interesting, innovative, or unexpected ways that you have seen Code2Flow used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Code2Flow?
When is Code2Flow the wrong choice?
What do you have planned for the future of Code2Flow?
Keep In Touch
Website
scottrogowski on GitHub
Picks
Tobias
Taking Vacation
Universal Studios, Florida
Scott
Service work
Closing Announcements
Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
Code2Flow
Colombia
Mongita
TI-83
Ruby
PHP
AST == Abstract Syntax Tree
Graphviz
Pylint
Robert Frost
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
46 min
February 21, 2022Scaling Knowledge Management For Technical Teams With Knowledge Repo
Summary
One of the most persistent challenges faced by organizations of all sizes is the recording and distribution of institutional knowledge. In technical teams this is exacerbated by the need to incorporate technical review feedback and manage access to data before publishing. When faced with this problem as an early data scientist at AirBnB, Chetan Sharma helped create the Knowledge Repo project as a solution. In this episode he shares the story behind its creation and growth, how and why it was released as open source, and the features that make it a compelling option for your own team’s knowledge management journey.
Announcements
Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Your host as usual is Tobias Macey and today I’m interviewing Chetan Sharma about Knowledge Repo, an open source framework for managing documentation for technical users
Interview
Introductions
How did you get introduced to Python?
EE + CS/AI + Stats degrees
Airbnb working on ML models
Knowledge Repo itself
Can you describe what Knowledge Repo is and the story behind it?
We started seeing interviewees use ipython notebooks, thought they were great
Wanted to push more people to use notebooks, but they weren’t very shareable, vettable
Existing notebook hosting services weren’t very good, and weren’t built for people who aren’t data stakeholders. It was especially poor with images, annoying cell blocks
Made a simple post processor to remove cell blocks, push the images to s3, and host on flask
Once we were pushing notebooks into a Github repo for hosting on a flask app, so many things became possible
Review cycles
Shareability / collaboration features
Indexing / searching
Concurrently, great work was happening on developing internal R packages / python libraries to provide consistent, branded aesthetics
What are some of the approaches that teams typically take for recording and sharing institutional knowledge?
Copy and paste to google docs, slides
Facebook was using facebook photo albums
untrustworthy, not discoverable, divorced from the code
What are the unique requirements that are introduced when attempting to record and distribute learnings related to data such as A/B experiments, analytical methods, data sets, etc.?
Reproducibility is a big one
Making sure the learnings are trustworthy (good data? no bugs?)
Distributing widely, across the org and across time
Experimentation
Experimentation is at the end of a research-design-build-measure cycle, strategic analysis is often before
Capturing all of the context
Can you describe how the Knowledge Repo project is architected?
Repositories: a store of posts, most commonly a github repo
Markdown as original lingua franca, eventually a KR specific “KR post” concept (which is still basically markdown)
Post processors
Convert whatever upstream file to markdown / KR post (Jupyter notebook, R Markdown, markdown were the original ones)
Handle images and other large assets, usually pushing them to cloud storage
Evolved to handle PDFs, googledocs, keynotes
What were the motivating factors for making it available as an open source project?
It was such a common problem. Even incredibly sophisticated data teams at Uber, Facebook, etc. were begging us to share the system.
What is the workflow for creating, sharing, and discovering information in an installation of Knowledge Repo?
Create a github repo for hosting strategic analysis
Use the KR script to create a stub/template for whatever format you’re working in
Do your work in Jupyter, etc.
Instead of using github scripts (git add) use knowledge scripts (knowledge add), which is basically the github scripts with postprocessors
Do typical Github workflows
See the result in the hosted knowledge repo app
What are some of the options available for extending or customizing an installation of Knowledge Repo?
More postprocessors! google docs, presentations, UX research, anything can be done in KR with a simple postprocessor to turn it to markdown/images/PDF
Tying the system to your internal data tools. For example, an experimentation system like Eppo or whatever you use for marketing campaigns
If you were to start over today, what are some of the ways that you might approach the solution to knowledge management differently?
Think of it more holistically:
What are the most interesting, innovative, or unexpected ways that you have seen Knowledge Repo used?
UX research
Writing up guide for acquihiring
Demonstrating of capabilities, data framework
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Knowledge Repo?
Strategic analysis needs to be elevated, this leads to paradigm changes
Organization problems are helped by tools like KR: eg. promotions
Meeting people’s tools/workflows where they are is powerful
When is Knowledge Repo the wrong choice?
Keep In Touch
LinkedIn
@chesharma87
Picks
Tobias
Learning Guitar
Chetan
Underrated cooking ingredients: chickpea flour, butter fried kimchi (in grilled cheese, nachos)
Closing Announcements
Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
Eppo
Data Engineering Podcast Episode
Knowledge Repo
IPython
Jupyter
Flask
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
40 min
February 14, 2022Simplify And Scale Your Software Development Cycles By Putting On Pants (Build Tool)
Summary
Software development is a complex undertaking due to the number of options available and choices to be made in every stage of the lifecycle. In order to make it more scaleable it is necessary to establish common practices and patterns and introduce strong opinions. One area that can have a huge impact on the productivity of the engineers engaged with a project is the tooling used for building, validating, and deploying changes introduced to the software. In this episode maintainers of the Pants build tool Eric Arellano, Stu Hood, and Andreas Stenius discuss the recent updates that add support for more languages, efforts made to simplify its adoption, and the growth of the community that uses it. They also explore how using Pants as the single entry point for all of your routine tasks allows you to spend your time on the decisions that matter.
Announcements
Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Building data integration workflows is time consuming and tedious, requiring an unpleasant amount of boilerplate code to do it right. Rivery is a managed platform for building our ELT pipelines that offers the industry’s first native integration with Python, allowing you to seamlessly load and export Pandas dataframes to and from all of your databases, services, and data warehouses with a few clicks and no extra code. Rivery is hosting a live demo of their first class Python support on February 22nd, and when you use the promo code "Python" during registration you will be entered to win a brand new series 7 apple watch. Go to pythonpodcast.com/rivery today to learn more and register.
Your host as usual is Tobias Macey and today I’m interviewing Eric Arellano, Stu Hood, and Andreas Stenius about the Pants build tool and all of the work that has gone into it recently
Interview
Introductions
How did you get introduced to Python?
Can you describe what Pants is and the story behind it?
What is the scope of concerns that Pants is focused on addressing?
What are some of the notable changes in the project and its ecosystem over the past 1 1/2 years?
How do you approach the work of defining the target scope of the Pants toolchain?
What are some of your guiding principles to decide when a feature request belongs in the core vs as a plugin?
What are some of the ergonomic improvements that you have added to simplify the work of getting started with Pants and adopting it across teams?
What are some of the challenges that teams run into as they start to scale the size of their monorepos? (e.g. project design, boilerplate reduction, etc.)
How are you managing the work of growing and supporting the community as you move beyond early adopters/experts into newcomers to Pants and programming?
How are you handling support for multiple language ecosystems?
What are some of the challenges involved with making Pants feel idiomatic for such a range of communities?
How does the use of Python as the plugin/extension syntax work for teams that don’t use it as their primary language?
What are the architectural changes that needed to be made for you to be capable of integrating with the different execution environments?
How would you characterize the level of feature coverage across the different supported languages?
Now that you have laid the foundation, how much effort is required to add new language targets?
What are the most interesting, innovative, or unexpected ways that you have seen Pants used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Pants?
When is Pants the wrong choice?
What do you have planned for the future of Pants?
Keep In Touch
Eric
LinkedIn
Eric-Arellano on GitHub
@earellanoaz on Twitter
Stu
LinkedIn
@stuhood on Twitter
stuhood on GitHub
Andreas
@andreasstenius on Twitter
kaos on GitHub
Picks
Tobias
Last Kingdom on Netflix
Eric
Getting Curious
Stu
Checks and Balance Podcast
Andreas
The Pragmatic Programmer
Links
Pants
Make
Earthly
Podcast Episode
MyPy
Podcast Episode
PyRight
Pylint
Flake8
Podcast Episode
Bazel
pre-commit
Podcast Episode
Underpants library
PyOxidizer
Podcast Episode
Eric PyCon Talk
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
59 min
February 06, 2022Achieve Repeatable Builds Of Your Software On Any Machine With Earthly
Summary
It doesn’t matter how amazing your application is if you are unable to deliver it to your users. Frustrated with the rampant complexity involved in building and deploying software Vlad A. Ionescu created the Earthly tool to reduce the toil involved in creating repeatable software builds. In this episode he explains the complexities that are inherent to building software projects and how he designed the syntax and structure of Earthly to make it easy to adopt for developers across all language environments. By adopting Earthly you can use the same techniques for building on your laptop and in your CI/CD pipelines.
Announcements
Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Your host as usual is Tobias Macey and today I’m interviewing Vlad A. Ionescu about Earthly, a syntax and runtime for software builds to reduce friction between development and delivery
Interview
Introductions
How did you get introduced to Python?
Can you describe what Earthly is and the story behind it?
What are the core principles that engineers should consider when designing their build and delivery process?
What are some of the common problems that engineers run into when they are designing their build process?
What are some of the challenges that are unique to the Python ecosystem?
What is the role of Earthly in the overall software lifecycle?
What are the other tools/systems that a team is likely to use alongside Earthly?
What are the components that Earthly might replace?
How is Earthly implemented?
What were the core design requirements when you first began working on it?
How have the design and goals of Earthly changed or evolved as you have explored the problem further?
What is the workflow for a Python developer to get started with Earthly?
How can Earthly help with the challenge of managing Javascript and CSS assets for web application projects?
What are some of the challenges (technical, conceptual, or organizational) that an engineer or team might encounter when adopting Earthly?
What are some of the features or capabilities of Earthly that are overlooked or misunderstood that you think are worth exploring?
What are the most interesting, innovative, or unexpected ways that you have seen Earthly used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Earthly?
When is Earthly the wrong choice?
What do you have planned for the future of Earthly?
Keep In Touch
LinkedIn
@VladAIonescu on Twitter
Website
Picks
Tobias
Shape Up book
Vlad
High Output Management by Andy Grove
Closing Announcements
Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
Earthly
Bazel
Pants
Podcast Episode
ARM
AWS Graviton
Apple M1 CPU
Qemu
Phoenix web framework for Elixir language
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
55 min
January 31, 2022Building A Detailed View Of Your Software Delivery Process With The Eiffel Protocol
Summary
The process of getting software delivered to an environment where users can interact with it requires many steps along the way. In some cases the journey can require a large number of interdependent workflows that need to be orchestrated across technical and organizational boundaries, making it difficult to know what the current status is. Faced with such a complex delivery workflow the engineers at Ericsson created a message based protocol and accompanying tooling to let the various actors in the process provide information about the events that happened across the different stages. In this episode Daniel Ståhl and Magnus Bäck explain how the Eiffel protocol allows you to build a tooling agnostic visibility layer for your software delivery process, letting you answer all of your questions about what is happening between writing a line of code and your users executing it.
Announcements
Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Your host as usual is Tobias Macey and today I’m interviewing Daniel Ståhl and Magnus Bäck about Eiffel, an open protocol for platform agnostic communication for CI/CD systems
Interview
Introductions
How did you get introduced to Python?
Can you describe what Eiffel is and the story behind it?
What are the goals of the Eiffel protocol and ecosystem?
What is the role of Python in the Eiffel ecosystem?
What are some of the types of questions that someone might ask about their CI/CD workflow?
How does Eiffel help to answer those questions?
Who are the personas that you would expect to interact with an Eiffel system?
Can you describe the core architectural elements required to integrate Eiffel into the software lifecycle?
How have the design and goals of the Eiffel protocol/architecture changed or evolved since you first began working on it?
What are some example workflows that an engineering/product team might build with Eiffel?
What are some of the challenges that teams encounter when integrating Eiffel into their delivery process?
What are the most interesting, innovative, or unexpected ways that you have seen Eiffel used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Eiffel?
When is Eiffel the wrong choice?
What do you have planned for the future of Eiffel?
Keep In Touch
Daniel
d-stahl-ericsson on GitHub
LinkedIn
Magnus
LinkedIn
magnusbaeck on GitHub
Picks
Tobias
Red Notice
Daniel
The Witcher
Magnus
Lego
Closing Announcements
Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
Eiffel
Ericsson
Axis Communications
Hudson CI framework
Spinnaker
Jenkins
Tekton
Gradle
Artifactory
JSON Schema
RabbitMQ
Prometheus
Continuous Delivery Foundation
CD Events
XKCD Competing Standards
Python Eiffel SDK
Pydantic
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
50 min
January 24, 2022Improve Your Productivity By Investing In Developer Experience Design For Your Projects
Summary
When we are creating applications we spend a significant amount of effort on optimizing the experience of our end users to ensure that they are able to complete the tasks that the system is intended for. A similar effort that we should all consider is optimizing the developer experience for ourselves and other engineers who contribute to the projects that we work on. Adam Johnson recently wrote a book on how to improve the developer experience for Django projects and in this episode he shares some of the insights that he has gained through that project and his work with clients to help you improve the experience that you and your team have when collaborating on software development.
Announcements
Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Your host as usual is Tobias Macey and today I’m interviewing Adam Johnson about optimizing your developer experience
Interview
Introductions
How did you get introduced to Python?
Can you describe what you mean by the term "developer experience"?
How does it compare to the concept of user experience design?
What are the main goals that you aim for through improving DX?
When considering DX, what are the categories of focus for improvement? (e.g. the experience of a given software project, the developer’s physical environment, their editing environment, etc.)
What are some of the most high impact optimizations that a developer can make?
What are some of the areas of focus that have the most variable impact on a developer’s experience of a project?
What are some of the most helpful tools or practices that you rely on in your own projects?
How does the size of a development team or the scale of an organization impact the decisions and benefits around DX improvements?
One of the perennial challenges with selecting a given tool or architectural pattern is the continually changing landscape of software. How have your choices for DX strategies changed or evolved over the years?
What are the most interesting, innovative, or unexpected developer experience tweaks that you have encountered?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on your book?
What are some of the potential pitfalls that individuals and teams need to guard against in their quest to improve developer experience for their projects?
What are some of the new tools or practices that you are considering incorporating into your own work?
Keep In Touch
@AdamChainz on Twitter
Website
adamchainz on GitHub
Picks
Tobias
Eternals movie
Adam
Fan of Eternals, enjoyed Neil Gaiman series
Also general MCU fan, watched it all in lockdown
Moon Knight trailer
Closing Announcements
Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
Boost Your Django DX
Rust
Ripgrep
Factory Boy
Mimesis
Podcast Episode
Language Server Protocol
EditorConfig
Starship Command Prompt
Pre-Commit
Podcast Episode
Flake8
Podcast Episode
DevDocs
Dash library documentation search tool
pyupgrade
StandardJS
Cython
Podcast Episode
The Phoenix Project
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
43 min