September 28, 2021

66: Tracing Production with Kai Wern Choong

39 minutes

We talk with Kai Wern Choong about his experience with tracing performance problems in production. We talk about what “tracing” means in a BEAM system, available tools like recon, great resources for learning how the tools work and general approaches for troubleshooting live production Elixir systems. Kai also shares his Livebook notebook setup for demonstrating tracing techniques in an interactive way.

Show Notes online - http://podcast.thinkingelixir.com/66

Elixir Community News

https://twitter.com/hexpm/status/1439268469296570368 – New version of Hex released that fixes issues with version downgrades and slow version resolutions. Run mix local.hex to upgrade.

https://hexdocs.pm/credo/1.6.0-rc.0/changelog.html#1-6-0 – Credo 1.6.0 was released with new "First Run Mode"

https://twitter.com/elixirphoenix/status/1438222537301843970 – Removing Phoenix compiler config change can speed up compilation times when on up-to-date versions

https://github.com/Qqwy/elixir-type_check – New library called TypeCheck

https://blog.oestrich.org/2021/09/introducing-aino/ – Eric Osterich is experimenting with an alternative HTTP framework for Elixir

https://github.com/oestrich/aino – The Aino project

https://techcrunch.com/2021/09/14/logistics-startup-stord-raises-90m-in-kleiner-perkins-led-round-becomes-a-unicorn-and-acquires-another-company/ – Stord, an Elixir company, raised $90M and became a "unicorn"

https://spawnfest.org/ – SpawnFest 2021 happened

https://github.com/spawnfest/eflambe – SpawnFest submission - rapid generation of flamegraphs

https://github.com/spawnfest/Sketch – SpawnFest submission - library for creating generative art with Elixir

https://github.com/spawnfest/lotus – SpawnFest submission - Lotus is a Surface UI wrapper for UIKit

https://github.com/spawnfest/Discovery – SpawnFest submission - Platform for hosting realtime, stateful servers with zero downtime deployment and horizontal scaling on Kubernetes

https://github.com/spawnfest/crash – SpawnFest submission - Crash is a simple (distributed) docker-on-docker Continuous Integration system written in Elixir

Do you have some Elixir news to share? Tell us at @ThinkingElixir or email at [email protected]

Discussion Resources

https://kaiwern.com/posts/2021/06/27/debugging-with-tracing-in-elixir-with-recon_trace/

https://www.naluri.life/

https://ferd.github.io/recon/recon.html

https://hex.pm/packages/recon

https://twitter.com/kw7oe/status/1409126300636254219

https://github.com/kw7oe/livebook-notebooks/blob/main/debugging-with-tracing-in-elixir-with-recon_trace.livemd – Livebook notebook showing tracing examples

https://www.youtube.com/watch?v=sR9h3DZAA74 – Debugging Live Systems on the BEAM talk by Jeffery Utter

https://www.youtube.com/watch?v=OR2Gc6_Le2U – Operable Erlang and Elixir talk by Fred Hebert (skip to 25:51, where he started talking about tracing)

https://kaiwern.com/posts/2020/11/02/debugging-with-tracing-in-elixir/ – Tracing with dbg modules

https://en.wikipedia.org/wiki/Tracing_(software)

https://erlang.org/doc/man/dbg.html – dbg module in Erlang

:dbg.fun2ms

https://kubernetes.io/blog/2017/12/using-ebpf-in-kubernetes/

https://github.com/cilium/cilium

https://opentelemetry.io/docs/erlang/

https://www.brendangregg.com/blog/2019-01-01/learn-ebpf-tracing.html

https://www.erlang-solutions.com/capabilities/wombatoam/

https://hex.pm/packages/recon_ex

https://erlang-in-anger.com/ – Erlang in Anger (Chapter 9)

https://www.thegreatcodeadventure.com/testing-genservers-with-erlang-trace/ – Testing GenServers with Erlang Trace

http://blog.plataformatec.com.br/2016/04/how-to-trace-elixir-nodes-with-erlyberly/ – How to trace Elixir nodes with Erlyberly - Plataformatec Blog

http://erlang.org/doc/man/seq_trace.html#sequential-tracing – Seq_trace

http://erlang.org/doc/man/erlang.html#trace-3 – Erlang:trace

https://github.com/andytill/erlyberly – Erlang tracing for the masses

https://github.com/tatsuya6502/recon_ex – Elixir wrapper for Recon, tools to diagnose Erlang VM safely in production

Guest Information

https://twitter.com/kw7oe – on Twitter

https://github.com/kw7oe/ – on Github

https://kaiwern.com/ – Blog

Find us online

Message the show - @ThinkingElixir

Email the show - [email protected]

Mark Ericksen - @brainlid

David Bernheisel - @bernheisel

Cade Ward - @cadebward

...more

View all episodes

By ThinkingElixir.com

4.9

3232 ratings