BSD Now

By JT PenningtonNews Education How To Tech News

BSD Now episodes

February 07, 2019284: FOSDEM 2019
We recap FOSDEM 2019, FreeBSD Foundation January update, OPNsense 19.1 released, the hardware-assisted virtualization challenge, ZFS and GPL terror, ClonOS 19.01-RELEASE, and more.
Headlines
FOSDEM 2019 Recap
Allan and I were at FOSDEM 2019 in Brussels, Belgium over the weekend.
On the Friday before, we held a FreeBSD Devsummit in a hotel conference room, with 25 people attending. We talked about various topics of interest to the project. You can find the notes on the wiki page.
Saturday was the first day of FOSDEM. The FreeBSD Project had a table next to the Illumos Project again. A lot of people visited our table, asked questions, or just said “Hi, I watch BSDNow.tv every week”. We handed out a lot of stickers, pens, swag, and flyers. There was also a full day BSD devroom, with a variety of talks that were well attended.
In the main conference track, Allan held a talk explaining how the ZFS ARC works. A lot of people attended the talk and had more questions afterwards. Another well attended talk was by Jonathan Looney about Netflix and FreeBSD.
Sunday was another day in the same format, but no bsd devroom. A lot of people visited our table, developers and users alike. A lot of meeting and greeting went on.
Overall, FOSDEM was a great success with FreeBSD showing a lot of presence. Thanks to all the people who attended and talked to us. Special thanks to the people who helped out at the FreeBSD table and Rodrigo Osorio for running the BSD devroom again.
FreeBSD Foundation Update, January 2019
Dear FreeBSD Community Member,
Happy New Year! It’s always exciting starting the new year with ambitious plans to support FreeBSD in new and existing areas. We achieved our fundraising goal for 2018, so we plan on funding a lot of work this year! Though it’s the new year, this newsletter highlights some of the work we accomplished in December. We also put together a list of technologies and features we are considering supporting, and are looking for feedback on what users want to help inform our 2019 development plans. Our advocacy and education efforts are in full swing as we prepare for upcoming conferences including FOSDEM, SANOG33, and SCaLE.
Finally, we created a year-end video to talk about the work we did in 2018. That in itself was an endeavor, so please take a few minutes to watch it! We’re working on improving the methods we use to inform the community on the work we are doing to support the Project, and are always open to feedback. Now, sit back, grab a refreshing beverage, and enjoy our newsletter!
Happy reading!!
Deb
OPNsense 19.1 released
For more than four years now, OPNsense is driving innovation through modularising and hardening the open source firewall, with simple and reliable firmware upgrades, multi-language support, HardenedBSD security, fast adoption of upstream software updates as well as clear and stable 2-Clause BSD licensing.
The 19.1 release, nicknamed “Inspiring Iguana”, consists of a total of 620 individual changes since 18.7 came out 6 months ago, spread out over 12 intermediate releases including the recent release candidates. That is the average of 2 stable releases per month, security updates and important bug fixes included! If we had to pick a few highlights it would be: The firewall alias API is finally in place. The migration to HardenedBSD 11.2 has been completed. 2FA now works with a remote LDAP / local TOTP combination. And the OpenVPN client export was rewritten for full API support as well.
These are the most prominent changes since version 18.7:
fully functional firewall alias API
PIE firewall shaper support
firewall NAT rule logging support
2FA via LDAP-TOTP combination
WPAD / PAC and parent proxy support in the web proxy
P12 certificate export with custom passwords
Dpinger is now the default gateway monitor
ET Pro Telemetry edition plugin[2]
extended IPv6 DUID support
Dnsmasq DNSSEC support
OpenVPN client export API
Realtek NIC driver version 1.95
HardenedBSD 11.2, LibreSSL 2.7
Unbound 1.8, Suricata 4.1
Phalcon 3.4, Perl 5.28
firmware health check extended to cover all OS files, HTTPS mirror default
updates are browser cache-safe regarding CSS and JavaScript assets
collapsible side bar menu in the default theme
language updates for Chinese, Czech, French, German, Japanese, Portuguese and Russian
API backup export, Bind, Hardware widget, Nginx, Ntopng, VnStat and Dnscrypt-proxy plugins
Here are the full changes against version 19.1-RC2:
ipsec: add firewall interface as soon as phase 1 is enabled
ipsec: phase 1 selection GUI JavaScript compatibility fix
monit: widget improvements and bug fix (contributed by Frank Brendel)
ui: fix regression in single host or network subnet select in static pages
plugins: os-frr 1.7 updates OSFP outbound rules (contributed by Fabian Franz)
plugins: os-telegraf 1.7.4 fixes packet filter input
plugins: os-theme-rebellion 1.8.2 adds image colour invert
plugins: os-vnstat 1.1[3]
plugins: os-zabbix-agent now uses Zabbix version 4.0
src: revert mmc_calculate_clock() as HS200/HS400 support breaks legacy support
src: update sqlite3-3.20.0 to sqlite3-3.26.0[4]
src: import tzdata 2018h, 2018i[5]
src: avoid unsynchronized updates to kn_status[6]
ports: ca_root_nss 3.42
ports: dhcp6c 20190128 prevent rawops double-free (contributed by Team Rebellion)
ports: sudo patch to fix listpw=never[7]
News Roundup
The hardware-assisted virtualization challenge
Over two years ago, I made a pledge to use NetBSD as my sole OS and only operating system, and to resist booting into any other OS until I had implemented hardware-accelerated virtualization in the NetBSD kernel (the equivalent of Linux’ KVM, or Hyper-V).
Today, I am here to report: Mission Accomplished!
It’s been a long road, but we now have hardware-accelerated virtualization in the kernel! And while I had only initially planned to get Oracle VirtualBox working, I have with the help of the Intel HAXM engine (the same backend used for virtualization in Android Studio) and a qemu frontend, successfully managed to boot a range of mainstream operating systems.
ZFS and GPL terror: How much freedom is there in Linux?
ZFS – the undesirable guest
ZFS is todays most advanced filesystem. It originated on the Solaris operating system and thanks to Sun’s decision to open it up, we have it available on quite a number of Unix-like operating systems. That’s just great! Great for everyone.
For everyone? Nope. There are people out there who don’t like ZFS. Which is totally fine, they don’t need to use it after all. But worse: There are people who actively hate ZFS and think that others should not use it. Ok, it’s nothing new that some random guys on the net are acting like assholes, trying to tell you what you must not do, right? Whoever has been online for more than a couple of days probably already got used to it. Unfortunately its still worse: One such spoilsport is Greg Kroah-Hartman, Linux guru and informal second-in-command after Linus Torvalds.
There have been some attempts to defend the stance of this kernel developer. One was to point at the fact that the “ZFS on Linux” (ZoL) port uses two kernel functions, __kernel_fpu_begin() and __kernel_fpu_end(), which have been deprecated for a very long time and that it makes sense to finally get rid of them since nothing in-kernel uses it anymore. Nobody is going to argue against that. The problem becomes clear by looking at the bigger picture, though:
The need for functions doing just what the old ones did has of course not vanished. The functions have been replaced with other ones. And those ones are deliberately made GPL-only. Yes, that’s right: There’s no technical reason whatsoever! It’s purely ideology – and it’s a terrible one.
ClonOS 19.01-RELEASE
ClonOS is a turnkey Open Source platform based on FreeBSD and the CBSD framework. ClonOS offers a complete web UI for easily controlling, deploying and managing FreeBSD jails containers and Bhyve/Xen hyperviser virtual environments.
ClonOS is currently the only platform available which allow both Xen and Bhyve hypervisor to coexist on the same host. Being a FreeBSD base platform, ClonOS ability to create and manage jails allows you to run FreeBSD applications without losing performance.
Features:
easy management via web UI interface
live Bhyve migration [coming soon, roadmap]
Bhyve management (create, delete VM)
Xen management (create, delete VM) [coming soon, roadmap]
connection to the “physical” guest console via VNC from the browser or directly
Real time system monitoring
access to load statistics through SQLite3 and beanstalkd
support for ZFS features (cloning, snapshots)
import/export of virtual environments
public repository with virtual machine templates
puppet-based helpers for configuring popular services
ClonOS is a free open-source FreeBSD-based platform for virtual environments creation and management. In the core:
FreeBSD OS as hoster platform
bhyve(8) as hypervisor engine
Xen as hypervisor engine
vale(4) as Virtual Ethernet Switch
jail(8) as container engine
CBSD Project as management tools
Puppet as configuration management
Beastie Bits
Florian Obser on unwind(8)
A low tech SMS gateway for fun and no profit
Netflix and FreeBSD : Using Open Source to Deliver Streaming Video
powerd++ 0.4.0 release
Is it time to rewrite the operating system in Rust?
Small change, big effect
Swedish BSD Meetup, Feb 19, 2019
Polish BSD User Group Meetup, Feb 21, 2019
Feedback/Questions
Casey - Cool new Digital Ocean Feature
Morgan - Jail w/differnet version of FreeBSD
Brad - FreeBSD Installer
Send questions, comments, show ideas/topics, or stories you want mentioned on the show to [email protected]
1 hr
January 31, 2019283: Graphical Interface-View
We’re at FOSDEM 2019 this week having fun. We’d never leave you in a lurch, so we have recorded an interview with Niclas Zeising of the FreeBSD graphics team for you. Enjoy.
##Interview - Niclas Zeising - [email protected] / @niclaszeising
Interview topic: FreeBSD Graphics Stack
BR: Welcome Niclas. Since this is your first time on BSDNow, can you tell us a bit about yourself and how you started with Unix/BSD?
AJ: What made you start working in the FreeBSD graphics stack?
BR: What is the current status with the FreeBSD graphics stack?
AJ: What challenges do you face in the FreeBSD graphics stack?
BR: How many people are working in the graphics team and what kind of help do you need there?
AJ: You’re also involved in FreeBSD ports and held a poudriere tutorial at last years EuroBSDcon. What kind of feedback did you get and will you give that tutorial again?
BR: You’ve been organizing the Stockholm BSD user group meeting. Can you tell us a bit about that, what’s involved, how is it structured?
AJ: What conferences do you go to where people could talk to you?
BR: Is there anything else you’d like to mention before we let you go?
##Feedback/Questions
Casey - TrueOS
Troels - zfs send vs zfs send -R
matclarke - Orphaned packages
Send questions, comments, show ideas/topics, or stories you want mentioned on the show to [email protected]
47 min
January 24, 2019282: Open the Rsync
Project Trident 18.12 released, Spotifyd on NetBSD, OPNsense 18.7.10 is available, Ultra EPYC AMD Powered Sun Ultra 24 Workstation, OpenRsync, LLD porting to NetBSD, and more.
##Headlines
###AsiaBSDCon 2019 Call for Papers
You have until Jan 30th to submit
Full paper requirement is relaxed a bit this year (this year ONLY!) due to the short submission window. You don’t need all 10-12 pages, but it is still preferred.
Send a message to [email protected] with your proposal. Could be either for a talk or a tutorial.
Two days of tutorials/devsummit and two days of conference during Sakura season in Tokyo, Japan
The conference is also looking for sponsors
If accepted, flight and hotel is paid for by the conference
###Project Trident 18.12 Released
Twitter account if you want to keep up on project news
Screenshots
Project Trident Community Telegram Channel
DistroWatch Page
LinuxActionNews Review
RoboNuggie’s in depth review
###Building Spotifyd on NetBSD
These are the steps I went through to build and run Spotifyd (this commit at the time of writing) on NetBSD AMD64. It’s a Spotify Connect client so it means I still need to control Spotify from another device (typically my phone), but the audio is played through my desktop… which is where my speakers and headphones are plugged in - it means I don’t have to unplug stuff and re-plug into my phone, work laptop, etc. This is 100% a “good enough for now solution” for me; I have had a quick play with the Go based microcontroller from spotcontrol and that allows a completely NetBSD only experience (although it is just an example application so doesn’t provide many features - great as a basis to build on though).
##News Roundup
###OPNsense 18.7.10 released
2019 means 19.1 is almost here. In the meantime accept this small
incremental update with goodies such as Suricata 4.1, custom passwords
for P12 certificate export as well as fresh fixes in the FreeBSD base.
A lot of cleanups went into this update to make sure there will be a
smooth transition to 19.1-RC for you early birds. We expect RC1 in 1-2
weeks and the final 19.1 on January 29.
###Introducing the Ultra EPYC AMD Powered Sun Ultra 24 Workstation
A few weeks ago, I got an itch to build a workstation with AMD EPYC. There are a few constraints. First, I needed a higher-clock part. Second, I knew the whole build would be focused more on being an ultra high-end workstation rather than simply utilizing gaming components. With that, I decided it was time to hit on a bit of nostalgia for our readers. Mainly, I wanted to do an homage to Sun Microsystems. Sun made the server gear that the industry ran on for years, and as a fun fact, if you go behind the 1 Hacker Way sign at Facebook’s campus, they left the Sun Microsystems logo. Seeing that made me wonder if we could do an ultimate AMD EPYC build in a Sun Microsystems workstation.
###OpenRsync
This is a clean-room implementation of rsync with a BSD (ISC) license. It is designed to be compatible with a modern rsync (3.1.3 is used for testing). It currently compiles and runs only on OpenBSD.
This project is still very new and very fast-moving.
It’s not ready for wide-spread testing. Or even narrow-spread beyond getting all of the bits to work. It’s not ready for strong attention. Or really any attention but by careful programming.
Many have asked about portability. We’re just not there yet, folks. But don’t worry, the system is easily portable. The hard part for porters is matching OpenBSD’s pledge and unveil.
###The first report on LLD porting
LLD is the link editor (linker) component of Clang toolchain. Its main advantage over GNU ld is much lower memory footprint, and linking speed. It is of specific interest to me since currently 8 GiB of memory are insufficient to link LLVM statically (which is the upstream default).
The first goal of LLD porting is to ensure that LLD can produce working NetBSD executables, and be used to build LLVM itself. Then, it is desirable to look into trying to build additional NetBSD components, and eventually into replacing /usr/bin/ld entirely with lld.
In this report, I would like to shortly summarize the issues I have found so far trying to use LLD on NetBSD.
###Ring in the new
It’s the second week of 2019 already, which means I’m curious what Nate is going to do with his series This week in usability … reset the numbering from week 1? That series is a great read, to keep up with all the little things that change in KDE source each week — aside from the release notes.
For the big ticket items of KDE on FreeBSD, you should read this blog instead.
In ports this week (mostly KDE, some unrelated):
KDE Plasma has been updated to the latest release, 5.14.5.
KDE Applications 18.12.1 were released today, so we’re right on top of them.
Marble was fixed for FreeBSD-running-on-Power9.
Musescore caught up on 18 months of releases.
Phonon updated to 4.10.1, along with its backends.
And in development, Qt WebEngine 5.12 has been prepared in the incongruously-named plasma-5.13 branch in Area51; that does contain all the latest bits described above, as well.
##Beastie Bits
NomadBSD 1.2-RC1 Released
ZFS - The First Enterprise Blockchain
Powersaving with DragonFly laptop
NetBSD reaches 100% reproducable builds
Potential Bhyve Web Interface?
LibGDX proof of concept on OpenBSD - Video
LiteCLI is a user-friendly CommandLine client for SQLite database
In honor of Donald Knuth’s 81 birthday Stanford uploaded 111 lectures on Youtube
Portland BSD Pizza Night - 2018-01-31 19:00 - Sweet Heart Pizza
Stockholm BSD February meetup
Polish BSD User Group: Jan 25 18:15 - 21:00
AsiaBSDcon 2019 CfP
##Feedback/Questions
Greg - VLANs and jails
Tara - ZFS on removable disks
Casey - Interview with Kirk McKusick
Send questions, comments, show ideas/topics, or stories you want mentioned on the show to [email protected]
1 hr 2 min
January 17, 2019281: EPYC Server Battle
SCP client vulnerabilities, BSDs vs Linux benchmarks on a Tyan EPYC Server, fame for the Unix inventors, Die IPv4, GhostBSD 18.12 released, Unix in pictures, and more.
##Headlines
###scp client multiple vulnerabilities
Overview
SCP clients from multiple vendors are susceptible to a malicious scp server performing
unauthorized changes to target directory and/or client output manipulation.
Description
Many scp clients fail to verify if the objects returned by the scp server match those
it asked for. This issue dates back to 1983 and rcp, on which scp is based. A separate
flaw in the client allows the target directory attributes to be changed arbitrarily.
Finally, two vulnerabilities in clients may allow server to spoof the client output.
Impact
Malicious scp server can write arbitrary files to scp target directory, change the
target directory permissions and to spoof the client output.
Details
The discovered vulnerabilities, described in more detail below, enables the attack
described here in brief.
The attacker controlled server or Man-in-the-Middle(*) attack drops .bash_aliases file to victim’s home directory when the victim performs scp operation from the server. The transfer of extra files is hidden by sending ANSI control sequences via stderr. For example:
user@local:~$ scp user@remote:readme.txt .
readme.txt 100% 494 1.6KB/s 00:00
user@local:~$
Once the victim launches a new shell, the malicious commands in .bash_aliases get executed.
*) Man-in-the-Middle attack does require the victim to accept the wrong host fingerprint.
###FreeBSD 12.0 vs. DragonFlyBSD 5.4 vs. TrueOS 18.12 vs. Linux On A Tyan EPYC Server
Last month when running FreeBSD 12.0 benchmarks on a 2P EPYC server I wasn’t able to run any side-by-side benchmarks with the new DragonFlyBSD 5.4 as this BSD was crashing during the boot process on that board. But fortunately on another AMD EPYC server available, the EPYC 1P TYAN Transport SX TN70A-B8026, DragonFlyBSD 5.4.1 runs fine. So for this first round of BSD benchmarking in 2019 are tests of FreeBSD 11.2, FreeBSD 12.0, DragonFlyBSD 5.4.1, the new TrueOS 18.12, and a few Linux distributions (CentOS 7, Ubuntu 18.04.1 LTS, and Clear Linux) on this EPYC 7601 server in a variety of workloads.
DragonFlyBSD 5.4.1 ran fine on this Tyan server and could boot fine unlike the issue encountered on the Dell PowerEdge R7425 for this particular BSD. But on the Tyan server, DragonFlyBSD 5.2.2 wouldn’t boot so only this latest DragonFlyBSD release series was used as part of the comparison.
A summary of the operating systems tested for this EPYC 7601 OS benchmark comparison included:
DragonFlyBSD 5.4.1 - The latest release of Matthew Dillon’s operating system while using the HAMMER2 file-system and GCC 8.1 compiler that is now the default system compiler for this BSD.
FreeBSD 11.2 - The previous stable release of FreeBSD. Installed with a ZFS file-system.
FreeBSD 12.0 - The latest stable release of FreeBSD and installed with its ZFS option.
TrueOS 18.12 - The latest release of the iX systems’ FreeBSD derivative. TrueOS 18.12 is based on FreeBSD 13.0-CURRENT and uses ZFS by default and was using the Clang 7.0.1 compiler compared to Clang 6.0.1 on FreeBSD 12.0.
CentOS Linux 7 - The latest EL7 operating system performance.
Ubuntu 18.04.1 LTS - The latest Ubuntu Long Term Support release.
Clear Linux 27120 - The latest rolling release as of testing out of Intel’s Open-Source Technology Center. Clear Linux often reflects as close to the gold standard for performance as possible with its insanely tuned software stack for offering optimal performance on x86_64 performance for generally showing best what the hardware is capable of.
Throughout all of this testing, the Tyan 2U server was kept to its same configuration of an AMD EPYC 7601 (32 cores / 64 threads) at stock speeds, 8 x 16GB DDR4-2666 ECC memory, and 280GB Intel Optane 900p SSD benchmarks.
##News Roundup
###National Inventors Hall of Fame honors creators of Unix
Dennis Ritchie (Posthumous) and Ken Thompson: UNIX Operating System
Thompson and Ritchie’s creation of the UNIX operating system and the C programming language were pivotal developments in the progress of computer science. Today, 50 years after its beginnings, UNIX and UNIX-like systems continue to run machinery from supercomputers to smartphones. The UNIX operating system remains the basis of much of the world’s computing infrastructure, and C language – written to simplify the development of UNIX – is one of the most widely used languages today.
###Die IPV4, Die
Imagine, it is 2019. Easy, ha? Imagine, it is 2019 and you want to turn off IPv4. Like, off off. Really off. Not disabling IPv6, but disabling IPv4.
Two steps back
You might be coming here wondering, why would anybody want to do what we are asking to be done. Well, it is dead simple: We are running data centers (like Data Center Light) with a lot of IPv6 only equipment. There simply is no need for IPv4. So why would we want to have it enabled?
Also, here at ungleich, we defined 2019 as the year to move away from IPv4.
The challenge
Do you like puzzles? Competitions? Challenges? Hacking? Well. If ANY of this is of your interest, here is a real challenge for you:
We offer a 100 CHF (roughly 100 USD) for anyone who can give us a detailed description of how to turn IPv4 completely off in an operating system and allowing it to communicate with IPv6 only. This should obviously include a tiny proof that your operating system is really unable to use IPv4 at all. Just flushing IPv4 addresses and keeping the IPv4 stack loaded, does not count.
###GhostBSD 18.12 released
GhostBSD 18.12 is an updated iso of GhostBSD 18.10 with some little changes to the live DVD/USB and with updated packages.
What has changed since 18.10
removed default call of kernel modules for AMD and Intel
replaced octopkg by software-station
added back gop hacks to the live system
added ghostbsd-drivers and ghostbsd-utils
we updated the packages to the latest build
###And Now for a laugh : #unixinpictures
##Beastie Bits
We are now closer to the Y2038 bug than the Y2K bug
OpenBSD Enterprise use
AT&T Unix Books
Process title and missing memory space
The History of a Security Hole
unbound-adblock: The ultimate network adblocker!
FreeBSD’s name/value pairs library
Pid Rollover
Booting OpenBSD kernels in EFI mode with QEMU
OpenBSD CVS commit: Make mincore lie
BSDCan 2019 CfP ending January 19 - Submit!
OpenZFS User Conference - April 18-19
FreeBSD Journal is a free publication now
##Feedback/Questions
Chris - Boot environments and SSDs
Jonathan - Bytes issued during a zpool scrub
Bostjan - ZFS Record Size and my mistakes
Send questions, comments, show ideas/topics, or stories you want mentioned on the show to [email protected]
1 hr 24 min
January 10, 2019Episode 280: FOSS Clothing | BSD Now 280
A EULA in FOSS clothing, NetBSD with more LLVM support, Thoughts on FreeBSD 12.0, FreeBSD Performance against Windows and Linux on Xeon, Microsoft shipping NetBSD, and more.
Headlines
A EULA in FOSS clothing?
There was a tremendous amount of reaction to and discussion about my blog entry on the midlife crisis in open source. As part of this discussion on HN, Jay Kreps of Confluent took the time to write a detailed response — which he shortly thereafter elevated into a blog entry.
Let me be clear that I hold Jay in high regard, as both a software engineer and an entrepreneur — and I appreciate the time he took to write a thoughtful response. That said, there are aspects of his response that I found troubling enough to closely re-read the Confluent Community License — and that in turn has led me to a deeply disturbing realization about what is potentially going on here.
To GitHub: Assuming that this is in fact a EULA, I think it is perilous to allow EULAs to sit in public repositories. It’s one thing to have one click through to accept a license (though again, that itself is dubious), but to say that a git clone is an implicit acceptance of a contract that happens to be sitting somewhere in the repository beggars belief. With efforts like choosealicense.com, GitHub has been a model in guiding projects with respect to licensing; it would be helpful for GitHub’s counsel to weigh in on their view of this new strain of source-available proprietary software and the degree to which it comes into conflict with GitHub’s own terms of service.
To foundations concerned with software liberties, including the Apache Foundation, the Linux Foundation, the Free Software Foundation, the Electronic Frontier Foundation, the Open Source Initiative, and the Software Freedom Conservancy: the open source community needs your legal review on this! I don’t think I’m being too alarmist when I say that this is potentially a dangerous new precedent being set; it would be very helpful to have your lawyers offer their perspectives on this, even if they disagree with one another. We seem to be in some terrible new era of frankenlicenses, where the worst of proprietary licenses are bolted on to the goodwill created by open source licenses; we need your legal voices before these creatures destroy the village!
NetBSD and LLVM
NetBSD entering 2019 with more complete LLVM support
I’m recently helping the NetBSD developers to improve the support for this operating system in various LLVM components. As you can read in my previous report, I’ve been focusing on fixing build and test failures for the purpose of improving the buildbot coverage.
Previously, I’ve resolved test failures in LLVM, Clang, LLD, libunwind, openmp and partially libc++. During the remainder of the month, I’ve been working on the remaining libc++ test failures, improving the NetBSD clang driver and helping Kamil Rytarowski with compiler-rt.
The process of upstreaming support to LLVM sanitizers has been finalized
I’ve finished the process of upstreaming patches to LLVM sanitizers (almost 2000LOC of local code) and submitted to upstream new improvements for the NetBSD support. Today out of the box (in unpatched version) we have support for a variety of compiler-rt LLVM features: ASan (finds unauthorized memory access), UBSan (finds unspecified code semantics), TSan (finds threading bugs), MSan (finds uninitialized memory use), SafeStack (double stack hardening), Profile (code coverage), XRay (dynamic code tracing); while other ones such as Scudo (hardened allocator) or DFSan (generic data flow sanitizer) are not far away from completeness.
The NetBSD support is no longer visibly lacking behind Linux in sanitizers, although there are still failing tests on NetBSD that are not observed on Linux. On the other hand there are features working on NetBSD that are not functional on Linux, like sanitizing programs during early initialization process of OS (this is caused by /proc dependency on Linux that is mounted by startup programs, while NetBSD relies on sysctl(3) interfaces that is always available).
News Roundup
Thoughts on FreeBSD 12.0
Playing with FreeBSD with past week I don’t feel as though there were any big surprises or changes in this release compared to FreeBSD 11. In typical FreeBSD fashion, progress tends to be evolutionary rather than revolutionary, and this release feels like a polished and improved incremental step forward. I like that the installer handles both UFS and ZFS guided partitioning now and in a friendly manner. In the past I had trouble getting FreeBSD’s boot menu to work with boot environments, but that has been fixed for this release.
I like the security options in the installer too. These are not new, but I think worth mentioning. FreeBSD, unlike most Linux distributions, offers several low-level security options (like hiding other users’ processes and randomizing PIDs) and I like having these presented at install time. It’s harder for people to attack what they cannot see, or predict, and FreeBSD optionally makes these little adjustment for us.
Something which stands out about FreeBSD, compared to most Linux distributions I run, is that FreeBSD rarely holds the user’s hand, but also rarely surprises the user. This means there is more reading to do up front and new users may struggle to get used to editing configuration files in a text editor. But FreeBSD rarely does anything unless told to do it. Updates rarely change the system’s behaviour, working technology rarely gets swapped out for something new, the system and its applications never crashed during my trial. Everything was rock solid. The operating system may seem like a minimal, blank slate to new users, but it’s wonderfully dependable and predictable in my experience.
I probably wouldn’t recommend FreeBSD for desktop use. It’s close relative, GhostBSD, ships with a friendly desktop and does special work to make end user applications run smoothly. But for people who want to run servers, possible for years without change or issues, FreeBSD is a great option. It’s also an attractive choice, in my opinion, for people who like to build their system from the ground up, like you would with Debian’s server install or Arch Linux. Apart from the base tools and documentation, there is nothing on a FreeBSD system apart from what we put on it.
FreeBSD 12.0 Performance Against Windows & Linux On An Intel Xeon Server
Last week I posted benchmarks of Windows Server 2019 against various Linux distributions using a Tyan dual socket Intel Xeon server. In this article are some complementary results when adding in the performance of FreeBSD 11.2 against the new FreeBSD 12.0 stable release for this leading BSD operating system. As some fun benchmarks to end out 2018, here are the results of FreeBSD 11.2/12.0 (including an additional run when using GCC rather than Clang) up against Windows Server and several enterprise-ready Linux distributions.
While FreeBSD 12.0 had picked up just one win of the Windows/Linux comparisons run, the FreeBSD performance is moving in the right direction. FreeBSD 12.0 was certainly faster than FreeBSD 11.2 on this dual Intel Xeon Scalable server based on a Tyan 1U platform. Meanwhile, to no surprise given the data last week, Clear Linux was by far the fastest out-of-the-box operating system tested.
I did run some extra benchmarks on FreeBSD 11.2/12.0 with this hardware: in total I ran 120 benchmarks for these BSD tests. Of the 120 tests, there were just 15 cases where FreeBSD 11.2 was faster than 12.0. Seeing FreeBSD 12.0 faster than 11.2 nearly 90% of the time is an accomplishment and usually with other operating systems we see more of a mixed bag on new releases with not such solidly better performance. It was also great seeing the competitive performance out of FreeBSD when using the Clang compiler for the source-based tests compared to the GCC8 performance. Additional data available via this OpenBenchmarking.org result file.
How NetBSD came to be shipped by Microsoft
Google cache in case the site is down
In 2000, Joe Britt, Matt Hershenson and Andy Rubin formed Danger Incorporated. Danger developed the world’s first recognizable smartphone, the Danger HipTop. T-Mobile sold the first HipTop under the brand name Sidekick in October of 2002.
Danger had a well developed kernel that had been designed and built in house. The kernel came to be viewed as not a core intellectual property and Danger started a search for a replacement. For business reasons, mostly to do with legal concerns over the Gnu Public License, Danger rejected Linux and began to consider BSD Unix as a replacement for the kernel.
In 2006 I was hired by Mike Chen, the manager of the kernel development group to investigate the feasibility of replacing the Danger kernel with a BSD kernel, to select the version of BSD to use, to develop a prototype and to develop the plan for adapting BSD to Danger’s requirements.
NetBSD was easily the best choice among the BSD variations at the time because it had well developed cross development tools. It was easy to use a NetBSD desktop running an Intel release to cross compile a NetBSD kernel and runtime for a device running an ARM processor. (Those interested in mailing list archaeology might be amused to investigate NetBSD technical mailing list for mail from picovex, particularly from Bucky Katz at picovex.)
We began product development on the specific prototype of the phone that would become the Sidekick LX2009 in 2007 and contracts for the phone were written with T-Mobile. We were about half way through the two year development cycle when Microsoft purchased Danger in 2008.
Microsoft would have preferred to ship the Sidekick running Windows/CE rather than NetBSD, but a schedule analysis performed by me, and another by an independent outside contractor, indicated that doing so would result in unacceptable delay.
Beastie Bits
Unleashed 1.2 Released

35th CCC - Taming the Chaos: Can we build systems that actually work?

Potholes to avoid when migrating to IPv6

XScreenSaver 5.42

SSH Examples and Tunnels

Help request - mbuf(9) - request for comment

NSA to release free Reverse Engineering Tool

Running FreeBSD on a Raspberry Pi3 using a custom image created with crochet and poudriere
Feedback/Questions
Dries - Lets talk a bit about VIMAGE jails

ohb - Question About ZFS Root Dataset

Micah - Active-Active NAS Sync recommendations
Send questions, comments, show ideas/topics, or stories you want mentioned on the show to [email protected]
53 min
January 03, 2019Episode 279: Future of ZFS | BSD Now 279
The future of ZFS in FreeBSD, we pick highlights from the FreeBSD quarterly status report, flying with the raven, modern KDE on FreeBSD, many ways to launch FreeBSD in EC2, GOG installers on NetBSD, and more.
Headlines
The future of ZFS in FreeBSD
The sources for FreeBSD’s ZFS support are currently taken directly from Illumos with local ifdefs to support the peculiarities of FreeBSD where the Solaris Portability Layer (SPL) shims fall short. FreeBSD has regularly pulled changes from Illumos and tried to push back any bug fixes and new features done in the context of FreeBSD. In the past few years the vast majority of new development in ZFS has taken place in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced that they will be moving to ZoL: https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means that there will be little to no net new development of Illumos. While working through the git history of ZoL I have also discovered that many races and locking bugs have been fixed in ZoL and never made it back to Illumos and thus FreeBSD. This state of affairs has led to a general agreement among the stakeholders that I have spoken to that it makes sense to rebase FreeBSD’s ZFS on ZoL. Brian Behlendorf has graciously encouraged me to add FreeBSD support directly to ZoL https://github.com/zfsonfreebsd/ZoF so that we might all have a single shared code base.
A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port Before it can be committed some additional functionality needs to be added to the FreeBSD opencrypto framework. These can be found at https://reviews.freebsd.org/D18520
This port will provide FreeBSD users with multi modifier protection, project quotas, encrypted datasets, allocation classes, vectorized raidz, vectorized checksums, and various command line improvements.
FreeBSD Quarterly Status Update
With FreeBSD having gone all the way to 12, it is perhaps useful to take a look back at all the things that have been accomplished, in terms of many visible changes, as well as all the things that happen behind the scenes to ensure that FreeBSD continues to offer an alternative in both design, implementation, and execution.
The things you can look forward to reading about are too numerous to summarize, but cover just about everything from finalizing releases, administrative work, optimizations and depessimizations, features added and fixed, and many areas of improvement that might just surprise you a little.
Please have a cup of coffee, tea, hot cocoa, or other beverage of choice, and enjoy this culmulative set of reports covering everything that’s been done since October, 2017.
—Daniel Ebdrup
News Roundup
One year of flying with the Raven: Ready for the Desktop?
It has been a little over one year now that I’m with the Ravenports project. Time to reflect my involvement, my expectations and hopes.
Ravenports
Ravenports is a universal packaging framework for *nix operating systems. For the user it provides easy access to binary packages of common software for multiple platforms. It has been the long-lasting champion on Repology’s top 10 repositories regarding package freshness (rarely dropping below 96 percent while all other projects keep below 90!).
For the porter it offers a well-designed and elegant means of writing cross-platform buildsheets that allow building the same version of the software with (completely or mostly) the same compile-time configuration on different operating systems or distributions.
And for the developer it means a real-world project that’s written in modern Ada (ravenadm) and C (pkg) – as well as some Perl for support scripts and make. Things feel very optimized and fast. Not being a programmer though, I cannot really say anything about the actual code and thus leave it to the interested reader’s judgement.
Modern KDE on FreeBSD
New stuff in the official FreeBSD repositories! The X11 team has landed a newer version of libinput, opening up the way for KDE Plasma 5.14 in ports. That’s a pretty big update and it may frighten people with a new wallpaper.
What this means is that the graphical stack is once again on-par with what Plasma upstream expects, and we can get back to chasing releases as soon as they happen, rather than gnashing our teeth at missing dependencies. The KDE-FreeBSD CI servers are in the process of being upgraded to 12-STABLE, and we’re integrating with the new experimental CI systems as well. This means we are chasing sensibly-modern systems (13-CURRENT is out of scope).
The many ways to launch FreeBSD in EC2
Talking to FreeBSD users recently, I became aware that while I’ve created a lot of tools, I haven’t done a very good job of explaining how, and more importantly when to use them. So for all of the EC2-curious FreeBSD users out there: Here are the many ways to launch and configure FreeBSD in EC2 — ranging from the simplest to the most complicated (but most powerful):
Launch FreeBSD and SSH in

Launch FreeBSD and provide user-data

Use the AMI Builder to create a customized FreeBSD AMI

Build a FreeBSD AMI from a modified FreeBSD source tree

Build your own disk image
I hope I’ve provided tools which help you to run FreeBSD in EC2, no matter how common or unusual your needs are. If you find my work useful, please consider supporting my work in this area; while this is both something I enjoy working on and something which is useful for my day job (Tarsnap, my online backup service), having support would make it easier for me to prioritize FreeBSD/EC2 issues over other projects.
Using the GOG.com installers for Linux, on NetBSD
GOG.com prefers that you use their GOG Galaxy desktop app to download, install and manage all of your GOG games. But customers always have the option to install the game on their own terms, with a platform-specific installer.
GOG offers these installers for Mac, Windows and/or Linux, depending on which platforms the game is available for.
The installers truly are platform-specific:

macOS games are distributed in a standard .pkg

Windows games are distributed in a setup wizard .exe

Linux games are distributed in a goofy shell archive
Of course, none of those are NetBSD. So, if I wanted to even attempt to play a game distributed by GOG.com on NetBSD, which one should I pick? The obvious choice is the Linux installer, since Linux is the most similar to NetBSD, right? Au contraire! In practice, I found that it is easier to download the Windows installer.
Here’s what I mean. For example, I ported the open source version of Aquaria to pkgsrc, but that package is only the game’s engine, not the multimedia data. The multimedia data is still copyrighted. Therefore, you need to get it from somewhere else. GOG is usually a good choice, because they distribute their games without DRM. And as mentioned earlier, picking the Linux installer seemed like a natural choice.
Now, actually PLAYING the games on NetBSD is a separate matter entirely. The game I’ve got here, though, my current obsession Pyre, is built with MonoGame and therefore could theoretically work on NetBSD, too, with the help of a library called FNA and a script for OpenBSD called fnaify. I do hope to create a pkgsrc package for FNA and port the fnaify script to NetBSD at some point.
Beastie Bits
Software as a Reflection of Values With Bryan Cantrill

Collection of bmc talks, updated 2018

wump: incorrect wumpus movement probability

Debugging Rust with VSCode on FreeBSD

SMB/CIFS on FreeBSD

BSD Tattoo

pkgsrc-2018Q4 branch announcement

toying with wireguard on openbsd

new USB audio class v2.0 driver

Todd Mortimer Removing ROP Gadgets from OpenBSD EuroBSDCon 2018

OpenBSD 6.5 release page is online

shell access to historical Unix versions in your browser
Feedback/Questions
Brad - ZFS Features and Upgrades

Andre - Splitting ZFS array

Michael - Priority/nice value for Jails?
Send questions, comments, show ideas/topics, or stories you want mentioned on the show to [email protected]
1 hr 34 min
December 27, 2018Episode 278: The Real McCoy | BSD Now 278
We sat down at BSDCan 2018 to interview Kirk McKusick about various topics ranging about the early years of Berkeley Unix, his continuing work on UFS, the governance of FreeBSD, and more.
##Interview - Kirk McKusick - [email protected]
25 years of FreeBSD
How Kirk got started in BSD, at the very beginning
Predicting the Future
How the code and community grew
The leadership of the project, and how it changed over time
UFS over the years (reading disks from 1982 in 2018)
Conferences
The rise and fall of Linux
The resurgence of FreeBSD
We want to extend a big thank you to the entire BSD community for making this show possible, and to all of our viewers for watching and providing the feedback that makes this show successful. We wish you all a happy and prosperous new year, and we’ll see you next week.
Send questions, comments, show ideas/topics, or stories you want mentioned on the show to [email protected]
50 min
December 24, 2018Episode 277: Nmap Level Up | BSD Now 277
The Open Source midlife crisis, Donald Knuth The Yoda of Silicon Valley, Certbot For OpenBSD's httpd, how to upgrade FreeBSD from 11 to 12, level up your nmap game, NetBSD desktop, and more.
##Headlines
###Open Source Confronts its midlife crisis
Midlife is tough: the idealism of youth has faded, as has inevitably some of its fitness and vigor. At the same time, the responsibilities of adulthood have grown. Making things more challenging, while you are navigating the turbulence of teenagers, your own parents are likely entering life’s twilight, needing help in new ways from their adult children. By midlife, in addition to the singular joys of life, you have also likely experienced its terrible sorrows: death, heartbreak, betrayal. Taken together, the fading of youth, the growth in responsibility and the endurance of misfortune can lead to cynicism or (worse) drastic and poorly thought-out choices. Add in a little fear of mortality and some existential dread, and you have the stuff of which midlife crises are made…
I raise this not because of my own adventures at midlife, but because it is clear to me that open source — now several decades old and fully adult — is going through its own midlife crisis. This has long been in the making: for years, I (and others) have been critical of service providers’ parasitic relationship with open source, as cloud service providers turn open source software into a service offering without giving back to the communities upon which they implicitly depend. At the same time, open source has been (rightfully) entirely unsympathetic to the proprietary software models that have been burned to the ground — but also seemingly oblivious as to the larger economic waves that have buoyed them.
So it seemed like only a matter of time before the companies built around open source software would have to confront their own crisis of confidence: open source business models are really tough, selling software-as-a-service is one of the most natural of them, the cloud service providers are really good at it — and their commercial appetites seem boundless. And, like a new cherry red two-seater sports car next to a minivan in a suburban driveway, some open source companies are dealing with this crisis exceptionally poorly: they are trying to restrict the way that their open source software can be used. These companies want it both ways: they want the advantages of open source — the community, the positivity, the energy, the adoption, the downloads — but they also want to enjoy the fruits of proprietary software companies in software lock-in and its monopolistic rents. If this were entirely transparent (that is, if some bits were merely being made explicitly proprietary), it would be fine: we could accept these companies as essentially proprietary software companies, albeit with an open source loss-leader. But instead, these companies are trying to license their way into this self-contradictory world: continuing to claim to be entirely open source, but perverting the license under which portions of that source are available. Most gallingly, they are doing this by hijacking open source nomenclature. Of these, the laughably named commons clause is the worst offender (it is plainly designed to be confused with the purely virtuous creative commons), but others (including CockroachDB’s Community License, MongoDB’s Server Side Public License, and Confluent’s Community License) are little better. And in particular, as it apparently needs to be said: no, “community” is not the opposite of “open source” — please stop sullying its good name by attaching it to licenses that are deliberately not open source! But even if they were more aptly named (e.g. “the restricted clause” or “the controlled use license” or — perhaps most honest of all — “the please-don’t-put-me-out-of-business-during-the-next-reInvent-keynote clause”), these licenses suffer from a serious problem: they are almost certainly asserting rights that the copyright holder doesn’t in fact have.
If I sell you a book that I wrote, I can restrict your right to read it aloud for an audience, or sell a translation, or write a sequel; these restrictions are rights afforded the copyright holder. I cannot, however, tell you that you can’t put the book on the same bookshelf as that of my rival, or that you can’t read the book while flying a particular airline I dislike, or that you aren’t allowed to read the book and also work for a company that competes with mine. (Lest you think that last example absurd, that’s almost verbatim the language in the new Confluent Community (sic) License.) I personally think that none of these licenses would withstand a court challenge, but I also don’t think it will come to that: because the vendors behind these licenses will surely fear that they wouldn’t survive litigation, they will deliberately avoid inviting such challenges. In some ways, this netherworld is even worse, as the license becomes a vessel for unverifiable fear of arbitrary liability.
let me put this to you as directly as possible: cloud services providers are emphatically not going to license your proprietary software. I mean, you knew that, right? The whole premise with your proprietary license is that you are finding that there is no way to compete with the operational dominance of the cloud services providers; did you really believe that those same dominant cloud services providers can’t simply reimplement your LDAP integration or whatever? The cloud services providers are currently reproprietarizing all of computing — they are making their own CPUs for crying out loud! — reimplementing the bits of your software that they need in the name of the service that their customers want (and will pay for!) won’t even move the needle in terms of their effort.
Worse than all of this (and the reason why this madness needs to stop): licenses that are vague with respect to permitted use are corporate toxin. Any company that has been through an acquisition can speak of the peril of the due diligence license audit: the acquiring entity is almost always deep pocketed and (not unrelatedly) risk averse; the last thing that any company wants is for a deal to go sideways because of concern over unbounded liability to some third-party knuckle-head. So companies that engage in license tomfoolery are doing worse than merely not solving their own problem: they are potentially poisoning the wellspring of their own community.
in the end, open source will survive its midlife questioning just as people in midlife get through theirs: by returning to its core values and by finding rejuvenation in its communities. Indeed, we can all find solace in the fact that while life is finite, our values and our communities survive us — and that our engagement with them is our most important legacy.
See the article for the rest
###Donald Knuth - The Yoda of Silicon Valley
For half a century, the Stanford computer scientist Donald Knuth, who bears a slight resemblance to Yoda — albeit standing 6-foot-4 and wearing glasses — has reigned as the spirit-guide of the algorithmic realm.
He is the author of “The Art of Computer Programming,” a continuing four-volume opus that is his life’s work. The first volume debuted in 1968, and the collected volumes (sold as a boxed set for about $250) were included by American Scientist in 2013 on its list of books that shaped the last century of science — alongside a special edition of “The Autobiography of Charles Darwin,” Tom Wolfe’s “The Right Stuff,” Rachel Carson’s “Silent Spring” and monographs by Albert Einstein, John von Neumann and Richard Feynman.
With more than one million copies in print, “The Art of Computer Programming” is the Bible of its field. “Like an actual bible, it is long and comprehensive; no other book is as comprehensive,” said Peter Norvig, a director of research at Google. After 652 pages, volume one closes with a blurb on the back cover from Bill Gates: “You should definitely send me a résumé if you can read the whole thing.”
The volume opens with an excerpt from “McCall’s Cookbook”:
Here is your book, the one your thousands of letters have asked us to publish. It has taken us years to do, checking and rechecking countless recipes to bring you only the best, only the interesting, only the perfect.
Inside are algorithms, the recipes that feed the digital age — although, as Dr. Knuth likes to point out, algorithms can also be found on Babylonian tablets from 3,800 years ago. He is an esteemed algorithmist; his name is attached to some of the field’s most important specimens, such as the Knuth-Morris-Pratt string-searching algorithm. Devised in 1970, it finds all occurrences of a given word or pattern of letters in a text — for instance, when you hit Command+F to search for a keyword in a document.
Now 80, Dr. Knuth usually dresses like the youthful geek he was when he embarked on this odyssey: long-sleeved T-shirt under a short-sleeved T-shirt, with jeans, at least at this time of year. In those early days, he worked close to the machine, writing “in the raw,” tinkering with the zeros and ones.
See the article for the rest
##News Roundup
###Let’s Encrypt: Certbot For OpenBSD’s httpd
Intro
Let’s Encrypt is “a free, automated, and open Certificate Authority”.
Certbot is “an easy-to-use automatic client that fetches and deploys SSL/TLS certificates for your web server”, well known as “the official Let’s Encrypt client”.
I remember well how excited I felt when I read Let’s Encrypt’s “Our First Certificate Is Now Live” in 2015.
How wonderful the goal of them is; it’s to “give people the digital certificates they need in order to enable HTTPS (SSL/TLS) for websites, for free” “to create a more secure and privacy-respecting Web”!
Since this year, they have begun to support even ACME v2 and Wildcard Certificate!
Well, in OpenBSD as well as other operating systems, it’s easy and comfortable to have their big help 😊
Environment
OS: OpenBSD 6.4 amd64
Web Server: OpenBSD’s httpd
Certification: Let’s Encrypt with Certbot 0.27
Reference: OpenBSD’s httpd
###FreeBSD 12 released: Here is how to upgrade FreeBSD 11 to 12
The FreeBSD project announces the availability of FreeBSD 12.0-RELEASE. It is the first release of the stable/12 branch. The new version comes with updated software and features for a wild variety of architectures. The latest release provides performance improvements and better support for FreeBSD jails and more. One can benefit greatly using an upgraded version of FreeBSD.
FreeBSD 12.0 supports amd64, i386, powerpc, powerpc64, powerpcspe, sparc64, armv6, armv7, and aarch64 architectures. One can run it on a standalone server or desktop system. Another option is to run it on Raspberry PI computer. FreeBSD 12 also runs on popular cloud service providers such as AWS EC2/Lightsail or Google compute VM.
New features and highlights:
OpenSSL version 1.1.1a (LTS)
OpenSSH server 7.8p1
Unbound server 1.8.1
Clang and co 6.0.1
The FreeBSD installer supports EFI+GELI as an installation option
VIMAGE FreeBSD kernel configuration option has been enabled by default. VIMAGE was the main reason I custom compiled FreeBSD for the last few years. No more custom compile for me.
Graphics drivers for modern ATI/AMD and Intel graphics cards are now available in the FreeBSD ports collection
ZFS has been updated to include new sysctl(s), vfs.zfs.arc_min_prefetch_ms and vfs.zfs.arc_min_prescient_prefetch_ms, which improve performance of the zpool scrub subcommand
The pf packet filter is now usable within a jail using vnet
KDE updated to version 5.12.5
The NFS version 4.1 includes pNFS server support
Perl 5.26.2
The default PAGER now defaults to less for most commands
The dd utility has been updated to add the status=progress option to match GNU/Linux dd command to show progress bar while running dd
FreeBSD now supports ext4 for read/write operation
Python 2.7
much more
###Six Ways to Level Up Your nmap Game
nmap is a network exploration tool and security / port scanner.
If you’ve heard of it, and you’re like me, you’ve most likely used it like this:
ie, you’ve pointed it at an IP address and observed the output which tells you the open ports on a host.
I used nmap like this for years, but only recently grokked the manual to see what else it could do. Here’s a quick look and some of the more useful things I found out.
Scan a Network
Scan All Ports
Get service versions
Use -A for more data
Find out what nmap is up to
Script your own scans with NSE
###[NetBSD Desktop]
Part 1: Manual NetBSD installation on GPT/UEFI
NetBSD desktop pt.2: Set up wireless networking on NetBSD with wpa_supplicant and dhcpcd
Part 3: Simple stateful firewall with NPF
Part 4: 4: The X Display Manager (XDM)
Part 5: automounting with Berkeley am-utils
##Beastie Bits
Call For Testing: ZFS on FreeBSD Project
DragonFlyBSD 5.4.1 release within a week
You Can’t Opt Out of the Patent System. That’s Why Patent Pandas Was Created!
Announcing Yggdrasil Network v0.3
OpenBSD Network Engineer Job listing
FreeBSD 12.0 Stable Version Released!
LibreSSL 2.9.0 released
Live stream test: Sgi Octane light bar repair / soldering!
Configure a FreeBSD Email Server Using Postfix, Dovecot, MySQL, DAVICAL and SpamAssassin
Berkeley smorgasbord
FOSDEM BSD Devroom schedule
##Feedback/Questions
Warren - Ep.273: OpenZFS on OS X
cogoman - tarsnap security and using SSDs in raid
Andrew - Portland BSD Pizza Night
Send questions, comments, show ideas/topics, or stories you want mentioned on the show to [email protected]
1 hr 17 min
December 13, 2018Episode 276: Ho, Ho, Ho - 12.0 | BSD Now 276
FreeBSD 12.0 is finally here, partly-cloudy IPsec VPN, KLEAK with NetBSD, How to create synth repos, GhostBSD author interview, and more.
##Headlines
###FreeBSD 12.0 is available
After a long release cycle, the wait is over: FreeBSD 12.0 is now officially available.
We’ve picked a few interesting things to cover in the show, make sure to read the full Release Notes
Userland:
Group permissions on /dev/acpi have been changed to allow users in the operator GID to invoke acpiconf(8) to suspend the system.
The default devfs.rules(5) configuration has been updated to allow mount_fusefs(8) with jail(8).
The default PAGER now defaults to less(1) for most commands.
The newsyslog(8) utility has been updated to reject configuration entries that specify setuid(2) or executable log files.
The WITH_REPRODUCIBLE_BUILD src.conf(5) knob has been enabled by default.
A new src.conf(5) knob, WITH_RETPOLINE, has been added to enable the retpoline mitigation for userland builds.
Userland applications:
The dtrace(1) utility has been updated to support if and else statements.
The legacy gdb(1) utility included in the base system is now installed to /usr/libexec for use with crashinfo(8). The gdbserver and gdbtui utilities are no longer installed. For interactive debugging, lldb(1) or a modern version of gdb(1) from devel/gdb should be used. A new src.conf(5) knob, WITHOUT_GDB_LIBEXEC has been added to disable building gdb(1). The gdb(1) utility is still installed in /usr/bin on sparc64.
The setfacl(1) utility has been updated to include a new flag, -R, used to operate recursively on directories.
The geli(8) utility has been updated to provide support for initializing multiple providers at once when they use the same passphrase and/or key.
The dd(1) utility has been updated to add the status=progress option, which prints the status of its operation on a single line once per second, similar to GNU dd(1).
The date(1) utility has been updated to include a new flag, -I, which prints its output in ISO 8601 formatting.
The bectl(8) utility has been added, providing an administrative interface for managing ZFS boot environments, similar to sysutils/beadm.
The bhyve(8) utility has been updated to add a new subcommand to the -l and -s flags, help, which when used, prints a list of supported LPC and PCI devices, respectively.
The tftp(1) utility has been updated to change the default transfer mode from ASCII to binary.
The chown(8) utility has been updated to prevent overflow of UID or GID arguments where the argument exceeded UID_MAX or GID_MAX, respectively.
Kernel:
The ACPI subsystem has been updated to implement Device object types for ACPI 6.0 support, required for some Dell, Inc. Poweredge™ AMD® Epyc™ systems.
The amdsmn(4) and amdtemp(4) drivers have been updated to attach to AMD® Ryzen 2™ host bridges.
The amdtemp(4) driver has been updated to fix temperature reporting for AMD® 2990WX CPUs.
Kernel Configuration:
The VIMAGE kernel configuration option has been enabled by default.
The dumpon(8) utility has been updated to add support for compressed kernel crash dumps when the kernel configuration file includes the GZIO option. See rc.conf(5) and dumpon(8) for additional information.
The NUMA option has been enabled by default in the amd64 GENERIC and MINIMAL kernel configurations.
Device Drivers:
The random(4) driver has been updated to remove the Yarrow algorithm. The Fortuna algorithm remains the default, and now only, available algorithm.
The vt(4) driver has been updated with performance improvements, drawing text at rates ranging from 2- to 6-times faster.
Deprecated Drivers:
The lmc(4) driver has been removed.
The ixgb(4) driver has been removed.
The nxge(4) driver has been removed.
The vxge(4) driver has been removed.
The jedec_ts(4) driver has been removed in 12.0-RELEASE, and its functionality replaced by jedec_dimm(4).
The DRM driver for modern graphics chipsets has been marked deprecated and marked for removal in FreeBSD 13. The DRM kernel modules are available from graphics/drm-stable-kmod or graphics/drm-legacy-kmod in the Ports Collection as well as via pkg(8). Additionally, the kernel modules have been added to the lua loader.conf(5) module_blacklist, as installation from the Ports Collection or pkg(8) is strongly recommended.
The following drivers have been deprecated in FreeBSD 12.0, and not present in FreeBSD 13.0: ae(4), de(4), ed(4), ep(4), ex(4), fe(4), pcn(4), sf(4), sn(4), tl(4), tx(4), txp(4), vx(4), wb(4), xe(4)
Storage:
The UFS/FFS filesystem has been updated to support check hashes to cylinder-group maps. Support for check hashes is available only for UFS2.
The UFS/FFS filesystem has been updated to consolidate TRIM/BIO_DELETE commands, reducing read/write requests due to fewer TRIM messages being sent simultaneously.
TRIM consolidation support has been enabled by default in the UFS/FFS filesystem. TRIM consolidation can be disabled by setting the vfs.ffs.dotrimcons sysctl(8) to 0, or adding vfs.ffs.dotrimcons=0 to sysctl.conf(5).
NFS:
The NFS version 4.1 server has been updated to include pNFS server support.
ZFS:
ZFS has been updated to include new sysctl(8)s, vfs.zfs.arc_min_prefetch_ms and vfs.zfs.arc_min_prescient_prefetch_ms, which improve performance of the zpool(8) scrub subcommand.
The new spacemap_v2 zpool feature has been added. This provides more efficient encoding of spacemaps, especially for full vdev spacemaps.
The large_dnode zpool feature been imported, allowing better compatibility with pools created under ZFS-on-Linux 0.7.x
Many bug fixes have been applied to the device removal feature. This feature allows you to remove a non-redundant or mirror vdev from a pool by relocating its data to other vdevs.
Includes the fix for PR 229614 that could cause processes to hang in zil_commit()
Boot Loader Changes:
The lua loader(8) has been updated to detect a list of installed kernels to boot.
The loader(8) has been updated to support geli(8) for all architectures and all disk-like devices.
The loader(8) has been updated to add support for loading Intel® microcode updates early during the boot process.
Networking:
The pf(4) packet filter is now usable within a jail(8) using vnet(9).
The pf(4) packet filter has been updated to use rmlock(9) instead of rwlock(9), resulting in significant performance improvements.
The SO_REUSEPORT_LB option has been added to the network stack, allowing multiple programs or threads to bind to the same port, and incoming connections load balanced using a hash function.
Again, read the release notes for a full list, check out the errata notices. A big THANKS to the entire release engineering team and all developers involved in the release, much appreciated!
###Abandon Linux. Move to FreeBSD or Illumos
If you use GNU/Linux and you are only on opensource, you may be doing it wrong. Here’s why.
Is your company based on opensource based software only? Do you have a bunch of developers hitting some kind of server you have installed for them to “do their thing”? Being it for economical reasons (remember to donate), being it for philosophycal ones, you may have skipped good alternatives. The BSD’s and Illumos.
I bet you are running some sort of Debian, openSuSE or CentOS. It’s very discouraging having entered into the IT field recently and discover many of the people you meet do not even recognise the name BSD. Naming Solaris seems like naming the evil itself. The problem being many do not know why. They can’t point anything specific other than it’s fading out. This has recently shown strong when Oracle officials have stated development for new features has ceased and almost 90 % of developers for Solaris have been layed off. AIX seems alien to almost everybody unless you have a white beard. And all this is silly.
And here’s why. You are certainly missing two important features that FreeBSD and Illumos derivatives are enjoying. A full virtualization technology, much better and fully developed compared to the LXC containers in the Linux world, such as Jails on BSD, Zones in Solaris/Illumos, and the great ZFS file system which both share.
You have probably heard of a new Linux filesystem named Btrfs, which by the way, development has been dropped from the Red Hat side. Trying to emulate ZFS, Oracle started developing Btrfs file system before they acquired Sun (the original developer of ZFS), and SuSE joined the effort as well as Red Hat. It is not as well developed as ZFS and it hasn’t been tested in production environments as extensively as the former has. That leaves some uncertainty on using it or not. Red Hat leaving it aside does add some more. Although some organizations have used it with various grades of success.
But why is this anyhow interesting for a sysadmin or any organization? Well… FreeBSD (descendant of Berkeley UNIX) and SmartOS (based on Illumos) aglutinate some features that make administration easier, safer, faster and more reliable. The dream of any systems administrator.
To start, the ZFS filesystem combines the typical filesystem with a volume manager. It includes protection against corruption, snapshots and copy-on-write clones, as well as volume manager.
Jails is another interesting piece of technology. Linux folks usually associate this as a sort of chroot. It isn’t. It is somehow inspired by it but as you may know you can escape from a chroot environment with a blink of an eye. Jails are not called jails casually. The name has a purpose. Contain processes and programs within a defined and totally controlled environment. Jails appeared first in FreeBSD in the year 2000. Solaris Zones debuted on 2005 (now called containers) are the now proprietary version of those.
There are some other technologies on Linux such as Btrfs or Docker. But they have some caveats. Btrfs hasn’t been fully developed yet and it’s hasn’t been proved as much in production environments as ZFS has. And some problems have arisen recently although the developers are pushing the envelope. At some time they will match ZFS capabilities for sure. Docker is growing exponentially and it’s one of the cool technologies of modern times. The caveat is, as before, the development of this technology hasn’t been fully developed. Unlike other virtualization technologies this is not a kernel playing on top of another kernel. This is virtualization at the OS level, meaning differentiated environments can coexist on a single host, “hitting” the same unique kernel which controls and shares the resources. The problem comes when you put Docker on top of any other virtualization technology such as KVM or Xen. It breaks the purpose of it and has a performance penalty.
I have arrived into the IT field with very little knowledge, that is true. But what I see strikes me. Working in a bank has allowed me to see a big production environment that needs the highest of the availability and reliability. This is, sometimes, achieved by bruteforce. And it’s legitime and adequate. Redundancy has a reason and a purpose for example. But some other times it looks, it feels, like killing flies with cannons. More hardware, more virtual machines, more people, more of this, more of that. They can afford it, so they try to maintain the cost low but at the end of the day there is a chunky budget to back operations.
But here comes reality. You’re not a bank and you need to squeeze your investment as much as possible. By using FreeBSD jails you can avoid the performance penalty of KVM or Xen virtualization. Do you use VMWare or Hyper-V? You can avoid both and gain in performance. Not only that, control and manageability are equal as before, and sometimes easier to administer. There are four ways to operate them which can be divided in two categories. Hardcore and Human Being. For the Hardcore use the FreeBSD handbook and investigate as much as you can. For the Human Being way there are three options to use. Ezjail, Iocage and CBSD which are frameworks or programs as you may call to manage jails. I personally use Iocage but I have also used Ezjail.
How can you use jails on your benefit? Ever tried to configure some new software and failed miserably? You can have three different jails running at the same time with different configurations. Want to try a new configuration in a production piece of hardware without applying it on the final users? You can do that with a small jail while the production environment is on in another bigger, chunkier jail.
Want to divide the hardware as a replica of the division of the team/s you are working with? Want to sell virtual machines with bare metal performance? Do you want to isolate some piece of critical software or even data in a more controlled environment? Do you have different clients and you want to use the same hardware but you want to avoid them seeing each other at the same time you maintain performance and reliability?
Are you a developer and you have to have reliable and portable snapshots of your work? Do you want to try new options-designs without breaking your previous work, in a timeless fashion? You can work on something, clone the jail and apply the new ideas on the project in a matter of seconds. You can stop there, export the filesystem snapshot containing all the environment and all your work and place it on a thumbdrive to later import it on a big production system. Want to change that image properties such as the network stack interface and ip? This is just one command away from you.
But what properties can you assign to a jail and how can I manage them you may be wondering. Hostname, disk quota, i/o, memory, cpu limits, network isolation, network virtualization, snapshots and the manage of those, migration and root privilege isolation to name a few. You can also clone them and import and export them between different systems. Some of these things because of ZFS. Iocage is a python program to manage jails and it takes profit from ZFS advantages.
But FreeBSD is not Linux you may say. No it is not. There are no run levels. The systemd factor is out of this equation. This is so since the begginning. Ever wondered where did vi come from? The TCP/IP stack? Your beloved macOS from Apple? All this is coming from the FreeBSD project. If you are used to Linux your adaptation period with any BSD will be short, very short. You will almost feel at home. Used to packaged software using yum or apt-get? No worries. With pkgng, the package management tool used in FreeBSD has almost 27.000 compiled packages for you to use. Almost all software found on any of the important GNU/Linux distros can be found here. Java, Python, C, C++, Clang, GCC, Javascript frameworks, Ruby, PHP, MySQL and the major forks, etc. All this opensource software, and much more, is available at your fingertips.
I am a developer and… frankly my time is money and I appreciate both much more than dealing with systems configuration, etc. You can set a VM using VMWare or VirtualBox and play with barebones FreeBSD or you can use TrueOS (a derivative) which comes in a server version and a desktop oriented one. The latter will be easier for you to play with. You may be doing this already with Linux. There is a third and very sensible option. FreeNAS, developed by iXSystems. It is FreeBSD based and offers all these technologies with a GUI. VMWare, Hyper-V? Nowadays you can get your hands off the CLI and get a decent, usable, nice GUI.
You say you play on the cloud. The major players already include FreeBSD in their offerings. You can find it in Amazon AWS or Azure (with official Microsoft support contracts too!). You can also find it in DigitalOcean and other hosting providers. There is no excuse. You can use it at home, at the office, with old or new hardware and in the cloud as well. You can even pay for a support contract to use it. Joyent, the developers of SmartOS have their own cloud with different locations around the globe. Have a look on them too.
If you want the original of ZFS and zones you may think of Solaris. But it’s fading away. But it really isn’t. When Oracle bouth Sun many people ran away in an stampide fashion. Some of the good folks working at Sun founded new projects. One of these is Illumos. Joyent is a company formed by people who developed these technologies. They are a cloud operator, have been recently bought by Samsung and have a very competent team of people providing great tech solutions. They have developed an OS, called SmartOS (based on Illumos) with all these features. The source from this goes back to the early days of UNIX. Do you remember the days of OpenSolaris when Sun opensourced the crown jewels? There you have it. A modern opensource UNIX operating system with the roots in their original place and the head planted on today’s needs.
In conclusion. If you are on GNU/Linux and you only use opensource software you may be doing it wrong. And missing goodies you may need and like. Once you put your hands on them, trust me, you won’t look back. And if you have some “old fashioned” admins who know Solaris, you can bring them to a new profitable and exciting life with both systems.
Still not convinced? Would you have ever imagined Microsoft supporting Linux? Even loving it? They do love now FreeBSD. And not only that, they provide their own image in the Azure Cloud and you can get Microsoft support, payed support if you want to use the platform on Azure. Ain’t it… surprising? Convincing at all?
PS: I haven’t mentioned both softwares, FreeBSD and SmartOS do have a Linux translation layer. This means you can run Linux binaries on them and the program won’t cough at all. Since the ABI stays stable the only thing you need to run a Linux binary is a translation between the different system calls and the libraries. Remember POSIX? Choose your poison and enjoy it.
###A partly-cloudy IPsec VPN
Audience
I’m assuming that readers have at least a basic knowledge of TCP/IP networking and some UNIX or UNIX-like systems, but not necessarily OpenBSD or FreeBSD. This post will therefore be light on details that aren’t OS specific and are likely to be encountered in normal use (e.g., how to use vi or another text editor.) For more information on these topics, read Absolute FreeBSD (3ed.) by Michael W. Lucas.
Overview
I’m redoing my DigitalOcean virtual machines (which they call droplets). My requirements are:
VPN
Road-warrior access, so I can use private network resources from anywhere.
A site-to-site VPN, extending my home network to my VPSes.
Hosting for public and private network services.
A proxy service to provide a public IP address to services hosted at home.
The last item is on the list because I don’t actually have a public IP address at home; my firewall’s external address is in the RFC 1918 space, and the entire apartment building shares a single public IPv4 address.1 (IPv6? Don’t I wish.) The end-state network will include one OpenBSD droplet providing firewall, router, and VPN services; and one FreeBSD droplet hosting multiple jailed services.
I’ll be providing access via these droplets to a NextCloud instance at home. A simple NAT on the DO router droplet isn’t going to work, because packets going from home to the internet would exit through the apartment building’s connection and not through the VPN. It’s possible that I could do work around this issue with packet tagging using the pf firewall, but HAProxy is simple to configure and unlikely to result in hard-to-debug problems. relayd is also an option, but doesn’t have the TLS parsing abilities of HAProxy, which I’ll be using later on.
Since this system includes jails running on a VPS, and they’ve got RFC 1918 addresses, I want them reachable from my home network. Once that’s done, I can access the private address space from anywhere through a VPN connection to the cloudy router.
The VPN itself will be of the IPsec variety. IPsec is the traditional enterprise VPN standard, and is even used for classified applications, but has a (somewhat-deserved) reputation for complexity, but recent versions of OpenBSD turn down the difficulty by quite a bit.
The end-state network should look like: https://d33wubrfki0l68.cloudfront.net/0ccf46fb057e0d50923209bb2e2af0122637e72d/e714e/201812-cloudy/endstate.svg
This VPN both separates internal network traffic from public traffic and uses encryption to prevent interception or tampering.
Once traffic has been encrypted, decrypting it without the key would, as Bruce Schneier once put it, require a computer built from something other than matter that occupies something other than space. Dyson spheres and a frakton of causality violation would possibly work, as would mathemagical technology that alters the local calendar such that P=NP.2 Black-bag jobs and/or suborning cloud provider employees doesn’t quite have that guarantee of impossibility, however. If you have serious security requirements, you’ll need to do better than a random blog entry.
##News Roundup
###KLEAK: Practical Kernel Memory Disclosure Detection
Modern operating systems such as NetBSD, macOS, and Windows isolate their kernel from userspace programs to increase fault tolerance and to protect against malicious manipulations [10]. User space programs have to call into the kernel to request resources, via system calls or ioctls. This communication between user space and kernel space crosses a security boundary. Kernel memory disclosures - also known as kernel information leaks - denote the inadvertent copying of uninitialized bytes from kernel space to user space. Such disclosed memory may contain cryptographic keys, information about the kernel memory layout, or other forms of secret data. Even though kernel memory disclosures do not allow direct exploitation of a system, they lay the ground for it.
We introduce KLEAK, a simple approach to dynamically detect kernel information leaks. Simply said, KLEAK utilizes a rudimentary form of taint tracking: it taints kernel memory with marker values, lets the data travel through the kernel and scans the buffers exchanged between the kernel and the user space for these marker values. By using compiler instrumentation and rotating the markers at regular intervals, KLEAK significantly reduces the number of false positives, and is able to yield relevant results with little effort.
Our approach is practically feasible as we prove with an implementation for the NetBSD kernel. A small performance penalty is introduced, but the system remains usable. In addition to implementing KLEAK in the NetBSD kernel, we applied our approach to FreeBSD 11.2. In total, we detected 21 previously unknown kernel memory disclosures in NetBSD-current and FreeBSD 11.2, which were fixed subsequently. As a follow-up, the projects’ developers manually audited related kernel areas and identified dozens of other kernel memory disclosures.
The remainder of this paper is structured as follows. Section II discusses the bug class of kernel memory disclosures. Section III presents KLEAK to dynamically detect instances of this bug class. Section IV discusses the results of applying KLEAK to NetBSD-current and FreeBSD 11.2. Section V reviews prior research. Finally, Section VI concludes this paper.
###How To Create Official Synth Repo
System Environment
Make sure /usr/dports is updated and that it contains no cruft (git pull; git status). Remove any cruft.
Make sure your ‘synth’ is up-to-date ‘pkg upgrade synth’. If you already updated your system you may have to build synth from scratch, from /usr/dports/ports-mgmt/synth.
Make sure /etc/make.conf is clean.
Update /usr/src to the current master, make sure there is no cruft in it
Do a full buildworld, buildkernel, installkernel and installworld
Reboot
After the reboot, before proceeding, run ‘uname -a’ and make sure you are now on the desired release or development kernel.
Synth Environment
/usr/local/etc/synth/ contains the synth configuration. It should contain a synth.ini file (you may have to rename the template), and you will have to create or edit a LiveSystem-make.conf file.
System requirements are hefty. Just linking chromium alone eats at least 30GB, for example. Concurrent c++ compiles can eat up to 2GB per process. We recommend at least 100GB of SSD based swap space and 300GB of free space on the filesystem.
synth.ini should contain this. Plus modify the builders and jobs to suit your system. With 128G of ram, 30/30 or 40/25 works well. If you have 32G of ram, maybe 8/8 or less.
; Take care when hand editing!

[Global Configuration]
profile_selected= LiveSystem

[LiveSystem]
Operating_system= DragonFly
Directory_packages= /build/synth/live_packages
Directory_repository= /build/synth/live_packages/All
Directory_portsdir= /build/synth/dports
Directory_options= /build/synth/options
Directory_distfiles= /usr/distfiles
Directory_buildbase= /build/synth/build
Directory_logs= /build/synth/logs
Directory_ccache= disabled
Directory_system= /
Number_of_builders= 30
Max_jobs_per_builder= 30
Tmpfs_workdir= true
Tmpfs_localbase= true
Display_with_ncurses= true
leverage_prebuilt= false
LiveSystem-make.conf should contain one line to restrict licensing to only what is allowed to be built as a binary package:
LICENSES_ACCEPTED= NONE
Make sure there is no other cruft in /usr/local/etc/synth/
In the example above, the synth working dirs are in “/build/synth”. Make sure the base directories exist. Clean out any cruft for a fresh build from-scratch:
rm -rf /build/synth/live_packages/*
rm -rf /build/synth/logs
mkdir /build/synth/logs
Run synth everything. I recommend doing this in a ‘screen’ session in case you lose your ssh session (assuming you are ssh’d into the build machine).
(optionally start a screen session)
synth everything
A full synth build takes over 24 hours to run on a 48-core box, around 12 hours to run on a 64-core box. On a 4-core/8-thread box it will take at least 3 days. There will be times when swap space is heavily used. If you have not run synth before, monitor your memory and swap loads to make sure you have configured the jobs properly. If you are overloading the system, you may have to ^C the synth run, reduce the jobs, and start it again. It will pick up where it left off.
When synth finishes, let it rebuild the database. You then have a working binary repo.
It is usually a good idea to run synth several times to pick up any stuff it couldn’t build the first time. Each of these incremental runs may take a few hours, depending on what it tries to build.
###Interview with founder and maintainer of GhostBSD, Eric Turgeon
Thanks you Eric for taking part. To start off, could you tell us a little about yourself, just a bit of background?
How did you become interested in open source?
When and how did you get interested in the BSD operating systems?
On your Twitter profile, you state that you are an automation engineer at iXsystems. Can you share what you do in your day-to-day job?
You are the founder and project lead of GhostBSD. Could you describe GhostBSD to those who have never used it or never heard of it?
Developing an operating system is not a small thing. What made you decide to start the GhostBSD project and not join another “desktop FreeBSD” related project, such as PC-BSD and DesktopBSD at the time?
How did you get to the name GhostBSD? Did you consider any other names?
You recently released GhostBSD 18.10? What’s new in that version and what are the key features? What has changed since GhostBSD 11.1?
The current version is 18.10. Will the next version be 19.04 (like Ubuntu’s version numbering), or is a new version released after the next stable TrueOS release
Can you tell us something about the development team? Is it yourself, or are there other core team members? I think I saw two other developers on your Github project page.
How about the relationship with the community? Is it possible for a community member to contribute, and how are those contributions handled?
What was the biggest challenge during development?
If you had to pick one feature readers should check out in GhostBSD, what is it and why?
What is the relationship between iXsystems and the GhostBSD project? Or is GhostBSD a hobby project that you run separately from your work at iXsystems?
What is the relationship between GhostBSD and TrueOS? Is GhostBSD TrueOS with the MATE desktop on top, or are there other modifications, additions, and differences?
Where does GhostBSD go from here? What are your plans for 2019?
Is there anything else that wasn’t asked or that you want to share?
##Beastie Bits
dialog(1) script to select audio output on FreeBSD
Erlang otp on OpenBSD
Capsicum
https://blog.grem.de/sysadmin/FreeBSD-On-rpi3-With-crochet-2018-10-27-18-00.html
Introduction to µUBSan - a clean-room reimplementation of the Undefined Behavior Sanitizer runtime
pkgsrcCon 2018 in Berlin - Videos
Getting started with drm-kmod
##Feedback/Questions
Malcolm - Show segment idea
Fraser - Question: FreeBSD official binary package options
Harri - BSD Magazine
Send questions, comments, show ideas/topics, or stories you want mentioned on the show to [email protected]
1 hr 11 min
December 09, 2018Episode 275: OpenBSD in Stereo | BSD Now 275
DragonflyBSD 5.4 has been released, down the Gopher hole with OpenBSD, OpenBSD in stereo with VFIO, BSD/OS the best candidate for legally tested open source Unix, OpenBGPD adds diversity to the routing server landscape, and more.
Headlines
DragonflyBSD 5.4 released
DragonFly version 5.4 brings a new system compiler in GCC 8, improved NUMA support, a large of number network and virtual machine driver updates, and updates to video support. This release is 64-bit only, as with previous releases.
The details of all commits between the 5.2 and 5.4 branches are available in the associated commit messages for 5.4.0rc and 5.4.0.
Big-ticket items

Much better support for asymmetric NUMA (Non-Uniform Memory Access) configurations. In particular, both the memory subsystem and the scheduler now understand the Threadripper 2990WX’s architecture. The scheduler will prioritize CPU nodes with direct-attached memory and the memory subsystem will normalize memory queues for CPU nodes without direct-attached memory (which improves cache locality on those CPUs).

Incremental performance work. DragonFly as a whole is very SMP friendly. The type of performance work we are doing now mostly revolves around improving fairness for shared-vs-exclusive lock clashes, reducing cache ping-ponging due to non-contending SMP locks (i.e. massive use of shared locks on shared resources), and so forth.

Major updates to dports brings us to within a week or two of FreeBSD’s ports as of this writing, in particular major updates to chromium, and making the whole mess work with gcc-8.

Major rewriting of the tty clist code and the tty locking code, significantly improving concurrency across multiple ttys and ptys.

GCC 8

DragonFly now ships with GCC 8.0, and runs as the default compiler. It is also now used for building dports.

GCC 4.7.4 and GCC 5.4.1 are still installed. 4.7.4 is our backup compiler, and 5.4.1 is still there to ensure a smooth transition, but should generally not be used. buildworld builds all three by default to ensure maximum compatibility.

Many passes through world sources were made to address various warnings and errors the new GCC brought with it.

HAMMER2

HAMMER2 is recommended as the default root filesystem in non-clustered mode.

Clustered support is not yet available.

Increased bulkfree cache to reduce the number of iterations required.

Fixed numerous bugs.

Improved support on low-memory machines.

Significant pre-work on the XOP API to help support future networked operations.

Details

Checksums
MD5 (dfly-x86_64-5.4.0_REL.img) = 7277d7cffc92837c7d1c5dd11a11b98f
MD5 (dfly-x86_64-5.4.0_REL.iso) = 6da7abf036fe9267479837b3c3078408
MD5 (dfly-x86_64-5.4.0_REL.img.bz2) = a77a072c864f4b72fd56b4250c983ff1
MD5 (dfly-x86_64-5.4.0_REL.iso.bz2) = 4dbfec6ccfc1d59c5049455db914d499

Downloads Links
DragonFly BSD is 64-bit only, as announced during the 3.8 release.
USB: dfly-x86_64-5.4.0_REL.img as bzip2 file

ISO: dfly-x86_64-5.4.0_REL.iso as bzip2 file

Uncompressed ISO: dfly-x86_64-5.4.0_REL.iso (For use with VPS providers as an install image.)
Down the Gopher hole with OpenBSD, Gophernicus, and TLS
In the early 2000s I thought I had seen the worst of the web - Java applets, Macromedia (>Adobe) Flash, animated GIFs, javascript snow that kept you warm in the winter by burning out your CPU, and so on. For a time we learned from these mistakes, and started putting the burden on the server-side - then with improvements in javascript engines we started abusing it again with JSON/AJAX and it all went down hill from there.
Like cloud computing, blockchains, machine learning and a tonne of other a la mode technologies around today - most users and service providers don’t need websites that consume 1GB of memory processing JS and downloading 50MB of compressed data just to read Alice’s one-page travel blog or Bob’s notes on porting NetBSD to his blood-pressure monitor.
Before the HTTP web we relied on Prestel/Minitel style systems, BBS systems, and arguably the most accessible of all - Gopher! Gopher was similar to the locally accessed AmigaGuide format, in that it allowed users to search and retrieve documents interactively, with links and cross-references. Its efficiency and distraction-free nature make it attractive to those who are tired of the invasive, clickbait, ad-filled, javascript-laden web2/3.x. But enough complaining and evangelism - here’s how to get your own Gopher Hole!
Gophernicus is a modern gopher daemon which aims to be secure (although it still uses inetd -_-); it’s even in OpenBSD ports so at least we can rely on it to be reasonably audited.
If you need a starting point with Gopher, SDF-EU’s wiki has a good article here.
https://sdfeu.org/w/tutorials:gopher
Finally, if you don’t like gopher(1) - there’s always lynx(1) or NCSA Mosaic!
https://cryogenix.net/NCSA_Mosaic_OpenBSD.html
I’ve added TLS support to Gophernicus so you don’t need to use stunnel anymore. The code is ugly and unpolished though so I wouldn’t recommend for production use.
https://github.com/0x16h/gophernicus

https://github.com/0x16h/gophernicus/blob/master/INSTALL.openbsd
News Roundup
OpenBSD in Stereo with Linux VFIO
I use a Huawei Matebook X as my primary OpenBSD laptop and one aspect of its hardware support has always been lacking: audio never played out of the right-side speaker. The speaker did actually work, but only in Windows and only after the Realtek Dolby Atmos audio driver from Huawei was installed. Under OpenBSD and Linux, and even Windows with the default Intel sound driver, audio only ever played out of the left speaker.
Now, after some extensive reverse engineering and debugging with the help of VFIO on Linux, I finally have audio playing out of both speakers on OpenBSD.
VFIO
The Linux kernel has functionality called VFIO which enables direct access to a physical device (like a PCI card) from userspace, usually passing it to an emulator like QEMU.
To my surprise, these days, it seems to be primarily by gamers who boot Linux, then use QEMU to run a game in Windows and use VFIO to pass the computer’s GPU device through to Windows.
By using Linux and VFIO, I was able to boot Windows 10 inside of QEMU and pass my laptop’s PCI audio device through to Windows, allowing the Realtek audio drivers to natively control the audio device. Combined with QEMU’s tracing functionality, I was able to get a log of all PCI I/O between Windows and the PCI audio device.
Using VFIO
To use VFIO to pass-through a PCI device, it first needs to be stubbed out so the Linux kernel’s default drivers don’t attach to it. GRUB can be configured to instruct the kernel to ignore the PCI audio device (8086:9d71) and explicitly enable the Intel IOMMU driver by adding the following to /etc/default/grub and running update-grub
With the audio device stubbed out, a new VFIO device can be created from it
Then the VFIO device (00:1f.3) can be passed to QEMU
I was using my own build of QEMU for this, due to some custom logging I needed (more on that later), but the default QEMU package should work fine. The events.txt was a file of all VFIO events I wanted logged (which was all of them).
Since I was frequently killing QEMU and restarting it, Windows 10 wanted to go through its unexpected shutdown routine each time (and would sometimes just fail to boot again). To avoid this and to get a consistent set of logs each time, I used qemu-img to take a snapshot of a base image first, then boot QEMU with that snapshot. The snapshot just gets thrown away the next time qemu-img is run and Windows always starts from a consistent state.
QEMU will now log each VFIO event which gets saved to a debug-output file.
With a full log of all PCI I/O activity from Windows, I compared it to the output from OpenBSD and tried to find the magic register writes that enabled the second speaker. After days of combing through the logs and annotating them by looking up hex values in the documentation, diffing runtime register values, and even brute-forcing it by mechanically duplicating all PCI I/O activity in the OpenBSD driver, nothing would activate the right speaker.
One strange thing that I noticed was if I booted Windows 10 in QEMU and it activated the speaker, then booted OpenBSD in QEMU without resetting the PCI device’s power in-between (as a normal system reboot would do), both speakers worked in OpenBSD and the configuration that the HDA controller presented was different, even without any changes in OpenBSD.
A Primer on Intel HDA
Most modern computers with integrated sound chips use an Intel High Definition Audio (HDA) Controller device, with one or more codecs (like the Realtek ALC269) hanging off of it. These codecs do the actual audio processing and communicate with DACs and ADCs to send digital audio to the connected speakers, or read analog audio from a microphone and convert it to a digital input stream. In my Huawei Matebook X, this is done through a Realtek ALC298 codec.
On OpenBSD, these HDA controllers are supported by the azalia(4) driver, with all of the per-codec details in the lengthy azalia_codec.c file. This file has grown quite large with lots of codec- and machine-specific quirks to route things properly, toggle various GPIO pins, and unmute speakers that are for some reason muted by default.
The azalia driver talks to the HDA controller and sets up various buffers and then walks the list of codecs. Each codec supports a number of widget nodes which can be interconnected in various ways. Some of these nodes can be reconfigured on the fly to do things like turning a microphone port into a headphone port.
The newer Huawei Matebook X Pro released a few months ago is also plagued with this speaker problem, although it has four speakers and only two work by default. A fix is being proposed for the Linux kernel which just reconfigures those widget pins in the Intel HDA driver. Unfortunately no pin reconfiguration is enough to fix my Matebook X with its two speakers.
While reading more documentation on the HDA, I realized there was a lot more activity going on than I was able to see through the PCI tracing.
For speed and efficiency, HDA controllers use a DMA engine to transfer audio streams as well as the commands from the OS driver to the codecs. In the output above, the CORBWP=0; size=256 and RIRBRP=0, size=256 indicate the setup of the CORB (Command Output Ring Buffer) and RIRB (Response Input Ring Buffer) each with 256 entries. The HDA driver allocates a DMA address and then writes it to the two CORBLBASE and CORBUBASE registers, and again for the RIRB.
When the driver wants to send a command to a codec, such as CORB_GET_PARAMETER with a parameter of COP_VOLUME_KNOB_CAPABILITIES, it encodes the codec address, the node index, the command verb, and the parameter, and then writes that value to the CORB ring at the address it set up with the controller at initialization time (CORBLBASE/CORBUBASE) plus the offset of the ring index. Once the command is on the ring, it does a PCI write to the CORBWP register, advancing it by one. This lets the controller know a new command is queued, which it then acts on and writes the response value on the RIRB ring at the same position as the command (but at the RIRB’s DMA address). It then generates an interrupt, telling the driver to read the new RIRBWP value and process the new results.
Since the actual command contents and responses are handled through DMA writes and reads, these important values weren’t showing up in the VFIO PCI trace output that I had gathered. Time to hack QEMU.
Logging DMA Memory Values in QEMU
Since DMA activity wouldn’t show up through QEMU’s VFIO tracing and I obviously couldn’t get Windows to dump these values like I could in OpenBSD, I could make QEMU recognize the PCI write to the CORBWP register as an indication that a command has just been written to the CORB ring.
My custom hack in QEMU adds some HDA awareness to remember the CORB and RIRB DMA addresses as they get programmed in the controller. Then any time a PCI write to the CORBWP register is done, QEMU fetches the new CORB command from DMA memory, decodes it into the codec address, node address, command, and parameter, and prints it out. When a PCI read of the RIRBWP register is requested, QEMU reads the response and prints the corresponding CORB command that it stored earlier.
With this hack in place, I now had a full log of all CORB commands and RIRB responses sent to and read from the codec:
An early version of this patch left me stumped for a few days because, even after submitting all of the same CORB commands in OpenBSD, the second speaker still didn’t work. It wasn’t until re-reading the HDA spec that I realized the Windows driver was submitting more than one command at a time, writing multiple CORB entries and writing a CORBWP value that was advanced by two. This required turning my CORB/RIRB reading into a for loop, reading each new command and response between the new CORBWP/RIRBWP value and the one previously seen.
Sure enough, the magic commands to enable the second speaker were sent in these periods where it submitted more than one command at a time.
Minimizing the Magic
The full log of VFIO PCI activity from the Windows driver was over 65,000 lines and contained 3,150 CORB commands, which is a lot to sort through. It took me a couple more days to reduce that down to a small subset that was actually required to activate the second speaker, and that could only be done through trial and error:
Boot OpenBSD with the full list of CORB commands in the azalia driver

Comment out a group of them

Compile kernel and install it, halt the QEMU guest

Suspend and wake the laptop, resetting PCI power to the audio device to reset the speaker/Dolby initialization and ensure the previous run isn’t influencing the current test (I’m guessing there is an easier to way to reset PCI power than suspending the laptop, but oh well)

Start QEMU, boot OpenBSD with the new kernel

Play an MP3 with mpg123 which has alternating left- and right-channel audio and listen for both channels to play
This required a dozen or so iterations because sometimes I’d comment out too many commands and the right speaker would stop working. Other times the combination of commands would hang the controller and it wouldn’t process any further commands. At one point the combination of commands actually flipped the channels around so the right channel audio was playing through the left speaker.
The Result
After about a week of this routine, I ended up with a list of 662 CORB commands that are needed to get the second speaker working. Based on the number of repeated-but-slightly-different values written with the 0x500 and 0x400 commands, I’m guessing this is some kind of training data and that this is doing the full Dolby/Atmos system initialization, not just turning on the second speaker, but I could be completely wrong.
In any case, the stereo sound from OpenBSD is wonderful now and I can finally stop downmixing everything to mono to play from the left speaker. In case you ever need to do this, sndiod can be run with -c 0:0 to reduce the channels to one.
Due to the massive size of the code needed for this quirk, I’m not sure if I’ll be committing it upstream in OpenBSD or just saving it for my own tree. But at least now the hardware support chart for my Matebook is all yeses for the things I care about.
I’ve also updated the Linux bug report that I opened before venturing down this path, hoping one of the maintainers of that HDA code that works at Intel or Realtek knew of a solution I could just port to OpenBSD. I’m curious to see what they’ll do with it.
Why BSD/OS is the best candidate for being the only tested legally open UNIX
Introduction
The UNIX® system is an old operating system, possibly older than many of the readers of this post. However, despite its age, it still has not been open sourced completely. In this post, I will try to detail which parts of which UNIX systems have not yet been open sourced. I will focus on the legal situation in Germany in particular, taking it representative of European law in general – albeit that is a stretch, knowing the diversity of European jurisdictions. Please note that familiarity with basic terms of copyright law is assumed.
Ancient UNIX
The term “Ancient UNIX” refers to the versions of UNIX up to and including Seventh Edition UNIX (1979) including the 32V port to the VAX. Ancient UNIX was created at Bell Laboratories, a subsidiary of AT&T at the time. It was later transferred of the AT&T UNIX Support Group, then AT&T Information Systems and finally the AT&T subsidiary UNIX System Laboratories, Inc. (USL). The legal situation differs between the United States of America and Germany.
In a ruling as part of the UNIX System Laboratories, Inc. v. Berkeley Software Design, Inc. (USL v. BSDi) case, a U.S. court found that USL had no copyright to the Seventh Edition UNIX system and 32V – arguably, by extension, all earlier versions of Ancient UNIX as well – because USL/AT&T had failed to affix copyright notices and could not demonstrate a trade secret. Due to the obsessive tendency of U.S. courts to consider themselves bound to precedents (cf. the infamous Pierson v. Post case), it can be reasonably expected that this ruling would be honored and applied in subsequent cases. Thus under U.S. law, Ancient UNIX can be safely assumed to belong in the public domain.
The situation differs in Germany. Unlike the U.S., copyright never needed registration in order to exist. Computer programs are works in the sense of the German 1965 Act on Copyright and Related Rights (Copyright Act, henceforth CopyA) as per CopyA § 2(1) no. 1. Even prior to the amendment of CopyA § 2(1) to include computer programs, computer programs have been recognized as copyrightable works by the German Supreme Court (BGHZ 112, 264 Betriebssystem, no. 19); CopyA § 137d(1) rightly clarifies that. The copyright holder at 1979 would still have been USL via Bell Labs and AT&T. Copyright of computer programs is transferred to the employer upon creation under CopyA § 69(1).
Note that this does not affect expiry (Daniel Kaboth/Benjamin Spies, commentary on CopyA §§ 69a‒69g, in: Hartwig Ahlberg/Horst-Peter Götting (eds.), Urheberrecht: UrhG, KUG, VerlG, VGG, Kommentar, 4th ed., C. H. Beck, 2018, no. 16 ad CopyA § 69b; cf. Bundestag-Drucksache [BT-Drs.] 12/4022, p. 10). Expiry occurs 70 years after the death of the (co-)author that died most recently as per CopyA § 65(1) and 64; this has been the case since at least the 1960s, meaning there is no way for copyright to have expired already (old version, as per Bundesgesetzblatt Part I No. 51 of September 16, 1965, pp. 1273‒1294).
In Germany, private international law applies the so-called “Territorialitätsprinzip” for intellectual property rights. This means that the effect of an intellectual property right is limited to the territory of a state (Anne Lauber-Rönsberg, KollisionsR, in: Hartwig Ahlberg/Horst-Peter Götting (eds.), ibid., pp. 2241 et seqq., no. 4). Additionally, the “Schutzlandprinzip” applies; this means that protection of intellectual property follows the lex loci protectionis, i.e. the law of the country for which protection is sought (BGH GRUR 2015, 264 HiHotel II, no. 25; BGH GRUR 2003, 328 Sender Felsberg, no. 24), albeit this is criticized in parts of doctrine (Lauber-Rönsberg, ibid., no. 10). The “Schutzlandprinzip” requires that the existence of an intellectual property right be verified as well (BGH ZUM 2016, 522 Wagenfeld-Leuchte II, no. 19).
Thus, in Germany, copyright on Ancient UNIX is still alive and well. Who has it, though? A ruling by the U.S. Court of Appeals, Tenth Circuit, in the case of The SCO Group, Inc. v. Novell, Inc. (SCO v. Novell) in the U.S. made clear that Novell owns the rights to System V – thus presumably UNIX System III as well – and Ancient UNIX, though SCO acquired enough rights to develop UnixWare/OpenServer (Ruling 10-4122 [D.C. No. 2:04-CV-00139-TS], pp. 19 et seq.). Novell itself was purchased by the Attachmate Group, which was in turn acquired by the COBOL vendor Micro Focus. Therefore, the rights to SVRX and – outside the U.S. – are with Micro Focus right now. If all you care about is the U.S., you can stop reading about Ancient UNIX here.
So how does the Caldera license factor into all of this? For some context, the license was issued January 23, 2002 and covers Ancient UNIX (V1 through V7 including 32V), specifically excluding System III and System V. Caldera, Inc. was founded in 1994. The Santa Cruz Operation, Inc. sold its rights to UNIX to Caldera in 2001, renamed itself to Tarantella Inc. and Caldera renamed itself The SCO Group. Nemo plus iuris ad alium transferre potest quam ipse habet; no one can transfer more rights than he has. The question now becomes whether Caldera had the rights to issue the Caldera license.
I’ve noted it above but it needs restating: Foreign decisions are not necessarily accepted in Germany due to the “Territorialitätsprinzip” and “Schutzlandprinzip” – however, I will be citing a U.S. ruling for its assessment of the facts for the sake of simplicity. As per ruling 10-4122, “The district court found the parties intended for SCO to serve as Novell’s agent with respect to the old SVRX licenses and the only portion of the UNIX business transferred outright under the APA [asset purchase agreement] was the ability to exploit and further develop the newer UnixWare system. SCO was able to protect that business because it was able to copyright its own improvements to the system. The only reason to protect the earlier UNIX code would be to protect the existing SVRX licenses, and the court concluded Novell retained ultimate control over that portion of the business under the APA.” The relevant agreements consist of multiple pieces:
the base Asset Purchase Agreement “APA” (Part I)
the base Asset Purchase Agreement “APA” (Part II)
the Operating Agremeent and Amendment 1 to the APA
the Amendment 2 to the APA
The APA dates September 19, 1995, from before the Caldera license. Caldera cannot possibly have acquired rights that The Santa Cruz Operation, Inc. itself never had. Furthermore, I’ve failed to find any mention of Ancient UNIX; all that is transferred is rights to SVRX. Overall, I believe that the U.S. courts’ assesment of the facts represents the situation accurately. Thus for all intents and purposes, UNIX up to and including System V remained with Novell/Attachmate/Micro Focus. Caldera therefore never had any rights to Ancient UNIX, which means it never had the rights to issue the Caldera license. The Caldera license is null and void – in the U.S. because the copyright has been lost due to formalities, everywhere else because Caldera never had the rights to issue it.
The first step to truly freeing UNIX would this be to get Micro Focus to re-issue the Caldera license for Ancient UNIX, ideally it would now also include System III and System V.
BSD/OS
Another operating system near UNIX is of interest. The USL v. BSDi lawsuit includes two parties: USL, which we have seen above, and Berkeley Software Design, Inc. BSDi sold BSD/386 (later BSD/OS), which was a derivative of 4.4BSD. The software parts of the BSDi company were acquired by Wind River Systems, whereas the hardware parts went to iXsystems. Copyright is not disputed there, though Wind River Systems ceased selling BSD/OS products 15 years ago, in 2003. In addition, Wind River System let their trademark on BSD expire, though this is without consequence for copyright.
BSD/OS is notable in the sense that it powered much of early internet infrastructure. Traces of its legacy can still be found on Richard Stevens’ FAQ.
To truly make UNIX history free, BSD/OS would arguably also need to see a source code release. BSD/OS at least in its earliest releases under BSDi would ship with source code, though under a non-free license, far from BSD or even GPL licensing.
System V
The fate of System V as a whole is difficult to determine. Various licenses have been granted to a number of vendors (Dell UNIX comes to mind; HP for HP-UX, IBM for AIX, SGI UNIX, etc.). Sun released OpenSolaris – notoriously, Oracle closed the source to Solaris again after its release –, which is a System V Release 4 descendant. However, this means nothing for the copyright or licensing status of System V itself. Presumably, the rights with System V still remain with Novell (now Micro Focus): SCO managed to sublicense rights to develop and sell UnixWare/OpenServer, themselves System V/III descendants, to unXis, Inc. (now known as Xinuos, Inc.), which implies that Xinuos is not the copyright holder of System V.
Obviously, to free UNIX, System V and its entire family of descendants would also need to be open sourced. However, I expect tremendous resistance on part of all the companies mentioned. As noted in the “Ancient UNIX” section, Micro Focus alone would probably be sufficient to release System V, though this would mean nothing for the other commercial System V derivatives.
Newer Research UNIX
The fate of Bell Labs would be a different one; it would go on to be purchased by Lucent, now part of Nokia. After commercial UNIX got separated out to USL, Research UNIX would continue to exist inside of Bell Labs. Research UNIX V8, V9 and V10 were not quite released by Alcatel-Lucent USA Inc. and Nokia in 2017.
However, this is merely a notice that the companies involved will not assert their copyrights only with respect to any non-commercial usage of the code. It is still not possible, over 30 years later, to freely use the V8 code.
Conclusion
In the U.S., Ancient UNIX is freely available. People located everywhere else, however, are unable to legally obtain UNIX code for any of the systems mentioned above. The exception being BSD/OS, assuming a purchase of a legitimate copy of the source code CD. This is deeply unsatisfying and I implore all involved companies to consider open sourcing (preferably under a BSD-style license) their code older than a decade, if nothing else, then at least for the sake of historical purposes. I would like to encourage everybody reading this to consider reaching out to Micro Focus and Wind River Systems about System V and BSD/OS, respectively. Perhaps the masses can change their minds.
A small note about patents: Some technologies used in newer iterations of the UNIX system (in particular the System V derivatives) may be encumbered with software patents. An open source license will not help against patent infringement claims. However, the patents on anything used in the historical operating systems will certainly have expired by now. In addition, European readers can ignore this entirely – software patents just aren’t a thing.
OpenBGPD - Adding Diversity to the Route Server Landscape
Introduction
As of last year, there was effectively only a single solution in the Route Server vendor market: the BIRD Internet routing daemon. NIC.CZ (the organisation developing BIRD) has done fantastic work on maintaining their BGP-4 implementation, however, it’s not healthy to have virtually every Internet Exchange Point (IXP) in the RIPE NCC service region depend on a single open source project. The current situation can be compared to the state of the DNS root nameservers back in 2002 - their dependence on the BIND nameserver daemon and the resulting development of NSD as an alternative by NLnet, in cooperation with the RIPE NCC.
OpenBGPD used to be one of the most popular Route Server implementations until the early 2010s. OpenBGPD’s main problem was that its performance couldn’t keep up with the Internet’s growth, so it lost market share. An analysis by Job Snijders suggested that a modernised OpenBGPD distribution would be a most viable option to regain diversity on the Route Server level.
Missing features in OpenBGPD
The following main missing features were identified in OpenBGPD:
Performance
In previous versions of OpenBGPD, the filtering performance didn’t allow proper filtering of all EBGP sessions. Current best practice at IXP Route Servers is to carefully evaluate and validate of all routes learned from EBGP peers. The OpenBGPD ruleset required to do correct filtering (in many deployment scenarios) was simply too lengthy - and negatively impacted service performance during configuration reloads. While filtering performance is the biggest bottleneck, general improvements to the Routing Information Base were also made to improve scalability. IXP Route Servers with a few hundred peering sessions are commonplace and adding new sessions shouldn’t impact the Route Servers’ service to other peers. We found that performance was the most pressing issue that needed to be tackled.
Lack of RPKI Origin Validation
As we’ve seen, Internet operators are moving to adopt RPKI based BGP Origin Validation. While it was theoretically possible to emulate RFC 6811-style Origin Validation in previous versions of OpenBGPD, the required configuration wasn’t optimised for performance and wasn’t user friendly. We believe that BGP Origin Validation should be as easy as possible - this requires BGP-4 vendors to implement native, optimised routines for Origin Validation. Of course, enabling Origin Validation shouldn’t have an impact on performance either when processing BGP updates or when updating the Route Origin Authorisation (ROA) table itself.
Portability
OpenBGPD is an integral part of OpenBSD, but IXPs may prefer to run their services infrastructure on an operating system of their choice. Making sure that there’s a portable OpenBGPD version which follows the OpenBSD project release cycle will give IXPs this option.
Development steps
By addressing the issues mentioned above, we could bring back OpenBGPD as a viable Route Server implementation.
Since I was one of the core OpenBGPD developers, I was asked if I wanted to pick up this project again. Thanks to the funding from the RIPE NCC Project Fund, this was possible. Starting in June 2018, I worked full time on this important community project. Over the last few months, many of the problems are already addressed and are now part of the OpenBSD 6.4 release. So far, 154 commits were made to OpenBGPD during the 6.4 development cycle - around 8% of all commits ever to OpenBGPD! This shows that due to funding and dedicated resources, a lot of work could be pushed into the latest release of OpenBGPD.
OpenBGPD 6.4
The OpenBGPD version, as part of OpenBSD 6.4 release, demonstrates great progress. Even though there have been many changes to the core of OpenBGPD, the released version is as solid and reliable as previous releases and the many bug fixes and improvements make this the best OpenBGPD release so far. The changes in the filter language allow users to write more efficient rulesets while the introduction of RPKI origination validation fixes an important missing feature. For IXPs, OpenBGPD now is an alternative again. There are still open issues, but the gap is closing!
Feature highlights
The following changes should be highlighted:
Introduction of background soft-reconfiguration on config reload. Running the soft-reconfiguration task in the background allows for new updates and withdraws to be processed at the same time. This improves convergence time - one of the key metrics for Route Servers.

BGP Origin Validation when a roa-set is configured Every EBGP route announcement is validated against the locally configured VRP table entries. Depending on the validation process’s outcome, the validation state is set to valid, invalid or not found. The filter language has been extended to allow checking for the origin validation state, and thanks to this, it is possible to deny invalid prefixes or regard valid prefixes different to the ones that aren’t found. The roa-set table is read from the configuration file and updated during configuration reloads. On production systems reloading the roa-set and applying it to all prefixes is done in a couple of seconds.

Fast prefix-set lookups In OpenBSD 6.3 prefix-sets got introduced in OpenBGPD. A prefix-set combines many prefix lookups into a single filter rule. The original implementation wasn’t optimised but now a fast trie lookup is used. Thanks to this, large IRR DB prefix tables can now be implemented efficiently.

Introduction of as-sets Similar to prefix-sets, as-sets help group many AS numbers into a single lookup. Thanks to this, large IRR DB origin AS tables can be implemented efficiently.
Introduction of origin-sets

Looking at the configurations of Route Servers doing full filtering, it was noticed that a common lookup was binding a prefix to an origin AS - similar to how a roa-set is used for RPKI. These origin-set tables are used to extend the IRR prefix lookup and generated from alternative sources.

Improving third party tools
Users can only benefit from the changes introduced in OpenBGPD 6.4 when the surrounding 3rd party tools are adjusted accordingly. Two opensource projects such as bgpq3 and arouteserver are frequently used by network operators and IXPs to generate BGP configurations. Thanks to our contributions to those projects, we were able to get them ready for all the new features in OpenBGPD.
bgpq3 was extended to create as-set and prefix-set tables based on IRR DB entries. This is replacing the old way of doing the same with a large amount of filter rules. Thanks to the quick response from the bgpq3 maintainer, it was possible to ship OpenBSD 6.4 with a bgpq3 package that includes all the new features.

arouteserver was adjusted to implement RPKI roa-set, as-set, prefix-set, and origin-set to generate a much better-performing configurations for the 6.4 version. With the v0.20.0 release of arouteserver, IXPs are able to generate an OpenBGPD configuration which is a ton faster but also implements the new functionalities. Looking at YYCIX (the resident IXP in Calgary, Canada) the ruleset generated by arouteserver was reduced from 370,000 rules to well under 6,000 rules. This resulted in the initial convergence time dropping from over 1 hour to less than 2 minutes, and subsequent configuration reloads are hitless and no longer noticeable.

What still needs to be done
A sizeable chunk of work still left on the table is the rework of the RIB data structures in OpenBGPD - these haven’t been changed since the initial design of OpenBGPD in 2003. There’s currently ongoing work (in small steps, to avoid jeopardising the stability of OpenBGPD) to modernise these data-structures. The goal is to provide better decoupling of the filter step from storing RIB database changes, to pave the way to multi-threaded operations at a later point.
Looking forward

Job Snijders oversaw this year’s fundraising and project management, he adds:
It’s been incredibly productive to create an environment where a core developer is allowed to work full time on the OpenBGPD code base. However, it’s important to note there still is room for a number of new features to help improve its operational capabilities (such as BMP, RFC 7313, ADD_PATH, etc). It’d be beneficial to the Internet community at large if we can extend Claudio Jeker’s involvement for another year. Open source software doesn’t grow on trees! Strategic investments are the only way to keep OpenBGPD’s roadmap aligned with Internet growth and operator requirements.
Beastie Bits
DragonFly - git: annotated tag v5.5.0 created

Torchlight 2 on NetBSD

Older, but still good USENIX Login Article on Capsicum

The Super Capsicumizer 9000

Dedicated and Virtual Server PXE provisioning tool

Cirrus CI have announced FreeBSD support

NetBSD PineBook Gameplay

BSDCan 2019 CfP is out

Allan’s first ZFS array, Zulu, turned 7 years old on Nov 29th
Feedback/Questions
Malcom - Installing Drivers in Development

Samir - Introduction to ZFS

Newnix - Drive Failures
Send questions, comments, show ideas/topics, or stories you want mentioned on the show to [email protected]
1 hr 25 min

About BSD Now

From the publisher's feed

Created by three guys who love BSD, we cover the latest news and have an extensive series of tutorials, as well as interviews with various people from all areas of the BSD community. It also serves as…