BSD Now

This week on the show, we’re loaded up with great stories ranging from System call fuzzing, a history of UNIX Pipes, speeding up MySQL imports and more. Stay tuned, BSDNow is coming your way right now.

One of the important developments in Unix was the introduction of pipes. Pipes had been suggested by McIlroy during the early days of creating Unix. Ritchie explains how "the idea, explained one afternoon on a blackboard, intrigued us but failed to ignite any immediate action. There were several objections to the idea as put....What a failure of imagination," he admits.(35) McIlroy concurs, describing how the initial effort to add pipes to Unix occurred about the same time in 1969 that Ritchie, Thompson and Canaday were outlining ideas for a file system. "That was when," he writes, "the simple pipeline as a way to combine programs, with data notationally propagating along a chain of (not necessarily concurrent) filters was articulated."(36) However, pipes weren't implemented in Unix until 1972.

Open Systems! Our Systems! How well those who were there remember the pipe-festooned garret where Unix took form. The excitement of creation drew people to work there amidst the whine of the computer's cool- ing fans, even though almost the same computer ac- cess, could be had from one's office or from home. Those raw quarters saw a procession of memorable events. The advent of software pipes precipitated a day-long orgy of one-liners....As people reveled in the power of functional composition in the large, which is even today unavailable to users of other systems.

When you deploy an application using Adhokku, Adhokku creates a new jail on the remote host and provisions it from a fixed clean state using the instructions in the Jailfile in your Git repository. All jails sit behind a reverse proxy that directs traffic to one of them based on the domain name or the IP address in the HTTP request. When a new jail has been provisioned for an application, Adhokku seamlessly reconfigures the reverse proxy to send traffic to it instead of the one currently active for that application.

The following instructions show how to get Adhokku and an example application running in a VM on your development machine using Vagrant. This process should require no FreeBSD-specific knowledge, through modifying the Jailfile to customize the application may.

This week on BSDNow, we’ve got voting news for you (No not that election), a closer look at

I have been lucky enough to participate in Google Summer of Code with the FreeBSD foundation. I was amazed by the community surrounding it which was noob friendly and very helpful (Thank you FreeBSD <3)

I wanted to make a starting tutorial for people to write a simple module for kernel before diving inside more complicated kernel shizzle

The kernel module that we will be working on is a simple event handler for the kernel. It will be composed of 2 parts, the event handling function, and the module declaration

The module event handler is a function that handles different events for the module. Like the module being loaded, unloaded or on system shutdown

Now that we have the events handling function ready. We need to declare the moduledata_t to be able to use it inside DECLARE_MODULE macro and load it into the kernel. It has the module name and a pointer to the event handling function

Lastly, we need to declare the module using the DECLARE_MODULE macro. Which has the following structure:

~~

name: The module name that will be used in the SYSINIT() call to identify the module.

On Friday, I committed support for Large Communities to OpenBGPD. This is a draft-RFC that I am pretty excited about.

Back in the early days of The Internet, when routers rode dinosaurs to work and nerds weren't cool, we wanted to signal to our network neighbours certain information about routes. To be fair, we still do. But, back then everyone had 16 bit ASNs, so there was a simple concept called 'communities'. This was a 32bit opaque value, that was traditionally split into two 16bit values. Conveniently, we were able to encode an "us" and a "them", and perform actions based on what our neighbours told us.

But, 16bits is pretty limiting. There could only be ~65'000 possible networks on The Internet total? Eeek. So, we created 32bit ASNs. 4 billion networks is seen as a quite reasonable limitation. However, you can't really encode a 32bit "us" and a 32bit "them" value into 32bits of total space. Something called "Extended Communities" was invented, but it tries to solve everything except the case of a 32bit ASN signalling to another 32bit ASN.

Enter Large Communities. This is 3 32bit values. The first one is the "owner" of the namespace. Normally, you would put in your own ASN, or the ASN that you wish to signal. The second two 32bit values are opaque and only have meaning from the originating operator, but normally people will use "myasn":"verb":"noun" Or "myasn":"noun":"verb". Either way, it fits very nicely.

Having previously ran a 32bit ASN, it became quickly obvious the lack of suitable communities was a critical problem. It was even the way to request an "old style" 16bit ASN from RIPE, "I need to use communities". Even the ability to say "do this to that ASN" was ugly, since you couldn't really communicate who the community was supposed to matter to. Clearly, we The Internet Community screwed up by not addressing this need earlier.

OpenBGPD in OpenBSD -current has support for Large Communities, and this will be available in the 6.1 release and later. This was based partially on a patch from Job Snijders, thanks!

This enhancement looks extremely important for the BeaST storage system as implementation of high available native ALUA in FreeBSD can potentially replace the BeaST arbitration mechanism (“Arbitrator”), which is completely described in the papers on the BeaST project page

ALUA in storage world terminology means Asymmetric Logical Unit Assignment. In simple words this set of technologies allows a host to access any LUN via both controllers of a storage system

As I still do not have any real hardware drive-enclosures, we will use Oracle Virtual Box and iSCSI protocol. I have already deployed this environment for the BeaST development, so we can use the similar, yet more simplified template for the renewed CTL HA testing purpose.

kern.cam.ctl.ha_role – configures default role for the node. So ctrl-a is set as 0 (primary node), ctrl-b – 1 (secondary node). The role also can be specified on per-LUN basis which allows to distribute LUNs over both controllers evenly.

The link states can be: 0 – not configured, 1 – configured but not established and 2 – established

sysctl kern.iscsi.fail_on_disconnection=1 on the client is needed to drop connection with one of the controllers in case of its failure

As we know that da0 and da1 on the client are the same drive, we can put them under multipathing control: gmultipath create -A HA /dev/da0 /dev/da1

The Sender Policy Framework (SPF) is unloved by some, because it conflicts with some long-established SMTP email use cases. But is it also just too hard to understand and to use correctly for application developers?

Then in August 2016, I tried to report a bug via the contact form at Altinn.no, the main tax authorities web site.

The report in itself was fairly trivial: The SMS alert I had just received about an invoice for taxes due contained one date, which turned out to be my birth date rather than the invoice due date. Not a major issue, but potentially confusing to the recipient until you actually log in and download the invoice as PDF and read the actual due date and other specifics.

The next time I checked my mail at bsdly.net, I found this bounce:

[email protected]: SMTP error from remote mail server after RCPT TO:: host mx.isp.as2116.net [193.75.104.7]: 550 5.7.23 SPF validation failed

which means that somebody, somewhere tried to send a message to [email protected], but the message could not be delivered because the sending machine did not match the published SPF data for the sender domain.

What happened is actually quite clear even from the part quoted above: the host mx.isp.as2116.net [193.75.104.7] tried to deliver mail on my behalf (I received the bounce, remember), and since I have no agreement for mail delivery with the owners and operators of that host, it is not in bsdly.net's SPF record either, and the delivery fails.

It looks like you have Sender Policy Framework (SPF) enabled on your mailserver, It is a known weakness of our contact form that mailervers with SPF are not supported.

The obvious answer should be, as you will agree if you're still reading: The form's developer should place the user's email address in the Reply-To: field, and send the message as its own, valid local user. That would solve the problem.

Yes, I'm well aware that SPF also breaks traditional forwarding of the type generally used by mailing lists and a few other use cases. Just how afraid should we be when those same developers come to do battle with the followup specifications such as DKIM and (shudder) the full DMARC specification?

This week on the show, we’ve got all sorts of goodies to discuss. Starting with, vmm, vkernels, raspberry pi and much more! Some iX folks are visiting from out of

“Raspberry Pi 3 limited support was committed to HEAD. Most of drivers should work with upstream dtb, RNG requires attention because callout mode seems to be broken and there is no IRQ in upstream device tree file. SMP is work in progress. There are some compatibility issue with VCHIQ driver due to some assumptions that are true only for ARM platform. “

The wait is over, 11.0 of FreeBSD has (officially) launched. We’ll have coverage of this, plus a couple looks back at UNIX history, and a crowd-favorite guest today.

Thus sysctl can iterate over the entire process list, copying out information to ps(1), without blocking. If we prevent processes from forking or exiting during this time, we get a consistent snapshot. The snapshot may be stale, but it will never show us a viewpoint that never happened.

Is there a way to trick ps on OpenBSD? Not everything is consistent. There’s a separate sysctl, KERN_PROC_ARGV, that reads the command line arguments for a process, but it only works on one process at a time. Processes can modify their own argv at any time.

if_iwm - Add basic powermanagement support via ifconfig wlan0 powersave.

I created a 64MB EFI partition at the front of the disk. Next, I created a 20GB primary partition at the beginning of the space, mounted as the root (/) filesystem.

I then added a 4096MB swap partition for Ubuntu. Finally, I used the rest of the free space to create a Reserved BIOS Boot Area FAT32 partition that was not associated with a mount point – this is where I will be installing OpenBSD.

I mashed through the defaults in the OpenBSD installer until I got to the disk partitioning. Since I told VMWare to make my hard drive an IDE one, I knew I was playing around with wd0 and not sd0 (my USB key). I dumped into fdisk by selecting to (E)dit the partition scheme and saw my setup from Linux. First was the EFI partition (I am guessing I’ll have to copy my bootx64.efi file to that at some point), second was the Linux etx4 partition, third was my Linux swap partition and fourth was a weird looking one that is the “Reserved BIOS Boot” partition. That’s the one I’ll fiddle with.

Issuing the command “edit 3” allowed me to fiddle with that partition #3 (remember, we start counting at zero). I set it’s type to “A6” (OpenBSD) and then took the defaults with the exception of naming it “OpenBSD”. A quick “write” followed by a “quit” allowed me to update my new partition and get back to the installer.

386BSD was last released back in 1994 with a series of articles in Dr. Dobb's Journal -- but then developers for this BSD-based operating system started migrating to both FreeBSD and NetBSD. An anonymous Slashdot reader writes: The last known public release was version 0.1. Until Wednesday, when Lynne Jolitz, one of the co-authors of 386BSD, released the source code to version 1.0 as well as 2.0 on Github.

386BSD takes us back to the days when you could count every file in your Unix distribution and more importantly, read and understand all of your OS source code. 386BSD is also the missing link between BSD and Linux. One can find fragments of Linus Torvalds's math emulation code in the source code of 386BSD. To quote Linus: "If 386BSD had been available when I started on Linux, Linux would probably never had happened.”

Though it was designed for Intel 80386 microprocessors, there's already instructions for launching it on the hosted hardware virtualization service Qemu.

“Jetpack is an experimental and incomplete implementation of the App Container Specification for FreeBSD. It uses jails as isolation mechanism, and ZFS for layered storage.”

Turning back the pages to the late 1970’s, Microsoft entered into an agreement with AT&T Corporation to license Unix from AT&T. While the company didn’t sell the OS to public, it licensed it to other OEM vendors like Intel, SCO, and Tandy.

As Microsoft had to face legal trouble due to “Unix” name, the company renamed it and came up with its own Unix distribution. So, AT&T licensed Unix to Redmond that was passed on to other OEMs as Xenix.

It’s interesting to recall a time when Microsoft enabled people to run Unix — an operating system originally designed for large and multiuser systems — on a microcomputer. Even though it came first, Unix was probably more powerful than MS-DOS.

In early 1980’s, IBM was looking for an OS to power its PC. As IBM didn’t want to maintain any ties with the recently split AT&T, Xenix was automatically rejected. To fulfill, the tech giant’s demand, Microsoft bought 86-DOS from Seattle Computer Products and managed to convince IBM to use it in their systems.

Slowly, Microsoft started losing interest in Xenix and traded the full rights of Xenix with SCO, a Xenix partner company. The company filed bankruptcy in 2007 before taking the Xenix legacy to the 21st century in the form of Open Server, previously known as SCO Unix and SCO Open Desktop.

This week on the show, we’ll be talking to Petra about the NetBSD foundation, about how they operate and assist NetBSD behind the scenes. That plus lots of news

The first one went for .$4,200.00

](https://utcc.utoronto.ca/~cks/space/blog/unix/TypingEOFEffects)

“Typing D causes the tty driver to immediately finish a read().”

Normally doing a read() from a terminal is line-buffered inside the tty driver; your program only wakes up when the tty driver sees the newline, at which point you get back the full

Typing D causes the tty driver to stop waiting for a newline and immediately return from the read() with however much of the line has been accumulated to date. If you haven't

(Other programs will immediately process the partial line with no buffering; cat is one example of this.)

Once you've typed D on a partial line, that portion of the line is immutable because it's already been given to the program. Most Unixes won't let you backspace over such partial

(Note that modern shells are not good examples of this, because they don't do line-buffered input; to support command line editing, they switch terminal input into an uninterpreted

““Kind of implemented” means that I was able to mount directory, traverse it, read file, calculate md5 sums and compare with host’s md5sum, create/remove directories,

I hit a roadblock with rename VOP: it involves some non-trivial locking logic and also there is a problem with cached paths. VBox hypervisor operates on full paths so we cache them

This week on BSDNow, we’re going to be hearing about Allan’s trip to EuroBSDCon, plus an Interview about “Bro on BSD”! Stay tuned, for your place to

I think, my main help here was a USB OTG driver, which I wrote before for another device (Ralink RT3052), then port it to R-Pi. But it was rewritten by Hans Peter Selasky. I do not know so much about USB as Hans knows.

Another useful part of my help is Xorg support. I did a simple Xorg video driver which uses framebuffer exported by virtual terminal subsystem. That is help to many guys to start use RPi as a simple desktop system.

FreeBSD is very powerful as a network server. All modern network features in one box, with very fast processing.

Another good side of FreeBSD is modularity. It is not required to write code to use some driver that was already written for another system, you can just define it in configuration files (kernel config, kernel hints, FDT). So if you want build a nice, R-Pi based, home server – use FreeBSD. If you want to play with devices attached to R-Pi’s GPIO – use FreeBSD.

The TrueOS desktop installer is basically the same as from the PC-BSD days, just re-branded. Still one of the easiest BSD graphical installers I've dealt with and makes it a breeze for setting up a FreeBSD-on-ZFS system by default.

This week on BSDNow, Allan is currently at EuroBSDCon! However due to the magic of video (or time travel), you still get a new episode. (You’re Welcome!). Stay tuned

This week on BSDNow! We’ve got Netflix + FreeBSD news to discuss, always a crowd pleaser, that plus EuroBSDCon is just around the corner. Stick around for your place

Since the beginning of the Open Connect program we have significantly increased the efficiency of our OCAs - from delivering 8 Gbps of throughput from a single server in 2012 to over 90 Gbps from a single server in 2016. We contribute to this effort on the software side by optimizing every aspect of the software for our unique use case - in particular, focusing on the open source FreeBSD operating system and the NGINX web server that run on the OCAs.

In the modern internet world, we have to focus not only on efficiency, but also security. There are many state-of-the-art security mechanisms in place at Netflix, including Transport Level Security (TLS) encryption of customer information, search queries, and other confidential data. We have always relied on pre-encoded Digital Rights Management (DRM) to secure our video streams. Over the past year, we’ve begun to use Secure HTTP (HTTP over TLS or HTTPS) to encrypt the transport of the video content as well. This helps protect member privacy, particularly when the network is insecure - ensuring that our members are safe from eavesdropping by anyone who might want to record their viewing habits.

The goal is to ensure that your government, ISP, and wifi sniffing neighbour cannot tell which Netflix videos you are watching

Netflix Open Connect serves over 125 million hours of content per day, all around the world. Given our scale, adding the overhead of TLS encryption calculations to our video stream transport had the potential to greatly reduce the efficiency of our global infrastructure.

We evaluated available and applicable ciphers and decided to primarily use the Advanced Encryption Standard (AES) cipher in Galois/Counter Mode (GCM), available starting in TLS 1.2. We chose AES-GCM over the Cipher Block Chaining (CBC) method, which comes at a higher computational cost. The AES-GCM cipher algorithm encrypts and authenticates the message simultaneously - as opposed to AES-CBC, which requires an additional pass over the data to generate keyed-hash message authentication code (HMAC). CBC can still be used as a fallback for clients that cannot support the preferred method.

All revisions of Open Connect Appliances also have Intel CPUs that support AES-NI, the extension to the x86 instruction set designed to improve encryption and decryption performance. We needed to determine the best implementation of AES-GCM with the AES-NI instruction set, so we investigated alternatives to OpenSSL, including BoringSSL and the Intel Intelligent Storage Acceleration Library (ISA-L).

Netflix and NGINX had previously worked together to improve our HTTP client request and response time via the use of sendfile calls to perform a zero-copy data flow from storage (HDD or SSD) to network socket, keeping the data in the kernel memory address space and relieving some of the CPU burden. The Netflix team specifically added the ability to make the sendfile calls asynchronous - further reducing the data path and enabling more simultaneous connections. However, TLS functionality, which requires the data to be passed to the application layer, was incompatible with the sendfile approach.

To retain the benefits of the sendfile model while adding TLS functionality, we designed a hybrid TLS scheme whereby session management stays in the application space, but the bulk encryption is inserted into the sendfile data pipeline in the kernel. This extends sendfile to support encrypting data for TLS/SSL connections.

We tested the BoringSSL and ISA-L AES-GCM implementations with our sendfile improvements against a baseline of OpenSSL (with no sendfile changes), under typical Netflix traffic conditions on three different OCA hardware types. Our changes in both the BoringSSL and ISA-L test situations significantly increased both CPU utilization and bandwidth over baseline - increasing performance by up to 30%, depending on the OCA hardware version. We chose the ISA-L cipher implementation, which had slightly better results. With these improvements in place, we can continue the process of adding TLS to our video streams for clients that support it, without suffering prohibitive performance hits.

Recent events have rocked the mobile computing world to its core. OpenBSD retired the zaurus port, leaving users in desperate need of a new device. And not long before that, Microsoft released the Anniversary Update to Windows 10, but with free space requirements such that it’s nigh impossible to install on cheap 32GB eMMC equipped devices such as the HP Stream series, leaving users searching for a new lightweight operating system. With necessity as both mother and father, the scene is set for a truly epic pairing. OpenBSD on the HP Stream 7.

The HP Stream line is a series of budget computers in a couple form factors. The Stream 11 is a fairly typical netbook. However, the Stream 7 and 8 are tablets. They look like cheap Android devices, but inside the case, they’re real boys, er PCs, with Intel Atom CPUs.

To install OpenBSD on such a device, we need a few parts. Obviously, the tablet itself. There’s a dearth of ports on these things, but there is a micro USB port. Attaching anything useful requires an OTG “on the go” cable that creates a type A port. Attaching more than one useful thing requires a mini hub. And completing the install requires one each USB stick, keyboard, and network adapter.

First, we need to prep the machine to boot from USB. Actually, before doing anything, make sure you have a full charge. It’s going to be battery only from here on out. Plug everything in. Flash drive, keyboard, and network into the hub, hub into the OTG cable, cable into the port on top of the Stream.

Turn on the machine while holding the volume down button. This launches a mini menu from which we can enter the BIOS. There’s a little on screen keyboard in the corner, so this can be done even without a keyboard attached, but the USB keyboard should work. We need to change two settings in the boot section. First, turn off secure boot. Second, switch boot order to prefer USB. Save and exit. The first reboot reveals a confirmation screen checking that we really want to disable secure boot. We must enter a PIN and press enter. Enter the PIN shown on the screen and press enter. And we are go.

Of course, there’s no X at this point. And definitely no touch screen. And no internal networking. However, by keeping our USB hub attached, we can drive the console and access the network. At least until the battery is depleted, even if we have no way of knowing how long that will be since we disabled all the ACPI devices, which also means no suspend or resume.

The Chromebook Pixel LS (2015) has an Intel Core i7 processor (Broadwell) at 2.4Ghz, 16Gb of RAM, a 2560x1700 400-nit IPS screen (239ppi), and Intel 802.11ac wireless. It has a Kingston 64Gib flash chip, of which about 54Gib can be used by OpenBSD when dual-booting with a 1Gb Chrome OS partition.

UNIX systems have two classes of user, the super user and regular users. The super user is super, and everybody else is not. This concentration of power keeps things simple, but also means that often too much power is granted. Usually we only need super user powers to perform one task. We would rather not have such power all the time. Think of the responsibility that would entail! Like the sudo command, doas allows for subdivision of super user privileges, granting them only for specific tasks.

I live in China. There, I can act as a bridge between Chinese companies and the FreeBSD community to help drive FreeBSD adoption. Through my leadership role in the FreeBSD Foundation, I will help promote FreeBSD in China and also represent the Foundation at conferences and events in my region.

I joined Microsoft and started to lead the project called FreeBSD Integration Service to get FreeBSD running well on Hyper-V and Azure. To promote our work and to understand the FreeBSD ecosystem, I started to participate in FreeBSD events where I was inspired by this technical community.

The patches I submitted to make ACPI and input devices work on that laptop led to a src commit bit in 2004. While I haven’t worked on ACPI or input devices since, I have been contributing to different areas of the kernel. Taking up maintainership of some ports I cared about also got me a ports commit bit after some time.

Primarily, I think I can be useful! I attend (and organize) a number of conferences around the world every year, particularly in regions that have a mostly “stealthy” FreeBSD community. While I clearly don’t need to be on the FreeBSD Foundation board to advocate for FreeBSD, joining as a director will provide an additional asset when working in areas of the world where organizational affiliations are meaningful.

Despite intending to eventually contribute their code to the FreeBSD Project as open source, many hardware vendors still find it very difficult to engage directly with the FreeBSD development community. The Foundation helps bridge that gap and helps facilitate collaboration between commercial vendors and the FreeBSD community.

I hope to make FreeBSD more visible in regions of the world where it is historically under-represented. I expect I will be attending even more conferences and getting myself invited to even more organizations.

In the beginning, by which we mean V7, Unix didn't have a pager at all. That was okay; Unix wasn't very visual in those days, partly because it was still sort of the era of the hard copy terminal. Then along came Berkeley and BSD. People at Berkeley were into CRT terminals, and so BSD Unix gave us things like vi and the first pager program, more (which showed up quite early, in 3BSD, although this isn't as early as vi, which appears in 2BSD). Calling a pager more is a little bit odd but it's a Unix type of name and from the beginning more prompted you with '--More--' at the bottom of the screen.

All of the Unix vendors that based their work on BSD Unix (like Sun and DEC) naturally shipped versions of more along with the rest of the BSD programs, and so more spread around the BSD side of things. However, more was by no means the best pager ever; as you might expect, it was actually a bit primitive and lacking in features. So fairly early on Mark Nudelman wrote a pager with somewhat more features and it wound up being called less as somewhat of a joke.

In a sane world, Unix vendors would have either replaced their version of more with the clearly superior less or at least updated their version of more to the 4.3 BSD version. Maybe less wouldn't have replaced more immediately, but certainly over say the next five years, when it kept on being better and most people kept preferring it when they had a choice.”

Oddly, FreeBSD has done the most sensible thing; they've outright replaced more with less. There is a /usr/bin/more but it's the same binary as less and as you can see the more manpage is just the less manpage. OpenBSD has done the same thing but has a specific manpage for more instead of just giving you the less manpage.

So, now you can see why I say that less is more, or more, or both, at several levels. less is certainly more than more, and sometimes less literally is more (or rather more is less, to put it the right way around).

This week on BSDNow, we’ll be talking to Diane Bruce about using it for Ham Radio Enthusiasts, the RPi3 and much more! That plus all the latest news from the week,

Ham Radio, RPi3 and more!