TechSNAP Video

We continue our take on ZFS as Jim and Wes dive in to snapshots, replication, and the magic on copy on write.

Plus some handy tools to manage your snapshots, rsync war stories, and more!

Links:

• sanoid: Policy-driven snapshot management and replication tools. — Sanoid is a policy-driven snapshot management tool for ZFS filesystems. When combined with the Linux KVM hypervisor, you can use it to make your systems functionally immortal.

Jim and Wes sit down to bust some ZFS myths and share their tips and tricks for getting the most out of the ultimate filesystem.

Plus when not to use ZFS, the surprising way your disks are lying to you, and more!

Links:

• ZFS - Ubuntu Wiki — ZFS is a combined file system and logical volume manager designed and implemented by a team at Sun Microsystems led by Jeff Bonwick and Matthew Ahrens.
• Performance tuning - OpenZFS — Make sure that you create your pools such that the vdevs have the correct alignment shift for your storage device's size. if dealing with flash media, this is going to be either 12 (4K sectors) or 13 (8K sectors).

We break down the ASUS Live Update backdoor and explore why these kinds of supply chain attacks are on the rise.

Plus an update from the linux vendor firmware service, your feedback, and more!

Links:

• Joren Verspeurt on Twitter — The explanation you gave for unsupervised wasn't correct, that was just using a net that was trained in a supervised way. Unsupervised learning doesn't involve labels at all. A good example: clustering. You say "there are x clusters" and it learns a way of grouping similar items.
• Hackers Hijacked ASUS Software Updates to Install Backdoors on Thousands of Computers — The researchers estimate half a million Windows machines received the malicious backdoor through the ASUS update server, although the attackers appear to have been targeting only about 600 of those systems.
• Malicious updates for ASUS laptops — A threat actor modified the ASUS Live Update Utility, which delivers BIOS, UEFI, and software updates to ASUS laptops and desktops, added a back door to the utility, and then distributed it to users through official channels.
• Asus Live Update Patch Now Availabile — Asus has emitted a non-spyware-riddled version of Live Update for people to install on its notebooks, which includes extra security features to hopefully detect any future tampering.
• ASUS response to the recent media reports regarding ASUS Live Update tool attack by Advanced Persistent Threat (APT) groups — ASUS has also implemented a fix in the latest version (ver. 3.6.8) of the Live Update software, introduced multiple security verification mechanisms to prevent any malicious manipulation in the form of software updates or other means, and implemented an enhanced end-to-end encryption mechanism. At the same time, we have also updated and strengthened our server-to-end-user software architecture to prevent similar attacks from happening in the future.
• The Messy Truth About Infiltrating Computer Supply Chains — The Defense Intelligence Agency believed that China’s capability at exploiting the BIOS “reflects a qualitative leap forward in exploitation that is difficult to detect”
• Inside the Unnerving CCleaner Supply Chain Attack — Security researchers at Cisco Talos and Morphisec made a worst nightmare-type disclosure: the ubiquitous computer cleanup tool CCleaner had been compromised by hackers for more than a month. The software updates users were downloading from CCleaner owner Avast—a security company itself—had been tainted with a malware backdoor. The incident exposed millions of computers and reinforced the threat of so-called digital supply chain attacks, situations where trusted, widely distributed software is actually infected by malicious code.
• ShadowPad: How Attackers hide Backdoor in Software used by Hundreds of Large Companies around the World — ShadowPad is an example of how dangerous and wide-scale a successful supply-chain attack can be. Given the opportunities for reach and data collection it gives to the attackers, most likely it will be reproduced again and again with some other widely used software component.
• Gaming industry still in the scope of attackers in Asia — Yet again, new supply-chain attacks recently caught the attention of ESET Researchers. This time, two games and one gaming platform application were compromised to include a backdoor.
• Microsoft Security Intelligence Report Volume 24 is now available — Software supply chain attacks are another trend that Microsoft has been tracking for several years. One supply chain tactic used by attackers is to incorporate a compromised component into a legitimate application or update package, which then is distributed to the users via the software. These attacks can be very difficult to detect because they take advantage of the trust that users have in their software vendors. The report includes several examples, including the Dofoil campaign, which illustrates how wide-reaching these types of attacks are and what we are doing to prevent and respond to them.
• Microsoft Security Intelligence Report Volume 24
• Supply Chain Attacks Spiked 78 Percent in 2018
• Supply Chain Security: A Talk by Bunnie Huang — I recently gave an invited talk about supply chain security at BlueHat IL 2019. I was a bit surprised at the level of interest it received, so I thought I’d share it here for people who might have missed it.
• Attack inception: Compromised supply chain within a supply chain poses new risk — The plot twist: The app vendor’s systems were unaffected. The compromise was traceable instead to a second software vendor that hosted additional packages used by the app during installation. This turned out be an interesting and unique case of an attack involving “the supply chain of the supply chain”.
• Supply Chain Attacks and Secure Software Updates — In general, a supply chain attack involves first hacking a trusted third party who provides a product or service to your target, and then using your newly acquired, privileged position to compromise your intended target.
• Bad USB, Very Bad USB — The best defense for this type of attack is to only use devices that do not have reprogrammable firmware. Outside of this, it is important to only use USB drives that you trust completely, because after plugging in an untrusted device, you will never know if there is an invisible threat running on your computer.
• Reflections on Trusting Trust by Ken Thompson
• LVFS Project Announcement - The Linux Foundation — The Linux Foundation welcomes the Linux Vendor Firmware Service (LVFS) as a new project. LVFS is a secure website that allows hardware vendors to upload firmware updates. It’s used by all major Linux distributions to provide metadata for clients, such as fwupdmgr, GNOME Software and KDE Discover.
• LVFS: Vendor Status
• Two new supply-chain attacks come to light in less than a week — Called “Colourama,” the package looked similar to Colorama, which is one of the top-20 most-downloaded legitimate modules in the Python repository. The doppelgänger Colourama package contained most of the legitimate functions of the legitimate module, with one significant difference: Colourama added code that, when run on Windows servers, installed a Visual Basic script.
• Malicious code found in npm package event-stream downloaded 8 million times in the past 2.5 months

Machine learning promises to change many industries, but with these changes come dangerous new risks. Join Jim and Wes as they explore some of the surprising ways bias can creep in and the serious consequences of ignoring these problems.

Links:

• Microsoft’s neo-Nazi sexbot was a great lesson for makers of AI assistants — What started out as an entertaining social experiment—get regular people to talk to a chatbot so it could learn while they, hopefully, had fun—became a nightmare for Tay’s creators. Users soon figured out how to make Tay say awful things. Microsoft took the chatbot offline after less than a day.
• Microsoft's Zo chatbot is a politically correct version of her sister Tay—except she’s much, much worse — A few months after Tay’s disastrous debut, Microsoft quietly released Zo, a second English-language chatbot available on Messenger, Kik, Skype, Twitter, and Groupme.
• How to make a racist AI without really trying | ConceptNet blog — Some people expect that fighting algorithmic racism is going to come with some sort of trade-off. There’s no trade-off here. You can have data that’s better and less racist. You can have data that’s better because it’s less racist. There was never anything “accurate” about the overt racism that word2vec and GloVe learned.
• Microsoft warned investors that biased or flawed AI could hurt the company’s image — Notably, this addition comes after a research paper by MIT Media Lab graduate researcher Joy Buolamwini showed in February 2018 that Microsoft’s facial recognition algorithm’s was less accurate for women and people of color. In response, Microsoft updated its facial recognition models, and wrote a blog post about how it was addressing bias in its software.
• AI bias: It is the responsibility of humans to ensure fairness — Amazon recently pulled the plug on its experimental AI-powered recruitment engine when it was discovered that the machine learning technology behind it was exhibiting bias against female applicants.
• California Police Using AI Program That Tells Them Where to Patrol, Critics Say It May Just Reinforce Racial Bias — “The potential for bias to creep into the deployment of the tools is enormous. Simply put, the devil is in the data,” Vincent Southerland, executive director of the Center on Race, Inequality, and the Law at NYU School of Law, wrote for the American Civil Liberties Union last year.

We reveal the shady password practices that are all too common at many utility providers, and hash out why salts are essential to proper password storage.

Plus the benefits of passphrases, and what you can do to keep your local providers on the up and up.

Links:

• Plain wrong: Millions of utility customers’ passwords stored in plain text | Ars Technica — In September of 2018, an anonymous independent security researcher (who we'll call X) noticed that their power company's website was offering to email—not reset!—lost account passwords to forgetful users. Startled, X fed the online form the utility account number and the last four phone number digits it was asking for. Sure enough, a few minutes later the account password, in plain text, was sitting in X's inbox.
• The LinkedIn Hack: Understanding Why It Was So Easy to Crack the Passwords | — LinkedIn stated that after the initial 2012 breach, they added enhanced protection, most likely adding the “salt” functionality to their passwords. However, if you have not changed your password since 2012, you do not have the added protection of a salted password hash. You may be asking yourself–what on earth are hashing and salting and how does this all work?
• How Developers got Password Security so Wrong — As time has gone on; developers have continued to store passwords insecurely, and users have continued to set them weakly. Despite this, no viable alternative has been created for password security.
• Adding Salt to Hashing: A Better Way to Store Passwords — A salt is added to the hashing process to force their uniqueness, increase their complexity without increasing user requirements, and to mitigate password attacks like rainbow tables.

Join Jim and Wes as they battle bufferbloat, latency spikes, and network hogs with some of their favorite tools for traffic shaping, firewalling, and QoS.

Plus the importance of sane defaults and why netdata belongs on every system.

Links:

• Why you want QoS - Netdata Documentation — One of the features the Linux kernel has, but it is rarely used, is its ability to apply QoS on traffic. Even most interesting is that it can apply QoS to both inbound and outbound traffic.

Jim and Wes are joined by OpenZFS developer Richard Yao to explain why the recent drama over Linux kernel 5.0 is no big deal, and how his fix for the underlying issue might actually make things faster.

Plus the nitty-gritty details of vectorized optimizations and kernel preemption, and our thoughts on the future of the relationship between ZFS and Linux.

Special Guest: Richard Yao.

Links:

• LinuxFest Northwest 2019 — Join a bunch of JB hosts and community celebrating the 20th anniversary!
• Choose Linux — The show that captures the excitement of discovering Linux.
• Linux 5.0: kernelfpu{begin,end} no longer exported — The latest kernels removed the old compatibility headers.
• ZFS On Linux Landing Workaround For Linux 5.0 Kernel Support — So while these symbols are important for SIMD vectorized checksums for ZFS in the name of performance, with Linux 5.0+ they are not going to be exported for use by non-GPL modules. ZFS On Linux developer Tony Hutter has now staged a change that would disable vector instructions on Linux 5.0+ kernels.
• Re: x86/fpu: Don't export __kernelfpu{begin,end}() — My tolerance for ZFS is pretty non-existant. Sun explicitly did not want their code to work on Linux, so why would we do extra work to get their code to work properly?
• The future of ZFS in FreeBSD — This state of affairs has led to a general agreement among the stakeholders that I have spoken to that it makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf has graciously encouraged me to add FreeBSD support directly so that we might all have a singleshared code base.
• Dephix: Kickoff to The Future — OpenZFS has grown over the last decade, and delivering our application on Linux provides great OpenZFS support while enabling higher velocity adoption of new environments.
• The future of ZFS on Linux [zfs-discuss] —

Honestly, I am beginning to think that my attempt to compromise with mainline gave the wrong impression. I am simply tired of this behavior by them and felt like reaching out to put an end to it. In a few weeks, we will likely be running on Linux 5.0 as if those symbols had never been removed because we will almost certainly have our own substitutes for them. Having to bloat our code because mainline won’t give us access to trivial functionality is annoying, but it is not the end of the world.

We welcome Jim to the show, and he and Wes dive deep into all things Let’s Encrypt.

The history, the clients, and the from-the-field details you'll want to know.

Links:

• Let’s Encrypt and CertBot – JRS Systems
• Automatic Certificate Management Environment (ACME) — The surprisingly readable IETF draft.
• How It Works - Let's Encrypt
• ACME Client Implementations
• Certbot — Certbot is EFF's tool to obtain certs from Let's Encrypt.
• acme-nginx: python acme client for nginx — A particularly simple client that is useful for understanding the protocol details.
• Caddy - The HTTP/2 Web Server with Automatic HTTPS
• mod_md: Let's Encrypt (ACME) support for Apache httpd
• Traefik - The Cloud Native Edge Router
• Looking Forward to 2019 - Let's Encrypt — We’re now serving more than 150 million websites while maintaining a stellar security and compliance track record. Most importantly though, the Web went from 67% encrypted page loads to 77% in 2018, according to statistics from Mozilla. This is an incredible rate of change!
• Let's Encrypt ACME v2 API Announcements — Now that the draft standard is in last-call and the pace of major changes has slowed, we’re able to release a “v2” API that is much closer to what will become the final ACME RFC.
• Let's Encrypt disables TLS-SNI-01 validation — The researcher noticed that "at least two" large hosting providers host many users on the same IP address and users are able to upload certificates for arbitrary names without proving they have control of a domain.
• A Technical Deep Dive on Using Certbot to Secure your Mailserver from the EFF — With the most recent release of Certbot v0.29.1, we’ve added some features which make it much easier to use with both Sendmail and Exim.

Wes and Jim have some great new SNAP in the works, in the meantime Chris stops by to keep you updated and share his favorite "hacker" story of the week.

Wes is joined by a special guest to take a look back on the growth and development of Azure in 2018 and discuss some of its unique strengths.

Special Guest: Chad M. Crowell.

Links:

• Under the sea, Microsoft tests a datacenter that’s quick to deploy, could provide internet connectivity for years
• An Azure Infrastructure Year in Review
• Azure File Sync now generally available
• Microsoft's Newest OS is Based on Linux
• Azure Sphere
• What is Azure Stack?
• Azure Outage Proves the Hard Way Availability Zones are a Good Idea
• Microsoft Azure Infrastructure and Deployment on Linux Academy — In this course, we will cover an introduction to the Azure portal, followed by how to build infrastructure and deploy that infrastructure in real world scenarios.
• Chad Crowell on Twitter