Griffin on Tech: Crowdstruck techies and Zuck’s open source AI gamble

This time last Friday, IT teams all over the world were looking forward to the end of the working week.

Then came the content update from cybersecurity firm Crowdstrike, and a sea of blue screening Windows machines throwing many of the world’s largest companies into chaos.

While fortunate timing with the rouge update doing its damage towards the end of the working week here lessened the impact, I’ve been talking to plenty of IT workers this week who had to scramble on Friday evening to manually apply fixes to machines running servers.

Crowdstrike yesterday published a detailed post-mortem of the event and what caused it, pinning the blame on a bug in test software for not properly validating the content update to its Falcon endpoint threat detection software. A 40-kilobyte sensor content file is what caused all the trouble.

Crowdstrike is now vowing to more thoroughly test its content updates, and improve its error handling. Perhaps the biggest criticism of the company from the tech community is that Crowdstrike was pushing out blanket updates across the world, all in one go. 

Interacting as it does with the Windows kernel, that was a high-risk approach. Crowdstrike has taken that feedback onboard and will implement a staggered deployment for future updates to avoid a similar disaster being felt across the world. Crowdstrike released two content updates last week, which it calls “Template Instances”.

“Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data,” says CrowdStrike.

Oops! Crowdstrike is also undertaking to “provide customers with greater control over the delivery of Rapid Response Content updates by allowing granular selection of when and where these updates are deployed”.

A lot of other software companies should be using this as an opportunity to look at their own software hygiene and how they test, validate and deploy software updates. Is Crowdstrike out of the woods? No way. 

Its move to give staff and firms they work with a $10 UberEats voucher to say sorry for the disruption shows how tone-deaf aspects of their response to the crisis has been. Many customers who had their Windows machines rendered inoperable by Crowdstrike’s faulty update will be seeking something more substantial as compensation for the cancelled flights, inability to process electronic payments, and delays to public health services that resulted last Friday.

What recourse to they have? It may be fairly limited according to Russell McVeagh, which issued an advisory bulletin on the incident.

“What recourse will be available will depend on the details of the cause of the outage and what more could have been done to avoid it occurring,” the law firm pointed out.

“The majority of customers will have contracted on supplier standard terms which typically provide minimal potential recourse for losses incurred as a result of outages of this nature. However, larger-scale organisations may have more contractual rights.”

It could go down as a test case in how serious disruption caused by a problem in the software supply chain are dealt with in the aftermath from a legal perspective. Expect to see a class action lawsuit or two in the highly litigious USA.

Meta goes big on opensource AI

A tanned and shaggy-haired Mark Zuckerberg interrupted his superyacht holiday this week to announced a new update to Meta’s Llama 3 large language model (Llama 3.1 405B) which like its predecessors is open source, so is freely available for anyone to deploy and use.

Zuckerberg has become an unlikely evangelist for open source software. This week he said AI models should follow the same path as the Linux operating system. It’s a marked departure from the closed, walled-garden approach that has characterised Facebook/Meta’s history.

“Today, Linux is the industry standard foundation for both cloud computing and the operating systems that run most mobile devices – and we all benefit from superior products because of it,” he wrote.

An open source push gives Meta clear differentiation from rivals like OpenAI and Google and should be of interest to New Zealand organisations looking to train their own models. As Zuckerberg points out:

“Many organisations don’t want to depend on models they cannot run and control themselves. They don’t want closed model providers to be able to change their model, alter their terms of use, or even stop serving them entirely,” he wrote. 

“They also don’t want to get locked into a single cloud that has exclusive rights to a model. Open source enables a broad ecosystem of companies with compatible toolchains that you can move between easily.”

What’s in it for Meta going open source?

As Zuckerberg points out, Llama needs to develop into an “ecosystem of tools” and getting the crowd to build them is more efficient. But he’s clearly still smarting from Apple’s changes to iOS14 that required apps to ask people for permission before tracking their activity across the web. That move cost Meta billions of dollars. 

“Between the way [Apple] tax developers, the arbitrary rules they apply, and all the product innovations they block from shipping, it’s clear that Meta and many other companies would be freed up to build much better services for people if we could build the best versions of our products and competitors were not able to constrain what we could build,” Zuckerberg wrote.

His approach to business suggests there’ll always be more for Meta than anyone else in whatever business moves he makes. But taking the open source path also gives Meta a big point of differentiation with its rivals and the ability to leverage the wisdom of the open source community.