Aussie’s Matilda model - A private sector crack at building a homegrown LLM

Maincode, the Sydney-based AI startup, has unveiled “Matilda,” what it hails as the country’s first foundation model built from scratch, and a bold play to ensure Australia’s AI future isn’t decided in Silicon Valley boardrooms.

“We have created Matilda, which is the first from scratch foundation model in Australia,” Maincode CEO and co-founder Dave Lemphers told an industry audience at the Tech Leaders forum in the Hunter Valley on Monday. Maincode is backed by Ed Craven, the founder of online casino and sports betting operator Stake.com.

“I think that it’s important for Australia to participate. Not only is it something we should be creating, it’s something that is good for Australia,” said Lemphers, a software engineer and startup founder who previously worked for Microsoft.

Maincode’s approach, he says, is about building models “done by Australians” and “here in Australia,” while critically maintaining transparency and control over the data and processes involved.

“It doesn’t mean that it’s wholly sourced by information available in Australia,” he explained.

“So part of training Matilda is training Matilda on a number of different sources, both globally and locally. The difference is that it’s not done post-training, so we’re not taking open weights and biases. We’re doing that from first principles. So we’re taking the data in its raw form. We’re curating, filtering through it. We’re selecting exactly what we want from those data sources, and then combining that in the pre-training phase,” he explained.

Maincode’s Dave Lemphers

Built to avoid the pitfalls of global rivals

Maincode’s timing follows a global push by the likes of OpenAI and Google to dominate the next era of AI. But Lemphers argues that Maincode’s “model factory” approach, which enables the rapid production of multiple, customisable LLMs, sets it apart from those firms, and from earlier local experiments that fizzled.

“We are the first true model factory in Australia with a very clear go-to-market and business model on how to develop LLMs from scratch, both for ourselves in terms of the Matilda foundation model, but also working with businesses in Australia to help them develop their own sovereign LLMs from scratch,” Lemphers said.

“That’s the differentiator, the commitment of capital, talent and the will and vision to make it a sustainable business.”

Unlike attempts to grab headlines with ever-larger models, Maincode emphasises focus and utility, said Lemphers.

“It’s our opinion that trying to build a massive, large language model actually doesn’t have the same economic advantage as building smaller, task-specific models. So we tend to focus our time on what actual business cases and even opportunities to advance Australia from a productivity perspective,” he said.

Eventually a ChatGPT-style Matilda chatbot will be made available for everyday consumer use.

Transparency, security, and control

Lemphers drew a sharp contrast between Maincode’s transparent, from-the-ground-up approach and the “black box” nature of many so-called open-source models, such as Meta’s Llama models.

“The challenge with open source is not a lot of it is truly open source. As a model builder and model factory, you kind of have to just build everything from scratch,” he said, warning of “a potential to be a time bomb inside” third-party models.

“If a user was to jailbreak or bypass the guardrails, they could potentially do some quite dangerous things. And again, I think that’s why it’s important to be working with sovereign entities who are held to local legal standards and are building locally in the community, because there’s just more transparency and more capability that way.”

Lemphers said earlier this month that companies like Maincode should pay to access content to train their models on, in contrast to the approach favoured by most LLM companies in the US, which argue that training data isn’t subject to copyright law.

Maincode has invested in its own GPUs infrastructure to develop and train its models, and Maincode claims on its website that when it comes to training data, “every piece is sourced ethically and legally, creating a dataset that is clean, representative, and truly ours”. But is the AI talent pool available in Australia to scale up LLM development?

Australia’s brain drain to Big Tech is well documented, but Lemphers believes Maincode’s mission and scale will attract the nation’s best. “We do have a huge advantage. We have an incredibly capable workforce. We have some of the smartest people coming out of PhD and advanced degree programmes, so much so that they end up going and working with companies like OpenAI and Meta,” he said.

“What’s really exciting about Maincode is we’re the only option, where everyone who comes gets to work on things you really can’t work on anywhere else.”

While the vision is ultimately to provide “a large language model capable of meeting the needs of the country”, Lemphers is pragmatic about the journey ahead.

“That’s a very long, resource-intensive process that can take us some years to get there,” he said. “In the meantime, what we are hoping is to work with Australian businesses to deploy smaller, far more capable, sovereign models that deal with the day-to-day tasks that actually matter to Australians.”

Matilda is open for beta access and is expected to be demoed at SXSW Sydney in October.