Underdog joins the fight
Underdog is the Mistral 7B model with unconventionally open source license. I'll also talk about AWS Bedrock, Anthropic, and Cloudflare Workers AI.
How have you been doing?
The last few weeks were full of exciting announcements. Here are a few most important highlights, relevant for product development.
Anthropic on AWS Bedrock
Anthropic partners with Amazon in a $4B deal, while Amazon Bedrock becomes available. This is a big deal, since you can now access second best LLM model through an API. There are no waitlists to join. Companies with allocated AWS budgets can start prototyping right away.
In addition to that, Anthropic offers fine-tuning of Claude model on AWS Bedrock. Unlike ChatGPT fine-tuning this looks like a real tuning, since you have to pay to keep the tuned model running.
Cloudflare Workers AI
Cloudflare announced Workers AI - ability to run low-power LLM models in datacenters all around the world. They are playing to the GDPR game here:
We’re also excited to support data localization in the future. To make this happen, we have an ambitious GPU rollout plan - we’re launching with seven sites today, roughly 100 by the end of 2023, and nearly everywhere by the end of 2024. Ultimately, this will empower developers to keep delivering killer AI features to their users, while staying compliant with their end users’ data localization requirements.
Normally that wouldn’t be such a big deal, since they are targeting only tiny 7B models like meta/llama-2-7b-chat-int8. This model is relatively weak and also compressed (quantised). However, coupled with the next announcement, there is a potential.
Mistral 7B - Surprisingly powerful
The French AI company Mistral has released a small 7B model that was also named Mistral.
They managed to push state of the art for small models even further!
First of all, capabilities of Mistral 7B outperform not only 13B models on product benchmarks, it also states competitive with 70B models. All of that, while being much cheaper to run.
Below is the preview of the upcoming Trustbit LLM Benchmark for October 2023.
I have already an independent confirmation of Mistral LLM outperforming Llama 13B Nous Hermes on information retrieval (RAG) tasks.
Second feature of Mistral is its license. You see, LLM Benchmark used to be dominated by two types of the models:
Cloud-hosted models with proprietary licenses (e.g. ChatGPT and Claude)
Models that you can run locally, based on Llama 2 (pretty much everything else).
However, Llama 2 has an interesting license. It is almost open source, coming with a few limitations:
you can’t use output of this model to train any other models;
companies with more than 700 million monthly users need a special permission to use it.
Meanwhile Mistral AI model has a familiar Apache 2.0 license, that is also used by software like Kubernetes, Kafka, and Docker. This license is already approved by the legal department of many companies, so the path to adoption will be more smooth.
There you have it: a small and powerful model with a true open-source license that can run on edge servers around the world at Cloudflare servers. I hope this puts more competitive pressure on the other model vendors.
ChatGPT goes multi-modal
What about ChatGPT and its multi-modal functionality? Isn’t it exciting to ask questions about images, and get good answers?
This is exciting but practically irrelevant, since the feature is limited only to the OpenAI applications. You can’t include these features in your own products, yet. There is no promise of this capability being available through the API any time soon.
If you already need multi-modal capabilities for LLMs, then you can get quite far with a combination of LLM and fine-tuned computer vision models. It even is easier than it sounds.
New content in ML Product Labs
As a subscriber of this newsletter you also have access to ML Product Labs. There is a new bit of content there: a catalogue of the most interesting applications of LLMs in products that I have been involved with (and can vouch for).
Thanks for reading ML Under the Hood! Subscribe for free to receive new posts and support my work.