Loopholes in AI Regulation

Governments will try to control large AI models, once they agree on the details. Their regulations will be irrelevant by then.

Let’s talk about the recent news, AI regulation, and long-term impact for building products with LLM/GPT.

I’ll lead with a joke.

OpenAI: US, we must license and control AI models
EU: everybody must license their AI models
EU: or pay tons of money
OpenAI: not fair!
EU: not our problem

This is exactly what happened recently :)

AI Regulation

Sam Altman (OpenAI CEO) has testified in U.S. Senate. He suggested that  government should consider licensing and testing requirements for AI models.

Meanwhile, the EU is working on an AI Act that also requires licensing of AI models, including Open Source ones. For example, GitHub would be liable for hosting an unlicensed Open Source model. The act has extraterritorial jurisdiction, allowing European governments to seek conflict with American developers and businesses.

What’s so funny about that? Sam Altman said that ChatGPT could leave the EU if it couldn’t comply with the regulations. "The current draft of the EU AI Act would be over-regulating, but we have heard it's going to get pulled back… They are still talking about it,” - he told Reuters.

EU officials didn’t like threats and said that the draft rules are not for negotiation.

While this looks like the stupidity of EU AI committees and double standards for Sam Altman, there is a logic behind that. Each player acts in a way that maximizes their benefits and lowers their risks. This means “slow things down and control them”.

• For Sam Altman, that is to get enough regulation to control innovation and hinder competition.

• For EU regulators, that is to slow down and regulate things that could cause unpredictable outcomes. They are mostly worried about use of generative AI to meddle with elections and spread disinformation.

Different players would have different understandings of what “slow down and control” means, especially the “who controls?” part. This causes misunderstanding and strong statements uttered publicly.

Ultimately they will probably settle their differences and come up with a set of mutually tolerable regulations. Open-sourcing large AI models could even become illegal.

Loopholes in AI Regulations

These regulations will have little impact.

Why? As I wrote earlier:

when every student, entrepreneur, and researcher is enthusiastic about the tech and wants to play with it on owned hardware, they will find a way. It will be clever, scrappy, and full of horrible hacks. But it will work.

Any regulation will become just another complication that needs to be worked around. In fact, there are good work-arounds already.

For example. Let’s apply regulations to models with 7B parameters and above. Sure, everybody will be making models with 6999999 parameters. Researchers will finally start improving model efficiency while keeping parameter counts the same.

Set a limit on model capabilities? Sure, just use in your products a dozen of tiny specialised models that each do their own task extremely well. Use really powerful AI models internally for training, but release only a collection of dumb and specialised models.

Make distribution of all unlicensed models completely illegal? Not a big problem either. While training of a large foundational model from the scratch can be expensive, fine-tuning of existing models can be done on a commodity hardware. You just need a GPU, good dataset and a training script.

Preparing training datasets here is the most time-consuming part. They are probably not going to be regulated by any acts that are coming up soon. So community will publish datasets and let consumers handle the final fine-tuning step.

Another approach, is to distribute model patches (relatively small in case of LoRA and QLoRA) that get applied to the foundational model in order to create a fine-tuned version.

This is exactly how explosion of LLaMA-based models happened. All these public models are distributed via patches.

For example, these are the instructions on setting up a popular Vicuna model:

We release Vicuna weights as delta weights to comply with the LLaMA model license. You can add our delta to the original LLaMA weights to obtain the Vicuna weights.

Find a way to clamp down model complexity altogether? Watch startups and businesses evacuate to the UK, Singapore, Africa, or China.

I wrote about the rate of innovation in the industry. Governments and large organisations can’t move that fast. They need months and even years to draft a paper and agree upon that. By the time a regulation is official, technology is already light-years ahead, making the regulation stale and irrelevant (and any damage - irreversible).

Impact of Irrelevant AI Regulations

Assuming that regulators will try to control easy targets (large models), while trying to maintain the looks of “supporting innovation”, here what is about to happen:

• The Cambrian explosion will continue. Models will become smaller and more specialised.

• Large powerful foundational models will be used internally as a tool to train and generate specialized models (aka “let’s scrape ChatGPT4 dialogues and train Vicuna”). You keep the large model to yourself and release only the tiny ones.

• Better technology to distill large models into small fine-tuned ones.

Obviously, I’m way out of my league with predictions like that. But let’s just wait and see what happens in the next months.

Meanwhile, here are some recent news to illustrate the trend:

• QLoRA technique that combines LoRA with a clever 4-bit quantisation. This allows us to take large language models, fine-tune and run them on commodity hardware. Details and source code are already available. The preliminary results look good.

• Google shared a new approach of using Reinforcement Learning to train a dedicated dialogue manager. This specialised model helps to create chatbots that think a few steps ahead while talking to a human, leading to more satisfactory open-ended conversations.

• Meta/Facebook released a new massively multilingual model. It can recognize 4000 languages, do text-to-speech and speech-to-text in 1100 languages. Works better than Whisper. Researchers of niche languages are already looking into creating compact and specialised models out of it.

Long story short, feel free to ignore all news about AI regulations and restricting Open Source AI models. There are still tons of ways around that.

Just gather data for your own domain and start taking a look at fine-tuning small AI models for your tasks.