8 years of cost reduction in 5 weeks: how Stanford’s Alpaca model changes everything, including the economics of OpenAI and GPT 4. The breakthrough, using self-instruct, has big implications for Apple’s secret large language model, Baidu’s ErnieBot, Amazon’s attempts and even governmental efforts, like the newly announced BritGPT.
I will go through how Stanford put the model together, why it costs so little, and demonstrate in action versus Chatgpt and GPT 4. And what are the implications of short-circuiting human annotation like this? With analysis of a tweet by Eliezer Yudkowsky, I delve into the workings of the model and the questions it rises.
Web Demo: https://alpaca-ai0.ngrok.io/
Alpaca: https://crfm.stanford.edu/2023/03/13/alpaca.html.
Ark Forecast: https://research.ark-invest.com/hubfs/1_Download_Files_ARK-I…_Final.pdf.
Eliezer Tweet:
I don’t think people realize what a big deal it is that Stanford retrained a LLaMA model, into an instruction-following form, by **cheaply** fine-tuning it on inputs and outputs **from text-davinci-003**.
It means: If you allow any sufficiently wide-ranging access to your AI…
— Eliezer Yudkowsky ⏹️ (@ESYudkowsky) March 14, 2023
Dear silly Internet saying “but knowledge distillation is already known”: the key idea here is that the fine-tuning above the base model is comparatively much *easier* and *cheaper* to extract and re-imbue, if you have a new base comparable to the earlier base model.
(Spelling…
— Eliezer Yudkowsky ⏹️ (@ESYudkowsky) March 14, 2023
Self-Instruct: https://arxiv.org/pdf/2212.10560.pdf.
InstructGPT: https://openai.com/research/instruction-following.
OpenAI Terms: https://openai.com/policies/terms-of-use.
MMLU Test: https://arxiv.org/pdf/2009.03300.pdf.
Apple LLM: https://www.nytimes.com/2023/03/15/technology/siri-alexa-goo…gence.html.
GPT 4 API: https://openai.com/pricing.
Llama Models: https://arxiv.org/pdf/2302.13971.pdf.
BritGPT: https://www.theguardian.com/technology/2023/mar/15/uk-to-inv…wn-britgpt.
Amazon: https://www.businessinsider.com/amazons-ceo-andy-jassy-on-ch…?r=US&IR=T
AlexaTM: https://arxiv.org/pdf/2208.01448.pdf.
Baidu Ernie: https://www.nytimes.com/2023/03/16/world/asia/china-baidu-chatgpt-ernie.html.
PaLM API: https://developers.googleblog.com/2023/03/announcing-palm-ap…suite.html.
Comments are closed.