Large Language Models Like ChatGPT Can Become More Efficient

Large language models (LLMs) such as OpenAI's GPT-3 and GPT-4 are becoming increasingly expensive to train and operate, prompting researchers to seek greater efficiency. GPT-3 had 175 billion parameters and cost at least $4.6 million to train; GPT-4 reportedly has about one trillion parameters with development costs above $100 million. Morgan Stanley estimated that processing half of Google search queries with a GPT-like model could cost $6 billion extra per year. The article reviews efficiency approaches: training smaller parameter models on more data, rounding numerical precision to lower hardware demands, low-rank adaptation, Google's teacher-student knowledge distillation, and a chip-aware attention algorithm from Stanford. Researchers at the University of Washington used these techniques to train Guanaco on a single GPU in one day. Meta also updated PyTorch, saying one line of code can double training speed. Experts believe significant architectural improvements are still possible.