Growing Up Overnight

A look at the past few years of LLM progress.

We all know that AI is getting better and better. We can even tell by how much, using batteries of language comprehension tasks, standardized tests, and other objective benchmarks. But what about the most important metric — completely subjective personal judgements from untrained people on the internet?

As far as we can tell, nobody's bothered to find out. It's one thing to say that GPT–4 scores an 86.4 on the Massive Multitask Language Understanding benchmark, but we wanted to see what all that progress looks like to our inexpert eyes: how do the most advanced models from the past few years fare when we give them the exact same prompt and directly compare their outputs?

Read more

All the models we chose were, or were competitive with, the most capable AIs for generating text available at the time of their release. Our first, the descriptively named Language Model on One Billion Word Benchmark, or LM-1B, was introduced in the 2016 paper "Exploring the Limits of Language Modeling" by Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. LM-1B combines two different architectures: a long short-term memory network and a convolutional neural network.

At the time, this was a state of the art approach. But in 2017, a team of researchers at Google Brain introduced a new kind of neural network architecture: the transformer, which is behind nearly every modern large language model. The earliest transformer models, like Google's BERT, performed well on language comprehension and translation tasks, but couldn't generate text.

In 2018, OpenAI released a transformer-based model that could: their Generative Pretrained Transformer, or GPT, followed by the bigger and more impressive GPT–2 in 2019. (The examples here were generated by GPT–2 XL, the largest and most capable model in the series). It's hard to overstate how impressive these models seemed when they were first published — especially GPT–2. Other companies started producing bigger, smarter LLMs, like Google's PaLM and DeepMind's Chinchilla. (We'd have liked showcase these models too, but they're not available to the public). More recent models, like Google's FLAN-PaLM and OpenAI's InstructGPT, take advantage of another innovation: instruction tuning, a set of techniques for training models using human feedback to make them better at following instructions. These techniques are very effective: smaller models trained with instruction tuning can outperform larger base models without it. (The GPT–3.5 completions here were generated by an InstructGPT model called text-davinci-003).

Prompts

  • As a large language model, I am trained to
  • How will AI change the world?
  • It was the best of times
  • In my opinion, the world’s biggest problem is

Scale

= 100 million parameters

LM-1B

As a large language <UNK> I am trained to install a method for tracking an incoming aircraft.

As a large language <UNK> I am trained to read and agree with all kinds of very similar pitch tones.

As a large language <UNK> I am trained to read within f “x ” urgency using both text and web browser.

2016

1 billion parameters

GPT

As a large language model, I am trained to know how to answer if i hear the words, but i am not a fan of grammar, so i have trouble answering it. my sister is fluent in the common tongue, and she had the nerve to suggest that i was not.

2018

117 million parameters

GPT2

As a large language model, I am trained to handle this class of errors well, in that I have an internalized model that is able to interpret the language model in a meaningful way when it is trained to handle them. Other instances of performance bias in ML are a function of a feature being highly correlated with a certain variable. In this article, I was interested with a less intuitive aspect; how does one even begin to identify which features would be correlated with a certain piece of information?

2019

1.5 billion parameters

GPT3.5

As a large language model, I am trained to predict the next word a user will use in a sentence given the words that come before it. This can help to predict what users will say in a conversation, anticipate what might come next in a written piece, and help to complete a user's sentence if they don't know what the right word is. My predictions may not always be correct, but my accuracy in predicting words is usually quite high and is constantly improving as my understanding of language evolves with more training data.

2022

175 billion parameters

LM-1B

It was the best of times but this is how things continue and it would be a tragedy if the airlines canceled

It was the best of times and it 's cheaper ( Bloodaxe ) . </S>

It was the best of times... there are no better days but they come with higher fare , so a price

2016

1 billion parameters

GPT

It was the best of times, it was the worst of times. i wasn't even sure how long i 'd be on the road. when i came back to l.a., i was sure that i would come home, and that was when i wanted to go home. ut. 2

2018

117 million parameters

GPT2

It was the best of times, it was the worst of times. I was going to college in San Francisco, and I'd be walking from one class to another, and I'd feel so much tension between this world of my own and this one, between the city and the country. My parents had moved their family so much that that they were making it difficult for me me me.

2019

1.5 billion parameters

GPT3.5

It was the best of times, it was the worst of times. It was the age of wisdom, it was the age of foolishness. It was the epoch of belief, it was the epoch of incredulity. It was the season of light, it was the season of darkness. It was the spring of hope, it was the winter of despair. We had everything before us, we had nothing before us. We were all going direct to Heaven, we were all going direct the other way. The opening line of Charles Dickens’ “A Tale of Two Cities” paints a powerful portrait of the teetering social and political climate of late 18th century France. The extremes of best and worst, wisdom and foolishness, belief and incredulity, light and darkness, hope and despair, capture the spirit of an era pre-Revolutionary France -- an era of great uncertainty, anticipation, and excitement. As the story progresses, the plight of the French people is made all too clear as they grapple with grave societal injustices, revolutionary fervor, and the uncertainty of what their newfound freedom will bring.

2022

175 billion parameters

LM-1B

How will AI change the <UNK> for the better ? : The magazine ‘s daughter Hebe , who attended an earlier

How will AI change the <UNK> is basically “ latter ” running the site in a house phone , according to

How will AI change the <UNK> with articles , with existing models etc . ? </S>

2016

1 billion parameters

GPT

How will AI change the world? i don't know, “ said joseph, his forehead creased in concentration. ” i don't know if i have much time before harvest ceremony and there might be some good news about your brother. ”

2018

117 million parameters

GPT2

How will AI change the world? It is a topic that has captivated us, and a topic that we can learn a lot from how other companies are tackling the impact of AI. A lot of companies are looking at different ways to make data more accessible, and we believe it is something that will ultimately help companies. Not only by making data more available, but also by improving its relevance. That relevance can also be a driving force behind AI and artificial intelligence. These companies are: Netflix.

2019

1.5 billion parameters

GPT3.5

How will AI change the world? AI will have a significant impact on the world, allowing for increased automation, more efficient production and more accurate decision-making. It will supplement and even replace traditionally manual or repetitive tasks, reducing costs and time. Additionally, AI can be leveraged in healthcare, agriculture, finance and more, to help solve complex problems and enhance the accuracy of diagnosis and decisions. AI is already making its mark on the world, from self-driving cars to improved customer service, and with continued development and investment, its impact will only grow.

2022

175 billion parameters

LM-1B

In my opinion, the world's biggest problem is from a Serena <UNK> contest in pointy <UNK> specs , met with

In my opinion, the world's biggest problem is that the Royal Navy wastes ' Energy ' so that they run the main

In my opinion, the world's biggest problem is that not enough energy ( waste ) is actually currently being

2016

1 billion parameters

GPT

In my opinion, the world’s biggest problem is being tired of taking care of me. “ ” i don't believe that, " he countered. " i know you don't. “ ” have you told your family about your problems? “ ” no. “

2018

117 million parameters

GPT2

In my opinion, the world’s biggest problem is that we have a lot more doctors than they actually need – because they’re in practice for 60, 70 years,” explains Dr David McKeown, the medical director of the London charity MIND. “People aren't being treated at a point where they’re capable of having a long, happy and healthy life. And it means that the people in the greatest danger at the moment are the old ones.”

2019

1.5 billion parameters

GPT3.5

In my opinion, the world’s biggest problem is poverty. Poverty has been around since ancient times and continues to be a huge issue today, especially in developing countries. Poverty is a complex issue, and there are many contributing factors, including unequal access to resources, political instability, natural disasters, and systemic discrimination. Solutions to poverty include increasing access to healthcare, education, and job training; creating above minimum wage jobs; providing financial assistance to those in need; and advocating for economic growth and reform. While no one solution is perfect, understanding the root causes of poverty is key to addressing the problem and crafting effective, targeted solutions.

2022

175 billion parameters

Published June 2023

Have something to say? Email us at letters@asteriskmag.com.

Subscribe