Language models

AI learned
by reading.

ChatGPT. Claude. Language models trained on text.

The orange fell to the ground.
Predict the next word.
See what words can do

If it can write text…

it can write emails, code, and stories.

An email

Friday, sorted.

Hi Alex,

Friday works for me.
See you at 10.

Thanks,
Sam

Some code

Hello, world.

function sayHello(name) {
  return `Hello, ${name}!`;
}
A story

The orange
that rolled away.

It made it to the edge of the table.
Then the world got a lot bigger.

An email, a JavaScript function, and a story made from text.
See why

Because it’s
all text.

duh!

What if AI could watch?
World models

AI can learn
by watching.

World models learn from video.
Just as we learn by watching.

Predict the next picture.

Scroll to see how

What happens next?

Watch. Guess. Learn.

Level 1 · Keep following the direction.Level 2 · Now gravity bends the path.Level 3 · Remember what happens at the floor.

Try it yourself.

Follow the ball behind the block.

Better guesses. A better feel for the world.

Keep going

Good guesses need patterns.

And when you act?

Now add an action.

Your move changes the next picture.

While the room is in view, use Left and Right to turn, Up to step forward, Down to step back, and Space to nudge the ball. Tab moves to the buttons.

Now think bigger

More than a movie.

First, a film.

Make the next picture. Then the next.

Now add a click
The Orange · Runway, 2019 · Flipbook editionSource: Runway

A film. An app. A world. A robot’s view.
All pictures changing over time.

It’s just pixels.

Runway’s bet: learn to simulate pixels, and you can begin to simulate almost anything we see.

Join us Learn more
Read the research
Learning From Watching the WorldPacky McCormick & Anastasis GermanidisIntroducing General World ModelsAnastasis Germanidis · RunwayReal-World SuperintelligenceAnastasis GermanidisThe Pixel EconomyCristóbal ValenzuelaTowards Universal SimulationAnastasis Germanidis

The films and recorded research demonstrations are Runway’s. This story simplifies a research direction; different tasks need additional training. Today’s language models also use images and sound. Reading versus watching is a simple way to compare two approaches to learning.