How Bramble Sees

A new job for Bramble

It is a quiet morning at the Ashdown Robotics Workshop. Bramble, the workshop's humanoid robot, stands by the long bench with its arms at its sides. Eleanor Price, one of the technicians, has spent months learning how Bramble's motors, joints and balance work. Today Henry Wren, the senior engineer, has a new task for her.

“Put a blue cup on the far bench,” Henry says, “and ask Bramble to fetch it.” Eleanor does. Bramble turns its head slowly, takes a few careful steps, and stops. It does not reach for the cup.

“It walks perfectly well,” says Eleanor. “Why doesn't it go and get the cup?”

“Because it doesn't know where the cup is,” Henry replies. “It knows how to move. It doesn't yet know how to look. That's what you and I are going to teach it, one small step at a time.”

Margaret Ellis, the workshop manager, looks up from her clipboard. “And while you're at it, teach it to notice people. I want it to stop long before it gets near anyone.”

What is computer vision?

Computer vision means getting useful information out of pictures with a computer program. The pictures come from a camera. The information might be “there is a blue cup”, “the cup is 30 cm to the left and 190 cm ahead”, “there is a crate in the way”, or “a person is standing in front of me”.

A robot uses vision to answer questions about the world around it so that it can act. It is not enough to take a picture: the robot must work something out from the picture, and then do something sensible with the answer.

From camera to action

Every vision system in this course follows the same chain of five links:

1. Camera. Light from the scene passes through a lens and lands on a sensor, a small chip covered in a grid of tiny light-measuring points (a phone camera has millions of them).
2. Numbers. Each point measures how much light reached it and turns that into a number. The camera hands the computer a grid of numbers, not a picture. You will see those numbers for yourself in the next lesson.
3. Picture processing. A program works through the numbers: it tidies them, looks for colours, edges and shapes, and measures what it finds. Most of this course is about this link.
4. Decisions. From the measurements, the program decides what is there and what to do: “that blob is the blue cup, it is a little to the left, so turn left a little.”
5. Action. The robot moves: it turns, steps, reaches or stops. Then it takes a new picture and the chain starts again.

That last point matters. A robot does not look once and then walk blindly. It looks, decides and acts over and over, several times a second, so that it can correct small mistakes as it goes. You will build a loop like this yourself in the last module.

You can see a red cup, a blue box, a green ball and a steel spanner on a wooden bench. Bramble cannot “see” any of that yet. All it has is 3072 small groups of numbers. Turning those numbers back into “red cup, blue box, green ball, spanner” is the job of the programs you will write.

How the workbench boxes work

The box above is a vision workbench. It holds a short program written in JavaScript, a common programming language. Press Run and the program runs in your browser; anything it shows or prints appears underneath.

• You can change the code and press Run again. Nothing you do can break Bramble or the page, so experiment freely.
• If you make a mistake, the box tells you what went wrong in plain words and which line it happened on.
• Lines starting with // are notes for you. The computer ignores them.

Try it now. In the box above, change "workbench" to "workbench-dim" and press Run. You will see the same bench with the lights turned down. Change it back when you have had a look.

Why seeing is hard for a robot

You can glance at a table and spot a cup instantly, without effort. It is tempting to think that must be easy for a computer too. It is not. Your brain has had years of practice and uses a large part of its power for seeing. A robot starts with nothing but numbers. Here are five things that make vision hard:

• Lighting. The same cup gives very different numbers in bright light, in dim light, in orange evening light or in shadow. A rule such as “the cup's pixels are about this bright” stops working the moment someone switches off a lamp.
• Clutter. Real benches are covered in things. The program must pick out the one object it wants from everything else, including the pattern of the wood itself.
• Look-alikes. A blue box and a blue cup can have exactly the same colour. A red sticker can look like a small red cup. The program needs more than one clue to tell them apart.
• Things partly hidden. When one object stands in front of another, the one behind shows only part of itself, and it looks smaller and a different shape.
• Movement. People walk past, balls roll and the robot itself moves, so the picture keeps changing. The program has to keep up and cope with blur.

Each of these problems has its own lesson or two later in the course. The good news is that you do not need to solve them all at once: a robot that works well in a tidy, well-lit workshop is a fine start, and you will make it tougher step by step.

Bramble's cameras

Bramble carries several cameras, each with its own job. This is Bramble's camera data sheet. You will come back to it often, so do not worry about remembering it now.

Some of these words (focal length, stereo, barrel bending) will be explained properly when you need them. For now, notice that Bramble has two kinds of eyes: cameras that measure light, giving colour or grey pictures, and a depth camera that measures distance.

A few other facts about Bramble will matter later. It is 1.6 m tall and about 44 cm wide. Its arm can reach a cup up to 65 cm from the middle of its body. And one rule is fixed: it must not move towards a person who is closer than 130 cm in front of it.

Here is what the front camera sees when Bramble stands in the workshop facing a bench across the room:

The blue cup Eleanor put out is the small blue patch on the bench, a little left of the middle. It is nearly 2 metres away, so it covers only a handful of pixels. Finding small, distant things like this reliably is one of the main goals of the course.

And here is the depth camera looking down a corridor with two crates in it. Instead of colours, every pixel holds a distance. Near things are drawn bright and far things dark:

The low crate is 220 cm ahead, the taller one 400 cm, and the end wall 700 cm. A robot that wants to walk down the corridor can read these numbers directly to find out what is in its way. You will do exactly that in Module 7.

What you will learn, module by module

Along the way you will help the rest of the workshop: Oliver Grant brings trays of parts to sort, Alice Bennett wants the stores counted, and Thomas Hale, the test engineer, makes sure that everything you build works every time, not just once.

Vision can be wrong: a word on safety

No vision system is perfect. Pictures can be too dark, a lens can be dirty, a look-alike can fool the program, and a person can step out from behind a crate. A robot that trusts every answer blindly will sooner or later do something it should not.

Safe robots are built on a simple idea: check, and stop when unsure. They look again before acting, they test whether an answer makes sense (is that “cup” the right size for its distance?), and when they cannot be sure, they stop or ask a person rather than guess. Margaret's rule about people is a good example: whatever the camera thinks it sees, Bramble keeps its distance from anyone in front of it.

You will meet this idea throughout the course, and Module 9 is given over to it. Good vision is not only about finding things; it is also about knowing when you have not found them.

Quick check

Choose an answer to see whether you are right and why.

Found a mistake on this page, or something unclear? Report a problem and mention “Robot Vision Lesson 1.1”.