Co-authored by Gabriel Greenberg.
Think of the last time you looked for an apartment: Most likely, a good number of the listings that you encountered came with floor plans. And by looking at these diagrams, you probably had no trouble finding out all sorts of things about the living spaces being advertised: the rough shape of each room; the location of all windows and doors. But how exactly did you reach these conclusions? And how do you immediately understand the route you’re being shown, when a helpful stranger in a foreign country traces a path with their index finger over a subway map? Or how do you look at a courtroom sketch and know that the defendant was wearing suspenders?
Whether they’re architectural renderings, Venn diagrams, or even the inkless images created by gestures, pictures can all be thought of as 2-D encodings of our 3-D world. We decipher these images so easily that we never even suspect we’re cracking a code of sorts; we recognize that a certain brushstroke represents an eyebrow, or certain lines forming a Y denote the corner of a cube. But what if someone could write out a codebook (so to speak) precise enough that even a machine, by consulting it, could draw and interpret drawings? Over the past few decades, philosophers, psychologists, and computer scientists have taken on this task, and found it less straightforward than one might think.