The Daily

Arts & Culture

Pictorial Grammar

December 21, 2010 | by

Co-authored by Gabriel Greenberg.

Think of the last time you looked for an apartment: Most likely, a good number of the listings that you encountered came with floor plans. And by looking at these diagrams, you probably had no trouble finding out all sorts of things about the living spaces being advertised: the rough shape of each room; the location of all windows and doors. But how exactly did you reach these conclusions? And how do you immediately understand the route you’re being shown, when a helpful stranger in a foreign country traces a path with their index finger over a subway map? Or how do you look at a courtroom sketch and know that the defendant was wearing suspenders?

Whether they’re architectural renderings, Venn diagrams, or even the inkless images created by gestures, pictures can all be thought of as 2-D encodings of our 3-D world. We decipher these images so easily that we never even suspect we’re cracking a code of sorts; we recognize that a certain brushstroke represents an eyebrow, or certain lines forming a Y denote the corner of a cube. But what if someone could write out a codebook (so to speak) precise enough that even a machine, by consulting it, could draw and interpret drawings? Over the past few decades, philosophers, psychologists, and computer scientists have taken on this task, and found it less straightforward than one might think.

Left: a portrait of George Bush; right: a drawing of an IKEA lamp.

Even the simplest sorts of pictures, line drawings, raise complicated questions. One could say that the marks in a line drawing simply demarcate boundaries between patches of color. As appealing as that rule of thumb might sound, it yields this largely unrecognizable portrait of George Bush. Other algorithms, the earliest ones pioneered in the 1970s, construed line segments and intersections as edges and corners. But lines represent many things: Some of the lines in this drawing of an IKEA lamp indicate edges; others show object boundaries. Groupings of lines can represent shadow.

Julian Opie’s Vera, Dancer, 2007

Most confounding of all, certain kinds of lines—termed “suggestive contours” by a team of Rutgers and Princeton researchers—show edges that would appear if the viewer shifted his or her vantage point. Look at the line segment at the bottom edge of the woman’s lip in Julian Opie’s Vera, Dancer, 2007. Opie’s depiction of Vera is undeniably stylized; still, we wouldn’t expect to find a black stripe tattooed under her lower lip in real life. Nor is it likely—unless she had an incredibly droopy pout—that the line depicts an actual edge. Instead, we interpret the mark as a suggestive contour: It’s the line that indicates precisely where an edge would appear, as carved out by the protruding curve of Vera’s lower lip if a viewer climbed a stepladder and looked down at her face. A similar suggestive contour appears in Picasso’s rendering of Igor Stravinsky. Though Picasso certainly wasn’t much of a realist, it’s still unlikely the line extending diagonally down from his friend’s ear is meant to imply that Stravinsky’s cheek buckled inward dramatically enough to create a kangaroo’s pocket of sorts. Rather, the line traces the edge delineated by his cheekbone that would become apparent if he swiveled his head slightly in the other direction.

Picasso’s rendering of Igor Stravinsky.

Though artists use suggestive contours in line drawings all the time, they usually do so without realizing the complexity of what they’re doing. In fact, the Rutgers and Princeton researchers have gone so far as to identify the precise algorithms that machines can use to create drawings with suggestive contours. (Those of you wanting to program your own computers, or to see how their machines fared, can find the documentation and images here.)

But how universal are the rules we use to understand pictures? Does the same codebook govern the Lascaux cave paintings and the vignettes in Delta’s emergency evacuation pamphlets? These questions are still open to debate. I recently came across a picture in the Metropolitan Museum of Art’s current Yuan Dynasty exhibition that illustrated a few of the answers. The work, by Wang Zhenpeng, follows some of the same rules as do contemporary Western pictures. Lines still indicate edges, visual boundaries, strands of hair. And yet there are distinct differences—the edges of a table represented by parallel lines that never converge.

“The World of Khubilai Khan: Chinese Art in the Yuan Dynasty” is up at the Metropolitan Museum of Art until January 2, 2011. Dawn Chan is the assistant editor at She briefly studied artificial intelligence and vision at the University of Zurich on a Fulbright grant. Gabriel Greenberg is writing his dissertation on pictorial semantics at the Department of Philosophy at Rutgers University.



‹ Previous:


  1. Jan Schindler | December 21, 2010 at 2:01 pm

    It seems to me that you really need to consult or be an artist to make such statements in this article. I have taught drawing for several years and Yes! I DO understand the complexity of what I am doing.

  2. Dawn Chan | December 21, 2010 at 2:04 pm

    I see what you’re saying. Perhaps if we’d said “mathematical complexity” that would have been more clear…

  3. Ryl Mandus | December 21, 2010 at 3:08 pm

    You’re kidding me, right? I don’t realize what I’m doing when I lay down a line or a curve, much less comprehend its mathematical complexities as I’m consciously performing the act? Golly, I guess all that time I spent studying perspective, composition, and the golden ratio was for nothing.

  4. Gabriel Greenberg | December 21, 2010 at 3:23 pm

    I think its very important here to distinguish between *tacit* and *explicit* knowledge. We all speak our mother-tongue fluently. So we know the rules of this language tacitly. But we cannot state the rules explicitly. Thus, despite speaking perfectly well, professional linguists still do not understand *many* of the rules governing very common languages like English and Chinese. The case of drawing is no different. Artists of course are expert users of pictorial “languages”. Of course they know these rules in one sense–tacitly. But no matter how expert, you or I will not be able to state all the rules explicitly. Professional computer vision scientists still have not uncovered all these rules. And it should go without saying that the rules of perspective are only the tip of the iceberg. As the article makes clear, the rules of line use are very subtle and have not yet been *explicitly* understood. There are also all kinds of complicated situations when the exact rules of perspective are actually violated.

  5. Anderson Silva | December 22, 2010 at 3:56 pm

    Thanks, Gabe, for putting things in perspective. The article was submitted for consideration; to start a dialogue, not to offend. It was directed at the population in general and not at specific, expertly trained artists who are far above such elementary subject matter. I interpreted Dawn’s tone as one of genuine appreciation and wonder; of respect for such a beautiful ability. It was, by no means, cause for capital letters and sarcasm.

  6. Dawn Chan | December 23, 2010 at 12:33 pm

    Thanks Anderson. Also, it might be worth noting in the post: We’re merely saying that most of us usually aren’t aware of the complex set of algorithms we’re following when we create or interpret a very particular type of line, defined by researchers as a “suggestive contour”—the details of which were first outlined in 2003 and can be found here. Jan, if you do understand the “suggestive contour” algorithms that De Carlo and his colleagues put forth, then that’s great! (And if not, my guess is that you probably use “suggestive contours” as effectively and powerfully, if intuitively.) But certainly, there are many other mathematically complex rules and strategies in art that have long been understood and employed for centuries—and Ryl, I agree that perspectival systems and the golden ratio are two good examples.

    In fact, I’m sure many artists are deeply familiar with most of the issues we highlighted in our post. The point isn’t that the existence of these rules, categories, and distinctions should come as news to artists. Rather, as researchers continue to hash out all these details precisely enough for machines to follow along, their task is turning out to be less and less trivial than it might first appear. Which to me makes it all the more marvelous that we create and interpret drawings as well as we do: a “beautiful ability,” to use Anderson’s words.

  7. Roger Conner | January 3, 2011 at 12:13 pm

    And again, we are still just scratching the surface even after the above clarifications. I have always been fascinated by the issue of what I call, for lack of a better term, emotive evoking in artwork. The simplist is the emoticon, examples such as 🙂 versus 🙁 As the drawing gets more complex, the emotive evoking becomes so subtle. Why do the simple line drawings of the old “Winnie The Pooh” book illustrations in pen and ink evoke emotion even greater than the newer more detailed color animations of today? Or simply think of “Nighthawks” by Hopper, or a good David Hockney drawing. If you wish to sense this, go to Google images and type in “david hockney Celia drawings” and soak it up. Could any machine hope to evoke the sensibility and emotion of those drawings?

  8. itidus20 | September 12, 2011 at 12:17 pm

    Computers can play chess but they can’t enjoy it. They can draw but they can’t express anything. They can interpret but they can’t see.

    So we can invent variants of chess and the computers will be confused again until they are taught to generalise their knowledge of chess onto variants.

    And in a similar way, just as the game of chess is an arbitrary set of rules with many variants, so humans possess the capacity to create these pictorial grammars.

    And at some stage we may find these artificial intelligences more of a nuisance and a parasite than something valuable.

    I guess what I’m saying is we don’t need computers interpreting our art. And on further thinking there is a danger that human pictorial grammar could be constrained by the limitations of AI if the AI becomes common enough, just as you can’t finger paint with a computer. You can fake it with a tablet but you can’t feel the paint on your fingers and experience the joy of having to wash it off.

    Just some outspoken thoughts, I wouldn’t be reading this article if I wasn’t fascinated by these things.

  9. itidus20 | September 12, 2011 at 12:36 pm

    That last post of mine i’m not sure if it needed to be said. But anwyay I found my way here from a wiki article about axonometric projection which used the term ‘pictorial grammar’. And I found the term again in an article about Piet Mondrian. *Scuttles away*

1 Pingbacks

  1. […] Paris Review – Pictorial Grammar , Dawn Chan […]

Leave a Comment