by Fred Abler

The invention of the digital computer forced us to realize that we need new languages – computer-programming languages – to communicate with machines. These languages are called structured or ‘constructed languages’, as opposed to our ‘natural language’.

Natural language refers to any ordinary language used for human communication. For architects and designers, our natural language is drawing. But to date, little progress has been made using drawings as a natural language to communicate with computers.

Now a team of university researchers from Rhode Island’s Brown University and the Technical University of Berlin have created software that analyzes users’ relatively crude, cartoony sketches – and figures out what it is that they’re trying to draw.

Fig 1 . A research system that recognizes cartoon sketches as they’re drawn in real time.

This research system does not yet have a name. ‘Pictionary’ is already taken, so I’m going to go with CARTOOGLE ™ as a working title. CARTOOGLE ™ correctly identifies the user’s drawing almost as often (56% of the time) as human viewers (73 %).

This feat of ‘semantic understanding’ is especially impressive when you consider the extreme breadth of drawings and representational styles CARTOOGLE™ must understand. To generate a visual reference library, researcher ended up with 250 object categories.

Then they used Mechanical Turk, a crowdsourcing marketplace run by Amazon, to hire people to sketch objects from each category — 20,000 sketches in all. Those data were then fed in to existing machine learning algorithms to sort out which category each sketch belonged to.

Fig 2 – ‘Funny Bunny’ drawings. A sample of drawings of a rabbit used by CARTOOGLE as a semantic reference library.

After seeing numerous examples of how various people drew a rabbit, for instance, CARTOOGLE would learn that combinations of specific shapes usually meant “rabbit.” Note that ‘ears and whiskers’ turned out to be a big hint.

In the future, it is hoped that the technology might be used for applications like sketch-based internet searches. Imagine SketchUp with a CARTOOGLE function that would automatically convert your sketches (simple or pristine) into 3D models!!

The research behind CARTOOGLE ™ is now available online, together with a video, a library of sample sketches, and other materials. The team is currently considering a ‘Pictionary’ type open source game to expand the systems’ drawing reference library.

In short – For Architects, drawing is our natural languages of choice. It’s silent, expressive and quick – no keyboarding required. The impact of CARTOOGLE may be ready conversion of our design sketches into objectified models, and much more ‘bandwidth’ for pen-based computing.

Thanks for reading and subscribing to FormFonts 3D Models.

Read More:

• Computer Program Can Identify Rough Sketches – Brown University
• How Do Humans Sketch Objects – tu-berline.de

CARTOOGLE ™ is a trademark of FormFonts 3D – All Rights Reserved.