CARTOOGLE ™ – Communicate with Computers via Rough Sketches

by Fred Abler

The invention of the digital computer forced us to realize that we need new languages – computer-programming languages – to communicate with machines. These languages are called structured or ‘constructed languages’, as opposed to our ‘natural language’.

Natural language refers to any ordinary language used for human communication. For architects and designers, our natural language is drawing. But to date, little progress has been made using drawings as a natural language to communicate with computers.

Now a team of university researchers from Rhode Island’s Brown University and the Technical University of Berlin have created software that analyzes users’ relatively crude, cartoony sketches – and figures out what it is that they’re trying to draw.

Fig 1 . A research system that recognizes cartoon sketches as they’re drawn in real time.

This research system does not yet have a name. ‘Pictionary’ is already taken, so I’m going to go with CARTOOGLE ™ as a working title. CARTOOGLE ™ correctly identifies the user’s drawing almost as often (56% of the time) as human viewers (73 %).

This feat of ‘semantic understanding’ is especially impressive when you consider the extreme breadth of drawings and representational styles CARTOOGLE™ must understand. To generate a visual reference library, researcher ended up with 250 object categories.

Then they used Mechanical Turk, a crowdsourcing marketplace run by Amazon, to hire people to sketch objects from each category — 20,000 sketches in all. Those data were then fed in to existing machine learning algorithms to sort out which category each sketch belonged to.

Fig 2 – ‘Funny Bunny’ drawings. A sample of drawings of a rabbit used by CARTOOGLE as a semantic reference library.

After seeing numerous examples of how various people drew a rabbit, for instance, CARTOOGLE would learn that combinations of specific shapes usually meant “rabbit.” Note that ‘ears and whiskers’ turned out to be a big hint.

In the future, it is hoped that the technology might be used for applications like sketch-based internet searches. Imagine SketchUp with a CARTOOGLE function that would automatically convert your sketches (simple or pristine) into 3D models!!

The research behind CARTOOGLE ™ is now available online, together with a video, a library of sample sketches, and other materials. The team is currently considering a ‘Pictionary’ type open source game to expand the systems’ drawing reference library.

In shortFor Architects, drawing is our natural languages of choice. It’s silent, expressive and quick – no keyboarding required. The impact of CARTOOGLE may be ready conversion of our design sketches into objectified models, and much more ‘bandwidth’ for pen-based computing.

Thanks for reading and subscribing to FormFonts 3D Models.

Read More:

CARTOOGLE ™ is a trademark of FormFonts 3D – All Rights Reserved.

About Formfonts 3D Models

Established in 2002, FormFonts 3D Models has served over 11,000 subscribers. FormFonts 3D Models offers over 50,000 of well-made and with the right-amount-of-polygons models and textures on a subscription basis. Throughout each month we upload new models for download. If you don’t see what you need, request it to be made. We offer models in the following formats: Google SketchUp ® | Autodesk®3ds Max®, Autocad®, Revit® & FBX
This entry was posted in 3D Models, 3D Tech, Architecture, GeoDesign, Reality Hacking, SketchUp, Theory & Concepts, Uncategorized and tagged , , , , , , , , , . Bookmark the permalink.

1 Response to CARTOOGLE ™ – Communicate with Computers via Rough Sketches

  1. Pingback: Human-Computer (or Robot) interface through Rough Sketches | Teresa Escrig

Comments are closed.