91�Թ�

Prehistoric cave drawings date back about 40,000 years. Ancient Egyptians used papyrus for paper. Artistic drawing flourished during the European Renaissance starting in the 14^th century.

Now, 91�Թ� computer scientists and Autodesk researchers are looking to take artistic vision from 2D to 3D with augmented reality technology.

“We tried out systems like Tilt Brush by Google, and we noticed it was extremely hard to draw in the air,” says Rahul Arora, a PhD student in the department of computer science. “One reason why is, we’ve been drawing on 2D paper. How can we transition those skills to the 3D world?”

Arora and Autodesk researchers Rubaiat Habib Kazi, Tovi Grossman – who joins the department computer science as an assistant professor this summer – George Fitzmaurice and Karan Singh, a 91�Թ� professor of computer science, created SymbiosisSketch, software for combining 2D and 3D sketching for detailed design of 3D objects.

Their research will be presented at the Association of Computing Machinery’s international conference on human factors in computing systems (ACM CHI) in Montreal beginning this weekend.

SymbiosisSketch uses a tablet to support the 2D action of drawing on a surface. An augmented reality headset, Microsoft HoloLens, brings the drawing to life in a 3D augmented view.

“You can think of the tablet as a camera you position in the air, and you draw within its field of view. So you're always creating 3D strokes even when you're drawing on the tablet," says Arora. "If you draw a line, you'll see it in real time on that surface, in the air. It’s like a drawing window you can move around in 3D space.”

Arora, who worked on the project in part during an internship at Autodesk Research, says the canvas size can increase, to draw very large objects, like cars or buildings, or decrease, to zoom in on drawing finer details, like a door handle.

The joint research is another example of the longstanding partnership between Autodesk, the 3D design, engineering and entertainment software powerhouse, and 91�Թ� computer science’s lab.

Arora says future applications of SymbiosisSketch could include use in animation and storyboarding, but for now they’re exploring state-of-the-art 3D sketching and fabrication.

“I'm currently working on a fabrication project," he says. "Let’s say you have a 3D sketch – what if you want to manufacture it to be strong and lightweight? How can we use some physical simulation to guide you?”

A Captain America figure battles an augmented reality drawing of an airplane (image courtesy of the researchers)

There are of course limitations to the current hardware technology, which lends itself well to a lab environment. A bulky motion capture system consisting of six ceiling-mounted infrared cameras tracks the pen’s movement. Also, AR hardware needs better speed and accuracy.  

But all of that is just a matter of time.

“As hardware improves, research on the software side moves forward as well,” says Arora.  

“This project is really looking forward in the future.”