Integrating Microsoft® Kinect™ with AutoCAD® - Autodesk

birthdaytestΤεχνίτη Νοημοσύνη και Ρομποτική

17 Νοε 2013 (πριν από 3 χρόνια και 8 μήνες)

100 εμφανίσεις

©

2011

Autodesk



1


Integrating Microsoft® Kinect™ with AutoCAD
®

Kean Walmsley

Senior Manager, Worldwide Developer Technical Services, Autodesk

©

2011

Autodesk



2


About the Presenter


Kean joined DevTech in August 1995, and has since worked
in a number of different positions and countries (UK, USA,
India and Switzerland). These days he spends most of his
time on management
-
related activities, but still manages to
keep involved at a technical level with our products.



Since July 2006 Kean has maintained a development
-
related blog at:
blogs.autodesk.com/through
-
the
-
interface



Kean holds a
Licence

d’Informatique

from
l’Université

de Paris
-
Sud

and a Masters in Computer Science from the University of Kent at
Canterbury.

©

2011

Autodesk



3


Class Summary

This class will focus on learning how to integrate
Kinect™
, Microsoft's ground
-
breaking controller for the Xbox 360®, with AutoCAD using .
NET.


We
will look at the various drivers, SDKs, and middleware available to .NET
developers wanting to work with Kinect, as well as the kind of capabilities a device
such as Kinect brings to 3D
products
such as
AutoCAD.


We
will cover two main uses of Kinect: as a 3D reality capture device, feeding point
clouds into AutoCAD, and as a user
-
input device, extending the next generation of
gesture
-
detection capabilities to AutoCAD .NET developers.

©

2011

Autodesk



4


Learning Objectives

At the end of this class, you will be able to:


Implement gestures to allow users to control drawing, modeling, and navigation
operations via
Kinect™


Describe industry trends in human
-
computer interaction and explain how devices
such as Kinect are changing the world


Integrate code that imports point cloud data from Kinect into AutoCAD


List the available drivers, SDKs, and middleware available for working with
Kinect

©

2011

Autodesk



5


Agenda


Introducing Kinect™


The Kinect Device


The Race to Hack Kinect


Drivers, SDKs &
Middleware


CL NUI Platform &
OpenKinect


OpenNI
™ / NITE™


Microsoft® Kinect SDK


Getting Started Coding
with Kinect for
AutoCAD®


Designing a Natural User Interface


Kinect Data Useful to 3D Applications


NUI Design Considerations






Point Clouds


Considerations


Doing a Jig


Interpreting Gestures


Skeleton Tracking


Implementing Gestures


Speech Recognition


Implementing
Voice Commands


Model Navigation


Implementing Model Navigation


©

2011

Autodesk



6


Introducing Kinect™

©

2011

Autodesk



7


©

2011

Autodesk



8


©

2011

Autodesk



9


©

2011

Autodesk



10


Image courtesy of Anya Quinn

©

2011

Autodesk



11


Image courtesy of Andre
Manoel

©

2011

Autodesk



12


Image courtesy of Microsoft

©

2011

Autodesk



13


The Race to Hack Kinect™

Nov 4
th
, 2010


Kinect for Xbox
360
® launched in North America


Adafruit

announced competition to reverse
-
engineer a Kinect driver

Nov 6
th


First hacked by
AlexP
, who chose not to publish his code


Went on to create
Code Laboratories™ NUI Platform

Nov 10
th


Héctor

Martín

announced by
Adafruit

as winner of the $3K bounty


Posted code to
github
, creating
libfreenect
/
OpenKinect

Dec 8
th


OpenNI


launched


Created by
PrimeSense
™, providers of Kinect technology to
Microsoft ®


©

2011

Autodesk



14


©

2011

Autodesk



15


©

2011

Autodesk



16


©

2011

Autodesk



17


Drivers, SDKs & Middleware

©

2011

Autodesk



18


Reverse
-
Engineered Drivers

Code
Laboratories™
NUI
Platform


Spawned from work by
AlexP


Free, closed source


Windows only


C/C++ and .NET

OpenKinect

/
libfreenect


Spawned from work
by
Héctor

Martín


Free, open source


Cross platform (Linux, Mac, Windows)


C/C++, .NET, Python,
ActionScript



Both tools are primarily
drivers

(i.e. no skeleton tracking, gesture recognition)

©

2011

Autodesk



19


OpenNI

/ NITE


For several months, the closest thing to “official” drivers



Multiple components required


OpenNI

SDK (open source)


NITE middleware for skeleton tracking and gesture handling (
c
losed source, for fee?)


SensorKinect

patched driver
(open
s
ource by
avin2
)


For .NET,
nKinect
, developed by
Boris Scheiman

(open
s
ource but 32
-
bit only)



Complex to get working


Matching builds of each component required


Plus a significant amount of manual configuration/tweaking

©

2011

Autodesk



20


Microsoft Kinect SDK


First Beta released June 16
th
, 2011


Projected commercial release early 2012



Overall significant benefits vs.
OpenNI


Simple component install (closed source)


32
-

and 64
-
bit versions available


Better skeleton tracking & real world mapping


Good sample code, documentation and support



Some disadvantages


More work needed for point cloud support


Previously relied on
nKinect

for heavy lifting


Currently doesn’t work inside a Virtual Machine


Commercial licensing costs?

©

2011

Autodesk



21


Getting Started
C
oding with Kinect for AutoCAD

1.
Make sure you have a USB/power adaptor


Comes with separate Kinect box


For compatibility with older Xbox 360 system


Does not come with a newer bundle


Must then be purchased separately


2.
Install the latest Microsoft Kinect SDK


Or
OpenNI

+ NITE +
SensorKinect

+
nKinect


3.
Test with the “Sample Skeletal Viewer”


4.
Create Class Library project


Add project references to AcMgd.dll, AcDbMgd.dll &
Microsoft.Research.Kinect.dll

©

2011

Autodesk



22


Demo |
Our first AutoCAD + Kinect app

©

2011

Autodesk



23


Designing a Natural User Interface (NUI)

©

2011

Autodesk



24


Kinect Data Useful to 3D Applications

Point cloud data


RGB + depth data


Good for “reality capture”

Audio


Speech recognition


Settings/commands

Skeleton data


Movement and gestures

Image courtesy of Kyle McDonald

©

2011

Autodesk



25


NUI Design
C
onsiderations

Do you want to display point cloud data in real
-
time?


Will the user be creating geometry in
-
place?


Do you need to interpret gestures?


Do you need to manipulate the current view?


How about spoken commands?


©

2011

Autodesk



26


Point Clouds

©

2011

Autodesk



27


Considerations


Performance a concern if displaying full point cloud interactively


640 x 480 = 307K points



Sample appropriately from the device


Each displayed pixel needs mapping to “skeleton space”


SkeletonEngine.DepthImageToSkeleton
()


C
olorize only when needed


NuiCamera.GetColorPixelCoordinatesFromDepthPixel
()



If high
point
density required,
consider clipping


Easy to
filter
point
set based on
clipping volume

©

2011

Autodesk



28


Doing a Jig


Jigs designed to get user input via keyboard/mouse


Graphics updated when messages
received


Enabling
graphics update for external devices…



changing
the cursor position by one
pixel forces an update



For interactive display in a jig, no need to colorize


WorldGeometry.Polypoint
() displays points using a single color



If defining paths, smooth off the jitters


Check distance against previous point



Avoid
expensive operations to generate display graphics


e.g. break complex sweeps/extrusions in segments

©

2011

Autodesk



29


Fully Colorization


Outside of a jig, full colorization is straightforward


e
.g. on completion of the jig


One approach is TXT → LAS


PCG



M
inor issue of color
mis
-
matched with depth


Strange shadowing occurs



Theoretically possible to combine data


Multiple
Kinects

or viewpoints

©

2011

Autodesk



30


Demo |
Jigging
p
oint clouds, polylines and solids

©

2011

Autodesk



31


Interpreting Gestures

©

2011

Autodesk



32


Skeleton Tracking


20 joints available



2 “players” can be actively tracked


6 additional players, passively tracked



Skeleton space has Y upwards


To transform to “real space”, need to


Swap Y with Z


Negate X (depending on capture settings)



No need if gestures aren’t “in place”


i.e. if not overlaying with point cloud data

©

2011

Autodesk



33


Implementing Gestures


From the
OnSkeletonFrameReady
() callback


Iterate through the
e.SkeletonFrame.Skeletons
, look for tracked Skeletons


Mor
e
e

work needed if supporting multiple participants


Access its Joints, select the entry you want, extract its Position



Transform them to the space you want


Relative to the model, if “in
-
place”


Can be left as is, if just navigating



Perform comparisons of relative points


e
.g. the distance between the hands is < 5cm



Extract vectors from elbow/shoulder to hand, if needing direction information


©

2011

Autodesk



34


Speech Recognition

©

2011

Autodesk



35


Implementing Voice Commands


Kinect

has an array of 4 microphones


Provides accurate positional data for voice commands


Can associate speech with a particular skeleton


The generated audio stream can also be processed


Microsoft Speech SDK provides speech recognition engine



Straightforward to integrate


Asynchronous callbacks for recognized words


Not on the GUI thread: take care with updating data


Useful for changing settings while gesturing


©

2011

Autodesk



36


Demo |
Jigging segmented solids

©

2011

Autodesk



37


Model Navigation

©

2011

Autodesk



38


Implementing Model Navigation


Collaborative walk
-

and fly
-
throughs


The Killer App for Kinect + AutoCAD
?



Need to adjust the current
ViewTableRecord



No need for a jig, as we’re not displaying in
-
place


Glyphs can be displayed via Transient Graphics



Consider constraining axes of movement, to avoid strangeness


Also worth adding a “home” gesture

Image courtesy of
Rafee

Jewell

©

2011

Autodesk



39


Demo |
Navigating an AutoCAD model

©

2011

Autodesk



40


Agenda


Introducing Kinect™


The Kinect Device


The Race to Hack Kinect


Drivers, SDKs &
Middleware


CL NUI Platform &
OpenKinect


OpenNI
™ / NITE™


Microsoft® Kinect SDK


Getting Started Coding
with Kinect for
AutoCAD®


Designing a Natural User Interface


Kinect Data Useful to 3D Applications


NUI Design Considerations






Point Clouds


Considerations


Doing a Jig


Interpreting Gestures


Skeleton Tracking


Implementing Gestures


Speech Recognition


Implementing
Voice Commands


Model Navigation


Implementing Model Navigation


©

2011

Autodesk



42


Autodesk
and AutoCAD are registered trademarks or trademarks of Autodesk, Inc., and/or its subsidiaries and/or affiliates in the USA a
nd/
or other countries. Microsoft, Kinect and Xbox 360 are either registered trademarks or trademarks of Microsoft Corporation in

th
e United States and/or other countries. Code Laboratories and the CL Logo are trademarks of Code Laboratories Incorporated
.
PrimeSense
, the
PrimeSense

logo, NITE and
OpenNI

are trademarks of
PrimeSense

or its affiliates. All other brand names, product names, or trademarks belong to their respective holders. Autodesk reserves
th
e right to alter product and services offerings, and specifications and pricing at any time without notice, and is not respon
sib
le for typographical or graphical errors that may appear in this document. © 2011 Autodesk, Inc. All rights reserved.