Kinect_Calibration_2012-01x - Facoltà di Ingegneria

abashedwhimsicalSoftware and s/w Development

Nov 2, 2013 (3 years and 9 months ago)

172 views

Kinect calibration


Ilya

Afanasyev







Facoltà di Ingegneria


Trento, 2012

1
/20

25/01/2012

Content

2
/20

1.
The description.

2.
About the existing software.

3.
Tests of Nicolas
Burrus

software binaries.

4.
Open Kinect drivers installation.

5.
Compilation of Nicolas
Burrus

source code.

6.
Results.

7.
Conclusions

8.
References.



25/01/2012

The description.

3
/20

The software for Kinect Calibration should be used
for:

1.
Saving the RGB images and IR
-
Depth data from Kinect sensor
under OS Windows.

2.
Calibration the Kinect sensor with obtaining [1]:

-

the intrinsic parameters (i.e. matrixes of intrinsic parameters
for IR camera (
K_ir
) & RGB camera (
K_rgb
)).

-

the extrinsic parameters (i.e. extrinsic mapping between IR
(depth) and RGB Kinect camera (R,T)).

-

distortion coefficients (
kc_ir
,
kc_rgb
).

-

the intrinsic and extrinsic parameters (i.e. matrixes of intrinsic
parameters for IR camera (
K_ir
) & RGB camera (
K_rgb
) and
extrinsic mapping between IR (depth) and RGB Kinect camera
(R,T)).

3.
Developing the interface of different algorithm with Kinect for
different real
-
time object recognition applications.




16/01/2012

About the existing software.

4
/20


The existing analogues of the software:

1.
-

The author doesn’t know analogues under Windows7/64 bit . It
can be created, basing on
Microsoft Kinect SDK
for developers
[6] under the platform
.NET Framework
ver. 4.0.

2.
-

There is the software developed by Nicolas Burrus [2] under
OS Linux / Ubuntu 10.10 / 32 bit, which is claimed to be
compiled also with Windows
-
32 bit. Can it be used under
Windows7/64 bit ??
-
> It seems that NO!!



25/01/2012

Microsoft Kinect SDK for developers
[6]

5
/20


1.
SDK (Software Development Kids) includes drivers,
APIs for raw sensor streams and human motion
tracking, installation documents, and resource
materials. It provides capabilities offered by the
Microsoft Kinect device connected to computers
under Windows 7 / Windows 8 to developers who
build applications with C++, C#, or Visual Basic by
using Microsoft Visual Studio 2010.


16/01/2012

Nicolas Burrus software

[2]

6
/20


Nicolas Burrus software [2]:


There are different versions of the Burrus software:

1. Test programs running
from

binaries
-
> Windows binaries
-
>
Kinect
RGBDemo

v0.6.1.

-
> supports
OpenNI
/
Nite

backends

[3]
and has the experimental infrared support with
OpenNI

(still
buggy)
-
> I tried to check this work for
OpenNI

backend under
Windows 64bit
-
> there is no the extrinsic mapping between IR
and RGB camera at Kinect calibration.

2.
Compiling from source
-
>

2.1
Compilation on Windows
-
>


with
OpenNI
/
Nite

backends

doesn’t give the extrinsic mapping
between IR and RGB camera at Kinect calibration.


with
Libfreenect

backend

-
> I wasn’t succeed to compile
under Windows 7/64 bit?!


2.2
Compilation on Linux (
Ubuntu

10.10
-

32 bit
)
-
> supports
Libfreenect

backend
-
> the only workable variant with getting all
calibration parameters (intrinsic, extrinsic and distortion) was
realized on Alberto
Fornaser’s

PC (
UniTN
)

16/01/2012

7
/20


1.
OpenNI

backend:

1.
Nicolas Burrus software [1]: Kinect
RGBDemo

v0.6.1.
-
> supports
OpenNI
/
Nite

backends

[2] and has the experimental infrared
support with
OpenNI

(still buggy)
-
> I tried to check this work for
OpenNI

backend under Windows 64bit.

2.
Install Sensor Kinect drivers under Windows 32 bit [4] and
OpenNI
/
Nite

modules (Win32) [3] according to the instruction [5]
and verified that they work properly.

3.
Use RGB
-
D Capture [1] to grab Kinect RGB and IR images with
intensity.raw and depth.raw files.
-
> the software doesn’t grab IR
images!!
-
> only RGB image, intensity.raw, depth.raw and
calibration file (calibration.yml without distortion parameters)!!

4.
“calibrate
-
openni
-
intrinsics

--
pattern
-
size 0.0405 grab1
calibration.yml”
-
> gives “openni_calibration.yml” with Intrinsic
matrix and distortion coefficients for Kinect RGB camera !!

5.
“calibrate
-
openni
-
depth.exe
--
pattern
-
size 0.0405 grab1” gives
partly processed figures with a message about errors.

6.
The software doesn’t produce a file with xyz
-

coordinates and
RGB
-
colors info of every point!!!





1
st

attempt

16/01/2012

The using of Nicolas Burrus software [2]

with
OpenNI

backend
under Windows

The comparison of Kinect calibration results
for

-

Libfreenect

backend (under OS Linux /
Ubuntu

10.10
-

32 bit)

-

OpenNI

backend (under OS Windows 32bit)

-


The same figures of Intrinsic
Matrixes for Kinect RGB and
Depth Cameras

Libfreenect

backend

OpenNI

backend

The Depth Camera Calibration
for
OpenNI

backend is

NOT available!!!

16/01/2012

The comparison of Kinect calibration results for

-

Libfreenect

backend (under OS Linux /
Ubuntu

10.10
-

32 bit)

-

OpenNI

backend (under OS Windows 32bit)

-


23/11/2011

There is NO extrinsic mapping
between Kinect Depth and RGB
cameras for
OpenNI

backend

Libfreenect

backend

OpenNI

backend

The reprojection of 3D points
from Kinect Depth Camera to
RGB image for
OpenNI

is NOT
possible!!

Install
OpenKinect

for Windows 7
[7]

10
/20

It is necessary to have:

-

Visual Studio 2010 (as a compiler on a 32 bit
Windows 7 installation).

-

Kinect Sensor drivers for Windows 7
(
OpenKinect
-
libfreenect

ofTheo
/
ofxKinect
) [9].

-

freenect

windows dependencies [8]:



libusb
-
win32 1.2.4.0 (Currently there is a
breaking change in
libusb

1.2.5.0, which I used!!);



pthreads
-
win32



Glut

17/01/2012

Kinect Sensor drivers installation

[7]

11
/20

-

Download & extract the latest
GitHub

repository [9].

-

Make sure your Kinect sensor is plugged into a
USB port on your computer.

-

Open Device Manager (Start


Control Panel


System and Security


System


Device
Manager)

-

Find “Human Interface Device” (Xbox NUI Motor)


Update Driver Software


uninstall the previous
Kinect drivers (if there are)


set path like
C:
\
Special
\
OpenKinect_libfreenect_v0_1_1_56
\
pl
atform
\
windows
\
inf
\
xbox
nui

camera
\



the
drivers will correspond to the processor type.


18/01/2012

Kinect Sensor drivers installation

[7]

12
/20

19/01/2012

Check the files:
libusb0.dll

(x64) in
C:
\
Windows
\
System32
\

and
libusb0.sys

in
C:
\
Windows
\
System32
\
drivers
\

and libusb0.dll (x86)

in
C:
\
Windows
\
SysWOW64
\

Cmake

configuration & Generation of a Visual
Studio 10 solution
[7]

13
/20

1.
Open
Cmake

GUI and set the Browse Source
path to /
libfreenect
/
openkinect
.

2.
Set the browse build path to
/
libfreenect
/
openkinect
/build

19/01/2012

The software should be configured when Kinect turned off!!

Cmake

configuration & Generation of a Visual
Studio 10 solution
[7]

14
/20

You will now need to point certain build paths to the
freenect

dependencies (
pthread
).

19/01/2012

The software should be configured when Kinect turned off!!

Cmake

configuration & Generation of a Visual
Studio 10 solution
[7]

15
/20

You will now need to point certain build paths to the
freenect

dependencies
(
libusb
).

19/01/2012

If take "
msvc
" instead of "
msvc_x64
"
-
> 1 fatal error of compilation less !!!

It doesn't matter if choose
THREADS
\
LIB
\

pthreadVC2.lib

or
pthreadVSE2.lib

Cmake

configuration & Generation of a Visual
Studio 10 solution
[7]

16
/20

19/01/2012

Run "Configure" and it should successfully complete. Click Generate
afterwards and then open the libfreenect.sln file from the build
folder the output was placed in
-

/
openkinect
/build/

Run Configure again. It should successfully complete the process
this time but we still need to add the Glut references.

Visual Studio 10 solution
[7]

17
/20

19/01/2012

RESULT:
Rebuild All: 13 succeeded, 2 failed, 2 skipped
.

You will receive a few errors when building the solution but it should compile
correctly regardless. The binaries will be placed in
build/bin/Debug.

Disable the Antivirus if it disturbs the creation of the VS10 solution!

Choose Visual C++ 2010 and then select Build Solution from the
Debug Menu or press F7.

The result of building the c for Windows 7
[7]

18
/20

The final step is to place a few .
dll

files in the
appropriate folder and then we can run some of the
samples provided:



Copy
freenect.dll

and
freenect_sync.dll

from
\
build
\
lib
\
Debug



Copy from dependencies folders: pthreadVC2.dll,
glut32.dll, libusb0.dll (choose the correct version x86/x64)



Copy
tiltdemo.exe
,
regtest.exe
,
glview.exe
,
glpclview.exe, glpclview.exe, regview.exe, hiview.exe
from
\
build
\
bin
\
Debug



It can be helpful to copy
freenect_sync.lib
,
freenect.lib
,
freenect.exp

from
\
build
\
lib
\
Debug
to
\
lib
\


Successful
start of demo
-
samples


Kinect works!

19/01/2012

Demo
-
samples of
OpenKinect

for Windows 7

19
/20



tiltdemo.exe
drives the Kinect motor
.



regview.exe

shows the combined


IR&RGB

video (
IR up to 1 m
)



regtest.exe

grabs bad
-
quality pictures:


--

registration_test_depth_registered.pgm
,


--

registration_test_rgb.ppm
,


--

registration_test_depth_raw.pgm
,


--

registration_test_depth_mm.pgm






glview.exe
gives IR&RGB


video in separate windows

(IR up to 3.5 m)


19/01/2012

Demo
-
samples of
OpenKinect

for Windows 7

20
/20



hiview.exe
gives IR&RGB


video in separate windows

(
IR

up to 3.5 m,
RGB

has hi
-
quality

& hi resolution)



glpclview.exe gives color

3D video with big shadows

behind the objects.



cppview.exe produces a

continuous list with messages

under the phrase:

“thread execution started”

19/01/2012

Compiling
RGBDemo
-
0.6.1 under Min GW

from
source with
OpenKinect

on Windows 7 [2]

21
/20

The step
-
by
-
step procedure for
Min GW
, in case using
libfreenect
:

1. Install QT
opensource

for Windows. This will also install Min
GW.

2. Add
C:
\
Special
\
QtSDK
\
mingw
\
bin
\

to the Path environment
variable: Start


Computer (properties)


Advanced System
Settings


Environmental Variables (button)


Variable: PATH


Edit


Add.

3. Install and run
cmake

on
rgbdemo



Disable the
NESTK_USE_OPENNI

cmake

variable



Could not find
module
FindOpenCV.cmake




Configuring incomplete,
errors occurred!


4. Ungrouped Entries





OpenCV_DIR

(NOT FOUND)


C:/Special/opencv
.



QT_QMAKE_EXECUTABLE (NOT FOUND)


C:
\
Special
\
QtSDK
\
Desktop
\
Qt
\
4.7.4
\
mingw
\
bin
\
qmake.exe


it is better to use qmake.exe from
QtSDK
\
mingw

??

19/01/2012

2
nd

attempt

Compiling
RGBDemo
-
0.6.1 under Min GW

from
source with
OpenKinect

on Windows 7 [2]

22
/20

The step
-
by
-
step procedure for
Min GW
, in case using
libfreenect
:



Open the CMakeLists.txt in Qt Creator
(from
C:
\
Special
\
RGBDemo
-
0.6.1
-
Source
\
RGBDemo
-
0.6.1
-
Source
\
CMakeLists.txt
open with
C:
\
Special
\
QtSDK
\
QtCreator
\
bin
\
qtcreator.exe
).


It creates the
folder:
qtcreator
-
build



Add QT
-
qmake

&
OpenCV

path to the Path
environment variable:

C:
\
Special
\
QtSDK
\
Desktop
\
Qt
\
4.7.4
\
mingw
\
bin
\

C:
\
Special
\
opencv
\



to the Path environment variable: Start


Computer
(properties)


Advanced System Settings


Environmental Variables (button)


Variable: PATH


Edit


Add.



20/01/2012

Compiling
RGBDemo
-
0.6.1 under Min GW

/ MSVS 2010
from source with
OpenKinect

on Windows 7 [2]

23
/20

The step
-
by
-
step procedure for
Min GW / MSVS2010
, in case
using
libfreenect
:



Cmake

2.8.6


Ungrouped Entries


PKG_CONFIG_EXECUTABLE


C:
\
Special
\
opencv
\
OpenCVFindPkgConfig.cmake

Results
:
Cmake

can’t compile the QT &
OpenCV_DIR



OpenCVConfig.cmake



We should use
Qt!!



Qt


Run
Cmake




CMAKE_MODULE_PATH
C:
\
Special
\
qtcreator
-
build
\



Run
Cmake




CUDA_TOOLKIT_ROOT_DIR not found or specified
” [10]



if set path to C:
\
Special
\
PCL_1_4_0
\

which contains CUDA
libraries, then we have the messages


CMake

Error: The source directory
"C:/Special/PCL_1_4_0/3rdParty/flann/bin" does not appear to
contain CMakeLists.txt.

Specify
--
help for usage, or press the help button on the
CMake

GUI
”.


20/01/2012

Burrus

software “
RGBDemo
-
0.6.1” is not installed on Window7
-
64bit!!

Conclusions

24
/2
0


1. Nicolas
Burrus

software with
OpenNI

binaries [2] works
under Windows 7
-
64 bit, but the software doesn’t give the
extrinsic mapping between Kinect Depth and RGB cameras
and doesn’t produce a file with
xyz
-

coordinates

and RGB
-
colors

info of every point
!!!

2. Although the drivers of
OpenKinect

has been successfully
installed under Windows 7
-
64 bit, the Nicolas
Burrus

source
code
RGBDemo
-
0.6.1
(with LGPL License)

for Kinect
Calibration [2] is

not compiled properly under Windows
7
-
64
bit
.


It should be used another software or another Kinect
drivers!!

3. Both
Microsoft Kinect SDK for developers [6]

and
OpenKinect drivers
[9] can be applicable to save RGB images
and IR
-
Depth data to calibrate Kinect sensor and use it under
Windows 7
-
64 bit, but no suitable software has not been
produced yet.

25/01/2012

5.

References

1.
Camera Calibration and 3d Reconstruction. //
OpenCV

(Open
Source Computer Vision) v2.1 documentation:
http://opencv.willowgarage.com/documentation/cpp/camera_cali
bration_and_3d_reconstruction.html

2.
Nicolas Burrus. Kinect.
RGBDemo
, calibrate and visualize
Kinect output. // http://nicolas.burrus.name/, 2011.

3.
OpenNI

Modules. // www.openni.org,

4.
SensorKinect

drivers. //
github

social coding.
https://github.com/avin2/SensorKinect.

5.
How
-
to: Successfully install Kinect on Windows (
OpenNI

and
NITE). //
Vangos

Pterneas

blog:

http://studentguru.gr/b/vangos/archive/2011/01/20/how
-
to
-
successfully
-
install
-
kinect
-
windows
-
openni
-
nite.aspx

6.
Microsoft Kinect SDK for developers.
http://www.microsoft.com/en
-
us/kinectforwindows/






25
/20

16/01/2012

5.

References

7.
Install
OpenKinect

for Windows 7 and XP. //
http://kinect.dashhacks.com/kinect
-
guides/2011/01/09/install
-
openkinect
-
windows
-
7
-
and
-
xp

8.
Getting Started using
OpenKinect
. //
http://openkinect.org/wiki/Getting_Started#Windows.

9.
https
:
//github
.
com/ofTheo/ofxKinect


10.

OpenCV
:

CUDA_TOOLKIT_ROOT_DIR

not

found

or

specified“
.

http
:
//xinyustudio
.
wordpress
.
com/
2011
/
11
/
30
/opencv
-
cuda_toolkit_root_dir
-
not
-
found
-
or
-
specified/





26
/20

16/01/2012