T Q ~Q

Adrian Buckley

Evan
Defibaugh

John Duhamel

Alex Kaiser

Darnell Parker

Craig
Riggins

T Q



~Q

Project Objectives

-
Relay real time Nintendo 64 session data
between multiple hosts for network play

-
Provide an interface for a Microsoft
Kinect

frontend in place of tradition controller

-
Utilize non
-
obtrusive electronics

-
Minimize cost and maximize functionality

-
Power consumption is a secondary concern




System Architecture

Client 1

Client 2

Server

Nintendo 64

Server

Client 1

Client 2

Player 1
Television

Player 2
Television

Player 1

Controller Signal

Player 2

Controller Signal

Player 2

Controller Signal

Player 1

Controller Signal

N64 RCA

Signal

N64 RCA

Signal

N64 RCA

Signal

N64 RCA

Signal

N64 RCA

Signal

Client Architecture

Regular N64
Controllers

Television

ARM Cortex
MCU

N64

Controller

Signal

Kinect

USB Signal

Digital

RCA Signal

Digital

Controller Signal

Ethernet IO

Microsoft
Kinect

Analog RCA

Signal


MCU takes input from 4 N64 controllers


Microsoft
Kinect

can track 2 people

Server Architecture

ARM Cortex
MCU

Digital

Controller Signal

Digital

RCA Signal

Ethernet IO

Nintendo 64

N64 Controller

Signal

Analog

RCA Signal


Can communicate with 1
-
4 clients


Moves controller signal from client to N64
and RCA signal from N64 to client

Unified Hardware

N64 Controllers /
Microsoft
Kinect

Television

ARM Cortex
MCU


Controller Signal

RCA

Signal

Controller

Signal

Ethernet IO

RCA Signal

Nintendo 64

Controller

Signal

R
CA


Signal


1 hardware architecture will be able to
operate as either the client or the server

Advanced System Architecture

Client 2

Client 1 /
Server

Nintendo 64

Client 1 /
Server

Client 2

Player 1
Television

Player 2
Television

Player 2

Controller Signal

Player 2

Controller Signal

Player 1

Controller Signal

N64 RCA

Signal

N64 RCA

Signal

N64 RCA

Signal

N64 RCA

Signal

Kinect Integrated Peripheral

Kinect Processor Linux Microcontroller


XBOX Kinect

Processor

Linux
Microcontroller

Kinect Sensor


I/O


2 Cameras (RGB and IR Depth)


IR Output


4 Array Microphone


USB Port


Possible Implementations

ROS:
BeagleBone


VS C# 2010: Netduino


VS C#
2010: PC



Implementing Kinect

via ROS on Beaglebone

ROS (Robot Operating System) is an open
source project focusing on interfacing
microprocessors with robots (in our case the
Kinect and a peripheral Linux microcontroller)

Kinect Processor

Control Node

The Netduino operates on Microsoft Visual
Studios C# 2010 via Net. Micro workforce

Implementing Kinect

via VS C# 2010 on
N
etduino

Minor Hiccups in Progression

Using the Robot OS


ROS tutorial has
incorrect/missing
info


Open source, thus
compatibility issues
from node to node


Unintuitive to
navigate


Using the Netduino

•
Netduino requires
Net. Micro
framework, where
Kinect requires Net.
Framework

•
Trouble converting
from standard to
micro

There is no problems implementing Kinect via PC

Last Resort and Additional info

As a last resort the Kinect can be routed
through a Windows PC for processing then
sent to microcontroller.

More ROS info

http://www.ros.org/wiki/kinect

(Install ROS)

http://www.ros.org/wiki/ROS/Tutorials

(ROS Tutorial)

http://beagleboard.org/bone

(
Beaglebone

Description)

http://www.ros.org/browse/list.php

(ROS software)



More Net. Micro workforce

http://docs.com/BTN4


(Installing
OpenNI
)

http://www.netduino.com/netduino/schematic.pdf



(
Netduino

Schematics)

http://informatix.miloush.net/Microframework/Arti
cles/WpfEmulators.aspx


(
.net

micro install)

http://informatix.miloush.net/microframework/Arti
cles/Reflector.aspx

(
.net

micro reflector)

http://netduino.com/netduino/specs.htm





(
Netduino

Specs)


Gesture and Speech Algorithms


Joint Data is returned in X, Y, Z, and W
(confidence) data


Evaluate position of joints relative to another
and trigger button pressed event


For audio grammar files must be written


Compare text output of audio and set trigger
for the controller or the sensor as being used


Output control trigger and button triggers

Kinect Processor Interface


The Processor will output triggers in the form
of buttons and sensor/controller triggers for
both players one and two


The triggers will be relayed to the Linux
microcontroller that will convert button trigger
to N64 controller output format and use the
control trigger to decide if forwarding the N64
controller data or sensor converted data

N64 Controller Tutorial

How the N64 Controller Sends Its Data


The N64 controller only has three pins: Ground, data, and
VCC. It takes a +3.6V supply.


The N64 uses a protocol in which bits are sent serially to and
from the controller as 4µs
-
wide pulses. ‘Low’ is 3 µs low and
1 µs high, and ‘High' is 1 µs low and 3 µs high.

N64 Data Polling

The N64 continually polls the controller using an 8
-
bit
command (0x01), and the controller reports the state of its
buttons and analog joystick with a 32
-
bit response: 16 bits for
the buttons, signed 8 bits for horizontal joystick position, and
signed 8 for vertical. Each bit packet ends with a stop bit (1).

Button Configuration

The buttons are mapped to their respective bits as follows:





Sample Joystick Positions and respective bits:


Bit

0

1

2

3

4

5

6

7

Button

A

B

Z

Start

D
-
Up

D
-
Down

D
-
Left

D
-
Right

Bit

8

9

10

11

12

13

14

15

Button

Null

Null

L

R

C
-
Up

C
-
Down

C
-
Left

C
-
Right

Left Extreme

1

0

0

0

0

0

0

1

Left Off Center

1

1

1

1

1

1

1

1

Right

Off Center

0

0

0

0

0

0

0

1

Right Extreme

0

1

1

1

1

1

1

1

Up

Off Center

0

0

0

0

0

0

0

1

Down Off

Center

1

1

1

1

1

1

1

1

Joystick Off
-
Center Left & Down

The Translator Box


The
Kinect

will take player movement/sound data and
send over button and trigger data to the “translator box,”
which then converts that data into the 64
-
bit
struct

format shown below.




There will be a physical master switch on the translator
box, to switch between
Kinect

control and N64 controller
control. When this switch is set to “
Kinect
,” the
translator will await trigger and button data from the
Kinect

to process.


If the switch is set to “N64,” the translator will need to
keep track of one or two controllers, simultaneously
converting their button presses into the shown format,
but with the “More Buttons” bit and time delay
consistently set to zero.


Button Data

Joystick

Data

Which Player?

More Buttons?

Time Delay

16

bits

16 bits

1 bit (0 or 1)

1 bit

Int

(30
-
bit)

Server to Translator to N64


Another Translator Box is needed to translate the
Server’s data into N64 controller data for the N64.


This translator will read in the controller data, and send
the correct button sequence via serial ports to the N64,
correctly timed. If the N64 polls the “controller,” but the
translator has not received button data from the server,
the translator will automatically respond with a 32
-
bit ‘0’
analog stream and continue doing so until new button
data appears.

Audio/Visual
Signal


The video output of the Nintendo 64 carries
visual information
via an analog signal, plus
two synchronizing signals for aligning the
video signal on a television.


The audio
output, also analog,
is split into
two signals for stereo speakers in a
television.

Audio/Visual
Waveforms

Video

Audio

Audio/Visual
Sampling


The audio and visual output of the Nintendo
64 must be digitized and formatted
for
input
into the server for transport over the internet.


Once the
audio/visual
information is sent
over
the
internet it must be converted back
into an analog signal to be displayed on a
player's television
.


A Series of Tubes
–

Hardware


Initial design and testing will be done on an
A
rduino

with an
E
thernet shield.


Ardiuno

is easy to program to send and receive
data packets.


After getting a fully functional system with the
Premade hardware we will move on to a
custom PCB with a Linux OS.

Our Internet Objectives


We need the data received from the server
to be real time


If there is missing data we just move to the next
data packet.


We know that the receiver is ready to receive
packets.


TCP/UDP


TCP/IP uses an established connection
between sender and receiver.


This requires an acknowledge signal from the
receiver which takes time.



UDP instead sends packets in a continuous
stream and doesn’t care if packets are
missing on the receiver side.


Instead of waiting for acknowledgement UDP just
tags the packet with the correct receiver IP
address and sends it on its way.

UDP in depth


UDP sends data with a header, and within
the header a 2byte portion telling the
receiver how long the data section will be.


This 2byte section allows for a maximum of 64Kb
section of data which is more than enough for
sending the n64 signal and frames of an RCA
signal.


The other parts of the header are receiver IP and
port number.

Putting it all Together


Using UDP because we need a real time
connection and don’t mind dropped packets.


Using a static IP for our server so that we
have something to aim at for our UDP
packets.


Using
Arduino

and Ethernet shield to
prototype and onto
custome

PCB once all
kinks are worked out.

Project Schedule

Longest Design
Period

11 Days for Testing

Technical Reference Manual

37 Days for Integration

Division of Labor

Task

Adrian

Evan

John

Alex

Darnell

Craig

Program Mgmt &
Documentation

P

S

S

Kinect

Programming

S

P

S

N64 Translation

S

P

S

Ethernet Interface

P

P


Cont牯lle爠偲g牡浭mng

S

S



Audio/Video

卩gnal
Conve牳ron

S



S

丶㐠
Ga浥play

A捴ion













P = Primary

S = Secondary

Advanced Parts List

TOTAL ESTIMATED PROJECT COST = $1,074

*NOTE*
–

ALL EXPENSES PAID TRIP TO LAS VEGAS FOR “ELECTRONICS



SEMINAR” NOT INCLUDED

Goals

Primary
:

•
Control Nintendo 64 game exclusively via an XBOX
Kinect


Secondary
:

•
Have one N64 controller communicate with one N64
console over the internet

•
Control the N64 game using either the Kinect or the N64
controller remotely


Tertiary
:

•
Play a complete N64 game with 2 players at one location
using the
Kinect
, and the other 2 players at a different
location using N64 controllers

•
Allow functionality with a variety of different controllers

Milestone 1 Objectives

Kinect Translation

•
Kinect translator sends simple gesture data to
the Linux µController, in correct format

Nintendo 64

•
Send properly timed Nintendo 64 analog signal
from the µController to the N64 console

Network Configuration

•
Send data packets across the network to and
from each user using the µController

Audio/Video

•
Convert analog input into a digital code


Milestone 2 Objectives

Kinect


Determine ideal
Kinect

gestures for
satisfactory user interface


Transmit verbal commands through
Kinect

Network Configuration


Send and receive N64 and audio/video
signals across server

Audio/Video


Output signal from server to television

Risks and Contingency Plan

Risks


Syncing the N64 timing sequence with the
Netduino

microcontroller’s internal timer


Busy internet path/lost data packets


Delay between
Kinect

/ N64 input and television
output


Data corruption or potential injury resulting from
falling or player collisions while playing with a
motion sensitive controller

Solutions


Use only N64 controllers (no
Kinect
) for game
control


Leave out server / direct connection from
microprocessor to N64


User plays video game at own risk!!





Questions???