ROS by Example: HTML5, Javascript Pi Face Tracker GUI

by Scott Bell

Overview

Advancements in browser technology now open the door to creating robust, platform independent GUI interfaces to your ROS projects. In this example, we create a GUI interface for the pi_face_tracker code. To make it all work, we leverage the following technologies:

rosbridge

mjpeg_server

html5 Canvas element

javascript

The pi_face_tracker_gui leverages the functionality of the pi_face_tracker code which exposes it's functionality through topics and services that we can interact with via javascript over rosbridge.


HTML and Javascript Note

When implementing javascript with html, you can either include the javascript as a section in the html page, or host it in a separate file and reference the path to it. For ease of visibility and usage, we will include our custom javascript in the same page.

We will also reference the rosbridge javascript which lives outside our html page using this line:

<script type="text/javascript" src="http://brown-ros-pkg.googlecode.com/svn/tags/brown-ros-pkg/rosbridge/ros.js"></script>


Components

The HTML page has the following main components which we will then discuss in turn:

Toggle buttons: buttons that run on off features that existed as hotkeys in the base pi_face_tracker

Dynamic ROI: GUI implementation of the opencv feature that lets you draw a rectangle over an image and then pass the coordinates of the rectangle to set a new ROI

Dynamic Resolution Adjustment: Using the output from the /camera/rgb/camera_info topic, we are able to dynamically read the camera resolution and adjust the canvas elements and location of the buttons relative to the height of the canvas.

Rosbridge: As mentioned before, it allows us to create javascript messages in JSON format that are then converted by the rosbridge to standard rostopic messages.

Mjpegserver: Used to create an image stream from the video source (in our case the Kinect) but it will also work with a standard webcam as well. The exciting bit about using the mjepgserver is that is is exposed via a socket connection, allowing the video source to be anywhere you can get to by IP address. This is important for future versions of pi_face_tracker_gui, which will working on mobile android enabled devices (or any device that can run google chrome or an equivalent browser).


Component and Features Code Details

Buttons: All of the buttons take the same basic format as the following code for the Toggle Text button.

<button type="button" onclick="toggle_markers()">Toggle Text</button>

This line of html creates a button that when selected, calls a custom method called toggle_markers()

The toggle_markers function:

function toggle_markers()
    {
        var connection = new ros.Connection("ws://127.0.0.1:9090");
        connection.setOnOpen(function (e) {
        connection.callService('/pi_face_tracker/key_command','["t"]',nop);         
        });
    }
}

This function like the others behind all the buttons creates a connection to rosbridge and once it establishes the connection, it then send the text “t” over the rostopic of /pi_face_tracker/key_command


Showing Video on a Canvas

An HTML5 canvas by default does not play streaming video. In order to make that work, we create the following function called init(), which we call on page load

function init(){
  kinect_img.src = "http://127.0.0.1:9191/stream?topic=/pi_face_tracker/image";

  //call redraw of video canvas every 100 ms
  setInterval(draw,100);
}

This function sets an image source to our streaming video and then uses the setInterval function to call the draw function every 100ms

function draw() {
  // draw video on single canvas
  var ctx = document.getElementById('video_canvas').getContext('2d');
  //ctx.clearRect(0,0,300,300); // clear canvas...not needed for video feed to work
  ctx.drawImage(kinect_img,0,0,width,height);
}

It first gets a handle (called a context) to the html object (ctx variable).

One you have the canvas elements context, you can call the drawImage function to display the current “grab” of the image on the canvas. By calling this every 100ms, you then get video.


Dynamically Setting the ROI

For this implementation, we needed the ability to draw a rectangle over the surface of the video image. To do so, we use 2 HTML5 Canvas elements. By default canvases are transparent, so we use the bottom canvas to display the video and another canvas of the exact same size that lays over the top, where we use the mouse to draw a rectangle. As you draw the rectangle to create a new ROI, you are really drawing on the top canvas and looking through at the video on the second canvas behind it.

We create the two canvas elements with the following html code

<canvas id="video_canvas" width="javascript:getWidth()" height="480" style="position: absolute; left: 0; top: 0; z-index: 0;"> </canvas>

<canvas id="drawing_canvas" width="640" height="480" style="position: absolute; left: 0; top: 0; z-index: 0;"></canvas>

They use a default size of 640x480, but we will then later adjust that based on information from the CameraInfo topic.


Drawing a Rectangle on a Canvas and Sending Coordinates to pi_face_tracker

When you press and release the mouse buttons while over the drawing canvas, it then calls functions which draw the rectangle. As the rectangle is drawn, we can get the rectangles coordinates relative to the canvas and send them over rosbridge back to the pi_face_tracker. Pi_face_tracker then uses that input to call opencv to draw a rectangle on top of the video coming into pi_face_tracker from the kinect. This is all accomplished through the following functions and code.

function mouseDown(e) {
  rect.startX = e.pageX - this.offsetLeft;
  rect.startY = e.pageY - this.offsetTop;
  drag = true;
}
function mouseUp() {
  drag = false;
  //clear rectangle on mouse up as opencv now has drawn it for us on the incoming image
  ctx.clearRect(0,0,canvas.width,canvas.height);
}
function mouseMove(e) {
  if (drag) {
    rect.w = (e.pageX - this.offsetLeft) - rect.startX;
    rect.h = (e.pageY - this.offsetTop) - rect.startY ;
    ctx.clearRect(0,0,canvas.width,canvas.height);
    draw_rect();
  }
}
function draw_rect() {
  // ctx.fillRect(rect.startX, rect.startY, rect.w, rect.h); //creates a filled in square
  ctx.strokeRect(rect.startX, rect.startY, rect.w, rect.h); //creates a square with just the outline
  //Call to pi face service to set roi
   var connection = new ros.Connection("ws://127.0.0.1:9090");
        connection.setOnOpen(function (e) {
        connection.callService('/pi_face_tracker/set_roi','{"roi":{"x_offset":' + rect.startX + ',"y_offset":' + rect.startY + ',"width":' + rect.w + ',"height":' + rect.h + ' }}',nop);
        
        });

}
function mouse_init() {
  canvas.addEventListener('mousedown', mouseDown, false);
  canvas.addEventListener('mouseup', mouseUp, false);
  canvas.addEventListener('mousemove', mouseMove, false);
}

mouse_init();


Dynamic Resolution Adjustment

Using the output from the /camera/rgb/camera_info topic, we are able to dynamically read the camera resolution and adjust the canvas elements and location of the buttons relative to the height of the canvas.

For this part, we will use rosbridge to READ info from a published topic, where so far we have only published to a topic over the bridge. For debugging purposes, we have also added some console output examples which you can see when you run in google Chrome by selecting Ctrl+Shift+J and then select the Console icon.

More Javascript Notes: A section of Javascript can be called on page load, as we do with the init() function, but from there, Javascript will then process from top down. So after the init() function is called, we then hit our custom Javascript below, when then sets up some global variables and executes and also contains various functions we have discussed.

<script type="application/x-javascript">
var kinect_img = new Image();

  //rectangle drawing code
    var canvas = document.getElementById('drawing_canvas');
    var ctx = canvas.getContext('2d');
    var rect = {};
    var drag = false;

  // set variables for adjusting canvas and video resolution settings
    var width = "320" //default width
    var height = "240"  //default height
        javascript:console.log("default width = " + width);
        javascript:console.log("default height = " + height);

    //get kinect video stream width and height values from rostopic via rosbridge
    dynamically_set_video_resolution();



The in-line function dynamically_set_video_resolution() gets called...

function dynamically_set_video_resolution() {
        javascript:console.log('console initialized');
       
        javascript:console.log('creating ROSProxy connection object...');
        var connection = null;
        try {
            connection = new ros.Connection("ws://127.0.0.1:9090");
        } catch (err) {
            javascript:console.log('Problem creating proxy connection object!');
            return;
        }
        javascript:console.log('created');
        javascript:console.log('connecting to 127.0.0.1:9090' + '...');
       
        connection.setOnClose(function (e) {
            javascript:console.log('connection closed');
        });
        connection.setOnError(function (e) {
            javascript:console.log('network error!');
        });
        connection.setOnOpen(function (e) {
            javascript:console.log('connected');
            javascript:console.log('initializing ROSProxy...');
            try {
                connection.callService('/rosjs/topics', '[]', nop);
            } catch (error) {
                javascript:console.log('Problem initializing ROSProxy!');
                return;
            }
            javascript:console.log('initialized');      
            javascript:console.log('running'); 
            javascript:console.log('trying to create topic Handler and read data back from topic');

            connection.addHandler('/camera/rgb/camera_info',function(msg) {
        width = msg.width;
        height = msg.height;

        javascript:console.log("dynamically adjusted width = " + width);
        javascript:console.log("dynamically adjusted height = " + height);

    document.getElementById("video_canvas").setAttribute("width",width)  
    document.getElementById("video_canvas").setAttribute("height",height)
    document.getElementById("drawing_canvas").setAttribute("width",width)
    document.getElementById("drawing_canvas").setAttribute("height",height)
    document.getElementById("button_location").setAttribute("style","position: absolute; left: 0; top: " + height + ";")

        connection.callService('/rosjs/unsubscribe','["/camera/rgb/camera_info",0]',function(rsp) {
        javascript:console.log('unsubscribed to /camera/rgb/camera_info');
    });

    });
    connection.callService('/rosjs/subscribe','["/camera/rgb/camera_info",0]',function(rsp) {
        javascript:console.log('subscribed to /camera/rgb/camera_info');
    });    
 
        });
    }


This function first creates a connection object to rosbridge and does some logging to the console.
If the connection is successfully created, the  connection.setOnOpen function is called.

In order to “listen” to a topic over the rosbridge, we have to first create a Handle to the connection object, which also sets the topic we wish to listen to

connection.addHandler('/camera/rgb/camera_info',function(msg) {
        width = msg.width;
        height = msg.height;

        javascript:console.log("dynamically adjusted width = " + width);
        javascript:console.log("dynamically adjusted height = " + height);

    document.getElementById("video_canvas").setAttribute("width",width)  
    document.getElementById("video_canvas").setAttribute("height",height)
    document.getElementById("drawing_canvas").setAttribute("width",width)
    document.getElementById("drawing_canvas").setAttribute("height",height)
    document.getElementById("button_location").setAttribute("style","position: absolute; left: 0; top: " + height + ";")

        connection.callService('/rosjs/unsubscribe','["/camera/rgb/camera_info",0]',function(rsp) {
        javascript:console.log('unsubscribed to /camera/rgb/camera_info');
    });

           
Once you have the handle you can then subscribe to the topic with the following code

connection.callService('/rosjs/subscribe','["/camera/rgb/camera_info",0]',function(rsp) {
        javascript:console.log('subscribed to /camera/rgb/camera_info');
    });

The interesting bit about this is we DON'T want to continuously listen to this topic as we only need the video resolution once to size the canvas and button locations. So forthis, we use the unsubscribe function.

The tricky bit was, it takes a moment for the listener to establish and we need to make sure we only grab the width and height from the topic AFTER the connect is created and that we DON'T unsubscribe until after the connection is made.

To ensure that, we place the unsubscribe function IN the handler, (our listener) when the Handler “hears” info being published to it's topic, it kicks off which then lets us grab the msg info off the topic and resize the canvas html. Then once “confidently” resized, we now unsubscribe from the topic.

Rosbridge and JSON message format

Rosbridge is implemented (sourced in) via the following HTML line:

<script type="text/javascript" src="http://brown-ros-pkg.googlecode.com/svn/tags/brown-ros-pkg/rosbridge/ros.js"></script>

For an overview and some good code example of rosbridge and JSON go to the following link:

http://code.google.com/p/brown-ros-pkg/wiki/Quick_start_rosbridge_and_ROS

Mjpegserver: The mjpegserver code can be installed from the following location at ROS http://www.ros.org/wiki/mjpeg_server

Line of HTML code that points image source at the mjpegserver video stream (grabs a single image each time it is called, thus need to call on a timer as discussed above to get video)


img.src = "http://127.0.0.1:9191/stream?topic=/camera/rgb/image_raw"


The port used here (9191) must match the port on which mjpeg_server is launched (see below), otherwise it can be any available port.


Running the Code

Once mjpeg server and pi_face_tracker are installed, you can run the pi_face_tracker and pi_face_tracker_gui with the following steps in separate terminal windows.


$ roscore

$ rosrun rosbridge rosbridge.py

$ rosrun mjpeg_server mjpeg_server _port:=9191

$ roslaunch ros2opencv openni_node.launch

$ roslaunch pi_face_tracker face_tracker_kinect.launch

Then download and open the pi_face_tracker_gui.html file in google Chrome and enjoy!