Gesture Recognition Engine

Welcome to the documentation for the gesture recognition engine.

Getting Started

To view the demo files:

  1. Import ALL the files into a NEW project
  2. Press the "Play" button in GameMaker:Studio (or Game Maker Studio 2).

To Import the basic Gesture Recognition SDK extension into your project:

  1. Open your existig project or create a new empty project
  2. Import only the folder: scrips/gesture_recognition.

Adding gestures

To recognize a gesture you need first to tell the engine what are the available gestures the user can draw. To do so, you can add all the template gestures with gr_template_add. You can use Game Maker paths, arrays of coordinates, arrays of points, ds_list of coordinates, or ds_list of points.

To start, try creating a new Game Maker path with a simple figure like a circle, or a "V" shape. Name that path path_hello_world,

Then, create a new empty object and add the path to the Gesture Recognition system passing a name for your template gesture, and the type of the points data and the points data source itself.

// Create event
gr_template_add("hello world", "path", path_hello_world);

To learn more about the different point data sources is recomended to check the included example. You can also check the documentation for gr_template_add.

Repeat the process for each template gesture.

Recording the user input

The next step is to record the user gesture. The engine provides all the gr_record_* functions for this task. For example:

// create
record = undefined;

// global mouse presed event
gr_record_destroy(record); // Destroy the previous gesture record
record = gr_record_create(); // And creates a new gesture record

// global mouse down event
if (record != undefined) {
gr_record_add_point(record, mouse_x, mouse_y);
}

To check all points are recorded inside the record object, you can draw the record on the screen with gr_draw_record.

// Draw event
gr_draw_gesture(3, c_black, true); // width of 3px, black with round corners
gr_draw_record(record); // Draw the record on screen

Recognizing the gesture

The last step is to recognize what gesture was draw by the user. To do that, in the global mouse released event we can use gr_record_recognize.

// global mouse released event
if (record != undefined) {
result = gr_record_recognize(record);
// Then, do stuff with the result of the recognition
}

The gr_record_recognize function returns a Recognition result object. Each recognition holds the name of each template gesture and the probability of the user gesture to be that gesture. For example:

  • name: circle. Accuracy: 0.97
  • name: square. Accuracy: 0.85
  • name: arrow. Accuracy: 0.52
  • name: wave. Accuracy: 0.22
  • And so on for each template

In this example, the figure the user has draw is most probably a circle.

To check the the most probable gesture drawn you can use gr_recognition_get_name. For example:

var name = gr_recognition_get_name(result, 0); // "circle"

The second parameter is optional and is the index of the result in the result list where 0 is the most accurate result, and gr_recognition_get_number(result) - 1 is the less accurate result.

In the same way, you can check the accuracy of each result with gr_recognition_get_accuracy.

var name = gr_recognition_get_accuracy(result, 0); // 0.97

Asynchronous recognition

If your game has a lot of figures you can use the asynchronous recognition mode to split the recognition process over multiple game steps. Check gr_recognition_async_process for a complete example.

Contact

For questions/Bugs/support or to show me the game you are making with Gesture Recognition Engine, please send me an email: