Week 12 - Interface and application programming

Assignment

Individual assignment

write an application that interfaces with an input &/or output device that you made

Group assignment

compare as many tool options as possible

Link to group page

Process

Since I might have bragged a bit about the fact that making stuff on the Unity is an easy piece of cake, our instructor Matteo kind of dared me to complete this week assignment with this environment.

Setting Unity and Arduino

Unity might be an easy place to start, but it also a very vast IDE to the point that it might give a paradox of choice. In this documentation, I won’t talk about how to use Unity and what its basic functions, since it’s the topic of a whole different subject, but I’ll spend when necessary more time explaining what certain features found within Unity means and are useful for my assignmnent.

First of all, I had to devise how to make the scripts inside Unity to talk out of the computer serial port, and then be sure to have a system that receive and interpretate that communication.

The first obvious direction to look was at the world of Arduino, and in fact there are many implementations of Unity and the famous board around the web which varies in functionality and efficiency.

The most complete resource up to date is a whole plug-in called Uniduino that in itself provides all the necessary methods to make Unity and Arduino boards (and possibly avr microcontroller) communicate. It is indeed complete and fully documented, but since it comes with a not-so convenient price I had to look further.

I found a very easy to pick tutorial by Alan Zucconi about how to integrate Unity and Arduino through serial communication, making use of and handy Arduino library called SerialCommand by Steven Cogswell. Although basic and with a lot of fixes to perform, this provided a good starting point.

Essentially, you have to divide the system in two parts. On the Arduino side, you have to implement a series of methods that are performed by the board as soon as it receives the related text string through serial communication. This is the main job of the SerialCommand library and the following self-explainatory snippet shows its methods put at work.

#include <SerialCommand.h>

SerialCommand sCmd; //  the serial command handler

void setup() {  
  Serial.begin(9600); // inits the serial port
  while (!Serial); // awaits serial activation 

  // adds a list of methods the command handler
  sCmd.addCommand("CMD1", Cmd1);
  sCmd.addCommand("CMD2", Cmd2);
  sCmd.addDefaultHandler(errorHandler); //default method for non-listed commands
}

void loop() {
  if (Serial.available() > 0) {  
    sCmd.readSerial(); // checks for commands in the serial port
  }
  delay(50);  
}

void Cmd1 () {
	//Performs something
}

void speakerHandler () {
	//Performs something else
}

void errorHandler () {
	// Error handling
}

On the Unity side, you have to adjust few settings in order to establish a communication with the environmnent and implement a similar way to talk through the serial port. As soon as you’ve made a new project, the first step is to add a C# assembly (assembly is the way C# compilers refers to libraries) called System.IO.Ports. To do so you have to go the menu Edit -> Settings -> Player and then in the Inspector tab look for the Optimization section and set the parameter Api Compatibilyt Level to Net 2.0.. In this way we will make use of a class called SerialPort that will provide all the functionalities needed for the serial communication.

The following lines of code explain briefly SerialPort main methods:

using System.IO.Ports;
SerialPort stream;

// Opens the serial port
stream = new SerialPort(port, baudrate);
stream.ReadTimeout = 50;
stream.Open();

// Sends a string to the port
string message = "foo";
stream.WriteLine(message);
stream.BaseStream.Flush();

// Reads a string from the port
int timeout = 500 // time before closing the communication in case of unfinished message
stream.ReadTimeout = timeout;
try {
	stream.ReadLine();
} catch (TimeoutException) {
	Debug.Log("Time out");
}

If you create a script making use of these functionalities and attach it to a GameObject inside Unity you can easily implement it to send and receive message from the serial port according to your needs.

First prototype

Unity makes it very easy to create quickly simple graphics interfaces to perform on, and to test my just learned functionalities I created a trivial point-and-click window that could send, receive and interact with an Arduino board. Specifically there are two buttons, one that when clicked sends a "PING" string to the port and another one that sends a "ECHO" string followed by another string argument (provided by an input text field). To better do so I made simple script that interfaces the buttons activities with another script that is separately devoted to the serial comminication. Here’s what it looked like:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.UI;


public class SerialHandler : MonoBehaviour {
	
	/// these two parameters are set from the Inspector editor of Unity
	public ArduinoConnector arduinoBoard;
	public Text outputText;

	public int timeOut = 500;

	void Start() {
		arduinoBoard.Open();
	}

	public void WriteToSerial(string msg) {
		arduinoBoard.WriteToArduino(msg);
	}

	public void ReadFromSerial() {
		outputText.text = arduinoBoard.ReadFromArduino(timeOut);
		Debug.Log(outputText.text);
	}

	public void WriteAndRead(string msg) {
		WriteToSerial(msg);
		ReadFromSerial();
	}
}

In this way I could possibly have many boards communicating with many entities through a class and not by touching their respective codes.

Here’s the script inspired the aformentioned tutorial for handling the serial communication:

using UnityEngine;
using System;
using System.Collections;
using System.IO.Ports;

public class ArduinoConnector : MonoBehaviour {
    // serial port
    public string port = "/dev/tty.usbmodem411";
    // baudrate
    public int baudrate = 9600;

    private SerialPort stream;

    public void Open () {
        // Opens the serial port
        stream = new SerialPort(port, baudrate);
        stream.ReadTimeout = 50;
        stream.Open();
    }

    // writes a string to the serial port
    public void WriteToArduino(string message) {
        // Send the request
        stream.WriteLine(message);
        stream.BaseStream.Flush();
    }

    // returns a string from the serial port
    public string ReadFromArduino(int timeout = 500) {
        stream.ReadTimeout = timeout;
        try {
            return stream.ReadLine();
        } catch (TimeoutException) {
			Debug.Log("Time out");
            return null;
        }
    }

    public void Close() {
        stream.Close();
    }
}

Notice that port and baudrate are set according to my own setting and should be replaced in case of different setups.

On the Arduino side, I had to implement a few lines to interpretate the possible commands the board can receive from the port. Similarly to the script at the beginning, this is what it looked like:

#include <SerialCommand.h>
SerialCommand sCmd;
int ledPin = 8;

void setup() {
  pinMode(ledPin, OUTPUT);
  digitalWrite(ledPin, LOW);
  Serial.begin(9600);
  while (!Serial);
  sCmd.addCommand("PING", pingHandler);
  sCmd.addCommand("ECHO", echoHandler);
  sCmd.addCommand("LED", ledHandler);
}

void loop() {
  if (Serial.available() > 0) {  
    sCmd.readSerial();
  }
  delay(50);  
}

// sends a 'PONG' message back
void pingHandler () {
  Serial.println("PONG");
}

// sends back the argument after the command string
void echoHandler () {
  char *arg;
  arg = sCmd.next();
  if (arg != NULL)
    Serial.println(arg);
  else
    Serial.println("nothing to echo");
}

// turns on or off an LED
void ledHandler () {
  if (digitalRead(ledPin) == HIGH)
  digitalWrite(ledPin, LOW);
	  else
  digitalWrite(ledPin, HIGH);
}

A very first test showed me that actually there was actually a communcation in place, but as I would have expected.

The interface didn’t answer always or not as I wanted it to be, even though apparently there was no mistake in the code or the communication setup (i.e.: in the GIF above the Ping button wasn’t always working and the Echo button replies with its own command at first and only after another hit of the button with the echoed string. In other cases the string was truncated or even absent). After many different unsuccesful tests I thought that perhaps I should have looked at what was going on under the hood and in fact it’s where the issue happened. The culprit was inside the SerialCommand Arduino library, specifically in the character that was used by the method to recognize the end of the communication. This is the code involved in this issue.

SerialCommand::SerialCommand() {
	usingSoftwareSerial=0;
	strncpy(delim," ",MAXDELIMETER);
	term='\r';
	numCommand=0;
	clearBuffer(); 
}

By the default constructor, the term parameter - which declares the character for the end of the communication - is set to '\r', but this is not something that can work for Unix-like systems like the one inside my Macbook. It might be possible that the library was wrote and intended to be used by a Windows user, since the aforementioned character is the command that makes return the carriage in serial communication for Windows systems along with a newline command, being it '\n'. In Unix-like systems it is used only the newline character.

In fact, after changing term to '\n', the communication worked just fine.

This system proved me that Unity can send a message to Arduino and can receive a message from it. To further explore the functionalities of this sytem, I made a new button to test if Unity could control the an Arduino pin and so I did it with a proper function an LED.

(pssst… I actually cheated on this, meaning that the control of the led is left to a method inside the Arduino board. A better proof-of-concept would have been sending a parameter from Unity - like the pin to turn on or off - and having the Arduino interpret).

Better use of Unity

As I said, Unity makes it very easy to create rapidly simple interactive interfaces, but it’s mostly renowned for designing and coding games. That’s why I figured that a best use of it for my assignment would be to create a game-like interface that would interact with an external device. It popped to me the idea of an arcade game where the player controls by by keyboard a character that jumps on different blocks to create different interactions with an external board, like turning on and off stuff like LEDs, speakers and motors. For this purpose, here comes Lien, Neil’s pixelated avatar.

With the help of a online raster editor called Piskel (specifically designed for drawing pixel art) I made a sprite to create the assets for a character to play with inside a new Unity project.

As said, Unity fits just fine for creating games and game-like applications, and offers many, many tools to help create a proper workflow to do so. In the case of a 2D platformer game, after setting up a prototype level where to run the scene, I proceeded to slice my sprite in different blocks of textures that will make the animations of different actions.

This is done within Unity with the help of the Sprite Editor tool, after setting the Sprite Mode of my texture to Multiple inside the Inspector tab.

Being the mine a very simple application, I had only three animations for three different states of the character: idle, walk and jump.

After slicing the sprites, I animated them through the mechanim system of the Animation window and I set up a specific Animator for Lien. This is the class used in Unity to associate different states to different animation clip, according to the relation you define among states with arrows inside the graphic editor and variables.

Then I proceeded to put together some elements inside my scene, which will represents the objects Lien will interact with. I placed three boxes that will correspond to three specific commands to be activated by the moment Lien touches them.

On the code side, the interaction is implemented with a script that is activated every time Lien enters a tight area surrounding the box. The scripts sends a command to the serial communication handler (like the one described before) which in turn sends it to the serial port. The script also change the texture of the box when it is touched.

Here’s the script:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class BoxTrigger : MonoBehaviour {
	
	public SpriteRenderer sprite;
	public Sprite onSprite;
	public Sprite offSprite;

	public string serialCommand;
	public SerialHandler serialHandler;

	bool boxState;

	void Start() {
		boxState = false;
	}

	// Activates when the box is touched
	void OnTriggerEnter2D (Collider2D other) {
		// Check if the box is touched by Lien
		if(other.tag == "Player") {
			updateState();
			serialHandler.WriteToSerial(serialCommand);
		}
	}

	// Same as before, but activated when Lien's departs from the box
	void OnTriggerExit2D (Collider2D other) {
		if(other.tag == "Player") {
			updateState();
			serialHandler.WriteToSerial(serialCommand);
		}
	}

	// Sprite update
	void updateState() {
		boxState = !boxState;
		if(boxState)
			sprite.sprite = onSprite;
		else
			sprite.sprite = offSprite;
	}
}

Visually, on the Inspector tab of Unity, when you attach the script to an object you get the following section.

In this way, I was able to use the same script for every box and just modify the public parameters (the most important for our communication system being Serial Command) in order to have different result.

From the Arduino side, techincally very little changed from the previous example and actually I was able to reduce the complexity of the sketch since I only had to read messages from Unity. After renaming the method according the message I would get from Unity, the job was done.

Tests and refinements

I won’t say that the whole process was smooth and flawless. Every time the word “game” is involved in programming you have to come to terms with the fact that you’re going to spend many, many loops on the code-run-debug cycle with often frustrating result. In fact most of the time what I had was more like the following GIF …

… than anything else.

But, at least, I was sure that the base component of my work, the serial communication, was solid and working fine and after many fixes of bugs, optimizations of physics and drawing of more texture I was able to produce an acceptable proof-of-concept test that (almost) worked just as I wanted.

Result

This assignments is far from finished. Due to personal issues I wasn’t able to work on it as much as I wanted and the result is that I didn’t try to use a board I designed myself, I hadn’t connected all the devices I devised for this application and - as suggested by our instructor Matteo - I could have implemented a series of switches that could make me control Lien from the Arduino.

But, running the risk of lacking modesty, the interface looks so rad!

Here’s the links to the parts of this project:

• Unity project;
• Arduino sketch;

Gist & Further development

Although it was very fun and bracing to work with Unity and integrate it with the Arduino IDE, this whole topic won’t have a great of for my final project’s purpose, because I’ve intedend it to be phisically interactive, with no other interactive device connected.

However, it was a very interesting workflow for prototyping things in that blurred lines between have hardware and software and the interfaces among them and surely will apply it in many other applications.

Tools and software used

• Unity
• Monodevelop
• Arduino UNO
• Arduino IDE
• Piskel
• Breadboard, wires, LEDs

Update 30/06/2018

In order to test the functioning of this interface with a custom board, I borrowed a board with an ATmega328p controller from my peer Dario (I couldn’t use other controllers because of libraries size, and Dario just had this board laying around unused ready to be used). In this board I uploaded the same code the Arduino (changing only the led pin) and on my Unity interface I just changed the communication port to COM 10(this time, I was testing it on a Windows system). Then I plugged the computer to the board with an FTDI connector…

…and run the interface on Unity.

)