Quassarian Viper

Quassarian Viper

Pile of Quarks as Homo Sapiens.

Deploying a Simple RASA Chatbot on Unity3d Project to Make a Chatbot for Android Devices

15 minute read

Creating a Chatbot on Android Using Unity3d and RASA

I might stop writing new blogs on this site so please visit dataqoil.com for more cool kinds of stuff.

Contents

Credits

The credit goes to higly motivating friends who taught me what they learned about RASA.

Introduction

How am I here?

Back on early 2020 I was learning Unity3d from its premium subscription and I was suddenly more interested on it. Also while I was doing internship, I learned how to make a commercial chatbots using RASA and deploy it on various platforms. Then I also got interested on text based game where there will be multiple choices and different stories that a player can have. I have not completed(but doing some) any Unity projects yet but I am sharing something useful here and I surely will use these concepts later on real world applications.

What will this blog be?

This blog will be a gentle introduction to the Unity3d and RASA to make chatbot. Or importantly, deploying a simple RASA chatbot on Android using Unity. I will also give gentle introduction to building Android installation(APK file). Importantly, I am not going to make next Alexa or Siri here. If you are here, then you probably know that Unity is a popular Game Development Platform. And Unity3d is C# based and I heard JavaScript can also be used. Unity3d has very great functionalities and the dev community so learning it will be easier than you think. If you are the one who wanted to learn game development for free but doesn’t know where? Then you can follow my below blog for guidance. I am not a pro here but a learner who shares what learns.

What will I built?

I will test the RASA chatbot framework with Unity’s powerful Game Development platform. I will be using Unity’s Networking concept on C#, I will only send a single message and then show the sent and received message on a Canvas. The testing will be done on Android Device, but first I will test it on Unity Remote. Unity Remote is a wonderful application available for Android Device, where we can test our developing aplications on UnityEditor.

Preparing System

My system is Intel I5 with 8GB RAM and Nvidia Geforce 720M, 250GB SSD. It will be very painful to run Unity on low specifications so the proper hardware is essential for this.

Preparing RASA

I am using:-

I am using old version of RASA because it is more stable and this current blog doesn’t requires much functionalities. If I have to write on steps then below steps will be best idea to prepare RASA.

Training RASA

RASA is understood by the system because it is added to system variables. So preparing a RASA project is very simple. From command, go to your working directory and there, rasa init. Then follow the command line steps it tells to do. Finally it will ask if we want to talk to our Assistant, then click yes and we are good to go. RASA creates a project folder on our directory and it includes everything we need on files. It uses some sort of LSTM networks to train a network model.

The files structures created by rasa is:-

  • main dir
    • data
      • nlu.md
      • stories.md
    • models
      • …..tar.gz
    • init.py
    • actions.py
    • config.yml
    • credentials.yml
    • domain.yml
    • endpoints.yml

Currently, we don’t need to understand any of these so I am heading towards next step.

Start RASA server

Starting a RASA server is essential for us because we will be sending a request from Unity’s event. So the only way to make listen our message is by webhooks/api.

  • Activate environment
  • From activated environment command window, go inside RASA project directory
  • Type and hit hit enter after the command rasa run
  • Done

The rasa server will be running on localhost:5005 we can change this but for now leave it as it is.

Preparing Unity3d

Please download UnityHub from its official website. Then it will allow us to add UnityEditor by adding, I am suggesting Unity 2018.3.17f1 to download. While installing Unity, please don’t forget to click on checkboxes like below. Installing MS Visual Studio is very helpful which is by default checked. image-title-here

Create a Unity Project

For simplicity, just create project with default settings but make sure where is your unity project is. Also give your project a good name.

Setting Build Settings

When opened the Unity window,

  • Go to File
  • Go to Build Settings
  • Click on the Android icon, it will take some time to change the settings to another platform. If you can’t then install android support manually.

image-title-here

  • Also go to Player Settings and name your project, comany, version. Player Settings is available under image-title-here

  • Also addd identificaitons image-title-here

  • Switch to 2d Mode by clicking on 2D image-title-here

The window and project structure I have is new, but I am writing this blog on existing project on new scene. So only focus you should make is on the things I stated. This Unity project I am working is a game named Roll a Ball which is still being developed. Also I am working on new Scene named as HTTPTest. SO you have to edit your Scene to HTTPTest from Main.

Creating Simple GUI on Unity

Now we will create a GUI where we will do all UI works.

Create a Canvas

  • Right click on Hierarchy
  • Go to UI
  • Go to Canvas

image-title-here

Installing TextMeshPro GUI

If you happend not to have TMP GUI, then it is essential for us.

  • Go to Windows on Unity’s Tabs
  • Go to Package Manager
  • Look for TextMesh Pro
  • If not there, make sure you are online
  • Then click on install and import it

Add Textbox for Incoming message field

  • Right click on Hierarchy
  • Go to UI
  • And click on Text-TextMeshPro
  • Rename it to IncommingMessage
  • On Inspector side of this TMP(Text Mesh Pro), edit its text to Your Incoming Message. image-title-here

Add Textbox fo Outgoing Message field

Follow above step or just duplicate (CTRL + d) above Game object and edit it.

EventSytem

If your hierarchy doesn’t contains a event system then the GUI wont function properly. So insert one from right click-> UI -> Event System.

Add Send Button

Again we will use TexMesh Pro.

  • From UI, just like above, add Button- TextMesh Pro
  • Name it Sender Button
  • expand it and on it lies, Text, on its TextMeshPro UGUI field, type Send. Then the text on button will change.

Scripting Unity

We will be using C# for writing codes here. We will use the concept of Networking on Unity, which is available by default fron Unity engine. I am using Visual Studio, which makes debugging so easy. I am not going to do tutorial on C# or any other scripting on Unity but my sole focus is to combine Unity and RASA to work full function.

Right under your Project Folder name on Project Window, there might be a folder named Scripts. If not present there then add on manually by right click->create->folder. And name it as Scripts. Following a Naming Convention makes Unity Learning much easier. And under Script folder create a new folder named HTTPTest. Where we will create our script files. There on HTTPTest folder, create a new Script file name HTTPTest from create menu.

Then select all our GUI Objects under Canvas. Click on HTTPTest script and drag that and drop that file on the bottom of inspector. Then ther must be the script available under inspector panel.

image-title-here

Coding Part

Open the HTTPTest.cs script file and lets start coding it. Initially there will be scripts but lets clear them all.

using System.Collections;
using UnityEngine;
using UnityEngine.Networking;
using TMPro;
using System;
using Newtonsoft.Json;
  • Collections is used to work with generic datatypes.
  • UnityEngine is used for MonoBehaviour
  • Networking to handle web requests
  • TMPro to work with TextMeshPro
  • Newtonsoft.Json for handeling JSON text.

Newtonsoft.Json

I am using Newtonsoft.Json a free opensource framework to work on JSON on Unity/C#. I am recommending you to install it. It is publicly and freely available on Unity Asset Store. Or you can download the file and then import to project. But installing and importing from Asset Store is much more efficent and easy choice.

Create a HTTPTest Class

By default when creating a Script, Unity makes a class for us but we cleared all earlier. so the class will include Start() and Update() method. Start is called once before first frame update and update is run on every frame.

public class HTTPTest : MonoBehaviour
{
    private string text;
    public TextMeshProUGUI displayIncomingText;
    public TextMeshProUGUI inputField;
    public TextMeshProUGUI displayOutgoingText;    
    void Start()
    {
        displayIncomingText = GetComponent<TextMeshProUGUI>();
        inputField = GetComponent<TextMeshProUGUI>();
        displayOutgoingText = GetComponent<TextMeshProUGUI>();
        Debug.Log(displayIncomingText.text);
    }
}
  • text is used within a class just to hold text data or we hold what HTTP request returns.
  • displayIncomingText to hold what our RASA model returns.
  • inputField to hold where we will type our message.
  • displayOutgoingText where we show our outgoiong text.

Save the script and go to Unity editor and press the play button. On the Console log, there must be seen text Your Incoming Message.

image-title-here

Create Class for Message Handeling

[Serializable]
public class MessageSender
{
    public string sender, message;
}
public class Message
{
    public string recipient_id, text, image; 
}

MessageSender

It is made serializable because we can later make it like JSON formatable.

  • sender is used to hold string sender, generally it will be username
  • message is used to hold string message sent by sender

Message

This class will handle incoming message sent by RASA channel.

Posting a Web Request

Note that the format of post must match with what a RASA webhook expects to be. Its format is available on RASA’s website but for simplification I am using below approach. {sender:sender_id, message: message text}

public void CallPostRequest(string txt)
    {
        MessageSender thisMSG = new MessageSender();
        thisMSG.sender = "Rasa";
        // read from the input field here.
        thisMSG.message = inputField.text;
        displayOutgoingText.text = inputField.text;
        string json = JsonUtility.ToJson(thisMSG);
        string a = "http://localhost:5005/webhooks/rest/webhook";
        StartCoroutine(PostRequest(a, json));
    }

This method will be called when a user hits a send button. What happens here?

  • Create a new instance for Message Sender,
  • Set sender to Rasa. (just for simplification)
  • Set message to text present on input field’s text.
  • Set display text i.e. outgoing message field.
  • Convert thisMSG’s attributes to JSON.
  • Start a thread or Coroutine named PostRequest that takes address of webhook and the JSON string.

Go to Unity editor and on the Button(Script) section of inspector. We will add event on button. Click on + sign and drag and drop IncomingMessage object on it. On the little bit right, add function under class HTTPTest, named CallPostRequest.

image-title-here

Post Request and Handeling Responses

IEnumerator PostRequest(string uri, string json)
    {
        var uwr = new UnityWebRequest(uri, "POST");
        byte[] jsonTOSend = new System.Text.UTF8Encoding().GetBytes(json);
        uwr.uploadHandler = (UploadHandler)new UploadHandlerRaw(jsonTOSend);
        uwr.downloadHandler = (DownloadHandler)new DownloadHandlerBuffer();
        uwr.SetRequestHeader("Content-Type", "application/json");
        yield return uwr.SendWebRequest();
        if (uwr.isNetworkError)
        {
            text = "Error";
        }
        else
        {
            text = uwr.downloadHandler.text;
            Debug.Log(text);            
        }
        displayIncomingText.text = text;
    }
  • Whenever this coroutine is run, the web request is made to the webhook with post method.
  • To send the post contents, we convert it onto byte array
  • Then upload a data using new instance of UploadHandler and send it the byte JSON.
  • Next prepare a download handler.
  • Next set header content. This webhook accepts json content type.
  • Yield the request.
  • Then check if network error occured, then set error text. Else show the returned text on log.
  • Finally, set the text field of displayIncomingText to the current text.

image-title-here

I have changed little color on the button and input field here but the focus must be on the console log and the incoming message text field. This is what RASA returned to us. Now is the overhead for us to parse this text and show only the part "Great, carry on!". Your result or response might be different than this one but only focus must be on the response. If the error is show then look over console log. Also leave a reply instead.

Parsing the incoming message

Our GUI doesn’t looks great here. Outgoing message is not shown well and neither the incoming message is on well form. Lets try to improve it. Drag and make outgoing message above the incoming one. Until now we were doing many things on wrong way. Lets improve it by adding a new empty game object.

  • Create an empty GameObject, name it Manager.
  • Create insert an HTTPTest script on Manager
  • Edit the event on button, fire the CallPostRequest of Manager’s HTTPTest.
  • Remove the HTTPTest Scripts from all other game objects.
  • Play.
  • Done.

It must be something like below.

image-title-here

IEnumerator PostRequest(string uri, string json)
    {
        var uwr = new UnityWebRequest(uri, "POST");
        byte[] jsonTOSend = new System.Text.UTF8Encoding().GetBytes(json);
        uwr.uploadHandler = (UploadHandler)new UploadHandlerRaw(jsonTOSend);
        uwr.downloadHandler = (DownloadHandler)new DownloadHandlerBuffer();
        uwr.SetRequestHeader("Content-Type", "application/json");
        yield return uwr.SendWebRequest();
        if (uwr.isNetworkError)
        {
            text = "Error";
        }
        else
        {
            text = uwr.downloadHandler.text;
            string ntext = "";
            string prev = "}";
            string now = ",";
            int prevInd = 1;
            int ind = 1;
            string empty = "" +
                "";
            foreach (char s in text)
            {
                if (prev == s.ToString())
                {
                    string m = text.Substring(prevInd, ind - prevInd);
                    ntext = ntext + JsonConvert.DeserializeObject<Message>(m).text;
                    Debug.Log(m);
                    prevInd = ind + 1;
                    ntext = ntext + empty;
                }
                ind = ind + 1;
            }
            int startInd = text.IndexOf("text") + 6;
            int length = text.Length - 3 - startInd;
            text = ntext;
        }
        displayIncomingText.text = text;
    }

Parsing the text string to JSON string was hard for me on C# so I manually tried to get the only part that contains the message. I did that by looking onto the response of webhook and counting where the key text, message lies. And also I deserialzed the attributes of Message. Some slicing, indexing, searching completes this operations. This came onto mind to solve this particular problem and I was stuck for hours due to this problem. So I hope someone finds a better way to do this.

Test on Android device

Now is the time for us to test it on our phone. So download the application named Unity Remote 5 and install it on phone. Open that application and follow the written instructions. That should be:-

  • Edit-> Project Settings->Editor

image-title-here

When pressing Play button on editor, now Scene must be loaded on phone. If you are testing it on phone, then it will not work for now. Because the keyboard is not shown on phone when clicking on the input field. So what I did instead was typed the message from Editor and pressed the send button from Unity Remote.

Building an APK file

If any USB Debugging enabled phone is connected then we can directly build and install apk on Android phone, just have to click on build and run. But Building an apk file first and then installing on phone can also be done. If the build finishes without any error, then it is fine else, make sure you are on internet (sometimes gradle build tools is expected to download). Build System must be internal not a gradle. But when we type and send the message, error is shown because we do not have a medium to connect with RASA server from phone. One way to connect RASA server for now is by laptop hotspot. Turning on laptop hotspot and then we have to change the URL on HTTPTest.cs file.

  • Turn on hotspot on computer
  • Turn on WiFi on phone
  • Edit the address

Final code looks like below:-

using System.Collections;
using UnityEngine;
using UnityEngine.Networking;
using TMPro;
using System;
using Newtonsoft.Json;
[Serializable]
public class MessageSender
{
    public string sender, message;
}
public class Message
{
    public string recipient_id, text, image;
}
public class HTTPTest : MonoBehaviour
{
    private string text;
    public TextMeshProUGUI displayIncomingText;
    public TextMeshProUGUI inputField;
    public TextMeshProUGUI displayOutgoingText;    
    void Start()
    {
        //displayIncomingText = GetComponent<TextMeshProUGUI>();
        //inputField = GetComponent<TextMeshProUGUI>();
        //displayOutgoingText = GetComponent<TextMeshProUGUI>();
        //Debug.Log(displayIncomingText.text);
    }
    public void CallPostRequest(string txt)
    {
        MessageSender thisMSG = new MessageSender();
        thisMSG.sender = "Rasa";
        // read from the input field here.
        thisMSG.message = inputField.text;
        displayOutgoingText.text = inputField.text;
        string json = JsonUtility.ToJson(thisMSG);
        string address = "http://localhost:5005/webhooks/rest/webhook";
        StartCoroutine(PostRequest(address, json));
    }
    IEnumerator PostRequest(string uri, string json)
    {
        var uwr = new UnityWebRequest(uri, "POST");
        byte[] jsonTOSend = new System.Text.UTF8Encoding().GetBytes(json);
        uwr.uploadHandler = (UploadHandler)new UploadHandlerRaw(jsonTOSend);
        uwr.downloadHandler = (DownloadHandler)new DownloadHandlerBuffer();
        uwr.SetRequestHeader("Content-Type", "application/json");
        yield return uwr.SendWebRequest();
        if (uwr.isNetworkError)
        {
            text = "Error";
        }
        else
        {
            text = uwr.downloadHandler.text;
            string ntext = "";
            string prev = "}";
            string now = ",";
            int prevInd = 1;
            int ind = 1;
            string empty = "" +
                "";
            foreach (char s in text)
            {
                if (prev == s.ToString())
                {
                    string m = text.Substring(prevInd, ind - prevInd);
                    ntext = ntext + JsonConvert.DeserializeObject<Message>(m).text;
                    Debug.Log(m);
                    prevInd = ind + 1;
                    ntext = ntext + empty;
                }
                ind = ind + 1;
            }
            int startInd = text.IndexOf("text") + 6;
            int length = text.Length - 3 - startInd;
            text = ntext;
        }
        displayIncomingText.text = text;
    }
}

Thank you so much for reading out. And I am still unsure that this blog has some errors. SO if you have any query about this project, please leave a comment or you can find me on Twitter as QuassarianViper.

What next?

  • Building a text based game using Chatbot.
  • Handling buttons, images, carousels etc.

Don’t forget to check out

Comments

You May Also Enjoy

Advent of Code 2022 with Python

4 minute read

The advent of Code is a yearly festival for programmers like me where we try to solve different stories to gain stars. I love these challenge because its fun...

Text Analysis with WordCloud in Python

5 minute read

WordCloud in Python can be done in different ways but one of the most popular and easier ones is using the package wordcloud. We can install it using the fol...

Simple Cryptography Algorithms in Python

2 minute read

Cryptography Algorithms have been around the world for more than centuries and there are still many inscriptions around various places in the world which we ...