Google Assistant and Mendix

Traditionally at TimeSeries, at the end of every year we look back at the year that has passed and close it with a bang: our annual End of the Year party! All colleagues (and their partners) gather for a nice dinner, an awesome party and most likely a fairly heavy hangover 😉

Every year we are surprised with a present in our hotel room. A gift for another year of hard work; a token of appreciation. Last November, that gift was our own (TimeSeries branded) Google Home!

Recap

As you might recall, in one of my previous blog posts, I dabbled in-home automation: retrieving measurement from my smart meter, creating a nice dashboard in Mendix and even automating lights, tv and other devices. This gift got me thinking to take it to the next level. I had already set up a Google Home Mini to do various tasks for me at that time, but now it was time to kick it up a notch: implementing Google Assistant in my Mendix project!

The idea

I remembered there was a Google Assistant module in the Mendix AppStore. I wanted to check whether it would be easy to implement it in my application and use the module to create a simple command. My idea was to ask Google Assistant about my energy consumption. Since that data was already present in the application, due to the link with my smart meter, the effort should be limited. Right?

Implementation

First things first though. It’s been a while since this project saw some action, so I wanted to upgrade the project to the newest Mendix version, which is 7.23.1 right now. The upgrade was as easy as opening the project in the new Modeler version and converting the project. No hassle. On to the fun part!

I downloaded the Google Assistant module from the Mendix AppStore and I read through the readme. I noticed the GitHub repo had a much more detailed explanation of what to do, so that became one of my more frequently visited bookmarks for the next few days.

Before we continue though, a shout-out to Conner Charlebois for his work on the Google Assistant module. Without his work, this project would have never come to fruition!

Google Dialogflow (the tool you use to create your Google Assistant Agent) has the ability to get the answers to a question through a webhook. This basically means it will execute a POST request to a REST service with the name of the intent (question asked) and any parameters. The webhook requires a specific response in the form of an HTTPResponse object. Thankfully all this is supplied by the Mendix module!

It took some trial and error, mainly because developing a valid Google Agent is a completely different skill. Dialogflow has a built-in testing tool, so you can check your questions and answers. And even better, you can have a look at the raw requests and responses received. Looking at these requests, I noticed that they did, in no way, match the JSON Snippets in Mendix.

That’s when I learned the Mendix Module was actually built for the V1 API, which is now legacy. The V2 API is completely different. This not only meant the request received by my REST service was different, but the actual response I had to supply had to be different as well.

Now, I could have chosen the easy way out, and set that little radio button to use the V1 API. There’s no fun in that though, so I chose the hard way. Which meant I had to overhaul the complete module to work with the V2 APIs.

After some hard work I managed to rebuild the JSON snippet and mapping, so that it would correctly map the request sent by Google to my Mendix application. All that I had to do was getting the correct data and passing that back in a ‘SimpleResponse’ with a ‘textToSpeech’ attribute. Since this response changed from the V1 API as well, I had to create this mapping myself and send back the HTTPResponse with the JSONString as its content.

Now, to actually use this Agent publicly, you would have to go through Google’s review process. And that means it would be publicly available to everyone who would say: OK Google, talk to Power Consumption (Power Consumption is the name of my agent). That’s not what I wanted, so I didn’t actually release it. Since it’s my personal account I can still use the test version though, and what do you know. It works! The image below is from the testing tool, but it works on every Google Assistant device which is connected to my account. How cool is that!

Conclusion

Summarizing: I created my own Google Agent, which would respond to me asking ‘What’s my Power Consumption?’ by getting the required data from my Mendix application.

All in all, this was a great learning experience where learning how to build a “Conversational AI” was the hardest. And this was only an agent with one intent.

The module now works with the V2 API, and supports a SimpleResponse type of response. Google Assistant supports many more types of responses, but they all have a completely different format and thus requires a separate mapping. So far, this works and that makes me a happy developer!

Working with Google Assistant has got me thinking this tool is really powerful when combining it with Mendix and I can’t wait to learn more!

Slight spoiler: Work has continued on the Google Assistant module, and currently allows for more rich responses, such as images, videos, tables, etc. Stay tuned for more information 😉

April 2, 2019
Paul Ketelaars