HoloLens + Vuforia:<\/strong><\/p>\n Pros:<\/p>\n Cons:<\/p>\n Zed Camera + Vuforia:<\/strong> Zed Camera + ArUco Markers<\/strong><\/p>\n Pros:<\/p>\n Cons:<\/p>\n Thus, we have decided to move forward with the Zed Camera attached to an Oculus Rift and uses ArUco markers to track the game pieces.<\/p>\n Planned AR Pipeline<\/strong><\/p>\n Conversation with the AR character is our most important mode of interaction\u00a0<\/em>between the player and the application. Therefore, speech recognition and processing is our other core feature.<\/p>\n Here is a portion interaction chart defining how the interaction between the agent and the player from our week’s weeks blog post. Thus, we need a 3-step system that can convert Speech to Text, process it and convert the processed data into speech. We make use of multiple available APIs to achieve this.<\/p>\n Step 1: Google Cloud Speech – Speech to Text<\/strong><\/p>\n We are really inclined to use Google Cloud\u00a0Speech API<\/a> to convert our users’ input into text which can later be processed by the “brain” of our program to give a corresponding response. We use the Unity Lowkey Parser<\/a> project to integrate this into a Unity project.<\/p>\n We loved it for many reasons, one being that it was able to understand multiple English accents. It is a cloud platform that uses neural networks and hence continuously learns from every user that uses these services in the world.<\/p>\n Step 2: Dialogflow for response interpretation<\/strong><\/p>\n Now that we have a\u00a0 text form of the spoken input, we need to process it and give back an appropriate response. This\u00a0processing\u00a0<\/em>is managed by a Google service called Dialogflow<\/a>. Dialogflow\u00a0contains a Language Understanding Model that is capable of taking in the given text dialog and interpret it based on the configurations it is set to. The configurations also contain the set responses that need to be returned in every case.<\/p>\n Step 3: Convert Response to Speech<\/strong><\/p>\n Finally, the character needs to speak out the interpreted response to the player. We first tried the IBM Watson to convert the text into speech. The text response wrapped as a JSON in the previous step is unwrapped and converted to SSML (Speech Synthesis Markup Language), a standard markup language for synthetic speech applications. IBM Watson is capable of reading this markup and converting it to an audio clip which gets streamed into Unity.<\/p>\n One major caveat with IBM Watson has been that it contains only one voice – Allison, which replicates the tone and accent of a female American speaker. We feel that this voice wouldn’t be as dyanamic or suited for the kind of character design we are looking at. Therefore, we are plannig to try a much newer (and hence, less documented) text to sound API owned by Amazon, called Amazon Polly<\/a> which also leverages its cloud AWS services.<\/p>\n We are also one week away from Quarters Open House. We have also been preparing ideas, plans and demos to show during quarters. We are also in the process of finalizing our branding materials (posters, half-sheets, and logos) this week!<\/p>\n Like mentioned in the last post, Ako explored both the passthrough AR and speech processing technologies last week. We did some prototyping, some vetting and, finally, settled on the design decisions to fit the tech. This was to make sure that we spend our next weeks focussing on the core essence of our project, which… \n
\n
\n<\/strong>Not supported due to the stereo nature of the video feed from the Zed Camera.<\/p>\n\n
\n
![\"\"](\"https:\/\/projects-old.etc.cmu.edu\/ako\/wp-content\/uploads\/2019\/02\/Steps-1-300x48.png\")
<\/p>\n![](\"https:\/\/i.imgur.com\/YtCUog8.gif\")
<\/p>\n![\"\"](\"https:\/\/projects-old.etc.cmu.edu\/ako\/wp-content\/uploads\/2019\/02\/Steps-2-e1549316561564-300x45.png\")
![](\"https:\/\/i.imgur.com\/yAY9Q9B.gif\")
<\/p>\n![\"\"](\"https:\/\/projects-old.etc.cmu.edu\/ako\/wp-content\/uploads\/2019\/02\/Steps-3-e1549324086444-300x43.png\")
<\/p>\n![](\"https:\/\/i.imgur.com\/myl7W04.gif\")
<\/p>\nThe Conversation Part<\/h2>\n
\n![\"\"](\"https:\/\/projects-old.etc.cmu.edu\/ako\/wp-content\/uploads\/2019\/01\/IMG_20190122_165543693-300x271.jpg\")
\nAs we can see, the\u00a0agent gets both\u00a0asked questions and answers questions.<\/em><\/p>\n![\"\"](\"https:\/\/projects-old.etc.cmu.edu\/ako\/wp-content\/uploads\/2019\/02\/google-sound-api-1.png\")
<\/p>\n![\"\"](\"https:\/\/projects-old.etc.cmu.edu\/ako\/wp-content\/uploads\/2019\/02\/dialogflow.png\")
<\/p>\n![\"\"](\"https:\/\/projects-old.etc.cmu.edu\/ako\/wp-content\/uploads\/2019\/02\/ibm-watson.png\")
<\/p>\nPreparing for Quarters<\/h2>\n
![\"\"](\"https:\/\/projects-old.etc.cmu.edu\/ako\/wp-content\/uploads\/2019\/02\/ako-poster-small.jpg\")
<\/p>\n![\"\"](\"https:\/\/projects-old.etc.cmu.edu\/ako\/wp-content\/uploads\/2019\/02\/ako-halfsheet-back0.png\")
<\/p>\n","protected":false},"excerpt":{"rendered":"
Read more<\/a><\/p>\n","protected":false},"author":2,"featured_media":188,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-160","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/projects-old.etc.cmu.edu\/ako\/index.php\/wp-json\/wp\/v2\/posts\/160"}],"collection":[{"href":"https:\/\/projects-old.etc.cmu.edu\/ako\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/projects-old.etc.cmu.edu\/ako\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/projects-old.etc.cmu.edu\/ako\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/projects-old.etc.cmu.edu\/ako\/index.php\/wp-json\/wp\/v2\/comments?post=160"}],"version-history":[{"count":14,"href":"https:\/\/projects-old.etc.cmu.edu\/ako\/index.php\/wp-json\/wp\/v2\/posts\/160\/revisions"}],"predecessor-version":[{"id":236,"href":"https:\/\/projects-old.etc.cmu.edu\/ako\/index.php\/wp-json\/wp\/v2\/posts\/160\/revisions\/236"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/projects-old.etc.cmu.edu\/ako\/index.php\/wp-json\/wp\/v2\/media\/188"}],"wp:attachment":[{"href":"https:\/\/projects-old.etc.cmu.edu\/ako\/index.php\/wp-json\/wp\/v2\/media?parent=160"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/projects-old.etc.cmu.edu\/ako\/index.php\/wp-json\/wp\/v2\/categories?post=160"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/projects-old.etc.cmu.edu\/ako\/index.php\/wp-json\/wp\/v2\/tags?post=160"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}