{"id":519,"date":"2020-04-25T18:16:52","date_gmt":"2020-04-25T18:16:52","guid":{"rendered":"https:\/\/projects-old.etc.cmu.edu\/memoiar\/?p=519"},"modified":"2020-05-08T16:06:54","modified_gmt":"2020-05-08T16:06:54","slug":"memoiar-post-mortem","status":"publish","type":"post","link":"https:\/\/projects-old.etc.cmu.edu\/memoiar\/index.php\/2020\/04\/25\/memoiar-post-mortem\/","title":{"rendered":"memoiAR Post Mortem"},"content":{"rendered":"\n<h1 class=\"wp-block-heading\"><strong>Introduction<\/strong><\/h1>\n\n\n\n<p><em>Rosenstrasse: We Choose Each Other<\/em> (<em>WCEO<\/em>) is a location-based, multiplayer roleplaying AR experience created by ETC project team MemoiAR under the direction of Jessica Hammer in the Spring of 2020. Based on Jessica Hammer&#8217;s board game <em>Rosenstrasse<\/em>: <em>We Choose Each Other<\/em> explores the future of storytelling in AR for the Snap Creative Challenge.<br><\/p>\n\n\n\n<p>The team consisted of the following ETC Students:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Programmer, Technical Artist<\/strong>: Ran \u201cRyan\u201d Zhang, 1st Year<\/li><li><strong>3D Artist<\/strong>: Qiyi \u201cChelsea\u201d Tang, 1st Year<\/li><li><strong>UX, Graphic Designer<\/strong>: Mingzhi \u201cMinz\u201d Cai, 1st Year<\/li><li><strong>Co-Producer, Game Designer<\/strong>: Roderick Luke \u201cDerek\u201d Chan, 1st Year<\/li><li><strong>Co-Producer, 3D Artist<\/strong>: Sally Im, 1st Year<\/li><\/ul>\n\n\n\n<p><strong><\/strong>Halfway through the semester, the project (along with all ETC projects) moved to working completely remotely in response to the 2020 COVID-19 epidemic. The following is a post-mortem analysis of what went well, what went not so well, what we learned, what we would do with more time, and the life of our work after the end of the semester.<\/p>\n\n\n\n<p>The remaining steps for memoiAR are to conclude final changes to the demo, conclude\u00a0 a documentation package for future development, and to remotely present the game to the Snap Creative Challenge Summit on Wednesday June 17th.\u00a0<br><br>Our client Jessica Hammer expressed a desire to pursue funding for potential future development of the concept (thus the documentation package). Ideally, a future team should be able to read the documentation and continue development on its design themes. Additionally, PhD candidate Erica Cruz expressed interest in pursuing academic publication of the design process of this project to interested HCI publications. The team plans to communicate with Erica further about the project following the conclusion of the semester.\u00a0<\/p>\n\n\n\n<h1 class=\"wp-block-heading\"><strong>What went well&nbsp;<\/strong><\/h1>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Productive and positive client relationship&nbsp;<\/strong><\/h2>\n\n\n\n<p>The team enjoyed a productive and positive relationship with our client, Jessica Hammer. The client was communicative, clear with her expectations and goals, as well as enthusiastically supportive of the team\u2019s progress (even after the transition to remote work and the pivot in final deliverables).<br><\/p>\n\n\n\n<p>Logistically, the client had an intimate understanding of ETC projects and deliverables and was able to tailor her goals to the unique work environment of the ETC. She knew what she could expect from 16 weeks of work from an interdisciplinary team of graduate students.<br><br><strong><\/strong><strong> <\/strong>Creatively, Jess was able to advise the project as a subject matter expert on the historical period. The game is historically accurate thanks to her input. Additionally, being an original designer of the <em>Rosenstrasse<\/em> game, Jess provided useful feedback and suggestions on interactive storytelling and roleplaying. She also made space for new interpretations of her original mechanics and story. At every client meeting, she was willing to discuss opposing design choices and playtest unorthodox designs.<br><\/p>\n\n\n\n<p><strong><\/strong>We recognize Jess Hammer to be an unorthodox client (i.e. not all ETC clients may be as involved or enthusiastic), but also recognize that much of what went well with the final product stems from her professional relationship to the project team.&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong><br><\/strong><strong>Genuinely Novel and Innovative Designs<\/strong><\/h2>\n\n\n\n<p>To quote one comment made early on in the semester, the project is like \u201ca theatrical play in which the actors don\u2019t know where they are supposed to be, what they\u2019re supposed to do, or what they\u2019re supposed to say.\u201d<br><\/p>\n\n\n\n<p>Several playtesters and faculty advisors expressed genuine interest in the idea of a multiplayer, location-based, roleplaying AR game. Despite the eclectic and seemingly-disparate parts of the design, the experience successfully functions on a technical level. Following the shift to remote work, the team was even able to create trailers of the experience that successfully explained the game\u2019s idiosyncratic design (even if the footage didn\u2019t show people actually playing the game or was \u201cfaked\u201d).&nbsp;<\/p>\n\n\n\n<p>Design innovations that we identified fall into the following categories:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Blending VR concepts with AR&nbsp;<\/strong><ul><li>Touching digital objects with the AR device: This was novel as many AR experiences focus on visuals overlaid onto reality as opposed to engaging a physical experience. Physicality in digital actions was adopted from lessons learned using VR trackers.<br><\/li><li>Multiplayer experiences using Networking: Since the focus of our game was on linear scenes and progression of on-screen text, we were able to keep both players in the same state of the game using simple TCP networking. Most commercial AR games struggle with networking since they attempt to track the location of 3D objects and use action-based mechanics vs. verbal mechanics.<br><\/li><\/ul><\/li><li><strong>Augmented Roleplaying<\/strong><ul><li>Players connecting with their partners: AR had been used in roleplaying games before (primarily to visualize board pieces or sets). We tried to place inter-player conversation at the heart of the experience and examined ways to meaningfully interact with it, beyond visualizing 3D assets.<br><\/li><li>Physically embodied actions: no AR games had used the device as a controller in addition to a camera. By taking the design paradigm of theatrical performance, the team was able to develop several scenes where players must assume a physical gesture in the real world in addition to imagining it.<br><\/li><\/ul><\/li><li><strong>Location-based AR<\/strong><ul><li>Diegetic markers and set pieces: One common criticism of AR is that the real world distracts from the digital world. We attempted to circumvent that problem by curating real-world visuals. The team recognizes that doing so placed constraints on where the game can be played (i.e. the game is best played at the Entertainment Technology Center).<\/li><\/ul><\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Shared creative vision between all team members<\/strong><\/h2>\n\n\n\n<p>Perhaps as a result of the unorthodox design of the game, the entire team was involved in all steps of design, from initial conception, to writing of scripts, and finally to execution. This made up for the lack of systematic and exact design documentation, since all members of the team would sit at the meetings where these design choices were made. Team members would also elect to be present for creative meetings rather than be notified later of the final design. Jess Hammer would later comment that the designs pitched to her \u201creflected the many hands that had helped make it.\u201d&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Remote work was mostly effective<\/strong><\/h2>\n\n\n\n<p>The team was able to transition fairly effectively to working from home. Since most of our documentation was already digital before the transition (e.g. Scrum boards, interaction scripts, meeting notes), our work process stayed mostly the same. Dependencies were communicated through instant message on Slack, tasks were tracked on Google Sheets, and assets were transferred using Perforce\/Google Drive.<br>&nbsp;<\/p>\n\n\n\n<p>One challenge to remote work was version control. Since our team only had 1 programmer, the team relied on him to manage all changes to the digital build using Git. Even though some teammates had experience using Unity, very few had used Git to control versioning on a Unity Project.<br><\/p>\n\n\n\n<p>On the whole, the team was typically able to maintain forward progress and meet deadlines during remote work.<br><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Playtest-Centric Pipeline was successful when we could playtest in person<\/strong><\/h2>\n\n\n\n<p>As a result of only having 1 programmer, the team successfully implemented a pipeline where the design of the game would remain 1 week ahead of the digital build of the game. In a given week and following the first creative meeting, the team\u2019s programmer would implement changes to the digital build while the Game designer and UX designer would create paper prototypes of future features.<br><\/p>\n\n\n\n<p>Generally speaking:<br>General Design \u2192 Paper build \u2192 Paper Playtest \u2192 Digital Build \u2192 Digital Playtest \u2192 Polish\/refine<br><\/p>\n\n\n\n<p>This typically resulted in more feedback and relatively higher-quality designs making it to the digital build. The pipeline appeared to work well prior to beginning remote work. Subsequently, the team planned to scale up production of the final 2 parts of the experience using this simultaneous workflow.<br><\/p>\n\n\n\n<p>Unfortunately, this pipeline broke down when paper playtests became impossible. While attempting to design remote playtest methods, digital work began to pile up. This forced the team to focus on the production of digital work to meet deadlines over polishing existing work. Since a significant portion of the game required players to physically move through a space together, many of the emotional beats remained theoretical and unpolished before becoming integrated into the digital build.&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Multiple Playtest Methods<\/strong><\/h2>\n\n\n\n<p>As a result of going remote, the team had to use playtest methods that captured parts of the experience, since the whole experience could not be played as previously designed.&nbsp;<\/p>\n\n\n\n<p><br>Types of playtests used are as follows:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Full playtest or \u201cDress-rehearsal\u201d playtest: This involved running people through the experience itself in 100% intended conditions. This is the conventional way to playtest a game and typically resulted in the best feedback and changes to the experience. Because our experience is relatively long, these playtests were conducted with few \u201cexpert\u201d players. This method prioritizes few, deep playtests over many, shallow playtests to look for high-quality expert opinion over volume of data.<br><\/li><li>Paper playtest: this involved sticky notes and paper cut outs to simulate on-screen UI and the specific logic of certain mechanics. Since creating a full build took up the Programmer\u2019s entire week-long sprint, the game designer and UX designer would use paper mock-ups and talk users through the intended experience. The key is to only provide as much detail as necessary to gauge players\u2019 instincts and desires. Most playtesters were willing to accept the low-fidelity of these playtests and provide feedback on a hypothetical high-fidelity version. The Tutorial\/Chapter 0 was playtested in this manner.<br><\/li><li>Composite video trailers: this involved editing together footage of live actors, recordings of the iPad app, and spatial diagrams to communicate the intended experience to remote audiences. These were time consuming and resource intensive to make; however, they were the most accurate and high-fidelity way to explain the experience to a playtester (short of actually playing the game); however, since all of the editing of the footage was accomplished by 1 member of the team, these trailers were difficult to change and modify to reflect changes in the experience itself.&nbsp;<\/li><\/ul>\n\n\n\n<h1 class=\"wp-block-heading\"><strong><\/strong><\/h1>\n\n\n\n<h1 class=\"wp-block-heading\"><strong>What didn\u2019t go well<\/strong><\/h1>\n\n\n\n<h2 class=\"wp-block-heading\"><strong><br><\/strong><strong>Technical Challenges<\/strong><\/h2>\n\n\n\n<p><strong>iPads <\/strong>were inelegant and uncomfortable to use for long periods of time. The project has never been executed fully with purely naive guests using the iPads as intended. iPads that run the experience best are also very expensive. Given the performance requirements, the experience becomes much less playable on lower-end iPads and apple devices.<br><\/p>\n\n\n\n<p><strong>&nbsp;iOS development <\/strong>also proved a great hurdle to remote development: only specific pre-registered devices can run the experience using testflight. While iPads are the premier tablet on the market, having to build the project on iOS increased build time and involved many iOS-specific steps before pushing the build to the iPad itself. Committing to iOS also meant that the game experienced weird glitches when switching platforms to Android.<br><\/p>\n\n\n\n<p><strong>Marker-based mobile AR<\/strong> in its current form is unreliable and prone to \u201cglitching\u201d out. The client directed the team (and the programmer) to pursue completion of the entire experience over ironing out the stability issues. The team understood that gameplay footage could be edited to avoid showing unsightly glitches caused by a combination of lighting, marker quality, robustness of Vuforia\u2019s marker-recognition functionality. We were also unable to test marker recognition outside of the programmer\u2019s own home due to iOS deployment issues.&nbsp;<br><\/p>\n\n\n\n<p><strong>A programming bottleneck <\/strong>occurred due to the team only having 1 programmer able to modify the Unity project and implement new digital features. Additionally, due to working remotely, the only member on the team capable of actually running the experience was the programmer. No one else on the team had iPads or an iMac to build the project. While we tried to design around simple theoretical implementation, the programmer still had to prioritize completion of new functionality over changing UI text, implementing sound effects, fixing lighting effects, etc.&nbsp;&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Complex and cumbersome design pillars&nbsp;<\/strong><\/h2>\n\n\n\n<p>While the team successfully understood the original <em>Rosenstrasse\u2019s<\/em> complex and moving emotional design, we struggled to turn the original\u2019s emotional design into elegant design principles. Our design pillars always felt <strong>confusing <\/strong>or <strong>difficult to make actionable <\/strong>given our project\u2019s goal to make a \u201cfull experience.\u201d Consequently, we stopped using the design pillars late into the semester.&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Complex and ambitious narrative left unpolished&nbsp;<\/strong><\/h2>\n\n\n\n<p>Having an <strong>ambitious and complex narrative <\/strong>meant that the team struggled to do justice to the story as a whole in all its nuances and complexities. While we had a sense of high-level narrative concepts (e.g. what happens in a part, what the general feeling of a part should be), we were never able to dig deep into specific moments of the story and what is supposed to go through the players\u2019 minds as they play those moments. Holocaust storytelling is in and of itself a great challenge. For the purposes of building as much of the experience as possible, we brushed broad strokes for the narrative and it shows. As much as the game is designed to <strong>immerse players <\/strong>historically in 1940s Berlin, <strong>the project also struggled with what that really meant in terms of the game\u2019s visuals, UI, and text itself<\/strong>. Arguably a more cohesive, intimate, and polished 10-minute experience would have accomplished our design goal of telling Max and Annaliese\u2019s story of \u201cwe choose each other\u201d better than the 30-minute experience.<br><\/p>\n\n\n\n<p>Additionally, the team\u2019s background and storytelling skills leaned towards theater, film, and studio art. These skills required access to the <strong>physical location in order to playtest<\/strong> and iterate. In contrast, a team with a dedicated writer or even script-writer might have had more success creating a story that could be refined without a physical location. The team believed that story and narrative would emerge from <strong>iterative playtests of the full experience<\/strong>. When those full-playtests were not possible, less and less progress was made to refine the narrative itself.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Roleplaying is still difficult for most players<\/strong><\/h2>\n\n\n\n<p><strong>Roleplaying<\/strong> remains a high-bar for most players of the game. The team never came to a satisfying answer to <strong>what successful roleplaying looks like<\/strong>. Does it look like a family of 4 walking into a museum to learn about the Holocaust? Does it look like improv actors holding iPads? When we played <em>Rosenstrasse<\/em>, we knew it was working because we didn\u2019t want to go home. We wanted to see what happened to their characters. We physically leaned over the table. We never had a chance to see what types of roleplaying emerged in our players and then choose which types of roleplaying behaviors to develop further.&nbsp;<br><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Playtesting was difficult<\/strong><\/h2>\n\n\n\n<p><strong>Niche and Changing Audience<\/strong>: The project began with the explicit goal of being deployed to the IMX conference in Barcelona and pivoted to laying down groundwork for future potential deployment to museums or other academic conferences. Our understanding of our audience was the overlap between academics, fans of history, and experienced roleplayers. This small audience required us to process all feedback through the lens of if our playtesters fell under our intended audience. Additionally, identifying this audience and having them play the game was difficult. Of all the playtests conducted this semester, maybe 5 individuals met the above criteria. Because the experience lives in one location, specificity on audience will drive successful designs on how best to incorporate them into the experience.&nbsp;<\/p>\n\n\n\n<p><strong>Remote playtesting of a location-based game<\/strong> plagued the second half of development. The team ultimately chose to not alter the design of the game to be easier to develop remotely. Since the experience is location-specific, the game was never truly and authentically playtested. This combined with iOS-specific development restrictions, meant that the game could never be deployed or played outside of the home of the programmer. While we did our best to work around these challenges (i.e. test android deployment, paper playtest through zoom, shift project goals to documentation, create trailers of edited footage, etc), final rounds of feedback pointed out the lack of polish in some parts of the game.&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Physical Materials do not exist<\/strong><\/h2>\n\n\n\n<p>Setting aside the problems with remote work, designing the physical materials of this game would have been a time-consuming and serious undertaking: it could be one teammate\u2019s entire job. Objectively, the team was unable to realize its design goal of an \u201cimmersive location-based\u201d experience beyond the high-level design phase. None of the team\u2019s expertise or technical skills lent themselves to building cardboard street signs or capturing the feeling of blackbox theater spaces. Even if we had gone to Barcelona, attendees of the conference would have likely been distracted by the walls, floors, and chairs of the venue itself.&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Audio-based mechanics were implemented very late<\/strong><\/h2>\n\n\n\n<p>While compelling, the proximity-based voice lines and sound-based mechanics remain unpolished and rough in their current state. A sound designer was also involved in the team very late in the semester. While audio is a crucial part of effective storytelling, we were unable to create the sounds of our game until very late in the semester. Additionally, we never created a unique musical identity for the game\u2019s trailers.&nbsp;<\/p>\n\n\n\n<h1 class=\"wp-block-heading\"><strong><\/strong><\/h1>\n\n\n\n<h1 class=\"wp-block-heading\"><strong>Lessons Learned&nbsp;<\/strong><\/h1>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Playtesting is key to a high-quality final product.&nbsp;<\/strong><\/h2>\n\n\n\n<p>We had wanted to playtest often, with clearly defined questions, and diverse playtesters. This was also the key to many of our first design iterations (e.g. sign shape matching, iPad as mirror, etc). The project suffered when it couldn\u2019t be playtested fully. While we developed a few different ways to playtest specific parts of the experience, the final product reflects a lack of polish and refinement that a final deliverable needs.&nbsp;<\/p>\n\n\n\n<p>Potentially, we could have pivoted the design of the project itself to be more easily playtested. Alternatively, if we had prioritized a narrative playtest tool, we could have addressed playtest questions sooner.&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Interactive storytelling is very complicated and can be one person\u2019s whole job.<\/strong><\/h2>\n\n\n\n<p>Conventional roleplaying is a strange and foreign concept to many people. The current project never came to a strong consensus about what a successful and emotionally moving playthrough of our game would look like. Text is also a necessary but insufficient medium to communicate narrative. It requires a lot of exactness to do effectively.&nbsp;<\/p>\n\n\n\n<p>Writing could also be the entire job of one team member. The team tried to disperse the responsibility of narrative design, but ultimately found most teammates focusing on their primary responsibilities before refining the narrative.&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Simultaneous production pipelines work best with familiar, repeatable processes<\/strong><\/h2>\n\n\n\n<p><strong><\/strong>While we had a plan to work on parts of the experience simultaneously, we underestimated how long polishing each part would take. While our team was able to execute and build new parts to the experience, our pipeline did not adequately budget time for incorporating feedback or testing. Additionally the paper design phase of our pipeline was never really able to playtest mechanics and designs before being incorporated into the digital product. Since the steps for developing Part 3 differed from the steps to develop Part 2, we wound up with inaccurate time and work estimates.&nbsp;&nbsp;&nbsp;<\/p>\n\n\n\n<h1 class=\"wp-block-heading\"><strong><\/strong><\/h1>\n\n\n\n<h1 class=\"wp-block-heading\"><strong>Last Steps and Wrap up for the Project<\/strong><\/h1>\n\n\n\n<p>The remaining steps for memoiAR are to conclude final changes to the demo, conclude&nbsp; a documentation package for future development, and to remotely present the game to the Snap Creative Challenge Summit on Wednesday June 17th.&nbsp;<br><\/p>\n\n\n\n<p>Our client Jessica Hammer expressed a desire to pursue funding for potential future development of the concept (thus the documentation package). Ideally, a future team should be able to read the documentation and continue development on its design themes. Additionally, PhD student Erica Principe Cruz expressed interest in pursuing academic publication of the design process of this project to interested HCI publications. The team plans to communicate with Erica further about the project following the conclusion of the semester.&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Introduction Rosenstrasse: We Choose Each Other (WCEO) is a location-based, multiplayer roleplaying AR experience created by ETC project team MemoiAR under the direction of Jessica Hammer in the Spring of 2020. Based on Jessica Hammer&#8217;s board game Rosenstrasse: We Choose Each Other explores the future of storytelling in AR for the Snap Creative Challenge. The<a class=\"more-link\" href=\"https:\/\/projects-old.etc.cmu.edu\/memoiar\/index.php\/2020\/04\/25\/memoiar-post-mortem\/\">Read more<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-519","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/projects-old.etc.cmu.edu\/memoiar\/index.php\/wp-json\/wp\/v2\/posts\/519"}],"collection":[{"href":"https:\/\/projects-old.etc.cmu.edu\/memoiar\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/projects-old.etc.cmu.edu\/memoiar\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/projects-old.etc.cmu.edu\/memoiar\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/projects-old.etc.cmu.edu\/memoiar\/index.php\/wp-json\/wp\/v2\/comments?post=519"}],"version-history":[{"count":2,"href":"https:\/\/projects-old.etc.cmu.edu\/memoiar\/index.php\/wp-json\/wp\/v2\/posts\/519\/revisions"}],"predecessor-version":[{"id":551,"href":"https:\/\/projects-old.etc.cmu.edu\/memoiar\/index.php\/wp-json\/wp\/v2\/posts\/519\/revisions\/551"}],"wp:attachment":[{"href":"https:\/\/projects-old.etc.cmu.edu\/memoiar\/index.php\/wp-json\/wp\/v2\/media?parent=519"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/projects-old.etc.cmu.edu\/memoiar\/index.php\/wp-json\/wp\/v2\/categories?post=519"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/projects-old.etc.cmu.edu\/memoiar\/index.php\/wp-json\/wp\/v2\/tags?post=519"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}