What are the tips for creating AR experiences? Roundtable Discussion with Participants of the "Local withAR Hackathon"

In early February 2022, withAR and Pretia Technologies (hereinafter Pretia) held the " Local withAR Hackathon," an event challenging collaborations between AR and other industries.

This event facilitated collaborations across diverse fields, including architects, local government officials, and creative company CEOs. Members of Dentsu Group's cross-organizational unit "XRX STUDIO" also participated as planners. Focusing on the theme of " tourism," a total of 17 AR solution prototypes were completed within one week.

This series explores the potential of AR × hackathons, featuring introductions to the works born from the hackathon. Part 2 presents a roundtable discussion with the participating engineer and Dentsu Inc. planners.

Our guests are: Mr. Takahiro Taki, CEO of U. Inc. (creator of " Tsurezure AR BALOOON "), and Ms. Hiromi Michiyori, Dentsu Inc. XR Art Director; Mr. Tsukasa Tada, Product Manager at Cloud Circus Inc. (creator of " LOOK BACK - SHINJUKU - "), and Mr. Tadahiko Maruyama, Dentsu Inc. Planner. Dentsu Inc.'s Yuto Shioda served as facilitator, asking about key points in creating the works and lessons learned from this hackathon.

Series Part 1: Nothing beats a moving proposal. AR × Hackathon for solving challenges

Expanding the "essence" of a city through technology

Shiota: I'm Yuuto Shiota from Dentsu Inc. Business Co-creation Bureau. This time, we'll delve deeper into the hackathon with engineers who actually created works at the "Local with AR Hackathon" and members of our XRX STUDIO. First, we'll hear from Mr. Taki and Mr. Michiyori, who created "Tsurezure AR BALOOON." Please briefly introduce yourselves.

Taki: I'm Taki from U. Our company reconstructs various technologies into entertainment, and we've recently been focusing on projects in the AR/VR field.

Michiyori: I'm Michiyori, working as an XR Art Director and AR Designer at Dentsu Inc. BX Creative Center. I've been involved in advertising production for many years, but in recent years I've expanded my focus to AR and am also part of the AR division at XRX STUDIO. For this hackathon, I assisted Mr. Taki's team with designing the experience, including visualization, and handled the art direction.

Shioda: Thank you. What kind of work is "Tsurezure AR BALOON" that we created this time?

Taki: It's an AR service that lets users attach messages to real-world spaces. We created it to expand the unique character of a place by allowing users to pin their thoughts and feelings right there, right then.

Shioda: Could you tell us about the background that led you to create this work?

Taki: When considering this theme of "revitalizing communities," the typical approach often involves uncovering a region's charm and visualizing its value. However, from my experience working with local governments as an IT PR advisor, I felt this approach can sometimes lead to every town becoming cookie-cutter.

I didn't want the future city shaped by XR's development to become just as formulaic as the real world . I wanted to challenge myself to use technology to define and expand a city's unique character. That's when I conceived an approach where that character and charm would emerge from the people living there and spending time there. For example, I'm from Kobe. There's a famous coffee shop near Sannomiya Station, and smelling its aroma makes me feel like I'm back home. I thought that if countless such anecdotes accumulated as murmurs in that place, its "essence" would become visible. That's the inspiration behind this work.

Shioda: Indeed, hometowns have that unique nostalgia and enduring charm. At the same time, new charms emerge as things change over time. What struck me as truly unique about this work is how "authenticity" emerges through everyone's murmurs, blending both the changing and the unchanging charms.

Michiyori: Exactly. The experience of words gathered in that place being shared across eras and time-space feels fresh. I was moved by how communication that would normally vanish in that place and moment can connect and expand beyond time. Even viewed from the perspective of solving local issues, aiming for a growth-oriented regional revitalization where everyone sustainably nurtures the town's charm is wonderful.

Building an essential common language for AR collaborations lacking a shared framework

Shioda: This time, you created the work using Pretia Technologies' AR cloud platform "Pretia." Were there any particular challenges or innovations during the production process?

Taki: Pretia's unique feature is its ability to scan the real world and perform spatial alignment. This inspired us to create a service where messages could be attached to spaces. The challenge was how the intensity of shadows changed depending on the time of day and weather. For example, if a shadow fell on part of a monument, altering its light and dark areas, the mapping recognition accuracy would decrease. To address this, we made operational adjustments like scanning at the same time of day whenever possible and avoiding peak foot traffic hours.

Shioda: I see. Michiyori-san, I believe you supported the project from a design and art direction perspective. Could you tell us what aspects you focused on?

Michiyori: Since the concept phase designed a very simple, refined experience , I focused on making that entire sequence feel as organic and human-like as possible. For instance, I added details to the speech bubble design to convey personality and increased the variety of shapes. I prioritized maintaining a sense of warmth so that reading the whispers wouldn't feel cold or distant.

On the other hand, objects displayed via AR tend to blend easily into the surrounding scenery, and monitors are susceptible to lighting conditions. Therefore, we consciously separated from the CMYK or RGB-based thinking typical of OOH or print media. We adjusted background colors and font designs to ensure readability when displayed in the real world via AR, carefully avoiding any user stress or discomfort.

Shiota: So, you needed to consider a design approach unique to AR?

Michiyori: Exactly. Just as approaches differ for print and OOH, AR has its own distinct characteristics and context. We focused on thinking about how to make information intake as natural as possible, almost like switching the OS in our minds to AR specifications.

Shioda: Thank you. Collaboration across corporate and industry boundaries is one of the appeals of AR hackathons. Could you share what you found challenging and what potential you saw from your perspectives?

Taki: Since AR lacks a common language, I felt establishing a shared vocabulary was essential for collaborating with members from different industries. This time, we focused on gathering and sharing case studies, having everyone experience AR firsthand, and bringing in concrete references to build that common ground.

Michiyori: Regarding design, we discussed detailed design rules with the developers right from the planning stage. For example, making speech bubble sizes variable based on text volume, adjusting font sizes and margins—establishing this shared understanding beforehand allowed for flexible implementation later, which I believe led to a better final product.

This time, collaborating with development experts and mutually influencing each other's growth allowed us to drive both individual development and the exploration of the AR field. Personally, I see this hackathon as a major catalyst for rethinking how we, as corporate professionals, can engage with cities and the lives of their residents, and how we can make that engagement meaningful.

An AR service offering simulated experiences of stories gathered at Shinjuku Station

Shioda: Next, I'd like to hear from Tada and Maruyama, who created "LOOK BACK - SHINJUKU -". First, please introduce yourselves.

Tada: I'm Tada from Cloud Circus, serving as Product Manager for AR promotion tools like "COCOAR" and "LESSAR." I primarily handle business and development planning, creating products aimed at making AR commonplace in society.

Maruyama: I'm Maruyama from Dentsu Inc. After graduating, I worked for about five years as a strategic planner, focusing on marketing and communication strategies. Later, I joined a department supporting clients with new business development. Currently, I'm in the Creative Bureau, concentrating on producing mass advertising like copy and commercials. At the same time, I often work on projects where technologies like AR and AI are central to the ideas.

Shioda: Thank you. "LOOK BACK - SHINJUKU -" was voted the top work by participants and visitors. Could you please reintroduce the work?

Tada: It's a work that visualizes past tweets posted in Shinjuku using AR. When you hold your smartphone up to the Docomo Tower in front of Shinjuku South Exit Station, the clock reverses, and a massive flock of blue birds descends. These birds then land at various locations throughout Shinjuku, and tweets linked to each specific spot appear.

Originally, when I wanted to express the value of Shinjuku Station, I thought its uniqueness lay not in objects but in experiences—that the memories spent in Shinjuku and the emotional movements felt at that time are what create its irreplaceable charm. Based on that idea, I worked on creating a story that flows smoothly from reality to unreality, and an experience that achieves a kind of "pseudo-synchronization" with the traces left by someone who was there, while receiving advice from Mr. Maruyama.

Maruyama: Deciding on the first day of the hackathon to "focus on creating a story that moves people's hearts rather than practicality" turned out to be the right choice. We chose Shinjuku as our subject because both Tada-san and I spent a long time there and each had our own foundational experiences tied to the place, so we believed we could create a realistic project.

Shioda: I think it was a work that perfectly delivered the "simulated synchronization" experience. Were there any difficult or technically challenging aspects during this production?

Tada: While AR object detection accuracy isn't quite there yet, I find VPS (Visual Positioning System) – the spatial recognition function – to be a very compelling technology. The real challenge in utilizing VPS was tracking outdoor environments. Scenery changes with day/night cycles and lighting conditions, and during this hackathon, snowfall completely transformed the landscape. This made me realize that when working outdoors, it's crucial to anchor key points in locations where changes are minimal.

Shioda: This time, we incorporated the Docomo Tower into the AR experience. Was there any difficulty in implementing AR on such a large outdoor structure from a distant location?

Tada: We recreated the real scene in Unity, but the human eye and camera lens perceive distance differently the farther apart they are. This creates a misalignment between the real world and the AR space. The tough part was that the only way to map it accurately was to do tedious on-site adjustments. However, these kinds of challenges only become apparent once you try, so I'm glad we took on the challenge.

Shared skill sets and values determine the quality of output

Shioda: Maruyama-san, you supported the project from a planner's perspective. Were there any particular approaches or points you focused on?

Maruyama:Similar to the advertising production process, I constantly considered how the audience experiencing this project would feel. As my role, I focused particularly on two things: organizing the context and story tailored to the experience, and preparing concrete ideas for expressive direction, backed by examples.

For example, with the clock visuals, we referenced expressions from movies and games, aligning our vision by saying things like, "This way of turning back time doesn't quite match the image," or "This feels closer to the sense of scale we want when the birds burst out."

Mr. Tada was incredibly reliable, quickly prototyping and testing ideas that came up in discussions. This allowed us to make fast decisions on what to prioritize building within the limited hackathon timeframe.

Shioda: Indeed, hackathons have short project timelines and involve collaborating with different members than usual, so prioritizing tasks and combining skills seem like critical points.

Maruyama: That's right. This time, we consciously set aside time upfront to start discussions from our core value criteria. I think that helped minimize moments of hesitation later in the process.

Personally, in my day-to-day work, I used to collaborate mostly with familiar members. But in recent years, I've had many opportunities to work with people outside the advertising industry, leading to more projects with members who have different values and cultures. Because of this, I want to place even greater importance going forward on aligning the team's understanding based on value criteria, just like we did this time.

Tada: AR is ultimately just one technology. If you can't determine what experiential value to offer users through it, and what approach is best to achieve that, you won't end up with a good product or service. AR technology tends to focus on augmenting physical objects, but I think it's crucial to include people on the project who can consider whether it's also augmenting human emotions.

Could the next killer content emerge from an AR hackathon?

Shiota: Finally, I'd like to ask our four participants to summarize their impressions of the "Local with AR Hackathon."

Taki: We saw truly diverse perspectives and ideas emerge. It was incredibly educational, not just technically, but also in terms of project management and presentation methods. That said, AR still doesn't have universally high demand, and I believe it's an area without clear answers. We need to keep searching for killer content, and in that sense, the hackathon approach felt extremely useful.

Tada: AR is a field where possibilities expand through the combination of skills. I believe it can develop into something even better if various players can join without feeling too much of a barrier. Especially in this hackathon, many participants were skilled not only in technology but also in idea generation and experience design. This made me reconsider the importance of creating experiences that resonate with end-users.

Michiyori: It was a great opportunity to consider how AR can contribute to businesses and society, and how it can expand human emotions and creativity. Also, hearing everyone's discussions really drove home the importance of a common language. When collaborating, discrepancies in technical terms, phrasing, or nuances inevitably slow things down. I want to personally experience existing AR works, business cases, and the platforms supporting them, so I can articulate these concepts clearly.

Maruyama: While this applies to hackathons in general, not just AR, I was reminded that hackathons are an excellent way to learn about unfamiliar technologies and challenges.

Even beginners, when trying to create something, will naturally encounter questions and ideas during the process. Having an environment where you can immediately consult peers or experts and get feedback is incredibly valuable. While this event focused on AR, I feel I gained some tangible understanding of what's currently possible with the technology and its future potential. Once again, thank you to everyone who organized this event.

Shioda: Thank you all for your valuable feedback.

---

Thank you for reading this article. If you're interested in utilizing the metaverse or AR, please feel free to contact us.

[Contact]
Dentsu Inc. XRX STUDIO xrxstudio@dentsu.co.jp