Redefine the "Experience" from the "Mirror World"
Paradigm shift from information to experience
──High dimensionality creates new experiences
Since 2017, NEC has hosted the NEC Future Creation Forum. Looking towards 2050 and a world beyond the singularity, the Forum has held numerous discussions among experts from Japan and abroad in order to envision the ideal future.
Expert panels held at previous NEC Future Creation Forum concluded that divisions at various levels of society are at the root of many of the challenges we confront today. As a means of overcoming these divisions, the forum has proposed a future concept of a “Will Resonate Society.” The project continues with the aim of building a society where people can produce diverse value as their dreams and ambitions resonate with others.
This year's expert panels on the topics of relationship, experience, value & trust, and learning/unlearning are aimed at deepening the discussion with an eye towards implementing ideas in society. Debate on each topic will be energized by the participation of an even wider variety of experts, such as legal scholars and game creators.
Participants of the 2nd Forum
Game Creator, CEO of Enhance, Inc. / Specially-appointed professor at Keio University Graduate School
Architect, noiz partner / gluon partner
Editor-in-Chief of WIRED Japan
Experience was the topic of the second session of the 2019 Forum. With the widespread adoption of VR (Virtual Reality) / AR (Augmented Reality) technology, experiences are rapidly emerging that are entirely distinct from conventional 2D media such as photographs and videos.
How will our experiences evolve as 5G becomes widespread and enables a “mirror world” where the digital and real worlds overlap? This topic is closely connected not only to the future of entertainment and media, but also to how we think about architecture, society, and interpersonal relationships.
Participating in the second expert session were Tetsuya Mizuguchi, a game creator who is CEO of Enhance, Inc. and a specially-appointed professor at Keio University Graduate School; architect Keisuke Toyoda, a noiz partner and gluon partner; Michiaki Matsushima, the editor-in-chief of WIRED Japan; and NEC Fellow Katsumi Emura. The discussion topic of “redefining experience from the mirror world” was addressed through two questions: “what new experiences will be possible in 2050?” and “how can we realize resonating minds in 2050?”
Common ground to connect the digital and real worlds
The discussion kicked off with the question of “what new experiences will be possible in 2050?” The progress of technologies such as IoT, AI, and VR/AR has already led to the emergence of many new experiences, but the very concept of experience is certain to change with the spread of 5G and the creation of a “mirror world” where the digital and real worlds merge. Toyoda's proposed concept of a “common ground” could be helpful to thinking about experiences in the future. The discussion began by considering how common ground could become the basis for connecting the digital and real worlds and creating new experiences.
Today's theme is "redefining experience from the mirror world." WIRED Founding Editor Kevin Kelly mentioned the mirror world in his lecture at the NEC Future Creation Forum held last year at the C&C User Forum & iEXPO 2018. This idea suggests that we will soon see the dramatic emergence of a new mirror world in which the real world we live in and a digital world containing digital versions of everything completely overlap. Today I would like to discuss what could take place in the mirror world as we approach 2050, and what kind of experiences might be possible there. The first question is “what new experiences will be possible in 2050?” Toyoda-san, you have proposed the concept of “common ground” in connection to mirror world experiences.
That's right. While it is difficult to explain common ground, you can think of it as a method for representing the world that allows humans and digital agents such as avatars to interact on equal footing. When we represent physical things digitally and create an entity like a digital twin, we might assume that the avatar-like digital agent is aware of the real world, but we still have not assembled sufficient data to make that possible. That is why there needs to be a more general-purpose platform and not just separate data created by different services or manufacturers.
Surprisingly we still seem to lack a common language to tie together the real, physical world where we live and the digital world of information.
We run into very significant limitations when trying to connect the digital and real worlds. Digitization itself is simple and we are able to simulate all sorts of things, but making the real and digital worlds interoperable is a tremendous challenge.
Absolutely. I think it's a challenge that requires a vision on the part of entire industries or society as a whole. Without a clearly shared vision, we won't know how to divide functions between the real and digital worlds, or what each should do.
From information design to experience design
On the other hand, Mizuguchi-san, you design experiences in which humans enter into an expanded reality through XR (X Reality) experiences like VR games. I think your approach to humans' physical senses could be helpful to conceptualizing Toyoda-san's idea of common ground.
When we consider the larger context, I think that we are undergoing a shift from an era of information design to an era of experience design. Information design began with the invention of letterpress printing, and we figured out how to communicate through limited or lower-resolution information. However, with the invention of films and television, the rise of computers and games, and now the spread of XR technology, the resolution of information has increased and we are on the cusp of entering into an era of experience. This signals a paradigm shift from the information age that has continued for about 600 years. It will become possible for people to share or publish their experiences in ways that they couldn't before. I continue to make games because they are a peerless experiential medium.
So we are moving from an information layer to an experience layer. Similarly, Emura-san has also proposed the concept of an “experience net.”
NEC proposed this technology at the NEC Future Creation Forum last year. Up to the present day, Japanese education has always provided information, but there are many things that can only be learned or understood through actual experience. Creating an “experience net” environment where people can engage in near-realistic simulated experiences would make it easier for children to think about what they really want to do. At last year's forum, the artist Sputniko! also proposed the idea that we need to increase the value that we assign to useless things. As advances in technology expand the work that can be entrusted to AI and robots, humans should end up having a lot more free time. If we make more time available for the sort of experiences that Mizuguchi-san describes, it could change the way individuals live their lives.
But it isn't good enough just to recreate the content of the information age for the experience age. Our creative ideas begin within the mind as multimodal and complex concepts, which we had no choice but to turn into flat pictures or letters. What we really need to do is share more multimodal images. It seems to me that in the coming years, the boundaries of architecture are going to expand greatly.
The undrawable architecture of the future
How we build buildings and create cities will also change. IT companies are already increasingly moving into real-world services and markets. Toyoda-san, how do you see experience relating to architecture and urban design in the future?
We need to think in more dimensions. Next-generation cities and environments won't be conceived only in three dimensions, so it will become increasingly difficult to draw plans. Designers in the 1970s and 1980s sketched many future cities with tubes running through the air, but we don't see that sort of thing as much recently.
People used to draw worlds like science fiction.
The future city will become a complex high-dimensional entity, so any attempt to draw it would only afford a provisional and fragmentary view. The value of the city will no longer manifest only in physical form, so it will be undrawable. Therefore, it will be necessary for the people designing cities and architecture to also have a sense of designing not just in three static dimensions, but an entire entity consisting of 10 to 20 dimensions.
At the scale of a single building, the purpose of the design will become more important. Different initial concepts will likely require different data to construct. Constructing a building today only requires the data for the building itself.
Yes, the construction industry only profits when they build, so digital data is only created and used for the purpose of construction. All of the data is for construction, so it cannot be used as data for an environment for digital agents like we are discussing. It is held within the architecture industry. For example, when creating a specialized building for VR/AR games, someone like Mizuguchi-san will need to be involved from the design stage. Just as we decide the positions of outlets and switches when designing today, there will be new conditions for design, such as making it easy to recognize corners or embedding markers. If that kind of design becomes a prerequisite for creating an environment where VR can be used, thinking of the construction of a physical building as the end of the design process would be an extraordinary waste. Data about the floors, walls, and materials of buildings is scattered throughout the architecture industry and not being used. Properly managing this data could speed up the implementation of various services by five years or more.