[सोचिका निशिदा द्वारा रैंडम ट्रैकिंग]The importance and potential of “Tempest 3D Audio” to ask the development of PS5

Talking about the PlayStation 5 (PS5), the focus is on high-performance graphics and SSD-optimized loading speeds. But there are other important factors as well.

“Sound”.

The PS5 is equipped with a hardware device dedicated to 3D audio called “Tempest 3D Audio” (later called “Tempest”). When details of the PS5 first surfaced in March last year, lead system architect Mark Cerny mentioned the content and importance of Tempest.

From the presentation when the PS5’s technical design was unveiled in March last year. Mark Cerny Introduces Tempest 3D Audio

So what kind of effect does Tempest produce? And what will happen in the future? I had the opportunity to interview people directly involved in development.

Mr. Hiromi indeed, the General Manager of the Global Product Planning Department, Sony Interactive Entertainment Platform Planning and Management Department, and Mr. Kenichi Imai, Deputy General Manager of the Software Development Division, responded.

Mr. Hiromi Wakai, General Manager, Global Product Planning Department, Sony Interactive Entertainment Platform Planning and Management Department

Mr. Kenichi Imai, Deputy General Manager of Sony Interactive Entertainment Software Development Division

What is the realization of 3D audio by PS5 with Tempest?

What is 3D audio? I do not think it is necessary to explain it more to the readers of this magazine, but I would like to review the time.

3D audio is a sound that gives the sound a three dimensional feel, such as reading. Similar to CG, we aim for a realistic mode of hearing by arranging at the location where the sound is emitted in 3D space and reproducing “how it sounds from the person in the center”. Such a mechanism is called “object-based audio”. “Dolby Atmos,” which is widely used in films and is also used in UHD BD and distribution, and “360 Reality Audio,” which is a 3D audio format that Sony developed in Japan Started, are also object-based.

Part of the reason why 3D audio is important in games is that “sound conditions often play, which greatly affects gameplay.” This is a general analogy, but in FPS, it is very valuable to be able to get information about where the enemy is and where the bullets are coming from. Therefore, it is used with PS3 / PS4, Xbox 360 / One generation and PC. He is the important factor.

However, there are some problems.

Simply put, the 3D audio of the PS5 Tempest is a technology intended to solve that problem.

This can be seen from the fact that Tempest was defined as a key differentiator in the PS5 project. Mr. Wakai and Mr. Imai explained their views at the planning level in this way.

Mr. Vakai (honorable title given below):PS5’s main message is “Play Has No Limits”. There was discussion from the planning stage of the PS5 or “what will happen to the next generation PlayStation”, “what kind of user experience is important”, but what came out was “immersive feeling.)” Was. . With 3D audio, you can feel like you’re in the game, so it was deployed in SSDs and haptech triggers (along with controllers), an important factor.

Imai:After the release of PS4, the “next” study began, but the question was “what to do if I want to increase the immersive spirit?” So I thought, “I want to shine a light on the sound.”

3D audio is very effective and in itself is very old. However, what should we do to increase that drowning sentiment further?

The resolution and number of 3D field objects that are configured as 3D audio increases make it finer and more immersive, but the amount of computation also increases, so is the limit for CPU, GPU, and general DSP . Processing with. I want to allocate as many resources to audio as possible while allocating computational resources to the game side. In short, we came to the conclusion that it is desirable to have dedicated hardware to get rid of resource limitations.

Tempest’s apparatus is open to the public, but Tempest’s performance is not revealed as a number. However, according to Mr Imai, it has “PS4 capability at times”. The following image is an image of the sound when using Tempest on the PS5 shown by SIE. By the way, the image itself has no information, but the point is that it shows the image that “closer to a more precise area, one can realize more accurate processing without being deprived of the game’s resources”.

An image provided by SIE that shows the number of expressions and direction of sound on the PS5.

Since this hardware is built into hardware from the point of view of software development, it is naturally necessary to develop it specifically for the PS5. However, the point is, “We support development as much as possible and update the library every day” (Mr. Imai).

Currently, 3D audio by Tempest is for games and is not used in Dolby Atmos during video playback. I would like to ask you to consider this point.

The value of “retail” + 3D audio that can be seen with simple headphones

So what should I do to enjoy the results? Mr. Imai said, “If the title is released by the SIE,” Demon Spirits “or” Marvel’s Spider-Man: Miles Morales “are easy to understand and recommend.” Returns “also. Good use of 3D audio.”

“Retail” is a title that was just released on April 30, but it is true that Tempest’s 3D audio effect is amazing.

The PS5 exclusive software “Return” was released by SIE on 30 April. In addition to the fun of the game, it is also perceptible as software that boldly uses Tempest’s 3D audio and DualEzine’s hybrid response.
(C) 2021 Sony Interactive Entertainment Europe. Developed by Housemark Oy, published by Sony Interactive Entertainment Europe Limited. “Return” is a trademark of Sony Interactive Entertainment Europe. All rights reserved.

As a game, it is a so-called third-person shooter, but has a rogue aspect of “returning to the original and starting over”, so understanding the situation around is very important. When you listen to it with 3D audio, in addition to being amazing in terms of production, you can get a good sense of the enemy’s position and how the bullets are being fired, which turns a lot of fun.

In addition, the use of the dedicated controller “Dualikin” is also quite high. In addition to being able to perceive the movements of moving, jumping and shooting in detail by vibration, the feeling of shooting is also felt by the “rigidity” of the trigger. This is a task in which you can experience “PS5 parity” with high-speed loading of 3D audio and SSDs. Well, the game is quite tough, so I have to take care.

Now, there is a limit to enjoy 3D audio in works such as “Return”. It means “you need to use headphones”.

The reason is also to improve the accuracy of “stereoscopic reproduction” of 3D audio.

If you want to reproduce 3D audio, the shortcut is to place a large number of speakers and make sounds from them. However, this limits the number of people who can use it. Therefore, the current PS5 system was developed with the objective of making it as easy as possible for many people to use 3D audio by devising a special mechanism based on the use of headphones. Mr. Imai also said, “The purpose is to be able to enjoy 3D audio simply by connecting the headphones to the controller.”

The PS5 controller has a 3.5mm headphone jack, where you can connect headphones to the wire or via USB to listen to 3D audio. Here the headphones connected to the headphone jack do not need to be special.

The PS5 controller “DualSense” has a headphone jack. Also used for 3D audio

The PS5 does not support Bluetooth audio headphones. The reason for this is “the controller is also Bluetooth. To avoid delays caused by many Bluetooth communications” (Mr. Really). If you only want to connect headphones wirelessly, then you need to use a real “PULSE 3D wireless headset” like a USB dongle,

Like the PS4, the PS5 does not support Bluetooth headphones.

Genuine wireless headphones “PULSE 3D wireless headset” released at the same time as the PS5

Despite the sound quality characteristics, there is “no major difference” in 3D audio quality, whether you use a PULSE 3D wireless headset or connect the headphones to the controller (Imai).

The “PULSE 3D Wireless Headset” was formulated as a real option from the time of release, the reason being “for users who don’t know which to choose, I want you to experience it first” ( Sri Vakai) are in existence.

3D audio with simple headphones, taking advantage of “HRTF”

So what mechanism is required to get 3D audio with ordinary headphones?

What unfolds here is the idea of ​​”head-related transfer function (HRTF)”. If you are a reader of this magazine, who is familiar with 3D audio, then you must have heard about this area too.

People usually hear voices with two ears. Everyday sounds are also heard as “three-dimensional sounds”, but one part is that they are felt in the brain simply by changes in sounds transmitted through the head, shoulders, ears, etc. This feature of changing the sound through the head and ears is “HRTF”. By using HRFT and changing the frequency characteristics of the sound, the sound transmitted from the headphones makes it easier to feel as 3D audio.

The problem is that HRTFs vary from person to person. It seems that it mainly depends on the shape of the ear, but there are many variations.

Example of HRTF. On the right is Mark Cerny’s data, but the data varies greatly from person to person.

Therefore, SIE measures HRTF from several people, collects average data from them, divides them into five patterns, and installs them in PS5. The number of people sampled is about “hundreds”.

Photograph of HRTF measurement in SIE. The HRTF is measured by sitting in the center of a large number of speakers and measuring the sound heard with a microphone in the ear.

The HTRF on the PS5 can be switched from “Settings”, but at that point, you can choose the one that you think is the most suitable for you when actually listening to the sound.

HRTF can be switched from 5 presets in PST settings

Waqai:I think it’s important how the sound reaches people’s ears. It is true that people listen to different things, so now we have adapted effects from multiple profiles and prepared them as five presets.

Imai:I was wondering how to choose 5 types. Ultimately it depends on the characteristics of the ear. This time, “I chose 5 attributes as the delegate”. I asked him a lot and picked one with a high rating.

Of course, the settings will continue to improve. We are also considering the introduction of personalized customization.

In fact, Tempest is also responsible for processing including HRTF optimization in hardware. “(HRTF optimization) is still a heavy load, so it is important to reduce processing with dedicated hardware,” (Mr. Imai).

“3D audio” for TV speakers is currently under development

Enjoying 3D audio with simple headphones is an important point in reducing obstruction.

But if you really want to lower the bar, eventually being able to experience 3D audio with an ordinary TV speaker is desirable. SIE is also working here.

Imai:We are also developing a virtual speaker for TV. We want to provide an experience that is as similar as possible to headphones.

However, the problem is the cancellation of crosstalk (elimination of sound mixing). This part is particularly important for TV adaptation. TV speakers are not an ideal environment to enjoy 3D audio, but in the end, the point is how close you can get to the ideal experience.

Although nature is different, we will continue to support sound bars and such.

The important thing is “I want to distribute 3D audio to everyone.” Only for those who have a great audio set … touch points won’t stretch. We want to gradually expand this to TV, which also includes types.