Museum Trailer
WPI PAX East 2022 Booth
The Jazz VR museum was one of the lucky projects to be selected and presented at WPI’s PAX East 2022 Booth!
Jazz History Database (JHDB)
The Jazz History Database is an organization at Worcester Polytechnic Institute (WPI) that stores jazz resources in its archives, including live recordings. The JHDB in collaboration with the Interactive Media and Game Development (IMGD) Department set out to build an interactive 3D environment in which a player can listen to jazz songs and interviews on a record player. We would build this environment for virtual reality (VR), where the player puts on a VR Oculus headset in which they would see and interact with the artifacts in the virtual museum.
Technology
We chose to use the Oculus Quest 2. Aside from having one onhand it is also one of the best plug and play VR headsets currently on the market and makes VR easily accessible. The Oculus can also be plugged into a PC using USB3 which allows for quick testing and a smoother development process.
Unreal Engine 5 was beta released in May of 2021 and it was an ambitious undertaking for this project.
Plugins:
Oculus OpenXR: OpenXR aims to ease VR development across multiple headsets by standardizing the inputs across devices. This project takes advantage of this system but we have yet to test it on headsets other than the Quest 2.
GPU lightmass: Stationary and static lights in the scene can take advantage of the lightmass system by precomputing lighting data and storing the information in lightmap textures. This process usually takes a while to compute; however, this plugin enables the process to be done on the GPU and uses DirectX 12 to speed up the process.
Design
Our initial concept began with brainstorming different spaces the player could visit with each space focusing on a different element of the JHDB. For this project we scoped to focus on one space in particular, the Listening Space, as a proof of concept.
Designing for VR
While we wanted players to browse and play music, using menus in VR can be quite underwhelming or even frustrating. Our solution was to design the environment to be used as an interface and take full advantage of the VR capabilities. While on a website a user may scroll through a list of songs and click on one to select it, in our Listening Space the user browses records on a shelf and can physically pick up a record of their choosing. Additionally the distance of objects from the player also needed to be considered. As This experience was designed to keep the player in a sitting position, the assets in the surrounding environment needed to remain within reach.
Iterations
VR Interaction
Utilizing the Oculus OpenXR and OpenXR plugins to do a lot of the heavy lifting inside the Unreal Engine VR Template, specifically the VRPawn and the GrabComponent Blueprints. The GrabComponent can be attached to any simple objects such as the vinyl record and enables the object to be picked up and thrown by the motion controllers. Additional behavior can then be added to the object for more complex interactions. For example, to create the vinyl records the GrabComponent was attached to a static mesh and a song variable was added so that a specific song could be attached to the object.
Vinyl Folders
The vinyl folder was added to organize and better manage the growing list of songs/records in the space. This blueprint consists of a text label, trigger volume, and placeholder. In addition the VinylFolder requires a reference to the VinylRecord blueprint and for a default song to be selected. The folder can then be placed within the space and on start the blueprint will initialize a vinyl in the location of the placeholder and assign it the selected song. The text label on the side of the folder will always display the name of the song of the last vinyl that it was holding. This means that if a new vinyl is placed into an empty folder the name will update to the correct song.
Skeletal Mesh Interaction
Working with a skeletal model can be really nice as like the Oculus hand models you have access to each bone individually and you can get a lot of complex behavior out of them. For this model the record plate was given a spinning animation while the record head is grabbable. The record player inherits from the GrabComponent but uses a custom grab type to avoid being picked up. Rather it uses the GrabComponent to get access to the MotionController. This way when the player tries to grab the record player the Blueprint sends the MotionController position to the RecordPlayer_AnimBlueprint so that the record head bone can point towards it.
When the vinyl is secured on the plate it will not initially play music, instead to get the vinyl to play the record head must be set in the proper position. Using the object’s forward vector and the record head bone forward vector the cross product can be used to find the range in which the head is over the plate. The same process must also be done with the up and right vectors to make sure the height and quadrant respectively are also within the desired range. A fun trick is to then use the possible forward vector cross product range to select the starting time of the song. As a result the record player will function very similarly to a real record player where the location of the head on the outside of the track will be the start of the song and the inside will be the end of the song.
VR Hand Animation
The Oculus hands are rigged and can be used to make multiple poses using Unreals animation sequence. Combined with a 1D blendspace you can interpolate between the poses with a 0-1 value. For our purpose this creates very satisfying feedback between the amount the player pulls on the grip of the oculus controller and the amount that the hand model closes. This simple system could be expanded into a 3D blendspace with multiple inputs for a variety of poses and in future iterations could make for some unique interactions and experiences.
