Abstract
-
Mixed reality refers to the integration of virtual reality into the real-world environment. This digital content can be interacted with in real time.
-
The emergence of mixed reality technology has been made possible by the introduction of head-mounted displays, which are being utilized across multiple surgical specialties. In upper extremity surgery, mixed reality has widespread applications in trauma, corrective surgery, arthroplasty, arthroscopy, and oncology.
-
Preoperatively, mixed reality allows for complex 3D planning.
-
Intraoperatively, surgeons can access this 3D data in a sterile environment. While in its infant stages, mixed reality is likely to become a powerful tool for intraoperative guidance and navigation.
-
Mixed reality can change the paradigm of communication, as it allows the sharing of visual data from the surgeon’s perspective, enabling remote assistance and participation.
Introduction
Surgical innovation aims to create value by improving patient outcomes and safety, enhancing surgical techniques, maximizing surgeon efficiency, and minimizing costs. Mixed reality (MR) has emerged as a groundbreaking technology with widespread applications across multiple medical specialties. In 1994, Paul Milgram first described the concept of MR as existing between reality and virtual reality (VR) in a continuum (1). VR refers to complete immersion in a digital world. In augmented reality (AR), virtual objects are superimposed on the real world and anchored in space. In MR, one can interact with these virtual objects in real time.
The emergence of MR has been made possible in recent years through the development of head-mounted displays, such as the Microsoft HoloLens, which was introduced in 2017. These devices provide hands-free holographic information within the user’s visual field and have various sensors that allow the user to interact with virtual objects through voice commands, gaze, and hand gestures. Recently, academic interest in head-mounted displays has increased dramatically (2). This is reflected economically, as the global AR/VR market is projected to grow 350% from $2.7 billion to nearly $10 billion in the next 5 years (3).
These new technologies continue to be adopted across surgical specialties, namely neurosurgery, general surgery, and urology (2, 4, 5, 6). There is also a growing number of applications within orthopedic surgery (7, 8, 9). While the physical world is 3D, most data used by orthopedic surgeons are 2D, from notes and imaging on pages and screens to intraoperative fluoroscopy. MR superimposes 3D data on a 3D world, thereby closing that gap. Preoperatively, MR allows enhanced 3D interaction with CT reconstructions. Intraoperatively, MR can link a preoperative plan with the patient’s anatomy and can be used for navigation, intraoperative guidance, and visualization (10). This has been shown to increase precision, reduce radiation exposure, and/or decrease operative time in various procedure types such as vertebroplasty (11, 12, 13, 14), superior ramus K-wire placement (15, 16), dynamic hip screw placement (17, 18), resection of bony neoplasms (19, 20), and cup placement in total hip arthroplasty (21, 22). Moreover, MR allows access to patient information while maintaining a sterile environment. It also shares information from the surgeon’s point of view and can be used for intraoperative remote guidance, participation, and surgical assistance (23). Postoperatively, AR can better educate patients and integrate them more actively and effectively into the rehabilitation plan (24). Lastly, MR is emerging as a powerful hybrid tool in orthopedic training, combining haptic, visual, and audio technology to create an immersive training environment (25, 26, 27).
While various studies have focused on the role of MR in shoulder arthroplasty (8, 23, 28), many other potential uses in upper extremity surgery remain untapped. As this technology emerges from its infancy, it has the potential to become a powerful tool in increasing surgeon precision and reducing surgical error. While robotic surgery may increase costs, MR head-mounted displays have the potential to be a more cost-saving and space-efficient alternative. This review will focus on a variety of MR applications in upper extremity surgery.
Trauma
While orthopedic trauma surgery is meticulously tailored to each injury, MR offers intuitive feedback, enhancing the precision and efficiency of orthopedic implant placement. This has extensive applications across orthopedic injuries. Previous research has demonstrated the utility of MR in decreasing operative time, reducing radiation exposure, and improving accuracy in superior ramus K-wire placement and dynamic hip screw placement (15, 16, 17, 18). These benefits can also be utilized in upper extremity trauma. 3D mapping via CT imaging can guide diagnosis, treatment plan selection, and surgical fixation design in fractures of the scapula (29, 30, 31), distal humerus (32), proximal radius, and scaphoid (33, 34). MR allows for the visualization of this 3D imaging in 3D virtual space and has powerful applications in preoperative visualization and planning as well as intraoperative access and guidance. The capacity to better inform the surgeon regarding complicated fracture patterns and deliver an accurate execution of a preoperative plan is perhaps the most useful area of MR in upper extremity surgery.
In the case of a complex fracture of the proximal humerus, MR allows for not only preoperative visualization of the fracture pattern but also virtual manipulation of fracture fragments, which can help plan the reduction technique (Fig. 1). 3D implant models can be made digitally with a high level of accuracy, and MR can be utilized to plan in a way that most current planning software does not otherwise allow. These models can be viewed virtually and manipulated to plan for appropriate implant size and positioning (Fig. 1C). Intraoperatively, interactive holograms can be manually dragged and positioned to provide real-time surgical guidance. For example, MR can be used to set the height and version of the humeral stem and assess tuberosity reduction (Fig. 1E). MR can be equally useful in percutaneous fixation of proximal humerus fractures. Holograms allow for a better understanding of fracture lines and displacement, and allow for the proper trajectory of K-wire fixation (Fig. 2).
Proximal humerus fracture treated with reverse shoulder arthroplasty using mixed reality. (A) Initial injury anteroposterior (AP) radiograph, (B) 3D CT reconstruction of fracture,and (C) 3D planning to preoperatively plan fracture reduction and stem height. (D) Intraoperative holograms in the operating room, dragged and positioned in space. (E) Positioned holograms to assist in glenoid pin placement, stem placement, and fracture reduction.
Citation: EFORT Open Reviews 9, 11; 10.1530/EOR-24-0080
Proximal humerus fracture treated with open reduction internal fixation using mixed reality. (A and B) Intraoperative holograms for use in preoperative planning and closed reduction of a proximal humerus fracture. (C and D) MR can guide percutaneous pin trajectory.
Citation: EFORT Open Reviews 9, 11; 10.1530/EOR-24-0080
Managing unstable fracture dislocations about the elbow can be challenging. Recently, the internal joint stabilizer (IJS; Skeletal Dynamics), which allows early postoperative motion, has been introduced (35). This device relies on the transepicondylar axis of the distal humerus to obtain an early stable concentric ulnohumeral range of motion. The axis pin placement can be a challenging part of the case but is critical for the outcome. Further surgical treatment of the unstable elbow may rely on the transepicondylar axis for the placement of screws, suture anchors, or tunnels for ligament reconstruction. Intraoperatively, the use of MR as a hologram in the operating field may improve 3D-spatial awareness and increase accuracy and efficiency in drilling the transepicondylar axis. The hologram can be placed immediately adjacent to the distal humerus in the surgical field, displaying the correct trajectory (Fig. 3B). Bone stock is sometimes a limited resource, and obtaining an accurate start point without multiple attempts may be critical. MR can provide preoperative planning and intraoperative assistance for challenging wire placement, proving invaluable in case efficiency and bone preservation. Additionally, MR can be used to preoperatively determine radial head prosthesis size and positioning for radial head arthroplasty (Fig. 3A).
Internal joint stabilizer (IJS) placement to treat elbow instability using mixed reality. (A) 3D preoperative planning of radial head prosthesis size and positioning. (B) Intraoperative MR-assisted guidance of transepicondylar axis pin.
Citation: EFORT Open Reviews 9, 11; 10.1530/EOR-24-0080
Osteotomy
One of the most common deformities in the upper extremity is cubitus varus, most commonly as a result of a pediatric supracondylar humerus fracture malunion. Although primarily a cosmetic issue, some patients may experience long-term sequelae, including pain, tardy ulnar nerve palsy, and elbow instability. In cases where the deformity is clinically significant, a closing wedge humeral osteotomy proves to be an effective treatment option. Regardless of the osteotomy method, the rates of complication with this procedure are notably high (36), and postoperative loss of correction is observed in 80% of patients (37). Preoperative planning and accurate correction guidance are therefore crucial.
Extensive research has been conducted on lower limb deformity and osteotomy. Computer-assisted surgery (31) and patient-specific instrumentation (PSI) (38, 39) have both been shown to increase accuracy in proximal tibial osteotomy. However, these methods require costly technology and preoperative preparation. Once the instrument is printed, the angle of the template cannot be adjusted during surgery if necessary. Both methods also necessitate the use of a CT scan, posing a radiation risk to the pediatric population. MR offers a user-friendly alternative by employing virtual wedges that can be created ad hoc at any angle. These wedges can be generated preoperatively for planning or intraoperatively for guidance. The holographic wedge can be accurately placed within the bone at the center of rotation of angulation (CORA) without risk of collision with tissues or other instruments and without enlarging surgical exposure, thus lowering the infection risk. Multiple wedges can be created during surgery, and their sizes can be modified through intuitive gestures of ‘stretching’ and ‘shrinking.’
The holographic wedge is not merely a flat template. Due to its polyhedral shape, the lateral surface can aid as a gauge for the oscillating saw (Fig. 4A and B). Our previous research has demonstrated that MR guidance provides superior accuracy and precision compared to visual control and printed templates (40). Results show correction in both the AP and lateral planes. Such instrumentation can be applied in numerous clinical scenarios regardless of the bone or joint in question, including derotational correction and shaping bone graft for open distal radius osteotomy (Fig. 4C).
Proximal humerus fracture treated with open reduction internal fixation using mixed reality. (A and B) Intraoperative holograms for use in preoperative planning and closed reduction of a proximal humerus fracture. (C and D) MR can guide percutaneous pin trajectory.
Citation: EFORT Open Reviews 9, 11; 10.1530/EOR-24-0080
Another major advantage of MR in osteotomy is during 3D planning. Some deformities require correction in multiple planes. Finding the proper axis of correction of angulation and CORA sometimes involves challenging calculations. These can be particularly difficult in cases of rotational malalignment. MR simplifies these processes by immersing the user in a truly 3D space, rather than projecting 3D objects on a 2D screen. Bilateral comparisons or even mirror images with color-coded models can expedite the planning process. Holographic measurement instruments such as calipers, protractors, and angular wedges can be employed for calculations. All planning can be completed beforehand and easily reproduced in the operating theater.
Arthroplasty
Shoulder arthroplasty has its challenges, particularly with glenohumeral deformity and, more specifically, glenoid bone loss when preparing for reconstruction of degenerative shoulder arthritis and cuff tear arthropathy. Glenoid deformities are not always fully appreciated on 2D CT scans, which has led to the use of 3D reconstructions to better evaluate glenoid vault shape, depth, and containment of the glenoid bone loss (41).
Accuracy is critical for some of the most challenging glenoids with severe bone loss, and access to bony fixation is crucial for successful surgery. It has been shown that the surgical technique impacts the survivorship of glenoid implants. Surgically addressing an excessively retroverted glenoid by excessive corrective reaming or simply leaving the glenoid component in retroversion can lead to glenoid component migration, loosening, and implant failure (42, 43). Therefore, the goals of glenoid reconstruction include the following: maximizing backside contact of the implant, minimizing bone reaming, restoring version and inclination, preventing cage/peg perforation, maximizing fixation, and preventing bony impingement of implants.
Preoperative planning software has been developed to achieve these goals by using CT scan images to better manage the version, inclination, and overhang of implants intraoperatively. 3D preoperative planning software and the use of patient-specific bone models and transfer devices improved the positioning accuracy of pin placement to guide the placement of glenoid components over the use of handheld instruments alone (44). Several cadaveric studies have demonstrated that AR is highly effective in achieving precise glenoid pin entry points and trajectories (45, 46), as well as accurate overall glenoid component placement (47).
Further developments have led to the concept of ‘live’ navigation surgery, which is performed intraoperatively. These systems utilize a 3D CT scan that is segmented for use in the preoperative planning of implants. Intraoperatively, the scan and preoperative plan provide step-by-step guidance throughout the case, including a real-time view of reaming, drilling, screw placement, and implant rotation. One caveat with navigation is that it requires the surgeon to physically look away from the operative field at a sterile electronic tablet to follow the live information that navigation provides (Fig. 5A). MR allows the surgeon to set up their own field of view as they see fit for each individual case (Fig. 5B and C). The view can include a 3D model of the glenoid or humerus, MRI or CT scan images, preoperative planning software, and virtual touch screens of the navigation system. Not only does MR allow access to more information in the sterile field, but it also eliminates the need to create and sterilize 3D-printed models. The surgeon’s focus stays within the surgical wound with improved ergonomics resulting in enhanced visual and tactile cues during the procedure (Fig. 5B).
(A) Using traditional navigation techniques, the surgeon must look away from the surgical field while placing the central pin. (B) Head-mounted displays allow the surgeon to look directly into the surgical incision while simultaneously viewing the navigation software, a 3D model of the glenoid humerus, and fluoroscopy within MR. (C) These MR features are shown from the surgeon’s point of view.
Citation: EFORT Open Reviews 9, 11; 10.1530/EOR-24-0080
Arthroscopy
Arthroscopy relies on optimal portal placement for both instrumentation and visualization. MR can aid in a variety of shoulder and elbow arthroscopic procedures, providing real-time access to preoperative measurements in the sterile field and continuous monitoring of the spatial orientation of arthroscopic instrumentation. This can help obtain ideal portal trajectories relative to patient anatomy, as well as improve surgeon ergonomics and access to visual information and various holographic tools. In our experience, MR has been used in arthroscopic rotator cuff repair, anatomic arthroscopic glenoid reconstruction (AAGR), acromioclavicular (AC) injury, and arthroscopic elbow release to aid in surgeon accuracy and efficiency.
MR can be employed in arthroscopic rotator cuff surgery, particularly in cases of massive cuff repair requiring augmentation with a patch. Accessing the subacromial space may not be equally straightforward in all shoulders, despite the consistent placement of portals relative to surface landmarks (48). MR can be used to plan optimal portal placement in both the preoperative and intraoperative settings. A virtual scapula and proximal humerus model can be manipulated to identify the optimal arm position to obtain the maximum footprint for greater tuberosity coverage. It can also help to visualize the optimal trajectories for the viewing and anchor placement portals (Fig. 6).
MR in arthroscopic rotator cuff surgery. (A) The patch, represented by the red line, is inserted through the posterior portal. (B) The virtual humerus is rotated to identify the optimal arm position to obtain the maximal greater tuberosity footprint.
Citation: EFORT Open Reviews 9, 11; 10.1530/EOR-24-0080
Managing glenoid bone loss in recurrent shoulder instability presents a significant hurdle for the treating surgeon (49). Arthroscopic techniques to reconstruct the glenoid have gained popularity in recent years after yielding excellent clinical outcomes (50). AAGR is a surgical technique that uses distal tibia allograft to reconstruct the glenoid, thereby allowing preservation of the patient’s native coracoid and subscapularis tendon. A successful technique requires meticulous surgical planning and execution, namely bone loss estimation, arthroscopic portal placement, graft preparation, insertion, positioning, and fixation.
Precise portal placement is crucial for the procedure’s success. The medial portal must be placed parallel to the glenoid surface, avoiding the conjoint tendon. The patient’s virtual scapula (containing the glenoid bone loss) can be positioned in space overlying the patient’s shoulder to simulate the arm’s operative position. MR can be used to holographically draw the patient’s conjoint tendon, and the virtual scapula can be manipulated from neutral to adduction to determine the optimal position to displace the conjoint tendon medially and aid in portal placement (Fig. 7A). Furthermore, the glenoid bone loss needs to be refreshed and cut at a perpendicular angle to accommodate the distal tibia allograft. A virtual divider tool can aid in the visualization of the amount of bone to be resected. Additionally, the distal tibia allograft must be contoured to align with the glenoid bone defect, ensuring anatomic restoration of the articular surface (Fig. 7C). This alignment can be assessed intraoperatively using a hologram display of the glenoid fossa model. The bone graft is placed over the defect, and any irregularities in the graft are matched to the glenoid surface precisely.
MR use in arthroscopic glenoid reconstruction using distal tibia allograft. (A) The arm can be manipulated to obtain an optimal medial portal position, which is visualized relative to the conjoint tendon, displayed in red. (B) A virtual divider tool can aid in the visualization of the amount of bone to be resected to outline a precise 90-degree bone cut. (C) A holographic model of the glenoid fossa is displayed next to the real-time distal tibia allograft to match the marked bone graft shaping. (D) The allograft is cut and positioned on its inserter. Final adjustments can be made when comparing it to the native glenoid.
Citation: EFORT Open Reviews 9, 11; 10.1530/EOR-24-0080
MR can also be useful in the arthroscopic management of AC injury. It has been reported that accurate drilling through the coracoid occurs in only 62–84% of cases, leading to a 20–38% re-drilling rate despite the use of guiding instruments (51, 52). While navigation has been shown to be helpful, it is not widely available. In such instances, MR imaging may aid in several ways. It is not yet a substitute for navigation and cannot replace standard guided instrumentation. However, it may significantly improve anatomical spatial awareness. Patient models can be accurately placed over the patient's body with the help of bone rendering. Virtual guides can then be placed to mimic the accurate direction of drilling through the clavicle and coracoid (Fig. 8). Holographic instrumentation can also facilitate measurements of the appropriate distance and location of the clavicle. Furthermore, MR can project a holographic virtual screen. AC joint surgery requires a lot of technical equipment, including an arthroscopic tower and C-arm. Visual access to control both the image intensifier and arthroscopic screen may be cumbersome. Virtual screens for both can be manually placed by the surgeon in the most convenient space in the operating field and adjusted to the size and distance from the eyes (Fig. 9), thereby maximizing surgeon efficiency.
(A and B) MR holograms guiding proper arthroscopic drilling trajectory of the coracoclavicular tunnel.
Citation: EFORT Open Reviews 9, 11; 10.1530/EOR-24-0080
Virtual screens can be placed in any space desired by the surgeon and adjusted in size and distance, thereby providing immediate and convenient access to visual information during arthroscopic surgery and reducing the burden of surgical equipment overload.
Citation: EFORT Open Reviews 9, 11; 10.1530/EOR-24-0080
Arthroscopic elbow release is a technically demanding procedure. A major challenge is the safe entry into the joint among numerous vital neurovascular structures. Additionally, one may have to debride outside the joint when performing capsular release and osteophyte or heterotopic ossification (HO) resection, putting these neurovascular structures at risk. 3D planning has historically been projected on 2D screens in the operating room. MR offers the major advantage of projecting 3D holographic models in 3D space overlying the patient’s anatomy.
Establishing the initial portals and viewing field during elbow arthroscopy requires a “nearly blind” entry into the joint. This requires a high level of hand-eye coordination, spatial orientation, and surgical skill. While training with either sawbones (53) or 3D-printed models (54) can enhance performance for novel elbow arthroscopists, MR offers particular advantages and can be utilized in real-time intraoperatively. Through MR, holograms can be overlaid onto the patient’s anatomy, offering anatomical guidance to aid in a better understanding of portal locations and approaches. Previous research has demonstrated that 3D models and image-enhanced navigation can improve precision during elbow arthroscopy (55, 56), namely with osteophyte and HO removal. MR adds value by allowing constant monitoring of the position and location of the deformity to be removed. Unlike traditional computer screens and printed models, MR allows the surgeon to manipulate the holographic model’s position in space with hand gestures without turning his or her head toward the back screen. The holographic model can be placed either inside the patient’s elbow or just in front of the arthroscopic screen (Fig. 10A). This capability proves particularly beneficial in identifying hidden loose bodies (Fig. 10B). MR also provides other tools as well, such as a holographic pointer, chisel, or protractor. The latter may be used for precise range of motion measurement before and after arthrolysis (Fig. 10D and E).
Arthroscopic elbow release using MR. (A) Elbow hologram overlayed over the patient’s anatomy intraoperatively to aid in elbow arthroscopic portal placement. (B and C) This can aid in the removal of hidden loose bodies. MR protractor tool to quantify passive elbow range of motion (D) preoperatively and (E) postoperatively.
Citation: EFORT Open Reviews 9, 11; 10.1530/EOR-24-0080
Oncology
Orthopedic oncology is a specialized branch of orthopedic surgery focused on the diagnosis and treatment of bone and soft tissue tumors, both benign and malignant. It involves a multidisciplinary approach, with collaboration among oncologists, radiologists, pathologists, and other specialists to provide comprehensive care to patients. MR can be extremely valuable in orthopedic oncology due to the complexity of cases and the distortion of the anatomy.
One of the primary challenges in orthopedic oncology is the complexity of tumor resection and reconstruction. Tumors located in critical anatomical areas such as the pelvis or spine pose significant surgical challenges due to their proximity to vital structures. This is also true in the upper extremity, where complicated neurovascular structures are often displaced by certain tumors. Balancing the goals of achieving oncological clearance while preserving function and quality of life for the patient requires careful planning and innovative surgical techniques.
MR can allow for more thorough 3D planning to assist the surgeon intraoperatively. Both MRI and CT scans can be used to create 3D models of both the tumor and nearby vital structures. For example, Fig. 11 demonstrates the MRI and 3D model of a forearm lipoma within the supinator muscle adjacent to the posterior interosseous nerve (PIN). The 3D model can be used for preoperative planning and moved to be holographically adjacent to the surgical field intraoperatively. The model includes the tumor, supinator muscle, and PIN and can be referenced intraoperatively for safer and more efficient surgical dissection.
(A) Preoperative MRI demonstrating a well-circumscribed forearm mass within the supinator muscle. (B) MRI was used for 3D preoperative modeling to identify the course of PIN relative to the mass. (C) Intraoperative hologram demonstrating MR assistance. (D) The 3D model is positioned within the surgical field, highlighting the mass (green), supinator (yellow), and PIN (purple).
Citation: EFORT Open Reviews 9, 11; 10.1530/EOR-24-0080
Communication and remote support
Communication is pivotal within surgical teams, and MR introduces a novel dimension to intraoperative communication. By sharing the same visual information through MR headsets, team members can see what the lead surgeon is seeing in both the real-time physical environment and the virtual environment, improving communication and decision-making.
Visual information does not necessarily have to be shared with someone physically in the operating room and instead can be broadcast to surgical assistants participating from a remote location. The lead surgeon’s visual information can be transmitted to co-surgeons, device representatives, or other experts elsewhere, and these consultants can provide their expertise as if they were physically present. Additionally, these consultants can interact not only verbally with the surgeon but can also annotate the surgeon’s visual field with cues or other messages. The visual field of surgical technicians or other assistants wearing HMDs can be interacted with to highlight instruments for the next surgical step, improving operative efficiency. This is particularly beneficial in rural or other geographic areas with limited access to specialized care, democratizing medical expertise. For example, this author used MR from the United States to assist another author in Brazil intraoperatively (57). Gregory et al. reported on surgeon experience in an international case series of 13 surgeries using MR headset technology and found that surgeons expressed high satisfaction levels (100%) and resounding intent (94.1%) to integrate MR into future clinical practice (58). Notably, all surgeons found the remote assist functionality and enhanced intraoperative communication to be useful for future practice (58).
The Metaverse, a collective VR shared space, is emerging as a powerful tool in orthopedic surgery. In contrast to MR, the VR space involves complete immersion in a digital world. The Metaverse enables the creation of a fully immersive surgical simulation environment where surgeons can practice and refine their skills without risk to patients. These simulations can mimic a wide range of scenarios, from routine procedures to rare and complex cases, offering a hands-on experience that traditional training methods cannot. In addition, these simulations can be accessed by surgeons in geographically disparate locations, allowing simultaneous training among surgeons across the globe and introducing a novel degree of collaboration not possible in the physical world.
Limitations
Despite the promising potential of MR in upper extremity surgery, several limitations exist. A recent review compared 3D planning, navigation, PSI, and MR in shoulder arthroplasty and found that each technology comes with unique advantages and disadvantages (59). The implementation of MR comes with increased costs, including the expense of acquiring HMDs, specialized software, and the necessary hardware infrastructure. These expenses can be prohibitive for some healthcare institutions, particularly smaller or resource-limited facilities. Additionally, ongoing maintenance and updates are required to keep the technology functioning optimally, further adding to costs. To date, no formal cost analyses examining MR in orthopedic surgery have been published, and the exact financial implications remain unknown. Future research should address this gap by comparing the costs and benefits of MR against other technological developments such as PSI, computer navigation, and robotics in both upper extremity surgery and orthopedic surgery as a whole.
The complexity of MR systems introduces another layer of challenges. Advances in MR technology necessitate education and training, which can be time-consuming and research-intensive. The learning curve associated with mastering MR tools can initially slow down surgical procedures as teams adapt to new technology. Additionally, incorporating MR into surgical workflows requires added time for surgical planning. Preoperative planning involves creating detailed virtual models, which can be time-consuming. Moreover, the reliance on digital data and technology introduces potential risks, such as technical malfunctions or software glitches. These drawbacks should be weighed against the potential advantages of MR, including more precise surgical execution, precision, and ultimately the potential for improved patient outcomes.
Conclusion
MR technology is at the forefront of a paradigm shift in orthopedic surgery, offering unprecedented opportunities for enhanced surgical planning and execution. The emergence of head-mounted displays allows the integration of a digital environment into the physical world, providing a platform for more intuitive, precise, and collaborative healthcare delivery. In upper extremity surgery, MR has widespread applications in trauma, arthroplasty, arthroscopy, and corrective surgery, both preoperatively and intraoperatively. While in its nascency, this technology promises to revolutionize the medical field, ultimately leading to better patient care and outcomes.
ICMJE Conflict of Interest Statement
The authors declare that the study was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. ST, IW, and JE and/or their direct family members have some financial interest in a mixed reality company (RSQ Holo). PL is an Associate Editor for EFORT Open Reviews, and was not involved with the peer review of this manuscript.
Funding Statement
This work did not receive any specific grant from any funding agency in the public, commercial, or not-for-profit sector.
References
- 1↑
Milgram P, Takemura H, Utsumi A, & Kishino F. Augmented reality: a class of displays on the reality-virtuality continuum. In SPIE Proceedings, pp 282–292. Ed Das H, 1995. (https://doi.org/10.1117/12.197321)
- 2↑
Rahman R, Wood ME, Qian L, Price CL, Johnson AA, & Osgood GM. Head-mounted display use in surgery: a systematic review. Surgical Innovation 2020 27 88–100. (https://doi.org/10.1177/1553350619871787)
- 3↑
Balasubramanian S. Augmented & virtual reality in healthcare is poised to become a nearly $10 billion dollar market. Forbes. Available at: https://www.forbes.com/sites/saibala/2022/07/23/augmented--virtual-reality-in-healthcare-is-poised-to-become-a-nearly-10-billion-dollar-market/?sh=2ad8fe9d2171.Published July 23, 2022.
- 4↑
Meola A, Cutolo F, Carbone M, Cagnazzo F, Ferrari M, & Ferrari V. Augmented reality in neurosurgery: a systematic review. Neurosurgical Review 2017 40 537–548. (https://doi.org/10.1007/s10143-016-0732-9)
- 5↑
Hallet J, Soler L, Diana M, Mutter D, Baumert TF, Habersetzer F, Marescaux J, & Pessaux P. Trans-thoracic minimally invasive liver resection guided by augmented reality. Journal of the American College of Surgeons 2015 220 e55–e60. (https://doi.org/10.1016/j.jamcollsurg.2014.12.053)
- 6↑
Ntourakis D, Memeo R, Soler L, Marescaux J, Mutter D, & Pessaux P. Augmented reality guidance for the resection of missing colorectal liver metastases: an initial experience. World Journal of Surgery 2016 40 419–426. (https://doi.org/10.1007/s00268-015-3229-8)
- 7↑
Jud L, Fotouhi J, Andronic O, Aichmair A, Osgood G, Navab N, & Farshad M. Applicability of augmented reality in orthopedic surgery - a systematic review. BMC Musculoskeletal Disorders 2020 21 103. (https://doi.org/10.1186/s12891-020-3110-2)
- 8↑
Daher M, Ghanimeh J, Otayek J, Ghoul A, Bizdikian AJ, & El Abiad R. Augmented reality and shoulder replacement: a state-of-the-art review article. JSES Reviews, Reports, and Techniques 2023 3 274–278. (https://doi.org/10.1016/j.xrrt.2023.01.008)
- 9↑
Verhey JT, Haglin JM, Verhey EM, & Hartigan DE. Virtual, augmented, and mixed reality applications in orthopedic surgery. International Journal of Medical Robotics + Computer Assisted Surgery 2020 16 e2067. (https://doi.org/10.1002/rcs.2067)
- 10↑
Lee SC, Fuerst B, Tateno K, Johnson A, Fotouhi J, Osgood G, Tombari F, & Navab N. Multi-modal imaging, model-based tracking, and mixed reality visualisation for orthopaedic surgery. Healthcare Technology Letters 2017 4 168–173. (https://doi.org/10.1049/htl.2017.0066)
- 11↑
Wu JR, Wang ML, Liu KC, Hu MH, & Lee PY. Real-time advanced spinal surgery via visible patient model and augmented reality system. Computer Methods and Programs in Biomedicine 2014 113 869–881. (https://doi.org/10.1016/j.cmpb.2013.12.021)
- 12↑
Abe Y, Sato S, Kato K, Hyakumachi T, Yanagibashi Y, Ito M, & Abumi K. A novel 3D guidance system using augmented reality for percutaneous vertebroplasty: technical note. Journal of Neurosurgery: Spine 2013 19 492–501. (https://doi.org/10.3171/2013.7.SPINE12917)
- 13↑
Navab N, Heining SM, & Traub J. Camera augmented mobile C-arm (CAMC): calibration, accuracy study, and clinical applications. IEEE Transactions on Medical Imaging 2010 29 1412–1423. (https://doi.org/10.1109/TMI.2009.2021947)
- 14↑
U-Thainual P, Fritz J, Moonjaita C, Ungi T, Flammang A, Carrino JA, Fichtinger G, & Iordachita I. MR image overlay guidance: system evaluation for preclinical use. International Journal of Computer Assisted Radiology and Surgery 2013 8 365–378. (https://doi.org/10.1007/s11548-012-0788-0)
- 15↑
Befrui N, Fischer M, Fuerst B, Lee SC, Fotouhi J, Weidert S, Johnson A, Euler E, Osgood G, Navab N, et al.3D augmented reality visualization for navigated osteosynthesis of pelvic fractures. Der Unfallchirurg 2018 121 264–270. (https://doi.org/10.1007/s00113-018-0466-y)
- 16↑
Fischer M, Fuerst B, Lee SC, Fotouhi J, Habert S, Weidert S, Euler E, Osgood G, & Navab N. Preclinical usability study of multiple augmented reality concepts for K-wire placement. International Journal of Computer Assisted Radiology and Surgery 2016 11 1007–1014. (https://doi.org/10.1007/s11548-016-1363-x)
- 17↑
van Duren BH, Sugand K, Wescott R, Carrington R, & Hart A. Augmented reality fluoroscopy simulation of the guide-wire insertion in DHS surgery: a proof of concept study. Medical Engineering and Physics 2018 55 52–59. (https://doi.org/10.1016/j.medengphy.2018.02.007)
- 18↑
Hiranaka T, Fujishiro T, Hida Y, Shibata Y, Tsubosaka M, Nakanishi Y, Okimura K, & Uemoto H. Augmented reality: the use of the PicoLinker smart glasses improves wire insertion under fluoroscopy. World Journal of Orthopedics 2017 8 891–894. (https://doi.org/10.5312/wjo.v8.i12.891)
- 19↑
Cho HS, Park MS, Gupta S, Han I, Kim HS, Choi H, & Hong J. Can augmented reality be helpful in pelvic bone cancer surgery? An in vitro study. Clinical Orthopaedics and Related Research 2018 476 1719–1725. (https://doi.org/10.1007/s11999.0000000000000233)
- 20↑
Cho HS, Park YK, Gupta S, Yoon C, Han I, Kim HS, Choi H, & Hong J. Augmented reality in bone tumour resection: an experimental study. Bone and Joint Research 2017 6 137–143. (https://doi.org/10.1302/2046-3758.63.BJR-2016-0289.R1)
- 21↑
Ogawa H, Hasegawa S, Tsukada S, & Matsubara M. A pilot study of augmented reality technology applied to the acetabular cup placement during total hip arthroplasty. Journal of Arthroplasty 2018 33 1833–1837. (https://doi.org/10.1016/j.arth.2018.01.067)
- 22↑
Fotouhi J, Alexander CP, Unberath M, Taylor G, Lee SC, Fuerst B, Johnson A, Osgood G, Taylor RH, Khanuja H, et al.Plan in 2-D, execute in 3-D: an augmented reality solution for cup placement in total hip arthroplasty. Journal of Medical Imaging 2018 5 021205. (https://doi.org/10.1117/1.JMI.5.2.021205)
- 23↑
Gregory TM, Gregory J, Sledge J, Allard R, & Mir O. Surgery guided by mixed reality: presentation of a proof of concept. Acta Orthopaedica 2018 89 480–483. (https://doi.org/10.1080/17453674.2018.1506974)
- 24↑
Yeo SM, Lim JY, Do JG, Lim JY, In Lee J, & Hwang JH. Effectiveness of interactive augmented reality-based telerehabilitation in patients with adhesive capsulitis: protocol for a multi-center randomized controlled trial. BMC Musculoskeletal Disorders 2021 22 386. (https://doi.org/10.1186/s12891-021-04261-1)
- 25↑
Lohre R, Bois AJ, Pollock JW, Lapner P, McIlquham K, Athwal GS, & Goel DP. Effectiveness of immersive virtual reality on orthopedic surgical skills and knowledge acquisition among senior surgical residents: a randomized clinical trial. JAMA Network Open 2020 3 e2031217. (https://doi.org/10.1001/jamanetworkopen.2020.31217)
- 26↑
Stefan P, Habert S, Winkler A, Lazarovici M, Fürmetz J, Eck U, & Navab N. A radiation-free mixed-reality training environment and assessment concept for C-arm-based surgery. International Journal of Computer Assisted Radiology and Surgery 2018 13 1335–1344. (https://doi.org/10.1007/s11548-018-1807-6)
- 27↑
Condino S, Turini G, Parchi PD, Viglialoro RM, Piolanti N, Gesi M, Ferrari M, & Ferrari V. How to build a patient-specific hybrid simulator for orthopaedic open surgery: benefits and limits of mixed-reality using the Microsoft HoloLens. Journal of Healthcare Engineering 2018 2018 5435097. (https://doi.org/10.1155/2018/5435097)
- 28↑
Kriechling P, Roner S, Liebmann F, Casari F, Fürnstahl P, & Wieser K. Augmented reality for base plate component placement in reverse total shoulder arthroplasty: a feasibility study. Archives of Orthopaedic and Trauma Surgery 2021 141 1447–1453. (https://doi.org/10.1007/s00402-020-03542-z)
- 29↑
Yimam HM, Dey R, Rachuene PA, Kauta NJ, Roche SJL, & Sivarasu S. Identification of recurring scapular fracture patterns using 3-dimensional computerized fracture mapping. Journal of Shoulder and Elbow Surgery 2022 31 571–579. (https://doi.org/10.1016/j.jse.2021.08.036)
- 30↑
Berritto D, Pinto A, Russo A, Urraro F, Laporta A, Belfiore MP, & Grassi R. Scapular fractures: a common diagnostic pitfall. Acta Bio-Medica 2018 89 102–110. (https://doi.org/10.23750/abm.v89i1-S.7014)
- 31↑
Cole PA, Schroder LK, Brahme IS, Thomas CN, Kuhn L, Zaehringer E, & Petersik A. Three-dimensional mapping of scapular body, neck, and glenoid fractures. Journal of Orthopaedic Trauma 2024 38 e48–e54. (https://doi.org/10.1097/BOT.0000000000002734)
- 32↑
Wang C, Zhu Y, Long H, Lin Z, Zhao R, Sun B, Zhao S, & Cheng L. Three-dimensional mapping of distal humerus fracture. Journal of Orthopaedic Surgery and Research 2021 16 545. (https://doi.org/10.1186/s13018-021-02691-0)
- 33↑
Ten Berg PWL, Dobbe JGG, & Streekstra GJ. Short report letter: three-dimensional printed anatomical models in scaphoid surgery. Journal of Hand Surgery 2018 43 101–102. (https://doi.org/10.1177/1753193417737935)
- 34↑
Ten Berg PWL, & Strackee SD. Three-dimensional planning of scaphoid malunion and nonunion correction. In Scaphoid Fractures: Evidence-Based Management, pp 221–228. Elsevier, 2018. (https://doi.org/10.1016/B978-0-323-48564-7.00025-3)
- 35↑
Heifner JJ, Chambers LR, Halpern AL, & Mercer DM. The internal joint stabilizer of the elbow: a systematic review of the clinical and biomechanical evidence. Journal of Hand Surgery Global Online 2024 6 62–67. (https://doi.org/10.1016/j.jhsg.2023.09.004)
- 36↑
Raney EM, Thielen Z, Gregory S, & Sobralske M. Complications of supracondylar osteotomies for cubitus varus. Journal of Pediatric Orthopedics 2012 32 232–240. (https://doi.org/10.1097/BPO.0b013e3182471d3f)
- 37↑
You C, Zhou Y, & Han J. Loss of correction in cubitus varus deformity after osteotomy. Medicine 2021 100 e27848. (https://doi.org/10.1097/MD.0000000000027848)
- 38↑
Zaffagnini S, Dal Fabbro G, Lucidi GA, Agostinone P, Belvedere C, Leardini A, & Grassi A. Personalised opening wedge high tibial osteotomy with patient-specific plates and instrumentation accurately controls coronal correction and posterior slope: results from a prospective first case series. Knee 2023 44 89–99. (https://doi.org/10.1016/j.knee.2023.07.011)
- 39↑
Aman ZS, DePhillipo NN, Peebles LA, Familiari F, LaPrade RF, & Dekker TJ. Improved accuracy of coronal alignment can be attained using 3D-printed patient-specific instrumentation for knee osteotomies: a systematic review of Level III and IV studies. Arthroscopy 2022 38 2741–2758. (https://doi.org/10.1016/j.arthro.2022.02.023)
- 40↑
Lubiatowski P, Stupnicki S, Nizinski J, Tuczynski P, Roszak K, Lubiatowski B, & Green A. Mixed reality guidance in humeral osteotomy provides superior precision and accuracy: validation and comparative study. Abstract presented at: ICSES 2023; September 5, 2023; Rome.
- 41↑
Scalise JJ, Codsi MJ, Bryan J, Brems JJ, & Iannotti JP. The influence of three-dimensional computed tomography images of the shoulder in preoperative planning for total shoulder arthroplasty. Journal of Bone and Joint Surgery 2008 90 2438–2445. (https://doi.org/10.2106/JBJS.G.01341)
- 42↑
Shapiro TA, McGarry MH, Gupta R, Lee YS, & Lee TQ. Biomechanical effects of glenoid retroversion in total shoulder arthroplasty. Journal of Shoulder and Elbow Surgery 2007 16 (3 Supplement) S90–S95. (https://doi.org/10.1016/j.jse.2006.07.010)
- 43↑
Nyffeler RW, Sheikh R, Atkinson TS, Jacob HAC, Favre P, & Gerber C. Effects of glenoid component version on humeral head displacement and joint reaction forces: an experimental study. Journal of Shoulder and Elbow Surgery 2006 15 625–629. (https://doi.org/10.1016/j.jse.2005.09.016)
- 44↑
Iannotti J, Baker J, Rodriguez E, Brems J, Ricchetti E, Mesiha M, & Bryan J. Three-dimensional preoperative planning software and a novel information transfer technology improve glenoid component positioning. Journal of Bone and Joint Surgery 2014 96 e71. (https://doi.org/10.2106/JBJS.L.01346)
- 45↑
Kriechling P, Loucas R, Loucas M, Casari F, Fürnstahl P, & Wieser K. Augmented reality through head-mounted display for navigation of baseplate component placement in reverse total shoulder arthroplasty: a cadaveric study. Archives of Orthopaedic and Trauma Surgery 2023 143 169–175. (https://doi.org/10.1007/s00402-021-04025-5)
- 46↑
Sanchez-Sotelo J, Berhouet J, Chaoui J, Freehill MT, Collin P, Warner J, Walch G, & Athwal GS. Validation of mixed-reality surgical navigation for glenoid axis pin placement in shoulder arthroplasty using a cadaveric model. Journal of Shoulder and Elbow Surgery 2024 33 1177–1184. (https://doi.org/10.1016/j.jse.2023.09.027)
- 47↑
Rojas JT, Jost B, Zipeto C, Budassi P, & Zumstein MA. Glenoid component placement in reverse shoulder arthroplasty assisted with augmented reality through a head-mounted display leads to low deviation between planned and postoperative parameters. Journal of Shoulder and Elbow Surgery 2023 32 e587–e596. (https://doi.org/10.1016/j.jse.2023.05.002)
- 48↑
Hoessly M, Bouaicha S, Jentzsch T, & Meyer DC. Angle of approach to the superior rotator cuff of arthroscopic instruments depends on the acromial morphology: an experimental study in 3D printed human shoulders. Journal of Orthopaedic Surgery and Research 2019 14 435. (https://doi.org/10.1186/s13018-019-1486-1)
- 49↑
Lukenchuk J, Thangarajah T, More K, Wong I, & Lo IKY. Arthroscopic anterior glenoid reconstruction using a distal tibial allograft positioned with an intra-articular guide and secured with double-button fixation. Arthroscopy Techniques 2022 11 e1053–e1057. (https://doi.org/10.1016/j.eats.2022.02.010)
- 50↑
McNeil D, Provencher M, & Wong IH. Arthroscopic anatomic glenoid reconstruction demonstrates its safety with short-term to medium-term results for anteroinferior shoulder instability: a systematic review. Journal of ISAKOS 2020 5 37–47. (https://doi.org/10.1136/jisakos-2019-000388)
- 51↑
Hoffmann M, Hartel M, Schroeder M, Reinsch O, Spiro AS, Ruecker AH, Grossterlinden L, Briem D, Rueger JM, & Petersen JP. Electromagnetic navigation provides high accuracy for transcoracoid-transclavicular drilling. Knee Surgery, Sports Traumatology, Arthroscopy 2014 22 2237–2242. (https://doi.org/10.1007/s00167-013-2607-8)
- 52↑
Theopold J, Weihs K, Löffler S, Marquass B, von Dercks N, Josten C, & Hepp P. Image-free navigated coracoclavicular drilling for the repair of acromioclavicular joint dislocation: a cadaver study. Archives of Orthopaedic and Trauma Surgery 2015 135 1077–1082. (https://doi.org/10.1007/s00402-015-2243-5)
- 53↑
Luceri F, Arrigoni P, Barco R, Cucchi D, Raj N, Frassoni S, & Randelli PS. Does Sawbone-based arthroscopy module (SBAM) can help elbow surgeons? Indian Journal of Orthopaedics 2021 55(Supplement 1) 182–188. (https://doi.org/10.1007/s43465-020-00133-3)
- 54↑
Trębacz P, Frymus J, Barteczko A, Pawlik M, Kurkowska A, & Czopowicz M. Accuracy of instrument portal placement using a custom-made 3d-printed aiming device versus free hand technique in canine elbow arthroscopy. Animals 2023 13 3592. (https://doi.org/10.3390/ani13233592)
- 55↑
Shigi A, Oka K, Tanaka H, Shiode R, & Murase T. Utility of a 3-dimensionally printed color-coded bone model to visualize impinging osteophytes for arthroscopic débridement arthroplasty in elbow osteoarthritis. Journal of Shoulder and Elbow Surgery 2021 30 1152–1158. (https://doi.org/10.1016/j.jse.2020.12.002)
- 56↑
Shiode R, Oka K, Shigi A, Miyamura S, Tanaka H, Mae T, & Murase T. Arthroscopic debridement of elbow osteoarthritis using CT-based computer-aided navigation systems is accurate. Arthroscopy, Sports Medicine, and Rehabilitation 2021 3 e1687–e1696. (https://doi.org/10.1016/j.asmr.2021.07.025)
- 57↑
Guillot C. Collaboration and preparation: what mixed reality surgery affords the operating room. HealthTech. Published online March 22, 2021. Available at: https://healthtechmagazine.net/article/2021/03/collaboration-and-preparation-what-mixed-reality-surgery-affords-operating-room.Accessed on 24 April 2024.
- 58↑
Gregory T, Gregory J, Dacheux C, & Hurst SA. Surgeon experience of mixed reality headset technology during the COVID-19 pandemic: a multicenter international case series in orthopedic surgery. BMJ Surgery, Interventions, and Health Technologies 2022 4 e000127. (https://doi.org/10.1136/bmjsit-2021-000127)
- 59↑
Can Kolac U, Paksoy A, & Akgün D. Three-dimensional planning, navigation, patient-specific instrumentation and mixed reality in shoulder arthroplasty: a digital orthopedic renaissance. EFORT Open Reviews 2024 9 517–527. (https://doi.org/10.1530/EOR-23-0200)
