return to Home page




Y. Zhao (Cornell Tech & MSR), E. CutrellC. HolzM. Ringel MorrisE.Ofek and A. Wilson

SeeingVR: A Set of Tools to Make Virtual Reality More Accessible to People with Low Vision

CHI 2019, to appear.

Current virtual reality applications do not support people who have low vision, i.e., vision loss that falls short of complete blindness but is not correctable by glasses. We present SeeingVR, a set of 14 tools that enhance a VR application for people with low vision by providing visual and audio augmentations. A user can select, adjust, and combine different tools based on their preferences. Nine of our tools modify an existing VR application post hoc via a plugin without developer effort. The rest require simple inputs from developers using a Unity toolkit we created that allows integrating all 14 of our low vision support tools during development. Our evaluation with 11 participants with low vision showed that SeeingVR enabled users to better enjoy VR and complete tasks more quickly and accurately. Developers also found our Unity toolkit easy and convenient to use.

Paper, Video, Microsoft Research Blog, Engadget RoadToVR


J. Lee (Kaist & MSR), M. Sinclair, M. Gonzalez-Franko, E. Ofek and C. Holz

TORC: A Virtual Reality Controller for In-Hand High-Dexterity Finger Interaction

CHI 2019, to appear.

Paper, Video

Recent hand-held controllers have explored a variety of haptic feedback sensations for users in virtual reality by producing both kinesthetic and cutaneous feedback from virtual objects. These controllers are grounded to the user's hand and can only manipulate objects through arm and wrist motions, not using the dexterity of their fingers as they would in real life. In this paper, we present TORC, a rigid haptic controller that renders virtual object characteristics and behaviors such as texture and compliance. Users hold and squeeze TORC using their thumb and two fingers and interact with virtual objects by sliding their thumb on TORC's trackpad. During the interaction, vibrotactile motors produce sensations to each finger that represent the haptic feel of squeezing, shearing or turning an object. Our evaluation showed that using TORC, participants could manipulate virtual objects more precisely (e.g., position and rotate objects in 3D) than when using a conventional VR controller.


P. Abtahi (Stanford & MSR), M. Gonzalez-Franco, E. Ofek, A. Steed (UCL)

CHI 2019, to appear.

I’m a Giant: Walking in Large Virtual Environments at High Speed Gains

Advances in tracking technology and wireless headsets enable walking as a means of locomotion in Virtual Reality. When exploring virtual environments larger than room-scale, it is often desirable to increase users’ perceived walking speed, for which we investigate three methods. (1) Ground-Level Scaling increases users’ avatar size, allowing them to walk farther. (2) Eye-Level Scaling enables users to walk through a World in Miniature, while maintaining a street-level view. (3) Seven-League Boots amplifies users’ movements along their walking path. We conduct a study comparing these methods and find that users feel most embodied using Ground-Level Scaling and consequently increase their stride length. Using Seven-League Boots, unlike the other two methods, diminishes positional accuracy at high gains, and users modify their walking behavior to compensate for the lack of control. We conclude with a discussion on each technique’s strength and weaknesses and the types of situation they might be appropriate for.

Paper, Video, Microsoft Research Blog


J. Hartmann (Waterloo & MSR), C. Holz, E. Ofek, & A. D. Wilson

Reality Check: Blending Virtual Environments with Situated Physical Reality

CHI 2019, to appear.

Today's virtual reality (VR) systems offer chaperone rendering techniques that prevent the user from colliding with physical objects. Without a detailed geometric model of the physical world, these techniques offer limited possibility for more advanced compositing between the real world and the virtual. We explore this using a realtime 3D reconstruction of the real world that can be combined with a virtual environment. RealityCheck allows users to freely move, manipulate, observe, and communicate with people and objects situated in their physical space without losing the sense of immersion or presence inside their virtual world. We demonstrate RealityCheck with seven existing VR titles, and describe compositing approaches that address the potential conflicts when rendering the real world and a virtual environment together. A study with frequent VR users demonstrate the affordances provided by our system and how it can be used to enhance current VR experiences.


L-P. Cheng (HPI), E. Ofek, C. Holz & A.D. Wilson

VRoamer: Generating On-The-Fly VR Experiences While Walking inside Large, Unknown Real-World Building Environments

IEEE VR 2019

Procedural generation in virtual reality (VR) has been used to adapt the virtual world to various indoor environments, fitting different geometries and interiors with virtual environments. However, such applications require that the physical environment be known or pre-scanned prior to use to then generate the corresponding virtual scene, thus restricting the virtual experience to a controlled space. In this paper, we present VRoamer, which enables users to walk unseen physical spaces for which VRoamer procedurally generates a virtual scene on-the-fly. Scaling to the size of office buildings, VRoamer extracts walkable areas and detects physical obstacles in real time using inside-out tracking, instantiates pre-authored virtual rooms if their sizes fit physically walkable areas or otherwise generates virtual corridors and doors that lead to undiscovered physical areas. The use of these virtual structures that connect pre-authored scenes on-the-fly allow VRoamer to (1) temporarily block users’ passage, thus slowing them down while increasing VRoamer’s insight into newly discovered physical areas, (2) prevent users from seeing changes beyond the current virtual scene, and (3) obfuscate the appearance of physical environments. VRoamer animates virtual objects to reflect dynamically discovered changes of the physical environment, such as people walking by or obstacles that become apparent only with closer proximity. In our feasibility evaluation of VRoamer, participants were able to quickly walk long distances through a procedurally generated dungeon experience and reported high levels of immersion in a post-hoc questionnaire.

Paper, Video, Microsoft Research Blog


J. Grubert (Coburg), E. Ofek, M. Pahud, P.O. Kristensson (Cambridge)

The Office of the Future: Virtual, Portable, and Global

IEEE Computer Graphics & Application, Nov/Dec 2018

Virtual reality has the potential to change the way we work. We envision the future office worker to be able to work productively everywhere solely using portable standard
input devices and immersive head-mounted displays. Virtual reality has the potential to enable this, by allowing users to create working environments of their choice and by relieving them from physical world limitations, such as constrained space or noisy environments. In this paper, we investigate opportunities and challenges for realizing this vision and discuss implications from recent findings of text entry in virtual reality as a core office task. 

Paper, Video1, Video2



C. C. Berger (CalTech), M. Goznalez-Franco, E. Eyal and K. Hinckley

Science Robotics 18 Apr 2018: Vol. 3, Issue 17, eaar7010

The Uncanny Valley of Haptics

During teleoperation and virtual reality experiences, enhanced haptic feedback incongruent with other sensory cues can reduce subjective realism, producing an uncanny valley of haptics

Paper, Video Microsoft Research Blog



C. Holz & E. Ofek

IEEE Engineering in Medicine and Biology Society (EMBC'18)  2018

Doubling the Signal Quality of Smartphone Camera Pulse Oximetry Using the Display Screen As a Controllable Selective Light Source. 

Recent smartphones have the potential to bring camera oximetry to everyone using their powerful sensors and the capability to process measurements in real-time, potentially augmenting people’s lives through always-available oximetry monitoring everywhere. The challenge of camera oximetry on smartphones is the low contrast between reflections from oxyhemoglobin and deoxyhemoglobin. In this paper, we show that this is the result of using the camera flash for illumination, which illuminates evenly across bands and thus leads to the diminished contrast in reflections. Instead, we propose capturing pulse using the front-facing camera and illuminating with the phone’s display, a selective illuminant in the red, green, and blue band. We evaluate the spectral characteristics of the phone display using a spectroradiometer in a controlled experiment, convolve them with the sensitivity curves of the phone’s camera, and show that the screen’s narrow-band display illumination increases the contrast between the reflections in the desired bands by a factor of two compared to flash illumination. Our preliminary evaluation showed further support for our approach and findings.


M. Pahud, E. Ofek, N. H. Riche, M. C. Hurter (ENAC) and J. Grubert (Coburg)

Data Visualization on Mobile Devices Workshop 2018

Mobiles as Portals for Interaction with Virtual Data Visualizations



I. Choi (Stanford), E. Ofek, H. Benko, M. Sinclair and C.Holz

CHI 2018

CLAW: A Multifunctional Handheld Haptic Controller for Grasping, Touching, and Triggering in Virtual Reality

CLAW is a handheld virtual reality controller that augments the typical controller functionality with force feedback and actuated movement to the index finger. Our controller enables three distinct interactions (grasping virtual object, touching virtual surfaces, and triggering) and changes its corresponding haptic rendering by sensing the differences in the user’s grasp.A servo motor coupled with a force sensor renders controllable forces to the index finger during grasping and touching. Using position tracking, a voice coil actuator at the index fingertip generates vibrations for various textures synchronized with finger movement. CLAW also supports a haptic force feedback in the trigger mode when the user holds a gun. We describe the design considerations for CLAW and evaluate its performance through two user studies. The first study obtained qualitative user feedback on the naturalness, effectiveness, and comfort when using the device. The second study investigated the ease of the transition between grasping and touching when using our device.

Paper, Video, Microsoft Research Blog, Engadget ("Microsoft’s mad scientists are making AR more tactile")Co. Design ("Crazy Microsoft is the best Microsoft"),  Gizmodo, Road To VR


E. Strasnick (Stanford), C. Holz, E. Ofek, M. Sinclair Hrvoje Benko

Haptic Links: Bimanual Haptics for Virtual Reality Using Variable Stiffness Actuation 

CHI 2018

We present Haptic Links, electro-mechanically actuated physical connections capable of rendering variable stiffness between two commodity handheld virtual reality (VR) controllers. When attached, Haptic Links can dynamically alter the forces perceived between the user’s hands to support the haptic rendering of a variety of two-handed objects and interactions. They can rigidly lock controllers in an arbitrary  configuration, constrain specific degrees of freedom or directions of motion, and dynamically set stiffness along a continuous range. We demonstrate and compare three prototype Haptic Links: Chain, Layer-Hinge, and Ratchet-Hinge. We then describe interaction techniques and scenarios leveraging the capabilities of each. Our user evaluation results confirm that users can perceive many two-handed  objects or interactions as more realistic with Haptic Links than with typical unlinked VR controllers.

Paper, Video,  Microsoft Research Blog, Engadget ("Microsoft’s mad scientists are making AR more tactile"), Co. Design ("Crazy Microsoft is the best Microsoft"), Gizmodo, Road To VR,  Slashgear, VRScout


E. Whitmire (UW), H. Benko, C. Holz, E. Ofek and Mike Sinclair

Haptic Revolver: Touch, Shear, Texture, and Shape Rendering on a Reconfigurable Virtual Reality Controller

Honorable Mentioned paper,

CHI 2018

We present Haptic Revolver, a handheld virtual reality controller that renders fingertip haptics when interacting with virtual surfaces. Haptic Revolver’s core haptic element is an actuated wheel that raises and lowers underneath the finger to render contact with a virtual surface. As the user’s finger moves along the surface of an object, the controller spins the wheel to render shear forces and motion under the fingertip. The wheel is interchangeable and can contain physical textures, shapes, edges, or active elements to provide different sensations to the user. Because the controller is spatially tracked, these physical features can be spatially registered with the geometry of the virtual environment and rendered on-demand. We evaluated Haptic Revolver in two studies to understand how wheel speed and direction impact perceived realism. We also report qualitative feedback from users who explored three application scenarios with our controller.

Paper, Video. Microsoft Research Blog, Engadget ("Microsoft’s mad scientists are making AR more tactile"), Co. Design ("Crazy Microsoft is the best Microsoft"), Gizmodo, Road To VR


J. Grubert (Cuborg), L. Witzani, E. Ofek, M. Pahud, M. Kraz and P.O. Kristensson (Cambridge)

Effects of Hand Representations for Typing in Virtual Reality

IEEE VR Mar. 2018

Alphanumeric text entry is a challenge for Virtual Reality (VR) applications. VR enables new capabilities, impossible in the real world, such as an unobstructed view of the keyboard, without occlusion by the user’s physical hands. Several hand representations have been proposed for typing in VR on standard physical keyboards. However, to date, these hand representations have not been compared regarding their performance and effects on presence for VR text entry. Our work addresses this gap by comparing existing hand representations with minimalistic fingertip visualization. We study the effects of four hand representations (no hand representation, inverse kinematic model, fingertip visualization using spheres and video inlay) on typing in VR using a standard physical keyboard with 24 participants. We found that the fingertip visualization and video inlay both resulted in statistically significant lower text entry error rates compared to no hand or inverse kinematic model representations. We found no statistical differences in text entry speed.

Paper, Video, Microsoft Research Blog



J. Grubert (Cuborg), L. Witzani, E. Ofek, M. Pahud, M. Kraz and P.O. Kristensson (Cambridge)

Text Entry in Immersive Head-Mounted Display-based Virtual Reality using Standard Keyboards

IEEE VR Mar. 2018

We study the performance and user experience of two popular mainstream text entry devices, desktop keyboards, and touchscreen keyboards, for use in Virtual Reality (VR) applications. We discuss the limitations arising from limited visual feedback and examine the efficiency of different strategies of use. We analyze a total of 24 hours of typing data in VR from 24 participants and find that novice users are able to retain about 60% of their typing speed on a desktop keyboard and about 40–45% of their typing speed on a touchscreen keyboard. We also find no significant learning effects, indicating that users can transfer their typing skills fast into VR. Besides investigating baseline performances, we study the position in which keyboards and hands are rendered in space. We find that this does not adversely affect performance for desktop keyboard typing and results in a performance trade-o for touchscreen keyboard typing. 

Paper, Video,  Microsoft Research Blog



C. Hurter, N. H. Riche, M. Pahud, E. Ofek, S. Drucker, B. Lee, D. Brown, C. Wong

Into the mixed reality data sphere: mapping user’s movements to data
exploration tools

#Immersive2017 Workshop on Immersive Analytics: Exploring Future Interaction and Visualization Technologies for Data Analytics

Workshop at IEEE VIS, 1st October, Phoenix.

In this paper, we propose to take advantage of user movements when exploring data in a virtual reality environment. As Human-Computer Interaction (HCI) relies on a fast and accurate binding between users actions and the system’s responses, we explore using the user location, as measured by the HoloLens head tracking, for data exploration, leaving the user hands free for data manipulation and selection. Since HoloLens has a very accurate measurement of user’s head movements, we investigated the available design space to efficiently map data exploration actions to these user head movements. To better investigate our design space, we implemented prototypes and explored new interaction techniques thanks to different data types and data exploration tasks.



CHI 2017

We propose a class of passive haptics that we call Sparse Haptic Proxy: a set of geometric primitives that simulate touch feedback in elaborate virtual reality scenes. Unlike previous passive haptics that replicate the virtual environment in physical space, a Sparse Haptic Proxy simulates a scene’s detailed geometry by redirecting the user’s hand to a matching primitive of the proxy. To bridge the divergence of the scene from the proxy, we augment an existing haptic retargeting technique with an on-the-fly target remapping: We predict users’ intentions during interaction in the virtual space by analyzing their gaze and hand motions, and consequently redirect their hand to a matching part of the proxy. \ We conducted three user studies on haptic retargeting technique and implemented a system from three main results: 1) The maximum angle participants found acceptable for retargeting their hand is 40°, with an average rating of 4.6 out of 5. 2) Tracking participants’ eye gaze reliably predicts their touch intentions (97.5%), even while simultaneously manipulating the user’s hand-eye coordination for retargeting. 3) Participants preferred minimized retargeting distances over better-matching surfaces of our Sparse Haptic Proxy when receiving haptic feedback for single-finger touch input. \ We demonstrate our system with two virtual scenes: a flight cockpit and a room quest game. While their scene geometries differ substantially, both use the same sparse haptic proxy to provide haptic feedback to the user during task completion.
Cited by more than 90 publications.



J. Grubert (Cuborg), E. Ofel, M. Pahud,

GlassHands: Interaction Around Unmodified Mobile Devices Using Sunglasses

ACM ISS 2016

We present a novel approach for extending the input space around unmodified mobile devices. Using built-in front facing cameras of unmodified handheld devices, GlassHands estimates hand poses and gestures through reflections in sunglasses, ski goggles or visors. Thereby, GlassHands creates an enlarged input space, rivaling input reach on large touch displays. We introduce the idea along with its technical concept and implementation. We demonstrate the feasibility and potential of our proposed approach in several application scenarios, such as map browsing or drawing using a set of interaction techniques previously possible only with modified mobile devices or on large touch displays. Our research is backed up with a user study.

Paper, Video, msPowerUser ("Microsoft’s latest Human Interaction Research will blow your mind") , CMWinBuzzer, IT News



H. Benko, C. Holz, M. Sinclair, E. Ofek

NormalTouch and TextureTouch: High-fidelity 3D Haptic Shape Rendering on Handheld Virtual Reality Controllers

Honorable Mentioned paper,

UIST 2016

We present an investigation of mechanically-actuated handheld controllers that render the shape of virtual objects through physical shape displacement, enabling users to feel 3D surfaces, textures, and forces that match the visual rendering.
We demonstrate two such controllers, NormalTouch and TextureTouch, which are tracked in 3D and produce spatially-registered haptic feedback to a user’s finger. NormalTouch haptically renders object surfaces and provides force feedback using a tiltable and extrudable platform. TextureTouch renders the shape of virtual objects including detailed surface structure through a 4×4 matrix of actuated pins. By moving our controllers around while keeping their finger on the actuated platform, users obtain the impression of a much larger 3D shape by cognitively integrating output sensations over time. Our evaluation compares the effectiveness of our controllers with the two de-facto standards in Virtual Reality controllers: device vibration and visual feedback only. We find that haptic feedback significantly increases the accuracy of VR interaction, most effectively by rendering high-fidelity shape output as in the case of our controllers.
Participants also generally found NormalTouch and TextureTouch realistic in conveying the sense of touch for a variety of 3D objects.

Paper, Video
Wired UK, Digital Thrends, Road to VR, Makezine

Cited by more than 80 publications.


B.  Nuernberger, E. Ofek, H. Benko &  A. D. Wilson

SnapToReality: Aligning Augmented Reality to the Real World

CHI 2016

Augmented Reality (AR) applications may require the precise alignment of virtual objects to the real world. We propose automatic alignment of virtual objects to physical constraints calculated from the real world in real time ("snapping to reality"). We demonstrate SnapToReality alignment techniques that allow users to position, rotate, and scale virtual content to dynamic, real world scenes. Our proof-of-concept prototype extracts 3D edge and planar surface constraints. We furthermore discuss the unique design challenges of snapping in AR, including the user's limited field of view, noise in constraint extraction, issues with changing the view in AR, visualizing constraints, and more. We also report the results of a user study evaluating SnapToReality, confirming that aligning objects to the real world is significantly faster when assisted by snapping to dynamically extracted constraints. Perhaps more importantly, we also found that snapping in AR enables a fresh and expressive form of AR content creation.

Paper, Video


M. Azmandian, M. Hancock, H. Benko, E. Ofek, A. D Wilson

Haptic Retargeting: Dynamic Repurposing of Passive Haptics for Enhanced Virtual Reality Experiences

CHI 2016

Manipulating a virtual object with appropriate passive haptic cues provides a satisfying sense of presence in virtual reality. However, scaling such experiences to support multiple virtual objects is a challenge as each one needs to be accompanied with a precisely-located haptic proxy object. We propose a solution that overcomes this limitation by hacking human perception. We have created a framework for repurposing passive haptics, called haptic retargeting, that leverages the dominance of vision when our senses conflict. With haptic retargeting, a single physical prop can provide passive haptics for multiple virtual objects. We introduce three approaches for dynamically aligning physical and virtual objects: world manipulation, body manipulation and a hybrid technique which combines both world and body manipulation. Our study results indicate that all our haptic retargeting techniques improve the sense of presence when compared to typical wand-based 3D control of virtual objects. Furthermore, our hybrid haptic retargeting achieved the highest satisfaction and presence scores while limiting the visible side-effects during interaction.

Paper, Video (Best of Videos, CHI 2016), Gamasutra, TechSpot, Wearable



T. Pajsa, H. Benko, A. D. Wilson  & E. Ofek

Room2Room: Enabling Life-Size Telepresence in a Projected Augmented Reality Environment

Best Paper,

CSCW 2016

Room2Room is a life-size telepresence system that leverages projected augmented reality to enable co-present interaction between two remote participants. Our solution recreates the experience of a face-to-face conversation by performing 3D capture of the local user with color + depth cameras and projecting their virtual copy into the remote space at life-size scale. This creates an illusion of the remote person’s physical presence in the local space, as well as a shared understanding of verbal and non-verbal cues (e.g., gaze, pointing) as if they were there. In addition to the technical details of our two prototype implementations, we contribute strategies for projecting remote participants onto physically plausible seating or standing locations, such that they form a natural and consistent conversational formation with the local participant. We also present observations and feedback from an evaluation with 7 pairs of participants on the usability of our solution for solving a collaborative, physical task.

paper, Video, MIT Tech Review, ZDnet



S. Korman, S. Avidan & E. Ofek

Peeking Template Matching for Depth Extension

ICCV 2015

We propose a method that extends a given depth image into regions in 3D that are not visible from the point of view of the camera. The algorithm detects repeated 3D structures in the visible scene and suggests a set of 3D extension hypotheses, which are then combined together through a global 3D MRF discrete optimization. The recovered global 3D surface is consistent with both the input depth map and the hypotheses.A key component of this work is a novel 3D template matcher that is used to detect repeated 3D structure in the scene and to suggest the hypotheses. A unique property of this matcher is that it can handle depth uncertainty. This is crucial because the matcher is required to “peek around the corner”, as it operates at the boundaries of the visible 3D scene where depth information is missing. The proposed matcher is fast and is guaranteed to find an approximation to the globally optimal solution.We demonstrate on real-world data that our algorithm is capable of completing a full 3D scene from a single depth image and can synthesize a full depth map from a novel viewpoint of the scene. In addition, we report results on an extensive synthetic set of 3D shapes, which allows us to evaluate the method both qualitatively and quantitatively.

paper, supliment materials, bibtex



 H. Benko, E. Ofek, Feng Zheng & A. D. Wilson 

FoveAR: Combining an Optically See-Through Near-Eye Display with Projector-Based Spatial Augmented Reality

UIST 2015

Optically see-through (OST) augmented reality glasses can overlay spatially-registered computer-generated content onto the real world. However, current optical designs and weight considerations limit their diagonal field of view to less than 40 degrees, making it difficult to create a sense of immersion or give the viewer an overview of the augmented reality space. We combine OST glasses with a projection-based spatial augmented reality display to achieve a novel display hybrid, called FoveAR, capable of greater than 100 degrees field of view, view dependent graphics, extended brightness and color, as well as interesting combinations of public and personal data display. We contribute details of our prototype implementation and an analysis of the interactive design space that our system enables. We also contribute four prototype experiences showcasing the capabilities of FoveAR as well as preliminary user feedback providing insights for enhancing future FoveAR experiences.

paper, video



B. Ens (Minetoba), E. Ofek, N. Bruce (Minetoba) & P. Irani (Minetoba)

Spatial Constancy of Surface-Embedded Layouts across Multiple Environments

The ACM Symposium on Spatial User Interaction (SUI)  2015

We introduce a layout manager that exploits the robust sensing capabilities of next-generation head-worn displays by embedding virtual application windows in the user’s surroundings. With the aim of allowing users to find applications quickly, our approach leverages spatial memory of a known body-centric configuration. The layout manager balances multiple constraints to keep layouts consistent across environments while observing geometric and visual features specific to each locale. We compare various constraint weighting schemas and discuss outcomes of this approach applied to models of two test environments.





 B. R. Jones, H. Benko, E. Ofek, A. D. Wilson 

IllumiRoom: Immersive Experiences Beyond the TV Screen

Communications of the ACM, Vol. 58 No. 6, June 2015

IllumiRoom is a proof-of-concept system that surrounds a television with projected light, bringing video games, and film experiences out of the TV screen and into the real world. IllumiRoom uses 3D scanning and projected light to change the appearance of the room, induce apparent motion, extend the field of view, and enable entirely new gaming experiences. Our system is entirely self-calibrating and is designed to work in any room. We present a detailed exploration of the design space of possible projected visualizations and we demonstrate ways to trigger and drive these visualizations from gaming content. We also contribute specific feedback from two groups of target users (10 gamers and 15 game designers); providing insights for enhancing game and film experiences beyond the TV.

Video, Popular Science



SurroundWeb: Microsoft's Plan To Cloak Your Living Room With Internet

J.Wilk, D. Molnar, E. Ofek, C. Rossbach, B. Livshits, A. Moshchuk, H.J.Wang & R. Gal

SurroundWeb: Mitigating Privacy Concers in a 3D Web Browser

IEEE Security & Privacy 2015

Immersive experiences that mix digital and real-world objects are becoming reality, but they raise serious privacy concerns as they require real-time sensor input. These experiences are already present on smartphones and game consoles via Kinect, and will eventually emerge on the web platform. However, browsers do not expose the display interfaces needed to render immersive experiences. Previous security research focuses on controlling application access to sensor  input alone, and do not deal with display interfaces. Recent research in human computer interactions has explored a variety of high-level rendering interfaces for immersive experiences, but these interfaces reveal sensitive data to the application. Bringing immersive experiences to the web requires a high-level interface that mitigates privacy concerns.
This paper presents SurroundWeb, the first 3D web browser, which provides the novel functionality of rendering web content onto a room while tackling many of the inherent privacy challenges. Following the principle of least privilege, we propose three abstractions for immersive rendering:1)the room skeleton lets applications place content in response to the physical dimensions and locations of renderable surfaces in a room; 2) the detection sandbox lets applications declaratively place content near recognized objects in the room without revealing if the object is present; and 3)satellite screens let applications display content across devices registered with SurroundWeb.Through user surveys, we validate that these abstractions limit the amount of revealed information to an acceptable degree.In addition, we show that a wide range of immersive experiences can be implemented with acceptable performance.

Video, Tech Report

Selected Media Coverage:
BBC (TV coverage)

The Telegraph
Daily Mail

T. Nguyen, M. Thaoduyen, S.Iqbal  & E. Ofek

The Known Stranger: Supporting Conversations between Strangers with Personalized Topic Suggestion

CHI 2015

For some people, it is often difficult to strike up a good conversation with new acquaintances. In this paper we report on the perceptions of wearable device users who were given real-time personalized topic suggestions during a conversation with a person they just met. The personalized topic suggestions were generated using a ranking recommendation algorithm, and were delivered via Google Glasses. We conducted experiments with 38 pairs of strangers, who received such suggestions while conversing for the first time. Participants found the suggestions to be helpful, but only at the right moments,
and for certain types of speakers. Our results contribute to the understanding of how communication interventions influence people’s experience and behaviours, and how wearable technologies can enhance interpersonal interactions. Our study also presents design implications for applications on wearable devices to facilitate conversations between strangers





B.Jones, R. Shdhi, M. Murdock, R. Mehra, H. Benko, A. Wilson, E.Ofek, B. MacIntyre, N. Raghuvanshi& L. Shapira

RoomAlive: Magical Experiences Enabled by Scalable, Adaptive Projector-Camera Units

UIST 2014

RoomAlive is a proof-of-concept prototype that transforms any room into an immersive, augmented entertainment experience. Our system enables new interactive projection mapping experiences that dynamically adapts content to any room. Users can touch, shoot, stomp, dodge and steer projected content that seamlessly co-exists with their existing physical environment. The basic building blocks of RoomAlive are projector-depth camera units, which can be combined through a scalable, distributed framework. The projector-depth camera units are individually auto-calibrating, self-localizing, and create a unified model of the room with no user intervention. We investigate the design space of gaming experiences that are possible with RoomAlive and explore methods for dynamically mapping content based on room layout and user position. Finally we showcase four experience prototypes that demonstrate the novel interactive experiences that are possible with RoomAlive and discuss the design challenges of adapting any game to any room.

Paper, Video

Cited by more than 180 publications.

RoomAlive: The Other Resident

Using RoomAlive capabilities, Microsoft researchers and summer intern Julian Kantor from the University of Southern California, Tomislav Pejsa from the University of Wisconson-Madison, Hrvoje BenkoAndy Wilson and myself have developed a video that is as frightening as it as inventive. For more about RoomAlive, check out this great FastCompany piece.


Selected Media Coverage:
The Washington Post


The Gradian


Creating a layout for an augmented reality (AR) application which embeds virtual objects in a physical environment is difficult as it must adapt to any physical space. We propose a rule-based framework for generating object layouts for AR applications. Under our framework, the developer of an AR application specifies a set of rules (constraints) which enforce self-consistency (rules regarding the inter-relationships of application components) and scene consistency (application components are consistent with the physical environment they are placed in). When a user enters a new environment, we create, in real-time, a layout for the application, which is consistent with the defined constraints (as much as possible). We find the optimal configurations for each object by solving a constraint-satisfaction problem. Our stochastic move making algorithm is domain-aware, and allows us to efficiently converge to a solution for most rule-sets. In the paper we demonstrate several augmented reality applications that automatically adapt to different rooms and changing circumstances in each room.






S. Jana, D. Molnar, A. Moshchuk, A. Dunn, B. Livshits, H. J. Wang, & E. Ofek

Enabling Fine-Grained Permissions for Augmented Reality Applications With Recognizers

22nd Usenix Security Symposium 2013

Augmented reality (AR) applications sense the environment, then render virtual objects on human senses. Examples include smart-phone applications that annotate storefronts with reviews and X-Box Kinect games that show avatars mimicking human movements. No current OS has special support for such applications. As a result, permissions for AR applications are necessarily coarse-grained: applications must ask for access to raw sensor feeds, such as video and audio. These raw feeds expose significant additional information beyond what applications need, including sensitive information such as the user's location, face, or surroundings.
Instead of exposing raw sensor data to applications directly, we introduce a new OS abstraction: the recognizer. A recognizer takes raw sensor data as input and exposes higher-level objects, such as a skeleton or a face, to applications. We propose a fine-grained permission system where applications request permissions at the granularity of recognizer objects. We analyze 87 shipping AR applications and found that a set of four core recognizers covers almost all current apps. We also introduce privacy goggles, a visualization of sensitive data exposed to an application. Surveys of 962 people establish a clear "privacy ordering" over recognizers and demonstrate that privacy goggles are effective at communicating application capabilities. We build a prototype on Windows that exposes nine recognizers to applications, including the Kinect skeleton tracker. Our prototype incurs negligible overhead for single applications, while improving performance of concurrent applications and enabling secure offloading of heavyweight recognizer computation.

Cited by more than 50 publications.


 Y. Eshet, S. Korman, E. Ofek & S. Avidan

DCSH - Matching Patches in RGBD Images

IEEE ICCV 2013, Sydney Australia 

We extend patch based methods to work on patches in 3D space. We start with Coherency Sensitive Hashing (CSH), which is an algorithm for matching patches between two RGB images, and extend it to work with RGBD images. This is done by warping all 3D patches to a common virtual plane in which CSH is performed. To avoid noise due to warping of patches of various normals and depths, we estimate a group of dominant planes and compute CSH on each plane separately, before merging the matching patches. The result is DCSH - an algorithm that matches world (3D) patches in order to guide the search for image plane matches. An independent contribution is an extension of CSH, which we term Social-CSH. It allows a major speedup of the k nearest neighbor (kNN) version of CSH - its runtime growing linearly, rather than quadratically, in k. Social-CSH is used as a subcomponent of DCSH when many NNs are required, as in the case of image denoising. We show the benefits of using depth information to image reconstruction and image denoising, demonstrated on several RGBD images.



L. D'Antoni, A. Dunn, S. Jana, T. Kohno, B. Livshits, D. Molnar,
A. Moshchuk, E. Ofek, F. Roesner, S. Saponas, M. Veanes & H. J. Wang

Operating System Support For Augmented Reality Applications

The 14th Workshop on Hot Topics in Operating Systems (HotOS XIV), 2013, New Mexcio, USA

Augmented reality (AR) takes natural user input (NUI), such as gestures, voice, and eye gaze, and produces digital visual overlays on top of reality seen by a user. Today, multiple shipping AR applications exist, most notably titles for the Microsoft Kinect and smartphone applications such as Layar, Wikitude, and Junaio. Despite this activity, little attention has been paid to operating system support for AR applications. Instead, each AR application today does its own sensing and rendering, with the help of  ser-level libraries like OpenCV or the Microsoft Kinect SDK.
In this paper, we explore how operating systems should evolve to support AR applications. Because AR applications work with fundamentally new inputs and outputs, an OS that supports AR applications needs to re-think the input and display abstractions exposed to applications. Unlike mouse and keyboard, which form explicit, separate channels for user input, NUI requires continuous sensing of the real-world environment, which often has sensitive data mixed with user input. Hence, the OS input abstractions must ensure that user privacy is not violated, and the OS must provide a fine grained permission system for access to recognized objects like a user's face and skeleton. In addition, because visual outputs of AR applications mix real-world and virtual objects, the synthetic window abstraction in traditional GUIs is no longer viable, and OSes must rethink the display abstractions and their management. We discuss research directions for solving these and other issues and building an OS that let multiple applications share one (augmented) reality.


B. Jones, H. Benko, E. Ofek & A. Wilson


ACM SIGCHI 2013, Paris, France

CHI 2013 Best Paper Award

CHI 2013 Golden Mouse award (Best Video show)

IllumiRoom is a proof-of-concept system that augments the area surrounding a television with projected visualizations to enhance traditional gaming experiences. We investigate how projected visualizations in the periphery can negate, include, or augment the existing physical environment and complement the content displayed on the television screen. Peripheral projected illusions can change the appearance of the room, induce apparent motion, extend the field of view, and enable entirely new physical gaming experiences. Our system is entirely self-calibrating and is designed to work in any room. We present a detailed exploration of the design space of peripheral projected illusions and we demonstrate ways to trigger and drive such illusions from gaming content. We also contribute specific feedback from two groups of target users (10 gamers and 15 game designers); providing insights for enhancing game experiences through peripheral projected illusions.

Cited by more than 200 publications.

Selected media coverage:


The Verge






E. Ofek, K. Strauss & S.T. Iqbal

Reducing Disruption from Subtle Information Delivery during a Conversation: Mode and Bandwidth Investigation

ACM SIGCHI 2013, Paris, France.

 With proliferation of mobile devices that provide ubiquitous access to information, the question arises of how distracting processing information can be in social settings, especially during a face-to-face conversation. At the same time, relevant information presented at opportune moments may help enhance conversation quality. In this paper, we investigate how much information users can consume dur-ing a conversation and what information delivery mode, via audio or visual aids, helps them effectively conceal the fact that they are receiving information. We observe that users can internalize more information while still disguising this fact the best when information is delivered visually in batches (multiple pieces of information at a time and per-form better on both dimensions if information is delivered while they are not speaking. Participants qualitatively did not prefer this mode as being the easiest to use, preferring modes that displayed one piece of information at a time.





M. Kroepfl, Y. Wexler & E. Ofek

Efficiently Locating Photographs in Many Panoramas

ACM SIGSPATIAL GIS 2010 (Full Paper)

 We present a method for efficient and reliable geo-positioning of images. It relies on image-based matching of the query images onto a trellis of existing images that provides accurate 5-DOF calibration (camera position and orientation without scale). As such it can handle any image input, including old historical images, matched against a whole city.  At that scale, care needs to be taken with the size of the database. We deviate from previous work by using panoramas to simultaneously reduce the database size and increase the coverage.  To reduce the likelihood of false matches, we restrict the range of angles for matched features. Furthermore, we enhance the RANSAC procedure to include two phases. The second phase includes guided feature matching to increase the likelihood of positive matches. Hence, we devise a matching confidence score that separates between true and false matches.   We demonstrate the algorithm on a large scale database covering a whole city and show its uses to vision-based augmented reality system.

TED talk by Blaise Aguera y Arcas.

Geek in Disguise

Search Engine Land

Tech Flash

Bing Blog




B. Zhang, Q. Li, H. Chao, B. Chen, E. Ofek & Y.Q. Xu

Annotating and Navigating Tourist Videos

ACM SIGSPATIAL GIS 2010 (Full paper)

 Due to the rapid increase in video capture technology, more and more tourist videos are captured every day, creating a challenge for organization and association with metadata.

In this paper, we present a novel system for annotating and navigating tourist videos. Placing annotations in a video is difficult because of the need to track the movement of the camera. Navigation of a regular video is also challenging due to the sequential nature of the media. To overcome these challenges, we introduce a system for registering videos to geo-referenced 3D models and analyzing the video contents.

We also introduce a novel scheduling algorithm for showing annotations in video. We show results in automatically annotated videos and in a map-based application for browsing videos. Our user study indicates the system is very useful.



B. Epshtein, Y. Wexler & E. Ofek 

Detecting Text in Natural Scenes with Stroke Width Transform

IEEE CVPR 2010 (Oral)

 We present a novel image operator that seeks to find the value of stroke width for each image pixel, and demonstrate its use on the task of text detection in natural images. The suggested operator is local and data dependent, which makes it fast and robust enough to eliminate the need for multi-scale computation or scanning windows. Extensive testing shows that the suggested scheme outperforms the latest published algorithms. Its simplicity allows the algorithm to detect texts in many fonts and languages.

Video of the CVPR talk

 Cited by more than 1250 publications.


R. Gal (Tel Aviv), Y. Wexler, H. Hoppe, E. Ofek & D. Cohen-Or

Seamless Montage for Texturing Models

EuroGraphics 2010 (Oral)

 We present an automatic method to recover high-resolution texture over an object shape by mapping detailed photographs onto its surface. Such high-resolution detail often reveals inaccuracies in geometry and registration, as well as lighting variations and surface reflections. Simple image projection results in visible seams on the surface. We minimize such seams using a global optimization that assigns compatible texture to adjacent triangles. The key idea is to search not only combinatorially over the source images, but also over a set of local image transformations that compensate for geometric misalignment. This broad search space is traversed using a discrete labeling algorithm, aided by a coarse-to-fine strategy. Our approach significantly improves resilience to acquisition errors, thereby allowing simple, easy creation of textured models for use in computer graphics.

Paper     Supplimental Materials    Optimization Animation

Cited by more than 110 publications.




 N. Villar, S. Izadi, H. Benko, J. Helmes, D. Rosenfeld, E. Ofek, J. Westhues, A. Butler, X. Cao, B. Chen &  S. Hodges

Mouse 2.0: Multi-Touch Meets the Mouse

Best Paper

UIST 2009 

 In this paper we explore the possibilities for augmenting the standard computer mouse with multi-touch capabilities so that it can sense the position of the user’s fingers and thereby complement traditional pointer-based desktop interactions with touch and gestures. We present five different multi-touch mouse implementations, each of which explores a different touch sensing strategy, which leads to differing form-factors and hence interaction possibilities. In addition to the detailed description of hardware and software implementations of our prototypes, we discuss the relative strengths, limitations and affordances of these different input devices as informed by the results of a preliminary user study.

Paper      Video     

 Cited by more than 80 publications.


B. Chen, B. Neubert, E. Ofek, O. Deussen & M. F. Cohen

Integrated Videos and Maps for Driving Directions

UIST 2009

 While onboard navigation systems are gaining in importance, maps are still the medium of choice for laying out a route to a destination and for way finding. However, even with a map, one is almost always more comfortable navigating a route the second time due to the visual memory of the route. To make the first time navigating a route feel more familiar, we present a system that integrates a map with a video automatically constructed from panoramic imagery captured at close intervals along the route. The routing information is used to create a variable speed video depicting the route. During playback of the video, the frame and field of view are dynamically modulated to highlight salient features along the route and connect them back to the map. A user interface is demonstrated to allow exploration of the combined map, video, and textual driving directions. We discuss the construction of the hybrid map and video interface. Finally, we report the results of a study that provides evidence of the effectiveness of such a system for route following.

Paper   Video


T. Yamaguchi, B. Wilburn , Z. Cao & E. Ofek
Video-based modeling of dynamic hair

Pacific-Rim Symposium on Image & Video Technology 2009, Tokyo, Japan





P. Mishra, E. Ofek & G. Kimchi
Validation of Vector Data using Oblique Images
ACM SIGGIS 2008, CA USA (Full Paper)

Oblique images are aerial photographs taken at oblique angles to the earth’s surface. Projections of vector and other geospatial data in these images depend on camera parameters, positions of the entities, surface terrain, and visibility. This paper presents a robust and scalable algorithm to detect inconsistencies in vector data using oblique images. The algorithm uses image descriptors to encode the local appearance of a geospatial entity in images. These image descriptors combine color, pixel-intensity gradients, texture, and steerable filter responses. A Support Vector Machine classifier is trained to detect image descriptors that are not consistent with underlying vector data, digital elevation maps, building models, and camera parameters. In this paper, we train the classifier on visible road segments and non-road data. Thereafter, the trained classifier detects inconsistencies in vectors, which include both occluded and misaligned road segments. The consistent road segments validate our vector, DEM, and 3-D model data for those areas while inconsistent segments point out errors. We further show that a search for descriptors that are consistent with visible road segments in the neighborhood of a misaligned road yields the desired road alignment that is consistent with pixels in the image.



J. Xiao (HKUST), T. Fang (HKUST). P. Tan (HKUST), Z. Peng (HKUST), E. Ofek & L. Quan (HKUST)
Image Based Facade Modeling

We propose in this paper a semi-automatic image-based approach that uses images captured along the streets, and relies on structure from motion to automatically recover the camera positions and point clouds as the initial stage for the modeling. We start a building facade as a flat rectangular plane or a developable surface, and the texture image of the flat facade is composited from the multiple visible images with handling of occluding objects. A facade is then decomposed and structured into a Directed Acyclic Graph of rectilinear elementary patches. The decomposition is carried out top-down by a recursive subdivision, and followed by a bottom-up merging with the detection of the architectural bilateral symmetry and repetitive patterns. Each subdivided patch of the flat facade is augmented with the depth that is optimized from the 3D points. Our system also allows the user to easily provide feedbacks in the 2D image space for the proposed decomposition and augmentation. Finally, our approach is demonstrated on a large number of facades from a variety of street-side images.
Paper     Video

Cited by more than 190 publications.



H. Wang, H. Hoppe, Y. Wexler and E. Ofek

Factoring Repeated Content Within and Among Images


We reduce transmission bandwidth and memory space for images by factoring their repeated content. A transform map and a condensed epitome are created such that all image blocks can be reconstructed from transformed epitome patches. The transforms may include affine deformation and color scaling to account for perspective and tonal variations across the image. The factored representation allows efficient random-access through a simple indirection, and can therefore be used for real-time texture mapping without expansion in memory. Our scheme is orthogonal to traditional image compression, in the sense that the epitome is amenable to further compression such as DXT. Moreover it allows a new mode of progressivity, whereby generic features appear before unique detail. Factoring is also effective across a collection of images, particularly in the context of image-based rendering. Eliminating redundant content lets us include textures that are several times as large in the same memory space.
Paper     Supplimental Materials     Data

Cited by more than 60 publications.





B. Epshtein, E. Ofek, Y. Wexler and P. Zhang

Hierarchical photo organization using geo-relevance

ACM GIS 2007, Seattle WA, USA (Full paper)

We present a novel framework for organizing large collections of images in a hierarchical way, based on scene semantics. Rather than score images directly, we use them to score the scene in order to identify typical views and important locations which we term Geo-Relevance. This is done by relating each image with its viewing frustum which can be readily computed for huge collections of images nowadays. The frustum contains much more information than only camera position that has been used so far. For example, it distinguishes between a photo of the Eiffel Tower and a photo of a garbage bin taken from the exact same place. The proposed framework enables a summarized display of the information and facilitates efficient browsing.

Cited by more than 50 publications.





N.Li, N. Moraveji, H. Kimura and E. Ofek

Improving the Experience of Controlling Avatars in Camera-Based Games Using Physical Input

The 14th ACM International Conference on Multimedia, Santa Barbara, CA, USA 2006

This paper investigates two methods of improving the user experience of camera-based interaction. First, problems that arise when avatars are designed to mimic a user’s physical actions are presented. Second, a solution is proposed: adding a layer of separation between user and avatar while retaining intuitive user control. Two methods are proposed for this separation: spatially and temporally. Implementations of these methods are then presented in the context of a simple game and evaluate their effect on performance and satisfaction. Results of a human subject experiment are presented, showing that reducing the amount of user control can maintain, and even improve, user satisfaction if the design of such a reduction is appropriate. This is followed by a discussion of how the findings inform camera-based game design.

Video - Whack-a-mole

Video - Boxing (Note that this is done with a 2D camera and yet, the body motion is recovered in real-time.



H. Jiang, E. Ofek, N. Moraveji and S. Yuanchun

Direct Pointer: Direct Manipulation for Large Display Interaction using Handheld Cameras

ACM CHI  2006

This paper describes the design and evaluation of a technique, Direct Pointer, that enables users to interact intuitively with large displays using cameras equipped on handheld devices, such as mobile phones and personal digital assistant (PDA). In contrast to many existing interaction methods that attempt to address the same problem, ours offers direct manipulation of the pointer position with continuous visual feedback. The primary advantage of this technique is that it only requires equipment that is readily available: an electronic display, a handheld digital camera, and a connection between the two. No special visual markers in the display content are needed, nor are fixed cameras pointing at the display. We evaluated the performance of Direct Pointer as an interaction product, showing that it performs as well as comparable techniques that require more sophisticated equipment.

Paper    Video

Cited by more than 50 publications.



Y. Matsushita, E. Ofek, X. Tang and H. Shum

Full Frame Video Stabilization with Motion Inpainting


Video stabilization is an important video enhancement technology which aims at removing annoying shaky motion from videos. We propose a practical and robust approach of video stabilization that produces full-frame stabilized videos with good visual quality. While most previous methods end up with producing low resolution stabilized videos, our completion method can produce full-frame videos by naturally filling in missing image parts by locally aligning image data of neighboring frames. To achieve this, motion inpainting is proposed to enforce spatial and temporal consistency of the completion in both static and dynamic image areas. In addition, image quality in the stabilized video is enhanced with a new practical deblurring algorithm. Instead of estimating point spread functions, our method transfers and interpolates sharper image pixels of neighboring frames to increase the sharpness of the frame. The proposed video completion and deblurring methods enabled us to develop a complete video stabilizer which can naturally keep the original image quality in the stabilized videos. The effectiveness of our method is confirmed by extensive experiments over a wide variety of videos.


Video - Stabilization & compleation of a dynamic scene

Video - Stabilization of a Beach scene

Video - Video Completion

Cited by more than 450 publications.





Y. Wei, E. Ofek, L. Quan and H. Shum

Modeling Hair from Multi Views


In this paper, we propose a novel image-based approach to model hair geometry from images taken at multiple viewpoints. Unlike previous hair modeling techniques that require intensive user interactions or rely on special capturing setup under controlled illumination conditions, we use a handheld camera to capture hair images under uncontrolled illumination conditions. Our multi-view approach is natural and flexible for capturing. It also provides inherent strong and accurate geometric constraints to recover hair models. In our approach, the hair fibers are synthesized from local image orientations. Each synthesized fiber segment is validated and optimally triangulated from all visible views. The hair volume and the visibility of synthesized fibers can also be reliably estimated from multiple views. Flexibility of acquisition, little user interaction, and high quality results of recovered complex hair models are the key advantages of our method.

Cited by more than 100 publications.



Description: Description: Description: Description: Description: Description: \\research\root\web\external\en-us\UM\People\eyalofek\SideBySide.gif

Y. Matsushita, E. Ofek, X. Tang and H. Shum

Video Completion with Motion Inpainting for Video Stabilization

IEEE CVPR  2005 (Oral)

Video stabilization is an important video enhancement technology which aims at removing annoying shaky motion from videos. We propose a practical and robust approach of video stabilization that produces full-frame stabilized videos with good visual quality. While most previous methods end up with producing low resolution stabilized videos, our completion method can produce full-frame videos by naturally filling in missing image parts by locally aligning image data of neighboring frames. To achieve this, motion inpainting is proposed to enforce spatial and temporal consistency of the completion in both static and dynamic image areas. In addition, image quality in the stabilized video is enhanced with a new practical deblurring algorithm. Instead of estimating point spread functions, our method transfers and interpolates sharper image pixels of neighboring frames to increase the sharpness of the frame. The proposed video completion and deblurring methods enabled us to develop a complete video stabilizer which can naturally keep the original image quality in the stabilized videos. The effectiveness of our method is confirmed by extensive experiments over a wide variety of videos.

Cited by more than 150 publications.



B. Chen (Stanford), E. Ofek, H. Shum and M. Levoy (Stanford)

Interactive Deformation of Light Fields


Paper   Video




X. Cao, E. Ofek, and D. Vronay

Evaluation of Alterantive Presentation Control Techniques

ACM CHI 2005

This paper describes the design and evaluation of a technique, Direct Pointer, that enables users to interact intuitively with large displays using cameras equipped on handheld devices, such as mobile phones and personal digital assistant (PDA). In contrast to many existing interaction methods that attempt to address the same problem, ours offers direct manipulation of the pointer position with continuous visual feedback. The primary advantage of this technique is that it only requires equipments that are readily available: a large electronic display, a handheld camera, and a connection between the two. No special visual markers in the display content are needed, nor are fixed cameras pointing at the display. We evaluated the performance of Direct Pointer as an interaction product, showing that it performs as well as comparable techniques that require more sophisticated equipment.




Description: Description: Description: Description: Description: Description: girl

R. Gvili, A. Kaplan, E. Ofek, G. Yahav

 Depth Key

Stereoscopic Displays and Applications: The Engineering Reality of Virtual Reality 2003  (Proceedings of SPIE/IS&T Volume 5006)

Video      Video2

Cited by more than 130 publications.




M. Op de Beeck, P. Wilinski, C. Fehn, P. Kauff, W. A. Ijsselsteijn, M. Pollefeys, Luc J Van Gool, E. Ofek, & I. Sexton

Towards an optimized 3D broadcast chain

TCom 2002: The Convergence of Information Technologies and Communications

Nov. 1, 2002


C. Fehn, P. Kauff, M. Op de Beeck, F. Ernst, W. IJsselsteijn, M. Pollefeys, L. Van Gool, E. Ofek and I. Sexton

An Evolutionary And Optimized Approach On 3D-TV

Proc. of IBC, Sep.  2002. Vol 2. pp. 357-365

In this paper we will present the concept of a system that allows for an evolutionary introduction of depth perception into the existing 2D digital TV framework. The work is part of the European Information Society Technologies (IST) project “Advanced Three-Dimensional Television System Technologies” (ATTEST), an activity where industries, research centers and universities have joined forces to design a backwards compatible, flexible and modular broadcast 3D-TV system, where all parts of the 3D processing chain are optimized to one another. This includes
content creation, coding and transmission, display and research in human 3D perception, which be will used to guide the development process. The goals of the project comprise the development of a novel broadcast 3D
camera, algorithms to convert existing 2D-video material into 3D, a 2Dcompatible coding and transmission scheme for 3D-video using MPEG-2/4/7 technologies and the design of two new auto stereoscopic displays

Cited by more than 200 publications.


   Description: Description: Description: Description: Description: Description: 3DTV

A. Redert, M. Op de Beeck, C. Fehn, W. IJsselsteijn, M. Pollefeys, L. J. Van Gool, E. Ofek, I. Sexton & P. Surman

ATTEST: Advanced Three-dimensional Television System Technologies.

3DPVT 2002

Cited by more than 250 publications.





E. Ofek and A. Rappoport

Interactive reflections on curved objects


Global view-dependent illumination phenomena, in particular reflections, greatly enhance the realism of computer-generated imagery. Current interactive rendering methods do not provide satisfactory support for reflections on curved objects.
In this paper we present a novel method for interactive computation of reflections on curved objects. We transform potentially reflected scene objects according to reflectors, to generate virtual objects. These are rendered by the graphics system as ordinary objects, creating a reflection image that is blended with the primary image. Virtual objects are created by tessellating scene objects and computing a virtual vertex for each resulting scene vertex. Virtual vertices are computed using a novel space subdivision, the reflection subdivision.
For general polygonal mesh reflectors, we present an associated approximate acceleration scheme, the explosion map. For specific types of objects (e.g., linear extrusions of planar curves) the reflection subdivision can be reduced to a 2-D one that is utilized more accurately and efficiently

Cited by more than 110 publications.




E. Ofek, E. Shilat, A. Rappoport, and M. Werman.

Multi-resolution Textures from Image Sequences

IEEE Computer Graphics and Applications 1997

 Cited by more than 80 publications.


ACM Copyright Notice

Copyright by the Association for Computing Machinery, Inc. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Publications Dept, ACM Inc., fax +1 (212) 869-0481, or The definitive version of this paper can be found at ACM's Digital Library

Eurographics Association Copyright Notice

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than Eurographics must be honored.

IEEE Copyright Notice

This material is posted here with permission of the IEEE. Internal or personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution must be obtained from the IEEE by writing to

Springer-Verlag Copyright Notice

The Author may publish his/her contribution on his/her personal Web page provided that he/she creates a link to the above mentioned volume of LNCS at the Springer-Verlag server or to the LNCS series Homepage (URL: and that together with this electronic version it is clearly pointed out, by prominently adding "© Springer-Verlag", that the copyright for this contribution is held by Springer.