Oculus VR + StereoVision (OpenCV)

Hey guys, i'm currently working a VR+AR-solution using a DK2 and a selfmade stereovision (2 cams with small HD / 60fps each) Until now i was using Unity + CSharp + OpenCV for calibration and image processing but after stereo-calibration it's super slow. Especially when it comes with the latest Unity+Oculus drivers i get around 15-25fps (displaying just the undistorted/rectified 2 videostreams on the oculus) which is ... super bad. I could start learning writing multithreaded-dll's now. But before doing so i'm thinking of a better/faster solution with Fabric. I'd have to learn KL ofcourse but that doesn't frightend me. It's more that i'd like to make sure that i'm going to have full "access" to all the OpenCV API features. In "perfect" world i'd like to get the most of out OpenCV, Oculus and my custom webcams within Farbic's fanastic multithreading features. So is this theoretically possible? And again are all OpenCV features "working" within a Fabric-workflow? Thanks so much in advance!

Comments

  • fgiordanafgiordana Posts: 19

    Hi Frankster,

    I'm going through a very similar experiment at the moment.
    I've successfully wrapped some OpenCV functionality as an external c++ extension and I've managed to do a simple chessboard tracking through my webcam and now I'm wrapping up the camera calibration part. Next step will be the rendering of digital content on the chessboard for my AR experiments.

    I'd say the workflow isn't perfect, but I haven't had any major problems so far with the wrapping of OpenCV. It's still a bit naive, I'm sure the data handling can be done in a more efficient way, but it's a start.

    For a high performance integration there is still a long way to go but if I can make something useful out of it I might stick it on GitHub at some point.

  • HelgeHelge Moderator, Fabric Employee Posts: 314 Fabric Employee

    Hey Frankster,

    to pick up where fgiordana left it - yes - it's absolutely feasable. You will need to wrap the libraries you need as KL extensions. We have already done that for Oculus, the extension is available here:

    https://github.com/fabric-engine/oculus

    You don't need to build it, we provide binaries for 2.0.0 as part of the download.

    OpenCV is not yet wrapped, if fgiordana can make it available at some point that's great, otherwise you always have the option of wrapping it yourself or wait for us to do it in the future.

    For the webcam streams it's very similar, if you can get access to the images in C++ then it should be fairly straight forward to wrap the C++ code into a KL extension and use it directly within Fabric and Canvas.

    For more information on how to write extensions, please refer to
    http://docs.fabric-engine.com/FabricEngine/latest/HTML/ExtensionAuthoringGuide/index.html#extrg

    I hope this helps.

    Research Engineer @ Fabric Software

Sign In or Register to comment.