DASH-3D

DASH-3D

Dynamic Adaptive 3D Streaming over HTTP

Thomas Forgione
November 29th, 2019
Jury
Gilles Gesquière President
Sidonie Christophe Reviewer
Gwendal Simon Reviewer
Maarten Wijnants Examiner
Wei Tsang Ooi Examiner
Vincent Charvillat Supervisor
Axel Carlier Co-supervisor
Géraldine Morin Co-supervisor

Streaming

Marina Bay, Singapore

Problems we are facing

Content preparation How to prepare content in order to perform efficient streaming?
Streaming policies How to determine, at a given time, which chunk to download depending on the user's interactions?
Interaction How to provide an interface allowing the users to navigate in the scene easily?

Contributions

3D Acquisition

Multi-view-stereo [AliceVision]
Lidar [nsheldon.github.io]

3D Compression and Streaming

Progressive mesh [Hoppe 1996]

3D Compression and Streaming

[Streaming QSplat]

3D Compression and Streaming

First use of DASH for 3D Streaming [Zampoglou 2016]

3D Visualisation

Integration

Nexus [Cignoni et al. 2005]
3DHOP [Potenziani et al. 2015]

Integration

[Google Earth / Google Maps]

DASH [Stockhammer 2011]

A youtube video with multiple resolutions
DASH-SRD [Niamut et al. 2016]

From video streaming to 3D streaming

Video 3D
Degrees of freedom 1 5 or 6
Data persistence 2 to 10 seconds ∞
Viewpoint dependency ✗ ✓

Content preparation

Space partitionning of the model
<AdaptationSet>
    <SupplementalProperty value="-8834.11230,2201.58853,-0.16950, 174.81540,-1344.47740,4767.83367" />
    <BaseURL>as1/</BaseURL>
    <Representation>
        <BaseURL>repr1/</BaseURL>
        <SegmentList>
            <SegmentURL area="2540342.3" size="120K" media="s0.obj" />
            <SegmentURL area="1124.4" size="162K" media="s1.obj" />
            <SegmentURL area="412.6" size="173K" media="s2.obj" />
            <SegmentURL area="270.3" size="147K" media="s3.obj" />
            <SegmentURL area="209.7" size="154K" media="s4.obj" />
            <SegmentURL area="119.8" size="165K" media="s5.obj" />
            <SegmentURL area="31.5" size="111K" media="s6.obj" />
        </SegmentList>
    </Representation>
</AdaptationSet>
A geometry adaptation set from the MPD
<AdaptationSet faces="90" area="36644.25529" average="156,158,157,254" mse="99.02242"/>
  <BaseURL>textures/ESPGR034.PNG/</BaseURL>
  <Representation>
    <BaseURL>64x64/</BaseURL>
    <SegmentList>
      <SegmentURL size="9288" mse="42.74764" media="texture.png" />
    </SegmentList>
  </Representation>
  <Representation>
    <BaseURL>128x128/</BaseURL>
    <SegmentList>
      <SegmentURL size="39385" mse="99.02242" media="texture.png" />
    </SegmentList>
  </Representation>
</AdaptationSet>
A texture adaptation set from the MPD
Model with average colors instead of textures

Streaming policy

The client first downloads the MPD
The client filters the content to download using the viewport
The client downloads the relevant content

Streaming policy

Greedy

𝑠 * = a r g m a x 𝑠 ∈ 𝑆 𝑢 𝑡 𝑖 𝑙 𝑖 𝑡 𝑦 ( 𝑠 , 𝑣 ( 𝑡 ) ) 𝑠 𝑖 𝑧 𝑒 ( 𝑠 )

Geometry utility

𝑢 𝑡 𝑖 𝑙 𝑖 𝑡 𝑦 𝑠 𝐺 , 𝑣 ( 𝑡 ) = 𝒜 𝑠 𝐺 𝒟 ( 𝑣 ( 𝑡 ) , ℬ ( 𝑠 𝐺 ) ) 2
With
  • 𝑠 𝐺 the considered geometry segment
  • 𝑣 ( 𝑡 ) the camera viewpoint at time 𝑡
  • 𝒜 𝑠 𝐺 the total area covered by 𝑠 𝐺
  • 𝒟 𝑣 ( 𝑡 ) , ℬ 𝑠 𝐺 2 the distance between the camera and the center of the bounding box containing 𝑠 𝐺

Texture utility

𝑢 𝑡 𝑖 𝑙 𝑖 𝑡 𝑦 𝑠 𝑇 , 𝑣 ( 𝑡 ) = 𝑚 𝑠 𝑒 𝑠 𝑇 𝑘 ∈ 𝐾 ( 𝑡 ) 𝒜 𝑠 𝐺 𝑘 ∩ 𝛥 ( 𝑇 , 𝑡 ) 𝒜 𝑠 𝐺 𝑘 𝑢 𝑡 𝑖 𝑙 𝑖 𝑡 𝑦 𝑠 𝐺 𝑘 , 𝑣 ( 𝑡 )
With
  • 𝑠 𝑇 the considred texture segment
  • 𝑣 ( 𝑡 ) the camera viewpoint at time 𝑡
  • 𝑚 𝑠 𝑒 𝑠 𝑇 the mean square error between the average color of the texture and 𝑠 𝑇
  • 𝐾 ( 𝑡 ) the set of segments received at time 𝑡
  • 𝒜 𝑠 𝐺 𝑘 the total area covered by 𝑠 𝐺 𝑘
  • 𝒜 𝑠 𝐺 𝑘 ∩ 𝛥 ( 𝑇 , 𝑡 ) the total area covered by texture 𝑇 in 𝑠 𝐺 𝑘

Streaming policy

Greedy

𝑠 * = a r g m a x 𝑠 ∈ 𝑆 𝑢 𝑡 𝑖 𝑙 𝑖 𝑡 𝑦 ( 𝑠 , 𝑣 ( 𝑡 ) ) 𝑠 𝑖 𝑧 𝑒 ( 𝑠 )

Streaming policy

Greedy predictive

𝑠 * = a r g m a x 𝑠 ∈ 𝑆 𝑢 𝑡 𝑖 𝑙 𝑖 𝑡 𝑦 ( 𝑠 , 𝑣 ( 𝑡 + 𝛿 𝑡 ) ) 𝑠 𝑖 𝑧 𝑒 ( 𝑠 )

Streaming policy

Proposed

𝑠 * = a r g m a x 𝑠 ∈ 𝑆 𝜒 𝑡 + 𝛿 𝑡 𝑢 𝑡 𝑖 𝑙 𝑖 𝑡 𝑦 ( 𝑠 , 𝑣 ( 𝑢 ) ) d 𝑢
A demo of DASH-3D

Evaluation

What we evaluate

Setup

Impact of the content preparation on the quality
Impact of the utility metric on the quality
Impact of the streaming policy on the quality

DASH-3D Interaction

Challenges for interaction

Screenshot of the desktop interface with bookmarks
Impact of bookmarks on discovery rate (MMSys 16)
Bookmark positions are known in advance
Screenshot of the color renderer
Difference between the default order and the optimized order
<AdaptationSet>
    <SupplementalProperty value="156.4909,1.6267,-146.2062,157.43106,1.5476,-146.5379" />
    <BaseURL>b1/</BaseURL>
    <Representation>
        <BaseURL>repr1/</BaseURL>
        <SegmentList>
            <SegmentURL media="thumbnail.jpg" />
            <SegmentURL media="order.json" />
        </SegmentList>
    </Representation>
</AdaptationSet>
A bookmark from the MPD
A demo of DASH-3D with bookmarks

Challenges for a mobile interface

Mobile inputs

Screenshot of the mobile interface with bookmarks

Evaluation

PSNR after a click on the bookmark path experiment
PSNR on top, percentage of users clicking on bottom, on the bookmark path experiment
PSNR after a click on the diamond experiment
PSNR on top, percentage of users clicking on bottom, on the diamond experiment

Contributions

Publications

Future work

Content preparation

Streaming

Client

Low resolution geometry
High resolution geometry
Simplistic textures
Detailed textures
Algorithm that determines the order of segments from a particular viewpoint
/