User:Brooke Vibber/Media codec alternatives

If we support MP4

If the decision to use transcoding to and from MP4 formats goes through, this will enable a few particular scenarios:

• no-install playback of video files on current versions of Windows and OS X running default browser (Internet Explorer, Safari)
• no-install playback of video files on all FOSS and "alternate" desktop browsers (Firefox, Chrome, Opera)
• no-install playback of video files on Android
• no-install playback of video files on iOS
  • All these would "just work" using either MP4 or WebM native support
• easy contribution of video files from desktop browser
• easy contribution of video files from mobile app
  • Most mobile devices and cameras record straight to an MP4-based format

If we don't support MP4

If we don't go with this proposal, what are our alternatives beyond simply not supporting video for some platforms?

• no-install playback of video files on current versions of Windows and OS X running default browser (Internet Explorer, Safari)
  • Difficult, not impossible: This can be achieved via #JavaScript playback, currently very experimental
  • Push vendors to support Theora/WebM
• no-install playback of video files on all FOSS and "alternate" browsers (Firefox, Chrome, Opera)
  • No problem. These browsers already play back WebM video.
• no-install playback of video files on Android
  • No problem. Android supports WebM natively in recentish versions.
• no-install playback of video files on iOS
  • Difficult, maybe not quite possible. #JavaScript playback is a possible route for the latest high-performance iOS devices, but many existing devices seem very slow at this, and we may only be able to get adequate performance by showing very low-resolution, poor-quality files.
  • Alternately, #Native app playback can do decoding much faster than JavaScript. This could be added to the iOS version of the Wikipedia app, which would be easier for users to install than desktop codecs. Leverage existing work for VLC for iOS, which has returned to the app store under more favorable licensing.
• easy contribution of video files from desktop browser
  • Dead in the water. JavaScript could be used to encode, but without being able to decode the existing file we may not be able to do much. Without server-side transcoding, additional client-side tools will be required as they are today to do the conversion.
• easy contribution of video files from mobile browser
  • Dead in the water. Note this applies to Android as well as iOS -- devices record to MP4-family formats by default, which is what their hardware codecs support natively.
• easy contribution of video files from mobile app
  • Difficult, not impossible: Transcoding could be done in an app, running an upload in the background if possible. #Native app contribution
• Power user tools for photographers?
  • For higher quality stuff see if Magic Lantern can be got to support WebM.

Possible work

If we're unable to decide to support MP4-family codecs for input or output on MediaWiki sites, continuing to push forward on multimedia support will require us to do some additional software development to make it easier for people to contribute and consume audio and video content in Wikimedia sites.

Note that the Wikimedia Foundation's Multimedia and Mobile teams would prefer to minimize such low-level work because we have limited resources! It would take time, effort, and money that could instead be spent on user-facing improvements. There is a real cost to going forward with these.

General limitations

Most mobile devices and desktop/laptop systems have some sort of integrated MP4-family codec acceleration, the exact details of which are hidden away by system drivers. This means that a pure-software codec implementation, even at the best of times, is likely to use more CPU time and energy to play back or encode video than using the default MP4-based formats.

This may be a serious problem for long-form playback or encoding, but is likely to be a manageable trade-off for shorter clips.

JavaScript playback

Wikimedia traditionally provided non-native audio/video playback using a Java applet ('Cortado') with a Java implementation of the Ogg Vorbis and Theora codecs. This is no longer usable on many systems, as Java applets are either unavailable (eg on iOS) or are so locked-down by default that it's hard to use them.

So, there's an opening for finding a new "no-install" playback component for people using browsers that don't support WebM natively...

There is very encouraging work going on in terms of alternate codecs using browsers' modern advanced JavaScript engines and graphics acceleration capabilities. See various posts about ORBX.js from Mozilla's Brendan Eich -- while that particular codec isn't (yet?) fully open it's a very encouraging example.

I've done some smaller-level research experiments using the Ogg Theora codec directly compiled into JavaScript.

This gives encouraging results on desktop machines. More serious work might attempt to replicate this with WebM's VP8 or newer VP9 codec, which provides better quality compression. Very serious work might see what kind of additional acceleration can be obtained using worker threads or WebGL shaders when available.

Complications:

• Performance: some systems are just too slow, especially in the mobile space
• Acceleration API availability: WebGL shaders can at minimum be used to perform colorspace conversion, and perhaps to actually perform some calculations. However, WebGL is not yet available on some major platforms (eg iOS)
• Sound synchronization: May not be trivial, especially on platforms that may require a Flash shim (Internet Explorer)
• Playback API, making it all work smoothly -- not likely to be trivial

Native app playback

I've done some experiments building the Ogg Theora codec for iOS and using it to perform native software decoding: https://wiki.xiph.org/IOS

This is an order of magnitude or so faster than JavaScript, and much more likely to be usable on mobile devices. There is also more direct access to device APIs -- vector instructions, GPU acceleration are more likely to be feasible here.

Complications:

• Sound synchronization: Not trivial to do right, but all the low-level tools are there to build on.
• Playback API, making it all work smoothly -- not likely to be trivial
  • These could be bypassed by using VLC for iOS, which has done this work already and is FOSS.
• Long tail of platforms: iOS work would be for iOS only, leaving smaller platforms like Windows Phone for more work another day

Native app contribution

With no server-side transcoding of MP4-family files recorded by typical mobile devices and cameras, contributing video files would also often require a client-side transcoding. This is not really feasible in a web browser, unless extended (eg Firefogg).

Note that uploading videos on the mobile web would be unwieldy anyway due to lack of background tasks and recovery -- so this is a case where people would be likely to want a native app anyway.

A native mobile app would also have access to the device's existing hardware-assisted MP4 codec, meaning that videos already recorded on the device as MP4 could be decoded, then encoded as WebM by the app and uploaded "transparently" with minimal user intervention.

However, we don't have a good idea what performance will look like on encoding. It'll be at least as slow as playback, and likely slower. If performed in parallel with the upload process, this might be relatively transparent to the user -- but we don't yet know whether network or transcoding is likely to be the limiting factor.

Complications:

• Performance characteristics unknown
• Transcoding always reduces picture quality