User:Brooke Vibber/Media handler and renderer thoughts

Some thoughts on media handlers

Key problem to solution mappings

At config time:

• tricky to add new file types from an extension if they're not in the core mime.types
  • add a registry of file extension -> mime type mapping overrides

At upload time:

• existing upload-time validation tends to be overzealous about .zip-based formats or otherwise gets confused
  • add an upload-time validation function in the Handler

At transform & parser time:

• srcset is inefficiently handled in Linker
  • allow handlers to do their own srcset sub-transforms
  • add suitable entries to imageinfo for additional resolutions for InstantCommons
• need reliable way to attach JS modules for file display progressive enhancement
  • for pre-rendered:
    • have a way to transfer list of modules/styles to load from a MediaTransformOutput to a ParserOutput when adding the rendering
  • for dynamic rendered:
    • hang tiny stub modules in on all page views, that hook the on-wikitext-added event and check if they need to load more
• Rendering remote files of locally-unknown type
  • add common infrastructure for iframe embedding like TMH has
  • expose iframe embed URL in imageinfo response; client side can use it if no local handler available
• Render files with backing type X but display type Y
  • mark photo spheres, panoramas etc with additional metadata that gets transferred across remote repos, so correct rendering gets picked
  • same can handle things like rendering a PNG/TIFF source to JPEG thumbnails vs PNG depending on photo vs diagram

At MMV popup activation time:

• Rendering remote files of locally-unknown type
  • Let MMV handle zoom of remote types it doesn't know locally via the iframe embedding

Background: current state of media bits

• mime type is primary identifier for picking a Handler class
• extension -> mime type and other -> mime type mappings are hidden and very hard to extend
• upload-time validation of file type & safety is hidden and very hard to extend
• File class encapsulates for a file from a given source repo:
  • how to get the Handler instance
  • wraps the Handler class's metadata & transform processes
    • caches metadata via DB
    • over InstantCommons, ForeignAPIFile speaks to repo over HTTP... up to once per rendering (cached)
• Handler classes encapsulate for a given file type:
  • how to extract metadata from the file
  • whether rendering to a thumbnail is possible
  • validating thumbnail parameters from wikitext or other source
  • performing some kind of backend rendering
  • returning a suitable Thumbnail subclass instance with a rendering
• MediaTransformOutput classes encapsulate for a given file type:
  • returning some HTML to actually show in wiki page renderings
  • returning direct references for flat 2d thumbnail image
• Linker class encapsulates for all types:
  • translating parameter strings into arrays to pass into the handler
  • asking the Handler for thumbnail renderings at 1x, 1.5x, 2x
    • combining the urls from the 1.5x and 2x renderings into the 1x MediaTransformOutput instance
    • note this is hella inefficient over InstantCommons -- 3 separate imageinfo requests get run
  • shoving the rendering into a pretty frame
• MMV extension encapsulates for some types:
  • how to copy the thumbnail into a big pretty viewer, then ask for a high-res thumbnail and switch to it when loaded
  • is based on file extension of the thumbnail, not on mime type
• ad-hoc JavaScript modules attached to particular types may handle:
  • progressive enhancement of the displayed thumbnail
    • TMH's replacement of bare <video>/<audio> or 'popup thumbnail' with fancy player widget
  • progressive enhancement of the MMV high-res view
    • 3d STL extension's in-progress plugin
    • currently MMV displayable-ness and plugin mapping are based on file extension of the thumbnail, which seems fragile
  • currently TMH jumps through some hoops to mark ParserOutput for later loading of the modules
  • no clear way to handle adding the modules dynamically when dynamically adding source through a runtime preview, etc

Ok that was kinda scary, huh?