gfx/video.h
Video: a file played inside an editor, and frames from a camera -- both as Bitmaps, drawn through Graphics like everything else.
#include <sonore/gfx/video.h>- 3types
- Guides: The native editor
On this page
Design notes
Why frames come back as Bitmaps
The reference's video component is a native child window: a platform media view parented into the editor. That is the one thing an editor here cannot have. Everything an editor shows is rasterised into one bitmap and blitted (gfx/bitmap.h says why), so a native view would sit on top of the drawing with its own z-order, ignore clipping and scrolling, and miss every check the rasteriser makes possible. So the platform decodes, the decoded picture is copied into a Bitmap, and VideoComponent::paint() draws it -- clipped, scrolled, scaled and composited exactly like any image.
The cost is a copy per frame: 8 MB for 1080p, a few milliseconds a second at 30 frames. For a picture in a plugin editor -- a reference video to score against, a preview -- that is the right trade. A 4K player is not what this is for.
Threads, and update()
Decoding happens off the message thread. VideoComponent::update() picks up the newest frame and is the ONE call that touches the component from the playback side: call it from the editor's tick (sync(), or a MultiTimer slot). It repaints when a frame arrived and fires onPlaybackEnded and onErrorOccurred there -- so every callback runs on the message thread, and nothing ever repaints a component from another thread.
CameraDevice is the opposite and says so: its frames arrive on the capture thread, as the reference's do, because a camera consumer is usually a processing step (a still, a level meter, a recorder) rather than a widget. A listener that wants to show one copies it and repaints from its tick.
The soundtrack
MUTED BY DEFAULT, and opt-in with setAudioEnabled(true). A plugin's audio belongs to the host: it is mixed, delayed for latency compensation and bounced. A video component that opened an output of its own would play next to the session rather than in it -- not in the bounce, not under the plugin's own gain, and on whatever device the OS says. For picture-to- sound work the soundtrack is usually already on a track in the session, and the picture is what the plugin is for.
When enabled: on Windows the soundtrack is decoded by the same Media Foundation reader and played through the SDK's own WASAPI output (the one the standalone uses), at volume, while playing at 1x. At any other speed it is silent rather than chipmunked -- a time-stretcher is not something a video player should smuggle in. On macOS AVPlayer plays it, and does pitch- corrected varispeed itself. The video clock is the master on Windows and audio starts from it; over a long file the two drift by the sound card's clock error (tens of ppm, a few milliseconds a minute) until the next seek or play re-aligns them.
Platforms
Windows: Media Foundation. IMFSourceReader with video processing converts whatever the installed decoders accept (H.264, HEVC with its extension, VP9, MJPEG...) to RGB32; a worker thread clocks the frames. Cameras through MFEnumDeviceSources and the same reader. RUN by modules_test.cpp.
macOS: AVFoundation, through the Objective-C runtime like window_cocoa.h, with every C function resolved by dlsym -- so a project links no framework it did not already. AVPlayer + AVPlayerItemVideoOutput for playback, AVAssetReader for VideoReader, AVCaptureSession for cameras. COMPILED against tests/au_shim, never run: no machine here can.
Linux: neither. Every entry point reports "not available on this platform" rather than pretending; GStreamer or V4L2 would be the way in.
Types 3
A video file, decoded frame by frame, in presentation order -- what VideoComponent plays on Windows, and on its own what a thumbnail strip, an offline analysis or a test needs: every frame, as fast as it decodes, with no clock in the way.
One thread at a time. On Windows, COM is initialised on the opening thread if nobody had (single-threaded apartment, the UI's kind) and released by close() if close() runs on that same thread.
Frame size is the decoded picture's VISIBLE size, cropped to the display aperture a codec padded to its block size (1920x1088 -> 1920x1080). Pixel aspect ratio and rotation metadata are not applied: a phone video shot upright decodes on its side, as it is stored.
class VideoComponent : public Component
A video file, played inside an editor. See the header comment for the threading rule (call update() from the editor's tick) and the soundtrack (muted until setAudioEnabled(true)).
A camera, delivering frames as Bitmaps to listeners -- ON THE CAPTURE THREAD (see the header comment). The bitmap is valid for the duration of the call; a listener that keeps one copies it.
removeListener() waits for a delivery in progress, so once it returns the listener is not called again and may be destroyed.