# Gapless and instant URI switching in playback elements

This document explains the various changes and improvements to the playback elements in order to support gapless playback and instantaneous URI switching.

Last Update: November 23rd 2022

# Background

The new `playbin3` element and its components (`uridecodebin3`, `decodebin3` and `urisourcebin`) are replacements to the legacy `playbin2` and `decodebin2` elements.

The goals of these new elements are to both allow new use-cases and improve performance (lower memory/cpu/io usage, lower latency). One of the key principles is also to re-use elements as much as possible. For example, when switching audio tracks the decoder can be re-used (if compatible).

The separation of roles was also more clearly split up into various new elements (from lowest-level to highest-level):

- `urisourcebin` handles choosing the right source elements for the given URI, and handles buffering (via `queue2`) if needed (for network sources for example).

- `parsebin` takes an input stream and figures out which demuxer, parsers and/or payloaders are needed to provide timed elementary streams.

- `decodebin3` internally uses `parsebin` to handle any input stream and will handle the decoding, inter-stream muxing interleave, stream selection and switching. It can also handle multiple inputs (such as an audio/video file and a separate subtitle file).

- `uridecodebin3` wraps `urisourcebin`s and `decodebin3` for any use-cases where one wishes to have decoded streams from given URIs.

- Finally `playbin3` combines `uridecodebin3` and `playsink` for providing a high-level convenience pipeline for playing back content.

This design has received many improvements over time:

- `decodebin3` was able to detect input changes (caps changes) and reconfigure the associated `parsebin` if incompatible. This allows use-cases where upstream is an HLS/DASH stream where codecs are different across bitrates. The playback remains seamless if the decoders are compatible.

- `decodebin3` was able to bypass the usage of `parsebin` altogether if the incoming stream is pull-based, provides a `GstStreamCollection` and is compatible with the decoders or output caps.

- `urisourcebin` can handle sources that handle buffering internally, avoiding dual-buffering.

- A new core query `GST_QUERY_SELECTABLE` was added so that (source) elements could notify `decodebin3` that they can handle stream selection and switching themselves.

- Several improvements were made to `playbin3` to allow complete stream type changes (such as going from playing audio+video to just audio or just video, and back), This allows temporarily disabling whole chains of elements when not needed.

# Limitation/Issue

Two limitations existed though, which are both related:

- Changing URI required bringing `playbin3` (and all contained elements) down to `GST_STATE_READY`, setting the uri, and then bringing all elements back to `GST_STATE_PAUSED`.

  - This meant that all elements contained within were either discarded (decoders, demuxers, parsers, sources, ...) or reset (sinks)... despite potentially being 100% compatible (ex: going from h264/aac to h264/aac).

- Gapless playback (i.e. automatically switching from one source to another, and removing any potential gap in the data arriving to the sinks) was implemented by pre-rolling a full `uridecodebin3` for the next item to play and switching the inputs to `playsink` when the original `uridecodebin3` was EOS.

  - This meant that none of the existing elements (demuxers, parsers, decoders, ..) contained in the original `uridecodebin3` were re-used.

Those two use-cases are the same thing: We want to change the URI (i.e. `urisourcebin`) but re-use as much as possible of existing elements (i.e. `decodebin3` and `playsink`). The only difference between the two use-cases is that changing URI should happen instantaneously in the first case, whereas in the second case it happens when the initial source is done (EOS).

Fixing this will allow:

- Reducing memory and cpu usage (no duplicate elements)

- Lowering latency (no longer re-instantiate/reconfigure elements and re-use compatible ones as fast as possible).

Another issue which is related, is figuring out the optimal time at which the next item should be prepared so that it has enough data to playback immediately:

- This shouldn't be too early, some URIs expire after a given time, or the user might change their mind in between
- This shouldn't be too late, otherwise we risk not having enough data to playback seamlessly.

# Changes

## parsebin in urisourcebin

In order to figure out the optimal time at which a switch should happen (i.e. a given amount of "time" before the end of the previous play entry), this can only be done on "timed" data (i.e. parsed elementary streams).

There is therefore a new option on `urisourcebin` : `parse-streams`, which if set to `TRUE` (non-default) will add a `parsebin` (if and where needed) so that `urisourcebin` only outputs elementary streams. A `multiqueue` will also be present to handle any interleave present (i.e. only queue up what is needed to offer coherent streams downstream).

If buffering is activated on `urisourcebin`, the `multiqueue` present after the `parsebin` will be configured in order to handle it (and post the appropriate buffering messages).

This offers the following benefits:

- `about-to-finish` can be emitted by `urisourcebin` as soon as `EOS` enters those `multiqueue`, which will be more precise than the previous usage (before `queue2` on non-timed data)

- buffering is much closer to the actual buffering amount (in time) which is specified on the properties.

- ALL scheduling downstream of `urisourcebin` is push-based, removing a lot of issues when trying to change scheduling modes (push vs pull) dynamically.

The `parse-streams` property is set to `TRUE` when used in `uridecodebin3`

## Only use a single uridecodebin3 in playbin3

Only a single `uridecodebin3` is in use in `playbin3` and the source pads it provides are directly linked to `playsink`.

There can only be at most one stream of each stream type (audio, video, text) on the output side of `uridecodebin3`. The exception to this is if the user/application configured a specific multi-sinkpad combiner element for a given stream type, in which case all streams of that given stream type are linked to that.

All uri-related properties are forwarded directly to `uridecodebin3`, which will handle switching the sources to the single `decodebin3` it contains.

## uridecodebin3 URI and source handling

The URI for a given entry are handled in a `GstPlayItem` structure which controls (via intermediary structures):

- The `urisourcebin` associated with the specified URI (and optional subtitle URI)

- The pads provided by those sources, and which states they are in (eos, blocked, ...) and the associated GstStream (if present)

- The buffering messages posted by those sources.

At any given point there is:

- A `input_play_item`, which is the play item currently feeding data into `decodebin3`

- A `output_play_itm`, which is the play item currently being outputted by `decodebin3`

Most of the time those two will be the same. But when switching play items (going from one URI to another, whether gapless or not) this switch will happen asynchronously.

## Switching inputs to decodebin3

The high-level goal is to add to `uridecodebin3` the capability of being able to change `GstPlayItem` with the same `decodebin3` either:

- When the previous `GstPlayItem` has finished and there is a pending next `GstPlayItem`. This is the "gapless" scenario.

- Or immediately switch to the given `GstPlayItem` without having to change state. This is the "instantaneous URI switch" scenario.

For this, the following points need to be solved:

1. both scenarios: Add a way for "next" `GstPlayItem` to be pre-rolled
2. gapless: Determining when the switch can happen
3. instant-uri: pre-roll next `GstPlayItem` and flush downstream (to make the switch as quick as possile)
4. both scenarios: Do the actual switch

### pre-rolling play items

In order to be able to re-use the same decoders (within `decodebin3`) as much as possible from the outside, we need to ensure that we feed the ideal "replacement" stream to the same `decodebin3` sink pad.

For example, if we are switching from an audio+video HLS source to another audio+video DASH source, we want to make sure we link the new `urisourcebin` source pad providing video to the `decodebin3` pad that was previously consuming the old video stream.

In order to do this, the `urisourcebin` we wish to switch to needs to be pre-rolled (set to PAUSED, new pads are set to be blocked, and we wait for a buffer/GAP to arrive on at least one of the pads).

At that point we will know the streams which are present in the new and old `urisourcebin`s and can unlink/relink compatible pads. If new sink pads are required they will be requested, and if old pads are no longer needed (for example switching from two streams to a single one) they will be removed.

> Note: Doing this also has the benefit that "replacing" the inputs to `decodebin3` are done from a new streaming thread, and not the old `urisourcebin` streaming thread which could cause deadlocks.

> Note: This "waiting" is only done when "switching", i.e. on sources which aren't in the current input play item. If the pads are from the current play entry they are linked/unlinked as soon as they are added/removed.

The moment at which the next play item is pre-rolled is done:

- When the current play item has posted `about-to-finish` and the user/application has set a new play item.

- When a new play item has been set and the `instant-uri` property has been set to TRUE.

When a play item is pre-rolled, it is marked as "active". There can only be one "active" play item in addition to the input play item.

### gapless: determining when the switch can happen

For gapless use-cases, we want to know the earliest time we can switch from one play item to another.

Since all streams coming from `urisourcebin parse-streams=True` are push-based, this is when the last EOS has been pushed through all pads of the source.

### Instantaneous URI switching

In order to be able to switch URI as soon as possible while re-using as many existing elements as possible, there is a new `instant-uri` boolean property on `uridecodebin3`/`playbin3`. The default value is FALSE.

If it is set to TRUE, the following happens whenever the `uri` property is set:

- On all pads of the current input play item:

  - `FLUSH_START` is sent to the downstream peer pads
  - The pad is made blocking
  - The pad is marked as EOS (i.e. as if EOS had been seen)

- And then again on all pads:

  - `FLUSH_STOP` is sent to the downstream peer pads

- Finally the new play item for the new URI is activated (pre-rolled).

  - Once it is pre-rolled it will switch over

This ensures all downstream elements are kept and are ready to receive the new data.

### Switching play items

Switching play items requires special attention since it needs to be done "atomically". We need to ensure it is done by a single thread. This is done by having a lock (`play_items_lock`) which is taken whenever we need to modify the list of play items and which play item is the current input/output.

We need to ensure the streaming thread(s) that were previously used are stopped. Since we are only dealing with push-based sources this is simple: we wait for the moment EOS is pushed on the last pad of the play item.

Another important consideration is that we need to ensure the thread that does the switch is not the previous streaming thread (it needs to be stopped).

In order to solve those issues, the actual replacement of the inputs will always happen from the streaming thread of the new play item, i.e. the one we wish to make the current input. This is done in a pad block probe on the new item source pad. Whenever a buffer (or GAP event) is received, we check whether we can switch:

- If the current input play item is completely EOS, the switch can happen immediately. This will always be the case in instant-uri scenario and if the current input play item is pull-based.

- If the current input play item is not completely EOS, the probe waits on the `GCond input_source_drained`. This is the case that will commonly happen in gapless push-based scenarios, since we are waiting for the current input play item to be finished.

Once the switch can happen, we unlink all pads from `decodebin3` and attempt to match compatible new source pads from `urisourcebin` to `decodebin3`. If new sink pads are required they are requested, and if some sink pads are no longer needed or do not match they are released.

Once all pads are linked, the new play item is set as the current play item.

## uridecodebin3 handles `about-to-finish` signalling

In regards to gapless playback, the API does not change. Users are still expected to listen to `about-to-finish` and set the next URI to play back.

One thing that needs to be taken care of is making sure we don't emit `about-to-finish` for play items which aren't currently used. This would end up in a situation where `about-to-finish` would cause a snowball effect of pending play items emitting it, which would cause a future entry to be created, prerolled and emitting it again.

For that reason, if a play item emits that signal but isn't the input or output play item, then it is just stored and not propagated upstream. When that play entry becomes the new input entry it will be propagated.

The results of the search are