Audio and sound effects

Audio plays one clip along the timeline. AudioManager schedules short cues, by name or in answer to another actor's events.

@motionactor/media has two ways to make sound. Audio is one clip that plays along the timeline: a voice-over, a music bed, room tone. AudioManager schedules short cues, such as a click per keystroke, from a named pack.

A clip

Audio plays its source through sourceTime, the seconds into the clip at each frame, like Video. A looped bed is one line:

import { function createScene<TSceneExports, const I extends SceneInputs = Readonly<Record<never, SceneInput>>, const L extends SceneLayers = Readonly<Record<never, SceneLayer<BackendRef<string>>>>>(script: SceneScript<I, L, TSceneExports>, options?: SceneDefinitionOptions<I, L>): SceneDefinition<I, L, TSceneExports>Declare a scene. Nothing compiles until a host builds it:
`scene.build({ viewport, fps, inputs })` runs the script as
`(ctx, inputs, viewport, layers)` and returns the compiled scene.
`ctx.spawn` places a root actor in the default layer; `layers.<key>.spawn`
places it in a declared one. `ctx.stage` stages another scene on this
scene's clock, and a staged scene may stage again.createScene } from "@motionactor/core";
import { class AudioAudio } from "@motionactor/media";
import { const audio: AssetType<"audio", "audio">audio } from "@motionactor/schema";

createScene<void, Readonly<Record<never, SceneInput>>, Readonly<Record<never, SceneLayer<BackendRef<string>>>>>(script: SceneScript<Readonly<Record<never, SceneInput>>, Readonly<Record<never, SceneLayer<BackendRef<string>>>>, void>, options?: SceneDefinitionOptions<Readonly<Record<never, SceneInput>>, Readonly<Record<never, SceneLayer<BackendRef<string>>>>> | undefined): SceneDefinition<...>Declare a scene. Nothing compiles until a host builds it:
`scene.build({ viewport, fps, inputs })` runs the script as
`(ctx, inputs, viewport, layers)` and returns the compiled scene.
`ctx.spawn` places a root actor in the default layer; `layers.<key>.spawn`
places it in a declared one. `ctx.stage` stages another scene on this
scene's clock, and a staged scene may stage again.createScene(function* (ctx: SceneContextctx) {
  ctx: SceneContextctx.CompileContext.spawn<Audio>(definition: ActorClass<Audio>, overrides: {
    content: {
        source: SignalInputValue<AssetRef<"audio">>;
    };
    volume?: SignalInputValue<number> | undefined;
    fade?: {
        in?: SignalInputValue<number> | undefined;
        out?: SignalInputValue<number> | undefined;
    } | undefined;
    timing?: {
        start?: SignalInputValue<number> | undefined;
        end?: SignalInputValue<number> | undefined;
    } | undefined;
    sourceTime?: SignalInputValue<number> | undefined;
    ... 18 more ...;
    parent?: undefined;
}, options?: {
    ...;
} | undefined): Audio (+1 overload)spawn(class AudioAudio, {
    content: {
    source: SignalInputValue<AssetRef<"audio">>;
}content: { source: SignalInputValue<AssetRef<"audio">>source: const audio: AssetType<"audio", "audio">audio.AssetType<"audio", "audio">.file(src: string, options?: {
    mimeType?: string;
}): AssetRef<"audio">file("music/bed.mp3") },
    volume?: SignalInputValue<number> | undefinedvolume: 0.6,
    fade?: {
    in?: SignalInputValue<number> | undefined;
    out?: SignalInputValue<number> | undefined;
} | undefinedfade: { in?: SignalInputValue<number> | undefinedin: 1, out?: SignalInputValue<number> | undefinedout: 2 },
  }).Audio.loop(params?: MediaLoopParams): voidloop();

  yield* ctx: SceneContextctx.CompileContext.wait(duration: Duration): Generator<Frame, void, unknown>wait(12);
}).SceneDefinition<Readonly<Record<never, SceneInput>>, Readonly<Record<never, SceneLayer<BackendRef<string>>>>, void>.build(args: SceneBuildInput<Readonly<Record<never, SceneInput>>>): CompiledScene<void, CompiledLayersOf<Readonly<Record<never, SceneLayer<BackendRef<string>>>>>>build({ viewport: SceneViewportviewport: { SceneViewport.width: numberwidth: 1920, SceneViewport.height: numberheight: 1080 }, fps: numberfps: 30 });

Member
`content.source`	The clip, an `audio` reference. `setSource(ref)` swaps it
`timing.start`, `timing.end`	The window of the clip to use, in seconds. `end: -1` is the clip’s end
`volume`, `muted`	Gain, and a switch. Both are signals, so `volume.tween(0, 2)` fades out
`fade.in`, `fade.out`	Seconds of ramp at the start and end of each play
`play({ from?, to?, rate? })`	Plays through the window. Takes that long
`loop({ window?, rate? })`	Loops the window from here on. Takes no time
`hold(seconds)`, `seekTo(time)`	Stays put, or jumps
`fastForward`, `rewind`, `rampTo`	Scrubs at a chosen speed, or eases between rates

Cues

A pack is a map from cue name to clip. defineSoundEffects keeps the names as a type, so a misspelt cue does not compile.

import { function defineSoundEffects<const TMap extends SoundEffectMap>(map: TMap): TMapdefineSoundEffects } from "@motionactor/media";
import { const audio: AssetType<"audio", "audio">audio } from "@motionactor/schema";

export const const keys: {
    readonly key: {
        readonly source: AssetRef<"audio">;
        readonly duration: 0.15;
        readonly volume: 0.3;
    };
    readonly enter: {
        readonly source: AssetRef<"audio">;
        readonly volume: 0.5;
    };
}keys = defineSoundEffects<{
    readonly key: {
        readonly source: AssetRef<"audio">;
        readonly duration: 0.15;
        readonly volume: 0.3;
    };
    readonly enter: {
        readonly source: AssetRef<"audio">;
        readonly volume: 0.5;
    };
}>(map: {
    readonly key: {
        readonly source: AssetRef<"audio">;
        readonly duration: 0.15;
        readonly volume: 0.3;
    };
    readonly enter: {
        readonly source: AssetRef<"audio">;
        readonly volume: 0.5;
    };
}): {
    readonly key: {
        readonly source: AssetRef<"audio">;
        readonly duration: 0.15;
        readonly volume: 0.3;
    };
    readonly enter: {
        readonly source: AssetRef<"audio">;
        readonly volume: 0.5;
    };
}defineSoundEffects({
  key: {
    readonly source: AssetRef<"audio">;
    readonly duration: 0.15;
    readonly volume: 0.3;
}key: {
    source: AssetRef<"audio">source: const audio: AssetType<"audio", "audio">audio.AssetType<"audio", "audio">.file(src: string, options?: {
    mimeType?: string;
}): AssetRef<"audio">file("sfx/key.mp3"),
    duration: 0.15duration: 0.15,
    volume: 0.3volume: 0.3,
  },
  enter: {
    readonly source: AssetRef<"audio">;
    readonly volume: 0.5;
}enter: { source: AssetRef<"audio">source: const audio: AssetType<"audio", "audio">audio.AssetType<"audio", "audio">.file(src: string, options?: {
    mimeType?: string;
}): AssetRef<"audio">file("sfx/enter.mp3"), volume: 0.5volume: 0.5 },
});

A cue may set duration in seconds (the whole clip when omitted), volume, playbackRate and loop.

Spawn an AudioManager, install the pack and play cues by name. play schedules the cue at the cursor and takes no time, so the script carries on while it sounds:

import { function createScene<TSceneExports, const I extends SceneInputs = Readonly<Record<never, SceneInput>>, const L extends SceneLayers = Readonly<Record<never, SceneLayer<BackendRef<string>>>>>(script: SceneScript<I, L, TSceneExports>, options?: SceneDefinitionOptions<I, L>): SceneDefinition<I, L, TSceneExports>Declare a scene. Nothing compiles until a host builds it:
`scene.build({ viewport, fps, inputs })` runs the script as
`(ctx, inputs, viewport, layers)` and returns the compiled scene.
`ctx.spawn` places a root actor in the default layer; `layers.<key>.spawn`
places it in a declared one. `ctx.stage` stages another scene on this
scene's clock, and a staged scene may stage again.createScene } from "@motionactor/core";
import { class AudioManagerAudioManager } from "@motionactor/media";
import { const keys: {
    readonly key: {
        readonly source: AssetRef<"audio">;
        readonly duration: 0.15;
        readonly volume: 0.3;
    };
    readonly enter: {
        readonly source: AssetRef<"audio">;
        readonly volume: 0.5;
    };
}keys } from "./sounds";

createScene<void, Readonly<Record<never, SceneInput>>, Readonly<Record<never, SceneLayer<BackendRef<string>>>>>(script: SceneScript<Readonly<Record<never, SceneInput>>, Readonly<Record<never, SceneLayer<BackendRef<string>>>>, void>, options?: SceneDefinitionOptions<Readonly<Record<never, SceneInput>>, Readonly<Record<never, SceneLayer<BackendRef<string>>>>> | undefined): SceneDefinition<...>Declare a scene. Nothing compiles until a host builds it:
`scene.build({ viewport, fps, inputs })` runs the script as
`(ctx, inputs, viewport, layers)` and returns the compiled scene.
`ctx.spawn` places a root actor in the default layer; `layers.<key>.spawn`
places it in a declared one. `ctx.stage` stages another scene on this
scene's clock, and a staged scene may stage again.createScene(function* (ctx: SceneContextctx) {
  const const sound: AudioManagersound = ctx: SceneContextctx.CompileContext.spawn<AudioManager>(definition: ActorClass<AudioManager>, overrides?: {
    enter?: undefined;
    enabled?: SignalInputValue<boolean> | undefined;
    track?: {
        cues?: SignalInputValue<AudioTrackCue[]> | undefined;
    } | undefined;
    playDefinition?: undefined;
    installSoundEffects?: undefined;
    bindSounds?: undefined;
    id?: undefined;
    cursor?: undefined;
    cursorSeconds?: undefined;
    sceneFps?: undefined;
    onInit?: undefined;
    onEnter?: undefined;
    onExit?: undefined;
    exit?: undefined;
    parent?: undefined;
} | undefined, options?: {
    ...;
} | undefined): AudioManager (+1 overload)spawn(class AudioManagerAudioManager);
  const const pack: InstalledSoundEffects<{
    readonly key: {
        readonly source: AssetRef<"audio">;
        readonly duration: 0.15;
        readonly volume: 0.3;
    };
    readonly enter: {
        readonly source: AssetRef<"audio">;
        readonly volume: 0.5;
    };
}>pack = const sound: AudioManagersound.AudioManager.installSoundEffects<{
    readonly key: {
        readonly source: AssetRef<"audio">;
        readonly duration: 0.15;
        readonly volume: 0.3;
    };
    readonly enter: {
        readonly source: AssetRef<"audio">;
        readonly volume: 0.5;
    };
}>({ map, }: {
    map: {
        readonly key: {
            readonly source: AssetRef<"audio">;
            readonly duration: 0.15;
            readonly volume: 0.3;
        };
        readonly enter: {
            readonly source: AssetRef<"audio">;
            readonly volume: 0.5;
        };
    };
}): InstalledSoundEffects<{
    readonly key: {
        readonly source: AssetRef<"audio">;
        readonly duration: 0.15;
        readonly volume: 0.3;
    };
    readonly enter: {
        readonly source: AssetRef<"audio">;
        readonly volume: 0.5;
    };
}>installSoundEffects({ map: {
    readonly key: {
        readonly source: AssetRef<"audio">;
        readonly duration: 0.15;
        readonly volume: 0.3;
    };
    readonly enter: {
        readonly source: AssetRef<"audio">;
        readonly volume: 0.5;
    };
}map: const keys: {
    readonly key: {
        readonly source: AssetRef<"audio">;
        readonly duration: 0.15;
        readonly volume: 0.3;
    };
    readonly enter: {
        readonly source: AssetRef<"audio">;
        readonly volume: 0.5;
    };
}keys });

  yield* const pack: InstalledSoundEffects<{
    readonly key: {
        readonly source: AssetRef<"audio">;
        readonly duration: 0.15;
        readonly volume: 0.3;
    };
    readonly enter: {
        readonly source: AssetRef<"audio">;
        readonly volume: 0.5;
    };
}>pack.InstalledSoundEffects<{ readonly key: { readonly source: AssetRef<"audio">; readonly duration: 0.15; readonly volume: 0.3; }; readonly enter: { readonly source: AssetRef<"audio">; readonly volume: 0.5; }; }>.play(cue: "key" | "enter", override?: Partial<AudioCueDefinition>): Generator<number, void, unknown>play("key");
  yield* ctx: SceneContextctx.CompileContext.wait(duration: Duration): Generator<Frame, void, unknown>wait(0.4);
  yield* const pack: InstalledSoundEffects<{
    readonly key: {
        readonly source: AssetRef<"audio">;
        readonly duration: 0.15;
        readonly volume: 0.3;
    };
    readonly enter: {
        readonly source: AssetRef<"audio">;
        readonly volume: 0.5;
    };
}>pack.InstalledSoundEffects<{ readonly key: { readonly source: AssetRef<"audio">; readonly duration: 0.15; readonly volume: 0.3; }; readonly enter: { readonly source: AssetRef<"audio">; readonly volume: 0.5; }; }>.play(cue: "key" | "enter", override?: Partial<AudioCueDefinition>): Generator<number, void, unknown>play("enter", { volume?: number | undefinedvolume: 1 });
}).SceneDefinition<Readonly<Record<never, SceneInput>>, Readonly<Record<never, SceneLayer<BackendRef<string>>>>, void>.build(args: SceneBuildInput<Readonly<Record<never, SceneInput>>>): CompiledScene<void, CompiledLayersOf<Readonly<Record<never, SceneLayer<BackendRef<string>>>>>>build({ viewport: SceneViewportviewport: { SceneViewport.width: numberwidth: 1920, SceneViewport.height: numberheight: 1080 }, fps: numberfps: 30 });

The second argument overrides the pack’s values for that one cue.

mergeSoundEffects(a, b) joins packs. @motionactor/remotion exports remotionSoundEffects, a pack of Remotion’s stock effects: whoosh, whip, pageTurn, uiSwitch, mouseClick, shutterModern, shutterOld, ding, bruh, vineBoom and windowsXpError.

Sounds from events

A keyboard actor that plays its own clicks is stuck with them. Have it emit an event instead, and let the scene choose the sound:

import { class Actor<TEvents extends ActorEventMap = {}>Actor, function createScene<TSceneExports, const I extends SceneInputs = Readonly<Record<never, SceneInput>>, const L extends SceneLayers = Readonly<Record<never, SceneLayer<BackendRef<string>>>>>(script: SceneScript<I, L, TSceneExports>, options?: SceneDefinitionOptions<I, L>): SceneDefinition<I, L, TSceneExports>Declare a scene. Nothing compiles until a host builds it:
`scene.build({ viewport, fps, inputs })` runs the script as
`(ctx, inputs, viewport, layers)` and returns the compiled scene.
`ctx.spawn` places a root actor in the default layer; `layers.<key>.spawn`
places it in a declared one. `ctx.stage` stages another scene on this
scene's clock, and a staged scene may stage again.createScene, function signal<T>(): RequiredSignal<T> (+1 overload)
module signalA timeline value on an actor, and an input to every spawn of it.
Without an initial value the signal is required: `ctx.spawn` must supply it.signal } from "@motionactor/core";
import { class AudioManagerAudioManager } from "@motionactor/media";
import { const keys: {
    readonly key: {
        readonly source: AssetRef<"audio">;
        readonly duration: 0.15;
        readonly volume: 0.3;
    };
    readonly enter: {
        readonly source: AssetRef<"audio">;
        readonly volume: 0.5;
    };
}keys } from "./sounds";

class class KeyboardKeyboard extends class Actor<TEvents extends ActorEventMap = {}>Actor<{ pressed: {
    key: string;
}pressed: { key: stringkey: string } }> {
  Keyboard.text: ActorFieldSignal<string>text = signal<string>(initial: string): ActorFieldSignal<string> (+1 overload)A timeline value on an actor, and an input to every spawn of it.
Without an initial value the signal is required: `ctx.spawn` must supply it.signal("");

  *Keyboard.type(word: string): Generator<number, void, unknown>type(word: stringword: string) {
    for (const const key: stringkey of word: stringword) {
      this.Keyboard.text: ActorFieldSignal<string>text.LiveSignal<string>.set(value: string): voidSet value instantly at current cursor (no yield needed)set(this.Keyboard.text: () => stringtext() + const key: stringkey);
      yield* this.Actor<{ pressed: { key: string; }; }>.emit: ActorEmitter<{
    pressed: {
        key: string;
    };
}>emit.pressed: (payload: {
    key: string;
}) => Generator<Frame, void, unknown>pressed({ key: stringkey });
      yield* this.Actor<{ pressed: { key: string; }; }>.runtime: ActorRuntimeOrchestration<this>runtime.ActorRuntimeOrchestration<this>.wait(duration: Duration): Generator<Frame, void, unknown>wait(0.1);
    }
    yield* this.Actor<{ pressed: { key: string; }; }>.emit: ActorEmitter<{
    pressed: {
        key: string;
    };
}>emit.pressed: (payload: {
    key: string;
}) => Generator<Frame, void, unknown>pressed({ key: stringkey: "Enter" });
  }
}

createScene<void, Readonly<Record<never, SceneInput>>, Readonly<Record<never, SceneLayer<BackendRef<string>>>>>(script: SceneScript<Readonly<Record<never, SceneInput>>, Readonly<Record<never, SceneLayer<BackendRef<string>>>>, void>, options?: SceneDefinitionOptions<Readonly<Record<never, SceneInput>>, Readonly<Record<never, SceneLayer<BackendRef<string>>>>> | undefined): SceneDefinition<...>Declare a scene. Nothing compiles until a host builds it:
`scene.build({ viewport, fps, inputs })` runs the script as
`(ctx, inputs, viewport, layers)` and returns the compiled scene.
`ctx.spawn` places a root actor in the default layer; `layers.<key>.spawn`
places it in a declared one. `ctx.stage` stages another scene on this
scene's clock, and a staged scene may stage again.createScene(function* (ctx: SceneContextctx) {
  const const sound: AudioManagersound = ctx: SceneContextctx.CompileContext.spawn<AudioManager>(definition: ActorClass<AudioManager>, overrides?: {
    enter?: undefined;
    id?: undefined;
    cursor?: undefined;
    cursorSeconds?: undefined;
    sceneFps?: undefined;
    onInit?: undefined;
    onEnter?: undefined;
    onExit?: undefined;
    exit?: undefined;
    parent?: undefined;
    enabled?: SignalInputValue<boolean> | undefined;
    track?: {
        cues?: SignalInputValue<AudioTrackCue[]> | undefined;
    } | undefined;
    playDefinition?: undefined;
    installSoundEffects?: undefined;
    bindSounds?: undefined;
} | undefined, options?: {
    ...;
} | undefined): AudioManager (+1 overload)spawn(class AudioManagerAudioManager);
  const const pack: InstalledSoundEffects<{
    readonly key: {
        readonly source: AssetRef<"audio">;
        readonly duration: 0.15;
        readonly volume: 0.3;
    };
    readonly enter: {
        readonly source: AssetRef<"audio">;
        readonly volume: 0.5;
    };
}>pack = const sound: AudioManagersound.AudioManager.installSoundEffects<{
    readonly key: {
        readonly source: AssetRef<"audio">;
        readonly duration: 0.15;
        readonly volume: 0.3;
    };
    readonly enter: {
        readonly source: AssetRef<"audio">;
        readonly volume: 0.5;
    };
}>({ map, }: {
    map: {
        readonly key: {
            readonly source: AssetRef<"audio">;
            readonly duration: 0.15;
            readonly volume: 0.3;
        };
        readonly enter: {
            readonly source: AssetRef<"audio">;
            readonly volume: 0.5;
        };
    };
}): InstalledSoundEffects<{
    readonly key: {
        readonly source: AssetRef<"audio">;
        readonly duration: 0.15;
        readonly volume: 0.3;
    };
    readonly enter: {
        readonly source: AssetRef<"audio">;
        readonly volume: 0.5;
    };
}>installSoundEffects({ map: {
    readonly key: {
        readonly source: AssetRef<"audio">;
        readonly duration: 0.15;
        readonly volume: 0.3;
    };
    readonly enter: {
        readonly source: AssetRef<"audio">;
        readonly volume: 0.5;
    };
}map: const keys: {
    readonly key: {
        readonly source: AssetRef<"audio">;
        readonly duration: 0.15;
        readonly volume: 0.3;
    };
    readonly enter: {
        readonly source: AssetRef<"audio">;
        readonly volume: 0.5;
    };
}keys });
  const const keyboard: Keyboardkeyboard = ctx: SceneContextctx.CompileContext.spawn<Keyboard>(definition: ActorClass<Keyboard>, overrides?: {
    enter?: undefined;
    type?: undefined;
    text?: SignalInputValue<string> | undefined;
    id?: undefined;
    cursor?: undefined;
    cursorSeconds?: undefined;
    sceneFps?: undefined;
    onInit?: undefined;
    onEnter?: undefined;
    onExit?: undefined;
    exit?: undefined;
    parent?: undefined;
} | undefined, options?: {
    id?: string;
} | undefined): Keyboard (+1 overload)spawn(class KeyboardKeyboard);

  const sound: AudioManagersound.AudioManager.bindSounds<{
    pressed: {
        key: string;
    };
}, {
    readonly key: {
        readonly source: AssetRef<"audio">;
        readonly duration: 0.15;
        readonly volume: 0.3;
    };
    readonly enter: {
        readonly source: AssetRef<"audio">;
        readonly volume: 0.5;
    };
}>(source: {
    id: string;
    emit: ActorEmitter<{
        pressed: {
            key: string;
        };
    }>;
}, pack: InstalledSoundEffects<{
    readonly key: {
        readonly source: AssetRef<"audio">;
        readonly duration: 0.15;
        readonly volume: 0.3;
    };
    readonly enter: {
        readonly source: AssetRef<"audio">;
        readonly volume: 0.5;
    };
}>, bindings: SoundBindings<...>): voidAttach sounds to another actor's semantic events. The actor stays
sound-agnostic — the event→cue mapping is a composition-level choice.bindSounds(const keyboard: Keyboardkeyboard, const pack: InstalledSoundEffects<{
    readonly key: {
        readonly source: AssetRef<"audio">;
        readonly duration: 0.15;
        readonly volume: 0.3;
    };
    readonly enter: {
        readonly source: AssetRef<"audio">;
        readonly volume: 0.5;
    };
}>pack, {
    pressed?: "key" | "enter" | ((payload: {
    key: string;
}) => SoundBindingResult<{
    readonly key: {
        readonly source: AssetRef<"audio">;
        readonly duration: 0.15;
        readonly volume: 0.3;
    };
    readonly enter: {
        readonly source: AssetRef<"audio">;
        readonly volume: 0.5;
    };
}>) | undefinedpressed: ({ key: stringkey }) => (key: stringkey === "Enter" ? "enter" : "key"),
  });

  yield* const keyboard: Keyboardkeyboard.Keyboard.type(word: string): Generator<number, void, unknown>type("hi"); // key at 0s, key at 0.1s, enter at 0.2s
}).SceneDefinition<Readonly<Record<never, SceneInput>>, Readonly<Record<never, SceneLayer<BackendRef<string>>>>, void>.build(args: SceneBuildInput<Readonly<Record<never, SceneInput>>>): CompiledScene<void, CompiledLayersOf<Readonly<Record<never, SceneLayer<BackendRef<string>>>>>>build({ viewport: SceneViewportviewport: { SceneViewport.width: numberwidth: 1920, SceneViewport.height: numberheight: 1080 }, fps: numberfps: 30 });

A binding is a cue name, or a function of the event’s payload that returns a cue name, { cue, ...overrides }, or null for silence. Both the event names and the cue names are checked against their types. bindSounds is a subscribe underneath, so use subscribe directly when an event should do more than play a cue.

Muting

sound.enabled.set(false) stops the manager scheduling cues from that point of the script. The check happens when play runs or when the event fires, so cues already scheduled still sound and a key pressed while it is off stays silent after it is turned back on.

Where it plays

Previews play Audio clips and scheduled cues alongside the picture, through the host’s media player. Exported video files have no sound yet. See Capture and export.