Build your own video platform

A guide for a developer who wants to ship a video product without building a renderer.

Everything below was checked against the live API before it was written. Field names, credit costs, scope names and status strings are what the server actually returns, not what a spec says it should. Where something is not built yet, it says so.

Base URL: https://api.vidmoat.com/v1. (/api/v1 resolves to the same place, but use /v1: it is the supported path, and it avoids a redirect, which is where Authorization headers get dropped.)

1. What Vidmoat gives you, and what it does not#

You get#

• A document format. A project is one JSON document: settings, tracks, clips. Every clip field that affects the output is in it: transforms, crops, keyframes, colour grade, effects, masks, audio DSP, transitions. It is readable, diffable, and you can hold it in your own database if you want to.
• A command reducer. ~80 named operations (addClip, sliceClip, addCaptions, cutRanges, autoDuck, addKeyframe, applyMotionPreset, addAdjustmentLayer, …) that take a document and produce a new one. Published verbatim at GET /v1/schema/commands.
• A render farm. POST /v1/renders queues a job; a worker composites the document in Chromium, mixes audio through ffmpeg, and hands you an MP4/WebM/GIF URL. You never install ffmpeg.
• Generative media. Text-to-image, text-to-video, text-to-speech, AI stickers, behind a credit meter, with refunds when the provider fails. xAI Grok is the default; images can also route to OpenAI or Vertex, speech to OpenAI, and video to HappyHorse or Wan.
• Transcription. Word-level timings in exactly the shape the addCaptions command consumes. Plus a free mode that distributes a script you already have across a known duration, with no provider call at all.
• Preview. Three representations: cheap JSON metadata + layout lint, a single rendered JPEG frame, and the whole interactive composition as HTML you can scrub client-side at zero server cost.
• A planner. POST /v1/ai/agent turns plain English into a validated command program and applies it.

You bring#

The product. The UI. The users. The onboarding, the pricing page, the retention loop, the reason anyone opens your app instead of CapCut.

What is not there#

Say these out loud before you design around them:

• Render webhooks are available in the developer console. Subscribe to render.completed and render.failed (plus credit, app and Telegram bot events), verify signatures and inspect delivery logs at /developer/webhooks. Public /v1/webhooks management is not shipped. Poll /v1/ai/jobs/{id} for video generation jobs and keep bounded render polling as a recovery path. See the webhook guide.
• ai.analyze is not exposed. The scope exists. The analysis (shot boundaries, loudness, voice activity) runs internally. There is no POST /v1/ai/analyze route. Do not build a cut-decision engine that assumes you can ask Vidmoat what is in a frame.
• No idempotency keys. There is no Idempotency-Key header. Commands are not idempotent: posting addClip twice gives you two clips. §8 explains what to do instead.
• No chunked upload. POST /v1/media/uploads buffers the whole file in memory and refuses anything over 32 MB. Bigger files go through POST /v1/media/import, which streams from a URL you host (up to your plan's upload cap or 512 MB, whichever is smaller, with a 120-second timeout).
• No browser-facing API. An API key is a server credential. It must never reach a browser, and CORS on api.vidmoat.com is configured for Vidmoat's own hosts, not for arbitrary third-party origins. Your web UI talks to your server; your server talks to Vidmoat. §4 shows the one place this matters architecturally.
• No frame-exact local scrubbing. The interactive preview is a real browser playing your media over the network, driven by the page's own clock. It is excellent for composition and timing decisions. It is not a broadcast monitor. If your users are colour-grading 4K ProRes and need guaranteed frame accuracy with no network, Vidmoat is the wrong choice: you want a local NLE, and no amount of API is going to fix that.
• No social publishing. No scope grants posting to Vidmoat Social, and none will.
• Browser recorder, inpainting and Flows are deliberately not in v1. Each would let one API caller starve everyone else on the box.

2. The mental model#

This is the one section worth reading twice, because everything else follows from it.

A project is a JSON document.

{
  "version": 2,
  "settings": { "name": "…", "aspectRatio": "9:16", "width": 1080, "height": 1920, "fps": 30, "background": "#000000" },
  "tracks": [ { "id": 0, "label": "V1", "type": "video", … }, … ],
  "clips":  [ { "id": "…", "type": "video", "start": 0, "duration": 6, "trackIndex": 0, "src": "…", … }, … ],
  "markers": [ … ]
}

Nothing is derived and stored. The timeline's duration is max(clip.start + clip.duration), computed on read. Compositing order is the position of a clip's track in the tracks array: a clip whose track is listed later draws on top. That is a real gotcha: trackIndex is a track id, not a layer number, and the two only coincide in the default document.

Every edit is a command through one reducer.

{ "op": "addClip", "type": "video", "src": "…", "start": 0, "duration": 6, "trackIndex": 0 }

One reducer is the only thing in the system that mutates a document. It generates ids, allowlists patch fields, normalises clips, resolves collisions, and applies the auto-fixes that model output actually needs.

The same reducer serves the editor, the agent and the API. The editor's window.vidmoat.dispatch calls it. The chat agent calls it. The MCP server's edit_project calls it. POST /v1/projects/{id}/commands calls it directly, as a library import, not over HTTP. PATCH /v1/projects/{id} calls it, even for a rename, because Project.name on the database row and document.settings.name are two copies of one fact and setProjectSettings is the only thing that keeps them equal.

Why this matters to you. An API edit and a hand edit cannot diverge. If your user opens the same project in Vidmoat's own editor, they see exactly what your API calls produced, with the same ids, the same normalisation and the same semantics. There is no "API version of the document" and no import/export step where fidelity leaks. That is the guarantee you are buying, and it is the reason a third party can build a real editor on this at all: you are not driving a sanitised subset, you are driving the actual thing.

Two consequences to design around:

1. Read the document you are about to mutate. GET /v1/projects/{id}?view=document returns it verbatim: the same JSON the reducer consumes. Nothing is reshaped, so your read and your write agree about field names.
2. The reducer may not do exactly what you asked. An overlapping clip is pushed to the next track index, and the track is created if it is missing. addClip defaults mediaDuration to duration if you omit it, which quietly breaks waveform windowing and looping. Always read the returned results[].data.clipId rather than assuming, and pass mediaDuration explicitly when you pass mediaStart.

3. Hello, render#

3.1 Get a key#

At https://developer.vidmoat.com/apps: create an app, accept the developer agreement, mint a key with the scopes you need.

• Key format: vmk_live_<48 hex> or vmk_test_<48 hex>. Older unprefixed vmk_<hex> keys still work.
• Live keys need three things: the developer agreement accepted, the app approved in review, and a plan with api_access (Studio, Team or Enterprise). Test keys are free on every plan, including free. Whether a key is test or live is fixed by how it was minted. Max 10 active keys per app.
• Scopes are chosen at mint time and clamped to the app's approved ceiling, and clamped again on every request, so narrowing an app's ceiling immediately narrows every key it already issued.
• Keys cannot mint keys. /api/developer/** is session-authenticated only.

Everything is Authorization: Bearer <key>.

3.2 Who am I#

export VM=https://api.vidmoat.com/v1
export KEY=vmk_live_…

curl -s $VM/me -H "Authorization: Bearer $KEY"

{
  "account": {
    "id": "…", "email": "…", "name": "…",
    "plan": { "id": "studio", "name": "Studio", "features": ["ai_console", "no_watermark", …] },
    "credits": { "remaining": 5840, "perMonth": 6000, "resetsAt": "2026-09-01T…" },
    "exports": { "used": 12, "limit": null, "resetsAt": "…" },
    "limits": { "maxProjects": null, "maxExportHeight": 2160, "maxUploadMb": 8192, "maxConcurrentRenders": 5 }
  },
  "credential": {
    "type": "api_key", "keyId": "…", "appId": "…",
    "environment": "live",
    "scopes": ["account.read", "projects.read", "projects.write", "render.read", "render.write"]
  }
}

credential.scopes is the answer to every 403 you are about to get. Read it first.

3.3 Create a project, already edited#

POST /v1/projects accepts a commands array, so a whole video can be one call. Returns 201.

PROJECT=$(curl -s -X POST $VM/projects \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{
    "name": "Hello render",
    "commands": [
      { "op": "setProjectSettings", "aspectRatio": "9:16", "fps": 30, "background": "#0b0b0f" },
      { "op": "addTextClip", "text": "Hello, render", "start": 0.2, "duration": 3,
        "style": { "fontSize": 96, "fontWeight": 800, "color": "#ffffff" } },
      { "op": "addShapeClip", "kind": "capsule", "start": 0, "duration": 3.5,
        "trackIndex": 0, "style": { "fill": "#FF4D2E", "width": 900, "height": 260 } }
    ]
  }' | jq -r '.project.id')

Response:

{
  "project": {
    "id": "clx…", "name": "Hello render",
    "createdAt": "…", "updatedAt": "…",
    "durationSec": 3.5, "clipCount": 2,
    "settings": { "aspectRatio": "9:16", "width": 1080, "height": 1920, "fps": 30, "background": "#0b0b0f" }
  },
  "results": [ { "op": "setProjectSettings", "ok": true }, { "op": "addTextClip", "ok": true, "data": { "clipId": "…" } }, … ],
  "blocked": [],
  "lint": [ { "severity": "note", "time": 0.2, "message": "…" } ],
  "suggestedPreviewTimes": [0.2, 1.75]
}

If a command in the program fails, the project is not rolled back. Each command succeeds or fails on its own, and the ones that succeeded are kept. You get the project plus a results array telling you which command was wrong: a debuggable state, rather than burning a project-quota slot on every retry.

3.4 Apply more commands#

curl -s -X POST $VM/projects/$PROJECT/commands \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{ "commands": [
        { "op": "applyMotionPreset", "clipId": "CLIP_ID", "preset": "bounce-in" },
        { "op": "setTransition", "clipId": "CLIP_ID", "type": "dissolve", "duration": 0.4 }
      ] }'

{
  "projectId": "clx…",
  "ok": true,
  "dryRun": false,
  "results": [ { "op": "applyMotionPreset", "ok": true }, … ],
  "blocked": [],
  "lint": [ { "severity": "warn", "time": 1.2, "message": "Text \"Hello, render\" runs off the right at ~1.2s. Reduce fontSize/scale or move it inward." } ],
  "suggestedPreviewTimes": [1.2, 0.2],
  "timeline": { "clipCount": 3, "durationSec": 3.5 },
  "hint": "1 layout warning(s): see lint[] …"
}

Notes that will save you an afternoon:

• 1 to 200 commands per request. Split larger programs; they apply in order.
• There is no all-or-nothing rollback. Each command succeeds or fails on its own in results[] (ok: false plus an error), and the batch is saved if at least one command succeeded. The top-level ok is true only when every command succeeded.
• blocked is not an error. Plan-gated ops (Pro effects/filters/ transitions, runScript) are dropped from the batch and named in blocked; everything else applies. A Creator-plan caller with one Studio-only effect in a fifty-command program gets forty-nine applied.
• lint comes back on every call and you should read it. Overlapping text, off-canvas elements, font sizes unreadable at output resolution. severity is "warn" or "note", capped at 10 entries. Numeric x/y/fontSize choices routinely look wrong on the real canvas, and this is the only cheap way to find out.
• dryRun: true runs the whole reducer and returns results + lint without persisting: nothing is saved.

3.5 Render and poll#

RENDER=$(curl -s -X POST $VM/renders \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d "{\"projectId\":\"$PROJECT\",\"format\":\"mp4\",\"quality\":\"standard\"}" \
  | jq -r '.render.id')

202, and note what is not in that request body: the document. You name a project you own and the server reads it. The internal /api/render takes the whole timeline inline; v1 deliberately does not, because that would weld the clip format into a public contract.

{
  "render": {
    "id": "clx…", "projectId": "clx…",
    "status": "PENDING", "progress": 0,
    "format": "mp4", "resolution": "full", "quality": "standard",
    "url": null, "error": null,
    "createdAt": "…", "updatedAt": "…"
  },
  "applied": { "resolution": "full", "watermark": false, "priority": true },
  "poll": "/api/v1/renders/clx…"
}

applied tells you the two things that would otherwise be silent surprises: a plan-capped resolution (maxHeight + cappedByPlan: true) and a baked-in watermark (free plans lack no_watermark).

while :; do
  sleep 4
  S=$(curl -s $VM/renders/$RENDER -H "Authorization: Bearer $KEY")
  echo "$S" | jq -r '.render | "\(.status) \(.progress)%"'
  case $(echo "$S" | jq -r '.render.status') in
    COMPLETED) echo "$S" | jq -r '.render.url'; break ;;
    FAILED|CANCELLED) echo "$S" | jq -r '.render.error'; exit 1 ;;
  esac
done

Status strings are uppercase: PENDING, PROCESSING, COMPLETED, FAILED, CANCELLED. The failure reason is on render.error (not errorMsg, which is the database column name; the serializer renames it).

Request-shape validation happens for accepted bodies: format ∈ {mp4, webm, gif}, quality ∈ {draft, standard, high}, resolution is "full" or anything else (which means half). A project with no clips is a 400.

4. Building an editor UI#

This is the section that decides whether you have a product or a script. The answer is yes, you can build a real editor UI on this, and here is exactly how.

4.0 The architecture constraint, first#

Your API key is a server credential. It cannot go in a browser, and api.vidmoat.com's CORS configuration is not written for arbitrary third-party origins. So:

browser (your UI)  ──►  your server  ──►  api.vidmoat.com/v1
       │                                        │
       └──────── media bytes, directly ─────────┘
                 /uploads/*  /exports/*  ACAO: *

The control plane goes through your server. The media plane does not: /uploads/, /exports/ and /fixtures/ on api.vidmoat.com are all served with Access-Control-Allow-Origin: *. That single header is what makes a genuine third-party editor possible, because it means your browser can fetch() the raw media, decode it, and read pixels and samples out of it without tainting a canvas or proxying gigabytes through your own box.

Give yourself a thin proxy and never think about it again:

// server: app/api/vm/[...path]/route.ts  (or the equivalent in your stack)
export async function POST(req: Request, { params }: { params: Promise<{ path: string[] }> }) {
  const { path } = await params;
  const res = await fetch(`https://api.vidmoat.com/v1/${path.join('/')}`, {
    method: 'POST',
    headers: { Authorization: `Bearer ${process.env.VIDMOAT_KEY!}`, 'Content-Type': 'application/json' },
    body: await req.text(),
  });
  // Pass the body and status through untouched: the error envelope's `code` is
  // what your client branches on, and flattening it to "request failed" throws
  // away the only stable machine-readable part.
  return new Response(await res.text(), {
    status: res.status,
    headers: { 'Content-Type': 'application/json' },
  });
}

4.1 Read the document, draw a timeline#

type Clip = {
  id: string;
  type: 'video' | 'audio' | 'text' | 'image' | 'shape' | 'sticker' | 'html' | 'adjustment';
  name: string; src: string;
  start: number;          // timeline seconds
  duration: number;       // timeline seconds
  trackIndex: number;     // a TRACK ID, not a layer index
  mediaStart: number;     // in-point in the source media
  mediaDuration: number;  // length of the source media
  speed: number; reverse: boolean; freezeAt: number | null; loop: boolean;
  x: number; y: number; scale: number; rotation: number; opacity: number;
  textStyle?: { content: string; fontSize: number; /* … */ };
  isCaption?: boolean;
  kfX: { time: number; value: number; easing?: string }[];  // … one array per keyframable prop
  volume: number; muted: boolean; fadeIn: number; fadeOut: number;
};

type TrackDef = { id: number; label: string; type: 'video' | 'audio' | 'text'; color: string; bg: string; muted: boolean; locked: boolean; visible: boolean };

type EditorDocument = {
  version: 2;
  settings: { name: string; aspectRatio: string; width: number; height: number; fps: number; background: string };
  tracks: TrackDef[];
  clips: Clip[];
  markers?: { id: string; time: number; label?: string; color?: string }[];
};

const { project } = await vm(`/projects/${id}?view=document`);
const doc: EditorDocument = project.document;

Three views exist and the difference is size, not taste:

view=document is refused on the list endpoint (GET /v1/projects): a page of 20 full documents is megabytes. You get view=clips and a note saying so.

Drawing the timeline is then arithmetic. Pixels-per-second is your zoom; the Vidmoat editor uses 10 to 300 px/s with a 46 px track height:

const PX_PER_SEC = 60;

function Timeline({ doc }: { doc: EditorDocument }) {
  const total = Math.max(...doc.clips.map(c => c.start + c.duration), 1);
  return (
    <div style={{ position: 'relative', width: total * PX_PER_SEC }}>
      {/* tracks in ARRAY order: index 0 is the bottom layer */}
      {doc.tracks.map((track, layer) => (
        <div key={track.id} style={{ position: 'relative', height: 46, background: track.bg }}>
          {doc.clips
            .filter(c => c.trackIndex === track.id)
            .map(c => (
              <div key={c.id} style={{
                position: 'absolute',
                left: c.start * PX_PER_SEC,
                width: Math.max(c.duration * PX_PER_SEC, 4),
                top: 2, bottom: 2,
                borderLeft: `2px solid ${track.color}`,
                background: `${track.color}22`,
                zIndex: layer,     // matches the renderer's own stacking
              }}>
                <span>{c.type === 'text' ? c.textStyle?.content : c.name}</span>
                {(c.type === 'audio' || c.type === 'video') && (
                  <Waveform src={c.src} clip={c} width={c.duration * PX_PER_SEC} color={track.color} />
                )}
              </div>
            ))}
        </div>
      ))}
    </div>
  );
}

4.2 Waveforms, client-side#

This is the part people assume needs a server. It does not, and Vidmoat's own editor does not use one either: it decodes the media in the browser with AudioContext.decodeAudioData and reduces it to peaks. Because /uploads/ sends Access-Control-Allow-Origin: *, your origin can do the identical thing.

const peakCache = new Map<string, number[]>();

/** ~300 MB ceiling: decodeAudioData needs the ENTIRE file in memory. Past that,
 *  the tab OOMs, so fail fast and fall back to a flat waveform instead. */
const MAX_DECODE_BYTES = 300 * 1024 * 1024;

export async function computePeaks(url: string, count = 400): Promise<number[]> {
  const hit = peakCache.get(url);
  if (hit) return hit;

  try {
    const res = await fetch(url);                       // cross-origin, allowed by ACAO: *
    if (!res.ok) throw new Error(`fetch ${res.status}`);
    if (Number(res.headers.get('content-length') || 0) > MAX_DECODE_BYTES) {
      res.body?.cancel();
      throw new Error('too large to decode');
    }
    const buf = await res.arrayBuffer();

    const ctx = new AudioContext();
    const audio = await ctx.decodeAudioData(buf.slice(0));  // slice(0): decodeAudioData detaches the buffer
    const chan = audio.getChannelData(0);

    const block = Math.floor(chan.length / count) || 1;
    const peaks: number[] = [];
    let max = 0.0001;
    for (let i = 0; i < count; i++) {
      let peak = 0;
      const start = i * block;
      for (let j = 0; j < block; j += 8) {                  // stride: 8x faster, visually identical
        const v = Math.abs(chan[start + j] || 0);
        if (v > peak) peak = v;
      }
      peaks.push(peak);
      if (peak > max) max = peak;
    }
    const norm = peaks.map(p => p / max);
    peakCache.set(url, norm);
    return norm;
  } catch {
    // A non-audio file or a decode failure is normal, not exceptional. Draw
    // something and keep the editor usable.
    const flat = Array.from({ length: count }, (_, i) => 0.2 + 0.12 * Math.abs(Math.sin(i * 0.4)));
    peakCache.set(url, flat);
    return flat;
  }
}

A trimmed clip must show its region of the file, not the whole file's. The window is a pair of fractions derived from mediaStart, duration and speed:

function mediaWindow(clip: Clip): { from: number; to: number } | undefined {
  const len = clip.mediaDuration;
  if (!len || len <= 0) return undefined;          // this is why you always pass mediaDuration
  const from = Math.max(0, Math.min(1, clip.mediaStart / len));
  const to   = Math.max(from, Math.min(1, (clip.mediaStart + clip.duration * clip.speed) / len));
  return { from, to };
}

function Waveform({ src, clip, width, color, height = 26 }: {
  src: string; clip: Clip; width: number; color: string; height?: number;
}) {
  const [peaks, setPeaks] = useState<number[] | null>(null);
  useEffect(() => {
    let alive = true;
    if (src) computePeaks(src, 400).then(p => { if (alive) setPeaks(p); }).catch(() => {});
    return () => { alive = false; };
  }, [src]);
  if (!peaks) return null;

  const win = mediaWindow(clip);
  const view = win
    ? peaks.slice(
        Math.floor(win.from * peaks.length),
        Math.max(Math.floor(win.from * peaks.length) + 1, Math.ceil(win.to * peaks.length)),
      )
    : peaks;

  const bars = Math.max(12, Math.min(400, Math.floor(width / 2)));
  const step = view.length / bars;
  const mid = height / 2;
  return (
    <svg width="100%" height={height} viewBox={`0 0 ${bars} ${height}`} preserveAspectRatio="none">
      {Array.from({ length: bars }, (_, i) => {
        const h = Math.max(1.2, (view[Math.floor(i * step)] ?? 0) * (height - 2));
        return <rect key={i} x={i + 0.15} y={mid - h / 2} width={0.7} height={h} fill={color} opacity={0.85} />;
      })}
    </svg>
  );
}

The same technique gives you the other analyses Vidmoat runs client-side, off the same decoded buffer: silence detection (RMS per 20 ms window, merge, drop regions under 0.4 s) and energy-flux beat detection. Both are a few dozen lines over the same peaks array. The point here is that none of it costs you a server round-trip or a credit, and it is the input to the cutRanges, trimSilence, addMarkers and sliceClip commands.

4.3 The composition preview#

GET /v1/projects/{id}/preview?format=html returns the whole composition: a self-contained HTML document with every clip as a positioned element, driven by one paused GSAP timeline registered at window.__timelines.main. This is the exact HTML the render worker drives, not an approximation of it.

GET /v1/projects/{id}/preview?format=html
→ text/html, Cache-Control: private, no-store
  X-Vidmoat-Media-Count: 7

Two things stop you from putting that URL straight in an <iframe src>:

1. You cannot attach an Authorization header to an iframe navigation.
2. Responses from the API host carry X-Frame-Options: DENY and Content-Security-Policy: frame-ancestors 'none'.

So fetch it on your server and serve it from your own origin. The HTML uses root-relative URLs (src="/uploads/…", <script src="/vendor/gsap.min.js">), which would resolve against your origin, so rebase them:

// server
const html = await fetch(`https://api.vidmoat.com/v1/projects/${id}/preview?format=html`, {
  headers: { Authorization: `Bearer ${process.env.VIDMOAT_KEY!}` },
}).then(r => r.text());

const rebased = html
  .replaceAll('src="/uploads/', 'src="https://api.vidmoat.com/uploads/')
  .replaceAll('src="/vendor/',  'src="https://api.vidmoat.com/vendor/');
// (Injecting <base href="https://api.vidmoat.com/"> into <head> also works and
//  is one line, but it rebases everything, including anything you add later.)

return new Response(rebased, { headers: { 'Content-Type': 'text/html; charset=utf-8' } });

Then scrub it:

const frame = document.querySelector('iframe')!;
const win = frame.contentWindow as any;

// Same-origin now, so the handle is reachable.
win.__timelines.main.seek(2.5);
win.__timelines.main.play();
win.__timelines.main.pause();

Wire that to your playhead and you have an interactive preview at zero server cost. Structure you can rely on inside the document:

• #root carries data-composition-id="main", data-start, data-duration, data-width, data-height, data-fps.
• Every timed element has class="clip" plus data-start, data-duration, data-track-index. Visibility is driven by the timeline; .clip starts visibility: hidden.
• Video clips are <video id="media-{clipId}" muted playsinline>. A video's sound is a separate <audio id="aud-{clipId}"> element, so muting picture and sound are independent.

One real limitation, stated plainly. The timeline tweens a video's currentTime only when the clip is non-trivially timed: speed !== 1, reverse, freezeAt, or loop. At speed 1 it does not, because the render worker seeks each video explicitly instead. So a bare seek(t) shows the right layout at t but may show the wrong frame of a normal-speed video. Do what the worker does:

function seekTo(win: any, t: number, doc: EditorDocument) {
  win.__timelines.main.seek(t);

  for (const c of doc.clips) {
    if (c.type !== 'video' || c.speed !== 1) continue;      // speed≠1 is already tweened
    if (t < c.start || t >= c.start + c.duration) continue;
    const el = win.document.getElementById(`media-${c.id}`) as HTMLVideoElement | null;
    if (!el) continue;
    const mediaT = (c.mediaStart ?? 0) + (t - c.start);
    if (Math.abs(el.currentTime - mediaT) > 0.02) el.currentTime = mediaT;
  }
}

data-media-start on the element carries the same number if you would rather read it out of the DOM than out of the document.

4.4 Frame previews, for thumbnails#

# JSON: metadata + lint + the frame as both a URL and a data URI
curl -s "$VM/projects/$PROJECT/preview?at=1.5" -H "Authorization: Bearer $KEY"

# Raw JPEG bytes
curl -s "$VM/projects/$PROJECT/preview?format=image&at=1.5" -H "Authorization: Bearer $KEY" -o frame.jpg

The image response carries X-Vidmoat-Frame-Time, X-Vidmoat-Frame-Width, X-Vidmoat-Frame-Height, so you do not need a second JSON call. The JSON response gives you frame.url (durable, for a UI) and frame.dataUri (bytes, for feeding a vision model without a second authenticated fetch).

Budget carefully:

• Every frame launches Chromium, serialised process-wide, and is rate limited to 10 per minute per key in its own bucket (api_v1_frame), separate from your plan RPM. Exceeding it is a 429 whose message tells you to use format=html instead.
• Rendered at half the project resolution. Enough to judge composition.
• at past the end is clamped, not rejected.
• Omitting both at and format=image gives you the cheap answer: duration, canvas, clipCount, lint, suggestedPreviewTimes, and the composition URL. No browser is launched. This is the call to make on every edit.

Do not build a filmstrip out of this endpoint. Ten frames a minute is a verification tool, not a thumbnail service. For a filmstrip, draw the video element to a canvas yourself in the browser. /uploads/ sends ACAO: *, so the canvas is not tainted and toDataURL works.

4.5 Optimistic edits, reconciled#

The reducer is deterministic, so you can apply an edit locally and post the same command. What you must not do is assume your local result is the server's: collision handling may have moved the clip to another track, plan gating may have dropped an op, and ids are generated server-side.

type CommandResult = { op: string; ok: boolean; error?: string; data?: Record<string, unknown> };

async function applyEdit(projectId: string, commands: unknown[], local: () => void, rollback: () => void) {
  local();                                          // optimistic: draw it now

  const res = await vm(`/projects/${projectId}/commands`, { commands });

  if (!res.ok) {
    // Per-command failures. `results` says WHICH one, and why.
    const bad = (res.results as CommandResult[]).filter(r => !r.ok);
    console.warn('rejected', bad);
    rollback();
  }

  if (res.blocked?.length) {
    // Not an error: a plan boundary. Everything else applied. Show an upsell,
    // do not show a failure.
    showUpgradePrompt(res.blocked);
  }

  // Reconcile from the server, always. Cheap: `clips` view, not `document`.
  const { project } = await vm(`/projects/${projectId}?view=clips`);
  setTimeline(project);

  // And surface the lint: this is the feedback your users cannot get any other way.
  setLint(res.lint as { severity: 'warn' | 'note'; time: number; message: string }[]);
  setSuggestedTimes(res.suggestedPreviewTimes as number[]);
}

For anything a user can spam (dragging a clip, scrubbing a slider), debounce into one command rather than one per frame: you have 20/120/600 requests per minute depending on the owner's plan (Team and Enterprise get 600), and 200 commands per request is plenty of room to batch.

5. Building an agentic platform#

Two routes. They are not competitors; they answer different questions.

5.1 Route A: Vidmoat's planner does the thinking#

curl -s -X POST $VM/ai/agent \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d "{\"projectId\":\"$PROJECT\",\"message\":\"add a bold title card for the first two seconds and duck the music under the voiceover\"}"

Body: { projectId, message, dryRun? }. message is required (max 2000 chars), projectId is required. Scope ai.agent, plan feature ai_console (every plan, including free), 5 credits.

{
  "projectId": "clx…",
  "applied": true,
  "planner": "deepseek",
  "message": "Added a title card and ducked the music.",
  "commands": [ { "op": "addTextClip", … }, { "op": "autoDuck", … } ],
  "results": [ … ],
  "blocked": [],
  "unresolved": [],
  "lint": [ … ],
  "suggestedPreviewTimes": [0.5, 1.9],
  "project": { … },
  "credits": { "charged": 5, "remaining": 5835 }
}

What it actually is, so you can price and reason about it:

• One turn, not a loop. One planning call plus at most one repair round-trip, then apply and return. The editor's agent runs a multi-step goal; this does not. Drive the loop yourself by reading results/unresolved and calling again. An endpoint that silently makes five provider calls is one you cannot budget.
• Provider chain: DeepSeek → Grok → Claude, whichever are configured, plus a built-in offline heuristic planner if none are. planner in the response tells you which answered ("heuristic" means no model provider is configured on the deployment).
• The repair round-trip is only accepted if it is better: strictly fewer validation failures than the first attempt. Anything it could not fix comes back in unresolved, rather than being swallowed.
• dryRun: true returns the plan without applying it. This is the interesting mode: see what the model intends, filter it, and post the parts you want to /commands yourself. The 5 credits are still spent: the model call happened, and that is what you paid for.
• No privileged path for model output. The commands go through the same reducer and the same plan gate as anything else. A hallucinated Studio-only effect on a Creator plan is dropped into blocked.
• Empty plan is not refunded. A model call happened. commands: [] with a message saying it could not turn that into commands. The 5 credits come back only when every planner fails outright, which is a 502 provider_error.

Use Route A when the natural-language step is your product's interface and you do not want to own a model relationship.

5.2 Route B: your own LLM emits commands#

GET /v1/schema/commands      →  { schema: { description, enums, commands } }

No scope required (a valid credential still is). This is COMMAND_SCHEMA verbatim: the same constant the editor, the chat agent, Flows and the MCP server read. There is no second copy to drift.

{
  "schema": {
    "description": "VidMoat editor command protocol. …Times are in seconds; positions in pixels at project resolution (origin = canvas center); keyframe times are relative to clip start.",
    "enums": {
      "clipType": ["video", "audio", "text", "image", "shape", "sticker"],
      "keyframableProp": ["opacity", "scale", "x", "y", "rotation", "volume", "cropL", …, "speed"],
      "easing": ["linear", "easeIn", …, "spring"],
      "transitionType": ["fade", "dissolve", "wipe", "slide", "push", "zoom", "spin", "blur", "flash", "glitch", "circle", "iris", "whip-pan", "radial", "blinds", "pixelate-out", "zoom-blur"],
      "shapeKind": ["rectangle", "circle", "triangle", "arrow", "line", "star", …],
      "aspectRatio": ["16:9", "9:16", "1:1", "4:5", "4:3", "21:9"],
      "motionPreset": [ … ], "filterPreset": [ … ], "effectType": [ … ], "blendMode": [ … ], "maskShape": [ … ], "textAnimation": [ … ]
    },
    "commands": {
      "addClip":     { "description": "Add a media clip to the timeline. src is a URL…", "params": { "type": "clipType", "src": "string", "start": "seconds", "trackIndex": "number", "duration": "seconds", "mediaStart": "seconds?", "mediaDuration": "seconds?", … }, "returns": { "clipId": "string" } },
      "sliceClip":   { "description": "Split a clip into N segments in ONE command (repeated splitClip does NOT work)…", … },
      "cutRanges":   { "description": "Cut arbitrary TIMELINE time ranges out of one clip (filler words, bad takes…)", … },
      "autoDuck":    { "description": "Duck music under speech: rewrites each music clip's volume keyframes…", … },
      "addCaptions": { "description": "Lay a word-level transcript out as timed, karaoke-styled caption clips…", … }
      // …~80 ops
    }
  }
}

The dependable pattern:

async function agenticEdit(projectId: string, goal: string) {
  const { schema } = await vm('/schema/commands');
  const { project } = await vm(`/projects/${projectId}?view=clips`);

  for (let attempt = 0; attempt < 3; attempt++) {
    const commands = await yourModel({
      system: [
        'Emit ONLY JSON: { "commands": [ { "op": "…", … } ] }.',
        'Use ONLY these ops and params:', JSON.stringify(schema.commands),
        'Enums:', JSON.stringify(schema.enums),
        'Times are seconds. Positions are pixels from the canvas CENTRE. Keyframe times are relative to clip start.',
        'Clips composite in tracks-LIST order: a clip on a track listed LATER draws ON TOP.',
      ].join('\n'),
      user: `Project: ${JSON.stringify(project)}\n\nGoal: ${goal}`,
    });

    // ── VALIDATE BEFORE YOU WRITE ────────────────────────────────────────────
    const check = await vm(`/projects/${projectId}/validate`, { commands });

    if (!check.valid) {
      // Feed the model its own errors. A clip id that does not exist is the most
      // common failure and it is mechanically detectable and mechanically fixable.
      goal = `${goal}\n\nYour previous plan failed validation: ${
        JSON.stringify(check.results.filter((r: any) => !r.ok))
      }`;
      continue;
    }

    // check.blocked is a PLAN answer, not a correctness answer. Do not retry it.
    if (check.blocked.length) notifyUpgrade(check.blocked);

    // check.lint is the layout of the HYPOTHETICAL outcome. Nothing was written.
    if (check.lint.some((l: any) => l.severity === 'warn')) {
      goal = `${goal}\n\nThat plan would produce layout problems: ${JSON.stringify(check.lint)}`;
      continue;
    }

    const applied = await vm(`/projects/${projectId}/commands`, { commands });
    return applied;
  }
  throw new Error('could not produce a valid plan');
}

Why validate-then-apply matters, concretely. POST /v1/projects/{id}/validate needs only projects.read. That is the whole reason it is a separate endpoint rather than a flag on /commands: a read-only key (the default the portal offers) can check a generated program against a real document without any write access. A self-correcting loop that mutates the project to discover whether its plan parsed is a loop that leaves debris behind on every failed attempt.

It also separates two answers a single boolean conflates:

• results[]: is each command valid against this document? Real clip ids, in-range times.
• blocked[]: does the owner's plan allow it? A Creator-plan caller whose Studio-only effect is perfectly well-formed should be told exactly that, not handed a validation failure to debug.

And lint[] is the third answer, the one nobody asks for and everybody needs: a program can be entirely valid and entirely allowed and still put your title off-canvas. Feed lint warnings back to the model the same way you feed validation errors. suggestedPreviewTimes then tells you which timestamps to actually look at.

5.3 Which route#

Nothing stops you using both: Route A for a quick-action bar, Route B for your core pipeline.

6. Auth: two models#

Model 1: API key#

Your server acts as itself. One Vidmoat account (yours) owns every project, every byte of media and every credit spent. Your users never hear the word Vidmoat.

Authorization: Bearer vmk_live_…

• You own the user relationship completely. No consent screen, no revocation you do not control, no OAuth to debug.
• You pay. Every render comes out of your export allowance, every generation out of your credit balance. Your unit economics are your problem, which is also the point: you can charge whatever you like on top.
• Every project belongs to you, so "let the user export their project" and "delete a user's data" are features you build.
• Live keys need a plan with api_access (Studio, Team or Enterprise), an accepted developer agreement and an app approved in review.

Right for: a SaaS product, a pipeline, an internal tool, anything where Vidmoat is an implementation detail.

Model 2: Sign in with Vidmoat (OAuth 2.0 + PKCE)#

Your users connect their own Vidmoat accounts. Full protocol details are on the Authentication page; the shape:

https://www.vidmoat.com/oauth/authorize
  ?response_type=code&client_id=…&redirect_uri=…
  &scope=account.read%20projects.write%20render.write%20render.read
  &state=…&code_challenge=…&code_challenge_method=S256
→ POST https://api.vidmoat.com/api/oauth/token  (code + code_verifier)
→ { access_token (1h), refresh_token (90d, rotates on use), scope }

• Their plan, their credits, their export allowance, their content. You are not reselling anything.
• Scoped consent, shown to the user. Spending scopes appear in a separate highlighted block and the Allow button tells the user that allowing includes spending their credits. Expect a materially lower approval rate when you ask for them; ask at the moment they are needed.
• For a client that belongs to a developer app, access tokens last 1 hour and refresh tokens 90 days, rotating on every use. OAuth tokens are always live, never test.
• Revocable, with no callback. The user revokes at https://www.vidmoat.com/oauth/connected and every token for your app is deleted immediately. invalid_grant on refresh is not retryable: it means "this user disconnected us".
• scope is required on the authorize call; there is no default. Store the granted scope from the token response, not the one you requested, because it may have been clamped to your app's ceiling.
• Read https://api.vidmoat.com/.well-known/oauth-authorization-server rather than hardcoding endpoints.
• Do not use /api/oauth/register (Dynamic Client Registration). It exists so MCP clients can self-register; it produces a client with no app identity, no branding and no entry on the user's connected-apps page.

Right for: a tool that augments Vidmoat, a marketplace integration, anything where the user already has (or should have) a Vidmoat account.

The honest trade-off#

Model 1 is simpler to ship, gives you total control of the experience, and puts the entire cost on you. Model 2 shifts the cost to the user and gives them a revoke button, at the price of a consent screen in your funnel, refresh-token lifecycle code, and a permanent dependency on a relationship you do not own.

Pick Model 1 if your product is the video. Pick Model 2 if your product is about the user's Vidmoat account.

Scopes are enforced#

Eighteen scopes, in full:

Keys minted before scopes existed carry every scope except webhooks.manage, plugins.invoke, telegram.users.read and telegram.users.write.

A missing scope is a 403 that names it:

{
  "error": {
    "code": "insufficient_scope",
    "message": "This key is missing the `ai.video` scope. Re-mint it with that scope at developer.vidmoat.com.",
    "docs_url": "https://developer.vidmoat.com/developer/docs/errors#insufficient_scope",
    "scope": "ai.video",
    "granted": ["account.read", "projects.read", "projects.write"]
  }
}

Nothing is leaked by that (the vocabulary is public and the caller already holds the credential), and it is the difference between a ten-second fix and a support thread.

A scope is necessary but never sufficient. Holding ai.video does not mean the call succeeds. The owner's plan must include ai_generative_media and they must have credits, or you get a 402. This is deliberate: without it, minting a key with a scope would be a way to buy Studio features for free.

403 is your bug. 402 is the account's balance. Do not retry either.

7. What it costs#

The credit table#

Every action that spends credits, and what it spends:

agent_step (3), plan_workflow (3) and browser_record (40) exist in the table but no v1 endpoint charges them.

Video bills per second#

This is the one that will surprise you if you read the old table price of 500.

credits = perSecond(provider, resolution) × durationSec

  grok (grok-imagine-video)    80 credits/second
  qwen (HappyHorse)           112 / 224 / 288 credits/second at 480p / 720p / 1080p
  wan  (Wan 3.0)               80 / 160 / 320 credits/second at 480p / 720p / 1080p

Choose with provider (grok, the default, qwen or wan). Through this API Grok always uses grok-imagine-video; the 1.5 model is not selectable.

So on grok-imagine-video: 5 s (the default) = 400 credits, 10 s = 800, 15 s = 1,200. The flat 500 was really a 15-second price that short clips overpaid for; per-second billing means margin does not depend on the caller's choice, and short clips got cheaper.

durationSec is a whole number of seconds, 5 by default. Grok takes 1 to 15 (out-of-range values are clamped), HappyHorse 3 to 15 and Wan 2 to 15. The same number is both billed and sent to the provider, so you cannot ask for 900 and be billed for 15 while getting 15, nor ask for 0.4 and be billed nothing.

The response tells you the basis so you can show a user a bill they can check:

{
  "id": "clx…", "status": "PROCESSING", "durationSec": 5,
  "credits": { "charged": 400, "remaining": 5440, "basis": "80 credits/second of generated video (grok-imagine-video)" },
  "poll": "/api/v1/ai/jobs/clx…"
}

Video is asynchronous (202) and one at a time per account: a second concurrent request is a 402 quota_exceeded, deliberately, because a caller looping video generation can spend a Studio allowance in under a minute. GET /v1/ai/jobs/{id} (scope ai.video) polls only video jobs; images, speech and stickers are synchronous. It polls through to the provider, persists the transition, and refunds on upstream failure exactly once (compare-and-swap on the job's own params, so two concurrent polls of a just-failed job cannot both pay out). A network blip talking to the provider comes back as still-PROCESSING with a warning, never as FAILED. Job statuses are PROCESSING, COMPLETED and FAILED.

Transcription#

• Free mode: { script, duration, start? } distributes text you already have across a span. No provider, no credits, and the output is the same words[] shape. For programmatic use (a voiceover you wrote, a script you are narrating) this is almost always the right call.
• ASR mode: { src, language? }. 10 credits per 5 minutes, rounded up. So a 12-minute podcast is 3 blocks = 30 credits.
• Hard cap: 30 minutes (MAX_CAPTION_SECONDS = 1800). Longer is a 400 telling you to split it and offset each part's word times by its start.
• The length is probed before you are charged. Unreadable length is a 400, not a guess: an unbounded bill and a genuinely broken source both deserve an error.
• If the deployment has no ASR provider, you get 501 not_configured before any charge.

Refunds#

Every generation endpoint charges before the provider call and refunds on failure (charge() hands back its own refund, so the amount given back cannot drift from the amount taken). Charging afterwards would let a caller disconnect mid-request and get the work free.

Test-mode keys spend nothing#

A vmk_test_… key returns deterministic fixtures. Ids are a hash of the request, so POST /v1/renders twice with the same arguments returns the same job id, and GET /v1/renders/{that id} answers from the id with no row ever existing, which is what lets you exercise your poll loop.

• Fixtures come from public/fixtures/v1/ and are real, valid media (render.mp4|webm|gif, image.jpg, video.mp4, speech.mp3, sticker.png) served with ACAO: *, so they decode in a <video> element on your origin.
• Test renders come back COMPLETED immediately with a sample fixture and test: true. They are never queued, use no export quota, and write no row.
• Generation returns sample fixtures and spends nothing; quoteOnly is ignored on a test key.
• The agent still needs a real projectId, then returns an empty plan without calling a model.
• Plugins are not called, and PATCH /v1/telegram/users/{user} is refused.
• Validation still runs first: a bad projectId still 404s, an empty project still 400s. A sandbox that accepts anything teaches nothing.
• Test mode is not a full sandbox. Only the paths that cost money or occupy a worker are stubbed. Projects, /commands, /media/uploads, /media/import and previews do real work with a test key: real rows, real files. Plan accordingly.
• A test-mode request that ever reaches the billing path throws a loud 500 rather than silently spending. That is a tripwire for a missing branch, not a path you should see.

Rate limits and concurrency#

• Team and Enterprise keys get 600 requests/min, the same as Studio (the higher of the owner's own plan and their team's plan applies).
• The limiter is keyed on the key id, not the user, over a 60-second sliding window, so two of your integrations cannot starve each other and a leaked key can be throttled by revoking it.
• Frame previews have their own extra bucket: 10/min per key.
• PATCH /v1/telegram/users/{user} is limited to 120/min per app.
• An app the platform has marked throttled drops to 6/min and keeps working; that is the whole point of throttled versus suspended.
• Every response that reaches the limiter carries X-RateLimit-Limit, X-RateLimit-Remaining and X-RateLimit-Reset. X-RateLimit-Reset is the window length in seconds (normally 60), not a Unix timestamp. A 429 or 503 also carries Retry-After. Refused requests do not count against the window.
• If the limiter's store (Redis) is down, v1 fails closed: every request gets 503 unavailable with Retry-After: 30, and nothing is run or charged.
• Inline uploads are capped at 32 MB regardless of plan (the path buffers in memory); the plan cap applies on top. Bigger files go via POST /v1/media/import, up to your plan's upload cap or 512 MB, whichever is smaller.

8. Production concerns#

Idempotency: there isn't any, so build it#

There is no Idempotency-Key header. Commands are not idempotent. What the platform does give you:

• Uploads dedupe by sha256. Re-POSTing the same bytes returns the existing file with deduplicated: true, so upload retries are safe and do not multiply your disk bill.
• Test fixture ids are deterministic, so retrying a test call is stable.
• Video refunds are claimed with a compare-and-swap, so concurrent polls cannot double-refund.

Everything else is on you. The pattern that works:

// Persist the server's ids BEFORE you retry anything.
const res = await vm(`/projects/${id}/commands`, { commands });
await db.saveClipIds(jobId, res.results.flatMap((r: any) => r.data?.clipId ?? r.data?.clipIds ?? []));

// On retry, check what already landed rather than replaying the program.
// A retried `addClip` is a duplicate clip, not a no-op.

For "create a project and edit it" specifically, use the commands array on POST /v1/projects: one call, one failure domain, and a partial failure leaves you a project plus a results array instead of an ambiguous half-state.

Polling cadence#

The limiter does not distinguish an eager poller from an attack. Both are requests against the same per-minute budget.

• Renders: 3 to 5 seconds is right. A PENDING render that has not started will not start faster because you asked twice.
• Video generation: 5 to 10 seconds. It takes tens of seconds at best.
• Budget it: on Creator (120/min), one client polling every 2 seconds burns 30 of your 120. Ten concurrent jobs at that cadence is 300/min and you are throttled.
• Back off on 429, honour Retry-After, and stop polling terminal states: COMPLETED, FAILED and (for renders) CANCELLED answer from the stored row, so re-polling is pure waste.
• A transient warning on a still-PROCESSING generation job means keep going, not give up.

402 handling#

Three distinct codes share the 402 status, and they need three different behaviours:

The only retryable one is concurrency, and even then with backoff.

The error envelope#

{ "error": { "code": "…", "message": "…", "docs_url": "https://developer.vidmoat.com/developer/docs/errors#…", /* extras */ } }

code is stable and machine-readable; message is prose written for a human and will be reworded. Branch on code. Never parse message.

Codes are added, never renamed. Treat an unknown code as a generic failure of its HTTP class.

Note that someone else's project id is a 404, not a 403: distinguishing them would confirm the id exists.

Vidmoat-Version is echoed on every response (default 2026-08-01). Nothing branches on it yet; send it anyway, so the day some behaviour becomes date-pinned your clients are already sending a version they can see acknowledged.

You are responsible for what your users generate#

Third-party apps mean content generated at Vidmoat's expense, hosted on Vidmoat's domain, by a user Vidmoat has no relationship with. So:

• Prompt filtering runs server-side, in the shared wrapper rather than per-endpoint, on every generation route. Any of prompt, text, message, script, caption in the body is checked before any charge. A refusal is 422 prompt_rejected and costs nothing.
• Your app can be suspended independently of your account. DeveloperApp.status is active | throttled | suspended; throttled drops you to 6 requests/minute, suspended returns 403 app_suspended on every call. One bad integration is stoppable without banning the person who wrote it.
• A monitored contact address, an accepted developer agreement and an app approved in review are required before a live key is issued. That is what makes takedown possible.
• Everything is attributed to the owner: UsageEvent and ApiRequestLog carry appId and keyId, and RenderJob does too. There is always a real Vidmoat user behind every byte.
• Do your own filtering in front of Vidmoat's. Server-side prompt safety is a backstop, not your moderation policy, and "the API let it through" is not a defence.

Observability#

GET /v1/usage?days=30 gives two breakdowns, and the second is the one you want:

{
  "window": { "days": 30, "since": "…" },
  "account": { "credits": 4210, "requests": 8134, "errors": 96, "successes": 8038,
               "byAction": { "auto_caption": { "count": 12, "credits": 180 }, … },
               "balance": { "remaining": 5440, "perMonth": 6000, "resetsAt": "…" } },
  "app":     { "appId": "…", "keyId": null, "credits": 3990, "requests": 7900, "errors": 88, "successes": 7812, "byAction": { … } }
}

account includes the owner's own editing in the browser. app covers only what came through this app's credentials, which is the number you need when you get a credit bill. credits is net, so refunds reduce it rather than reading as a second charge. days caps at 90.

9. Worked example: a podcast episode into five captioned vertical clips#

This exercises transcription, the command vocabulary, per-clip projects, preview verification and rendering. It is roughly the smallest thing that is a real product.

Inputs: a URL to a 40-minute episode (audio or video), and a way to pick interesting spans (your model, your heuristic, or a human).

Scopes: media.write projects.write projects.read render.write render.read ai.transcribe.

Cost: the episode is over the 30-minute transcription cap, so it must be split. Two halves of 20 minutes = 4 blocks each = 40 credits each = 80 credits total. Then 5 renders against the export allowance. No generative credits at all.

const VM = 'https://api.vidmoat.com/v1';
const KEY = process.env.VIDMOAT_KEY!;

async function vm(path: string, body?: unknown, method = body ? 'POST' : 'GET') {
  const res = await fetch(`${VM}${path}`, {
    method,
    headers: { Authorization: `Bearer ${KEY}`, ...(body ? { 'Content-Type': 'application/json' } : {}) },
    body: body ? JSON.stringify(body) : undefined,
  });
  const json = await res.json();
  if (!res.ok) {
    const e = json.error ?? {};
    throw Object.assign(new Error(`${e.code}: ${e.message}`), { code: e.code, status: res.status, detail: e });
  }
  return json;
}

type Word = { text: string; start: number; end: number };
type CommandResult = { op: string; ok: boolean; error?: string; data?: Record<string, unknown> };

// ─── 1. Get the episode into the media store ─────────────────────────────────
// import, not upload: uploads buffer in memory and cap at 32 MB. import streams,
// and runs the SSRF gate on your URL.
async function ingest(episodeUrl: string) {
  const { media } = await vm('/media/import', { url: episodeUrl });
  //  { url, filename, mediaType, contentType, bytes, durationSec }
  if (!media.durationSec) throw new Error('could not determine episode length');
  return media as { url: string; mediaType: 'video' | 'audio' | 'image'; durationSec: number };
}

// ─── 2. Transcribe, in ≤30-minute chunks ─────────────────────────────────────
// The cap is 1800s per request, enforced from a probe BEFORE you are charged.
// Splitting means re-offsetting each chunk's word times by its own start.
const CHUNK = 1500; // 25 min, comfortably under the cap

async function transcribeEpisode(src: string, durationSec: number): Promise<Word[]> {
  const words: Word[] = [];
  for (let offset = 0; offset < durationSec; offset += CHUNK) {
    // NOTE: /v1/ai/transcriptions has no time-range parameter. You must hand it a
    // src that IS the chunk: cut the episode into parts on your side, import each,
    // and transcribe each. (`start` on the endpoint belongs to `script` mode, not
    // ASR mode.)
    const chunkUrl = await yourChunker(src, offset, Math.min(CHUNK, durationSec - offset));
    const { media } = await vm('/media/import', { url: chunkUrl });

    const r = await vm('/ai/transcriptions', { src: media.url });
    //  { words, provider, mediaSeconds, credits: { charged, remaining, basis } }
    console.log(`chunk @${offset}s: ${r.credits.charged} credits (${r.credits.basis})`);

    for (const w of r.words as Word[]) {
      words.push({ text: w.text, start: w.start + offset, end: w.end + offset });
    }
  }
  return words;
}

// ─── 3. Pick five spans ──────────────────────────────────────────────────────
// Your model, your heuristic, your editor. The transcript with real timings is
// the whole input. Keep them 20 to 60 s: TikTok's sweet spot, and short enough that
// a bad pick is cheap.
type Span = { start: number; end: number; hook: string };

async function pickSpans(words: Word[]): Promise<Span[]> { /* … five of them … */ }

// ─── 4. One project per clip ─────────────────────────────────────────────────
// One project per clip, not one project with five outputs: a render renders a
// project, so five outputs means five projects. It also means one bad clip does
// not block the other four.
async function buildClip(source: { url: string; mediaType: string; durationSec: number }, span: Span, words: Word[], index: number) {
  const duration = Math.round((span.end - span.start) * 100) / 100;

  // Words that fall inside the span, rebased to TIMELINE seconds.
  //
  // addCaptions takes TIMELINE times: it sets each caption clip's `start` from
  // the first word of its line, and stores per-word offsets relative to that
  // clip. So subtract the span start; do NOT pass raw episode times.
  const spanWords = words
    .filter(w => w.end > span.start && w.start < span.end)
    .map(w => ({
      text: w.text,
      start: Math.max(0, w.start - span.start),
      end: Math.min(duration, w.end - span.start),
    }))
    .filter(w => w.end > w.start);

  // ── 4a. Everything that needs no clip id: one call, one failure domain ─────
  // `commands` on POST /v1/projects means create-and-edit is atomic enough:
  // a partial failure leaves you a project plus a results[] naming the bad
  // command, instead of an ambiguous half-state across two requests.
  const created = await vm('/projects', {
    name: `Clip ${index + 1}: ${span.hook.slice(0, 40)}`,
    commands: [
      { op: 'setProjectSettings', aspectRatio: '9:16', fps: 30, background: '#000000' },

      // The source clip. mediaStart selects the window; mediaDuration MUST be the
      // real source length. addClip defaults it to `duration`, which silently
      // breaks waveform windowing and looping downstream.
      {
        op: 'addClip',
        type: source.mediaType === 'video' ? 'video' : 'audio',
        src: source.url,
        name: 'Episode',
        start: 0,
        duration,
        trackIndex: 0,
        mediaStart: span.start,
        mediaDuration: source.durationSec,
      },

      // The hook. Positions are pixels from the canvas CENTRE and +y is DOWN, so
      // a negative y is the upper third of a 1080×1920 canvas.
      {
        op: 'addTextClip',
        text: span.hook,
        start: 0,
        duration: Math.min(3, duration),
        style: { fontSize: 84, fontWeight: 800, color: '#ffffff', align: 'center', outline: '#000000', outlineWidth: 6 },
        patch: { y: -560 },
      },

      { op: 'addCaptions', words: spanWords, preset: 'Karaoke', maxWordsPerLine: 4, y: 620 },
    ],
  });
  const project = created.project;

  // ── 4b. Anything that needs a clip id, in a second call ────────────────────
  // You cannot reference a clip you have not created yet, so ops that take a
  // clipId are a follow-up. Read the id out of results[]; do not guess it, and
  // do not assume the clip landed on the trackIndex you asked for.
  const sourceClipId = (created.results as CommandResult[])
    .find(r => r.op === 'addClip')?.data?.clipId as string | undefined;

  let applied = created;
  if (sourceClipId && source.mediaType === 'video') {
    // A vertical crop that keeps the bottom strip clear for the captions.
    applied = await vm(`/projects/${project.id}/commands`, {
      commands: [{ op: 'applyLayout', clipId: sourceClipId, layout: 'caption-safe' }],
    });
  }

  // ── 4c. READ THE LINT ──────────────────────────────────────────────────────
  const warnings = (applied.lint as { severity: string; time: number; message: string }[])
    .filter(l => l.severity === 'warn');
  if (warnings.length) {
    console.warn(`clip ${index + 1}:`, warnings.map(w => `${w.time}s ${w.message}`).join('\n'));
    // Typical fixes: updateClip with a smaller fontSize or a pulled-in x/y, or
    // retime an overlapping caption. Do it before rendering, not after.
  }

  // ── 4d. Look at a frame ────────────────────────────────────────────────────
  // 10/minute across the whole key, so ONE frame per clip: the first suggested
  // time, which is where the lint says the problem is.
  const at = (applied.suggestedPreviewTimes as number[])[0] ?? 0.5;
  const preview = await vm(`/projects/${project.id}/preview?at=${at}`);
  await saveThumbnail(index, preview.frame.dataUri);   // also your social cover image

  return project.id as string;
}

// ─── 5. Render and poll ──────────────────────────────────────────────────────
async function render(projectId: string) {
  const queued = await vm('/renders', { projectId, format: 'mp4', quality: 'high' });
  if (queued.applied.watermark) console.log('watermarked: owner has no no_watermark feature');
  if (queued.applied.cappedByPlan) console.log(`capped to ${queued.applied.maxHeight}px by plan`);

  for (;;) {
    await new Promise(r => setTimeout(r, 4000));
    const { render } = await vm(`/renders/${queued.render.id}`);
    if (render.status === 'COMPLETED') return render.url as string;
    if (render.status === 'FAILED' || render.status === 'CANCELLED') throw new Error(render.error ?? 'render failed');
  }
}

// ─── put it together ─────────────────────────────────────────────────────────
export async function podcastToClips(episodeUrl: string) {
  const source = await ingest(episodeUrl);
  const words = await transcribeEpisode(source.url, source.durationSec);
  const spans = await pickSpans(words);

  const out: string[] = [];
  for (const [i, span] of spans.entries()) {
    const projectId = await buildClip(source, span, words, i);
    // Serial, not Promise.all: concurrent renders are capped at 1/3/5 by plan and
    // a 402 quota_exceeded here is not a failure worth surfacing to a user.
    out.push(await render(projectId));
  }
  return out;
}

What this example is teaching, beyond the code#

• Transcription has no time-range parameter. Chunking is your job, and so is re-offsetting word times. The 30-minute cap is not negotiable and it is enforced from a probe before you are charged, so you find out cheaply.
• addCaptions takes timeline times. It derives each caption clip's start from its line's first word and stores per-word offsets relative to that. Feed it raw episode times and every caption lands 20 minutes into a 40-second clip.
• Positions are pixels from the canvas centre, +y down. Getting this wrong is the single most common way an otherwise-correct program produces a wrong frame, which is why lint exists and why you should look at one frame per clip.
• Read results[].data.clipId. You cannot know a clip's id before you create it, and collision handling may not even have put it on the track you asked for.
• Serialise your renders. Concurrency is 1/3/5 by plan.
• Free credits where you can get them. If your product generates the narration itself, use { script, duration } transcription mode: it costs nothing, needs no provider, and returns the same words[].

Extensions that are one command each#

Once the pipeline above works, most of what a competitor charges for is a single op:

• Beat-cut b-roll: detectBeats client-side (§4.2) → addMarkers → sliceClip times=[…].
• Cut filler words: the transcript already has word timings → cutRanges with ripple: true, which closes the gaps and pulls later clips left.
• Music under speech: addClip the bed, then autoDuck. One command rewrites the music's volume keyframes to dip wherever voice plays, with smooth ramps. A flat music bed over dialogue is the tell of an amateur edit.
• One global look: addAdjustmentLayer, then setColor on the returned clip id. Grades everything below it instead of every clip individually.
• Trim dead air: detectSilence client-side → trimSilence.
• A designed title card: addHtmlElement with your own HTML/CSS. System fonts, gradients, inline SVG and emoji work; scripts and external resources do not (it rasterises through an SVG foreignObject).

Appendix A: every shipped endpoint#

Everything that is live under /v1 today, and nothing that is not.

Not shipped: /v1/ai/analyze, /v1/webhooks. Do not design against either.

Deliberately excluded, permanently or for now: browser recorder (holds a CPU core for ~3 minutes on the box serving the API), inpainting (one global CPU worker capped at 1/user), Flows (an orchestration product, not a primitive), Vidmoat Social (a moderation liability with no revenue; no scope grants it), channels, CDS, admin, billing (internal surfaces with no versioning promise).

Cursor pagination: GET /v1/projects, /v1/renders and /v1/media all return { data, hasMore, nextCursor }. Cursors are opaque base64; do not parse them. limit defaults to 20 and caps at 100.

Appendix B: the versioning contract#

Additive changes ship into v1: new fields, new endpoints, new enum values. Your client must ignore unknown fields and tolerate unknown enum values and error codes. That is the promise you are making in exchange for the one Vidmoat is making, which is that no field already present in a v1 response is removed or repurposed. Breaking changes create /v2, with v1 supported for at least twelve months.

Send Vidmoat-Version: 2026-08-01. It is reserved for date-pinned behaviour inside v1 and is echoed back today.

Questions: developers@vidmoat.com. The portal, keys and interactive docs are at https://developer.vidmoat.com.