Social Video Designer

[To Build] · agent__social-video-designer · Runway Gen-3 Turbo / Kling 1.6

Generates short-form video clips (5–10 seconds) for social media posts with reel, story, or tiktok formats. Uses an image-to-video approach: the existing image pipeline generates a brand-consistent still frame first, then a video model animates it into a clip. Uploads the final MP4 to DigitalOcean Spaces, creates a Media row, and advances the post to client_review.

Related: Social Post Designer · Social Post Writer · AI Image Generation

Overview

Why Two Phases

Video generation APIs are asynchronous — submission returns a job ID, and the result is only available after 30 seconds to 5 minutes. This makes the single-job image pattern unsuitable. The solution is a two-job chain:

Phase 1 — social-video-designer (this worker)
  → Generate brand still frame (reuses image pipeline)
  → Submit still to video API → get jobId
  → Store jobId + poll schedule on SocialPost
  → Enqueue phase-2 poller

Phase 2 — social-video-poller (companion worker)
  → Poll video API every 30s until ready
  → Download MP4 → upload to Spaces
  → Create Media row → advance status

This keeps Phase 1 fast (<30s) and moves the wait entirely into Phase 2.

Supported Formats

Static, carousel, and text formats are not handled by this worker — they remain with the image designer.

Triggers

The dispatch logic lives in apps/api/src/routers/dm/social.ts — the dm-approve handler checks platformFormat and routes to the correct designer.

Input

interface SocialVideoDesignerJobData {
  tenantId:          string;
  activityId:        string;
  wakeReason?:       "new_task" | "rejection";
  reviewerFeedback?: string;
}

Output

No structured return value. Side effects on Phase 2 completion:

• SocialPost.status → "client_review"
• Deliverable.status → "needs_approval"
• Media row created with mediaType: "social_video", mimeType: "video/mp4"
• In-app notification sent to client

On failure (video generation error or timeout):

• SocialPost.status → "dm_review"
• Deliverable.status → "failed"
• Reserved credits released

Pipeline Flow

DM approves copy (reel/story/tiktok format)
  → enqueueSocialVideoDesigner()
    → status: design_pending
      → Credits reserved (15 cr)
        ── Phase 1 ──────────────────────────────────────
        → Generate brand still frame (image pipeline)
        → Submit still + motion prompt to Runway/Kling
        → Receive videoJobId
        → Store videoJobId on SocialPost.designJobId
        → Enqueue social-video-poller (delay: 45s)
        ── Phase 2 (poller) ─────────────────────────────
        → Poll video API every 30s (max 15 attempts)
        → On ready: download MP4
        → Upload to Spaces → Media row created
        → status: client_review
        → Client notified

Client rejects video
  → Old Media row unlinked (preserved in library)
    → Re-enqueued with wakeReason: "rejection"
      → New video generated with rejection feedback

How It Works

Phase 1 — Generate Still Frame

1. Load SocialPost via activityId
2. Load BrandAssets for the tenant
3. Set SocialPost.status → "design_pending", Deliverable.status → "generating"
4. Get dimensions from getDimensions(platform, "reel") → { width: 1080, height: 1920 }
5. Reserve 15 credits via reserveCredits()
6. Reuse buildImagePrompt() from social-post-designer with contentType, brand colours, platform tone — this produces the base still frame
7. Call OpenAIImagesProvider.generateImage() with size "1024x1536" (portrait)
8. Composite brand text overlay and logo onto the still (same as image pipeline)

Phase 2 — Animate Still into Video

9. Build motion prompt (see Motion Prompt Construction below)
10. Submit { imageBuffer, motionPrompt, duration } to the video provider
11. Receive videoJobId
12. Store videoJobId in SocialPost.designJobId (new field — see Schema Change below)
13. Enqueue social-video-poller job with 45-second initial delay

Phase 3 — Poll and Upload (social-video-poller)

14. Call video provider status endpoint with videoJobId
15. If pending or processing: reschedule poll in 30s (max 15 attempts = ~8 min total)
16. If failed: release credits, revert status, throw
17. If succeeded: download MP4 from provider URL
18. Upload MP4 to Spaces via streaming multipart (uploadVideoToSpaces())
19. Create Media row with mediaType: "social_video", mimeType: "video/mp4"
20. Consume 15 credits
21. Advance SocialPost.status → "client_review"
22. Notify client

Motion Prompt Construction

The motion prompt instructs the video model how to animate the still frame. It is separate from the image prompt — the image prompt describes the visual composition; the motion prompt describes movement only.

const MOTION_MAP: Record<string, string> = {
  educational:        "Slow, confident zoom-in on the subject. Subtle environment movement in the background. Professional and calm.",
  promotional:        "Smooth product reveal with gentle rotation. Light flare sweep left to right. Dynamic but clean.",
  inspirational:      "Cinematic slow push-in toward the horizon. Warm light shifts from golden to soft white. Uplifting.",
  announcement:       "Bold graphic elements pulse once. Background shifts subtly. Energetic but brief.",
  engagement:         "Natural handheld-style micro-movement. People in scene gesture naturally. Warm and authentic.",
  "behind-the-scenes":"Handheld feel, slight sway. Scene continues naturally as if candid. Authentic.",
  ugc:                "Natural movement, slight handheld sway. Lifestyle context continues naturally.",
};
 
const PLATFORM_MOTION: Record<string, string> = {
  instagram: "Smooth, elegant motion. 5 seconds. Professional social media aesthetic.",
  tiktok:    "Punchy, dynamic motion. 5–8 seconds. Youth-friendly energy.",
  facebook:  "Calm, approachable motion. 5 seconds. Friendly and accessible.",
};
 
function buildMotionPrompt(opts: {
  platform:    string;
  contentType: string;
  reviewerFeedback?: string | null;
}): string {
  const motion   = MOTION_MAP[opts.contentType] ?? "Gentle natural motion. Professional.";
  const platform = PLATFORM_MOTION[opts.platform] ?? "5 seconds. Professional quality.";
  const feedback = opts.reviewerFeedback
    ? `Previous video was rejected: "${opts.reviewerFeedback}". Address this specifically.`
    : "";
 
  return [motion, platform, feedback, "No text or logo movement. No cuts. Single continuous shot."]
    .filter(Boolean)
    .join(" ");
}

Key constraints in the motion prompt:

• "No text or logo movement" — text and logo are baked into the still frame; they must not animate independently
• "No cuts" — single shot only; Runway/Kling can hallucinate cuts
• Duration is specified per platform
• Motion is content-type appropriate (calm for educational, punchy for TikTok)

Provider Selection

RUNWAY_API_KEY set?
  → Runway Gen-3 Alpha Turbo     ← preferred (best quality)
  → else: FAL_API_KEY set?
    → Kling 1.6 via fal.ai       ← fallback (cheaper, slightly lower quality)
    → else: throw (job fails)

Runway Gen-3 Alpha Turbo

• API style: REST + polling (GET /v1/tasks/{taskId})
• Input: image URL or base64 + text prompt
• Output: MP4 URL (expires 24h — must download immediately)
• Duration: 5 or 10 seconds
• Typical turnaround: 30–90 seconds
• Cost: ~$0.25 per 5s clip

Kling 1.6 via fal.ai

• API style: REST + webhook or polling (GET /v1/queue/status/{requestId})
• Input: image URL + prompt
• Output: MP4 URL
• Duration: 5 seconds
• Typical turnaround: 45–120 seconds
• Cost: ~$0.20 per 5s clip

Schema Changes Required

SocialPost model — new fields

designJobId    String?   // video provider job ID (stored during Phase 1, cleared on completion)
designProvider String?   // "runway" | "kling" (for retry routing)

Media model — new fields / new values

// New mediaType value
mediaType   String   // existing: "social_poster" | NEW: "social_video"
mimeType    String?  // "image/png" (existing) | NEW: "video/mp4"
duration    Int?     // video duration in seconds (null for images)

Migration

// packages/db/prisma/schema.prisma
model SocialPost {
  // ... existing fields ...
  designJobId    String?
  designProvider String?
}
 
model Media {
  // ... existing fields ...
  mimeType    String?
  duration    Int?
}

Storage Path Convention

DigitalOcean Spaces bucket: leadmetrics-media
Path: tenants/{tenantId}/social-videos/{socialPostId}/{nanoid}.mp4

Media.mediaType:   "social_video"
Media.mimeType:    "video/mp4"
Media.platform:    post.platform
Media.format:      "reel" | "story"
Media.slideIndex:  null (videos are single-file, no slides)
Media.duration:    5 or 10 (seconds)
Media.generatedBy: "social-video-designer"
Media.prompt:      motion prompt string (stored for audit)

Video files must be uploaded using streaming multipart upload, not uploadToSpaces() (which buffers entirely in memory). A new uploadVideoToSpaces() function is needed in packages/storage.

Credit Costs

Credits are reserved at Phase 1 start and consumed at Phase 2 completion. If Phase 2 times out or fails, all reserved credits are released.

HITL Gates

Guardrails

Environment Variables

Required in apps/servers/agents/.env:

# Runway Gen-3 (primary)
RUNWAY_API_KEY=...

# fal.ai / Kling (fallback)
FAL_API_KEY=...

# Still frame generation (inherited from image designer)
AZURE_IMAGE_ENDPOINT=...
AZURE_IMAGE_API_KEY=...

# DigitalOcean Spaces (required for upload)
DO_SPACES_KEY=...
DO_SPACES_SECRET=...
DO_SPACES_BUCKET=...
DO_SPACES_ENDPOINT=...
DO_SPACES_CDN_URL=...
DO_SPACES_REGION=...

Error Handling

API Endpoints Needed

POST /dm/v1/social/:id/dm-approve
  → existing, but needs routing logic:
    if (platformFormat in ["reel", "story", "tiktok"]) enqueueSocialVideoDesigner()
    else enqueueSocialPostDesigner()

POST /dm/v1/social/:id/design-reject
  → existing, needs same routing:
    if (platformFormat in ["reel", "story", "tiktok"]) enqueueSocialVideoDesigner({ wakeReason: "rejection" })
    else enqueueSocialPostDesigner({ wakeReason: "rejection" })

No new endpoints required — the existing dm-approve and design-reject handlers need format-aware routing added.

Dashboard UI Changes

• Social post detail page — when Media.mimeType === "video/mp4", render <video> tag instead of <img> with autoplay, muted, loop
• Status badge — add "generating_video" as a distinct status variant with a spinner (distinct from "design_pending" which is used for images)
• Poster carousel — rename to MediaCarousel; handle both image and video media types

Implementation Phases

Open Questions

1. Duration selection — Should the DM choose 5s vs 10s at approval time, or is it fixed per platform? TikTok may benefit from 8–10s; Stories must be ≤15s per platform rules.
2. Audio — Runway and Kling can add ambient sound. Should generated videos be silent (client adds music) or ambient-sound-on by default? Silent is safer for brand control.
3. Aspect ratio for Facebook Reels — Facebook Reels support both 9:16 and 1:1. Currently defaulting to 9:16. Needs a DM choice at approval if both are needed.
4. Thumbnail extraction — Should the first frame be extracted and stored as a social_poster Media row for use as a preview before the video loads?