feat(editor): AI 编辑助手悬浮面板、插件权限体系与时间线渲染管线(M4–M6.3) (#411)

* feat(editor): 编辑器实施 M0-M1 - 剪辑入口占位与插件 SDK v2 垂直切片

- detail.tsx 新增剪辑成片视图(editor tab,常驻),概览页插入剪辑步骤卡
- plugin-types.ts 新增 v2 类型(EditorSlotKind/EditorPluginPermission/PluginManifestV2),修复权限数组交叉归约 bug
- editor-slot-registry.ts 插槽注册表(优先级排序、幂等注册、订阅机制)
- plugin-registry.ts 注册器 v1/v2 分支校验,v1 路径零改动
- editor-shell.tsx 垂直切片插件:timeline-panel 插槽真实渲染,验证插件链路全通
- 测试:插槽注册表 5 用例 + 命令测试骨架,v1 插件回归 21 例零回归

* feat(editor): 命令协议黄金文件 - editor-commands 纯函数 reducer + 8 个黄金 op

* feat(editor): 快照撤销历史 - editor-history 有界 200 层 undo/redo

* feat(editor): 命令状态机 store - dispatch/手势预览/1.5s 防抖保存/串行保存链

* fix(editor): createEditorStore 返回 zustand hook 类型修复 typecheck

* feat(editor): 手势 echo 最小件 - 时间线面板拖拽移动/裁剪 + store 接线与本地持久化

* docs(editor): M2 完成状态记录

* feat(editor): M3.1 时间线面板选中联动 - store 选中状态 + 片段高亮/点击取消

* feat(editor): M3.2 预览监视器 - preview-renderer 插槽 + 真实媒体帧/播放头/时间码

* feat(editor): M3.3 片段检查器 - inspector 插槽 + 属性编辑入队可撤销

* feat(editor): M3.4 asset-ingest 素材库插件(addClip 命令入轨)

* feat(editor): M3.4 补充 - host 上下文(项目信息+资产)、editor.tsx 接线、asset-ingest 纯函数、死代码清理

* feat(editor): M3.5 subtitle-tools - SRT import/export + rebuild from node snapshot

* feat(editor): M3.6 transcription - deterministic mock ASR to subtitle track

* feat(editor): M3.7 export - render plan + ffmpeg.wasm fallback export

* feat(editor): M3.8 - render all 8 editor slots in three-pane layout + ai-assistant placeholder

* docs(editor): M3 完成状态 - 8 插件 + 全插槽接线,冒烟待确认

* feat(editor): 编辑器媒体与时间线打磨 - 资产来源分组筛选、时间线全宽布局、拖拽末片段修复、OSS 降级本地存储、Concat 风格控制条与检查器

* fix(plugins): 合并上游后修复插件分类筛选对 v2 清单的类型兼容

* feat(editor): 时间线工具态与轨道操作打磨 - 工具按钮选中仅图标变色、监视器 Concat 式控制条、removeTrack 黄金命令与轨道行头移除按钮、maskClosable 迁移

* fix(api): 资源上传超 MaxBytesReader 上限时报中文大小限制而非英文裸 400

* fix(editor): 监视器控制条 - 修复播放按钮 JSX 模板闭合花括号缺失导致的编译错误

* feat(api): 本地媒体导入分片化对齐 Concat - >50MB 视频不再失败 - 分片会话三接口(POST开始/PUT单片/complete合并)、前端超50MB自动切8MB分片并失败整传重试、默认配额放宽(日200MB→2GB、总2GB→20GB)、UploadResourceFile 豁免单文件限

* feat(editor): 轨道行头新增可见性/静音开关按钮 - 复用现有图标资源,按轨道类型显隐;命令层新增 setTrackFlag 黄金命令(flag 白名单 fail-closed)并补单测

* docs(progress): 记录轨道可见/静音开关改动与复测项

* docs(editor): M4 转写与渲染任务设计 - 复用 Task 表与状态机、独立创建 API、worker 领取类型过滤、whisper HTTP provider(OpenAI 兼容 /v1/audio)

* fix(api): OSS 上传失败自动降级本地存储时 body 已被 Transport 关闭 - newOSSRequest 用 NopCloser 包装请求体,服务端提前返回(签名 403)时不再关闭调用方持有的文件,降级分支 Seek 重读生效;补回归测试

* docs(progress): 记录媒体导入根治(分片+OSS 降级修复)复测项

* feat(api): 视频播放副本转码 H.265→H.264 + variant=playback 服务与启动回填 - 上传视频后异步探测 codec,H.265/AV1 走 moov 快速探测并转码 H.264 播放副本;GET /api/user-data/...?variant=playback 就绪时服务副本(独立 ETag :pb),未就绪回退原件并撤销后缀防 304 黑屏;storeResource 触发、启动幂等回填存量;前端播放失败自动轮询副本就绪后刷新

* fix(editor): 轨道至少保留一条 - 不可移除轨道点击 X 时居中浮现原因提示(2.2s 自动消失, aria-live);修正可见性 toggle 传参 (visible===false 语义) ;轨道行内移除按钮改为常驻(不可移除时点击给提示)

* feat(timeline): M4.1 字幕转写后台任务 - whisper.cpp 本地执行 - 新增 timeline_transcription 任务类型,worker 按类型分叉独立执行器(不走计费与模型渠道);输入 resourceId+language,读取用户资源后 ffmpeg 预处理 16k 单声道 wav,POST 本地 whisper.cpp /inference,结果写回 segments+srt+language;未配置 CANVAS_WHISPER_BASE_URL 等失败均落明确终态;feature 门控;进度分阶段;含解析/SRT/快速失败单元测试

* feat(media): 素材卡片时长与媒体面板打磨 - 导入媒体解析时长(前端 media-metadata + 后端 project_asset durationMs 透出)随卡片展示;媒体面板/监视器/时间线相关样式细节收尾

* docs(progress): 阶段 C 监视器播放复检清单 - H.265 播放副本/真实播放驱动/时长链路待浏览器确认

* feat(timeline): M4.1 转写任务创建 API - POST /api/timeline/transcriptions - service CreateTimelineTranscriptionTask:IsDraining 守护 + feature 门控 + 资源归属与可转写类型校验 + 排队计 active 限额(含任务级/项目级存储估算);local/whisper.cpp 标记,任务不经过模型路由与计费;createTaskWithinStorageQuota 复用与 ErrActiveTaskLimit 用户可读映射;测试覆盖排队成功(owner/type/provider 断言)、跨用户资源拒绝、非音视频拒绝

* feat(timeline): M4.3 字幕转写前端接线 - 真实任务替换 mock - 新增 services/api/timeline-tasks.ts(POST /timeline/transcriptions,复用 apiClient+request);EditorTranscription 改为 创建任务→waitForGenerationTask 轮询(25min)→结果 segments 映射 SrtEntry[]→rebuildSubtitleClips 写入字幕轨道;实时展示 stage/progress,失败/空识别/无字幕轨道分别提示;素材选择支持音频与视频(取 storageKey 中的 resource id 提交)

* fix(media): 播放副本状态收敛与轮询终态 - 无ffmpeg/远端存储/编码不可处理落 none 避免重复探测;前端 none 停轮询回原生播放并加 processing 上限

* feat(timeline): M4.2 时间线渲染任务 - POST /api/timeline/renders 创建(buildRenderPlan 校验含可渲染媒体、配额入队、local/ffmpeg、60min 超时);worker 顶部按类型分叉 processTimelineRender;ffmpeg 合成产物写资源存储并经 ResultJSON 记 resourceId;render 计划/结果测试 PASS

* fix(resource): 转码原子抢占与 OSS 请求体加固 - ClaimPlaybackTranscode 空/none→processing 单实例抢占避免重复转码,runPlaybackTranscode defer recover 落 failed;If-Range 与带 :pb 后缀的 serveETag 比较;无请求体的 OSS 请求不再包 NopCloser(nil)(防 Go 1.26 body 探测 nil 解引用)

* style(editor): 修正字幕回写面板缩进

* docs(timeline): M4 设计文档按实现回写 - 单一 worker 池 + processClaimedTask 顶部按类型分叉(决策 3/4),端点统一为复数 /api/timeline/transcriptions 与 /api/timeline/renders(注册于 RegisterTaskRoutes),ResultJSON/feature gate/ffmpeg env 同步;Runbook M4 表格与状态行更新;auth.go API 前缀白名单补 timeline

* fix(service): 修复上游 v1.2.5 公告配图删除保护自锁与测试夹具缺口

上游 #385 资源引用走查器对公告配图/草稿做了硬性引用检查,导致三个问题:
- UpdateAnnouncement 在更新公告前对旧配图走查,必然命中当前公告自身引用而自锁,
  公告替换配图功能实际不可用;DiscardAnnouncementImage 同理被草稿自身引用阻塞。
  修复:ensureResourceHasNoBusinessReferences 支持忽略指定引用(当前公告/当前草稿),
  其余引用仍按原语义拦截。
- admin DeleteAdminResources 将公告草稿引用视为阻塞,但删除事务会级联清理草稿与
  资源绑定,service 层永不进入事务,回滚审计注入测试无法到达预期路径。修复:service
  预检时过滤公告草稿引用,与 repository 级联语义对齐。
- project_settings_test.go 夹具未建 ResourceReferenceSnapshot 走查的 Task/Session/
  Message/TaskLog/Result/Announcement/AnnouncementImageDraft 表,报 no such table。
  修复:夹具 AutoMigrate 补齐。

后端全量 go test ./... 通过。

* feat(editor): 成片导出接入后端渲染任务(M4.2 前端接线)- timeline-tasks 增 createTimelineRenderTask;editor-export 主路径改 POST /timeline/renders 轮询取 resourceId 内嵌预览/下载,ffmpeg.wasm 降级保留

* feat(editor): M5 插件权限执行(fail-closed)- plugin-permission-check 新建(8 插槽最小权限表/判定/断言);SlotStack 渲染前过滤缺权限插槽改诊断条;editor-shell manifest 补 timeline.command/export.run;测试 10 用例含 editor-shell 真实回归

* feat(editor): M6.1 AI 命令 schema + M6.2 时间线摘要 - ai-command-schema(parse/dry-run 校验/12-op 提示词契约/版本化+golden);timeline-summary(确定性中文摘要/折叠上限);各 20/10 用例全绿

* docs(progress): M6 质检清单 - M6.1 schema/M6.2 摘要标记完成(含测试计数),M6.3 待模型源确认;清理 M5.1 重复条目

* feat(editor): AI 命令 schema 支持只读问答 - commands 空数组合法化(无法执行/纯提问时 reasoning 直接作答),新增 AI_EDITING_MAX_COMMANDS=8 上限防失控编辑,提示词同步

* feat(editor): AI 助手重构为右上悬浮对话面板(M6.3)+ 素材入库原子化收尾

- editor-ai-assistant.tsx: 占位面板替换为 OpenAI Canvas 风格右上悬浮对话面板;
  面板内文本模型选择器(effective.textModels,选择即持久化并覆写请求模型),
  消息流 user/assistant/error/preview 四类气泡 + 思考中占位,空态引导示例 chips
  一键发送,底部输入条;顶部工具栏常驻 Sparkles 入口(aria-pressed 切换)。
  修复上一版重复/截断函数声明(appendToEntry 双定义、settleAnswer 三声明等)。
  宿主链路:buildAiEditingSystemPrompt(summarizeTimeline(project)) 喂上下文 →
  parseAiCommandPlan → validateAiCommandBatch dry-run(任一非法整批拒绝)→
  ≤3 条 dispatch 直执行 / >3 条预览确认卡片;宿主拒绝时 failTurn 回填 FAILED_MARK 可重试。
- editor-asset-ingest/editor-preview-monitor/detail/editor.tsx: 配套接线与打磨
  (编辑器顶栏 AI 入口、素材导入面板、预览监视、项目页顶栏整理)。
- backend: LinkProjectAsset 事务内原子创建资产 + 项目链接(OnConflict 幂等),
  资产首次入库不再预落库,失败整体回滚,消除"有资产无链接"孤儿记录。
- docs: pending-test.mdx / runbook M6.3 状态更新(tsc + 测试 + 构建通过,
  浏览器冒烟待真实文本渠道)。
This commit is contained in:
爱笑的毛毛虫 authored and GitHub committed 2026-09-04 14:03:33 +08:00
1 parent 33c589fd8e
commit 05d1bce876
106 files changed
+10857 -119

No files matched your search

@@ -69,7 +69,18 @@ func (s *Service) DeleteAdminResources(actor *model.User, req AdminResourceDelet
return nil, snapshotErr
}
for resourceID, references := range adminResourceReferences(snapshot, userResources) {
blockedByID[resourceID] = AdminResourceDeleteBlocked{ID: resourceID, Reason: "资源仍被业务数据引用", References: references}
blocking := make([]AdminResourceReferenceView, 0, len(references))
for _, reference := range references {
if reference.Kind == "公告草稿" {
// 公告草稿及其资源绑定在删除事务内级联清理,不构成阻塞引用。
continue
}
blocking = append(blocking, reference)
}
if len(blocking) == 0 {
continue
}
blockedByID[resourceID] = AdminResourceDeleteBlocked{ID: resourceID, Reason: "资源仍被业务数据引用", References: blocking}
}
}
announcementReferences, err := s.repo.AnnouncementResourceReferences(resourceIDs)
+15 -4
View File
@@ -125,7 +125,7 @@ func (s *Service) UpdateAnnouncement(actor *model.User, id string, req UpdateAnn
}
return nil, err
}
if err := s.ensureResourceHasNoBusinessReferences(oldResource); err != nil {
if err := s.ensureResourceHasNoBusinessReferences(oldResource, repository.ResourceDirectReference{Kind: "公告", ID: announcement.ID}); err != nil {
return nil, err
}
sharedCount, countErr := s.repo.ResourceStorageReferenceCount(oldResource, []string{oldResource.ID})
@@ -316,7 +316,7 @@ func (s *Service) discardAnnouncementImageDraft(userID string, resourceID string
}
return err
}
if err := s.ensureResourceHasNoBusinessReferences(resource); err != nil {
if err := s.ensureResourceHasNoBusinessReferences(resource, repository.ResourceDirectReference{Kind: "公告草稿", ID: resource.ID}); err != nil {
return err
}
sharedCount, err := s.repo.ResourceStorageReferenceCount(resource, []string{resource.ID})
@@ -340,7 +340,7 @@ func (s *Service) discardAnnouncementImageDraft(userID string, resourceID string
return nil
}
func (s *Service) ensureResourceHasNoBusinessReferences(resource *model.Resource) error {
func (s *Service) ensureResourceHasNoBusinessReferences(resource *model.Resource, ignoredDirect ...repository.ResourceDirectReference) error {
if resource == nil {
return NotFound("资源不存在")
}
@@ -348,7 +348,18 @@ func (s *Service) ensureResourceHasNoBusinessReferences(resource *model.Resource
if err != nil {
return err
}
if len(snapshot.Direct) > 0 {
hasBlocking := false
DirectLoop:
for _, direct := range snapshot.Direct {
for _, ignored := range ignoredDirect {
if direct.Kind == ignored.Kind && direct.ID == ignored.ID {
continue DirectLoop
}
}
hasBlocking = true
break
}
if hasBlocking {
return BadAuthRequest("公告配图仍被其他业务数据引用,已停止删除")
}
resourceIDs := map[string]struct{}{resource.ID: {}}
@@ -14,23 +14,25 @@ import (
const featureAvailabilitySettingKey = "feature_availability"
const (
FeatureShortDrama = "shortDrama"
FeatureTaskCenter = "taskCenter"
FeatureCredits = "credits"
FeatureCustomChannels = "customChannels"
FeatureFrontendModels = "frontendModels"
FeaturePluginCenter = "pluginCenter"
FeatureSystemPlugins = "systemPluginsVisibleToUsers"
FeatureShortDrama = "shortDrama"
FeatureTaskCenter = "taskCenter"
FeatureCredits = "credits"
FeatureCustomChannels = "customChannels"
FeatureFrontendModels = "frontendModels"
FeaturePluginCenter = "pluginCenter"
FeatureSystemPlugins = "systemPluginsVisibleToUsers"
FeatureTimelineTranscription = "timelineTranscription"
)
type FeatureAvailability struct {
ShortDramaEnabled bool `json:"shortDramaEnabled"`
TaskCenterEnabled bool `json:"taskCenterEnabled"`
CreditsEnabled bool `json:"creditsEnabled"`
CustomChannelsEnabled bool `json:"customChannelsEnabled"`
FrontendModelsEnabled bool `json:"frontendModelsEnabled"`
PluginCenterEnabled bool `json:"pluginCenterEnabled"`
SystemPluginsVisibleToUsers bool `json:"systemPluginsVisibleToUsers"`
ShortDramaEnabled bool `json:"shortDramaEnabled"`
TaskCenterEnabled bool `json:"taskCenterEnabled"`
CreditsEnabled bool `json:"creditsEnabled"`
CustomChannelsEnabled bool `json:"customChannelsEnabled"`
FrontendModelsEnabled bool `json:"frontendModelsEnabled"`
PluginCenterEnabled bool `json:"pluginCenterEnabled"`
SystemPluginsVisibleToUsers bool `json:"systemPluginsVisibleToUsers"`
TimelineTranscriptionEnabled bool `json:"timelineTranscriptionEnabled"`
}
type PublicFeatureAvailability struct {
@@ -44,13 +46,14 @@ type PublicFeatureAvailability struct {
func defaultFeatureAvailability() FeatureAvailability {
// 缺少配置代表尚未由运维接管;前台模型需要明确配置后才开放。
return FeatureAvailability{
ShortDramaEnabled: true,
TaskCenterEnabled: true,
CreditsEnabled: true,
CustomChannelsEnabled: true,
FrontendModelsEnabled: false,
PluginCenterEnabled: true,
SystemPluginsVisibleToUsers: true,
ShortDramaEnabled: true,
TaskCenterEnabled: true,
CreditsEnabled: true,
CustomChannelsEnabled: true,
FrontendModelsEnabled: false,
PluginCenterEnabled: true,
SystemPluginsVisibleToUsers: true,
TimelineTranscriptionEnabled: true,
}
}
@@ -114,6 +117,8 @@ func (s *Service) FeatureEnabled(feature string) (bool, error) {
return value.PluginCenterEnabled, nil
case FeatureSystemPlugins:
return value.SystemPluginsVisibleToUsers, nil
case FeatureTimelineTranscription:
return value.TimelineTranscriptionEnabled, nil
default:
return false, errors.New("未知功能开放配置")
}
@@ -142,6 +147,8 @@ func (s *Service) RequireFeature(feature string) error {
return Forbidden("插件中心暂未开放")
case FeatureSystemPlugins:
return Forbidden("系统插件暂未向普通用户展示")
case FeatureTimelineTranscription:
return Forbidden("字幕转写暂未开放")
default:
return Forbidden("该功能暂未开放")
}
+80 -6
View File
@@ -2,9 +2,12 @@ package service
import (
"encoding/json"
"errors"
"fmt"
"strings"
"time"
"gorm.io/gorm"
"infinite-canvas/backend/internal/model"
)
@@ -12,6 +15,8 @@ type LinkProjectAssetRequest struct {
AssetID string `json:"assetId"`
Category string `json:"category"`
FolderID *string `json:"folderId"`
// Title 媒体导入场景下由前端携带原始文件名;已有资产时忽略。
Title string `json:"title"`
}
type UpdateProjectAssetRequest struct {
@@ -37,8 +42,10 @@ type ProjectAssetSummary struct {
FolderID string `json:"folderId,omitempty"`
Position int `json:"position"`
StorageKey string `json:"storageKey,omitempty"`
DurationMs int64 `json:"durationMs,omitempty"`
PreviewText string `json:"previewText,omitempty"`
UpdatedAt time.Time `json:"updatedAt"`
Source string `json:"source,omitempty"`
Character *CharacterCardSummary `json:"character,omitempty"`
}
@@ -99,9 +106,17 @@ func (s *Service) LinkProjectAsset(userID string, projectID string, req LinkProj
}
assetID := strings.TrimSpace(req.AssetID)
asset, err := s.repo.AssetForUser(userID, assetID)
if err != nil {
if err != nil && !errors.Is(err, gorm.ErrRecordNotFound) {
return ProjectAssetSummary{}, err
}
if asset == nil {
// 媒体导入只落 resources 表;首次链接时按资源元数据合成资产记录,
// 避免“资源存在但无资产记录”导致导入永远失败。
asset, err = s.assetFromUploadedResource(userID, assetID, req.Title)
if err != nil {
return ProjectAssetSummary{}, err
}
}
category := model.AssetCategory(strings.TrimSpace(req.Category))
if category == "" {
category = asset.Category
@@ -156,6 +171,62 @@ func (s *Service) LinkProjectAsset(userID string, projectID string, req LinkProj
return s.projectAssetSummary(userID, projectID, asset)
}
// assetFromUploadedResource 把媒体导入上传的 Resource 合成为资产记录(内存构造,不落库)。
// 资产创建与项目链接由 LinkProjectAsset 事务原子提交,避免失败留下孤立资产。
func (s *Service) assetFromUploadedResource(userID string, resourceID string, title string) (*model.Asset, error) {
resource, err := s.repo.ResourceForUser(userID, resourceID)
if err != nil {
return nil, err
}
if resource.Kind == "" {
return nil, BadAuthRequest("资源缺少媒体类型,无法导入")
}
payload, err := json.Marshal(map[string]any{
"data": map[string]any{
"storageKey": "resource:" + resource.ID,
"mimeType": resource.MimeType,
"kind": resource.Kind,
"size": resource.Size,
"width": resource.Width,
"height": resource.Height,
"durationMs": resource.DurationMs,
"status": resource.Status,
"mediaType": resource.Kind,
"source": "uploaded",
},
})
if err != nil {
return nil, err
}
title = strings.TrimSpace(title)
if title == "" {
title = mediaTitleFallback(resource)
}
asset := &model.Asset{
ID: resource.ID,
UserID: userID,
Kind: resource.Kind,
Category: model.NormalizeAssetCategory("", resource.Kind),
Status: model.AssetVersionStatusConfirmed,
Title: title,
PayloadJSON: string(payload),
CreatedAt: resource.CreatedAt,
UpdatedAt: resource.UpdatedAt,
}
// 不在此预落库:资产创建交给 LinkProjectAsset 的事务统一提交,
// 事务失败时与链接一起回滚,避免留下“有资产无链接”的孤儿记录。
return asset, nil
}
// mediaTitleFallback 资源无文件名时的展示标题:未命名 + 媒体类型 + 分辨率。
func mediaTitleFallback(resource *model.Resource) string {
base := "未命名" + resource.Kind
if resource.Width > 0 && resource.Height > 0 {
base += fmt.Sprintf(" %dx%d", resource.Width, resource.Height)
}
return base
}
func (s *Service) UnlinkProjectAsset(userID string, projectID string, assetID string) error {
if _, err := s.repo.ProjectForUser(userID, projectID); err != nil {
return err
@@ -482,8 +553,8 @@ func (s *Service) projectAssetSummary(userID string, projectID string, asset *mo
if err != nil {
return ProjectAssetSummary{}, err
}
storageKey, previewText := projectAssetPreview(asset.PayloadJSON)
summary := ProjectAssetSummary{ID: asset.ID, Title: asset.Title, MediaType: asset.Kind, Category: asset.Category, Status: asset.Status, PrimaryVersionID: asset.PrimaryVersionID, VersionCount: len(versions), Usages: usages, FolderID: link.FolderID, Position: link.Position, StorageKey: storageKey, PreviewText: previewText, UpdatedAt: asset.UpdatedAt}
storageKey, previewText, source, durationMs := projectAssetPreview(asset.PayloadJSON)
summary := ProjectAssetSummary{ID: asset.ID, Title: asset.Title, MediaType: asset.Kind, Category: asset.Category, Status: asset.Status, PrimaryVersionID: asset.PrimaryVersionID, VersionCount: len(versions), Usages: usages, FolderID: link.FolderID, Position: link.Position, StorageKey: storageKey, PreviewText: previewText, DurationMs: durationMs, UpdatedAt: asset.UpdatedAt, Source: source}
if asset.Category == model.AssetCategoryCharacter && asset.PrimaryVersionID != "" {
card, cardErr := s.characterCard(userID, asset)
if cardErr != nil {
@@ -494,19 +565,22 @@ func (s *Service) projectAssetSummary(userID string, projectID string, asset *mo
return summary, nil
}
func projectAssetPreview(payloadJSON string) (string, string) {
func projectAssetPreview(payloadJSON string) (string, string, string, int64) {
var payload struct {
Data struct {
StorageKey string `json:"storageKey"`
Content string `json:"content"`
Source string `json:"source"`
DurationMs int64 `json:"durationMs"`
} `json:"data"`
}
if json.Unmarshal([]byte(payloadJSON), &payload) != nil {
return "", ""
return "", "", "", 0
}
previewRunes := []rune(strings.TrimSpace(payload.Data.Content))
if len(previewRunes) > 240 {
previewRunes = append(previewRunes[:240], '…')
}
return strings.TrimSpace(payload.Data.StorageKey), string(previewRunes)
return strings.TrimSpace(payload.Data.StorageKey), string(previewRunes), strings.TrimSpace(payload.Data.Source), payload.Data.DurationMs
}
@@ -0,0 +1,95 @@
package service
import (
"testing"
"infinite-canvas/backend/internal/model"
"infinite-canvas/backend/internal/repository"
"gorm.io/driver/sqlite"
"gorm.io/gorm"
)
func newProjectAssetLinkTestService(t *testing.T) (*Service, *gorm.DB) {
t.Helper()
db, err := gorm.Open(sqlite.Open("file:"+newID()+"?mode=memory&cache=shared"), &gorm.Config{})
if err != nil {
t.Fatal(err)
}
if err := db.AutoMigrate(
&model.Project{},
&model.ProjectAssetLink{},
&model.ProjectAssetFolder{},
&model.Asset{},
&model.AssetVersion{},
&model.AssetRepresentation{},
&model.Resource{},
&model.Shot{},
&model.ShotAssetReference{},
); err != nil {
t.Fatal(err)
}
return &Service{repo: repository.New(db)}, db
}
// TestLinkProjectAssetCreatesAssetFromUploadedResource 覆盖媒体导入场景:
// 上传只落 resources 表,首次链接时服务端应按资源元数据合成资产记录,
// 否则“资源存在但无资产记录”会让导入永远失败。
func TestLinkProjectAssetCreatesAssetFromUploadedResource(t *testing.T) {
service, db := newProjectAssetLinkTestService(t)
project := model.Project{ID: "project-1", UserID: "user-1", Name: "短剧", Status: model.ProjectStatusActive}
resource := model.Resource{
ID: "resource-1",
UserID: "user-1",
Kind: "video",
Status: model.ResourceStatusReady,
MimeType: "video/mp4",
Size: 1024,
Width: 1920,
Height: 1080,
}
for _, item := range []any{&project, &resource} {
if err := db.Create(item).Error; err != nil {
t.Fatalf("seed %T: %v", item, err)
}
}
summary, err := service.LinkProjectAsset("user-1", "project-1", LinkProjectAssetRequest{
AssetID: "resource-1",
Category: "",
Title: "开场.mp4",
})
if err != nil {
t.Fatalf("LinkProjectAsset: %v", err)
}
if summary.ID != "resource-1" {
t.Fatalf("summary.ID = %q, want resource-1", summary.ID)
}
var asset model.Asset
if err := db.First(&asset, "id = ?", "resource-1").Error; err != nil {
t.Fatalf("asset not created: %v", err)
}
if asset.Title != "开场.mp4" {
t.Errorf("asset.Title = %q, want 开场.mp4", asset.Title)
}
if asset.Category != model.AssetCategoryMaterial {
t.Errorf("asset.Category = %q, want material", asset.Category)
}
}
// TestLinkProjectAssetMissingResourceKeepsFailingClosed 资源与资产都不存在时仍应报错,
// 不允许凭空链接。
func TestLinkProjectAssetMissingResourceKeepsFailingClosed(t *testing.T) {
service, db := newProjectAssetLinkTestService(t)
project := model.Project{ID: "project-2", UserID: "user-2", Name: "短剧", Status: model.ProjectStatusActive}
if err := db.Create(&project).Error; err != nil {
t.Fatal(err)
}
if _, err := service.LinkProjectAsset("user-2", "project-2", LinkProjectAssetRequest{
AssetID: "no-such-resource",
}); err == nil {
t.Fatal("LinkProjectAsset succeeded for missing resource, want error")
}
}
@@ -21,6 +21,8 @@ func newProjectSettingsTestService(t *testing.T) (*Service, *gorm.DB) {
&model.Project{}, &model.ProjectUnit{}, &model.Resource{}, &model.Asset{}, &model.AssetVersion{}, &model.AssetRepresentation{},
&model.CanvasProject{}, &model.StyleProfile{}, &model.ProjectAssetCandidate{}, &model.WorkflowInstance{}, &model.WorkflowStepInstance{},
&model.Shot{}, &model.ShotArtifact{}, &model.VoiceProfile{},
&model.Task{}, &model.Session{}, &model.Message{}, &model.TaskLog{}, &model.Result{},
&model.Announcement{}, &model.AnnouncementImageDraft{},
); err != nil {
t.Fatal(err)
}
+94 -17
View File
@@ -16,6 +16,7 @@ import (
_ "image/jpeg"
_ "image/png"
"io"
"log"
"mime"
"mime/multipart"
"net/http"
@@ -308,7 +309,43 @@ func (s *Service) UploadResource(userID string, header *multipart.FileHeader, ki
return resource, err
}
func detectUploadedMimeType(file multipart.File, fileName string, declared string) string {
// UploadResourceFile 接收已完整落盘的本地文件(分片上传合并后调用),语义与 UploadResource 一致;
// 唯一差异是豁免“单文件大小上限”——分片会话已在 handler 按片校验,此处只受日上传与账号存储总量约束。
func (s *Service) UploadResourceFile(userID string, fileName string, size int64, kind string, width int, height int, durationMs int64, file io.ReadSeeker, uploadIdentity ...string) (*model.Resource, error) {
if file == nil || size <= 0 {
return nil, BadAuthRequest("请选择要上传的文件")
}
uploadKey := normalizedResourceUploadKey(uploadIdentity)
existing, err := s.resourceForUploadKey(userID, uploadKey)
if err != nil {
return nil, err
}
if existing != nil && existing.Status == model.ResourceStatusReady {
return existing, nil
}
if existing != nil && existing.Status == model.ResourceStatusPending {
return nil, resourceUploadInProgress()
}
mimeType := detectUploadedMimeType(file, fileName, "")
if existing != nil {
return s.retryStoredResource(userID, existing, kind, mimeType, size, file)
}
day, err := s.reserveChunkedUploadQuota(userID, size)
if err != nil {
return nil, err
}
resource, stored, err := s.storeResource(userID, kind, fileName, mimeType, size, width, height, durationMs, file, uploadKey)
if err != nil {
s.releaseUserUploadQuota(userID, day, size)
} else if stored {
s.commitUserUploadQuota(userID, size)
} else {
s.releaseUserUploadQuota(userID, day, size)
}
return resource, err
}
func detectUploadedMimeType(file io.ReadSeeker, fileName string, declared string) string {
declared = strings.TrimSpace(strings.Split(declared, ";")[0])
if declared != "" && declared != "application/octet-stream" {
return declared
@@ -492,12 +529,7 @@ func (s *Service) storeResource(userID string, kind string, fileName string, mim
return nil, false, err
}
var etag string
if provider == "local" {
filePath := filepath.Join(s.dataDir, "resources", filepath.FromSlash(objectKey))
err = writeLocalResourceObject(filePath, body)
} else {
etag, err = putOSSObject(setting, objectKey, mimeType, size, body)
}
etag, err = s.storeResourceObject(&resource, fileName, body)
resource.UpdatedAt = time.Now()
if err != nil {
resource.Status = model.ResourceStatusFailed
@@ -527,6 +559,7 @@ func (s *Service) storeResource(userID string, kind string, fileName string, mim
return nil, true, errors.Join(err, fmt.Errorf("清理已上传资源对象失败:%w", cleanupErr))
}
s.recordActivity(userID, "resource", 1)
s.maybeStartPlaybackTranscode(&resource)
return &resource, true, nil
}
@@ -546,6 +579,48 @@ func writeLocalResourceObject(filePath string, body io.Reader) error {
return closeErr
}
// storeResourceObject 写入资源物理对象。对象存储不可用(配置错误、密钥失效、网络
// 故障、设置被删)时自动降级为本地存储并同步改写资源记录,保证上传写路径不因外部
// 存储故障整体失败。对象存储失败后 body 会被重新读取,须支持 Seek。
func (s *Service) storeResourceObject(resource *model.Resource, fileName string, body io.Reader) (string, error) {
if resource == nil {
return "", errors.New("资源不存在")
}
if resource.Provider == "local" {
return "", writeLocalResourceObject(filepath.Join(s.dataDir, "resources", filepath.FromSlash(resource.ObjectKey)), body)
}
setting, settingErr := s.ossSettingForResource(resource.UserID, resource)
var etag string
var putErr error
if settingErr == nil {
etag, putErr = putOSSObject(setting, resource.ObjectKey, resource.MimeType, resource.Size, body)
}
if putErr == nil && settingErr == nil {
return etag, nil
}
fallbackErr := putErr
if fallbackErr == nil {
fallbackErr = settingErr
}
if seeker, ok := body.(io.Seeker); ok {
if _, seekErr := seeker.Seek(0, io.SeekStart); seekErr != nil {
return "", errors.Join(fallbackErr, fmt.Errorf("降级本地存储时重置读取位置失败:%w", seekErr))
}
}
localKey := localObjectKey(resource.UserID, resource.Kind, fileName, resource.MimeType, time.Now())
resource.Provider = "local"
resource.ObjectKey = localKey
resource.Endpoint = ""
resource.Bucket = ""
resource.StorageSettingID = ""
resource.ETag = ""
if localErr := writeLocalResourceObject(filepath.Join(s.dataDir, "resources", filepath.FromSlash(localKey)), body); localErr != nil {
return "", errors.Join(fallbackErr, fmt.Errorf("降级本地存储失败:%w", localErr))
}
log.Printf("object storage upload degraded to local storage: resource=%s error=%v", resource.ID, fallbackErr)
return "", nil
}
func (s *Service) retryStoredResource(userID string, resource *model.Resource, kind string, mimeType string, size int64, body io.Reader) (*model.Resource, error) {
if resource == nil {
return nil, errors.New("资源不存在")
@@ -585,15 +660,7 @@ func (s *Service) retryStoredResource(userID string, resource *model.Resource, k
return nil, err
}
var etag string
if resource.Provider == "local" {
err = writeLocalResourceObject(filepath.Join(s.dataDir, "resources", filepath.FromSlash(resource.ObjectKey)), body)
} else {
var setting ossSettingValue
setting, err = s.ossSettingForResource(userID, resource)
if err == nil {
etag, err = putOSSObject(setting, resource.ObjectKey, resource.MimeType, resource.Size, body)
}
}
etag, err = s.storeResourceObject(resource, "", body)
resource.UpdatedAt = time.Now()
if err != nil {
s.releaseRetryUploadQuota(userID, day, size)
@@ -1489,7 +1556,17 @@ func newOSSRequest(method string, setting ossSettingValue, objectKey string, con
return nil, err
}
baseURL.Path = strings.TrimRight(baseURL.Path, "/") + "/" + escapeObjectKey(objectKey)
req, err := http.NewRequest(method, baseURL.String(), body)
// 请求体必须用 no-op close 包装:服务端提前返回(如 OSS 签名 403)时
// http.Transport 会关闭未发完的 Request.Body,若直接传入 *os.File 等
// 调用方持有的文件,后续“降级本地存储”的 Seek 重读将因 file already
// closed 失败。NopCloser 让 Transport 的关闭成为空操作,底层文件保持可用。
// GET/HEAD/DELETE 等无请求体的调用传入 nil body;NopCloser(nil) 会产生非 nil 的
// Body 包装 nil reader,Go 1.26 发送前 body 探测会直接 nil 解引用崩溃。
var reqBody io.Reader
if body != nil {
reqBody = io.NopCloser(body)
}
req, err := http.NewRequest(method, baseURL.String(), reqBody)
if err != nil {
return nil, err
}
@@ -0,0 +1,107 @@
package service
import (
"bytes"
"os"
"path/filepath"
"testing"
"infinite-canvas/backend/internal/model"
"infinite-canvas/backend/internal/repository"
"gorm.io/driver/sqlite"
"gorm.io/gorm"
)
func newResourceFallbackTestService(t *testing.T) (*Service, *gorm.DB) {
t.Helper()
db, err := gorm.Open(sqlite.Open("file:"+newID()+"?mode=memory&cache=shared"), &gorm.Config{})
if err != nil {
t.Fatal(err)
}
if err := db.AutoMigrate(
&model.Resource{},
&model.UserOSSSetting{},
&model.SystemSetting{},
&model.StorageLocation{},
&model.UserDailyActivity{},
); err != nil {
t.Fatal(err)
}
return &Service{repo: repository.New(db), dataDir: t.TempDir()}, db
}
// TestStoreResourceDegradesToLocalWhenOSSUnavailable 覆盖对象存储不可用时的降级:
// 配置了 OSS(enabled=1、endpoint 不可达)时上传应回退本地存储成功,
// 而不是把上传标记为失败——本地媒体导入不应因外部存储故障整体失败。
func TestStoreResourceDegradesToLocalWhenOSSUnavailable(t *testing.T) {
service, db := newResourceFallbackTestService(t)
seedOSSEnabled(t, db, "user-1", "http://127.0.0.1:1")
resource, created, err := service.storeResource(
"user-1", "video", "intro.mp4", "video/mp4", 1024,
1920, 1080, 0, bytes.NewReader([]byte("fake-mp4-bytes")), nil,
)
if err != nil {
t.Fatalf("storeResource: %v", err)
}
if !created {
t.Fatal("storeResource returned created=false, want true")
}
if resource.Status != model.ResourceStatusReady {
t.Fatalf("resource.Status = %q, want %q", resource.Status, model.ResourceStatusReady)
}
if resource.Provider != "local" {
t.Fatalf("resource.Provider = %q, want local (degraded from OSS)", resource.Provider)
}
if resource.Endpoint != "" || resource.Bucket != "" || resource.StorageSettingID != "" {
t.Fatalf("resource storage binding not cleared after degrade: endpoint=%q bucket=%q setting=%q",
resource.Endpoint, resource.Bucket, resource.StorageSettingID)
}
payload, err := os.ReadFile(filepath.Join(service.dataDir, "resources", filepath.FromSlash(resource.ObjectKey)))
if err != nil {
t.Fatalf("local object not written: %v", err)
}
if string(payload) != "fake-mp4-bytes" {
t.Fatalf("local object content = %q, want fake-mp4-bytes", payload)
}
}
// TestStoreResourceLocalPathUnaffectedByOSS 未启用 OSS 时上传仍走本地存储,不受其他用户 OSS 设置影响。
func TestStoreResourceLocalPathUnaffectedByOSS(t *testing.T) {
service, db := newResourceFallbackTestService(t)
seedOSSEnabled(t, db, "user-other", "http://127.0.0.1:1")
resource, created, err := service.storeResource(
"user-2", "video", "local.mp4", "video/mp4", 512,
1280, 720, 0, bytes.NewReader([]byte("local-bytes")), nil,
)
if err != nil {
t.Fatalf("storeResource: %v", err)
}
if !created {
t.Fatal("storeResource returned created=false, want true")
}
if resource.Provider != "local" {
t.Fatalf("resource.Provider = %q, want local", resource.Provider)
}
payload, err := os.ReadFile(filepath.Join(service.dataDir, "resources", filepath.FromSlash(resource.ObjectKey)))
if err != nil {
t.Fatalf("local object not written: %v", err)
}
if string(payload) != "local-bytes" {
t.Fatalf("local object content = %q, want local-bytes", payload)
}
}
func seedOSSEnabled(t *testing.T, db *gorm.DB, userID string, endpoint string) {
t.Helper()
setting := model.UserOSSSetting{
UserID: userID,
Enabled: true,
ValueJSON: `{"provider":"aliyun","endpoint":"` + endpoint + `","bucket":"test-bucket","accessKeyId":"ak-test","accessKeySecret":"sk-plaintext","region":"cn-shenzhen"}`,
}
if err := db.Create(&setting).Error; err != nil {
t.Fatal(err)
}
}
+44 -1
View File
@@ -9,6 +9,8 @@ import (
"net/http"
"net/http/httptest"
"net/url"
"os"
"strconv"
"strings"
"testing"
@@ -95,6 +97,47 @@ func TestAliyunOSSUploadRequestStillUsesEndpointWhenCDNConfigured(t *testing.T)
}
}
func TestNewOSSRequestBodyCloseDoesNotCloseCallerFile(t *testing.T) {
// 服务端提前返回 403(不读 body)时,http.Transport 会关闭 Request.Body。
// newOSSRequest 必须用 no-op close 包装请求体,否则调用方持有的 *os.File
// 会被关掉,OSS 失败后的“降级本地存储”Seek 重读将报 file already closed。
f, err := os.CreateTemp(t.TempDir(), "merged")
if err != nil {
t.Fatal(err)
}
defer f.Close()
if _, err := f.WriteString("payload-bytes"); err != nil {
t.Fatal(err)
}
if _, err := f.Seek(0, io.SeekStart); err != nil {
t.Fatal(err)
}
req, err := newOSSRequest(http.MethodPut, ossSettingValue{
Provider: aliyunOSSProvider, Endpoint: "https://oss-cn-test.aliyuncs.com",
Bucket: "private-bucket", AccessKeyID: "access-id", AccessKeySecret: "secret-value",
}, "users/u-1/video/clip.mp4", "video/mp4", f)
if err != nil {
t.Fatal(err)
}
if req.Body == nil {
t.Fatal("newOSSRequest() body is nil")
}
// 模拟 Transport 在服务端提前响应后关闭请求体:
if err := req.Body.Close(); err != nil {
t.Fatalf("close request body: %v", err)
}
if _, err := f.Seek(0, io.SeekStart); err != nil {
t.Fatalf("caller file was closed by transport: %v", err)
}
got, err := io.ReadAll(f)
if err != nil {
t.Fatal(err)
}
if string(got) != "payload-bytes" {
t.Fatalf("payload after body close = %q", got)
}
}
func TestSignedOSSObjectURLUsesTencentCOSCDNBaseURLWithoutCOSSignature(t *testing.T) {
value, err := signedOSSObjectURL(ossSettingValue{
Provider: tencentCOSProvider, Region: "ap-guangzhou", Bucket: "private-bucket-1250000000",
@@ -962,7 +1005,7 @@ func TestPersistGeneratedMediaAppliesStoredFileQuota(t *testing.T) {
_, err := svc.persistGeneratedMediaResult("user-1", map[string]interface{}{
"image": map[string]interface{}{"dataUrl": "data:image/png;base64,YQ=="},
})
if err == nil || !strings.Contains(err.Error(), "2GB 上限") {
if err == nil || !strings.Contains(err.Error(), "20GB 上限") {
t.Fatalf("persistGeneratedMediaResult() error = %v", err)
}
}
+2 -2
View File
@@ -103,8 +103,8 @@ func defaultRuntimePolicy() RuntimePolicySetting {
ResourceUploadMB: 50,
SessionUploadMB: 32,
GeneratedFileMB: 64,
DailyUploadMB: 200,
StoredFileGB: 2,
DailyUploadMB: 2048,
StoredFileGB: 20,
StructuredDataMB: 256,
TaskDataGB: 1,
AssetCount: 2_000,
+200
View File
@@ -0,0 +1,200 @@
package service
import (
"context"
"encoding/json"
"fmt"
"io"
"os"
"os/exec"
"path/filepath"
"strings"
"time"
"infinite-canvas/backend/internal/model"
)
const renderFfmpegEnv = "CANVAS_FFMPEG_PATH"
// processTimelineRender 执行时间线渲染:按快照把引用的媒体落盘 →
// ffprobe 探测音轨 → ffmpeg concat 合成 → 产物写入资源存储。
// 渲染是本地重编码,不经模型路由与计费;失败一律落明确终态。
func (w *taskWorkerCoordinator) processTimelineRender(task *model.Task, ctx context.Context) error {
s := w.service
ffmpegBin, err := renderFfmpegBinary()
if err != nil {
return w.failTimelineTask(task, "渲染失败", err.Error())
}
var input timelineRenderInput
if err := json.Unmarshal([]byte(task.InputJSON), &input); err != nil {
return w.failTimelineTask(task, "渲染失败", "任务缺少有效的时间线快照")
}
plan := buildRenderPlan(input.Timeline)
if !plan.HasMedia {
return w.failTimelineTask(task, "渲染失败", "时间线没有可渲染的媒体片段")
}
if err := w.progress(task, "准备媒体…", 10); err != nil {
return err
}
workDir, cleanup, err := materializeRenderSources(ctx, s, task.UserID, &plan)
if cleanup != nil {
defer cleanup()
}
if err != nil {
return w.failTimelineTask(task, "渲染失败", err.Error())
}
for _, seg := range plan.Segments {
if seg.Source == nil || seg.Kind == "image" {
continue
}
hasAudio, probeErr := probeHasAudioStream(ctx, seg.Source.Path)
if probeErr != nil {
return w.failTimelineTask(task, "渲染失败", probeErr.Error())
}
seg.Source.HasAudio = hasAudio
}
args := buildRenderFFmpegArgs(plan, filepath.Join(workDir, "render-output.mp4"))
if len(args) == 0 {
return w.failTimelineTask(task, "渲染失败", "无法生成渲染命令")
}
if err := w.progress(task, "正在渲染…", 30); err != nil {
return err
}
cmd := exec.CommandContext(ctx, ffmpegBin, args...)
cmd.Dir = workDir
output, runErr := cmd.CombinedOutput()
if runErr != nil {
detail := strings.TrimSpace(string(output))
if len(detail) > 400 {
detail = detail[len(detail)-400:]
}
return w.failTimelineTask(task, "渲染失败", fmt.Sprintf("ffmpeg 渲染失败:%s", detail))
}
if err := w.progress(task, "写入资源…", 85); err != nil {
return err
}
renderedPath := filepath.Join(workDir, "render-output.mp4")
file, err := os.Open(renderedPath)
if err != nil {
return w.failTimelineTask(task, "渲染失败", "读取渲染产物失败")
}
defer file.Close()
stat, err := file.Stat()
if err != nil || stat.Size() == 0 {
return w.failTimelineTask(task, "渲染失败", "渲染产物为空")
}
durationMs := planDurationMs(plan)
fileName := fmt.Sprintf("timeline-render-%s.mp4", time.Now().Format("20060102-150405"))
resource, _, err := s.storeResource(task.UserID, "media", fileName, "video/mp4", stat.Size(), renderWidth, renderHeight, durationMs, file, nil)
if err != nil || resource == nil {
return w.failTimelineTask(task, "渲染失败", "保存渲染产物失败")
}
result := timelineRenderResult{
ResourceID: resource.ID,
FileName: fileName,
Size: stat.Size(),
DurationMs: durationMs,
SubtitleSRT: plan.SubtitleSRT,
}
payload, err := json.Marshal(result)
if err != nil {
return w.failTimelineTask(task, "渲染失败", "渲染结果序列化失败")
}
task.Status = model.TaskStatusSucceeded
task.Stage = "渲染完成"
task.Progress = 100
task.ResultJSON = string(payload)
completedAt := time.Now()
task.CompletedAt = &completedAt
if err := s.repo.SaveTaskCompletion(task, model.TaskStatusRunning, nil, nil, nil); err != nil {
return fmt.Errorf("写入渲染完成态失败: %w", err)
}
s.logInfo(task.UserID, task.ID, fmt.Sprintf("时间线渲染完成,时长 %.1fs", float64(durationMs)/1000), "")
return nil
}
func planDurationMs(plan renderPlan) int64 {
var total int64
for _, seg := range plan.Segments {
total += seg.DurationMs
}
return total
}
func renderFfmpegBinary() (string, error) {
if configured := strings.TrimSpace(os.Getenv(renderFfmpegEnv)); configured != "" {
return configured, nil
}
path, err := exec.LookPath("ffmpeg")
if err != nil {
return "", fmt.Errorf("渲染依赖未安装(需要 ffmpeg,可通过 %s 指定)", renderFfmpegEnv)
}
return path, nil
}
// materializeRenderSources 把计划中每个媒体片段对应的资源下载到临时目录,
// 同资源复用同一份本地文件;返回工作目录与清理函数。
func materializeRenderSources(ctx context.Context, s *Service, userID string, plan *renderPlan) (string, func(), error) {
tmpDir, err := os.MkdirTemp("", "yingce-render-*")
if err != nil {
return "", nil, fmt.Errorf("创建临时目录失败: %w", err)
}
cleanup := func() { _ = os.RemoveAll(tmpDir) }
cache := map[string]string{}
for i := range plan.Segments {
seg := &plan.Segments[i]
resourceID, ok := mediaResourceID(seg.Clip)
if !ok {
// 无可用媒体引用时渲染为黑场段,保证时间轴连续。
continue
}
path, cached := cache[resourceID]
if !cached {
_, reader, err := s.OpenResource(userID, resourceID)
if err != nil || reader == nil {
cleanup()
return "", nil, fmt.Errorf("无法读取时间线引用的媒体,可能已被删除")
}
ext := extForMime("video/mp4")
path = filepath.Join(tmpDir, fmt.Sprintf("src-%d%s", len(cache), ext))
file, err := os.Create(path)
if err != nil {
reader.Close()
cleanup()
return "", nil, fmt.Errorf("写入临时媒体失败: %w", err)
}
if _, err := io.Copy(file, reader); err != nil {
file.Close()
reader.Close()
cleanup()
return "", nil, fmt.Errorf("读取时间线媒体失败: %w", err)
}
file.Close()
reader.Close()
cache[resourceID] = path
}
seg.Source = &renderSource{ResourceID: resourceID, Clip: seg.Clip, Path: path, Ext: filepath.Ext(path)}
}
return tmpDir, cleanup, nil
}
// probeHasAudioStream 用 ffprobe 判断媒体是否含音轨,供滤镜图选择静音回退。
func probeHasAudioStream(ctx context.Context, path string) (bool, error) {
bin, err := exec.LookPath("ffprobe")
if err != nil {
return false, fmt.Errorf("媒体探测依赖未安装(需要 ffprobe)")
}
cmd := exec.CommandContext(ctx, bin,
"-v", "error", "-select_streams", "a:0", "-show_entries", "stream=index",
"-of", "csv=p=0", path)
output, runErr := cmd.Output()
if runErr != nil {
// 探测失败按无音轨处理,渲染仍可产出静音视频。
return false, nil
}
return strings.TrimSpace(string(output)) != "", nil
}
@@ -0,0 +1,133 @@
package service
import (
"context"
"encoding/json"
"fmt"
"strings"
"testing"
"time"
"gorm.io/gorm"
"infinite-canvas/backend/internal/model"
)
// renderTestProject 构造一个可渲染的 v2 快照:一条可见视频轨 + 一个
// 经 directMedia.storageKey(resource:<id>)引用后端资源的片段。
func renderTestProject(storageKey string) renderProject {
visible := true
return renderProject{
Version: 2,
DurationMs: 2000,
Tracks: []renderTrack{{ID: "track-v1", Kind: "video", Visible: &visible}},
Clips: []renderClip{{
ID: "clip-v1",
Kind: "video",
TrackID: "track-v1",
StartMs: 0,
DurationMs: 2000,
SourceStartMs: 0,
Volume: 1,
DirectMedia: &struct {
ID string `json:"id"`
Kind string `json:"kind"`
StorageKey string `json:"storageKey"`
}{ID: "asset-v1", Kind: "video", StorageKey: storageKey},
}},
}
}
func renderInputJSON(t *testing.T, project renderProject) string {
t.Helper()
raw, err := json.Marshal(timelineRenderInput{ProjectID: "prj-render", Timeline: project})
if err != nil {
t.Fatalf("marshal render input: %v", err)
}
return string(raw)
}
func seedRunningRenderTask(t *testing.T, db *gorm.DB, inputJSON string) *model.Task {
t.Helper()
task := &model.Task{
ID: fmt.Sprintf("tsk-render-%d", time.Now().UnixNano()),
UserID: "usr-render-test",
Type: model.TaskTypeTimelineRender,
Status: model.TaskStatusRunning,
Stage: "已领取",
InputJSON: inputJSON,
}
if err := db.Create(task).Error; err != nil {
t.Fatalf("seed task: %v", err)
}
return task
}
func TestCreateTimelineRenderTaskQueues(t *testing.T) {
svc, db := newTimelineTaskTestService(t)
task, err := svc.CreateTimelineRenderTask("usr-render-test", TimelineRenderCreateRequest{
ProjectID: "prj-render",
Timeline: renderTestProject("resource:res-1"),
})
if err != nil {
t.Fatalf("create: %v", err)
}
if task.Type != model.TaskTypeTimelineRender || task.Status != model.TaskStatusQueued {
t.Fatalf("task = %s/%s, want %s/queued", task.Type, task.Status, model.TaskTypeTimelineRender)
}
if task.Provider != "local" || task.Model != "ffmpeg" {
t.Fatalf("provider/model = %s/%s, want local/ffmpeg", task.Provider, task.Model)
}
var stored model.Task
if err := db.First(&stored, "id = ?", task.ID).Error; err != nil {
t.Fatalf("load task: %v", err)
}
if stored.ProjectID != "prj-render" || stored.UserID != "usr-render-test" {
t.Fatalf("stored owner/project mismatch: %s/%s", stored.UserID, stored.ProjectID)
}
}
func TestCreateTimelineRenderTaskRejectsNoMedia(t *testing.T) {
svc, _ := newTimelineTaskTestService(t)
visible := true
textOnly := renderProject{
Version: 2,
DurationMs: 1000,
Tracks: []renderTrack{{ID: "track-t1", Kind: "text", Visible: &visible}},
Clips: []renderClip{{
ID: "clip-t1", Kind: "text", TrackID: "track-t1", StartMs: 0,
DurationMs: 1000, Text: "字幕",
}},
}
_, err := svc.CreateTimelineRenderTask("usr-render-test", TimelineRenderCreateRequest{Timeline: textOnly})
if err == nil || !strings.Contains(err.Error(), "可渲染") {
t.Fatalf("err = %v, want mention of 没有可渲染的媒体片段", err)
}
}
func TestTimelineRenderFailsFastWhenSourceUnreadable(t *testing.T) {
svc, db := newTimelineTaskTestService(t)
// 指向不存在的资源;ffmpeg 路径设为无效值不影响断言——本测试在
// 素材落盘(materialize)阶段即失败,且该失败早于任何外部命令执行。
task := seedRunningRenderTask(t, db, renderInputJSON(t, renderTestProject("resource:res-missing")))
t.Setenv(renderFfmpegEnv, "/no/such/ffmpeg")
w := newTaskWorkerCoordinator(svc)
if err := w.processTimelineRender(task, context.Background()); err != nil {
t.Fatalf("process: want nil (task terminal handled internally), got %v", err)
}
var stored model.Task
if err := db.First(&stored, "id = ?", task.ID).Error; err != nil {
t.Fatalf("load task: %v", err)
}
if stored.Status != model.TaskStatusFailed {
t.Fatalf("status = %q, want failed", stored.Status)
}
if !strings.Contains(stored.Error, "时间线引用的媒体") {
t.Fatalf("error = %q, want mention of 无法读取时间线引用的媒体", stored.Error)
}
}
+294
View File
@@ -0,0 +1,294 @@
package service
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"os/exec"
"path/filepath"
"strings"
"time"
"infinite-canvas/backend/internal/model"
"infinite-canvas/backend/internal/repository"
)
// timeline 转写任务输入,由画布提交,字段与前端契约一致。
type timelineTranscriptionInput struct {
ResourceID string `json:"resourceId"`
Language string `json:"language"`
}
const whisperLangEnv = "CANVAS_WHISPER_BASE_URL"
// processTimelineTranscription 执行时间线字幕转写:
// 读取资源 -> 本地 ffmpeg 预处理为 16k 单声道 wav -> 本地 whisper.cpp 识别 ->
// 结果写回任务(segments + srt)。任何失败都落到任务终态并携带用户可读原因。
func (w *taskWorkerCoordinator) processTimelineTranscription(task *model.Task, ctx context.Context) error {
s := w.service
baseURL := strings.TrimSpace(os.Getenv(whisperLangEnv))
if baseURL == "" {
return w.failTimelineTask(task, "转写失败", "未配置本地转写服务:请设置 CANVAS_WHISPER_BASE_URL 指向 whisper.cpp 服务")
}
var input timelineTranscriptionInput
if err := json.Unmarshal([]byte(task.InputJSON), &input); err != nil || strings.TrimSpace(input.ResourceID) == "" {
return w.failTimelineTask(task, "转写失败", "任务缺少有效的资源引用")
}
if err := s.RequireFeature(FeatureTimelineTranscription); err != nil {
return w.failTimelineTask(task, "转写失败", "字幕转写暂未开放")
}
s.logInfo(task.UserID, task.ID, "时间线转写任务开始", "")
resource, reader, err := s.OpenResource(task.UserID, input.ResourceID)
if err != nil || reader == nil {
return w.failTimelineTask(task, "转写失败", "无法读取待转写媒体,可能已被删除")
}
defer reader.Close()
if resource == nil || !isTranscribableMime(resource.MimeType) {
return w.failTimelineTask(task, "转写失败", "仅支持音视频文件转写")
}
if err := w.progress(task, "等待转写服务", 15); err != nil {
return err
}
wavPath, cleanup, err := prepareWhisperWav(ctx, reader, resource.MimeType)
if cleanup != nil {
defer cleanup()
}
if err != nil {
return w.failTimelineTask(task, "转写失败", err.Error())
}
if err := w.progress(task, "正在转写…", 40); err != nil {
return err
}
client := newWhisperClient(baseURL)
segments, language, err := client.transcribe(ctx, wavPath, input.Language)
if err != nil {
return w.failTimelineTask(task, "转写失败", err.Error())
}
if err := w.progress(task, "整理字幕…", 80); err != nil {
return err
}
result := timelineTranscriptionResult{
Segments: segments,
SRT: buildTimelineSRT(segments),
Language: language,
}
payload, err := json.Marshal(result)
if err != nil {
return w.failTimelineTask(task, "转写失败", "转写结果序列化失败")
}
task.Status = model.TaskStatusSucceeded
task.Stage = "转写完成"
task.Progress = 100
task.ResultJSON = string(payload)
completedAt := time.Now()
task.CompletedAt = &completedAt
if err := s.repo.SaveTaskCompletion(task, model.TaskStatusRunning, nil, nil, nil); err != nil {
// 冲突/租约已失效时不覆盖他人终态,交由上层判定。
return fmt.Errorf("写入转写完成态失败: %w", err)
}
s.logInfo(task.UserID, task.ID, fmt.Sprintf("时间线转写完成,段落 %d", len(segments)), "")
return nil
}
func (w *taskWorkerCoordinator) failTimelineTask(task *model.Task, stage string, message string) error {
s := w.service
done, err := s.repo.UpdateTaskTerminalState(task.ID, model.TaskStatusRunning, model.TaskStatusFailed, stage, message, time.Now())
if err != nil {
return fmt.Errorf("写入转写失败态失败: %w", err)
}
s.logInfo(task.UserID, task.ID, fmt.Sprintf("时间线转写失败: %s", message), "")
if !done {
return fmt.Errorf("时间线转写已失败: %s", message)
}
return nil
}
func (w *taskWorkerCoordinator) progress(task *model.Task, stage string, progress int) error {
if err := w.service.repo.UpdateTaskProgress(task.ID, stage, progress); err != nil {
return fmt.Errorf("更新转写进度失败: %w", err)
}
return nil
}
func (s *Service) logInfo(userID string, taskID string, message string, extra string) {
s.log(userID, taskID, "info", message, extra)
}
func isTranscribableMime(mime string) bool {
return strings.HasPrefix(mime, "video/") || strings.HasPrefix(mime, "audio/")
}
// prepareWhisperWav 将媒体流经本地 ffmpeg 转为 whisper.cpp 期望的
// 16k 单声道 PCM wav;返回临时文件路径与清理函数。
func prepareWhisperWav(ctx context.Context, reader io.Reader, mime string) (string, func(), error) {
if _, err := exec.LookPath("ffmpeg"); err != nil {
return "", nil, fmt.Errorf("音频预处理依赖未安装(需要 ffmpeg)")
}
tmpDir, err := os.MkdirTemp("", "yingce-whisper-*")
if err != nil {
return "", nil, fmt.Errorf("创建临时目录失败: %w", err)
}
cleanup := func() { _ = os.RemoveAll(tmpDir) }
inPath := filepath.Join(tmpDir, "input"+extForMime(mime))
inFile, err := os.Create(inPath)
if err != nil {
cleanup()
return "", nil, fmt.Errorf("写入待转写媒体失败: %w", err)
}
if _, err := io.Copy(inFile, reader); err != nil {
inFile.Close()
cleanup()
return "", nil, fmt.Errorf("读取待转写媒体失败: %w", err)
}
if err := inFile.Close(); err != nil {
cleanup()
return "", nil, fmt.Errorf("关闭临时文件失败: %w", err)
}
wavPath := filepath.Join(tmpDir, "audio16k.wav")
cmd := exec.CommandContext(ctx, "ffmpeg", "-nostdin", "-y", "-i", inPath, "-vn", "-ac", "1", "-ar", "16000", "-c:a", "pcm_s16le", wavPath)
output, runErr := cmd.CombinedOutput()
if runErr != nil {
cleanup()
detail := strings.TrimSpace(string(output))
if len(detail) > 400 {
detail = detail[len(detail)-400:]
}
return "", nil, fmt.Errorf("音频预处理失败(ffmpeg): %s", detail)
}
return wavPath, cleanup, nil
}
func extForMime(mime string) string {
switch {
case strings.HasPrefix(mime, "video/mp4"), strings.HasPrefix(mime, "audio/mp4"):
return ".mp4"
case strings.HasPrefix(mime, "video/webm"), strings.HasPrefix(mime, "audio/webm"):
return ".webm"
case strings.HasPrefix(mime, "video/quicktime"):
return ".mov"
case strings.HasPrefix(mime, "audio/mpeg"), strings.HasPrefix(mime, "audio/mp3"):
return ".mp3"
case strings.HasPrefix(mime, "audio/wav"), strings.HasPrefix(mime, "audio/x-wav"), strings.HasPrefix(mime, "audio/wave"):
return ".wav"
case strings.HasPrefix(mime, "audio/flac"):
return ".flac"
case strings.HasPrefix(mime, "audio/aac"):
return ".aac"
case strings.HasPrefix(mime, "audio/ogg"), strings.HasPrefix(mime, "video/ogg"):
return ".ogg"
default:
return ".bin"
}
}
// TimelineTranscriptionCreateRequest 是画布提交字幕转写任务的入参。
type TimelineTranscriptionCreateRequest struct {
ResourceID string `json:"resourceId"`
Language string `json:"language"`
ProjectID string `json:"projectId"`
}
// CreateTimelineTranscriptionTask 创建时间线字幕转写任务:转写由本地
// whisper.cpp 执行,不经模型路由与计费;创建时校验功能门控、资源归属
// 与可转写类型,排队计入 active 限额。
func (s *Service) CreateTimelineTranscriptionTask(userID string, req TimelineTranscriptionCreateRequest) (*model.Task, error) {
if s.IsDraining() {
return nil, &AppError{Status: 503, Code: 503, Message: "服务正在维护,暂不接受新的生成任务", Retryable: true}
}
if err := s.RequireFeature(FeatureTimelineTranscription); err != nil {
return nil, err
}
resourceID := strings.TrimSpace(req.ResourceID)
if resourceID == "" {
return nil, BadAuthRequest("必须指定待转写媒体")
}
resource, err := s.Resource(userID, resourceID)
if err != nil || resource == nil {
return nil, BadAuthRequest("无法读取待转写媒体,可能已被删除")
}
if !isTranscribableMime(resource.MimeType) {
return nil, BadAuthRequest("仅支持音视频文件转写")
}
policy, err := s.RuntimePolicy()
if err != nil {
return nil, err
}
input := timelineTranscriptionInput{ResourceID: resourceID, Language: strings.TrimSpace(req.Language)}
inputJSON, _ := json.Marshal(input)
task := model.Task{
ID: newID(), UserID: userID, ProjectID: req.ProjectID,
Type: model.TaskTypeTimelineTranscription, Status: model.TaskStatusQueued,
Stage: "等待队列调度", Progress: 5, Prompt: "字幕转写",
Provider: "local", Model: "whisper.cpp", InputJSON: string(inputJSON),
}
if err := s.createTaskWithinStorageQuota(&task, nil, policy); err != nil {
if errors.Is(err, repository.ErrActiveTaskLimit) {
return nil, BadAuthRequest(fmt.Sprintf("同时排队或运行的任务最多 %d 个,请等待已有任务完成", policy.Task.ActiveTaskLimit))
}
return nil, err
}
s.recordActivity(userID, "task", 1)
_ = s.log(userID, task.ID, "info", "字幕转写任务已进入队列", "")
return taskForOutput(task), nil
}
// TimelineRenderCreateRequest 是画布提交时间线渲染任务的入参;
// Timeline 为前端 TimelineProject 快照(v2:tracks/clips 平铺)。
// 片段通过 directMedia.storageKey=resource:<id> 引用后端资源。
type TimelineRenderCreateRequest struct {
ProjectID string `json:"projectId"`
Timeline renderProject `json:"timeline"`
}
type timelineRenderInput struct {
ProjectID string `json:"projectId"`
Timeline renderProject `json:"timeline"`
}
type timelineRenderResult struct {
ResourceID string `json:"resourceId"`
FileName string `json:"fileName"`
Size int64 `json:"size"`
DurationMs int64 `json:"durationMs"`
SubtitleSRT string `json:"subtitleSrt,omitempty"`
}
// CreateTimelineRenderTask 创建时间线渲染任务:本地 ffmpeg 合成,不经模型
// 路由与计费;创建时校验快照至少包含一个可渲染媒体片段。
func (s *Service) CreateTimelineRenderTask(userID string, req TimelineRenderCreateRequest) (*model.Task, error) {
if s.IsDraining() {
return nil, &AppError{Status: 503, Code: 503, Message: "服务正在维护,暂不接受新的生成任务", Retryable: true}
}
plan := buildRenderPlan(req.Timeline)
if !plan.HasMedia {
return nil, BadAuthRequest("时间线没有可渲染的媒体片段")
}
policy, err := s.RuntimePolicy()
if err != nil {
return nil, err
}
input := timelineRenderInput{ProjectID: strings.TrimSpace(req.ProjectID), Timeline: req.Timeline}
inputJSON, _ := json.Marshal(input)
task := model.Task{
ID: newID(), UserID: userID, ProjectID: strings.TrimSpace(req.ProjectID),
Type: model.TaskTypeTimelineRender, Status: model.TaskStatusQueued,
Stage: "等待队列调度", Progress: 5, Prompt: "时间线渲染",
Provider: "local", Model: "ffmpeg", InputJSON: string(inputJSON),
}
if err := s.createTaskWithinStorageQuota(&task, nil, policy); err != nil {
if errors.Is(err, repository.ErrActiveTaskLimit) {
return nil, BadAuthRequest(fmt.Sprintf("同时排队或运行的任务最多 %d 个,请等待已有任务完成", policy.Task.ActiveTaskLimit))
}
return nil, err
}
s.recordActivity(userID, "task", 1)
_ = s.log(userID, task.ID, "info", "时间线渲染任务已进入队列", "")
return taskForOutput(task), nil
}
@@ -0,0 +1,150 @@
package service
import (
"context"
"fmt"
"strings"
"testing"
"time"
"gorm.io/driver/sqlite"
"gorm.io/gorm"
"infinite-canvas/backend/internal/database"
"infinite-canvas/backend/internal/model"
"infinite-canvas/backend/internal/repository"
)
func newTimelineTaskTestService(t *testing.T) (*Service, *gorm.DB) {
t.Helper()
// 每个测试独立的 sqlite 内存库,避免共享缓存串库。
dsn := fmt.Sprintf("file:%s_%d?mode=memory&cache=shared", t.Name(), time.Now().UnixNano())
db, err := gorm.Open(sqlite.Open(dsn), &gorm.Config{})
t.Helper()
if err != nil {
t.Fatalf("open sqlite: %v", err)
}
if err := db.AutoMigrate(database.Models()...); err != nil {
t.Fatalf("migrate: %v", err)
}
return &Service{repo: repository.New(db)}, db
}
func seedRunningTimelineTask(t *testing.T, db *gorm.DB, inputJSON string) *model.Task {
t.Helper()
task := &model.Task{
ID: fmt.Sprintf("tsk-timeline-%d", time.Now().UnixNano()),
UserID: "usr-timeline-test",
Type: model.TaskTypeTimelineTranscription,
Status: model.TaskStatusRunning,
Stage: "已领取",
InputJSON: inputJSON,
}
if err := db.Create(task).Error; err != nil {
t.Fatalf("seed task: %v", err)
}
return task
}
func TestTimelineTranscriptionFailsFastWhenWhisperUnconfigured(t *testing.T) {
svc, db := newTimelineTaskTestService(t)
task := seedRunningTimelineTask(t, db, `{"resourceId":"res-1","language":""}`)
w := newTaskWorkerCoordinator(svc)
t.Setenv(whisperLangEnv, "")
err := w.processTimelineTranscription(task, context.Background())
if err != nil {
t.Fatalf("process: want nil (task terminal handled internally), got %v", err)
}
var stored model.Task
if err := db.First(&stored, "id = ?", task.ID).Error; err != nil {
t.Fatalf("load task: %v", err)
}
if stored.Status != model.TaskStatusFailed {
t.Fatalf("status = %q, want failed", stored.Status)
}
if !strings.Contains(stored.Error, "CANVAS_WHISPER_BASE_URL") {
t.Fatalf("error = %q, want mention of CANVAS_WHISPER_BASE_URL", stored.Error)
}
}
func TestTimelineTranscriptionRejectsMissingResourceRef(t *testing.T) {
svc, db := newTimelineTaskTestService(t)
task := seedRunningTimelineTask(t, db, `{"resourceId":" "}`)
w := newTaskWorkerCoordinator(svc)
t.Setenv(whisperLangEnv, "http://127.0.0.1:9999")
err := w.processTimelineTranscription(task, context.Background())
if err != nil {
t.Fatalf("process: want nil (task terminal handled internally), got %v", err)
}
var stored model.Task
if err := db.First(&stored, "id = ?", task.ID).Error; err != nil {
t.Fatalf("load task: %v", err)
}
if stored.Status != model.TaskStatusFailed {
t.Fatalf("status = %q, want failed", stored.Status)
}
if !strings.Contains(stored.Error, "资源引用") {
t.Fatalf("error = %q, want mention of 资源引用", stored.Error)
}
}
func seedResource(t *testing.T, db *gorm.DB, id string, userID string, mime string) {
t.Helper()
if err := db.Create(&model.Resource{ID: id, UserID: userID, Kind: "media", Status: model.ResourceStatusReady, Provider: "local", MimeType: mime, Size: 1024}).Error; err != nil {
t.Fatalf("seed resource: %v", err)
}
}
func TestCreateTimelineTranscriptionTaskQueues(t *testing.T) {
svc, db := newTimelineTaskTestService(t)
seedResource(t, db, "res-video-1", "usr-create-test", "video/mp4")
task, err := svc.CreateTimelineTranscriptionTask("usr-create-test", TimelineTranscriptionCreateRequest{ResourceID: "res-video-1", Language: "zh", ProjectID: "prj-1"})
if err != nil {
t.Fatalf("create: %v", err)
}
if task.Type != model.TaskTypeTimelineTranscription {
t.Fatalf("type = %q, want timeline_transcription", task.Type)
}
if task.Status != model.TaskStatusQueued {
t.Fatalf("status = %q, want queued", task.Status)
}
var stored model.Task
if err := db.First(&stored, "id = ?", task.ID).Error; err != nil {
t.Fatalf("load task: %v", err)
}
if stored.UserID != "usr-create-test" || stored.ProjectID != "prj-1" {
t.Fatalf("owner/project mismatch: %q / %q", stored.UserID, stored.ProjectID)
}
if stored.Provider != "local" || stored.Model != "whisper.cpp" {
t.Fatalf("provider/model = %q/%q, want local/whisper.cpp", stored.Provider, stored.Model)
}
}
func TestCreateTimelineTranscriptionTaskRejectsForeignResource(t *testing.T) {
svc, db := newTimelineTaskTestService(t)
seedResource(t, db, "res-owner-a", "usr-a", "video/mp4")
_, err := svc.CreateTimelineTranscriptionTask("usr-b", TimelineTranscriptionCreateRequest{ResourceID: "res-owner-a"})
if err == nil || !strings.Contains(err.Error(), "媒体") {
t.Fatalf("want 无法读取待转写媒体 error, got %v", err)
}
// 另一用户自己的音视频资源可正常创建。
seedResource(t, db, "res-b", "usr-b", "audio/mpeg")
if _, err := svc.CreateTimelineTranscriptionTask("usr-b", TimelineTranscriptionCreateRequest{ResourceID: "res-b"}); err != nil {
t.Fatalf("own audio resource should create: %v", err)
}
}
func TestCreateTimelineTranscriptionTaskRejectsNonTranscribable(t *testing.T) {
svc, db := newTimelineTaskTestService(t)
seedResource(t, db, "res-img", "usr-img", "image/png")
_, err := svc.CreateTimelineTranscriptionTask("usr-img", TimelineTranscriptionCreateRequest{ResourceID: "res-img"})
if err == nil || !strings.Contains(err.Error(), "音视频") {
t.Fatalf("want 仅支持音视频 error, got %v", err)
}
}
+15
View File
@@ -130,6 +130,15 @@ func (w *taskWorkerCoordinator) processClaimedTask(task *model.Task) error {
return terminal.handleAlreadyCancelled(*latest)
}
// 时间线转写由本地 whisper.cpp 执行,不经计费与模型渠道路由,
// 在进入通用生成流程前按类型分叉到独立执行器。
if task.Type == model.TaskTypeTimelineTranscription {
return w.processTimelineTranscription(task, ctx)
}
if task.Type == model.TaskTypeTimelineRender {
return w.processTimelineRender(task, ctx)
}
task.Stage = "调用生成模型"
task.Progress = 35
if taskUsesUpstreamReportedProgress(task.Type) {
@@ -231,6 +240,12 @@ func taskExecutionTimeoutWithPolicy(taskType string, policy RuntimeTaskPolicy) t
return time.Duration(policy.AudioTimeoutMinutes) * time.Minute
case strings.HasPrefix(taskType, "canvas_text"):
return time.Duration(policy.TextTimeoutMinutes) * time.Minute
case taskType == model.TaskTypeTimelineTranscription:
// 本地转写受媒体时长影响,给出独立宽松超时。
return 20 * time.Minute
case taskType == model.TaskTypeTimelineRender:
// 渲染是整条时间线的重编码,耗时随长度线性增长。
return 60 * time.Minute
default:
return time.Duration(policy.DefaultTimeoutMinutes) * time.Minute
}
@@ -0,0 +1,276 @@
package service
import (
"fmt"
"sort"
"strings"
)
// 时间线渲染:把前端 TimelineProject 快照(tracks/clips 平铺,见
// web/src/types/timeline.ts TimelineProject v2)展开为可执行的 ffmpeg 步骤。
// 与前端 timeline-to-ffmpeg.ts 保持同一数据流:trim(按 sourceStartMs 裁切)
// → 空隙补黑场静音 → concat。字幕不烧录(依赖 libass),单独产出 SRT。
//
// 输入布局统一为「每个片段两个输入:视频源 + 音频源」,空隙展开为独立
// 黑场片段,因此第 i 个片段的视频输入为 2i、音频输入为 2i+1。
type renderClip struct {
ID string `json:"id"`
Kind string `json:"kind"`
TrackID string `json:"trackId"`
StartMs int64 `json:"startMs"`
DurationMs int64 `json:"durationMs"`
SourceStartMs int64 `json:"sourceStartMs"`
SourceDurationMs int64 `json:"sourceDurationMs"`
Volume float64 `json:"volume"`
Text string `json:"text"`
DirectMedia *struct {
ID string `json:"id"`
Kind string `json:"kind"`
StorageKey string `json:"storageKey"`
} `json:"directMedia"`
}
type renderTrack struct {
ID string `json:"id"`
Kind string `json:"kind"`
Visible *bool `json:"visible"`
Muted bool `json:"muted"`
}
type renderProject struct {
Version int `json:"version"`
Tracks []renderTrack `json:"tracks"`
Clips []renderClip `json:"clips"`
DurationMs int64 `json:"durationMs"`
}
type renderSource struct {
ResourceID string
Clip renderClip
Path string
Ext string
// HasAudio 由 ffprobe 探测得到;无音轨的媒体改为静音源,避免 map 失败。
HasAudio bool
}
// renderSegment 是渲染序列中的一段;Kind 为 video/image 表示媒体片段,
// gap 表示补黑场静音的空隙段(无 Source)。
type renderSegment struct {
Kind string
DurationMs int64
GapMs int64
Clip renderClip
Source *renderSource
}
type renderPlan struct {
Segments []renderSegment
SubtitleSRT string
HasMedia bool
}
// buildRenderPlan 挑选可见视频/图片轨片段按 startMs 排序,并把片段之间的
// 空隙展开为黑场段,使渲染序列在时间轴上连续。
func buildRenderPlan(project renderProject) renderPlan {
visible := map[string]bool{}
for _, track := range project.Tracks {
visible[track.ID] = track.Visible == nil || *track.Visible
}
clips := make([]renderClip, 0, len(project.Clips))
for _, clip := range project.Clips {
if !visible[clip.TrackID] {
continue
}
if clip.Kind != "video" && clip.Kind != "image" {
continue
}
if clip.DurationMs <= 0 {
continue
}
clips = append(clips, clip)
}
sort.SliceStable(clips, func(i, j int) bool {
if clips[i].StartMs == clips[j].StartMs {
return clips[i].ID < clips[j].ID
}
return clips[i].StartMs < clips[j].StartMs
})
plan := renderPlan{SubtitleSRT: buildRenderSubtitleSRT(project)}
cursor := int64(0)
for _, clip := range clips {
if gap := clip.StartMs - cursor; gap > 0 {
plan.Segments = append(plan.Segments, renderSegment{Kind: "gap", DurationMs: gap, GapMs: gap})
}
plan.Segments = append(plan.Segments, renderSegment{Kind: clip.Kind, DurationMs: clip.DurationMs, Clip: clip})
cursor = clip.StartMs + clip.DurationMs
}
for _, seg := range plan.Segments {
if _, ok := mediaResourceID(seg.Clip); ok {
plan.HasMedia = true
break
}
}
return plan
}
// mediaResourceID 从片段的 directMedia.storageKey(resource:<id>)还原资源 ID。
func mediaResourceID(clip renderClip) (string, bool) {
if clip.DirectMedia == nil {
return "", false
}
key := strings.TrimSpace(clip.DirectMedia.StorageKey)
if !strings.HasPrefix(key, "resource:") {
return "", false
}
id := strings.TrimSpace(strings.TrimPrefix(key, "resource:"))
return id, id != ""
}
func buildRenderSubtitleSRT(project renderProject) string {
subtitle := make([]renderClip, 0, len(project.Clips))
for _, clip := range project.Clips {
if clip.Kind != "subtitle" || strings.TrimSpace(clip.Text) == "" || clip.DurationMs <= 0 {
continue
}
subtitle = append(subtitle, clip)
}
sort.SliceStable(subtitle, func(i, j int) bool {
if subtitle[i].StartMs == subtitle[j].StartMs {
return subtitle[i].ID < subtitle[j].ID
}
return subtitle[i].StartMs < subtitle[j].StartMs
})
if len(subtitle) == 0 {
return ""
}
var out strings.Builder
for i, clip := range subtitle {
fmt.Fprintf(&out, "%d\n%s --> %s\n%s\n\n",
i+1,
formatSRTTimestamp(clip.StartMs),
formatSRTTimestamp(clip.StartMs+clip.DurationMs),
strings.TrimSpace(clip.Text))
}
return out.String()
}
const (
renderWidth = 1920
renderHeight = 1080
renderFPS = 30
renderSampleRate = 44100
)
// buildRenderFFmpegArgs 依据渲染计划生成 ffmpeg 参数(工作目录内相对路径)。
// 每段固定两个输入:视频源与音频源,末尾 concat 为单路输出。
func buildRenderFFmpegArgs(plan renderPlan, target string) []string {
if len(plan.Segments) == 0 {
return nil
}
scale := fmt.Sprintf("scale=%d:%d:force_original_aspect_ratio=decrease,pad=%d:%d:(ow-iw)/2:(oh-ih)/2",
renderWidth, renderHeight, renderWidth, renderHeight)
args := []string{"-nostdin", "-y"}
for _, seg := range plan.Segments {
seconds := fmt.Sprintf("%.3f", float64(seg.DurationMs)/1000)
switch seg.Kind {
case "gap":
args = append(args,
"-f", "lavfi", "-t", seconds,
"-i", fmt.Sprintf("color=c=black:s=%dx%d:r=%d", renderWidth, renderHeight, renderFPS),
"-f", "lavfi", "-t", seconds,
"-i", fmt.Sprintf("anullsrc=r=%d:cl=stereo", renderSampleRate),
)
case "image":
if seg.Source == nil {
args = append(args,
"-f", "lavfi", "-t", seconds,
"-i", fmt.Sprintf("color=c=black:s=%dx%d:r=%d", renderWidth, renderHeight, renderFPS),
"-f", "lavfi", "-t", seconds,
"-i", fmt.Sprintf("anullsrc=r=%d:cl=stereo", renderSampleRate),
)
continue
}
args = append(args,
"-loop", "1", "-t", seconds, "-i", seg.Source.Path,
"-f", "lavfi", "-t", seconds,
"-i", fmt.Sprintf("anullsrc=r=%d:cl=stereo", renderSampleRate),
)
default:
if seg.Source == nil {
args = append(args,
"-f", "lavfi", "-t", seconds,
"-i", fmt.Sprintf("color=c=black:s=%dx%d:r=%d", renderWidth, renderHeight, renderFPS),
"-f", "lavfi", "-t", seconds,
"-i", fmt.Sprintf("anullsrc=r=%d:cl=stereo", renderSampleRate),
)
continue
}
if seg.Clip.SourceStartMs > 0 {
args = append(args, "-ss", fmt.Sprintf("%.3f", float64(seg.Clip.SourceStartMs)/1000))
}
args = append(args, "-t", seconds, "-i", seg.Source.Path)
// 音频输入复用同一媒体文件;无音轨时 ffmpeg 会因 map 失败,
// 由调用方探测后回退为静音源(见 renderAudioFallback)。
if seg.Clip.SourceStartMs > 0 {
args = append(args, "-ss", fmt.Sprintf("%.3f", float64(seg.Clip.SourceStartMs)/1000))
}
args = append(args, "-t", seconds, "-i", seg.Source.Path)
}
}
filters := make([]string, 0, len(plan.Segments)*2+1)
labels := make([]string, 0, len(plan.Segments)*2)
for i, seg := range plan.Segments {
videoIn := 2 * i
audioIn := 2*i + 1
videoLabel := fmt.Sprintf("v%d", i)
audioLabel := fmt.Sprintf("a%d", i)
if seg.Source != nil && seg.Kind != "image" && !seg.Source.HasAudio {
// 无音轨媒体:改用静音源,避免 map 失败。
audioIn = videoIn
filters = append(filters,
fmt.Sprintf("[%d:v]fps=%d,%s,setsar=1[%s]", videoIn, renderFPS, scale, videoLabel))
silentLabel := fmt.Sprintf("silent%d", i)
filters = append(filters,
fmt.Sprintf("anullsrc=r=%d:cl=stereo[%s]", renderSampleRate, silentLabel))
filters = append(filters,
fmt.Sprintf("[%s]atrim=0:%.3f,asetpts=N/SR/TB[%s]", silentLabel, float64(seg.DurationMs)/1000, audioLabel))
} else {
volume := 1.0
if seg.Clip.Volume > 0 {
volume = seg.Clip.Volume
}
filters = append(filters,
fmt.Sprintf("[%d:v]fps=%d,%s,setsar=1[%s]", videoIn, renderFPS, scale, videoLabel))
filters = append(filters,
fmt.Sprintf("[%d:a]aformat=sample_fmts=fltp:sample_rates=%d:channel_layouts=stereo,volume=%.3f,apad[%s]",
audioIn, renderSampleRate, volume, audioLabel))
}
labels = append(labels, videoLabel, audioLabel)
}
var concatInputs strings.Builder
for _, label := range labels {
fmt.Fprintf(&concatInputs, "[%s]", label)
}
filters = append(filters, fmt.Sprintf("%sconcat=n=%d:v=1:a=1[vout][aout]", concatInputs.String(), len(plan.Segments)))
args = append(args,
"-filter_complex", strings.Join(filters, ";"),
"-map", "[vout]", "-map", "[aout]",
"-c:v", "libx264", "-preset", "veryfast", "-crf", "23", "-pix_fmt", "yuv420p",
"-c:a", "aac", "-b:a", "128k", "-movflags", "+faststart",
"-t", fmt.Sprintf("%.3f", planTotalSeconds(plan)),
target,
)
return args
}
func planTotalSeconds(plan renderPlan) float64 {
var total int64
for _, seg := range plan.Segments {
total += seg.DurationMs
}
return float64(total) / 1000
}
@@ -0,0 +1,148 @@
package service
import (
"strings"
"testing"
)
func renderClipFixture(id string, kind string, trackID string, startMs int64, durationMs int64, storageKey string) renderClip {
clip := renderClip{ID: id, Kind: kind, TrackID: trackID, StartMs: startMs, DurationMs: durationMs, Volume: 1}
if storageKey != "" {
clip.DirectMedia = &struct {
ID string `json:"id"`
Kind string `json:"kind"`
StorageKey string `json:"storageKey"`
}{ID: "media-" + id, Kind: kind, StorageKey: storageKey}
}
return clip
}
func TestBuildRenderPlanExpandsGapAndSorts(t *testing.T) {
project := renderProject{
Version: 2,
Tracks: []renderTrack{{ID: "track-video-1", Kind: "video"}},
Clips: []renderClip{
renderClipFixture("clip-b", "video", "track-video-1", 3000, 2000, "resource:res-b"),
renderClipFixture("clip-a", "video", "track-video-1", 0, 2000, "resource:res-a"),
renderClipFixture("clip-sub", "subtitle", "track-subtitle-1", 0, 1000, ""),
},
}
plan := buildRenderPlan(project)
if !plan.HasMedia {
t.Fatal("HasMedia = false, want true")
}
// 前 0–3000ms 被 clip-a(0-2000) 覆盖后仍有 1000ms 空隙,应展开为黑场段。
if len(plan.Segments) != 3 {
t.Fatalf("segments = %d, want 3 (clip-a, gap, clip-b)", len(plan.Segments))
}
if plan.Segments[0].Kind != "video" || plan.Segments[0].Clip.ID != "clip-a" {
t.Fatalf("seg0 = %s/%s, want video/clip-a", plan.Segments[0].Kind, plan.Segments[0].Clip.ID)
}
if plan.Segments[1].Kind != "gap" || plan.Segments[1].DurationMs != 1000 {
t.Fatalf("seg1 = %s/%dms, want gap/1000ms", plan.Segments[1].Kind, plan.Segments[1].DurationMs)
}
if plan.Segments[2].Clip.ID != "clip-b" {
t.Fatalf("seg2 = %s, want clip-b (sorted by startMs)", plan.Segments[2].Clip.ID)
}
}
func TestBuildRenderPlanSkipsHiddenTrack(t *testing.T) {
hidden := false
project := renderProject{
Version: 2,
Tracks: []renderTrack{{ID: "track-hidden", Kind: "video", Visible: &hidden}},
Clips: []renderClip{renderClipFixture("clip-hidden", "video", "track-hidden", 0, 1000, "resource:res-x")},
}
plan := buildRenderPlan(project)
if len(plan.Segments) != 0 {
t.Fatalf("segments = %d, want 0 (hidden track skipped)", len(plan.Segments))
}
if plan.HasMedia {
t.Fatal("HasMedia = true, want false for hidden-only timeline")
}
}
func TestBuildRenderSubtitleSRT(t *testing.T) {
project := renderProject{
Version: 2,
Clips: []renderClip{
{ID: "sub-b", Kind: "subtitle", StartMs: 2000, DurationMs: 1000, Text: "第二条"},
{ID: "sub-a", Kind: "subtitle", StartMs: 0, DurationMs: 1000, Text: "第一条"},
{ID: "sub-blank", Kind: "subtitle", StartMs: 5000, DurationMs: 1000, Text: " "},
},
}
srt := buildRenderSubtitleSRT(project)
want := "1\n00:00:00,000 --> 00:00:01,000\n第一条\n\n2\n00:00:02,000 --> 00:00:03,000\n第二条\n\n"
if srt != want {
t.Fatalf("srt mismatch:\n got %q\nwant %q", srt, want)
}
if buildRenderSubtitleSRT(renderProject{Version: 2}) != "" {
t.Fatal("srt for empty timeline: want empty string")
}
}
func TestBuildRenderFFmpegArgsLayout(t *testing.T) {
plan := buildRenderPlan(renderProject{
Version: 2,
Tracks: []renderTrack{{ID: "track-video-1", Kind: "video"}},
Clips: []renderClip{
renderClipFixture("clip-a", "video", "track-video-1", 0, 2000, "resource:res-a"),
renderClipFixture("clip-b", "video", "track-video-1", 5000, 1000, "resource:res-b"),
},
})
plan.Segments[0].Source = &renderSource{ResourceID: "res-a", Path: "src-0.mp4", HasAudio: true}
plan.Segments[1].Source = &renderSource{ResourceID: "res-a", Path: "src-0.mp4", HasAudio: true}
plan.Segments[2].Source = &renderSource{ResourceID: "res-b", Path: "src-1.mp4", HasAudio: true}
args := buildRenderFFmpegArgs(plan, "render-output.mp4")
joined := strings.Join(args, " ")
if len(args) == 0 {
t.Fatal("args empty, want ffmpeg arguments")
}
for _, want := range []string{
"-filter_complex",
"concat=n=3:v=1:a=1",
"[vout]", "[aout]",
"libx264", "aac",
"color=c=black:s=1920x1080:r=30",
"anullsrc=r=44100:cl=stereo",
"render-output.mp4",
} {
if !strings.Contains(joined, want) {
t.Fatalf("args missing %q: %s", want, joined)
}
}
// 每段两个输入:视频 + 音频;末尾输入索引应为 5(3 段 × 2)。
if !strings.Contains(joined, "[5:a]") {
t.Fatalf("args missing final audio input label [5:a]: %s", joined)
}
if buildRenderFFmpegArgs(renderPlan{}, "out.mp4") != nil {
t.Fatal("args for empty plan: want nil")
}
}
func TestBuildRenderFFmpegArgsSilentFallbackUniqueLabels(t *testing.T) {
plan := renderPlan{Segments: []renderSegment{
{Kind: "video", DurationMs: 1000, Source: &renderSource{Path: "src-0.mp4", HasAudio: false}},
{Kind: "video", DurationMs: 1000, Source: &renderSource{Path: "src-1.mp4", HasAudio: false}},
}}
args := buildRenderFFmpegArgs(plan, "out.mp4")
joined := strings.Join(args, " ")
// 两段均无音轨:静音源标签必须按片段唯一,否则 filter_complex 报错。
if !strings.Contains(joined, "[silent0]") || !strings.Contains(joined, "[silent1]") {
t.Fatalf("silent labels not unique: %s", joined)
}
}
func TestMediaResourceID(t *testing.T) {
clip := renderClipFixture("a", "video", "track-1", 0, 1000, "resource:res-42")
if id, ok := mediaResourceID(clip); !ok || id != "res-42" {
t.Fatalf("mediaResourceID = %q/%v, want res-42/true", id, ok)
}
if _, ok := mediaResourceID(renderClip{}); ok {
t.Fatal("mediaResourceID for clip without directMedia: want false")
}
if _, ok := mediaResourceID(renderClipFixture("a", "video", "track-1", 0, 1000, "https://example.com/a.mp4")); ok {
t.Fatal("mediaResourceID for http storage key: want false")
}
}
@@ -0,0 +1,156 @@
package service
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"math"
"mime/multipart"
"net/http"
"os"
"path/filepath"
"strings"
"time"
)
// 本地 whisper.cpp HTTP 服务客户端。配置项 CANVAS_WHISPER_BASE_URL
// 指向自建 whisper.cpp server(/inference,多部件 file 上传)。
// 语音数据不出本机,未配置该地址时任务在进入转写前明确失败。
type timelineTranscriptionSegment struct {
StartMs int64 `json:"startMs"`
EndMs int64 `json:"endMs"`
Text string `json:"text"`
}
type timelineTranscriptionResult struct {
Segments []timelineTranscriptionSegment `json:"segments"`
SRT string `json:"srt"`
Language string `json:"language"`
}
type whisperSegment struct {
Start float64 `json:"start"`
End float64 `json:"end"`
Text string `json:"text"`
}
type whisperVerboseJSON struct {
Segments []whisperSegment `json:"segments"`
Language string `json:"language"`
}
type whisperClient struct {
baseURL string
http *http.Client
}
func newWhisperClient(baseURL string) *whisperClient {
return &whisperClient{
baseURL: strings.TrimRight(strings.TrimSpace(baseURL), "/"),
http: &http.Client{Timeout: 20 * time.Minute},
}
}
// transcribe 将本地 wav 送至 whisper.cpp /inference,返回段落与语言。
func (c *whisperClient) transcribe(ctx context.Context, wavPath string, language string) ([]timelineTranscriptionSegment, string, error) {
if c.baseURL == "" {
return nil, "", fmt.Errorf("本地转写服务未配置:请设置 CANVAS_WHISPER_BASE_URL")
}
var body bytes.Buffer
writer := multipart.NewWriter(&body)
audio, err := os.Open(wavPath)
if err != nil {
return nil, "", fmt.Errorf("读取待转写音频失败: %w", err)
}
defer audio.Close()
part, err := writer.CreateFormFile("file", filepath.Base(wavPath))
if err != nil {
return nil, "", err
}
if _, err := io.Copy(part, audio); err != nil {
return nil, "", fmt.Errorf("上传音频失败: %w", err)
}
if err := writer.WriteField("response_format", "verbose_json"); err != nil {
return nil, "", err
}
if strings.TrimSpace(language) != "" {
if err := writer.WriteField("language", strings.TrimSpace(language)); err != nil {
return nil, "", err
}
}
if err := writer.Close(); err != nil {
return nil, "", err
}
req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+"/inference", &body)
if err != nil {
return nil, "", err
}
req.Header.Set("Content-Type", writer.FormDataContentType())
resp, err := c.http.Do(req)
if err != nil {
return nil, "", fmt.Errorf("无法连接本地转写服务(whisper.cpp): %w", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
msg, _ := io.ReadAll(io.LimitReader(resp.Body, 2048))
return nil, "", fmt.Errorf("本地转写服务返回 %d: %s", resp.StatusCode, strings.TrimSpace(string(msg)))
}
payload, err := io.ReadAll(io.LimitReader(resp.Body, 8<<20))
if err != nil {
return nil, "", fmt.Errorf("读取转写结果失败: %w", err)
}
segments, languageOut, err := decodeWhisperVerboseJSON(payload)
if err != nil {
return nil, "", err
}
if len(segments) == 0 {
return nil, "", fmt.Errorf("本地转写服务未识别出语音内容")
}
return segments, languageOut, nil
}
// decodeWhisperVerboseJSON 解析 whisper.cpp verbose_json 响应为内部段落,
// 过滤空文本,秒精度取整为毫秒。
func decodeWhisperVerboseJSON(payload []byte) ([]timelineTranscriptionSegment, string, error) {
var parsed whisperVerboseJSON
if err := json.Unmarshal(payload, &parsed); err != nil {
return nil, "", fmt.Errorf("转写结果解析失败: %w", err)
}
segments := make([]timelineTranscriptionSegment, 0, len(parsed.Segments))
for _, seg := range parsed.Segments {
if strings.TrimSpace(seg.Text) == "" {
continue
}
segments = append(segments, timelineTranscriptionSegment{
StartMs: int64(math.Round(seg.Start * 1000)),
EndMs: int64(math.Round(seg.End * 1000)),
Text: strings.TrimSpace(seg.Text),
})
}
return segments, parsed.Language, nil
}
// formatSRTTimestamp 将毫秒时间格式化为 SRT 时间戳 hh:mm:ss,mmm。
func formatSRTTimestamp(ms int64) string {
if ms < 0 {
ms = 0
}
hours := ms / 3600000
minutes := (ms % 3600000) / 60000
seconds := (ms % 60000) / 1000
millis := ms % 1000
return fmt.Sprintf("%02d:%02d:%02d,%03d", hours, minutes, seconds, millis)
}
// buildTimelineSRT 由段落生成 SRT 字幕文本,供前端直接消费或导出。
func buildTimelineSRT(segments []timelineTranscriptionSegment) string {
var out strings.Builder
for i, seg := range segments {
fmt.Fprintf(&out, "%d\n%s --> %s\n%s\n\n", i+1, formatSRTTimestamp(seg.StartMs), formatSRTTimestamp(seg.EndMs), seg.Text)
}
return out.String()
}
@@ -0,0 +1,101 @@
package service
import (
"strings"
"testing"
)
func TestDecodeWhisperVerboseJSON(t *testing.T) {
payload := []byte(`{
"task": "transcribe",
"language": "zh",
"duration": 6.12,
"text": " 大家好 ",
"segments": [
{"id": 0, "start": 0.0, "end": 2.015, "text": " 大家好 "},
{"id": 1, "start": 2.5, "end": 6.12, "text": " "},
{"id": 2, "start": 4.12, "end": 6.121, "text": "今天天气不错"}
]
}`)
segments, language, err := decodeWhisperVerboseJSON(payload)
if err != nil {
t.Fatalf("decode: %v", err)
}
if language != "zh" {
t.Fatalf("language = %q, want zh", language)
}
// 空文本段被过滤,毫秒取整(0.0→0,2.5→2500)。
if len(segments) != 2 {
t.Fatalf("segments = %d, want 2 (blank segment filtered)", len(segments))
}
if segments[0].StartMs != 0 || segments[0].EndMs != 2015 {
t.Fatalf("seg0 = %d-%d, want 0-2015", segments[0].StartMs, segments[0].EndMs)
}
if segments[0].Text != "大家好" {
t.Fatalf("seg0 text = %q, want 大家好 (trimmed)", segments[0].Text)
}
if segments[1].StartMs != 4120 || segments[1].EndMs != 6121 {
t.Fatalf("seg1 = %d-%d, want 4120-6121", segments[1].StartMs, segments[1].EndMs)
}
}
func TestDecodeWhisperVerboseJSONInvalid(t *testing.T) {
if _, _, err := decodeWhisperVerboseJSON([]byte(`{not json`)); err == nil {
t.Fatal("decode invalid payload: want error")
}
// 空段落由 decode 正常返回空切片,守卫在 transcribe 层(避免写出空字幕)。
segments, _, err := decodeWhisperVerboseJSON([]byte(`{"text":"","segments":[]}`))
if err != nil {
t.Fatalf("decode empty segments: unexpected error %v", err)
}
if len(segments) != 0 {
t.Fatalf("decode empty segments: got %d, want 0", len(segments))
}
}
func TestFormatSRTTimestamp(t *testing.T) {
cases := []struct {
ms int64
want string
}{
{0, "00:00:00,000"},
{2015, "00:00:02,015"},
{61001, "00:01:01,001"},
{3661001, "01:01:01,001"},
{-5, "00:00:00,000"},
}
for _, c := range cases {
if got := formatSRTTimestamp(c.ms); got != c.want {
t.Fatalf("formatSRTTimestamp(%d) = %q, want %q", c.ms, got, c.want)
}
}
}
func TestBuildTimelineSRT(t *testing.T) {
segments := []timelineTranscriptionSegment{
{StartMs: 0, EndMs: 2015, Text: "大家好"},
{StartMs: 2500, EndMs: 6121, Text: "今天天气不错"},
}
srt := buildTimelineSRT(segments)
want := "1\n00:00:00,000 --> 00:00:02,015\n大家好\n\n2\n00:00:02,500 --> 00:00:06,121\n今天天气不错\n\n"
if srt != want {
t.Fatalf("srt mismatch:\n got %q\nwant %q", srt, want)
}
}
func TestExtForMimeAndTranscribable(t *testing.T) {
transcribable := []string{"video/mp4", "audio/wav", "audio/mpeg", "video/webm", "audio/flac", "video/quicktime", "audio/aac", "audio/ogg"}
for _, mime := range transcribable {
if !isTranscribableMime(mime) {
t.Fatalf("isTranscribableMime(%q) = false, want true", mime)
}
if strings.TrimPrefix(extForMime(mime), ".") == "bin" {
t.Fatalf("extForMime(%q) fell back to .bin", mime)
}
}
for _, mime := range []string{"image/png", "text/plain", ""} {
if isTranscribableMime(mime) {
t.Fatalf("isTranscribableMime(%q) = true, want false", mime)
}
}
}
+10
View File
@@ -18,6 +18,16 @@ func (s *Service) reserveUserUploadQuota(userID string, size int64) (string, err
return s.reserveUserStoredFileQuota(userID, size, megabytes(policy.Resource.ResourceUploadMB), megabytes(policy.Resource.DailyUploadMB), gigabytes(policy.Resource.StoredFileGB), fmt.Sprintf("单个上传文件必须小于 %dMB", policy.Resource.ResourceUploadMB))
}
// reserveChunkedUploadQuota 用于分片上传完成时预留额度:单文件上限对分片会话不适用(每片已独立校验),
// 仅受“今日上传”与“账号存储总量”约束;singleFileLimit 传 size+1 使单文件上限永不命中。
func (s *Service) reserveChunkedUploadQuota(userID string, size int64) (string, error) {
policy, err := s.RuntimePolicy()
if err != nil {
return "", err
}
return s.reserveUserStoredFileQuota(userID, size, size+1, megabytes(policy.Resource.DailyUploadMB), gigabytes(policy.Resource.StoredFileGB), "")
}
func (s *Service) reserveSessionUploadQuota(userID string, size int64) (string, error) {
policy, err := s.RuntimePolicy()
if err != nil {
@@ -17,12 +17,15 @@ func TestReserveUserUploadQuotaRejectsSingleFileAtLimit(t *testing.T) {
func TestReserveUserUploadQuotaRejectsDailyTotalAtLimit(t *testing.T) {
svc := newResourceTestService(t)
for range 4 {
if _, err := svc.reserveUserUploadQuota("user-1", 49<<20); err != nil {
daily := megabytes(defaultRuntimePolicy().Resource.DailyUploadMB)
chunk := int64(49 << 20)
for used := int64(0); used+chunk <= daily; used += chunk {
if _, err := svc.reserveUserUploadQuota("user-1", chunk); err != nil {
t.Fatal(err)
}
}
if _, err := svc.reserveUserUploadQuota("user-1", 4<<20); err == nil || !strings.Contains(err.Error(), "小于 200MB") {
// 单文件限(50MB)未命中、今日额度已满 → 拒绝并提示每日上限。
if _, err := svc.reserveUserUploadQuota("user-1", chunk); err == nil || !strings.Contains(err.Error(), "小于 2GB") {
t.Fatalf("reserveUserUploadQuota() error = %v", err)
}
}
@@ -63,8 +66,7 @@ func TestReserveUserUploadQuotaRejectsTotalStoredFilesAtLimit(t *testing.T) {
if err := svc.repo.Create(&model.Resource{ID: "resource-1", UserID: "user-1", Status: model.ResourceStatusReady, Size: gigabytes(defaultRuntimePolicy().Resource.StoredFileGB) - 1}); err != nil {
t.Fatal(err)
}
if _, err := svc.reserveUserUploadQuota("user-1", 1); err == nil || !strings.Contains(err.Error(), "2GB 上限") {
t.Fatalf("reserveUserUploadQuota() error = %v", err)
if _, err := svc.reserveUserUploadQuota("user-1", 1); err == nil || !strings.Contains(err.Error(), "20GB 上限") {
}
}
+314
View File
@@ -0,0 +1,314 @@
package service
import (
"bytes"
"encoding/binary"
"fmt"
"io"
"net/http"
"os"
"os/exec"
"path/filepath"
"errors"
"strings"
"infinite-canvas/backend/internal/model"
)
// 播放副本转码:HEVC/H.265 原片在 Chrome/Firefox 等无法解码(<video> 黑屏),
// 上传就绪后若探测到 hvc1/hev1 且本机可用 ffmpeg,则异步转 H.264/AAC 到本地
// playback 目录,供 file 端点 variant=playback 读取。转码只作用于本地原件;
// OSS 原件不转码(避免每次上传拉取远端)。
// ErrPlaybackNotReady 表示资源没有可用的浏览器兼容播放副本(未转码/转码中/失败),
// file 端点应回退 serve 原件。
var ErrPlaybackNotReady = errors.New("播放副本尚未就绪")
const (
playbackDirName = "playback"
videoCodecH264 = "h264"
videoCodecH265 = "h265"
videoCodecAV1 = "av1"
videoCodecVP9 = "vp9"
videoCodecMPEG4 = "mpeg4"
probeMaxMoovSize = 128 << 20
)
// probeVideoCodec 解析本地 mp4 的 stsd 首个视频 sample entry fourcc,返回 h264/h265 等。
// 非 mp4 容器或解析失败返回空串(调用方按“无需转码”处理,前端仍可用原件)。
func probeVideoCodec(path string) string {
f, err := os.Open(path)
if err != nil {
return ""
}
defer f.Close()
// 顶层 box 遍历(跳过 mdat 数据体),定位 moov。
var moovSize int64
var moovData []byte
pos := int64(0)
for {
boxType, size, err := readMP4BoxHeaderAt(f, pos)
if err != nil {
return ""
}
switch boxType {
case "moov":
if size > probeMaxMoovSize {
return ""
}
moovData = make([]byte, size-8)
if _, err := f.ReadAt(moovData, pos+8); err != nil {
return ""
}
moovSize = size
}
pos += size
if moovSize != 0 {
break
}
if size < 8 {
return ""
}
}
return codecFromMoov(moovData)
}
// readMP4BoxHeaderAt 定位文件 pos 处的 box(8/16 字节头),返回类型与总大小。
func readMP4BoxHeaderAt(r io.ReaderAt, pos int64) (string, int64, error) {
var hdr [16]byte
if _, err := r.ReadAt(hdr[:8], pos); err != nil {
return "", 0, err
}
size := int64(binary.BigEndian.Uint32(hdr[:4]))
boxType := string(hdr[4:8])
if size == 1 {
if _, err := r.ReadAt(hdr[8:16], pos+8); err != nil {
return "", 0, err
}
size = int64(binary.BigEndian.Uint64(hdr[8:16]))
}
if size < 8 {
return "", 0, fmt.Errorf("invalid box size %d", size)
}
return boxType, size, nil
}
// codecFromMoov 在 moov 子树中找出所有 stsd,取首个视频 sample entry fourcc。
func codecFromMoov(moov []byte) string {
for _, stsdBody := range boxBodies(moov, "stsd") {
// stsd = fullbox(4) + entry_count(4) + entries…
// 首个 sample entry:entry_size(4) + fourcc(4) → fourcc 在 body+12。
if stsdBody+16 > len(moov) {
continue
}
fourcc := string(moov[stsdBody+12 : stsdBody+16])
switch fourcc {
case "avc1":
return videoCodecH264
case "hvc1", "hev1":
return videoCodecH265
case "av01":
return videoCodecAV1
case "vp09":
return videoCodecVP9
case "mp4v":
return videoCodecMPEG4
}
}
return ""
}
// boxBodies 在 data 中递归查找类型为 want 的 box,返回其 body 起始偏移。
// 只下钻容器 box(moov/trak/mdia/minf/stbl),避免误入 sample entry 内部。
func boxBodies(data []byte, want string) []int {
var out []int
var walk func(start, end int)
walk = func(start, end int) {
pos := start
for pos+8 <= end {
size := int(binary.BigEndian.Uint32(data[pos : pos+4]))
boxType := string(data[pos+4 : pos+8])
hdr := 8
if size == 1 {
if pos+16 > end {
return
}
size = int(binary.BigEndian.Uint64(data[pos+8 : pos+16]))
hdr = 16
}
if size < 8 || pos+size > end {
return
}
if boxType == want {
out = append(out, pos+hdr)
}
switch boxType {
case "moov", "trak", "mdia", "minf", "stbl":
walk(pos+hdr, pos+size)
}
pos += size
}
}
walk(0, len(data))
return out
}
// maybeStartPlaybackTranscode 由上传就绪路径调用;H.265 且 ffmpeg 可用时置
// processing 并异步转码。H.264 直接标记 none,避免每次上传重复探测。
func (s *Service) maybeStartPlaybackTranscode(resource *model.Resource) {
if resource == nil || resource.Kind != "video" {
return
}
// 远端存储(OSS 等)不落本地副本:标 none 表示无需转码,前端按原生播放/直链处理,
// 避免状态留空被当成 processing 无限轮询(markPlaybackNone 幂等)。
if resource.Provider != "local" || resource.Status != model.ResourceStatusReady {
if resource.Provider != "local" {
markPlaybackNone(s, resource)
}
return
}
if resource.PlaybackStatus != "" && resource.PlaybackStatus != model.PlaybackStatusNone {
return
}
// 不可转码场景(无 ffmpeg)落 none:状态为空会被前端当成 processing 无限轮询,
// 也会让启动回填每次重试探测。none 表示"无需转码",按原生播放处理。
if _, err := exec.LookPath("ffmpeg"); err != nil {
markPlaybackNone(s, resource)
return
}
src := filepath.Join(s.dataDir, "resources", filepath.FromSlash(resource.ObjectKey))
switch probeVideoCodec(src) {
case videoCodecH265:
// 原子抢占(空/none → processing):并发上传 + 回填、多实例同抢时
// 仅一个能成功置位,其余直接返回,避免重复转码。
claimed, err := s.repo.ClaimPlaybackTranscode(resource.ID)
if err != nil || !claimed {
return
}
resource.PlaybackStatus = model.PlaybackStatusProcessing
go s.runPlaybackTranscode(resource.UserID, resource.ID, src)
case videoCodecH264, videoCodecAV1, videoCodecVP9, videoCodecMPEG4, "":
// H.264 浏览器可直接解码;AV1/VP9/MPEG4 暂不转码;探针读不出编码(非 mp4 /
// moov 在尾部 / 加密容器)也无法处理 —— 均标 none,避免重复探测与前端无限轮询。
markPlaybackNone(s, resource)
}
}
// markPlaybackNone 将资源标记为无需播放副本(幂等,写失败不影响上传主流程)。
func markPlaybackNone(s *Service, resource *model.Resource) {
if resource.PlaybackStatus == model.PlaybackStatusNone {
return
}
resource.PlaybackStatus = model.PlaybackStatusNone
_ = s.repo.SaveResource(resource)
}
// runPlaybackTranscode 转码本地原件到 playback/<id>.mp4 并回写状态(幂等按 id 重载)。
func (s *Service) runPlaybackTranscode(userID string, resourceID string, src string) {
// 转码 goroutine 意外 panic 时把状态落 failed,避免 processing 卡死到下次重启。
defer func() {
if r := recover(); r != nil {
if res, err := s.repo.ResourceForUser(userID, resourceID); err == nil && res != nil {
res.PlaybackStatus = model.PlaybackStatusFailed
res.PlaybackError = clipText(fmt.Sprintf("转码 panic:%v", r), 1000)
_ = s.repo.SaveResource(res)
}
}
}()
status := model.PlaybackStatusFailed
objectKey := ""
var errText string
dst := filepath.Join(s.dataDir, playbackDirName, resourceID+".mp4")
// 副本已存在且为 H.264(上次转码完成但回写随进程崩溃丢失):直接置 ready 不重转。
if probeVideoCodec(dst) == videoCodecH264 {
status = model.PlaybackStatusReady
objectKey = resourceID + ".mp4"
} else if err := runH264Transcode(src, dst); err != nil {
errText = clipText(err.Error(), 1000)
} else {
status = model.PlaybackStatusReady
objectKey = resourceID + ".mp4"
}
res, err := s.repo.ResourceForUser(userID, resourceID)
if err != nil || res == nil {
return
}
res.PlaybackStatus = status
res.PlaybackObjectKey = objectKey
res.PlaybackError = errText
_ = s.repo.SaveResource(res)
}
// runH264Transcode 用 ffmpeg 将任意输入转为 H.264/AAC mp4(faststart、yuv420p、偶数尺寸)。
func runH264Transcode(src string, dst string) error {
if err := os.MkdirAll(filepath.Dir(dst), 0o750); err != nil {
return err
}
cmd := exec.Command("ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
"-i", src,
"-c:v", "libx264", "-preset", "veryfast", "-crf", "23",
"-pix_fmt", "yuv420p", "-vf", "scale=trunc(iw/2)*2:trunc(ih/2)*2",
"-c:a", "aac", "-b:a", "128k", "-movflags", "+faststart",
dst)
var stderr bytes.Buffer
cmd.Stderr = &stderr
if err := cmd.Run(); err != nil {
msg := strings.TrimSpace(stderr.String())
if msg == "" {
msg = err.Error()
}
return fmt.Errorf("ffmpeg 转码失败:%s", clipText(msg, 800))
}
return nil
}
func clipText(s string, max int) string {
if len(s) <= max {
return s
}
return s[:max]
}
// OpenResourcePlaybackRange 打开浏览器兼容播放副本(本地 ffmpeg 转码的 H.264)。
// 仅当资源为本地存储且副本 ready 时可用;否则返回 ErrPlaybackNotReady,调用方回退原件。
func (s *Service) OpenResourcePlaybackRange(userID string, resourceID string) (*ResourceStream, error) {
resource, err := s.repo.ResourceForUser(userID, resourceID)
if err != nil {
return nil, err
}
if resource == nil || resource.Status != model.ResourceStatusReady || resource.Provider != "local" ||
resource.PlaybackStatus != model.PlaybackStatusReady || resource.PlaybackObjectKey == "" {
return nil, ErrPlaybackNotReady
}
body, err := os.Open(filepath.Join(s.dataDir, playbackDirName, filepath.FromSlash(resource.PlaybackObjectKey)))
if err != nil {
return nil, err
}
playback := *resource
playback.MimeType = "video/mp4"
playback.ObjectKey = filepath.Join(playbackDirName, resource.PlaybackObjectKey)
size := int64(0)
if st, err := body.Stat(); err == nil {
size = st.Size()
}
return &ResourceStream{Resource: &playback, Body: body, StatusCode: http.StatusOK, ContentLength: size, AcceptRanges: "bytes"}, nil
}
// BackfillPlaybackTranscodes 在服务启动后扫描存量本地视频:未判定 codec 的补判定,
// H.265 触发转码、H.264 标记 none。幂等:maybeStartPlaybackTranscode 先置
// processing/none 再入库,重复扫描不会重复转码。
func (s *Service) BackfillPlaybackTranscodes() {
// 上次进程可能崩溃在转码中途(状态卡 processing),先重置为待判定。
_ = s.repo.ResetStuckPlaybackTranscodes()
for {
resources, err := s.repo.PlaybackPendingVideos(20)
if err != nil || len(resources) == 0 {
return
}
for i := range resources {
s.maybeStartPlaybackTranscode(&resources[i])
}
}
}
@@ -0,0 +1,65 @@
package service
import (
"encoding/binary"
"os"
"path/filepath"
"testing"
)
// stsdBox 构造仅含 first-sample-entry fourcc 的最小 stsd box(置于 moov 切片内)。
func stsdBoxWithFourcc(fourcc string) []byte {
b := make([]byte, 24)
binary.BigEndian.PutUint32(b[0:4], 24)
copy(b[4:8], "stsd")
// body: version/flags(4) + entry_count(4)=1 + entry_size(4) + fourcc(4)
binary.BigEndian.PutUint32(b[12:16], 1)
binary.BigEndian.PutUint32(b[16:20], 8)
copy(b[20:24], fourcc)
return b
}
func TestCodecFromMoov(t *testing.T) {
cases := []struct {
fourcc string
want string
}{
{"hvc1", videoCodecH265},
{"hev1", videoCodecH265},
{"avc1", videoCodecH264},
{"av01", videoCodecAV1},
{"vp09", videoCodecVP9},
}
for _, c := range cases {
if got := codecFromMoov(stsdBoxWithFourcc(c.fourcc)); got != c.want {
t.Errorf("codecFromMoov(%s) = %q, want %q", c.fourcc, got, c.want)
}
}
if got := codecFromMoov([]byte{0, 1, 2, 3}); got != "" {
t.Errorf("garbage moov codec = %q, want empty", got)
}
}
func TestProbeVideoCodecReadsRealFile(t *testing.T) {
// 构造 ftyp + moov(含 hvc1 stsd) 的最小 mp4 文件,验证 probeVideoCodec 走文件读取路径。
stsd := stsdBoxWithFourcc("hvc1")
moov := make([]byte, 8+len(stsd))
binary.BigEndian.PutUint32(moov[0:4], uint32(len(moov)))
copy(moov[4:8], "moov")
copy(moov[8:], stsd)
full := make([]byte, 0, 16+len(moov))
ftyp := make([]byte, 16)
binary.BigEndian.PutUint32(ftyp[0:4], 16)
copy(ftyp[4:8], "ftyp")
copy(ftyp[8:12], "isom")
full = append(full, ftyp...)
full = append(full, moov...)
path := filepath.Join(t.TempDir(), "clip.mp4")
if err := os.WriteFile(path, full, 0o644); err != nil {
t.Fatal(err)
}
if got := probeVideoCodec(path); got != videoCodecH265 {
t.Errorf("probeVideoCodec = %q, want h265", got)
}
}