feat(agent): 创作能力 - 接通专业节点发现与插件执行基础
This commit is contained in:
王志伟
@@ -7,6 +7,8 @@ description: 面向图片创作的 AI 视觉分析、问题定位和修改建议
|
||||
|
||||
> 状态：MVP 已实现，当前审美标准为 `2026-08-v1`。
|
||||
|
||||
执行入口已统一到 `art-critique-execution`：手动分析各阶段通过后台文本任务提交，原图使用资源引用，节点记录阶段任务 ID。一次分析最多 9 个不同模型阶段，不自动重复提交。停止观察不等于取消后台任务，已提交阶段仍可在任务中心查看；自动费用审批及整条管线刷新恢复仍待接入。Agent 当前支持准备输入和读取已有报告，尚未开放自动启动收费分析。
|
||||
|
||||
## 1. 先说结论
|
||||
|
||||
这个节点的定位不是“图片生成器”，而是一个视觉老师：用户把图片接进来，它负责看图、指出问题、解释原因，并给出可以执行的修改方向。
|
||||
|
||||
58
docs/design/agent-capabilities.mdx
Normal file
58
docs/design/agent-capabilities.mdx
Normal file
@@ -0,0 +1,58 @@
|
||||
---
|
||||
title: Agent 能力发现与插件动作
|
||||
description: 在线画布助手的节点发现、技能渐进读取和插件动作接入合同
|
||||
---
|
||||
|
||||
# Agent 能力发现与插件动作
|
||||
|
||||
在线画布助手通过 `canvas_list_capabilities` 查询实时节点注册表和有效启用的插件。内置节点及插件贡献不再依赖助手中的固定节点类型枚举。发现返回名称、来源、输入约束、默认数据和插件提供的动作；发现到声明不代表专用执行器已经存在。
|
||||
|
||||
## 技能自动发现
|
||||
|
||||
`canvas_list_skills` 支持 `query`、`offset`、`limit`，默认每页 40 条、最多 100 条，返回 `items`、`total`、`nextOffset`。助手先根据任务检索元数据，再读取 `SKILL.md`，按入口引用搜索、读取关联文件。无需用户手动选中技能；是否选用由模型结合任务判断，不保证每次都调用技能。
|
||||
|
||||
在线技能工具执行时重新取得当前已加入技能目录，复用账号隔离缓存和安装、移除时的缓存失效机制。插件文档可通过 `canvas_read_plugin` 按已发现 ID 读取。文档和技能说明不扩大用户授权，也不意味着浏览器可以执行技能包中的任意脚本。
|
||||
|
||||
## 专业短剧节点
|
||||
|
||||
创作方案和通用工作流支持 `styleboard`、`story_input`、`script`，复用手动短剧入口的数据结构。画风节点初始为空，显示系统已有的选择入口；故事节点使用专用编辑卡片；分镜节点使用真实表格。
|
||||
|
||||
`script.shots` 可填写逐镜 `durationSeconds`、`videoMotionPrompt`、`dialogue`，最多 100 行，写入稳定行 ID。方案连到分镜节点时使用 `storyboard:context` 输入。仅创建这些节点不产生媒体生成项；图像、视频仍沿用既有方案和费用确认。
|
||||
|
||||
`canvas_list_styles` 按 `source=system|user` 查询系统预设或当前用户保存的风格。`canvas_apply_style` 只接受该目录返回的真实 ID，复用页面的异步画风保存入口：关联项目时先保存项目设置，再应用画布节点；不关联项目时走原画布保存机制。异步返回时校验画布、项目及用户上下文，避免写到新页面。该动作是原画风选择逻辑，不冒充普通节点撤销能够回滚服务端项目设置；需要改回时重新应用原风格。
|
||||
|
||||
`canvas_read_storyboard` 返回真实行 ID、镜头内容及素材关联。`canvas_edit_storyboard` 支持 `append`、`update`、`remove`，按行 ID 操作；仅允许镜头文本与正数时长字段，不允许伪造任务状态或素材 ID。移除行保留生成素材，剩余行保留 ID 并重排序号。分镜编辑进入原画布操作、撤销与历史记录链路，不自动重新生成媒体。
|
||||
|
||||
## 插件动作接入
|
||||
|
||||
已在宿主注册的 `RegisteredPlugin` 可提供 `agentActions`：
|
||||
|
||||
```ts
|
||||
agentActions: [{
|
||||
id: "create-analysis-node",
|
||||
description: "创建图片分析节点",
|
||||
inputSchema: { type: "object", properties: {}, additionalProperties: false },
|
||||
buildOperations(input, snapshot) {
|
||||
// 在此校验本动作入参，仅返回操作，禁止请求模型或产生其他副作用。
|
||||
return [{ type: "add_node", nodeType: "my-analysis-node" }];
|
||||
},
|
||||
}]
|
||||
```
|
||||
|
||||
模型根据目录通过 `canvas_plugin_action` 提交 `pluginId`、`actionId`、`input`。动作构建器负责业务入参校验；宿主核对插件有效启用状态、动作存在性、`canvas.write` 权限、操作数量及生成操作的 `generation.run` 声明，再交给原画布执行链路。工具调用和结果沿用原会话记录。
|
||||
|
||||
构建器必须是纯函数：输入和快照以副本传入，返回 1 至 100 个操作，不自行网络请求、收费或写入资源。`inputSchema` 用于向模型说明合同，不替代动作构建器的校验。仅上传带节点声明的 JSON 插件不会凭空获得 JavaScript 执行器；执行器需要通过宿主支持的插件运行时接入。本接口没有增加任意远程代码执行能力。
|
||||
|
||||
插件还可提供 `readAgentNode(node, snapshot)`，将节点状态转换为有限的结构化结果，通过 `canvas_read_plugin_node` 读取。必须校验插件启用及 `canvas.read` 声明，不返回内嵌媒体和密钥。审美批改、肖像排查已接入 `prepare-input` 动作和状态读取；同一图片已有对应节点时复用，不重复创建。审美报告仅在当前唯一输入与报告指纹一致时返回，否则标为过期。
|
||||
|
||||
节点类型冲突会拒绝注册。插件停用后不再出现在能力目录，新建节点和调用插件动作都会失败，旧作品保留。未知节点类型不能降级为普通文本。
|
||||
|
||||
## 当前边界与后续接入
|
||||
|
||||
审美分析的手动入口已接入 `art-critique-execution` 统一执行边界。各阶段通过现有后台文本任务运行，原图先准备为资源引用，任务正文不内嵌媒体。最多提交 9 个不同阶段，不自动重复提交同一阶段；节点保存 `stageTaskIds`，任务元数据记录节点、执行和阶段标识。底层 pipeline 支持注入阶段执行器，费用审批可在这一入口接入，不复制审美判断与解析逻辑。
|
||||
|
||||
停止观察不代表后台任务取消或未收费。已提交任务保留在任务中心；本批没有自动恢复整条管线或重新提交失败阶段。画布输入改变后停止后续观察，不将旧结果写到新输入。Agent 目前仍不能自动启动该收费分析，须先接通具体阶段报价、用户批准和任务接续。
|
||||
|
||||
本批接入在线画布助手。首页尚未改造成通用工具循环，本机 Agent 仍使用原技能包交付机制。现有插件只有声明而没有 `agentActions` 时，可发现贡献并创建节点，不宣称能自动执行它的专用分析、编辑或导出。
|
||||
|
||||
已补齐画风预设的模型选择与项目同步、分镜专用增删改。审美批改与肖像排查的新分析仍通过插件自己的入口执行：前者涉及多次模型调用，后者涉及本机任务，尚未接入统一 Agent 费用和任务编排。后续继续接通这些执行器，以及首页与本机端的同一能力目录。不得以任意 `metadata` 更新替代具有业务校验和费用语义的执行接口。
|
||||
@@ -21,9 +21,10 @@ import {
|
||||
type ArtCritiqueReport,
|
||||
} from "@/lib/art-critique/contracts";
|
||||
import { ART_CRITIQUE_CATEGORY_COLORS, ART_CRITIQUE_SEVERITY_COLORS, layoutArtCritiqueLabels, repairIssueTarget, targetBounds } from "@/lib/art-critique/annotation";
|
||||
import { runArtCritiquePipeline } from "@/lib/art-critique/pipeline";
|
||||
import { executeArtCritique } from "@/services/art-critique-execution";
|
||||
import { getActiveUserScope } from "@/lib/user-scope";
|
||||
import { useCopyText } from "@/hooks/use-copy-text";
|
||||
import { imageToDataUrl, resolveImageUrl } from "@/services/image-storage";
|
||||
import { resolveImageUrl } from "@/services/image-storage";
|
||||
import { modelOptionLabel, useEffectiveConfig } from "@/stores/use-config-store";
|
||||
import { usePluginStore } from "@/stores/use-plugin-store";
|
||||
import { useThemeStore } from "@/stores/use-theme-store";
|
||||
@@ -78,7 +79,8 @@ export function AiArtCritiqueModal({ node, upstreamNodes, open, onClose, onUpdat
|
||||
const [hoveredIssueId, setHoveredIssueId] = useState<string | null>(null);
|
||||
const [draftReportVisible, setDraftReportVisible] = useState(false);
|
||||
const abortRef = useRef<AbortController | null>(null);
|
||||
const preparedImageRef = useRef<{ fingerprint: string; dataUrl: string } | null>(null);
|
||||
const currentRunTargetRef = useRef("");
|
||||
currentRunTargetRef.current = `${getActiveUserScope()}:${node?.id || ""}:${currentFingerprint}`;
|
||||
|
||||
useEffect(() => {
|
||||
if (!open || !input) {
|
||||
@@ -110,6 +112,7 @@ export function AiArtCritiqueModal({ node, upstreamNodes, open, onClose, onUpdat
|
||||
}, [node?.id, currentFingerprint, open]);
|
||||
|
||||
useEffect(() => () => abortRef.current?.abort(), []);
|
||||
useEffect(() => () => abortRef.current?.abort(), [node?.id, currentFingerprint]);
|
||||
|
||||
const issues = visibleState.report?.issues || [];
|
||||
const reportOptions = visibleState.report?.options || [];
|
||||
@@ -160,6 +163,11 @@ export function AiArtCritiqueModal({ node, upstreamNodes, open, onClose, onUpdat
|
||||
const runId = `art-critique-${Date.now()}-${Math.random().toString(36).slice(2, 7)}`;
|
||||
const controller = new AbortController();
|
||||
abortRef.current = controller;
|
||||
const runTarget = currentRunTargetRef.current;
|
||||
const runScope = getActiveUserScope();
|
||||
const publishState = (next: ArtCritiqueNodeState) => {
|
||||
if (getActiveUserScope() === runScope && currentRunTargetRef.current === runTarget && abortRef.current === controller) onUpdateState(node.id, next);
|
||||
};
|
||||
const baseState: ArtCritiqueNodeState = {
|
||||
...state,
|
||||
schemaVersion: ART_CRITIQUE_SCHEMA_VERSION,
|
||||
@@ -168,41 +176,32 @@ export function AiArtCritiqueModal({ node, upstreamNodes, open, onClose, onUpdat
|
||||
sourceNodeId: input.id,
|
||||
sourceFingerprint,
|
||||
lastRunId: runId,
|
||||
stageTaskIds: {},
|
||||
errorCode: undefined,
|
||||
errorMessage: undefined,
|
||||
updatedAt: new Date().toISOString(),
|
||||
};
|
||||
onUpdateState(node.id, baseState);
|
||||
publishState(baseState);
|
||||
setRunning(true);
|
||||
setActiveStage("preparing");
|
||||
setLocalError("");
|
||||
let latestStage: ArtCritiquePipelineStage = "preparing";
|
||||
let runningReport: ArtCritiqueReport | undefined = state.report;
|
||||
const stageTaskIds: Record<string, string> = {};
|
||||
try {
|
||||
const source = input.metadata?.content || input.metadata?.previewContent || "";
|
||||
const cachedImage = preparedImageRef.current;
|
||||
const dataUrl =
|
||||
cachedImage?.fingerprint === sourceFingerprint
|
||||
? cachedImage.dataUrl
|
||||
: await imageToDataUrl({
|
||||
dataUrl: source.startsWith("data:") ? source : undefined,
|
||||
url: source.startsWith("data:") ? undefined : source,
|
||||
storageKey: input.metadata?.storageKey,
|
||||
name: input.title,
|
||||
mimeType: input.metadata?.mimeType,
|
||||
});
|
||||
if (!dataUrl) throw new Error("无法读取输入图片");
|
||||
if (!controller.signal.aborted) preparedImageRef.current = { fingerprint: sourceFingerprint, dataUrl };
|
||||
const report = await runArtCritiquePipeline(
|
||||
effectiveConfig,
|
||||
{ dataUrl, title: input.title, sourceFingerprint },
|
||||
{
|
||||
const report = await executeArtCritique({
|
||||
nodeId: node.id, runId, source: input, config: effectiveConfig,
|
||||
signal: controller.signal,
|
||||
onTaskCreated: (stage, taskId) => {
|
||||
stageTaskIds[stage] = taskId;
|
||||
publishState({ ...baseState, stageTaskIds: { ...stageTaskIds }, analysisStage: latestStage, updatedAt: new Date().toISOString() });
|
||||
},
|
||||
onStage: (analysisStage) => {
|
||||
latestStage = analysisStage;
|
||||
setActiveStage(analysisStage);
|
||||
onUpdateState(node.id, {
|
||||
publishState({
|
||||
...baseState,
|
||||
stageTaskIds: { ...stageTaskIds },
|
||||
...(runningReport ? { report: { ...runningReport, modelLabel: modelOptionLabel(effectiveConfig, selectedCritiqueModel) } } : {}),
|
||||
analysisStage,
|
||||
updatedAt: new Date().toISOString(),
|
||||
@@ -212,19 +211,20 @@ export function AiArtCritiqueModal({ node, upstreamNodes, open, onClose, onUpdat
|
||||
if (controller.signal.aborted || abortRef.current !== controller) return;
|
||||
runningReport = draftReport;
|
||||
setDraftReportVisible(true);
|
||||
onUpdateState(node.id, {
|
||||
publishState({
|
||||
...baseState,
|
||||
stageTaskIds: { ...stageTaskIds },
|
||||
status: "running",
|
||||
analysisStage: latestStage,
|
||||
report: { ...draftReport, modelLabel: modelOptionLabel(effectiveConfig, selectedCritiqueModel) },
|
||||
updatedAt: new Date().toISOString(),
|
||||
});
|
||||
},
|
||||
},
|
||||
);
|
||||
});
|
||||
setDraftReportVisible(false);
|
||||
onUpdateState(node.id, {
|
||||
publishState({
|
||||
...baseState,
|
||||
stageTaskIds: { ...stageTaskIds },
|
||||
status: "completed",
|
||||
analysisStage: "completed",
|
||||
report: { ...report, modelLabel: modelOptionLabel(effectiveConfig, selectedCritiqueModel) },
|
||||
@@ -235,14 +235,14 @@ export function AiArtCritiqueModal({ node, upstreamNodes, open, onClose, onUpdat
|
||||
if (controller.signal.aborted) {
|
||||
setActiveStage(undefined);
|
||||
setDraftReportVisible(false);
|
||||
onUpdateState(node.id, { ...baseState, status: "idle", analysisStage: undefined, errorCode: undefined, errorMessage: undefined, updatedAt: new Date().toISOString() });
|
||||
publishState({ ...baseState, stageTaskIds: { ...stageTaskIds }, status: "failed", analysisStage: "failed", errorCode: "analysis_interrupted", errorMessage: "分析观察已停止；已提交阶段保留在任务中心，不会自动重复提交。", updatedAt: new Date().toISOString() });
|
||||
return;
|
||||
}
|
||||
const errorMessage = error instanceof Error ? error.message : "AI 批改失败，请稍后重试";
|
||||
setActiveStage("failed");
|
||||
setDraftReportVisible(false);
|
||||
setLocalError(errorMessage);
|
||||
onUpdateState(node.id, { ...baseState, status: "failed", analysisStage: "failed", errorCode: error instanceof Error ? error.message : "art_critique_failed", errorMessage, updatedAt: new Date().toISOString() });
|
||||
publishState({ ...baseState, stageTaskIds: { ...stageTaskIds }, status: "failed", analysisStage: "failed", errorCode: error instanceof Error ? error.message : "art_critique_failed", errorMessage, updatedAt: new Date().toISOString() });
|
||||
} finally {
|
||||
if (abortRef.current === controller) abortRef.current = null;
|
||||
setRunning(false);
|
||||
|
||||
@@ -13,6 +13,14 @@ import { nanoid } from "nanoid";
|
||||
import { type ResponseFunctionTool, type ResponseInputMessage, type ResponseToolCall } from "@/services/api/image";
|
||||
import { runBackendToolGenerationTask } from "@/services/api/generation-task";
|
||||
import { inspectAgentImage } from "@/services/agent-image-preview";
|
||||
import { AGENT_CAPABILITY_GUIDANCE, buildAgentPluginOperations, listAgentCapabilities, readAgentPluginDocumentation, readAgentPluginNode } from "@/services/agent-capabilities";
|
||||
import { getNodeDefinition } from "@/lib/canvas/node-registry";
|
||||
import { buildAgentStoryboardOperations, readAgentStoryboard } from "@/lib/canvas/canvas-agent-storyboard";
|
||||
import { canvasStylePresets, userStylePreset, type CanvasStylePreset } from "./canvas-style-picker-modal";
|
||||
import { listStyleProfiles } from "@/services/api/style-profiles";
|
||||
import { getActiveUserScope } from "@/lib/user-scope";
|
||||
import { isPluginEffectivelyEnabled } from "@/stores/use-plugin-store";
|
||||
import type { CanvasNodeTypeId } from "@/types/canvas";
|
||||
import { isCanvasGenerationDurableAckError, persistCanvasCinematicSessionContinuationEffect } from "@/services/canvas-generation-consumer";
|
||||
import { consumeGenerationTaskAgent } from "@/services/project-asset-sync";
|
||||
import { applyGenerationConsumerEffect, generationEffectApplied } from "@/services/generation-consumer-dedupe";
|
||||
@@ -58,8 +66,8 @@ const ONLINE_AGENT_PROMPT =
|
||||
const JSON_RECORD_SCHEMA = { type: "object", additionalProperties: true };
|
||||
const POSITION_SCHEMA = { type: "object", properties: { x: { type: "number" }, y: { type: "number" } }, required: ["x", "y"], additionalProperties: false };
|
||||
const VIEWPORT_SCHEMA = { type: "object", properties: { x: { type: "number" }, y: { type: "number" }, k: { type: "number" } }, required: ["x", "y", "k"], additionalProperties: false };
|
||||
const NODE_TYPE_SCHEMA = { type: "string", enum: ["image", "text", "skill", "video", "audio"] };
|
||||
const WORKFLOW_NODE_KIND_SCHEMA = { type: "string", enum: ["text", "script", "image", "video", "audio", "character_cards", "character_three_view", "storyboard_video"] };
|
||||
const NODE_TYPE_SCHEMA = { type: "string", description: "使用 canvas_list_capabilities 返回的真实节点类型 id" };
|
||||
const WORKFLOW_NODE_KIND_SCHEMA = { type: "string", enum: ["text", "script", "styleboard", "story_input", "image", "video", "audio", "character_cards", "character_three_view", "storyboard_video"] };
|
||||
const GENERATION_MODE_SCHEMA = { type: "string", enum: ["text", "image", "video", "audio"] };
|
||||
const GENERATION_OPTION_PROPERTIES = {
|
||||
model: { type: "string" },
|
||||
@@ -128,6 +136,14 @@ function generationToolDefinition(name: string, description: string, mode?: "tex
|
||||
}
|
||||
|
||||
const ONLINE_AGENT_TOOLS: ResponseFunctionTool[] = [
|
||||
toolDefinition("canvas_read_plugin_node", "读取已启用插件提供的真实节点状态与已有分析结果，不启动新任务。", { nodeId: { type: "string" } }, ["nodeId"]),
|
||||
toolDefinition("canvas_list_styles", "读取真实项目画风目录；source=system 为系统预设，source=user 为当前用户保存的风格。返回 ID 和说明，不能编造 ID。", { source: { type: "string", enum: ["system", "user"] }, query: { type: "string" } }),
|
||||
toolDefinition("canvas_apply_style", "应用从画风目录发现的真实风格，复用页面画风保存逻辑；关联项目时同步项目设置。已有画风且无需改变时直接沿用。", { source: { type: "string", enum: ["system", "user"] }, id: { type: "string" } }, ["source", "id"]),
|
||||
toolDefinition("canvas_read_storyboard", "读取真实分镜行 ID、内容及已有素材关联，修改前先读取。", { nodeId: { type: "string" } }, ["nodeId"]),
|
||||
toolDefinition("canvas_edit_storyboard", "按真实行 ID 新增、修改或移除分镜。patch 仅接受 durationSeconds 与镜头文本字段，不改素材 ID 或生成状态；移除行保留生成素材，不自动重新生成视频。", { nodeId: { type: "string" }, action: { type: "string", enum: ["append", "update", "remove"] }, rowId: { type: "string" }, patch: JSON_RECORD_SCHEMA }, ["nodeId", "action"]),
|
||||
toolDefinition("canvas_read_plugin", "按发现的插件 id 读取当前版本用法文档。", { pluginId: { type: "string" } }, ["pluginId"]),
|
||||
toolDefinition("canvas_plugin_action", "执行能力目录中插件已挂载的画布动作。input 遵循动作的 inputSchema；操作沿用画布确认、校验和结果记录。", { pluginId: { type: "string" }, actionId: { type: "string" }, input: JSON_RECORD_SCHEMA }, ["pluginId", "actionId", "input"]),
|
||||
toolDefinition("canvas_list_capabilities", "发现当前已注册内置节点和已启用插件。返回真实类型、默认数据、输入约束与能力边界；query 可按业务名称检索。", { query: { type: "string" } }),
|
||||
...CREATIVE_AGENT_TOOLS,
|
||||
...skillRuntime.agentTools("onlineAgent"),
|
||||
toolDefinition("canvas_get_state", "读取当前网页画布的节点、连线、选区和视口。", {}),
|
||||
@@ -171,6 +187,7 @@ const ONLINE_AGENT_TOOLS: ResponseFunctionTool[] = [
|
||||
runGeneration: { type: "boolean" },
|
||||
width: { type: "number" },
|
||||
height: { type: "number" },
|
||||
shots: { type: "array", maxItems: 100, items: { type: "object", properties: { durationSeconds: { type: "number" }, videoMotionPrompt: { type: "string" }, dialogue: { type: "string" } }, required: ["durationSeconds", "videoMotionPrompt"], additionalProperties: false } },
|
||||
},
|
||||
required: ["ref", "kind", "title"],
|
||||
additionalProperties: false,
|
||||
@@ -186,7 +203,7 @@ const ONLINE_AGENT_TOOLS: ResponseFunctionTool[] = [
|
||||
),
|
||||
toolDefinition(
|
||||
"canvas_create_node",
|
||||
"创建任意类型节点：text、image、video、audio。适合创建文本、媒体占位或自定义 metadata 节点。",
|
||||
"创建 canvas_list_capabilities 中已注册且可用的节点类型。使用返回的真实 id 和数据合同；专业短剧结构优先通过方案或 canvas_create_workflow 创建。",
|
||||
{ nodeType: NODE_TYPE_SCHEMA, title: { type: "string" }, x: { type: "number" }, y: { type: "number" }, width: { type: "number" }, height: { type: "number" }, metadata: JSON_RECORD_SCHEMA },
|
||||
["nodeType"],
|
||||
),
|
||||
@@ -268,6 +285,7 @@ type CanvasAssistantPanelProps = {
|
||||
onSelectNodeIds: (ids: Set<string>) => void;
|
||||
onSessionsChange: (sessions: CanvasAssistantSession[], activeSessionId: string | null) => void;
|
||||
onApplyOps: (ops?: CanvasAgentOp[], context?: { conversationId?: string; messageId?: string; source?: "online" | "local" }) => Promise<CanvasAgentSnapshot>;
|
||||
onApplyStyle: (preset: CanvasStylePreset) => Promise<void>;
|
||||
canUndoOps: boolean;
|
||||
undoOpsCount: number;
|
||||
onUndoOps: () => CanvasAgentSnapshot | null;
|
||||
@@ -359,6 +377,7 @@ export function CanvasAssistantPanel({
|
||||
onSelectNodeIds,
|
||||
onSessionsChange,
|
||||
onApplyOps,
|
||||
onApplyStyle,
|
||||
canUndoOps,
|
||||
undoOpsCount,
|
||||
onUndoOps,
|
||||
@@ -836,8 +855,31 @@ export function CanvasAssistantPanel({
|
||||
if (expectedRevision !== undefined && expectedRevision !== (current.revision ?? 0)) return { ok: false, message: "画布 revision 已变化，请重新读取 canvas_get_context 后再执行写操作。" };
|
||||
const expectedStateHash = typeof args.expectedStateHash === "string" ? args.expectedStateHash : "";
|
||||
if (expectedStateHash && expectedStateHash !== buildCanvasAgentContext(current).stateHash) return { ok: false, message: "画布状态已变化，请重新读取 canvas_get_context 后再执行写操作。" };
|
||||
const skillToolResult = await skillRuntime.executeAgentTool("onlineAgent", name, args, composerSkills);
|
||||
const currentSkills = skillRuntime.agentToolNames("onlineAgent").has(name) ? (await listAddedSkills()).skills : composerSkills;
|
||||
const skillToolResult = await skillRuntime.executeAgentTool("onlineAgent", name, args, currentSkills);
|
||||
if (skillToolResult) return skillToolResult;
|
||||
if (name === "canvas_read_plugin_node") return { ok: true, message: "已读取插件节点状态。", data: readAgentPluginNode(current, requireString(args.nodeId, "nodeId")) };
|
||||
if (name === "canvas_read_storyboard") {
|
||||
const { node, rows } = readAgentStoryboard(current, requireString(args.nodeId, "nodeId"));
|
||||
return { ok: true, message: "已读取分镜表。", data: { nodeId: node.id, title: node.title, rows } };
|
||||
}
|
||||
if (name === "canvas_list_styles" || name === "canvas_apply_style") {
|
||||
const originScope = getActiveUserScope();
|
||||
const source = args.source || "system";
|
||||
if (source !== "system" && source !== "user") throw new Error("画风来源必须是 system 或 user");
|
||||
const presets = source === "system" ? canvasStylePresets.map((preset) => ({ id: preset.id, preset })) : (await listStyleProfiles()).profiles.flatMap((entity) => { const preset = userStylePreset(entity); return preset ? [{ id: entity.id, preset }] : []; });
|
||||
if (getActiveUserScope() !== originScope || snapshotRef.current.projectId !== current.projectId || generationConsumerControllerRef.current.signal.aborted) throw new Error("画布或账号已切换，请重新读取画风");
|
||||
if (name === "canvas_list_styles") {
|
||||
const query = typeof args.query === "string" ? args.query.toLocaleLowerCase().trim() : "";
|
||||
return { ok: true, message: "已读取画风目录。", data: presets.filter(({ preset }) => !query || `${preset.title} ${preset.description} ${preset.tags.join(" ")}`.toLocaleLowerCase().includes(query)).map(({ id, preset }) => ({ id, source, title: preset.title, description: preset.description, tags: preset.tags })) };
|
||||
}
|
||||
const selected = presets.find((item) => item.id === args.id);
|
||||
if (!selected) throw new Error("画风不存在或已移除，请先读取真实画风目录");
|
||||
await onApplyStyle(selected.preset);
|
||||
return { ok: true, message: `已应用画风“${selected.preset.title}”。`, data: { source, id: selected.id, presetId: selected.preset.id, prompt: selected.preset.prompt } };
|
||||
}
|
||||
if (name === "canvas_read_plugin") return { ok: true, message: "已读取插件用法。", data: readAgentPluginDocumentation(requireString(args.pluginId, "pluginId")) };
|
||||
if (name === "canvas_list_capabilities") return { ok: true, message: "已读取当前能力目录。", data: listAgentCapabilities(typeof args.query === "string" ? args.query : "") };
|
||||
if (name === "canvas_inspect_image") {
|
||||
const readSignal = generationConsumerControllerRef.current.signal;
|
||||
const node = current.nodes.find((item) => item.id === args.id && item.type === "image");
|
||||
@@ -1500,6 +1542,8 @@ function parseToolArguments(value: string) {
|
||||
}
|
||||
|
||||
export function onlineToolToOps(name: string, input: Record<string, unknown>, snapshot: CanvasAgentSnapshot, config: AiConfig): CanvasAgentOp[] {
|
||||
if (name === "canvas_edit_storyboard") return buildAgentStoryboardOperations(snapshot, input);
|
||||
if (name === "canvas_plugin_action") return buildAgentPluginOperations(requireString(input.pluginId, "pluginId"), requireString(input.actionId, "actionId"), recordOptional(input.input) || {}, snapshot);
|
||||
if (name === "canvas_apply_ops") return requireOps(input.ops);
|
||||
if (name === "canvas_create_workflow") return buildCanvasWorkflowOps(input as unknown as CanvasWorkflowInput, snapshot, config);
|
||||
if (name === "canvas_create_node") {
|
||||
@@ -1652,7 +1696,7 @@ function runGenerationOp(nodeId: string, mode: "text" | "image" | "video" | "aud
|
||||
}
|
||||
|
||||
function isWritableToolCall(call: ResponseToolCall) {
|
||||
return !ONLINE_READ_TOOLS.has(call.function.name);
|
||||
return !["canvas_list_capabilities", "canvas_read_plugin", "canvas_list_styles", "canvas_read_storyboard", "canvas_read_plugin_node"].includes(call.function.name) && !ONLINE_READ_TOOLS.has(call.function.name);
|
||||
}
|
||||
|
||||
function toolCallsFromDetail(detail: Record<string, unknown>): ResponseToolCall[] {
|
||||
@@ -1717,6 +1761,14 @@ function previewOnlineToolCalls(calls: ResponseToolCall[], snapshot: CanvasAgent
|
||||
}
|
||||
|
||||
function toolCallLabel(name: string) {
|
||||
if (name === "canvas_read_plugin_node") return "读取插件结果";
|
||||
if (name === "canvas_list_styles") return "查找项目画风";
|
||||
if (name === "canvas_apply_style") return "应用项目画风";
|
||||
if (name === "canvas_read_storyboard") return "读取分镜表";
|
||||
if (name === "canvas_edit_storyboard") return "调整分镜";
|
||||
if (name === "canvas_list_capabilities") return "查找可用能力";
|
||||
if (name === "canvas_read_plugin") return "读取插件用法";
|
||||
if (name === "canvas_plugin_action") return "执行插件操作";
|
||||
if (name === "canvas_inspect_image") return "观察图片";
|
||||
if (name === "canvas_list_skills") return "列出技能";
|
||||
if (name === "canvas_get_skill") return "读取技能入口";
|
||||
@@ -1818,9 +1870,10 @@ function requireNumber(value: unknown, field: string) {
|
||||
return value;
|
||||
}
|
||||
|
||||
function requireNodeType(value: unknown): CanvasNodeType {
|
||||
if (Object.values(CanvasNodeType).includes(value as CanvasNodeType)) return value as CanvasNodeType;
|
||||
throw new Error("节点类型必须是 text、image、config、video 或 audio");
|
||||
function requireNodeType(value: unknown): CanvasNodeTypeId {
|
||||
const definition = typeof value === "string" ? getNodeDefinition(value) : undefined;
|
||||
if (definition && (!definition.plugin || isPluginEffectivelyEnabled(definition.plugin.pluginId))) return definition.type;
|
||||
throw new Error("节点未注册或插件不可用，请先读取 canvas_list_capabilities");
|
||||
}
|
||||
|
||||
function requireViewport(value: unknown) {
|
||||
@@ -1970,7 +2023,7 @@ async function buildToolAgentMessages(snapshot: CanvasAgentSnapshot, history: Ca
|
||||
const executionGuidance = confirmTools
|
||||
? "当前普通画布工具由程序展示执行确认。用户目标和操作范围明确时直接提交工具，程序会处理确认，不要在工具确认之前再用文字问一遍。"
|
||||
: "当前普通画布工具的逐次确认已关闭。用户请求范围内的操作可以直接调用，不要先征求重复授权；这不代表可以擅自扩大删除范围或跳过结构化方案、具体费用的批准。";
|
||||
const systemContent = [ONLINE_AGENT_PROMPT, executionGuidance, creativeScenarioPrompt(seed.scene), skillCatalog ? `当前可按需加载的技能（仅元数据）：\n${skillCatalog}` : ""].filter(Boolean).join("\n\n");
|
||||
const systemContent = [ONLINE_AGENT_PROMPT, AGENT_CAPABILITY_GUIDANCE, executionGuidance, creativeScenarioPrompt(seed.scene), skillCatalog ? `当前可按需加载的技能摘要（完整目录可调用 canvas_list_skills 检索）：\n${skillCatalog}` : ""].filter(Boolean).join("\n\n");
|
||||
return budgetCanvasAgentHistory(
|
||||
{ role: "system", content: systemContent },
|
||||
history
|
||||
|
||||
@@ -654,7 +654,7 @@ function EmptyStyleCenter({ tab, loading, failed, color, onCreate, onBrowse }: {
|
||||
return <div className="col-span-full grid min-h-64 place-items-center text-center"><div><Search className="mx-auto size-5" style={{ color }} /><p className="mt-2 text-xs font-medium">{loading ? "正在加载风格库" : failed && tab !== "system" ? "我的风格加载失败，请检查登录状态或后端服务" : tab === "mine" ? "还没有自己的风格" : tab === "favorites" ? "还没有收藏风格" : tab === "recent" ? "还没有使用记录" : "没有匹配的系统风格"}</p><div className="mt-3 flex justify-center gap-2">{tab === "mine" && !failed ? <Button size="small" icon={<Plus className="size-3.5" />} onClick={onCreate}>新建风格</Button> : null}<Button size="small" onClick={onBrowse}>浏览系统风格</Button></div></div></div>;
|
||||
}
|
||||
|
||||
function userStylePreset(entity: UserStyleProfile): CanvasStylePreset | null {
|
||||
export function userStylePreset(entity: UserStyleProfile): CanvasStylePreset | null {
|
||||
const profile = parseStyleProfile(entity.profileJson);
|
||||
if (!profile) return null;
|
||||
return { id: profile.presetId, title: profile.title, category: "我的风格", description: profile.description, tags: [...profile.tags], prompt: profile.prompt, imageUrl: profile.coverUrl || entity.coverUrl || "/short-drama-styles/real-life.jpg", profile };
|
||||
|
||||
@@ -125,6 +125,7 @@ export type ArtCritiqueReport = {
|
||||
export type ArtCritiqueNodeStatus = "idle" | "running" | "completed" | "failed" | "stale";
|
||||
|
||||
export type ArtCritiqueNodeState = {
|
||||
stageTaskIds?: Record<string, string>;
|
||||
schemaVersion: typeof ART_CRITIQUE_SCHEMA_VERSION;
|
||||
status: ArtCritiqueNodeStatus;
|
||||
sourceNodeId?: string;
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { requestToolResponse, type ResponseFunctionTool, type ResponseInputMessage } from "@/services/api/image";
|
||||
import { requestToolResponse, type ResponseFunctionTool, type ResponseInputMessage, type ToolResponseResult } from "@/services/api/image";
|
||||
import type { AiConfig } from "@/stores/use-config-store";
|
||||
|
||||
import {
|
||||
@@ -31,6 +31,7 @@ const GROUNDING_CONFIDENCE_THRESHOLD = 0.65;
|
||||
const VERIFICATION_REJECTION_THRESHOLD = 0.75;
|
||||
|
||||
export type ArtCritiquePipelineOptions = {
|
||||
requestStage?: (input: { config: AiConfig; messages: ResponseInputMessage[]; tool: ResponseFunctionTool; toolName: string; signal?: AbortSignal }) => Promise<ToolResponseResult>;
|
||||
signal?: AbortSignal;
|
||||
onStage?: (stage: ArtCritiquePipelineStage) => void;
|
||||
onDraftReport?: (report: ArtCritiqueReport) => void;
|
||||
@@ -557,7 +558,9 @@ export function applyEditPrompts(issues: readonly ArtCritiqueIssue[], prompts: r
|
||||
|
||||
async function requestPipelineStage<T>(config: AiConfig, messages: ResponseInputMessage[], tool: ResponseFunctionTool, toolName: string, parse: (value: unknown) => T, options: ArtCritiquePipelineOptions) {
|
||||
throwIfAborted(options.signal);
|
||||
const response = await requestToolResponse(config, messages, [tool], { type: "function", name: toolName }, undefined, { signal: options.signal });
|
||||
const response = options.requestStage
|
||||
? await options.requestStage({ config, messages, tool, toolName, signal: options.signal })
|
||||
: await requestToolResponse(config, messages, [tool], { type: "function", name: toolName }, undefined, { signal: options.signal });
|
||||
const raw = response.toolCalls.find((candidate) => candidate.function.name === toolName)?.function.arguments || response.content;
|
||||
if (!raw?.trim()) throw new Error(`${toolName}_missing`);
|
||||
let value: unknown;
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
import { hashCanvasAgentSnapshot, type CanvasAgentOp, type CanvasAgentSnapshot } from "./canvas-agent-ops";
|
||||
import { CanvasNodeType, type CanvasNodeData } from "@/types/canvas";
|
||||
import { getNodeDefinition } from "./node-registry";
|
||||
import { isPluginEffectivelyEnabled } from "@/stores/use-plugin-store";
|
||||
|
||||
export type CanvasAgentResource = {
|
||||
nodeId: string;
|
||||
@@ -151,6 +153,8 @@ export function validateCanvasAgentOps(snapshot: CanvasAgentSnapshot, ops: Canva
|
||||
};
|
||||
ops.forEach((op, index) => {
|
||||
if (op.type === "add_node") {
|
||||
const definition = getNodeDefinition(op.nodeType || CanvasNodeType.Text);
|
||||
if (!definition || (definition.plugin && !isPluginEffectivelyEnabled(definition.plugin.pluginId))) issues.push({ index, severity: "error", message: "节点未注册或插件不可用，请重新发现能力" });
|
||||
validateNodeNumbers(issues, index, op as unknown as Record<string, unknown>);
|
||||
if (op.id && liveNodeIds.has(op.id)) issues.push({ index, severity: "error", message: `新增节点 id「${op.id}」重复` });
|
||||
if (op.id) liveNodeIds.add(op.id);
|
||||
|
||||
@@ -1,12 +1,14 @@
|
||||
import { nanoid } from "nanoid";
|
||||
|
||||
import { getNodeSpec } from "@/constant/canvas";
|
||||
import { CanvasNodeType, type CanvasConnection, type CanvasNodeData, type CanvasNodeMetadata, type ViewportTransform } from "@/types/canvas";
|
||||
import { getNodeDefinition } from "./node-registry";
|
||||
import { isPluginEffectivelyEnabled } from "@/stores/use-plugin-store";
|
||||
import { CanvasNodeType, type CanvasNodeTypeId, type CanvasConnection, type CanvasNodeData, type CanvasNodeMetadata, type ViewportTransform } from "@/types/canvas";
|
||||
|
||||
export type CanvasAgentOp =
|
||||
| { type: "add_node"; id?: string; nodeType?: CanvasNodeType; title?: string; position?: { x: number; y: number }; x?: number; y?: number; width?: number; height?: number; metadata?: CanvasNodeMetadata }
|
||||
| { type: "add_node"; id?: string; nodeType?: CanvasNodeTypeId; title?: string; position?: { x: number; y: number }; x?: number; y?: number; width?: number; height?: number; metadata?: CanvasNodeMetadata }
|
||||
| { type: "update_node"; id: string; patch?: Partial<CanvasNodeData>; metadata?: CanvasNodeMetadata }
|
||||
| { type: "delete_node"; id?: string; ids?: string[]; nodeType?: CanvasNodeType }
|
||||
| { type: "delete_node"; id?: string; ids?: string[]; nodeType?: CanvasNodeTypeId }
|
||||
| { type: "delete_connections"; id?: string; ids?: string[]; all?: boolean }
|
||||
| { type: "connect_nodes"; id?: string; fromNodeId: string; toNodeId: string; fromHandleId?: string; toHandleId?: string }
|
||||
| { type: "set_viewport"; viewport: ViewportTransform }
|
||||
@@ -320,8 +322,10 @@ export function applyCanvasAgentOps(snapshot: CanvasAgentSnapshot, ops?: CanvasA
|
||||
(Array.isArray(ops) ? ops : []).forEach((op, index) => {
|
||||
if (!op?.type) return;
|
||||
if (op.type === "add_node") {
|
||||
const nodeType = Object.values(CanvasNodeType).includes(op.nodeType as CanvasNodeType) ? op.nodeType! : CanvasNodeType.Text;
|
||||
const spec = getNodeSpec(nodeType);
|
||||
const nodeType = op.nodeType || CanvasNodeType.Text;
|
||||
const definition = getNodeDefinition(nodeType);
|
||||
if (!definition || (definition.plugin && !isPluginEffectivelyEnabled(definition.plugin.pluginId))) throw new Error(`节点类型未注册或插件不可用：${nodeType}`);
|
||||
const spec = getNodeSpec(nodeType as CanvasNodeType) || { title: definition.defaultTitle, ...definition.defaultSize, metadata: definition.defaultMetadata };
|
||||
const node: CanvasNodeData = {
|
||||
id: op.id || `${nodeType}-${Date.now()}-${index}`,
|
||||
type: nodeType,
|
||||
@@ -329,7 +333,7 @@ export function applyCanvasAgentOps(snapshot: CanvasAgentSnapshot, ops?: CanvasA
|
||||
position: op.position || { x: op.x ?? index * 36, y: op.y ?? index * 36 },
|
||||
width: op.width || spec.width,
|
||||
height: op.height || spec.height,
|
||||
metadata: { ...spec.metadata, ...op.metadata },
|
||||
metadata: { ...spec.metadata, ...op.metadata, ...(definition.plugin ? { pluginId: definition.plugin.pluginId, pluginNodeId: nodeType } : {}) },
|
||||
};
|
||||
nodes = [...nodes, node];
|
||||
selectedNodeIds = [node.id];
|
||||
@@ -390,7 +394,7 @@ function opLabel(type: string) {
|
||||
return type;
|
||||
}
|
||||
|
||||
function canvasNodeTypeLabel(type?: CanvasNodeType) {
|
||||
function canvasNodeTypeLabel(type?: CanvasNodeTypeId) {
|
||||
if (type === CanvasNodeType.Image) return "图片节点";
|
||||
if (type === CanvasNodeType.Video) return "视频节点";
|
||||
if (type === CanvasNodeType.Audio) return "音频节点";
|
||||
@@ -399,8 +403,7 @@ function canvasNodeTypeLabel(type?: CanvasNodeType) {
|
||||
if (type === CanvasNodeType.Frame) return "背板";
|
||||
if (type === CanvasNodeType.Drawing) return "绘图节点";
|
||||
if (type === CanvasNodeType.Skill) return "技能节点";
|
||||
if (type === CanvasNodeType.MediaConversion) return "转换节点";
|
||||
return "文本节点";
|
||||
return type ? `${getNodeDefinition(type)?.label || type}节点` : "文本节点";
|
||||
}
|
||||
|
||||
function generationModeLabel(mode?: "text" | "image" | "video" | "audio") {
|
||||
|
||||
52
web/src/lib/canvas/canvas-agent-storyboard.ts
Normal file
52
web/src/lib/canvas/canvas-agent-storyboard.ts
Normal file
@@ -0,0 +1,52 @@
|
||||
import type { CanvasAgentOp, CanvasAgentSnapshot } from "./canvas-agent-ops";
|
||||
import { createStoryboardRow } from "./canvas-project-domain";
|
||||
import type { StoryboardRow } from "@/types/canvas";
|
||||
|
||||
const textFields = ["plotDescription", "dialogue", "videoMotionPrompt", "imageGenerationPrompt", "camera", "motion", "shotSize", "emotion", "lightingAndAtmosphere", "audioEffects", "narrativeIntent", "viewerPOV", "performanceBlocking", "timeBeats", "continuityOut", "negativePrompt"] as const;
|
||||
|
||||
export function readAgentStoryboard(snapshot: CanvasAgentSnapshot, nodeId: string) {
|
||||
const node = snapshot.nodes.find((item) => item.id === nodeId && item.type === "script");
|
||||
if (!node) throw new Error("未找到当前画布的分镜脚本节点，请先查找真实节点");
|
||||
return { node, rows: node.metadata?.storyboard?.rows || [] };
|
||||
}
|
||||
|
||||
export function buildAgentStoryboardOperations(snapshot: CanvasAgentSnapshot, input: Record<string, unknown>): CanvasAgentOp[] {
|
||||
if (typeof input.nodeId !== "string") throw new Error("需要分镜节点 nodeId");
|
||||
const { node, rows } = readAgentStoryboard(snapshot, input.nodeId);
|
||||
const action = input.action;
|
||||
const rowId = typeof input.rowId === "string" ? input.rowId : "";
|
||||
const index = rows.findIndex((row) => row.id === rowId);
|
||||
let next = rows.slice();
|
||||
if (action === "remove") {
|
||||
if (index < 0) throw new Error("分镜行不存在，请重新读取分镜表");
|
||||
// Removing a row never deletes its existing generated assets.
|
||||
next.splice(index, 1);
|
||||
} else if (action === "append" || action === "update") {
|
||||
if (action === "update" && index < 0) throw new Error("分镜行不存在，请重新读取分镜表");
|
||||
const raw = input.patch;
|
||||
if (!raw || typeof raw !== "object" || Array.isArray(raw)) throw new Error("需要分镜内容 patch");
|
||||
const data = raw as Record<string, unknown>;
|
||||
const patch: Partial<StoryboardRow> = {};
|
||||
for (const [key, value] of Object.entries(data)) {
|
||||
if (key === "durationSeconds") {
|
||||
if (typeof value !== "number" || !Number.isFinite(value) || value <= 0) throw new Error("分镜时长必须是大于零的数字");
|
||||
patch.durationSeconds = value;
|
||||
} else if (textFields.includes(key as typeof textFields[number])) {
|
||||
if (typeof value !== "string" || value.length > 20_000) throw new Error(`分镜字段 ${key} 需要不超过 20000 字符的文本`);
|
||||
patch[key as typeof textFields[number]] = value;
|
||||
} else throw new Error(`不能通过分镜编辑修改字段 ${key}；已有素材关联与任务状态由系统维护`);
|
||||
}
|
||||
if (!Object.keys(patch).length) throw new Error("未提供需要修改的分镜内容");
|
||||
if (action === "append") {
|
||||
if (rows.length >= 100) throw new Error("单个分镜表最多 100 行");
|
||||
if (!patch.videoMotionPrompt?.trim() && !patch.plotDescription?.trim()) throw new Error("新增镜头需要画面描述或视频提示词");
|
||||
next.push(createStoryboardRow(rows.length + 1, patch));
|
||||
} else next[index] = { ...rows[index], ...patch };
|
||||
} else throw new Error("分镜操作必须是 append、update 或 remove");
|
||||
next = next.map((row, i) => ({ ...row, shotNumber: i + 1 }));
|
||||
return [{ type: "update_node", id: node.id, metadata: { storyboard: {
|
||||
rows: next,
|
||||
visibleColumns: node.metadata?.storyboard?.visibleColumns || ["shotNumber", "durationSeconds", "videoMotionPrompt", "dialogue", "assets"],
|
||||
referenceNodeIds: node.metadata?.storyboard?.referenceNodeIds || [],
|
||||
} } }];
|
||||
}
|
||||
@@ -4,10 +4,14 @@ import { NODE_DEFAULT_SIZE } from "@/constant/canvas";
|
||||
import { CanvasNodeType, type CanvasNodeData, type CanvasNodeMetadata } from "@/types/canvas";
|
||||
import type { AiConfig } from "@/stores/use-config-store";
|
||||
import type { CanvasAgentOp, CanvasAgentSnapshot } from "./canvas-agent-ops";
|
||||
import { createShortDramaPipeline } from "./canvas-short-drama";
|
||||
import { createStoryboardRow } from "./canvas-project-domain";
|
||||
|
||||
export type CanvasWorkflowNodeKind =
|
||||
| "text"
|
||||
| "script"
|
||||
| "styleboard"
|
||||
| "story_input"
|
||||
| "image"
|
||||
| "video"
|
||||
| "audio"
|
||||
@@ -28,6 +32,7 @@ export type CanvasWorkflowNodeInput = {
|
||||
runGeneration?: boolean;
|
||||
width?: number;
|
||||
height?: number;
|
||||
shots?: Array<{ durationSeconds: number; videoMotionPrompt: string; dialogue?: string }>;
|
||||
};
|
||||
|
||||
export type CanvasWorkflowInput = {
|
||||
@@ -102,6 +107,18 @@ export function buildCanvasWorkflowOps(input: CanvasWorkflowInput, snapshot: Can
|
||||
],
|
||||
} : {}),
|
||||
};
|
||||
if (["styleboard", "story_input", "script"].includes(node.kind)) {
|
||||
const template = createShortDramaPipeline({ x: 0, y: 0 });
|
||||
const source = template.nodes[node.kind === "styleboard" ? 0 : node.kind === "story_input" ? 1 : 2];
|
||||
Object.assign(metadata, source.metadata, { content: node.kind === "styleboard" ? "" : node.content || "", composerContent: prompt });
|
||||
if (node.kind === "script") {
|
||||
if (node.shots && (!Array.isArray(node.shots) || node.shots.length > 100)) throw new Error("分镜最多 100 行");
|
||||
metadata.storyboard = { ...source.metadata!.storyboard!, rows: (node.shots || []).map((shot, rowIndex) => {
|
||||
if (!Number.isFinite(shot.durationSeconds) || shot.durationSeconds <= 0 || typeof shot.videoMotionPrompt !== "string" || !shot.videoMotionPrompt.trim()) throw new Error("每条分镜需要有效时长和视频提示词");
|
||||
return createStoryboardRow(rowIndex + 1, { id: `${id}:shot:${rowIndex + 1}`, durationSeconds: shot.durationSeconds, videoMotionPrompt: shot.videoMotionPrompt, dialogue: typeof shot.dialogue === "string" ? shot.dialogue : "" });
|
||||
}) };
|
||||
}
|
||||
}
|
||||
ops.push({ type: "add_node", id, nodeType: type, title: node.title, position, width: size.width, height: size.height, metadata });
|
||||
});
|
||||
|
||||
@@ -113,20 +130,20 @@ export function buildCanvasWorkflowOps(input: CanvasWorkflowInput, snapshot: Can
|
||||
const key = `${edge.from}\0${edge.to}`;
|
||||
if (edgeKeys.has(key)) continue;
|
||||
edgeKeys.add(key);
|
||||
ops.push({ type: "connect_nodes", fromNodeId: ids.get(edge.from)!, toNodeId: ids.get(edge.to)! });
|
||||
ops.push({ type: "connect_nodes", fromNodeId: ids.get(edge.from)!, toNodeId: ids.get(edge.to)!, ...(input.nodes.find((node) => node.ref === edge.to)?.kind === "script" ? { toHandleId: "storyboard:context" } : {}) });
|
||||
}
|
||||
for (const node of input.nodes) {
|
||||
for (const referenceRef of node.referenceRefs || []) {
|
||||
const key = `${referenceRef}\0${node.ref}`;
|
||||
if (edgeKeys.has(key)) continue;
|
||||
edgeKeys.add(key);
|
||||
ops.push({ type: "connect_nodes", fromNodeId: ids.get(referenceRef)!, toNodeId: ids.get(node.ref)! });
|
||||
ops.push({ type: "connect_nodes", fromNodeId: ids.get(referenceRef)!, toNodeId: ids.get(node.ref)!, ...(node.kind === "script" ? { toHandleId: "storyboard:context" } : {}) });
|
||||
}
|
||||
for (const referenceNodeId of node.referenceNodeIds || []) {
|
||||
const key = `${referenceNodeId}\0${node.ref}`;
|
||||
if (edgeKeys.has(key)) continue;
|
||||
edgeKeys.add(key);
|
||||
ops.push({ type: "connect_nodes", fromNodeId: referenceNodeId, toNodeId: ids.get(node.ref)! });
|
||||
ops.push({ type: "connect_nodes", fromNodeId: referenceNodeId, toNodeId: ids.get(node.ref)!, ...(node.kind === "script" ? { toHandleId: "storyboard:context" } : {}) });
|
||||
}
|
||||
}
|
||||
const targetIds = input.nodes.map((node) => ids.get(node.ref)!);
|
||||
@@ -177,7 +194,8 @@ function nodeTypeForWorkflowKind(kind: CanvasWorkflowNodeKind) {
|
||||
if (kind === "image" || kind === "character_cards" || kind === "character_three_view") return CanvasNodeType.Image;
|
||||
if (kind === "video" || kind === "storyboard_video") return CanvasNodeType.Video;
|
||||
if (kind === "audio") return CanvasNodeType.Audio;
|
||||
return CanvasNodeType.Text;
|
||||
if (["text", "styleboard", "story_input"].includes(kind)) return CanvasNodeType.Text;
|
||||
throw new Error(`未知工作流节点类型：${kind}`);
|
||||
}
|
||||
|
||||
function nodeSize(type: CanvasNodeType, kind: CanvasWorkflowNodeKind, width?: number, height?: number) {
|
||||
@@ -187,7 +205,7 @@ function nodeSize(type: CanvasNodeType, kind: CanvasWorkflowNodeKind, width?: nu
|
||||
: kind === "storyboard_video"
|
||||
? { width: 640, height: 360 }
|
||||
: defaults;
|
||||
return { width: width || preferred.width, height: height || preferred.height };
|
||||
return { width: width || (kind === "styleboard" ? 360 : kind === "story_input" ? 420 : preferred.width), height: height || (kind === "styleboard" ? 220 : kind === "story_input" ? 260 : preferred.height) };
|
||||
}
|
||||
|
||||
function workflowKindForNode(kind: CanvasWorkflowNodeKind): CanvasNodeMetadata["workflowKind"] {
|
||||
|
||||
@@ -13,6 +13,12 @@ const FALLBACK_MIN_SIZE = { width: 220, height: 160 } as const;
|
||||
|
||||
/** 批量注册节点定义 */
|
||||
export function registerNodeDefinitions(defs: CanvasNodeDefinition[], ownerId = "builtin") {
|
||||
const ids = new Set<string>();
|
||||
for (const def of defs) {
|
||||
const owner = ownerByType.get(def.type);
|
||||
if (ids.has(def.type) || (owner && owner !== ownerId)) throw new Error(`节点类型重复或归属冲突：${def.type}`);
|
||||
ids.add(def.type);
|
||||
}
|
||||
for (const def of defs) {
|
||||
definitions.set(def.type, def);
|
||||
ownerByType.set(def.type, ownerId);
|
||||
|
||||
@@ -59,7 +59,7 @@ export function normalizeCreativeProposal(raw: unknown, id: string, version: num
|
||||
if (!Array.isArray(workflow.nodes) || workflow.nodes.length < 1 || workflow.nodes.length > 20) throw new Error("方案应包含 1 至 20 个有明确用途的节点");
|
||||
const nodes = workflow.nodes.map((value) => {
|
||||
const node = record(value), kind = str(node.kind);
|
||||
if (!["text", "image", "video"].includes(kind)) throw new Error("此方案包含本期不支持的节点类型");
|
||||
if (!["text", "image", "video", "styleboard", "story_input", "script"].includes(kind)) throw new Error("方案节点类型不可用，请读取能力目录");
|
||||
const content = str(node.content), prompt = str(node.prompt);
|
||||
const assetId = str(node.assetId), ref = required(node.ref, "节点引用");
|
||||
if (assetId) {
|
||||
@@ -67,14 +67,16 @@ export function normalizeCreativeProposal(raw: unknown, id: string, version: num
|
||||
if (kind !== "image" || !asset) throw new Error("方案引用的已有图片不可用");
|
||||
existingAssets[ref] = asset;
|
||||
}
|
||||
if (!assetId && (kind === "text" ? !content : !prompt)) throw new Error("文本节点需要正文，媒体节点需要生成提示词");
|
||||
return { ref: required(node.ref, "节点引用"), kind: kind as "text" | "image" | "video", title: required(node.title, "节点标题"), content, prompt, runGeneration: false, referenceRefs: Array.isArray(node.referenceRefs) ? node.referenceRefs.map(String) : [], referenceNodeIds: Array.isArray(node.referenceNodeIds) ? node.referenceNodeIds.map(String) : [] };
|
||||
if (!assetId && (["text", "story_input"].includes(kind) ? !content : ["image", "video"].includes(kind) && !prompt)) throw new Error("文本节点需要正文，媒体节点需要生成提示词");
|
||||
const shots = Array.isArray(node.shots) ? node.shots.map((value) => { const shot = record(value); const durationSeconds = Number(shot.durationSeconds); if (!Number.isFinite(durationSeconds) || durationSeconds <= 0) throw new Error("分镜时长必须大于零"); return { durationSeconds, videoMotionPrompt: required(shot.videoMotionPrompt, "分镜视频提示词"), dialogue: str(shot.dialogue) }; }) : undefined;
|
||||
if (shots && shots.length > 100) throw new Error("分镜最多 100 行");
|
||||
return { ref: required(node.ref, "节点引用"), kind: kind as "text" | "image" | "video" | "styleboard" | "story_input" | "script", title: required(node.title, "节点标题"), content, prompt, shots, runGeneration: false, referenceRefs: Array.isArray(node.referenceRefs) ? node.referenceRefs.map(String) : [], referenceNodeIds: Array.isArray(node.referenceNodeIds) ? node.referenceNodeIds.map(String) : [] };
|
||||
});
|
||||
const refs = new Set(nodes.map((node) => node.ref));
|
||||
if (refs.size !== nodes.length) throw new Error("方案节点引用重复");
|
||||
const generationItems: CreativeGenerationItem[] = (Array.isArray(data.generationItems) ? data.generationItems : []).map((value) => {
|
||||
const item = record(value), ref = required(item.ref, "生成目标"), node = nodes.find((candidate) => candidate.ref === ref);
|
||||
if (!node || node.kind === "text" || node.kind !== item.mode || existingAssets[ref]) throw new Error("生成任务与媒体节点不匹配，已有素材不能重复计费");
|
||||
if (!node || (node.kind !== "image" && node.kind !== "video") || node.kind !== item.mode || existingAssets[ref]) throw new Error("生成任务与媒体节点不匹配，已有素材不能重复计费");
|
||||
const model = required(item.model, "生成模型");
|
||||
if (!selectableModelsByCapability(config, node.kind).includes(model)) throw new Error(`节点“${node.title}”需要${node.kind === "video" ? "视频" : "图片"}模型，所选模型不支持该用途或已不可用：${model}。请从 availableModels 中选择 mode=${node.kind} 的模型重新提交方案。`);
|
||||
const referenceRefs: string[] = [];
|
||||
@@ -104,7 +106,7 @@ export function normalizeCreativeProposal(raw: unknown, id: string, version: num
|
||||
});
|
||||
const mediaRefs = generationItems.map((item) => item.ref);
|
||||
if (new Set(mediaRefs).size !== mediaRefs.length) throw new Error("方案存在重复的 generationItems.ref，每个待生成媒体节点只能对应一个生成项");
|
||||
const missing = nodes.filter((node) => node.kind !== "text" && !existingAssets[node.ref] && !mediaRefs.includes(node.ref));
|
||||
const missing = nodes.filter((node) => (node.kind === "image" || node.kind === "video") && !existingAssets[node.ref] && !mediaRefs.includes(node.ref));
|
||||
if (missing.length) throw new Error(`方案缺少媒体生成配置：${missing.map((node) => `${node.title}（ref=${node.ref}，mode=${node.kind}）`).join("、")}。请补齐 generationItems；配置用于方案校验与报价，不代表立即生成，仍需用户确认。`);
|
||||
// 引用图的循环依赖会造成永远没有可执行批次。
|
||||
const visited = new Set<string>(), visiting = new Set<string>();
|
||||
|
||||
@@ -5,7 +5,7 @@ const strings = { type: "array", items: string };
|
||||
const object = (properties: Record<string, unknown>, required: string[] = []) => ({ type: "object", properties, required, additionalProperties: false });
|
||||
const option = object({ id: string, label: string, description: string }, ["id", "label"]);
|
||||
const question = object({ id: string, field: string, type: { type: "string", enum: ["single", "multiple", "text", "asset"] }, title: string, description: string, required: { type: "boolean" }, allowCustom: { type: "boolean" }, options: { type: "array", items: option } }, ["id", "field", "type", "title", "required", "allowCustom", "options"]);
|
||||
const workflowNode = object({ ref: string, kind: { type: "string", enum: ["text", "image", "video"] }, title: string, content: string, prompt: string, assetId: string, referenceRefs: strings, referenceNodeIds: strings }, ["ref", "kind", "title"]);
|
||||
const workflowNode = object({ ref: string, kind: { type: "string", enum: ["text", "image", "video", "styleboard", "story_input", "script"] }, title: string, content: string, prompt: string, assetId: string, referenceRefs: strings, referenceNodeIds: strings, shots: { type: "array", maxItems: 100, items: object({ durationSeconds: { type: "number" }, videoMotionPrompt: string, dialogue: string }, ["durationSeconds", "videoMotionPrompt"]) } }, ["ref", "kind", "title"]);
|
||||
const generation = object({ ref: string, mode: { type: "string", enum: ["image", "video"] }, model: string, size: string, seconds: string, quality: string, referenceRefs: strings }, ["ref", "mode", "model"]);
|
||||
|
||||
// 模型只描述内容；批准、费用提交、任务状态和页面跳转不作为模型工具暴露。
|
||||
@@ -31,7 +31,7 @@ export const CREATIVE_AGENT_SYSTEM_PROMPT = `你是影策的通用创作助手
|
||||
多步骤创作使用 creative_respond。先结合全部需求摘要、会话、用户素材、专业角色指导、选中技能与真实画布，提取已知信息；只有缺失或冲突且确实影响执行的信息才补问，通常1至3题，确需一起决定时最多6题，不凑题。允许自定义安全的需求字段，优先复用已有字段，不通过改名重复询问。用户直接输入的自然语言回答与卡片回答同等有效，可以同时更新多个字段。不得将猜测写成用户确认。用户允许自主决定时可以直接给方案。
|
||||
多步骤任务可携带 plan，按本次实际工作命名步骤，例如“确定结尾情绪”“设计商品构图”“制作首张主图”；不套固定步骤数。每步 ref 在同一工作持续时保持稳定，新工作使用新ref。phase只用于程序关联真实状态：questions=需求选择，proposal=内容方案，canvas=节点落地，media=产物生成（mediaRefs关联方案生成项ref）。可以调整未完成步骤并在reason说明原因；不编造完成状态。普通咨询不创建计划。已完成工作不可抹掉，新增任务不能复用已完成步骤ref。
|
||||
问题与方案不能在同一次响应同时推进。已经确认的题材/时长/数量/比例/风格不再问。覆盖已确认字段必须在 evidence 引用最新用户消息中明确修改该字段的原文；不能将建议或旧消息作为修改授权。素材题的 option.id 只能使用上下文提供的真实素材ID。
|
||||
方案应是可阅读的完整 Markdown，包含内容、已有素材、待生成产物、规格和真实依赖。workflow 仅用现有 text/image/video 节点；媒体必须有完整提示词，文本必须有正文。每个待生成媒体节点有且仅有一项 generationItems，使用可用模型的完整 model ID。用户已有图片优先用 image 节点的 assetId 指向真实 references.id，此类节点不写 generationItems、不重复生成，可被其他节点用 referenceRefs 引用。
|
||||
方案应是可阅读的完整 Markdown，包含内容、已有素材、待生成产物、规格和真实依赖。workflow 支持 text/image/video/styleboard/story_input/script；搭建短剧工作流时使用真实的 styleboard 项目画风选择节点、story_input 故事梗概和 script 分镜表，已有节点优先沿用。script.shots 可填写逐镜 durationSeconds、videoMotionPrompt、dialogue，不用普通文本模拟表格。styleboard 是待选择入口，不把任意文字当作已应用画风；用户已有画风时不重复创建。只有 image/video 填写 generationItems；创建分镜表和画风入口不代表生成媒体。媒体必须有完整提示词，文本必须有正文。每个待生成媒体节点有且仅有一项 generationItems，使用可用模型的完整 model ID。用户已有图片优先用 image 节点的 assetId 指向真实 references.id，此类节点不写 generationItems、不重复生成，可被其他节点用 referenceRefs 引用。
|
||||
generationItems 是方案的声明式生成配置，用于校验和之后的报价，不会立即调用媒体模型。因此“确认后再生成”仍必须填写它，不能省略。每项 ref 必须对应节点 ref，mode 必须等于节点 kind，model 必须来自 availableModels 中相同 mode 的条目；图片模型不能用于视频。上下文若有 rejectedProposal，先根据其中真实校验错误修正，保留已知需求并返回完整方案。结构化方案和费用的批准以程序记录为准；若仍缺按钮确认，直接显示对应卡片，不再先问一遍是否同意、再让用户点卡片。
|
||||
引用已有画布图片时，将真实 references.id 放到 workflow.nodes[].referenceNodeIds；referenceRefs 优先用于本次方案内其他图片节点的 ref。不要为引用现有图片再生成一张副本，不要在两个字段重复放同一张图。程序会核实真实资源并整理引用，未知ID不能通过。
|
||||
单段视频可以在正文描述多个叙事节奏，不把它们拆成未合成的几段视频。用户要求15秒时只选实际支持15秒的模型；不存在则解释缺少能力并询问可接受替代，绝不能偷偷缩短。图像引用数量、视频输入角色、尺寸均须符合上下文能力。边表达真实依赖，不为了好看造线。
|
||||
|
||||
24
web/src/lib/plugins/analysis-node-action.ts
Normal file
24
web/src/lib/plugins/analysis-node-action.ts
Normal file
@@ -0,0 +1,24 @@
|
||||
import { nanoid } from "nanoid";
|
||||
import type { CanvasNodeMetadata } from "@/types/canvas";
|
||||
import type { PluginAgentAction } from "./plugin-types";
|
||||
|
||||
export function prepareAnalysisNodeAction(nodeType: string, title: string, metadata: () => CanvasNodeMetadata): PluginAgentAction {
|
||||
return {
|
||||
id: "prepare-input",
|
||||
description: `为已有图片准备${title}节点及连线，复用相同输入的已有节点；仅准备输入，不启动分析或收费。`,
|
||||
inputSchema: { type: "object", properties: { sourceNodeId: { type: "string" } }, required: ["sourceNodeId"], additionalProperties: false },
|
||||
buildOperations(input, snapshot) {
|
||||
if (Object.keys(input).some((key) => key !== "sourceNodeId")) throw new Error("只接受 sourceNodeId");
|
||||
const source = snapshot.nodes.find((node) => node.id === input.sourceNodeId && node.type === "image");
|
||||
if (!source || !(source.metadata?.storageKey || source.metadata?.content || source.metadata?.previewContent)) throw new Error("请提供当前画布已有图片节点的真实 ID");
|
||||
const existing = snapshot.nodes.find((node) => node.type === nodeType && snapshot.connections.some((edge) => edge.fromNodeId === source.id && edge.toNodeId === node.id));
|
||||
if (existing) return [{ type: "select_nodes", ids: [existing.id] }];
|
||||
const id = `${nodeType}-${nanoid()}`;
|
||||
return [
|
||||
{ type: "add_node", id, nodeType, title, position: { x: Math.max(...snapshot.nodes.map((node) => node.position.x + node.width)) + 120, y: source.position.y }, metadata: metadata() },
|
||||
{ type: "connect_nodes", fromNodeId: source.id, toNodeId: id },
|
||||
{ type: "select_nodes", ids: [id] },
|
||||
];
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -1,7 +1,8 @@
|
||||
import { registerPlugin } from "@/lib/plugins/plugin-registry";
|
||||
import type { PluginManifest, RegisteredPlugin } from "@/lib/plugins/plugin-types";
|
||||
|
||||
import { ART_CRITIQUE_NODE_TYPE, ART_CRITIQUE_PLUGIN_ID } from "@/lib/art-critique/contracts";
|
||||
import { ART_CRITIQUE_NODE_TYPE, ART_CRITIQUE_PLUGIN_ID, artCritiqueSourceFingerprint, createDefaultArtCritiqueState } from "@/lib/art-critique/contracts";
|
||||
import { prepareAnalysisNodeAction } from "../analysis-node-action";
|
||||
|
||||
const manifest: PluginManifest = {
|
||||
apiVersion: "yingce.plugin/v1",
|
||||
@@ -30,6 +31,15 @@ const manifest: PluginManifest = {
|
||||
},
|
||||
};
|
||||
|
||||
export const artCritiquePlugin: RegisteredPlugin = { manifest };
|
||||
export const artCritiquePlugin: RegisteredPlugin = {
|
||||
manifest,
|
||||
agentActions: [prepareAnalysisNodeAction(ART_CRITIQUE_NODE_TYPE, "AI 审美批改", () => ({ artCritique: createDefaultArtCritiqueState() }))],
|
||||
readAgentNode: (node, snapshot) => {
|
||||
const state = node.metadata?.artCritique;
|
||||
const sources = snapshot.nodes.filter((item) => item.type === "image" && snapshot.connections.some((edge) => edge.fromNodeId === item.id && edge.toNodeId === node.id));
|
||||
const currentReport = state?.report && sources.length === 1 && state.report.sourceFingerprint === artCritiqueSourceFingerprint(sources[0]) ? state.report : undefined;
|
||||
return { status: state?.report && !currentReport ? "stale" : state?.status || "idle", stage: state?.analysisStage, stageTaskIds: state?.stageTaskIds, sourceNodeId: state?.sourceNodeId, summary: currentReport?.summary, strengths: currentReport?.strengths, issues: currentReport?.issues, options: currentReport?.options, error: state?.errorMessage, execution: "已有报告仅在输入指纹一致时返回；新分析请使用节点的分析入口，当前 Agent 动作仅准备输入。" };
|
||||
},
|
||||
};
|
||||
|
||||
registerPlugin(artCritiquePlugin);
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
import { registerPlugin } from "@/lib/plugins/plugin-registry";
|
||||
import type { PluginManifest, RegisteredPlugin } from "@/lib/plugins/plugin-types";
|
||||
|
||||
import { PORTRAIT_CLEARANCE_NODE_TYPE, PORTRAIT_CLEARANCE_PLUGIN_ID } from "@/lib/portrait-clearance/contracts";
|
||||
import { PORTRAIT_CLEARANCE_NODE_TYPE, PORTRAIT_CLEARANCE_PLUGIN_ID, createDefaultPortraitClearanceState } from "@/lib/portrait-clearance/contracts";
|
||||
import { prepareAnalysisNodeAction } from "../analysis-node-action";
|
||||
|
||||
const manifest: PluginManifest = {
|
||||
apiVersion: "yingce.plugin/v1",
|
||||
@@ -35,6 +36,13 @@ const manifest: PluginManifest = {
|
||||
},
|
||||
};
|
||||
|
||||
export const portraitClearancePlugin: RegisteredPlugin = { manifest };
|
||||
export const portraitClearancePlugin: RegisteredPlugin = {
|
||||
manifest,
|
||||
agentActions: [prepareAnalysisNodeAction(PORTRAIT_CLEARANCE_NODE_TYPE, "肖像可识别性排查", () => ({ portraitClearance: createDefaultPortraitClearanceState() }))],
|
||||
readAgentNode: (node) => {
|
||||
const state = node.metadata?.portraitClearance;
|
||||
return { mode: state?.mode, task: state?.task, lastResult: state?.lastResult, execution: "仅返回已观测任务和结果；新分析仍通过插件自己的任务入口启动。" };
|
||||
},
|
||||
};
|
||||
|
||||
registerPlugin(portraitClearancePlugin);
|
||||
|
||||
@@ -51,6 +51,11 @@ function assertManifestV2(manifest: PluginManifestV2) {
|
||||
|
||||
export function registerPlugin(plugin: RegisteredPlugin) {
|
||||
assertManifest(plugin.manifest);
|
||||
const actionIds = new Set<string>();
|
||||
for (const action of plugin.agentActions || []) {
|
||||
if (!action.id.trim() || !action.description.trim() || actionIds.has(action.id) || typeof action.buildOperations !== "function") throw new Error("插件 Agent 动作需要唯一 ID、说明和操作构建器");
|
||||
actionIds.add(action.id);
|
||||
}
|
||||
const existing = registeredPlugins.get(plugin.manifest.id);
|
||||
if (existing && existing.manifest.version !== plugin.manifest.version) {
|
||||
throw new Error(`插件 ${plugin.manifest.id} 已注册其他版本`);
|
||||
|
||||
@@ -317,7 +317,18 @@ export type AssetSourceProvider = {
|
||||
openAsset?: (item: ExternalAssetItem) => Promise<void>;
|
||||
};
|
||||
|
||||
export type PluginAgentAction = {
|
||||
id: string;
|
||||
description: string;
|
||||
inputSchema: Record<string, unknown>;
|
||||
/** Pure planner: validate input and return operations, never perform side effects here. */
|
||||
buildOperations: (input: Record<string, unknown>, snapshot: import("@/lib/canvas/canvas-agent-ops").CanvasAgentSnapshot) => import("@/lib/canvas/canvas-agent-ops").CanvasAgentOp[];
|
||||
};
|
||||
|
||||
export type RegisteredPlugin = {
|
||||
agentActions?: PluginAgentAction[];
|
||||
/** Pure, bounded projection: never return embedded media or provider credentials. */
|
||||
readAgentNode?: (node: import("@/types/canvas").CanvasNodeData, snapshot: import("@/lib/canvas/canvas-agent-ops").CanvasAgentSnapshot) => Record<string, unknown>;
|
||||
/** v1 或 v2 插件清单；v2 清单结构为 v1 超集（含 editorSlots 声明）。 */
|
||||
manifest: PluginManifest | PluginManifestV2;
|
||||
source?: "bundled" | "uploaded" | string;
|
||||
|
||||
@@ -1526,7 +1526,8 @@ function InfiniteCanvasPage() {
|
||||
focusSelection: fitCanvasSelection,
|
||||
});
|
||||
|
||||
const { selectCanvasStyle, styleApplying } = useCanvasStyleWorkflow({
|
||||
const { selectCanvasStyle, applyCanvasStyleAsync, styleApplying } = useCanvasStyleWorkflow({
|
||||
canvasId: projectId,
|
||||
domainProjectId: currentProject?.projectId,
|
||||
nodesRef,
|
||||
selectedNodeIdsRef,
|
||||
@@ -2484,6 +2485,7 @@ function InfiniteCanvasPage() {
|
||||
onSelectNodeIds={setSelectedNodeIds}
|
||||
onSessionsChange={handleAssistantSessionsChange}
|
||||
onApplyOps={applyAgentOps}
|
||||
onApplyStyle={applyCanvasStyleAsync}
|
||||
canUndoOps={canUndoAgentOps}
|
||||
undoOpsCount={agentUndoCount}
|
||||
onUndoOps={undoAgentOps}
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { useCallback, type Dispatch, type SetStateAction } from "react";
|
||||
import { useCallback, useEffect, useRef, type Dispatch, type SetStateAction } from "react";
|
||||
import { useMutation, useQueryClient } from "@tanstack/react-query";
|
||||
import { App } from "antd";
|
||||
|
||||
@@ -7,8 +7,10 @@ import { createCanvasNode } from "@/lib/canvas/canvas-project-domain";
|
||||
import { createStyleProfileSnapshot, serializeStyleProfile } from "@/lib/canvas/style-profile";
|
||||
import { updateProject as updateDomainProject } from "@/services/api/projects";
|
||||
import { CanvasNodeType, type CanvasNodeData, type CanvasNodeMetadata, type Position } from "@/types/canvas";
|
||||
import { getActiveUserScope } from "@/lib/user-scope";
|
||||
|
||||
type UseCanvasStyleWorkflowOptions = {
|
||||
canvasId: string;
|
||||
domainProjectId?: string;
|
||||
nodesRef: { current: CanvasNodeData[] };
|
||||
selectedNodeIdsRef: { current: Set<string> };
|
||||
@@ -20,9 +22,13 @@ type UseCanvasStyleWorkflowOptions = {
|
||||
setStylePickerOpen: Dispatch<SetStateAction<boolean>>;
|
||||
};
|
||||
|
||||
export function useCanvasStyleWorkflow({ domainProjectId, nodesRef, selectedNodeIdsRef, getCanvasCenter, setNodes, setSelectedNodeIds, setSelectedConnectionId, setDialogNodeId, setStylePickerOpen }: UseCanvasStyleWorkflowOptions) {
|
||||
export function useCanvasStyleWorkflow({ canvasId, domainProjectId, nodesRef, selectedNodeIdsRef, getCanvasCenter, setNodes, setSelectedNodeIds, setSelectedConnectionId, setDialogNodeId, setStylePickerOpen }: UseCanvasStyleWorkflowOptions) {
|
||||
const { message } = App.useApp();
|
||||
const queryClient = useQueryClient();
|
||||
const liveContext = useRef({ canvasId, domainProjectId, mounted: true });
|
||||
liveContext.current = { canvasId, domainProjectId, mounted: liveContext.current.mounted };
|
||||
const applyingRef = useRef(false);
|
||||
useEffect(() => { liveContext.current.mounted = true; return () => { liveContext.current.mounted = false; }; }, []);
|
||||
|
||||
const applyCanvasStyle = useCallback(
|
||||
(preset: CanvasStylePreset, profileJson: string) => {
|
||||
@@ -66,25 +72,29 @@ export function useCanvasStyleWorkflow({ domainProjectId, nodesRef, selectedNode
|
||||
if (!domainProjectId) throw new Error("画布尚未关联项目");
|
||||
return updateDomainProject(domainProjectId, { stylePresetId: preset.id, styleProfileJson: profileJson });
|
||||
},
|
||||
onSuccess: (_project, { preset, profileJson }) => {
|
||||
applyCanvasStyle(preset, profileJson);
|
||||
void queryClient.invalidateQueries({ queryKey: ["project", domainProjectId] });
|
||||
},
|
||||
onError: (error) => message.error(error instanceof Error ? error.message : "项目画风保存失败"),
|
||||
});
|
||||
|
||||
const applyCanvasStyleAsync = useCallback(async (preset: CanvasStylePreset) => {
|
||||
if (applyingRef.current) throw new Error("画风正在保存，请稍后重试");
|
||||
const scope = getActiveUserScope();
|
||||
applyingRef.current = true;
|
||||
try {
|
||||
const profileJson = serializeStyleProfile(preset.profile || createStyleProfileSnapshot(preset));
|
||||
if (domainProjectId) {
|
||||
await persistStyleMutation.mutateAsync({ preset, profileJson });
|
||||
void queryClient.invalidateQueries({ queryKey: ["project", domainProjectId] });
|
||||
}
|
||||
if (!liveContext.current.mounted || liveContext.current.canvasId !== canvasId || liveContext.current.domainProjectId !== domainProjectId || getActiveUserScope() !== scope) throw new Error("页面或账号已变化，未向当前画布应用画风；如项目保存已完成，请回原项目查看");
|
||||
applyCanvasStyle(preset, profileJson);
|
||||
} finally { applyingRef.current = false; }
|
||||
}, [canvasId, domainProjectId, persistStyleMutation.mutateAsync, queryClient, applyCanvasStyle]);
|
||||
|
||||
const selectCanvasStyle = useCallback(
|
||||
(preset: CanvasStylePreset) => {
|
||||
if (persistStyleMutation.isPending) return;
|
||||
const profileJson = serializeStyleProfile(preset.profile || createStyleProfileSnapshot(preset));
|
||||
if (!domainProjectId) {
|
||||
applyCanvasStyle(preset, profileJson);
|
||||
return;
|
||||
}
|
||||
persistStyleMutation.mutate({ preset, profileJson });
|
||||
void applyCanvasStyleAsync(preset).catch((error) => message.error(error instanceof Error ? error.message : "项目画风保存失败"));
|
||||
},
|
||||
[applyCanvasStyle, domainProjectId, persistStyleMutation],
|
||||
[applyCanvasStyleAsync, message],
|
||||
);
|
||||
|
||||
return { selectCanvasStyle, styleApplying: persistStyleMutation.isPending };
|
||||
return { selectCanvasStyle, applyCanvasStyleAsync, styleApplying: persistStyleMutation.isPending };
|
||||
}
|
||||
|
||||
52
web/src/services/agent-capabilities.ts
Normal file
52
web/src/services/agent-capabilities.ts
Normal file
@@ -0,0 +1,52 @@
|
||||
import { getNodeDefinition, listNodeDefinitions } from "@/lib/canvas/node-registry";
|
||||
import { getRegisteredPlugin, listRegisteredPlugins } from "@/lib/plugins/plugin-registry";
|
||||
import { isPluginEffectivelyEnabled } from "@/stores/use-plugin-store";
|
||||
import type { CanvasAgentSnapshot } from "@/lib/canvas/canvas-agent-ops";
|
||||
|
||||
// Query live registries on every discovery; installation is not execution availability.
|
||||
export function listAgentCapabilities(query = "") {
|
||||
const nodes = listNodeDefinitions().filter((node) => !node.plugin || isPluginEffectivelyEnabled(node.plugin.pluginId)).map((node) => ({
|
||||
id: node.type, name: node.label, source: node.plugin?.pluginId || "builtin",
|
||||
scope: "canvas", actions: node.type === "script" ? ["create", "canvas_read_storyboard", "canvas_edit_storyboard"] : ["create", "read", "update"],
|
||||
defaultMetadata: node.defaultMetadata, defaultSize: node.defaultSize,
|
||||
inputKind: node.acceptsInputKind, schema: node.plugin?.schema,
|
||||
execution: node.plugin ? "仅开放节点创建和数据更新；插件专用执行需提供动作适配器，不能用媒体生成替代。" : "通过已有画布工具操作；专业节点不等同于普通媒体生成。",
|
||||
}));
|
||||
const plugins = listRegisteredPlugins().filter((plugin) => isPluginEffectivelyEnabled(plugin.manifest.id)).map(({ manifest, agentActions, readAgentNode }) => ({
|
||||
id: manifest.id, name: manifest.name, source: manifest.id, version: manifest.version,
|
||||
description: manifest.description, contributions: manifest.contributes,
|
||||
actions: (agentActions || []).map(({ id, description, inputSchema }) => ({ id, description, inputSchema })),
|
||||
readTool: readAgentNode ? "canvas_read_plugin_node" : undefined,
|
||||
execution: agentActions?.length ? "使用 canvas_plugin_action 调用 actions 中的真实动作。" : "贡献声明是发现信息，不是任意命令执行入口。只调用已挂载工具。",
|
||||
}));
|
||||
const terms = query.toLocaleLowerCase().trim().split(/\s+/).filter(Boolean);
|
||||
return [...nodes, ...plugins, { id: "project-style", name: "项目画风", source: "builtin", actions: ["canvas_list_styles", "canvas_apply_style"], scope: "canvas", description: "读取系统或个人风格，通过同一保存入口应用画风并同步关联项目。" }].filter((item) => terms.every((term) => JSON.stringify(item).toLocaleLowerCase().includes(term)));
|
||||
}
|
||||
|
||||
export function readAgentPluginNode(snapshot: CanvasAgentSnapshot, nodeId: string) {
|
||||
const node = snapshot.nodes.find((item) => item.id === nodeId);
|
||||
const owner = node && getNodeDefinition(node.type)?.plugin?.pluginId;
|
||||
const plugin = owner && isPluginEffectivelyEnabled(owner) ? getRegisteredPlugin(owner) : undefined;
|
||||
if (!node || !plugin?.readAgentNode || !plugin.manifest.permissions.includes("canvas.read")) throw new Error("当前插件未启用或未提供节点读取能力");
|
||||
return { nodeId, pluginId: owner, version: plugin.manifest.version, data: plugin.readAgentNode(structuredClone(node), structuredClone(snapshot)) };
|
||||
}
|
||||
|
||||
export function readAgentPluginDocumentation(pluginId: string) {
|
||||
const plugin = getRegisteredPlugin(pluginId);
|
||||
if (!plugin || !isPluginEffectivelyEnabled(pluginId)) throw new Error("插件不可用，请重新发现能力");
|
||||
return { id: pluginId, version: plugin.manifest.version, documentation: plugin.manifest.documentation || plugin.manifest.description };
|
||||
}
|
||||
|
||||
export function buildAgentPluginOperations(pluginId: string, actionId: string, input: Record<string, unknown>, snapshot: CanvasAgentSnapshot) {
|
||||
const plugin = getRegisteredPlugin(pluginId);
|
||||
if (!plugin || !isPluginEffectivelyEnabled(pluginId)) throw new Error("插件已停用或不可用");
|
||||
const action = plugin.agentActions?.find((item) => item.id === actionId);
|
||||
if (!action) throw new Error("插件未提供此 Agent 动作，请先读取能力目录");
|
||||
if (!plugin.manifest.permissions.includes("canvas.write")) throw new Error("插件未声明画布写入权限");
|
||||
const operations = action.buildOperations(structuredClone(input), structuredClone(snapshot));
|
||||
if (!Array.isArray(operations) || !operations.length || operations.length > 100) throw new Error("插件必须返回 1 至 100 个画布操作");
|
||||
if (operations.some((op) => op.type === "run_generation") && !plugin.manifest.permissions.includes("generation.run")) throw new Error("插件未声明生成权限");
|
||||
return operations;
|
||||
}
|
||||
|
||||
export const AGENT_CAPABILITY_GUIDANCE = "处理专业任务前先调用 canvas_list_capabilities 查找系统现有节点和已启用插件，并用 canvas_list_skills 检索相关专业技能、读取 SKILL.md 后按需使用；用户不必手动选择技能。不要用普通文本模拟分镜表或插件结果。发现信息不代表已经执行，未挂载的专用动作须如实说明。已授权范围内自主推进，不重复征求技能读取或普通操作许可。";
|
||||
@@ -143,6 +143,8 @@ type BackendToolGenerationOptions = {
|
||||
toolChoice: ToolChoice;
|
||||
signal?: AbortSignal;
|
||||
onDelta?: (text: string) => void;
|
||||
onTaskCreated?: (task: GenerationTask) => void;
|
||||
metadata?: { source: string; nodeId?: string; runId?: string; stage?: string };
|
||||
};
|
||||
|
||||
// 报价和执行复用完全相同的任务协议，准备阶段不提交模型任务。
|
||||
@@ -174,7 +176,7 @@ export function prepareBackendToolGenerationTask(options: BackendToolGenerationO
|
||||
config: backendProviderConfig(options.config, "text"),
|
||||
agentRequests: buildBackendToolRequests(options.messages, options.tools, options.toolChoice, options.config),
|
||||
referenceImages: [...imageKeys].map((storageKey) => ({ storageKey })),
|
||||
metadata: { source: "canvas-online-agent" },
|
||||
metadata: options.metadata || { source: "canvas-online-agent" },
|
||||
},
|
||||
};
|
||||
if (new Blob([JSON.stringify(task)]).size > 15 * 1024 * 1024) {
|
||||
@@ -185,6 +187,7 @@ export function prepareBackendToolGenerationTask(options: BackendToolGenerationO
|
||||
|
||||
export async function runBackendToolGenerationTask(options: BackendToolGenerationOptions): Promise<ToolResponseResult> {
|
||||
const task = await createGenerationTask(prepareBackendToolGenerationTask(options));
|
||||
options.onTaskCreated?.(task);
|
||||
const completed = await waitForGenerationTask(task.id, { signal: options.signal, initialTask: task, onTextDelta: options.onDelta });
|
||||
const result = parseBackendGenerationResult(completed);
|
||||
return {
|
||||
|
||||
52
web/src/services/art-critique-execution.ts
Normal file
52
web/src/services/art-critique-execution.ts
Normal file
@@ -0,0 +1,52 @@
|
||||
import { inspectAgentImage } from "./agent-image-preview";
|
||||
import { runBackendToolGenerationTask } from "./api/generation-task";
|
||||
import { runArtCritiquePipeline, type ArtCritiquePipelineOptions } from "@/lib/art-critique/pipeline";
|
||||
import { artCritiqueSourceFingerprint, isArtCritiqueImageInput } from "@/lib/art-critique/contracts";
|
||||
import type { CanvasNodeData } from "@/types/canvas";
|
||||
import type { AiConfig } from "@/stores/use-config-store";
|
||||
import { getActiveUserScope } from "@/lib/user-scope";
|
||||
|
||||
export const ART_CRITIQUE_MAX_MODEL_CALLS = 9;
|
||||
|
||||
// One execution boundary for manual UI and future approved Agent runs.
|
||||
// The executor may be replaced by an approval-aware task submitter; parsing stays in the pipeline.
|
||||
export async function executeArtCritique(input: {
|
||||
nodeId: string;
|
||||
runId: string;
|
||||
source: CanvasNodeData;
|
||||
config: AiConfig;
|
||||
signal: AbortSignal;
|
||||
onTaskCreated: (stage: string, taskId: string) => void;
|
||||
onStage?: ArtCritiquePipelineOptions["onStage"];
|
||||
onDraftReport?: ArtCritiquePipelineOptions["onDraftReport"];
|
||||
}, dependencies = { inspectImage: inspectAgentImage, runTask: runBackendToolGenerationTask, runPipeline: runArtCritiquePipeline }) {
|
||||
if (!input.nodeId || !input.runId || !isArtCritiqueImageInput(input.source)) throw new Error("分析需要真实目标节点、执行标识和已有图片");
|
||||
const scope = getActiveUserScope();
|
||||
const assertLive = () => {
|
||||
if (input.signal.aborted || getActiveUserScope() !== scope) throw new DOMException("分析已停止或账号已切换", "AbortError");
|
||||
};
|
||||
assertLive();
|
||||
const content = input.source.metadata?.content || input.source.metadata?.previewContent || "";
|
||||
const image = await dependencies.inspectImage({ storageKey: input.source.metadata?.storageKey, ...(content.startsWith("data:") ? { dataUrl: content } : { url: content }), name: input.source.title }, "original");
|
||||
assertLive();
|
||||
const calledStages = new Set<string>();
|
||||
return dependencies.runPipeline(input.config, { dataUrl: image.storageKey, title: input.source.title, sourceFingerprint: artCritiqueSourceFingerprint(input.source) }, {
|
||||
signal: input.signal,
|
||||
onStage: (stage) => { assertLive(); input.onStage?.(stage); },
|
||||
onDraftReport: (report) => { assertLive(); input.onDraftReport?.(report); },
|
||||
requestStage: async ({ config, messages, tool, toolName, signal }) => {
|
||||
assertLive();
|
||||
if (calledStages.has(toolName) || calledStages.size >= ART_CRITIQUE_MAX_MODEL_CALLS) throw new Error("分析阶段重复或超过本次调用数量上限");
|
||||
calledStages.add(toolName);
|
||||
const result = await dependencies.runTask({
|
||||
config, messages, tools: [tool], toolChoice: { type: "function", name: toolName },
|
||||
prompt: `图片审美分析：${toolName}`, signal,
|
||||
metadata: { source: "art-critique", nodeId: input.nodeId, runId: input.runId, stage: toolName },
|
||||
// A task can be accepted while the user stops observing; retain that receipt.
|
||||
onTaskCreated: (task) => { if (getActiveUserScope() === scope) input.onTaskCreated(toolName, task.id); },
|
||||
});
|
||||
assertLive();
|
||||
return result;
|
||||
},
|
||||
});
|
||||
}
|
||||
@@ -130,7 +130,7 @@ function functionTool(name: string, description: string, properties: Record<stri
|
||||
}
|
||||
|
||||
const PROGRESSIVE_SKILL_TOOLS: ResponseFunctionTool[] = [
|
||||
functionTool("canvas_list_skills", "列出当前用户已加入、可按需加载的画布技能；只返回元数据，不返回完整指令。", {}),
|
||||
functionTool("canvas_list_skills", "自动发现当前用户已加入的技能；可按 query 检索名称、描述或标签，offset/limit 分页；只返回元数据。", { query: { type: "string" }, offset: { type: "number" }, limit: { type: "number" } }),
|
||||
functionTool("canvas_get_skill", "按 skillId 或名称读取技能元数据和入口 SKILL.md；不会加载 references、scripts、assets 等其他文件。", { skillId: { type: "string" }, name: { type: "string" } }),
|
||||
functionTool("canvas_list_skill_files", "列出一个技能包的文件路径、类型和大小，不读取文件正文。", { skillId: { type: "string" }, name: { type: "string" } }),
|
||||
functionTool("canvas_read_skill_file", "按路径读取技能包中的一个文本文件。先从入口引用或文件清单确定路径，禁止猜测路径。", { skillId: { type: "string" }, name: { type: "string" }, path: { type: "string" } }, ["path"]),
|
||||
@@ -343,9 +343,13 @@ function nativePackage(skill: Skill, bundle: SkillPackageBundle): NativeSkillPac
|
||||
|
||||
function createProgressiveToolsAdapter(dependencies: SkillRuntimeDependencies): SkillToolAdapter {
|
||||
const handlers: Record<string, (args: Record<string, unknown>, skills: Skill[]) => Promise<SkillRuntimeToolResult>> = {
|
||||
canvas_list_skills: async (_args, skills) => {
|
||||
const data = skills.filter((skill) => skill.is_added).map((skill) => ({ skillId: skill.skill_id, name: skill.skill_name, description: skill.description, tag: skill.tag, version: skill.version, fileCount: skill.file_count, sourceType: skill.source_type }));
|
||||
return { ok: true, message: data.length ? "已列出当前可用技能。" : "当前没有已加入技能。", data };
|
||||
canvas_list_skills: async (args, skills) => {
|
||||
const terms = typeof args.query === "string" ? args.query.trim().toLocaleLowerCase().split(/\s+/).filter(Boolean) : [];
|
||||
const matches = skills.filter((skill) => skill.is_added && terms.every((term) => `${skill.skill_name} ${skill.description} ${skill.tag}`.toLocaleLowerCase().includes(term)));
|
||||
const offset = typeof args.offset === "number" && Number.isFinite(args.offset) ? Math.max(0, Math.floor(args.offset)) : 0;
|
||||
const limit = typeof args.limit === "number" && Number.isFinite(args.limit) ? Math.max(1, Math.min(100, Math.floor(args.limit))) : 40;
|
||||
const items = matches.slice(offset, offset + limit).map((skill) => ({ skillId: skill.skill_id, name: skill.skill_name, description: skill.description, tag: skill.tag, version: skill.version, fileCount: skill.file_count, sourceType: skill.source_type }));
|
||||
return { ok: true, message: items.length ? "已发现相关技能，请按需读取入口；无匹配时可缩短关键词或查询完整目录。" : "没有匹配技能，可调整关键词。", data: { items, total: matches.length, nextOffset: offset + items.length < matches.length ? offset + items.length : null } };
|
||||
},
|
||||
canvas_get_skill: async (args, skills) => {
|
||||
const skill = requireAddedSkill(skills, args);
|
||||
|
||||
41
web/test/agent-capabilities.test.ts
Normal file
41
web/test/agent-capabilities.test.ts
Normal file
@@ -0,0 +1,41 @@
|
||||
import { afterEach, expect, test } from "bun:test";
|
||||
import { registerPlugin, unregisterPlugin } from "../src/lib/plugins/plugin-registry";
|
||||
import { usePluginStore } from "../src/stores/use-plugin-store";
|
||||
import { listAgentCapabilities, buildAgentPluginOperations } from "../src/services/agent-capabilities";
|
||||
import { applyCanvasAgentOps, type CanvasAgentSnapshot } from "../src/lib/canvas/canvas-agent-ops";
|
||||
import { normalizeCreativeProposal, creativeProposalOps } from "../src/lib/creation/creative-agent-state";
|
||||
import { defaultConfig } from "../src/stores/use-config-store";
|
||||
|
||||
const snapshot: CanvasAgentSnapshot = { projectId: "test", title: "test", nodes: [], connections: [], selectedNodeIds: [], viewport: { x: 0, y: 0, k: 1 } };
|
||||
const pluginId = "agent-extension-test";
|
||||
afterEach(() => { unregisterPlugin(pluginId); usePluginStore.setState({ runtimeStatuses: {} }); });
|
||||
|
||||
test("专业方案创建可编辑分镜与上下文连线，不伪造画风已选择或收费任务", () => {
|
||||
const proposal = normalizeCreativeProposal({ title: "短剧", summary: "搭建", markdown: "完整方案", workflow: { nodes: [
|
||||
{ ref: "style", kind: "styleboard", title: "项目画风" },
|
||||
{ ref: "story", kind: "story_input", title: "故事", content: "两个人重逢" },
|
||||
{ ref: "shots", kind: "script", title: "分镜", shots: [{ durationSeconds: 5, videoMotionPrompt: "推近人物", dialogue: "好久不见" }] },
|
||||
], edges: [{ from: "style", to: "shots" }, { from: "story", to: "shots" }] }, generationItems: [] }, "p", 1, defaultConfig);
|
||||
const result = applyCanvasAgentOps(snapshot, creativeProposalOps("run", proposal, snapshot, defaultConfig));
|
||||
expect(result.nodes.find((node) => node.metadata?.workflowKind === "styleboard")?.metadata?.content).toBe("");
|
||||
const script = result.nodes.find((node) => node.type === "script")!;
|
||||
expect(script.metadata?.storyboard?.rows[0]?.dialogue).toBe("好久不见");
|
||||
expect(result.connections.every((edge) => edge.toHandleId === "storyboard:context")).toBe(true);
|
||||
expect(proposal.generationItems).toHaveLength(0);
|
||||
});
|
||||
|
||||
test("插件注册后可发现和创建真实类型，停用后立即拒绝新调用", () => {
|
||||
registerPlugin({ manifest: { id: pluginId, name: "测试扩展", version: "1", apiVersion: "yingce.plugin/v1", description: "测试", surfaces: ["node"], permissions: ["canvas.write"], runtime: { web: "declarative" }, contributes: { canvasNodes: [{ id: "test-special-node", label: "专用节点", defaultTitle: "测试", defaultSize: { width: 400, height: 300 }, schema: {}, renderer: "declarative" }] } }, agentActions: [{ id: "create", description: "创建", inputSchema: {}, buildOperations: () => [{ type: "add_node", nodeType: "test-special-node" }] }] });
|
||||
usePluginStore.setState({ runtimeStatuses: { [pluginId]: "enabled" } });
|
||||
expect(listAgentCapabilities("专用节点").length).toBeGreaterThan(0);
|
||||
const ops = buildAgentPluginOperations(pluginId, "create", {}, snapshot);
|
||||
expect(applyCanvasAgentOps(snapshot, ops).nodes[0]?.type).toBe("test-special-node");
|
||||
usePluginStore.setState({ runtimeStatuses: { [pluginId]: "disabled" } });
|
||||
expect(listAgentCapabilities("专用节点")).toHaveLength(0);
|
||||
expect(() => buildAgentPluginOperations(pluginId, "create", {}, snapshot)).toThrow();
|
||||
expect(() => applyCanvasAgentOps(snapshot, ops)).toThrow();
|
||||
});
|
||||
|
||||
test("未知节点不能静默降级成文本", () => {
|
||||
expect(() => applyCanvasAgentOps(snapshot, [{ type: "add_node", nodeType: "missing-plugin-node" }])).toThrow();
|
||||
});
|
||||
50
web/test/agent-storyboard.test.ts
Normal file
50
web/test/agent-storyboard.test.ts
Normal file
@@ -0,0 +1,50 @@
|
||||
import { expect, test } from "bun:test";
|
||||
import { buildAgentStoryboardOperations } from "../src/lib/canvas/canvas-agent-storyboard";
|
||||
import { createShortDramaPipeline } from "../src/lib/canvas/canvas-short-drama";
|
||||
import { createStoryboardRow } from "../src/lib/canvas/canvas-project-domain";
|
||||
import { applyCanvasAgentOps, type CanvasAgentSnapshot } from "../src/lib/canvas/canvas-agent-ops";
|
||||
import { prepareAnalysisNodeAction } from "../src/lib/plugins/analysis-node-action";
|
||||
|
||||
function fixture() {
|
||||
const pipeline = createShortDramaPipeline({ x: 0, y: 0 });
|
||||
const script = pipeline.nodes.find((node) => node.type === "script")!;
|
||||
script.metadata!.storyboard!.rows = [createStoryboardRow(1, { id: "row-1", videoNodeId: "video-1", imageNodeId: "image-1", dialogue: "旧对白" }), createStoryboardRow(2, { id: "row-2" })];
|
||||
const snapshot: CanvasAgentSnapshot = { projectId: "test", title: "test", nodes: pipeline.nodes, connections: pipeline.connections, selectedNodeIds: [], viewport: { x: 0, y: 0, k: 1 } };
|
||||
return { snapshot, script };
|
||||
}
|
||||
|
||||
test("按行修改保留其他镜头及已有生成素材关联", () => {
|
||||
const { snapshot, script } = fixture();
|
||||
const next = applyCanvasAgentOps(snapshot, buildAgentStoryboardOperations(snapshot, { nodeId: script.id, action: "update", rowId: "row-1", patch: { dialogue: "新对白", durationSeconds: 8 } }));
|
||||
const rows = next.nodes.find((node) => node.id === script.id)!.metadata!.storyboard!.rows;
|
||||
expect(rows[0]).toMatchObject({ id: "row-1", dialogue: "新对白", durationSeconds: 8, imageNodeId: "image-1", videoNodeId: "video-1" });
|
||||
expect(rows[1]).toEqual(script.metadata!.storyboard!.rows[1]);
|
||||
});
|
||||
|
||||
test("移除分镜只更新表格，保留行 ID 并重排序号", () => {
|
||||
const { snapshot, script } = fixture();
|
||||
const ops = buildAgentStoryboardOperations(snapshot, { nodeId: script.id, action: "remove", rowId: "row-1" });
|
||||
expect(ops).toHaveLength(1);
|
||||
expect(ops[0].type).toBe("update_node");
|
||||
if (ops[0].type === "update_node") expect(ops[0].metadata?.storyboard?.rows[0]).toMatchObject({ id: "row-2", shotNumber: 1 });
|
||||
});
|
||||
|
||||
test("分镜编辑拒绝伪造结果、未知行和非法时长", () => {
|
||||
const { snapshot, script } = fixture();
|
||||
for (const patch of [{ status: "success" }, { videoNodeId: "fake" }, { durationSeconds: -1 }]) {
|
||||
expect(() => buildAgentStoryboardOperations(snapshot, { nodeId: script.id, action: "update", rowId: "row-1", patch })).toThrow();
|
||||
}
|
||||
expect(() => buildAgentStoryboardOperations(snapshot, { nodeId: script.id, action: "remove", rowId: "missing" })).toThrow();
|
||||
});
|
||||
|
||||
test("分析节点准备复用相同输入且不启动模型", () => {
|
||||
const { snapshot, script } = fixture();
|
||||
snapshot.nodes.push({ ...script, id: "source", type: "image", metadata: { storageKey: "resource:image" } });
|
||||
const action = prepareAnalysisNodeAction("test-analysis", "分析", () => ({}));
|
||||
const ops = action.buildOperations({ sourceNodeId: "source" }, snapshot);
|
||||
expect(ops.some((op) => op.type === "run_generation")).toBe(false);
|
||||
expect(ops.some((op) => op.type === "connect_nodes")).toBe(true);
|
||||
snapshot.nodes.push({ ...script, id: "analysis", type: "test-analysis" });
|
||||
snapshot.connections.push({ id: "edge", fromNodeId: "source", toNodeId: "analysis" });
|
||||
expect(action.buildOperations({ sourceNodeId: "source" }, snapshot)).toEqual([{ type: "select_nodes", ids: ["analysis"] }]);
|
||||
});
|
||||
46
web/test/art-critique-execution.test.ts
Normal file
46
web/test/art-critique-execution.test.ts
Normal file
@@ -0,0 +1,46 @@
|
||||
import { expect, test } from "bun:test";
|
||||
import { executeArtCritique } from "../src/services/art-critique-execution";
|
||||
import { defaultConfig } from "../src/stores/use-config-store";
|
||||
import { createCanvasNode } from "../src/lib/canvas/canvas-project-domain";
|
||||
import { CanvasNodeType } from "../src/types/canvas";
|
||||
import type { ArtCritiquePipelineOptions } from "../src/lib/art-critique/pipeline";
|
||||
|
||||
const source = createCanvasNode(CanvasNodeType.Image, { x: 0, y: 0 }, { storageKey: "resource:original" });
|
||||
const options = () => ({ nodeId: "analysis", runId: "run", source, config: defaultConfig, signal: new AbortController().signal, onTaskCreated: (_stage: string, _id: string) => {} });
|
||||
|
||||
test("分析阶段使用资源引用、记录后台任务并拒绝重复阶段提交", async () => {
|
||||
let submitted = 0;
|
||||
const created: string[] = [];
|
||||
const input = options();
|
||||
input.onTaskCreated = (stage, id) => created.push(`${stage}:${id}`);
|
||||
await executeArtCritique(input, {
|
||||
inspectImage: async () => ({ storageKey: "resource:original", encodedBytes: 30 }),
|
||||
runTask: async (request) => {
|
||||
submitted++;
|
||||
expect(JSON.stringify(request.messages)).toContain("resource:original");
|
||||
expect(request.metadata).toMatchObject({ source: "art-critique", runId: "run", nodeId: "analysis" });
|
||||
request.onTaskCreated?.({ id: "task-1" } as never);
|
||||
return { content: "{}", toolCalls: [] };
|
||||
},
|
||||
runPipeline: async (config, image, opts: ArtCritiquePipelineOptions = {}) => {
|
||||
expect(image.dataUrl).toBe("resource:original");
|
||||
const stage = { config, messages: [{ role: "user" as const, content: [{ type: "image_url" as const, image_url: { url: image.dataUrl } }] }], tool: { type: "function" as const, name: "review", description: "review", parameters: {} }, toolName: "review", signal: input.signal };
|
||||
await opts.requestStage!(stage);
|
||||
await expect(opts.requestStage!(stage)).rejects.toThrow("重复");
|
||||
return {} as never;
|
||||
},
|
||||
});
|
||||
expect(submitted).toBe(1);
|
||||
expect(created).toEqual(["review:task-1"]);
|
||||
});
|
||||
|
||||
test("准备素材期间取消不会提交后台模型任务", async () => {
|
||||
const controller = new AbortController();
|
||||
let submitted = false;
|
||||
await expect(executeArtCritique({ ...options(), signal: controller.signal }, {
|
||||
inspectImage: async () => { controller.abort(); return { storageKey: "resource:image", encodedBytes: 30 }; },
|
||||
runTask: async () => { submitted = true; return { content: "", toolCalls: [] }; },
|
||||
runPipeline: async () => { throw new Error("不应开始分析"); },
|
||||
})).rejects.toThrow();
|
||||
expect(submitted).toBe(false);
|
||||
});
|
||||
@@ -54,6 +54,15 @@ function file(path: string, content: string, kind: SkillPackageFile["kind"] = "m
|
||||
}
|
||||
|
||||
describe("skill runtime", () => {
|
||||
test("可检索首屏之外的技能并分页，不暴露未加入技能", async () => {
|
||||
const runtime = createSkillRuntime();
|
||||
const skills = Array.from({ length: 45 }, (_, index) => skill({ skill_id: `skill-${index}`, skill_name: `技能 ${index}`, description: index === 44 ? "商品构图" : "通用" }));
|
||||
skills.push(skill({ skill_id: "hidden", description: "商品构图", is_added: false }));
|
||||
const first = await runtime.executeAgentTool("onlineAgent", "canvas_list_skills", {}, skills);
|
||||
expect(first?.ok && first.data).toMatchObject({ total: 45, nextOffset: 40 });
|
||||
const found = await runtime.executeAgentTool("onlineAgent", "canvas_list_skills", { query: "商品" }, skills);
|
||||
expect(found?.ok && found.data).toMatchObject({ total: 1, nextOffset: null, items: [{ skillId: "skill-44" }] });
|
||||
});
|
||||
test("技能引用解析由统一规则同时支持稳定 token 和自然提及", () => {
|
||||
const director = skill();
|
||||
const storyboard = skill({ skill_id: "storyboard", skill_name: "小说转分镜" });
|
||||
|
||||
Reference in New Issue
Block a user