Compare commits
4 Commits
23bf383c04
...
c3716d7a6c
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c3716d7a6c | ||
|
|
a2d8a774ca | ||
|
|
b6eaa2a1b1 | ||
|
|
28ba7da64a |
@@ -6,7 +6,10 @@
|
|||||||
"Bash(npm run check:*)",
|
"Bash(npm run check:*)",
|
||||||
"Bash(npx vitest:*)",
|
"Bash(npx vitest:*)",
|
||||||
"Bash(node -e:*)",
|
"Bash(node -e:*)",
|
||||||
"Bash(npm run build:*)"
|
"Bash(npm run build:*)",
|
||||||
|
"Bash(npm run test:*)",
|
||||||
|
"Bash(python -m py_compile:*)",
|
||||||
|
"Bash(node --check:*)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"enableAllProjectMcpServers": true,
|
"enableAllProjectMcpServers": true,
|
||||||
|
|||||||
137
DEMAND-queryFiled.md
Normal file
137
DEMAND-queryFiled.md
Normal file
@@ -0,0 +1,137 @@
|
|||||||
|
|
||||||
|
|
||||||
|
**Role:** 资深前端架构师 (Svelte 5 + TypeScript)
|
||||||
|
|
||||||
|
**Project Pivot (重大架构调整):**
|
||||||
|
我们将 "Excel2JSON Mapper" 升级为一个 **"ETL 配置生成器 (ETL Blueprint Generator)"**。
|
||||||
|
前端的任务是生成一个包含 **[源数据 + 处理逻辑]** 的 **Job Bundle (JSON 文件)**,用户将使用该文件配合 Python 脚本在后端执行实际的数据抓取和入库。
|
||||||
|
|
||||||
|
### Phase 2: ETL 配置生成器需求文档
|
||||||
|
|
||||||
|
#### 1. 项目概述
|
||||||
|
|
||||||
|
这是一个基于 Svelte 5 的单页应用。用户上传 Excel,配置字段映射规则(包含静态重命名和动态 API 获取规则),最后导出为一个标准化的 JSON 任务包 (`job_bundle.json`)。
|
||||||
|
|
||||||
|
#### 2. 用户界面与交互 (UI/UX)
|
||||||
|
|
||||||
|
##### 2.1 顶部工具栏
|
||||||
|
|
||||||
|
* **导入/导出配置:** 支持保存当前所有的映射规则。
|
||||||
|
* **导出任务包 (Export Job Bundle):** 這是核心操作。点击后下载 `job_bundle.json` 文件(包含数据+配置)。
|
||||||
|
* **提交设置 (Submission Settings):** 一个模态框,配置最终数据推送到哪里。
|
||||||
|
* `Target URL`: 最终数据接收接口 (e.g., `https://api.db.com/bulk-insert`).
|
||||||
|
* `Method`: POST / PUT.
|
||||||
|
* `Batch Size`: 批次大小 (默认 50).
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
##### 2.2 主体区域 (左右分栏)
|
||||||
|
|
||||||
|
* **左侧 (Source):** Excel 表格预览。
|
||||||
|
* **右侧 (Preview):** 静态映射后的 JSON 预览(仅展示前 20 条以保证性能)。
|
||||||
|
|
||||||
|
##### 2.3 核心功能:列配置 (Column Configuration)
|
||||||
|
|
||||||
|
在左侧表格区域,除了点击现有表头修改映射外,新增 **"添加计算列 (Add Computed Column)"** 功能。
|
||||||
|
|
||||||
|
**新增类型:`API_FETCH` (动态 API 字段)**
|
||||||
|
当用户选择此类型时,弹出一个详细配置面板:
|
||||||
|
|
||||||
|
1. **Target Key:** 最终生成的 JSON 字段名 (例如 `user_balance`)。
|
||||||
|
2. **Request URL (支持模板变量):**
|
||||||
|
* 允许使用 `{{ColumnName}}` 语法引用当前行的 Excel 数据。
|
||||||
|
* *示例:* `https://api.example.com/users/{{用户ID}}/detail`
|
||||||
|
* *UI 交互:* 输入框旁应有“插入变量”按钮,点击列出所有可用 Excel 表头。
|
||||||
|
|
||||||
|
|
||||||
|
3. **Request Method:** 下拉选择 `GET` (默认) 或 `POST`。
|
||||||
|
4. **Headers:** Key-Value 编辑器 (用于传 `Authorization`, `Content-Type` 等)。
|
||||||
|
5. **Request Body (仅 POST):**
|
||||||
|
* 多行文本域,支持 JSON 格式。
|
||||||
|
* 同样支持 `{{ColumnName}}` 模板变量替换。
|
||||||
|
|
||||||
|
|
||||||
|
6. **Response Extractor (取值路径):**
|
||||||
|
* 指定从接口返回的 JSON 中提取哪个字段。
|
||||||
|
* 支持 `lodash.get` 风格的点号路径。
|
||||||
|
* *示例:* 接口返回 `{ "data": { "balance": 100 } }`,用户填 `data.balance`。
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
#### 3. 核心输出:Job Bundle 数据结构
|
||||||
|
|
||||||
|
请严格按照以下 TypeScript 接口定义生成导出的 JSON 文件:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// 1. 静态映射规则
|
||||||
|
interface StaticRule {
|
||||||
|
type: 'static';
|
||||||
|
source: string; // Excel 原表头
|
||||||
|
target: string; // JSON 目标 Key
|
||||||
|
dataType: 'string' | 'number' | 'boolean' | 'date';
|
||||||
|
format?: string; // 日期格式化字符串
|
||||||
|
}
|
||||||
|
|
||||||
|
// 2. 动态 API 获取规则 (本次新增核心)
|
||||||
|
interface ApiEnrichmentRule {
|
||||||
|
type: 'api_fetch';
|
||||||
|
target_key: string; // JSON 目标 Key
|
||||||
|
url_template: string; // "https://api.com/{{id}}"
|
||||||
|
method: 'GET' | 'POST';
|
||||||
|
headers?: Record<string, string>;
|
||||||
|
body_template?: string; // POST body 模板
|
||||||
|
response_path: string; // "data.result.value"
|
||||||
|
fallback_value?: any; // 默认值 (null/0)
|
||||||
|
}
|
||||||
|
|
||||||
|
// 3. 提交配置
|
||||||
|
interface SubmissionConfig {
|
||||||
|
target_url: string;
|
||||||
|
method: 'POST' | 'PUT';
|
||||||
|
batch_size: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
// 4. 最终导出的 Job Bundle 结构
|
||||||
|
interface JobBundle {
|
||||||
|
meta: {
|
||||||
|
version: string;
|
||||||
|
generated_at: string;
|
||||||
|
};
|
||||||
|
config: {
|
||||||
|
static_rules: StaticRule[];
|
||||||
|
enrichment_rules: ApiEnrichmentRule[];
|
||||||
|
submission: SubmissionConfig;
|
||||||
|
};
|
||||||
|
source_data: Record<string, any>[]; // 经过静态映射后的基础数据列表
|
||||||
|
}
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 4. 开发任务清单
|
||||||
|
|
||||||
|
1. **Store 设计:** 更新 Svelte Store 以存储 `enrichmentRules` 和 `submissionConfig`。
|
||||||
|
2. **UI 组件:**
|
||||||
|
* 开发 `ApiConfigModal.svelte`: 用于录入 URL、Headers、Body 等复杂信息。
|
||||||
|
* 实现变量插入辅助功能 (点击列名自动插入 `{{...}}`)。
|
||||||
|
|
||||||
|
|
||||||
|
3. **导出逻辑:** 编写 `generateJobBundle` 函数。
|
||||||
|
* **步骤 1:** 根据 `static_rules` 转换 Excel 原始数据,生成基础 JSON 数组。
|
||||||
|
* **步骤 2:** 将基础数据、API 规则、提交配置组装成 `JobBundle` 格式。
|
||||||
|
* **步骤 3:** 触发浏览器下载 `job_bundle.json`。
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
#### 5. 特别说明 (给 AI 的提示)
|
||||||
|
|
||||||
|
* **No Runtime Fetch:** 前端代码 **不需要** 执行 `fetch` 去调用用户配置的 API(避免 CORS)。前端只负责把 URL 字符串保存到 JSON 里。
|
||||||
|
* **Template Validation:** 在 UI 上简单校验 URL 模板格式(检查是否包含 `{}`),但不做逻辑校验。
|
||||||
|
* **Preview Limitations:** 右侧预览仅展示静态映射的结果。对于 API 字段,可以在预览中显示一个占位符(如 `[Pending API Fetch]`)。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 后续步骤
|
||||||
|
|
||||||
|
Claude 完成这个前端代码后,你可以再发一条指令让它写对应的 Python 执行脚本:
|
||||||
|
|
||||||
|
> "前端已经完成了。现在请帮我写一个 Python 脚本 (`executor.py`)。它读取上面定义的 `job_bundle.json`,使用 http请求 执行 `enrichment_rules` 里的请求(注意替换 URL 中的 {{变量}}),最后把结果推送到 `submission` 定义的接口。"
|
||||||
@@ -106,9 +106,10 @@ npm run preview
|
|||||||
- 当 `excludeIfEmpty = true` 时,空值字段不会出现在输出 JSON 中
|
- 当 `excludeIfEmpty = true` 时,空值字段不会出现在输出 JSON 中
|
||||||
- 若目标字段包含 `.`,会按路径写入嵌套对象
|
- 若目标字段包含 `.`,会按路径写入嵌套对象
|
||||||
|
|
||||||
## <20><><EFBFBD><EFBFBD><EFBFBD><EFBFBD>ͼ
|
|
||||||
|
|
||||||

|
## Screenshots
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
|||||||
201
scripts/node/enricher.mjs
Normal file
201
scripts/node/enricher.mjs
Normal file
@@ -0,0 +1,201 @@
|
|||||||
|
/**
|
||||||
|
* enricher.mjs — API 数据补全脚本 (Node.js)
|
||||||
|
*
|
||||||
|
* 读取前端生成的 job_bundle.json,执行 enrichment_rules 中的 API 请求,
|
||||||
|
* 将结果合并到 source_data 中,输出最终完整的 JSON 文件。
|
||||||
|
*
|
||||||
|
* 用法:
|
||||||
|
* node enricher.mjs job_bundle.json
|
||||||
|
* node enricher.mjs job_bundle.json -o enriched_data.json
|
||||||
|
* node enricher.mjs job_bundle.json --concurrency 10
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { readFileSync, writeFileSync } from "fs";
|
||||||
|
import { basename, join, dirname } from "path";
|
||||||
|
|
||||||
|
// ── helpers ──
|
||||||
|
|
||||||
|
function resolvePath(obj, path) {
|
||||||
|
const keys = path.split(".");
|
||||||
|
let current = obj;
|
||||||
|
for (const key of keys) {
|
||||||
|
if (current == null) return null;
|
||||||
|
if (Array.isArray(current)) {
|
||||||
|
const idx = Number(key);
|
||||||
|
if (Number.isNaN(idx) || idx < 0 || idx >= current.length) return null;
|
||||||
|
current = current[idx];
|
||||||
|
} else if (typeof current === "object") {
|
||||||
|
current = current[key];
|
||||||
|
} else {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return current ?? null;
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderTemplate(template, row) {
|
||||||
|
return template.replace(/\{\{(.+?)\}\}/g, (_, key) => {
|
||||||
|
const val = row[key];
|
||||||
|
return val != null ? String(val) : "";
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── fetch one ──
|
||||||
|
|
||||||
|
async function fetchOne(rule, row, rowIndex, semaphore) {
|
||||||
|
const targetKey = rule.target_key;
|
||||||
|
const fallback = rule.fallback_value ?? null;
|
||||||
|
const url = renderTemplate(rule.url_template, row);
|
||||||
|
const method = (rule.method || "GET").toUpperCase();
|
||||||
|
|
||||||
|
const headers = {};
|
||||||
|
if (rule.headers) {
|
||||||
|
for (const [k, v] of Object.entries(rule.headers)) {
|
||||||
|
headers[k] = renderTemplate(v, row);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** @type {RequestInit} */
|
||||||
|
const opts = { method, headers };
|
||||||
|
|
||||||
|
if (method === "POST" && rule.body_template) {
|
||||||
|
const body = renderTemplate(rule.body_template, row);
|
||||||
|
try {
|
||||||
|
JSON.parse(body);
|
||||||
|
headers["Content-Type"] = headers["Content-Type"] || "application/json";
|
||||||
|
opts.body = body;
|
||||||
|
} catch {
|
||||||
|
opts.body = body;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
await semaphore.acquire();
|
||||||
|
try {
|
||||||
|
const resp = await fetch(url, opts);
|
||||||
|
if (!resp.ok) {
|
||||||
|
console.log(` [WARN] Row ${rowIndex} | ${targetKey} | HTTP ${resp.status} <- ${url}`);
|
||||||
|
return { rowIndex, targetKey, value: fallback };
|
||||||
|
}
|
||||||
|
const data = await resp.json();
|
||||||
|
const value = resolvePath(data, rule.response_path);
|
||||||
|
return { rowIndex, targetKey, value: value ?? fallback };
|
||||||
|
} catch (e) {
|
||||||
|
console.log(` [ERROR] Row ${rowIndex} | ${targetKey} | ${e.message}`);
|
||||||
|
return { rowIndex, targetKey, value: fallback };
|
||||||
|
} finally {
|
||||||
|
semaphore.release();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── semaphore ──
|
||||||
|
|
||||||
|
function createSemaphore(max) {
|
||||||
|
let current = 0;
|
||||||
|
/** @type {(() => void)[]} */
|
||||||
|
const queue = [];
|
||||||
|
return {
|
||||||
|
acquire() {
|
||||||
|
if (current < max) {
|
||||||
|
current++;
|
||||||
|
return Promise.resolve();
|
||||||
|
}
|
||||||
|
return new Promise((resolve) => queue.push(resolve));
|
||||||
|
},
|
||||||
|
release() {
|
||||||
|
current--;
|
||||||
|
if (queue.length > 0) {
|
||||||
|
current++;
|
||||||
|
queue.shift()();
|
||||||
|
}
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── main ──
|
||||||
|
|
||||||
|
function parseArgs() {
|
||||||
|
const args = process.argv.slice(2);
|
||||||
|
const opts = { bundle: "", output: "", concurrency: 5 };
|
||||||
|
for (let i = 0; i < args.length; i++) {
|
||||||
|
if (args[i] === "-o" || args[i] === "--output") {
|
||||||
|
opts.output = args[++i];
|
||||||
|
} else if (args[i] === "--concurrency") {
|
||||||
|
opts.concurrency = Number(args[++i]) || 5;
|
||||||
|
} else if (args[i] === "--help" || args[i] === "-h") {
|
||||||
|
console.log("Usage: node enricher.mjs <job_bundle.json> [-o output.json] [--concurrency N]");
|
||||||
|
process.exit(0);
|
||||||
|
} else if (!opts.bundle) {
|
||||||
|
opts.bundle = args[i];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!opts.bundle) {
|
||||||
|
console.error("Error: Please provide a job_bundle.json path.");
|
||||||
|
console.error("Usage: node enricher.mjs <job_bundle.json>");
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
return opts;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function main() {
|
||||||
|
const opts = parseArgs();
|
||||||
|
|
||||||
|
const raw = readFileSync(opts.bundle, "utf-8");
|
||||||
|
const bundle = JSON.parse(raw);
|
||||||
|
|
||||||
|
const meta = bundle.meta || {};
|
||||||
|
const config = bundle.config || {};
|
||||||
|
const sourceData = bundle.source_data || [];
|
||||||
|
const rules = config.enrichment_rules || [];
|
||||||
|
|
||||||
|
console.log(`=== Job Bundle v${meta.version || "?"} ===`);
|
||||||
|
console.log(`Generated: ${meta.generated_at || "?"}`);
|
||||||
|
console.log(`Rows: ${sourceData.length}`);
|
||||||
|
console.log(`Static rules: ${(config.static_rules || []).length}`);
|
||||||
|
console.log(`Enrichment rules: ${rules.length}`);
|
||||||
|
console.log();
|
||||||
|
|
||||||
|
if (rules.length === 0) {
|
||||||
|
console.log("No enrichment rules configured. Outputting source data as-is.");
|
||||||
|
} else {
|
||||||
|
const totalCalls = sourceData.length * rules.length;
|
||||||
|
console.log(`Enriching ${sourceData.length} rows x ${rules.length} rule(s) = ${totalCalls} API calls`);
|
||||||
|
console.log(`Concurrency: ${opts.concurrency}`);
|
||||||
|
console.log();
|
||||||
|
|
||||||
|
const semaphore = createSemaphore(opts.concurrency);
|
||||||
|
const tasks = [];
|
||||||
|
for (let rowIdx = 0; rowIdx < sourceData.length; rowIdx++) {
|
||||||
|
for (const rule of rules) {
|
||||||
|
tasks.push(fetchOne(rule, sourceData[rowIdx], rowIdx, semaphore));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const results = await Promise.all(tasks);
|
||||||
|
|
||||||
|
let errorCount = 0;
|
||||||
|
for (const r of results) {
|
||||||
|
if (r.value === undefined) errorCount++;
|
||||||
|
sourceData[r.rowIndex][r.targetKey] = r.value;
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log();
|
||||||
|
console.log(`Done. ${totalCalls - errorCount}/${totalCalls} calls succeeded.`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const output = {
|
||||||
|
meta,
|
||||||
|
submission: config.submission || {},
|
||||||
|
data: sourceData,
|
||||||
|
};
|
||||||
|
|
||||||
|
const outName = opts.output || opts.bundle.replace(/\.json$/i, "_enriched.json");
|
||||||
|
writeFileSync(outName, JSON.stringify(output, null, 2), "utf-8");
|
||||||
|
|
||||||
|
console.log(`\nEnriched data saved to: ${outName}`);
|
||||||
|
console.log(`Next step: node submitter.mjs ${outName}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
main().catch((e) => {
|
||||||
|
console.error(e);
|
||||||
|
process.exit(1);
|
||||||
|
});
|
||||||
177
scripts/node/submitter.mjs
Normal file
177
scripts/node/submitter.mjs
Normal file
@@ -0,0 +1,177 @@
|
|||||||
|
/**
|
||||||
|
* submitter.mjs — 数据提交脚本 (Node.js)
|
||||||
|
*
|
||||||
|
* 读取 enricher.mjs 输出的 JSON 文件,按批次提交到目标接口。
|
||||||
|
* 记录提交成功和失败的记录到单独的日志文件。
|
||||||
|
*
|
||||||
|
* 用法:
|
||||||
|
* node submitter.mjs enriched_data.json
|
||||||
|
* node submitter.mjs enriched_data.json --batch-size 100
|
||||||
|
* node submitter.mjs enriched_data.json --url https://api.example.com/import
|
||||||
|
* node submitter.mjs enriched_data.json --dry-run
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { readFileSync, writeFileSync } from "fs";
|
||||||
|
|
||||||
|
// ── args ──
|
||||||
|
|
||||||
|
function parseArgs() {
|
||||||
|
const args = process.argv.slice(2);
|
||||||
|
const opts = { input: "", url: "", method: "", batchSize: 0, dryRun: false };
|
||||||
|
for (let i = 0; i < args.length; i++) {
|
||||||
|
if (args[i] === "--url") {
|
||||||
|
opts.url = args[++i];
|
||||||
|
} else if (args[i] === "--method") {
|
||||||
|
opts.method = args[++i];
|
||||||
|
} else if (args[i] === "--batch-size") {
|
||||||
|
opts.batchSize = Number(args[++i]) || 0;
|
||||||
|
} else if (args[i] === "--dry-run") {
|
||||||
|
opts.dryRun = true;
|
||||||
|
} else if (args[i] === "--help" || args[i] === "-h") {
|
||||||
|
console.log("Usage: node submitter.mjs <enriched.json> [--url URL] [--method POST|PUT] [--batch-size N] [--dry-run]");
|
||||||
|
process.exit(0);
|
||||||
|
} else if (!opts.input) {
|
||||||
|
opts.input = args[i];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!opts.input) {
|
||||||
|
console.error("Error: Please provide an enriched JSON file path.");
|
||||||
|
console.error("Usage: node submitter.mjs <enriched.json>");
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
return opts;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── submit ──
|
||||||
|
|
||||||
|
async function submitBatch(url, method, batch, batchIndex, totalBatches) {
|
||||||
|
try {
|
||||||
|
const resp = await fetch(url, {
|
||||||
|
method,
|
||||||
|
headers: { "Content-Type": "application/json" },
|
||||||
|
body: JSON.stringify(batch),
|
||||||
|
});
|
||||||
|
const body = await resp.text();
|
||||||
|
const ok = resp.status < 400;
|
||||||
|
const tag = ok ? "OK" : "FAIL";
|
||||||
|
console.log(` Batch ${batchIndex}/${totalBatches}: HTTP ${resp.status} ${tag} (${batch.length} records)`);
|
||||||
|
return { ok, status: resp.status, body, data: batch };
|
||||||
|
} catch (e) {
|
||||||
|
const msg = e.message;
|
||||||
|
console.log(` Batch ${batchIndex}/${totalBatches}: ERROR - ${msg} (${batch.length} records)`);
|
||||||
|
return { ok: false, status: 0, body: msg, data: batch };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── main ──
|
||||||
|
|
||||||
|
async function main() {
|
||||||
|
const opts = parseArgs();
|
||||||
|
|
||||||
|
const raw = readFileSync(opts.input, "utf-8");
|
||||||
|
const payload = JSON.parse(raw);
|
||||||
|
|
||||||
|
const data = payload.data || [];
|
||||||
|
const submission = payload.submission || {};
|
||||||
|
|
||||||
|
const targetUrl = opts.url || submission.target_url || "";
|
||||||
|
const method = (opts.method || submission.method || "POST").toUpperCase();
|
||||||
|
const batchSize = opts.batchSize || submission.batch_size || 50;
|
||||||
|
|
||||||
|
if (!targetUrl) {
|
||||||
|
console.error("Error: No target URL configured. Use --url or set in job bundle.");
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log("=== Submitter ===");
|
||||||
|
console.log(`Input: ${opts.input} (${data.length} records)`);
|
||||||
|
console.log(`Target: ${method} ${targetUrl}`);
|
||||||
|
console.log(`Batch size: ${batchSize}`);
|
||||||
|
console.log();
|
||||||
|
|
||||||
|
// 分批
|
||||||
|
const batches = [];
|
||||||
|
for (let i = 0; i < data.length; i += batchSize) {
|
||||||
|
batches.push(data.slice(i, i + batchSize));
|
||||||
|
}
|
||||||
|
const totalBatches = batches.length;
|
||||||
|
|
||||||
|
if (opts.dryRun) {
|
||||||
|
console.log(`[DRY RUN] Would submit ${data.length} records in ${totalBatches} batch(es)`);
|
||||||
|
for (let i = 0; i < batches.length; i++) {
|
||||||
|
console.log(` Batch ${i + 1}/${totalBatches}: ${batches[i].length} records`);
|
||||||
|
}
|
||||||
|
console.log("\nDry run complete. No data was sent.");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** @type {object[]} */
|
||||||
|
const successRecords = [];
|
||||||
|
/** @type {object[]} */
|
||||||
|
const failedRecords = [];
|
||||||
|
/** @type {object[]} */
|
||||||
|
const failedDetails = [];
|
||||||
|
|
||||||
|
console.log(`Submitting ${data.length} records in ${totalBatches} batch(es)...\n`);
|
||||||
|
|
||||||
|
for (let i = 0; i < batches.length; i++) {
|
||||||
|
const result = await submitBatch(targetUrl, method, batches[i], i + 1, totalBatches);
|
||||||
|
if (result.ok) {
|
||||||
|
successRecords.push(...result.data);
|
||||||
|
} else {
|
||||||
|
failedRecords.push(...result.data);
|
||||||
|
failedDetails.push({
|
||||||
|
batch_index: i + 1,
|
||||||
|
status: result.status,
|
||||||
|
response: result.body.slice(0, 500),
|
||||||
|
record_count: result.data.length,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 结果统计
|
||||||
|
console.log();
|
||||||
|
console.log("=== Result ===");
|
||||||
|
console.log(`Success: ${successRecords.length} records`);
|
||||||
|
console.log(`Failed: ${failedRecords.length} records`);
|
||||||
|
|
||||||
|
const timestamp = new Date().toISOString().replace(/[-:T]/g, "").slice(0, 15);
|
||||||
|
const baseName = opts.input.replace(/\.json$/i, "");
|
||||||
|
|
||||||
|
if (successRecords.length > 0) {
|
||||||
|
const successFile = `${baseName}_success_${timestamp}.json`;
|
||||||
|
writeFileSync(successFile, JSON.stringify(successRecords, null, 2), "utf-8");
|
||||||
|
console.log(`\nSuccess log: ${successFile}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (failedRecords.length > 0) {
|
||||||
|
const failedFile = `${baseName}_failed_${timestamp}.json`;
|
||||||
|
const report = {
|
||||||
|
summary: {
|
||||||
|
total_failed: failedRecords.length,
|
||||||
|
failed_batches: failedDetails.length,
|
||||||
|
target_url: targetUrl,
|
||||||
|
timestamp,
|
||||||
|
},
|
||||||
|
batch_errors: failedDetails,
|
||||||
|
failed_records: failedRecords,
|
||||||
|
};
|
||||||
|
writeFileSync(failedFile, JSON.stringify(report, null, 2), "utf-8");
|
||||||
|
console.log(`Failed log: ${failedFile}`);
|
||||||
|
|
||||||
|
const retryFile = `${baseName}_retry_${timestamp}.json`;
|
||||||
|
const retryPayload = { submission, data: failedRecords };
|
||||||
|
writeFileSync(retryFile, JSON.stringify(retryPayload, null, 2), "utf-8");
|
||||||
|
console.log(`\nTo retry failed records:`);
|
||||||
|
console.log(` node submitter.mjs ${retryFile}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (successRecords.length === 0 && failedRecords.length === 0) {
|
||||||
|
console.log("\nNo records to submit.");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
main().catch((e) => {
|
||||||
|
console.error(e);
|
||||||
|
process.exit(1);
|
||||||
|
});
|
||||||
186
scripts/py/enricher.py
Normal file
186
scripts/py/enricher.py
Normal file
@@ -0,0 +1,186 @@
|
|||||||
|
"""
|
||||||
|
enricher.py — API 数据补全脚本
|
||||||
|
|
||||||
|
读取前端生成的 job_bundle.json,执行 enrichment_rules 中的 API 请求,
|
||||||
|
将结果合并到 source_data 中,输出最终完整的 JSON 文件。
|
||||||
|
|
||||||
|
用法:
|
||||||
|
python enricher.py job_bundle.json
|
||||||
|
python enricher.py job_bundle.json -o enriched_data.json
|
||||||
|
python enricher.py job_bundle.json --concurrency 10
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import aiohttp
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_path(obj, path: str):
|
||||||
|
"""按点号路径从 dict/list 中取值,类似 lodash.get"""
|
||||||
|
keys = path.split(".")
|
||||||
|
current = obj
|
||||||
|
for key in keys:
|
||||||
|
if isinstance(current, dict) and key in current:
|
||||||
|
current = current[key]
|
||||||
|
elif isinstance(current, list):
|
||||||
|
try:
|
||||||
|
current = current[int(key)]
|
||||||
|
except (ValueError, IndexError):
|
||||||
|
return None
|
||||||
|
else:
|
||||||
|
return None
|
||||||
|
return current
|
||||||
|
|
||||||
|
|
||||||
|
def render_template(template: str, row: dict) -> str:
|
||||||
|
"""将 {{变量名}} 替换为行数据中的对应值"""
|
||||||
|
def replacer(match: re.Match) -> str:
|
||||||
|
key = match.group(1)
|
||||||
|
value = row.get(key, "")
|
||||||
|
return str(value) if value is not None else ""
|
||||||
|
|
||||||
|
return re.sub(r"\{\{(.+?)\}\}", replacer, template)
|
||||||
|
|
||||||
|
|
||||||
|
async def fetch_one(
|
||||||
|
session: aiohttp.ClientSession,
|
||||||
|
rule: dict,
|
||||||
|
row: dict,
|
||||||
|
row_index: int,
|
||||||
|
semaphore: asyncio.Semaphore,
|
||||||
|
) -> tuple[int, str, object]:
|
||||||
|
"""对单行数据执行一条 enrichment rule 的 API 请求"""
|
||||||
|
target_key = rule["target_key"]
|
||||||
|
fallback = rule.get("fallback_value")
|
||||||
|
|
||||||
|
url = render_template(rule["url_template"], row)
|
||||||
|
|
||||||
|
headers = {}
|
||||||
|
if rule.get("headers"):
|
||||||
|
for k, v in rule["headers"].items():
|
||||||
|
headers[k] = render_template(v, row)
|
||||||
|
|
||||||
|
method = rule.get("method", "GET").upper()
|
||||||
|
|
||||||
|
body = None
|
||||||
|
if method == "POST" and rule.get("body_template"):
|
||||||
|
body = render_template(rule["body_template"], row)
|
||||||
|
|
||||||
|
async with semaphore:
|
||||||
|
try:
|
||||||
|
kwargs: dict = {"headers": headers}
|
||||||
|
if body is not None:
|
||||||
|
try:
|
||||||
|
kwargs["json"] = json.loads(body)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
kwargs["data"] = body
|
||||||
|
|
||||||
|
async with session.request(method, url, **kwargs) as resp:
|
||||||
|
if resp.status >= 400:
|
||||||
|
print(f" [WARN] Row {row_index} | {target_key} | HTTP {resp.status} <- {url}")
|
||||||
|
return row_index, target_key, fallback
|
||||||
|
|
||||||
|
data = await resp.json(content_type=None)
|
||||||
|
value = resolve_path(data, rule["response_path"])
|
||||||
|
if value is None:
|
||||||
|
value = fallback
|
||||||
|
return row_index, target_key, value
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print(f" [ERROR] Row {row_index} | {target_key} | {type(e).__name__}: {e}")
|
||||||
|
return row_index, target_key, fallback
|
||||||
|
|
||||||
|
|
||||||
|
async def run_enrichments(
|
||||||
|
source_data: list[dict],
|
||||||
|
rules: list[dict],
|
||||||
|
concurrency: int,
|
||||||
|
) -> list[dict]:
|
||||||
|
"""对所有行执行所有 enrichment rules"""
|
||||||
|
if not rules:
|
||||||
|
print("No enrichment rules configured. Outputting source data as-is.")
|
||||||
|
return source_data
|
||||||
|
|
||||||
|
semaphore = asyncio.Semaphore(concurrency)
|
||||||
|
tasks = []
|
||||||
|
|
||||||
|
total_calls = len(source_data) * len(rules)
|
||||||
|
print(f"Enriching {len(source_data)} rows x {len(rules)} rule(s) = {total_calls} API calls")
|
||||||
|
print(f"Concurrency: {concurrency}")
|
||||||
|
print()
|
||||||
|
|
||||||
|
async with aiohttp.ClientSession() as session:
|
||||||
|
for row_idx, row in enumerate(source_data):
|
||||||
|
for rule in rules:
|
||||||
|
tasks.append(fetch_one(session, rule, row, row_idx, semaphore))
|
||||||
|
|
||||||
|
results = await asyncio.gather(*tasks, return_exceptions=True)
|
||||||
|
|
||||||
|
error_count = 0
|
||||||
|
for result in results:
|
||||||
|
if isinstance(result, Exception):
|
||||||
|
print(f" [ERROR] Unexpected: {result}")
|
||||||
|
error_count += 1
|
||||||
|
continue
|
||||||
|
row_idx, target_key, value = result
|
||||||
|
source_data[row_idx][target_key] = value
|
||||||
|
|
||||||
|
print()
|
||||||
|
print(f"Done. {total_calls - error_count}/{total_calls} calls succeeded.")
|
||||||
|
return source_data
|
||||||
|
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
parser = argparse.ArgumentParser(description="Enrich job_bundle.json with API data and output final JSON.")
|
||||||
|
parser.add_argument("bundle", help="Path to job_bundle.json")
|
||||||
|
parser.add_argument("-o", "--output", type=str, default=None, help="Output file path (default: <bundle>_enriched.json)")
|
||||||
|
parser.add_argument("--concurrency", type=int, default=5, help="Max concurrent API requests (default: 5)")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
bundle_path = Path(args.bundle)
|
||||||
|
if not bundle_path.exists():
|
||||||
|
print(f"Error: File not found: {bundle_path}")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
with open(bundle_path, "r", encoding="utf-8") as f:
|
||||||
|
bundle = json.load(f)
|
||||||
|
|
||||||
|
meta = bundle.get("meta", {})
|
||||||
|
config = bundle.get("config", {})
|
||||||
|
source_data = bundle.get("source_data", [])
|
||||||
|
|
||||||
|
print(f"=== Job Bundle v{meta.get('version', '?')} ===")
|
||||||
|
print(f"Generated: {meta.get('generated_at', '?')}")
|
||||||
|
print(f"Rows: {len(source_data)}")
|
||||||
|
print(f"Static rules: {len(config.get('static_rules', []))}")
|
||||||
|
print(f"Enrichment rules: {len(config.get('enrichment_rules', []))}")
|
||||||
|
print()
|
||||||
|
|
||||||
|
enriched = await run_enrichments(
|
||||||
|
source_data,
|
||||||
|
config.get("enrichment_rules", []),
|
||||||
|
args.concurrency,
|
||||||
|
)
|
||||||
|
|
||||||
|
# 构建输出,保留 submission 配置
|
||||||
|
output = {
|
||||||
|
"meta": meta,
|
||||||
|
"submission": config.get("submission", {}),
|
||||||
|
"data": enriched,
|
||||||
|
}
|
||||||
|
|
||||||
|
output_path = Path(args.output) if args.output else bundle_path.with_name(bundle_path.stem + "_enriched.json")
|
||||||
|
with open(output_path, "w", encoding="utf-8") as f:
|
||||||
|
json.dump(output, f, ensure_ascii=False, indent=2)
|
||||||
|
|
||||||
|
print(f"\nEnriched data saved to: {output_path}")
|
||||||
|
print(f"Next step: python submitter.py {output_path}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
asyncio.run(main())
|
||||||
161
scripts/py/submitter.py
Normal file
161
scripts/py/submitter.py
Normal file
@@ -0,0 +1,161 @@
|
|||||||
|
"""
|
||||||
|
submitter.py — 数据提交脚本
|
||||||
|
|
||||||
|
读取 enricher.py 输出的 JSON 文件,按批次提交到目标接口。
|
||||||
|
记录提交成功和失败的记录到单独的日志文件。
|
||||||
|
|
||||||
|
用法:
|
||||||
|
python submitter.py enriched_data.json
|
||||||
|
python submitter.py enriched_data.json --batch-size 100
|
||||||
|
python submitter.py enriched_data.json --url https://api.example.com/import # 覆盖 URL
|
||||||
|
python submitter.py enriched_data.json --dry-run # 只输出不提交
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
from datetime import datetime
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import aiohttp
|
||||||
|
|
||||||
|
|
||||||
|
async def submit_batch(
|
||||||
|
session: aiohttp.ClientSession,
|
||||||
|
url: str,
|
||||||
|
method: str,
|
||||||
|
batch: list[dict],
|
||||||
|
batch_index: int,
|
||||||
|
total_batches: int,
|
||||||
|
) -> tuple[bool, int, str, list[dict]]:
|
||||||
|
"""提交一个批次,返回 (success, status_code, response_text, batch_data)"""
|
||||||
|
try:
|
||||||
|
async with session.request(method, url, json=batch) as resp:
|
||||||
|
status = resp.status
|
||||||
|
body = await resp.text()
|
||||||
|
ok = status < 400
|
||||||
|
tag = "OK" if ok else "FAIL"
|
||||||
|
print(f" Batch {batch_index}/{total_batches}: HTTP {status} {tag} ({len(batch)} records)")
|
||||||
|
return ok, status, body, batch
|
||||||
|
except Exception as e:
|
||||||
|
msg = f"{type(e).__name__}: {e}"
|
||||||
|
print(f" Batch {batch_index}/{total_batches}: ERROR - {msg} ({len(batch)} records)")
|
||||||
|
return False, 0, msg, batch
|
||||||
|
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
parser = argparse.ArgumentParser(description="Submit enriched data to target API in batches.")
|
||||||
|
parser.add_argument("input", help="Path to enriched JSON file (from enricher.py)")
|
||||||
|
parser.add_argument("--url", type=str, default=None, help="Override target URL from config")
|
||||||
|
parser.add_argument("--method", type=str, default=None, choices=["POST", "PUT"], help="Override HTTP method")
|
||||||
|
parser.add_argument("--batch-size", type=int, default=None, help="Override batch size")
|
||||||
|
parser.add_argument("--dry-run", action="store_true", help="Print what would be submitted without sending")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
input_path = Path(args.input)
|
||||||
|
if not input_path.exists():
|
||||||
|
print(f"Error: File not found: {input_path}")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
with open(input_path, "r", encoding="utf-8") as f:
|
||||||
|
payload = json.load(f)
|
||||||
|
|
||||||
|
data = payload.get("data", [])
|
||||||
|
submission = payload.get("submission", {})
|
||||||
|
|
||||||
|
target_url = args.url or submission.get("target_url", "")
|
||||||
|
method = args.method or submission.get("method", "POST")
|
||||||
|
batch_size = args.batch_size or submission.get("batch_size", 50)
|
||||||
|
|
||||||
|
if not target_url:
|
||||||
|
print("Error: No target URL configured. Use --url or set in job bundle.")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
print(f"=== Submitter ===")
|
||||||
|
print(f"Input: {input_path} ({len(data)} records)")
|
||||||
|
print(f"Target: {method} {target_url}")
|
||||||
|
print(f"Batch size: {batch_size}")
|
||||||
|
print()
|
||||||
|
|
||||||
|
batches = [data[i : i + batch_size] for i in range(0, len(data), batch_size)]
|
||||||
|
total_batches = len(batches)
|
||||||
|
|
||||||
|
if args.dry_run:
|
||||||
|
print(f"[DRY RUN] Would submit {len(data)} records in {total_batches} batch(es)")
|
||||||
|
for i, batch in enumerate(batches, 1):
|
||||||
|
print(f" Batch {i}/{total_batches}: {len(batch)} records")
|
||||||
|
print("\nDry run complete. No data was sent.")
|
||||||
|
return
|
||||||
|
|
||||||
|
success_records: list[dict] = []
|
||||||
|
failed_records: list[dict] = []
|
||||||
|
failed_details: list[dict] = []
|
||||||
|
|
||||||
|
print(f"Submitting {len(data)} records in {total_batches} batch(es)...\n")
|
||||||
|
|
||||||
|
async with aiohttp.ClientSession() as session:
|
||||||
|
for i, batch in enumerate(batches, 1):
|
||||||
|
ok, status, body, batch_data = await submit_batch(
|
||||||
|
session, target_url, method, batch, i, total_batches
|
||||||
|
)
|
||||||
|
if ok:
|
||||||
|
success_records.extend(batch_data)
|
||||||
|
else:
|
||||||
|
failed_records.extend(batch_data)
|
||||||
|
failed_details.append({
|
||||||
|
"batch_index": i,
|
||||||
|
"status": status,
|
||||||
|
"response": body[:500],
|
||||||
|
"record_count": len(batch_data),
|
||||||
|
})
|
||||||
|
|
||||||
|
# 结果统计
|
||||||
|
print()
|
||||||
|
print(f"=== Result ===")
|
||||||
|
print(f"Success: {len(success_records)} records")
|
||||||
|
print(f"Failed: {len(failed_records)} records")
|
||||||
|
|
||||||
|
# 写入日志文件
|
||||||
|
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||||
|
base_name = input_path.stem
|
||||||
|
|
||||||
|
if success_records:
|
||||||
|
success_file = input_path.with_name(f"{base_name}_success_{timestamp}.json")
|
||||||
|
with open(success_file, "w", encoding="utf-8") as f:
|
||||||
|
json.dump(success_records, f, ensure_ascii=False, indent=2)
|
||||||
|
print(f"\nSuccess log: {success_file}")
|
||||||
|
|
||||||
|
if failed_records:
|
||||||
|
failed_file = input_path.with_name(f"{base_name}_failed_{timestamp}.json")
|
||||||
|
report = {
|
||||||
|
"summary": {
|
||||||
|
"total_failed": len(failed_records),
|
||||||
|
"failed_batches": len(failed_details),
|
||||||
|
"target_url": target_url,
|
||||||
|
"timestamp": timestamp,
|
||||||
|
},
|
||||||
|
"batch_errors": failed_details,
|
||||||
|
"failed_records": failed_records,
|
||||||
|
}
|
||||||
|
with open(failed_file, "w", encoding="utf-8") as f:
|
||||||
|
json.dump(report, f, ensure_ascii=False, indent=2)
|
||||||
|
print(f"Failed log: {failed_file}")
|
||||||
|
print(f"\nTo retry failed records:")
|
||||||
|
print(f' python submitter.py {failed_file} --url "{target_url}"')
|
||||||
|
# 生成一个可直接重试的文件
|
||||||
|
retry_file = input_path.with_name(f"{base_name}_retry_{timestamp}.json")
|
||||||
|
retry_payload = {
|
||||||
|
"submission": submission,
|
||||||
|
"data": failed_records,
|
||||||
|
}
|
||||||
|
with open(retry_file, "w", encoding="utf-8") as f:
|
||||||
|
json.dump(retry_payload, f, ensure_ascii=False, indent=2)
|
||||||
|
print(f" or: python submitter.py {retry_file}")
|
||||||
|
|
||||||
|
if not success_records and not failed_records:
|
||||||
|
print("\nNo records to submit.")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
asyncio.run(main())
|
||||||
298
src/lib/components/ApiConfigModal.svelte
Normal file
298
src/lib/components/ApiConfigModal.svelte
Normal file
@@ -0,0 +1,298 @@
|
|||||||
|
<script lang="ts">
|
||||||
|
import type { ApiEnrichmentRule } from '$lib/types.js';
|
||||||
|
|
||||||
|
let {
|
||||||
|
rule,
|
||||||
|
headers,
|
||||||
|
onsave,
|
||||||
|
onclose
|
||||||
|
}: {
|
||||||
|
rule?: ApiEnrichmentRule;
|
||||||
|
headers: string[];
|
||||||
|
onsave: (rule: ApiEnrichmentRule) => void;
|
||||||
|
onclose: () => void;
|
||||||
|
} = $props();
|
||||||
|
|
||||||
|
// svelte-ignore state_referenced_locally — intentionally using initial values only
|
||||||
|
let targetKey = $state(rule?.target_key ?? '');
|
||||||
|
// svelte-ignore state_referenced_locally
|
||||||
|
let urlTemplate = $state(rule?.url_template ?? '');
|
||||||
|
// svelte-ignore state_referenced_locally
|
||||||
|
let method = $state<'GET' | 'POST'>(rule?.method ?? 'GET');
|
||||||
|
// svelte-ignore state_referenced_locally
|
||||||
|
let headerEntries = $state<{ key: string; value: string }[]>(
|
||||||
|
rule?.headers ? Object.entries(rule.headers).map(([key, value]) => ({ key, value })) : []
|
||||||
|
);
|
||||||
|
// svelte-ignore state_referenced_locally
|
||||||
|
let bodyTemplate = $state(rule?.body_template ?? '');
|
||||||
|
// svelte-ignore state_referenced_locally
|
||||||
|
let responsePath = $state(rule?.response_path ?? '');
|
||||||
|
// svelte-ignore state_referenced_locally
|
||||||
|
let fallbackValue = $state(rule?.fallback_value != null ? String(rule.fallback_value) : '');
|
||||||
|
|
||||||
|
let showUrlVars = $state(false);
|
||||||
|
let showBodyVars = $state(false);
|
||||||
|
|
||||||
|
let urlInput: HTMLInputElement | undefined = $state();
|
||||||
|
let bodyTextarea: HTMLTextAreaElement | undefined = $state();
|
||||||
|
|
||||||
|
function insertVariable(field: 'url' | 'body', header: string) {
|
||||||
|
const variable = `{{${header}}}`;
|
||||||
|
if (field === 'url') {
|
||||||
|
if (urlInput) {
|
||||||
|
const start = urlInput.selectionStart ?? urlTemplate.length;
|
||||||
|
urlTemplate = urlTemplate.slice(0, start) + variable + urlTemplate.slice(urlInput.selectionEnd ?? start);
|
||||||
|
} else {
|
||||||
|
urlTemplate += variable;
|
||||||
|
}
|
||||||
|
showUrlVars = false;
|
||||||
|
} else {
|
||||||
|
if (bodyTextarea) {
|
||||||
|
const start = bodyTextarea.selectionStart ?? bodyTemplate.length;
|
||||||
|
bodyTemplate = bodyTemplate.slice(0, start) + variable + bodyTemplate.slice(bodyTextarea.selectionEnd ?? start);
|
||||||
|
} else {
|
||||||
|
bodyTemplate += variable;
|
||||||
|
}
|
||||||
|
showBodyVars = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function addHeader() {
|
||||||
|
headerEntries = [...headerEntries, { key: '', value: '' }];
|
||||||
|
}
|
||||||
|
|
||||||
|
function removeHeader(index: number) {
|
||||||
|
headerEntries = headerEntries.filter((_, i) => i !== index);
|
||||||
|
}
|
||||||
|
|
||||||
|
function save() {
|
||||||
|
if (!targetKey.trim() || !urlTemplate.trim() || !responsePath.trim()) return;
|
||||||
|
|
||||||
|
const headersObj: Record<string, string> = {};
|
||||||
|
for (const entry of headerEntries) {
|
||||||
|
if (entry.key.trim()) {
|
||||||
|
headersObj[entry.key.trim()] = entry.value;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const newRule: ApiEnrichmentRule = {
|
||||||
|
type: 'api_fetch',
|
||||||
|
target_key: targetKey.trim(),
|
||||||
|
url_template: urlTemplate.trim(),
|
||||||
|
method,
|
||||||
|
response_path: responsePath.trim()
|
||||||
|
};
|
||||||
|
|
||||||
|
if (Object.keys(headersObj).length > 0) newRule.headers = headersObj;
|
||||||
|
if (method === 'POST' && bodyTemplate.trim()) newRule.body_template = bodyTemplate.trim();
|
||||||
|
if (fallbackValue !== '') newRule.fallback_value = fallbackValue;
|
||||||
|
|
||||||
|
onsave(newRule);
|
||||||
|
}
|
||||||
|
|
||||||
|
const isValid = $derived(targetKey.trim() !== '' && urlTemplate.trim() !== '' && responsePath.trim() !== '');
|
||||||
|
</script>
|
||||||
|
|
||||||
|
<!-- svelte-ignore a11y_no_static_element_interactions a11y_click_events_have_key_events -->
|
||||||
|
<!-- svelte-ignore a11y_click_events_have_key_events -->
|
||||||
|
<div class="fixed inset-0 z-50 flex items-center justify-center bg-black/40" onclick={onclose} role="presentation">
|
||||||
|
<!-- svelte-ignore a11y_no_static_element_interactions a11y_click_events_have_key_events -->
|
||||||
|
<!-- svelte-ignore a11y_click_events_have_key_events -->
|
||||||
|
<div class="max-h-[90vh] w-full max-w-lg overflow-y-auto rounded-lg bg-white p-6 shadow-xl" onclick={(e) => e.stopPropagation()}>
|
||||||
|
<div class="mb-4 flex items-center justify-between">
|
||||||
|
<h3 class="text-lg font-semibold text-gray-800">
|
||||||
|
{rule ? '编辑' : '添加'} API 字段
|
||||||
|
</h3>
|
||||||
|
<button onclick={onclose} aria-label="关闭" class="cursor-pointer text-gray-400 hover:text-gray-600">
|
||||||
|
<svg class="h-5 w-5" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||||
|
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M6 18L18 6M6 6l12 12" />
|
||||||
|
</svg>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="space-y-4">
|
||||||
|
<!-- Target Key -->
|
||||||
|
<div>
|
||||||
|
<label for="api-target-key" class="mb-1 block text-sm font-medium text-gray-700">Target Key</label>
|
||||||
|
<input
|
||||||
|
id="api-target-key"
|
||||||
|
type="text"
|
||||||
|
bind:value={targetKey}
|
||||||
|
placeholder="user_balance"
|
||||||
|
class="w-full rounded-md border border-gray-300 px-3 py-2 text-sm focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
|
||||||
|
/>
|
||||||
|
<p class="mt-1 text-xs text-gray-400">最终 JSON 中的字段名</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Request URL -->
|
||||||
|
<div>
|
||||||
|
<label for="api-url" class="mb-1 block text-sm font-medium text-gray-700">Request URL</label>
|
||||||
|
<div class="flex gap-1">
|
||||||
|
<input
|
||||||
|
id="api-url"
|
||||||
|
type="text"
|
||||||
|
bind:this={urlInput}
|
||||||
|
bind:value={urlTemplate}
|
||||||
|
placeholder={"https://api.example.com/users/{{用户ID}}/detail"}
|
||||||
|
class="flex-1 rounded-md border border-gray-300 px-3 py-2 text-sm focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
|
||||||
|
/>
|
||||||
|
<div class="relative">
|
||||||
|
<button
|
||||||
|
onclick={() => { showUrlVars = !showUrlVars; showBodyVars = false; }}
|
||||||
|
class="cursor-pointer whitespace-nowrap rounded-md border border-gray-300 px-2 py-2 text-xs text-gray-600 hover:bg-gray-50"
|
||||||
|
title="插入变量"
|
||||||
|
>
|
||||||
|
{{x}}
|
||||||
|
</button>
|
||||||
|
{#if showUrlVars}
|
||||||
|
<div class="absolute right-0 z-10 mt-1 max-h-48 w-40 overflow-y-auto rounded-md border border-gray-200 bg-white py-1 shadow-lg">
|
||||||
|
{#each headers as h (h)}
|
||||||
|
<button
|
||||||
|
onclick={() => insertVariable('url', h)}
|
||||||
|
class="w-full cursor-pointer px-3 py-1.5 text-left text-xs text-gray-700 hover:bg-blue-50"
|
||||||
|
>
|
||||||
|
{h}
|
||||||
|
</button>
|
||||||
|
{/each}
|
||||||
|
</div>
|
||||||
|
{/if}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<p class="mt-1 text-xs text-gray-400">支持 {{列名}} 模板变量</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Request Method -->
|
||||||
|
<div>
|
||||||
|
<label for="api-method" class="mb-1 block text-sm font-medium text-gray-700">Request Method</label>
|
||||||
|
<select
|
||||||
|
id="api-method"
|
||||||
|
bind:value={method}
|
||||||
|
class="w-full rounded-md border border-gray-300 px-3 py-2 text-sm focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
|
||||||
|
>
|
||||||
|
<option value="GET">GET</option>
|
||||||
|
<option value="POST">POST</option>
|
||||||
|
</select>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Headers -->
|
||||||
|
<div>
|
||||||
|
<div class="mb-1 flex items-center justify-between">
|
||||||
|
<span class="text-sm font-medium text-gray-700">Headers</span>
|
||||||
|
<button onclick={addHeader} class="cursor-pointer text-xs text-blue-600 hover:text-blue-700">
|
||||||
|
+ 添加
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
{#if headerEntries.length > 0}
|
||||||
|
<div class="space-y-2">
|
||||||
|
{#each headerEntries as entry, i (i)}
|
||||||
|
<div class="flex items-center gap-1">
|
||||||
|
<input
|
||||||
|
type="text"
|
||||||
|
bind:value={entry.key}
|
||||||
|
placeholder="Key"
|
||||||
|
class="flex-1 rounded border border-gray-300 px-2 py-1.5 text-xs focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
|
||||||
|
/>
|
||||||
|
<input
|
||||||
|
type="text"
|
||||||
|
bind:value={entry.value}
|
||||||
|
placeholder="Value"
|
||||||
|
class="flex-1 rounded border border-gray-300 px-2 py-1.5 text-xs focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
|
||||||
|
/>
|
||||||
|
<button
|
||||||
|
onclick={() => removeHeader(i)}
|
||||||
|
class="cursor-pointer flex-shrink-0 text-gray-400 hover:text-red-500"
|
||||||
|
aria-label="删除此 Header"
|
||||||
|
>
|
||||||
|
<svg class="h-4 w-4" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||||
|
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M6 18L18 6M6 6l12 12" />
|
||||||
|
</svg>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
{/each}
|
||||||
|
</div>
|
||||||
|
{:else}
|
||||||
|
<p class="text-xs text-gray-400">暂无 Header</p>
|
||||||
|
{/if}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Body (POST only) -->
|
||||||
|
{#if method === 'POST'}
|
||||||
|
<div>
|
||||||
|
<div class="mb-1 flex items-center justify-between">
|
||||||
|
<label for="api-body" class="text-sm font-medium text-gray-700">Request Body</label>
|
||||||
|
<div class="relative">
|
||||||
|
<button
|
||||||
|
onclick={() => { showBodyVars = !showBodyVars; showUrlVars = false; }}
|
||||||
|
class="cursor-pointer text-xs text-blue-600 hover:text-blue-700"
|
||||||
|
>
|
||||||
|
插入变量
|
||||||
|
</button>
|
||||||
|
{#if showBodyVars}
|
||||||
|
<div class="absolute right-0 z-10 mt-1 max-h-48 w-40 overflow-y-auto rounded-md border border-gray-200 bg-white py-1 shadow-lg">
|
||||||
|
{#each headers as h (h)}
|
||||||
|
<button
|
||||||
|
onclick={() => insertVariable('body', h)}
|
||||||
|
class="w-full cursor-pointer px-3 py-1.5 text-left text-xs text-gray-700 hover:bg-blue-50"
|
||||||
|
>
|
||||||
|
{h}
|
||||||
|
</button>
|
||||||
|
{/each}
|
||||||
|
</div>
|
||||||
|
{/if}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<textarea
|
||||||
|
id="api-body"
|
||||||
|
bind:this={bodyTextarea}
|
||||||
|
bind:value={bodyTemplate}
|
||||||
|
rows="4"
|
||||||
|
placeholder={'{"user_id": "{{用户ID}}"}'}
|
||||||
|
class="w-full rounded-md border border-gray-300 px-3 py-2 font-mono text-xs focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
|
||||||
|
></textarea>
|
||||||
|
</div>
|
||||||
|
{/if}
|
||||||
|
|
||||||
|
<!-- Response Path -->
|
||||||
|
<div>
|
||||||
|
<label for="api-response-path" class="mb-1 block text-sm font-medium text-gray-700">Response Extractor</label>
|
||||||
|
<input
|
||||||
|
id="api-response-path"
|
||||||
|
type="text"
|
||||||
|
bind:value={responsePath}
|
||||||
|
placeholder="data.balance"
|
||||||
|
class="w-full rounded-md border border-gray-300 px-3 py-2 text-sm focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
|
||||||
|
/>
|
||||||
|
<p class="mt-1 text-xs text-gray-400">从接口返回 JSON 中提取值的路径(如 data.result.value)</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Fallback Value -->
|
||||||
|
<div>
|
||||||
|
<label for="api-fallback" class="mb-1 block text-sm font-medium text-gray-700">Fallback Value</label>
|
||||||
|
<input
|
||||||
|
id="api-fallback"
|
||||||
|
type="text"
|
||||||
|
bind:value={fallbackValue}
|
||||||
|
placeholder="null"
|
||||||
|
class="w-full rounded-md border border-gray-300 px-3 py-2 text-sm focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
|
||||||
|
/>
|
||||||
|
<p class="mt-1 text-xs text-gray-400">API 请求失败时的默认值</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="mt-6 flex justify-end gap-2">
|
||||||
|
<button
|
||||||
|
onclick={onclose}
|
||||||
|
class="cursor-pointer rounded-md border border-gray-300 px-4 py-2 text-sm text-gray-700 hover:bg-gray-50"
|
||||||
|
>
|
||||||
|
取消
|
||||||
|
</button>
|
||||||
|
<button
|
||||||
|
onclick={save}
|
||||||
|
disabled={!isValid}
|
||||||
|
class="cursor-pointer rounded-md bg-blue-600 px-4 py-2 text-sm font-medium text-white hover:bg-blue-700 disabled:cursor-not-allowed disabled:opacity-50"
|
||||||
|
>
|
||||||
|
保存
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
@@ -1,15 +1,23 @@
|
|||||||
<script lang="ts">
|
<script lang="ts">
|
||||||
import type { MappingConfig, RowData } from '$lib/types.js';
|
import type { MappingConfig, RowData, ApiEnrichmentRule } from '$lib/types.js';
|
||||||
import ColumnConfig from './ColumnConfig.svelte';
|
import ColumnConfig from './ColumnConfig.svelte';
|
||||||
|
|
||||||
let {
|
let {
|
||||||
headers,
|
headers,
|
||||||
rows,
|
rows,
|
||||||
mappings = $bindable()
|
mappings = $bindable(),
|
||||||
|
enrichmentRules = [],
|
||||||
|
onaddapi,
|
||||||
|
oneditapi,
|
||||||
|
ondeleteapi
|
||||||
}: {
|
}: {
|
||||||
headers: string[];
|
headers: string[];
|
||||||
rows: RowData[];
|
rows: RowData[];
|
||||||
mappings: MappingConfig[];
|
mappings: MappingConfig[];
|
||||||
|
enrichmentRules?: ApiEnrichmentRule[];
|
||||||
|
onaddapi?: () => void;
|
||||||
|
oneditapi?: (index: number) => void;
|
||||||
|
ondeleteapi?: (index: number) => void;
|
||||||
} = $props();
|
} = $props();
|
||||||
|
|
||||||
let activeConfigIndex = $state<number | null>(null);
|
let activeConfigIndex = $state<number | null>(null);
|
||||||
@@ -37,16 +45,67 @@
|
|||||||
|
|
||||||
<div class="flex h-full flex-col overflow-hidden">
|
<div class="flex h-full flex-col overflow-hidden">
|
||||||
<div class="flex-shrink-0 border-b border-gray-200 bg-gray-50 px-4 py-2">
|
<div class="flex-shrink-0 border-b border-gray-200 bg-gray-50 px-4 py-2">
|
||||||
|
<div class="flex items-center justify-between">
|
||||||
<h3 class="text-sm font-semibold text-gray-700">
|
<h3 class="text-sm font-semibold text-gray-700">
|
||||||
Excel 数据
|
Excel 数据
|
||||||
<span class="ml-2 text-xs font-normal text-gray-400">
|
<span class="ml-2 text-xs font-normal text-gray-400">
|
||||||
{rows.length} 行 × {headers.length} 列
|
{rows.length} 行 × {headers.length} 列
|
||||||
|
{#if enrichmentRules.length > 0}
|
||||||
|
+ {enrichmentRules.length} API 字段
|
||||||
|
{/if}
|
||||||
{#if rows.length > maxPreviewRows}
|
{#if rows.length > maxPreviewRows}
|
||||||
(显示前 {maxPreviewRows} 行)
|
(显示前 {maxPreviewRows} 行)
|
||||||
{/if}
|
{/if}
|
||||||
</span>
|
</span>
|
||||||
</h3>
|
</h3>
|
||||||
|
{#if onaddapi}
|
||||||
|
<button
|
||||||
|
onclick={onaddapi}
|
||||||
|
class="inline-flex cursor-pointer items-center gap-1 rounded-md bg-purple-50 px-2 py-1 text-xs font-medium text-purple-700 hover:bg-purple-100"
|
||||||
|
>
|
||||||
|
<svg class="h-3.5 w-3.5" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||||
|
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M12 4v16m8-8H4" />
|
||||||
|
</svg>
|
||||||
|
添加 API 字段
|
||||||
|
</button>
|
||||||
|
{/if}
|
||||||
</div>
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Enrichment rules bar -->
|
||||||
|
{#if enrichmentRules.length > 0}
|
||||||
|
<div class="flex flex-shrink-0 flex-wrap items-center gap-2 border-b border-gray-200 bg-purple-50/50 px-4 py-2">
|
||||||
|
<span class="text-xs font-medium text-purple-600">API 字段:</span>
|
||||||
|
{#each enrichmentRules as rule, i (i)}
|
||||||
|
<span class="inline-flex items-center gap-1 rounded-full bg-purple-100 px-2.5 py-0.5 text-xs font-medium text-purple-700">
|
||||||
|
{rule.target_key}
|
||||||
|
<span class="text-purple-400">({rule.method})</span>
|
||||||
|
{#if oneditapi}
|
||||||
|
<button
|
||||||
|
onclick={() => oneditapi?.(i)}
|
||||||
|
class="cursor-pointer text-purple-400 hover:text-purple-600"
|
||||||
|
title="编辑"
|
||||||
|
>
|
||||||
|
<svg class="h-3 w-3" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||||
|
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M11 5H6a2 2 0 00-2 2v11a2 2 0 002 2h11a2 2 0 002-2v-5m-1.414-9.414a2 2 0 112.828 2.828L11.828 15H9v-2.828l8.586-8.586z" />
|
||||||
|
</svg>
|
||||||
|
</button>
|
||||||
|
{/if}
|
||||||
|
{#if ondeleteapi}
|
||||||
|
<button
|
||||||
|
onclick={() => ondeleteapi?.(i)}
|
||||||
|
class="cursor-pointer text-purple-400 hover:text-red-500"
|
||||||
|
title="删除"
|
||||||
|
>
|
||||||
|
<svg class="h-3 w-3" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||||
|
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M6 18L18 6M6 6l12 12" />
|
||||||
|
</svg>
|
||||||
|
</button>
|
||||||
|
{/if}
|
||||||
|
</span>
|
||||||
|
{/each}
|
||||||
|
</div>
|
||||||
|
{/if}
|
||||||
|
|
||||||
<div class="flex-1 overflow-auto">
|
<div class="flex-1 overflow-auto">
|
||||||
<table class="w-full text-sm">
|
<table class="w-full text-sm">
|
||||||
@@ -76,6 +135,17 @@
|
|||||||
{/if}
|
{/if}
|
||||||
</th>
|
</th>
|
||||||
{/each}
|
{/each}
|
||||||
|
<!-- API enrichment columns (virtual) -->
|
||||||
|
{#each enrichmentRules as rule (rule.target_key)}
|
||||||
|
<th class="border-b border-r border-purple-200 bg-purple-50 px-3 py-2 text-left font-medium text-purple-600">
|
||||||
|
<div class="flex items-center gap-1">
|
||||||
|
<svg class="h-3 w-3 flex-shrink-0" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||||
|
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M13 10V3L4 14h7v7l9-11h-7z" />
|
||||||
|
</svg>
|
||||||
|
<span class="truncate" title="{rule.target_key} (API)">{rule.target_key}</span>
|
||||||
|
</div>
|
||||||
|
</th>
|
||||||
|
{/each}
|
||||||
</tr>
|
</tr>
|
||||||
</thead>
|
</thead>
|
||||||
<tbody>
|
<tbody>
|
||||||
@@ -86,6 +156,12 @@
|
|||||||
<span class="block max-w-[200px] truncate">{displayValue(row[header])}</span>
|
<span class="block max-w-[200px] truncate">{displayValue(row[header])}</span>
|
||||||
</td>
|
</td>
|
||||||
{/each}
|
{/each}
|
||||||
|
<!-- API placeholder cells -->
|
||||||
|
{#each enrichmentRules as _ (_.target_key)}
|
||||||
|
<td class="border-b border-r border-purple-100 bg-purple-50/30 px-3 py-1.5 text-xs italic text-purple-400">
|
||||||
|
[Pending API Fetch]
|
||||||
|
</td>
|
||||||
|
{/each}
|
||||||
</tr>
|
</tr>
|
||||||
{/each}
|
{/each}
|
||||||
</tbody>
|
</tbody>
|
||||||
|
|||||||
@@ -13,6 +13,7 @@
|
|||||||
defaultOpen?: boolean;
|
defaultOpen?: boolean;
|
||||||
} = $props();
|
} = $props();
|
||||||
|
|
||||||
|
// svelte-ignore state_referenced_locally — intentionally using initial value only
|
||||||
let open = $state(defaultOpen);
|
let open = $state(defaultOpen);
|
||||||
|
|
||||||
const valueType = $derived(
|
const valueType = $derived(
|
||||||
|
|||||||
88
src/lib/components/SubmissionSettings.svelte
Normal file
88
src/lib/components/SubmissionSettings.svelte
Normal file
@@ -0,0 +1,88 @@
|
|||||||
|
<script lang="ts">
|
||||||
|
import type { SubmissionConfig } from '$lib/types.js';
|
||||||
|
|
||||||
|
let {
|
||||||
|
config = $bindable(),
|
||||||
|
onclose
|
||||||
|
}: {
|
||||||
|
config: SubmissionConfig;
|
||||||
|
onclose: () => void;
|
||||||
|
} = $props();
|
||||||
|
|
||||||
|
let localConfig = $state<SubmissionConfig>({ ...config });
|
||||||
|
|
||||||
|
function save() {
|
||||||
|
config.target_url = localConfig.target_url;
|
||||||
|
config.method = localConfig.method;
|
||||||
|
config.batch_size = localConfig.batch_size;
|
||||||
|
onclose();
|
||||||
|
}
|
||||||
|
</script>
|
||||||
|
|
||||||
|
<!-- svelte-ignore a11y_no_static_element_interactions a11y_click_events_have_key_events -->
|
||||||
|
<!-- svelte-ignore a11y_click_events_have_key_events -->
|
||||||
|
<div class="fixed inset-0 z-50 flex items-center justify-center bg-black/40" onclick={onclose} role="presentation">
|
||||||
|
<!-- svelte-ignore a11y_no_static_element_interactions a11y_click_events_have_key_events -->
|
||||||
|
<!-- svelte-ignore a11y_click_events_have_key_events -->
|
||||||
|
<div class="w-full max-w-md rounded-lg bg-white p-6 shadow-xl" onclick={(e) => e.stopPropagation()}>
|
||||||
|
<div class="mb-4 flex items-center justify-between">
|
||||||
|
<h3 class="text-lg font-semibold text-gray-800">提交设置</h3>
|
||||||
|
<button onclick={onclose} aria-label="关闭" class="cursor-pointer text-gray-400 hover:text-gray-600">
|
||||||
|
<svg class="h-5 w-5" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||||
|
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M6 18L18 6M6 6l12 12" />
|
||||||
|
</svg>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="space-y-4">
|
||||||
|
<div>
|
||||||
|
<label for="target-url" class="mb-1 block text-sm font-medium text-gray-700">Target URL</label>
|
||||||
|
<input
|
||||||
|
id="target-url"
|
||||||
|
type="text"
|
||||||
|
bind:value={localConfig.target_url}
|
||||||
|
placeholder="https://api.db.com/bulk-insert"
|
||||||
|
class="w-full rounded-md border border-gray-300 px-3 py-2 text-sm focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div>
|
||||||
|
<label for="method" class="mb-1 block text-sm font-medium text-gray-700">Method</label>
|
||||||
|
<select
|
||||||
|
id="method"
|
||||||
|
bind:value={localConfig.method}
|
||||||
|
class="w-full rounded-md border border-gray-300 px-3 py-2 text-sm focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
|
||||||
|
>
|
||||||
|
<option value="POST">POST</option>
|
||||||
|
<option value="PUT">PUT</option>
|
||||||
|
</select>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div>
|
||||||
|
<label for="batch-size" class="mb-1 block text-sm font-medium text-gray-700">Batch Size</label>
|
||||||
|
<input
|
||||||
|
id="batch-size"
|
||||||
|
type="number"
|
||||||
|
bind:value={localConfig.batch_size}
|
||||||
|
min="1"
|
||||||
|
class="w-full rounded-md border border-gray-300 px-3 py-2 text-sm focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="mt-6 flex justify-end gap-2">
|
||||||
|
<button
|
||||||
|
onclick={onclose}
|
||||||
|
class="cursor-pointer rounded-md border border-gray-300 px-4 py-2 text-sm text-gray-700 hover:bg-gray-50"
|
||||||
|
>
|
||||||
|
取消
|
||||||
|
</button>
|
||||||
|
<button
|
||||||
|
onclick={save}
|
||||||
|
class="cursor-pointer rounded-md bg-blue-600 px-4 py-2 text-sm font-medium text-white hover:bg-blue-700"
|
||||||
|
>
|
||||||
|
保存
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
@@ -1,5 +1,14 @@
|
|||||||
import dayjs from 'dayjs';
|
import dayjs from 'dayjs';
|
||||||
import type { MappingConfig, RowData, DataType, MappingTemplate } from './types.js';
|
import type {
|
||||||
|
MappingConfig,
|
||||||
|
RowData,
|
||||||
|
DataType,
|
||||||
|
MappingTemplate,
|
||||||
|
ApiEnrichmentRule,
|
||||||
|
SubmissionConfig,
|
||||||
|
StaticRule,
|
||||||
|
JobBundle
|
||||||
|
} from './types.js';
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Check if a value is considered empty.
|
* Check if a value is considered empty.
|
||||||
@@ -195,3 +204,47 @@ export function exportTemplate(mappings: MappingConfig[]): MappingTemplate {
|
|||||||
return entry;
|
return entry;
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Convert enabled MappingConfigs to StaticRule format for Job Bundle.
|
||||||
|
*/
|
||||||
|
function toStaticRules(mappings: MappingConfig[]): StaticRule[] {
|
||||||
|
return mappings
|
||||||
|
.filter((m) => m.enabled)
|
||||||
|
.map((m) => {
|
||||||
|
const rule: StaticRule = {
|
||||||
|
type: 'static',
|
||||||
|
source: m.source,
|
||||||
|
target: m.target,
|
||||||
|
dataType: m.type
|
||||||
|
};
|
||||||
|
if (m.type === 'date' && m.format) rule.format = m.format;
|
||||||
|
return rule;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Generate a complete Job Bundle for export.
|
||||||
|
*/
|
||||||
|
export function generateJobBundle(
|
||||||
|
rows: RowData[],
|
||||||
|
mappings: MappingConfig[],
|
||||||
|
enrichmentRules: ApiEnrichmentRule[],
|
||||||
|
submissionConfig: SubmissionConfig
|
||||||
|
): JobBundle {
|
||||||
|
const sourceData = convertData(rows, mappings);
|
||||||
|
const staticRules = toStaticRules(mappings);
|
||||||
|
|
||||||
|
return {
|
||||||
|
meta: {
|
||||||
|
version: '1.0.0',
|
||||||
|
generated_at: new Date().toISOString()
|
||||||
|
},
|
||||||
|
config: {
|
||||||
|
static_rules: staticRules,
|
||||||
|
enrichment_rules: enrichmentRules,
|
||||||
|
submission: submissionConfig
|
||||||
|
},
|
||||||
|
source_data: sourceData
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|||||||
@@ -33,3 +33,45 @@ export type MappingTemplate = Pick<
|
|||||||
MappingConfig,
|
MappingConfig,
|
||||||
'source' | 'target' | 'type' | 'format' | 'excludeIfEmpty' | 'defaultValue'
|
'source' | 'target' | 'type' | 'format' | 'excludeIfEmpty' | 'defaultValue'
|
||||||
>[];
|
>[];
|
||||||
|
|
||||||
|
/** Static mapping rule in Job Bundle output */
|
||||||
|
export interface StaticRule {
|
||||||
|
type: 'static';
|
||||||
|
source: string;
|
||||||
|
target: string;
|
||||||
|
dataType: DataType;
|
||||||
|
format?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Dynamic API enrichment rule */
|
||||||
|
export interface ApiEnrichmentRule {
|
||||||
|
type: 'api_fetch';
|
||||||
|
target_key: string;
|
||||||
|
url_template: string;
|
||||||
|
method: 'GET' | 'POST';
|
||||||
|
headers?: Record<string, string>;
|
||||||
|
body_template?: string;
|
||||||
|
response_path: string;
|
||||||
|
fallback_value?: unknown;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Submission configuration for final data push */
|
||||||
|
export interface SubmissionConfig {
|
||||||
|
target_url: string;
|
||||||
|
method: 'POST' | 'PUT';
|
||||||
|
batch_size: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Final exported Job Bundle structure */
|
||||||
|
export interface JobBundle {
|
||||||
|
meta: {
|
||||||
|
version: string;
|
||||||
|
generated_at: string;
|
||||||
|
};
|
||||||
|
config: {
|
||||||
|
static_rules: StaticRule[];
|
||||||
|
enrichment_rules: ApiEnrichmentRule[];
|
||||||
|
submission: SubmissionConfig;
|
||||||
|
};
|
||||||
|
source_data: Record<string, unknown>[];
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,9 +1,11 @@
|
|||||||
<script lang="ts">
|
<script lang="ts">
|
||||||
import type { MappingConfig, RowData, MappingTemplate } from '$lib/types.js';
|
import type { MappingConfig, RowData, MappingTemplate, ApiEnrichmentRule, SubmissionConfig } from '$lib/types.js';
|
||||||
import { parseExcelFile } from '$lib/excel.js';
|
import { parseExcelFile } from '$lib/excel.js';
|
||||||
import { convertData, createDefaultMappings, applyTemplate, exportTemplate } from '$lib/converter.js';
|
import { convertData, createDefaultMappings, applyTemplate, exportTemplate, generateJobBundle } from '$lib/converter.js';
|
||||||
import ExcelTable from '$lib/components/ExcelTable.svelte';
|
import ExcelTable from '$lib/components/ExcelTable.svelte';
|
||||||
import JsonPreview from '$lib/components/JsonPreview.svelte';
|
import JsonPreview from '$lib/components/JsonPreview.svelte';
|
||||||
|
import ApiConfigModal from '$lib/components/ApiConfigModal.svelte';
|
||||||
|
import SubmissionSettings from '$lib/components/SubmissionSettings.svelte';
|
||||||
|
|
||||||
// State
|
// State
|
||||||
let headers = $state<string[]>([]);
|
let headers = $state<string[]>([]);
|
||||||
@@ -14,6 +16,17 @@
|
|||||||
let isDragOver = $state(false);
|
let isDragOver = $state(false);
|
||||||
let copySuccess = $state(false);
|
let copySuccess = $state(false);
|
||||||
|
|
||||||
|
// ETL state
|
||||||
|
let enrichmentRules = $state<ApiEnrichmentRule[]>([]);
|
||||||
|
let submissionConfig = $state<SubmissionConfig>({
|
||||||
|
target_url: '',
|
||||||
|
method: 'POST',
|
||||||
|
batch_size: 50
|
||||||
|
});
|
||||||
|
let showApiConfig = $state(false);
|
||||||
|
let showSubmissionSettings = $state(false);
|
||||||
|
let editingRuleIndex = $state<number | null>(null);
|
||||||
|
|
||||||
// Split panel
|
// Split panel
|
||||||
let splitPercent = $state(50);
|
let splitPercent = $state(50);
|
||||||
let jsonCollapsed = $state(false);
|
let jsonCollapsed = $state(false);
|
||||||
@@ -50,28 +63,41 @@
|
|||||||
let debounceTimer: ReturnType<typeof setTimeout>;
|
let debounceTimer: ReturnType<typeof setTimeout>;
|
||||||
|
|
||||||
$effect(() => {
|
$effect(() => {
|
||||||
// $state.snapshot forces deep read of all nested properties,
|
|
||||||
// so changes to e.g. mappings[i].target will trigger this effect
|
|
||||||
const snapshotMappings = $state.snapshot(mappings);
|
const snapshotMappings = $state.snapshot(mappings);
|
||||||
|
const snapshotRules = $state.snapshot(enrichmentRules);
|
||||||
const currentRows = rows;
|
const currentRows = rows;
|
||||||
const dataReady = hasData;
|
const dataReady = hasData;
|
||||||
|
|
||||||
clearTimeout(debounceTimer);
|
clearTimeout(debounceTimer);
|
||||||
debounceTimer = setTimeout(() => {
|
debounceTimer = setTimeout(() => {
|
||||||
convertedJson = dataReady ? convertData(currentRows, snapshotMappings) : [];
|
if (dataReady) {
|
||||||
|
const base = convertData(currentRows, snapshotMappings);
|
||||||
|
// Add API placeholders for preview
|
||||||
|
if (snapshotRules.length > 0) {
|
||||||
|
convertedJson = base.map((row) => {
|
||||||
|
const enriched = { ...row };
|
||||||
|
for (const rule of snapshotRules) {
|
||||||
|
enriched[rule.target_key] = '[Pending API Fetch]';
|
||||||
|
}
|
||||||
|
return enriched;
|
||||||
|
});
|
||||||
|
} else {
|
||||||
|
convertedJson = base;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
convertedJson = [];
|
||||||
|
}
|
||||||
}, 150);
|
}, 150);
|
||||||
});
|
});
|
||||||
|
|
||||||
// Manual JSON editing — overrides converted output until next mapping/data change
|
// Manual JSON editing
|
||||||
let manualJson = $state<string | null>(null);
|
let manualJson = $state<string | null>(null);
|
||||||
|
|
||||||
const jsonString = $derived(manualJson ?? JSON.stringify(convertedJson, null, 2));
|
const jsonString = $derived(manualJson ?? JSON.stringify(convertedJson, null, 2));
|
||||||
|
|
||||||
function onJsonEdited(edited: string) {
|
function onJsonEdited(edited: string) {
|
||||||
manualJson = edited;
|
manualJson = edited;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Clear manual override when conversion changes
|
|
||||||
$effect(() => {
|
$effect(() => {
|
||||||
convertedJson;
|
convertedJson;
|
||||||
manualJson = null;
|
manualJson = null;
|
||||||
@@ -179,6 +205,43 @@
|
|||||||
errorMessage = '复制失败,请手动复制';
|
errorMessage = '复制失败,请手动复制';
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Job Bundle export
|
||||||
|
function handleExportJobBundle() {
|
||||||
|
const bundle = generateJobBundle(rows, mappings, enrichmentRules, submissionConfig);
|
||||||
|
const blob = new Blob([JSON.stringify(bundle, null, 2)], { type: 'application/json' });
|
||||||
|
const url = URL.createObjectURL(blob);
|
||||||
|
const a = document.createElement('a');
|
||||||
|
a.href = url;
|
||||||
|
a.download = fileName ? fileName.replace(/\.[^.]+$/, '_job_bundle.json') : 'job_bundle.json';
|
||||||
|
a.click();
|
||||||
|
URL.revokeObjectURL(url);
|
||||||
|
}
|
||||||
|
|
||||||
|
// API enrichment rules management
|
||||||
|
function openAddApiConfig() {
|
||||||
|
editingRuleIndex = null;
|
||||||
|
showApiConfig = true;
|
||||||
|
}
|
||||||
|
|
||||||
|
function openEditApiConfig(index: number) {
|
||||||
|
editingRuleIndex = index;
|
||||||
|
showApiConfig = true;
|
||||||
|
}
|
||||||
|
|
||||||
|
function deleteApiRule(index: number) {
|
||||||
|
enrichmentRules = enrichmentRules.filter((_, i) => i !== index);
|
||||||
|
}
|
||||||
|
|
||||||
|
function handleSaveApiRule(rule: ApiEnrichmentRule) {
|
||||||
|
if (editingRuleIndex !== null) {
|
||||||
|
enrichmentRules = enrichmentRules.map((r, i) => (i === editingRuleIndex ? rule : r));
|
||||||
|
} else {
|
||||||
|
enrichmentRules = [...enrichmentRules, rule];
|
||||||
|
}
|
||||||
|
showApiConfig = false;
|
||||||
|
editingRuleIndex = null;
|
||||||
|
}
|
||||||
</script>
|
</script>
|
||||||
|
|
||||||
<div
|
<div
|
||||||
@@ -231,8 +294,37 @@
|
|||||||
导出配置
|
导出配置
|
||||||
</button>
|
</button>
|
||||||
|
|
||||||
|
<div class="mx-2 h-6 w-px bg-gray-200"></div>
|
||||||
|
|
||||||
|
<!-- Submission Settings -->
|
||||||
|
<button
|
||||||
|
onclick={() => (showSubmissionSettings = true)}
|
||||||
|
disabled={!hasData}
|
||||||
|
class="inline-flex items-center gap-1 rounded-md border border-gray-300 bg-white px-3 py-1.5 text-sm text-gray-700 hover:bg-gray-50 disabled:cursor-not-allowed disabled:opacity-50 cursor-pointer"
|
||||||
|
>
|
||||||
|
<svg class="h-4 w-4" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||||
|
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M10.325 4.317c.426-1.756 2.924-1.756 3.35 0a1.724 1.724 0 002.573 1.066c1.543-.94 3.31.826 2.37 2.37a1.724 1.724 0 001.066 2.573c1.756.426 1.756 2.924 0 3.35a1.724 1.724 0 00-1.066 2.573c.94 1.543-.826 3.31-2.37 2.37a1.724 1.724 0 00-2.573 1.066c-.426 1.756-2.924 1.756-3.35 0a1.724 1.724 0 00-2.573-1.066c-1.543.94-3.31-.826-2.37-2.37a1.724 1.724 0 00-1.066-2.573c-1.756-.426-1.756-2.924 0-3.35a1.724 1.724 0 001.066-2.573c-.94-1.543.826-3.31 2.37-2.37.996.608 2.296.07 2.572-1.065z" />
|
||||||
|
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M15 12a3 3 0 11-6 0 3 3 0 016 0z" />
|
||||||
|
</svg>
|
||||||
|
提交设置
|
||||||
|
</button>
|
||||||
|
|
||||||
<div class="flex-1"></div>
|
<div class="flex-1"></div>
|
||||||
|
|
||||||
|
<!-- Export Job Bundle -->
|
||||||
|
<button
|
||||||
|
onclick={handleExportJobBundle}
|
||||||
|
disabled={!hasData}
|
||||||
|
class="inline-flex items-center gap-1 rounded-md bg-emerald-600 px-3 py-1.5 text-sm font-medium text-white hover:bg-emerald-700 disabled:cursor-not-allowed disabled:opacity-50 cursor-pointer"
|
||||||
|
>
|
||||||
|
<svg class="h-4 w-4" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||||
|
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M20 7l-8-4-8 4m16 0l-8 4m8-4v10l-8 4m0-10L4 7m8 4v10M4 7v10l8 4" />
|
||||||
|
</svg>
|
||||||
|
导出任务包
|
||||||
|
</button>
|
||||||
|
|
||||||
|
<div class="mx-1 h-6 w-px bg-gray-200"></div>
|
||||||
|
|
||||||
<!-- JSON operations -->
|
<!-- JSON operations -->
|
||||||
<button
|
<button
|
||||||
onclick={handleDownloadJson}
|
onclick={handleDownloadJson}
|
||||||
@@ -261,6 +353,19 @@
|
|||||||
复制 JSON
|
复制 JSON
|
||||||
{/if}
|
{/if}
|
||||||
</button>
|
</button>
|
||||||
|
|
||||||
|
<!-- GitHub -->
|
||||||
|
<a
|
||||||
|
href="https://github.com/meowrain/Excel2JSON"
|
||||||
|
target="_blank"
|
||||||
|
rel="noopener noreferrer"
|
||||||
|
class="ml-1 inline-flex items-center rounded-md border border-gray-300 bg-white p-1.5 text-gray-600 hover:bg-gray-50 hover:text-gray-900"
|
||||||
|
title="GitHub"
|
||||||
|
>
|
||||||
|
<svg class="h-5 w-5" fill="currentColor" viewBox="0 0 24 24">
|
||||||
|
<path d="M12 .297c-6.63 0-12 5.373-12 12 0 5.303 3.438 9.8 8.205 11.385.6.113.82-.258.82-.577 0-.285-.01-1.04-.015-2.04-3.338.724-4.042-1.61-4.042-1.61C4.422 18.07 3.633 17.7 3.633 17.7c-1.087-.744.084-.729.084-.729 1.205.084 1.838 1.236 1.838 1.236 1.07 1.835 2.809 1.305 3.495.998.108-.776.417-1.305.76-1.605-2.665-.3-5.466-1.332-5.466-5.93 0-1.31.465-2.38 1.235-3.22-.135-.303-.54-1.523.105-3.176 0 0 1.005-.322 3.3 1.23.96-.267 1.98-.399 3-.405 1.02.006 2.04.138 3 .405 2.28-1.552 3.285-1.23 3.285-1.23.645 1.653.24 2.873.12 3.176.765.84 1.23 1.91 1.23 3.22 0 4.61-2.805 5.625-5.475 5.92.42.36.81 1.096.81 2.22 0 1.606-.015 2.896-.015 3.286 0 .315.21.69.825.57C20.565 22.092 24 17.592 24 12.297c0-6.627-5.373-12-12-12" />
|
||||||
|
</svg>
|
||||||
|
</a>
|
||||||
</header>
|
</header>
|
||||||
|
|
||||||
<!-- Error message -->
|
<!-- Error message -->
|
||||||
@@ -289,7 +394,15 @@
|
|||||||
>
|
>
|
||||||
<!-- Left: Excel Table -->
|
<!-- Left: Excel Table -->
|
||||||
<div class="overflow-hidden" style="width: {jsonCollapsed ? '100%' : `${splitPercent}%`}">
|
<div class="overflow-hidden" style="width: {jsonCollapsed ? '100%' : `${splitPercent}%`}">
|
||||||
<ExcelTable {headers} {rows} bind:mappings />
|
<ExcelTable
|
||||||
|
{headers}
|
||||||
|
{rows}
|
||||||
|
bind:mappings
|
||||||
|
{enrichmentRules}
|
||||||
|
onaddapi={openAddApiConfig}
|
||||||
|
oneditapi={openEditApiConfig}
|
||||||
|
ondeleteapi={deleteApiRule}
|
||||||
|
/>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{#if !jsonCollapsed}
|
{#if !jsonCollapsed}
|
||||||
@@ -356,3 +469,20 @@
|
|||||||
</div>
|
</div>
|
||||||
{/if}
|
{/if}
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
<!-- Modals -->
|
||||||
|
{#if showApiConfig}
|
||||||
|
<ApiConfigModal
|
||||||
|
rule={editingRuleIndex !== null ? enrichmentRules[editingRuleIndex] : undefined}
|
||||||
|
{headers}
|
||||||
|
onsave={handleSaveApiRule}
|
||||||
|
onclose={() => { showApiConfig = false; editingRuleIndex = null; }}
|
||||||
|
/>
|
||||||
|
{/if}
|
||||||
|
|
||||||
|
{#if showSubmissionSettings}
|
||||||
|
<SubmissionSettings
|
||||||
|
bind:config={submissionConfig}
|
||||||
|
onclose={() => (showSubmissionSettings = false)}
|
||||||
|
/>
|
||||||
|
{/if}
|
||||||
|
|||||||
Reference in New Issue
Block a user