Compare commits

...

6 Commits

20 changed files with 2333 additions and 98 deletions

View File

@@ -6,7 +6,10 @@
"Bash(npm run check:*)",
"Bash(npx vitest:*)",
"Bash(node -e:*)",
"Bash(npm run build:*)"
"Bash(npm run build:*)",
"Bash(npm run test:*)",
"Bash(python -m py_compile:*)",
"Bash(node --check:*)"
]
},
"enableAllProjectMcpServers": true,

View File

@@ -4,14 +4,15 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
## Project Overview
**excel2json** — a SvelteKit web application for converting Excel files to JSON. Built with Svelte 5, SvelteKit 2, TypeScript, and Tailwind CSS 4.
**excel2json** — a SvelteKit web application for converting Excel files to JSON with visual mapping configuration. Built with Svelte 5, SvelteKit 2, TypeScript, and Tailwind CSS 4.
## Commands
- `npm run dev` — start dev server
- `npm run dev` — start dev server (typically http://localhost:5173)
- `npm run build` — production build
- `npm run preview` — preview production build
- `npm run check` — type-check with svelte-check
- `npm run check:watch` — type-check in watch mode
- `npm run test` — run all tests once
- `npm run test:unit` — run tests in watch mode
- `npm run test:unit -- --run --testNamePattern="pattern"` — run a single test by name
@@ -22,10 +23,11 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
- **Styling**: Tailwind CSS 4 with `@tailwindcss/forms` and `@tailwindcss/typography` plugins, configured via `src/routes/layout.css`
- **Adapter**: `@sveltejs/adapter-auto`
- **TypeScript**: strict mode enabled
- **Dependencies**: `xlsx` (SheetJS) for Excel parsing, `dayjs` for date handling
### Testing
Two Vitest project configurations in `vite.config.ts`:
Two Vitest project configurations in `vitest.config.ts`:
- **`client`** — browser tests using Playwright (headless Chromium). Files matching `src/**/*.svelte.{test,spec}.{js,ts}`. Uses `vitest-browser-svelte` for component rendering.
- **`server`** — Node.js unit tests. Files matching `src/**/*.{test,spec}.{js,ts}` (excluding `.svelte.` test files).
@@ -38,6 +40,32 @@ All tests require assertions (`expect.requireAssertions: true`).
- Shared library code goes in `src/lib/` (aliased as `$lib`)
- Use Svelte 5 runes syntax, not legacy Svelte 4 patterns
### Core Application Structure
**Single-page application** with split-pane layout:
- `src/routes/+page.svelte` — Main page containing all application logic
- `src/lib/components/ExcelTable.svelte` — Left panel showing Excel data with column configuration
- `src/lib/components/JsonPreview.svelte` — Right panel showing JSON output with syntax highlighting
- `src/lib/components/ColumnConfig.svelte` — Modal for column mapping configuration
- `src/lib/components/ApiConfigModal.svelte` — API enrichment rules configuration
- `src/lib/components/SubmissionSettings.svelte` — Data submission settings
**Core libraries:**
- `src/lib/excel.ts` — Excel/CSV reading and parsing
- `src/lib/converter.ts` — Mapping conversion core logic
- `src/lib/types.ts` — TypeScript type definitions (`MappingConfig`, `RowData`, etc.)
**Data flow:** File upload → Excel parsing → Mapping configuration → JSON conversion → Preview/Export
**Key features:**
- Nested object support via dot notation (e.g., `user.address.city`)
- Date formatting including Excel serial date compatibility
- Empty value handling (exclude field or use default value)
- Template import/export for mapping configurations
- API enrichment for dynamic data fetching
## Svelte MCP Server
A Svelte MCP server is available for Svelte 5 / SvelteKit documentation lookup and code validation. When writing Svelte code:

137
DEMAND-queryFiled.md Normal file
View File

@@ -0,0 +1,137 @@
**Role:** 资深前端架构师 (Svelte 5 + TypeScript)
**Project Pivot (重大架构调整):**
我们将 "Excel2JSON Mapper" 升级为一个 **"ETL 配置生成器 (ETL Blueprint Generator)"**。
前端的任务是生成一个包含 **[源数据 + 处理逻辑]** 的 **Job Bundle (JSON 文件)**,用户将使用该文件配合 Python 脚本在后端执行实际的数据抓取和入库。
### Phase 2: ETL 配置生成器需求文档
#### 1. 项目概述
这是一个基于 Svelte 5 的单页应用。用户上传 Excel配置字段映射规则包含静态重命名和动态 API 获取规则),最后导出为一个标准化的 JSON 任务包 (`job_bundle.json`)。
#### 2. 用户界面与交互 (UI/UX)
##### 2.1 顶部工具栏
* **导入/导出配置:** 支持保存当前所有的映射规则。
* **导出任务包 (Export Job Bundle):** 這是核心操作。点击后下载 `job_bundle.json` 文件(包含数据+配置)。
* **提交设置 (Submission Settings):** 一个模态框,配置最终数据推送到哪里。
* `Target URL`: 最终数据接收接口 (e.g., `https://api.db.com/bulk-insert`).
* `Method`: POST / PUT.
* `Batch Size`: 批次大小 (默认 50).
##### 2.2 主体区域 (左右分栏)
* **左侧 (Source):** Excel 表格预览。
* **右侧 (Preview):** 静态映射后的 JSON 预览(仅展示前 20 条以保证性能)。
##### 2.3 核心功能:列配置 (Column Configuration)
在左侧表格区域,除了点击现有表头修改映射外,新增 **"添加计算列 (Add Computed Column)"** 功能。
**新增类型:`API_FETCH` (动态 API 字段)**
当用户选择此类型时,弹出一个详细配置面板:
1. **Target Key:** 最终生成的 JSON 字段名 (例如 `user_balance`)。
2. **Request URL (支持模板变量):**
* 允许使用 `{{ColumnName}}` 语法引用当前行的 Excel 数据。
* *示例:* `https://api.example.com/users/{{用户ID}}/detail`
* *UI 交互:* 输入框旁应有“插入变量”按钮,点击列出所有可用 Excel 表头。
3. **Request Method:** 下拉选择 `GET` (默认) 或 `POST`
4. **Headers:** Key-Value 编辑器 (用于传 `Authorization`, `Content-Type` 等)。
5. **Request Body (仅 POST):**
* 多行文本域,支持 JSON 格式。
* 同样支持 `{{ColumnName}}` 模板变量替换。
6. **Response Extractor (取值路径):**
* 指定从接口返回的 JSON 中提取哪个字段。
* 支持 `lodash.get` 风格的点号路径。
* *示例:* 接口返回 `{ "data": { "balance": 100 } }`,用户填 `data.balance`
#### 3. 核心输出Job Bundle 数据结构
请严格按照以下 TypeScript 接口定义生成导出的 JSON 文件:
```typescript
// 1. 静态映射规则
interface StaticRule {
type: 'static';
source: string; // Excel 原表头
target: string; // JSON 目标 Key
dataType: 'string' | 'number' | 'boolean' | 'date';
format?: string; // 日期格式化字符串
}
// 2. 动态 API 获取规则 (本次新增核心)
interface ApiEnrichmentRule {
type: 'api_fetch';
target_key: string; // JSON 目标 Key
url_template: string; // "https://api.com/{{id}}"
method: 'GET' | 'POST';
headers?: Record<string, string>;
body_template?: string; // POST body 模板
response_path: string; // "data.result.value"
fallback_value?: any; // 默认值 (null/0)
}
// 3. 提交配置
interface SubmissionConfig {
target_url: string;
method: 'POST' | 'PUT';
batch_size: number;
}
// 4. 最终导出的 Job Bundle 结构
interface JobBundle {
meta: {
version: string;
generated_at: string;
};
config: {
static_rules: StaticRule[];
enrichment_rules: ApiEnrichmentRule[];
submission: SubmissionConfig;
};
source_data: Record<string, any>[]; // 经过静态映射后的基础数据列表
}
```
#### 4. 开发任务清单
1. **Store 设计:** 更新 Svelte Store 以存储 `enrichmentRules``submissionConfig`
2. **UI 组件:**
* 开发 `ApiConfigModal.svelte`: 用于录入 URL、Headers、Body 等复杂信息。
* 实现变量插入辅助功能 (点击列名自动插入 `{{...}}`)。
3. **导出逻辑:** 编写 `generateJobBundle` 函数。
* **步骤 1:** 根据 `static_rules` 转换 Excel 原始数据,生成基础 JSON 数组。
* **步骤 2:** 将基础数据、API 规则、提交配置组装成 `JobBundle` 格式。
* **步骤 3:** 触发浏览器下载 `job_bundle.json`
#### 5. 特别说明 (给 AI 的提示)
* **No Runtime Fetch:** 前端代码 **不需要** 执行 `fetch` 去调用用户配置的 API避免 CORS。前端只负责把 URL 字符串保存到 JSON 里。
* **Template Validation:** 在 UI 上简单校验 URL 模板格式(检查是否包含 `{}`),但不做逻辑校验。
* **Preview Limitations:** 右侧预览仅展示静态映射的结果。对于 API 字段,可以在预览中显示一个占位符(如 `[Pending API Fetch]`)。
---
### 后续步骤
Claude 完成这个前端代码后,你可以再发一条指令让它写对应的 Python 执行脚本:
> "前端已经完成了。现在请帮我写一个 Python 脚本 (`executor.py`)。它读取上面定义的 `job_bundle.json`,使用 http请求 执行 `enrichment_rules` 里的请求(注意替换 URL 中的 {{变量}}),最后把结果推送到 `submission` 定义的接口。"

162
DEMAND-字典映射.md Normal file
View File

@@ -0,0 +1,162 @@
这次更新加入了 **“字典映射 (Dictionary Mapping)”** 功能,允许用户对有限的枚举值进行转换(如“是/否”转“Y/N”
我特别在文档中强调了**“自动扫描”**功能——让 UI 自动读取 Excel 列中的前几十行去重数据,自动列出所有可能的选项,用户只需要填右边的目标值即可,极大提升体验。
---
### 复制下面的内容发送给 Claude
---
**Role:** 资深前端架构师 (Svelte 5 + TypeScript)
**Project Context:**
我们正在开发 "Excel2JSON ETL Blueprint Generator"。
前端负责解析 Excel、配置映射规则含静态重命名、字典转换、动态 API、生成预览并最终导出包含 **[处理后的基础数据 + API 执行逻辑]** 的 `job_bundle.json`
**Current Task:**
我们需要在 **v2.0 (API Enrichment)** 的基础上,新增 **v2.1 (Dictionary/Enum Mapping)** 功能。
### Phase 2.1: 字典映射与高级 ETL 配置需求文档
#### 1. 核心功能变更
在静态字段映射Static Mapping阶段新增 **"值映射 (Value Map)"** 功能。
* **场景:** Excel 中某列是状态词(如 "开/关", "男/女"),目标 JSON 需要代码(如 `1/0`, `M/F`)。
* **逻辑:** 解析 Excel -> **字典替换** -> 类型转换 -> 生成基础 JSON -> (后续由 Python 处理 API).
#### 2. 用户界面与交互 (UI/UX)
##### 2.1 列配置面板升级
点击 Excel 表头配置时,除了修改 `Target Key``Data Type`,新增一个 **"Value Mapping" (值映射)** 开关/折叠面板。
**面板内容:**
1. **自动扫描 (Auto-Scan):**
* UI 自动读取该列的前 50 行数据,提取所有**唯一值 (Unique Values)**。
* 显示一个“映射表”:左侧是 `Source Value` (Excel 原值),右侧是 `Target Value` (输入框)。
2. **手动添加:** 允许用户手动增加新的映射对(防止前 50 行没覆盖到所有情况)。
3. **默认值 (Fallback):**
* 如果单元格的值不在映射表中,怎么处理?
* 选项: `Keep Original` (保留原值) / `Set to Null` / `Custom Value` (自定义默认值)。
##### 2.2 预览逻辑 (Preview Logic)
* 右侧 JSON 预览必须**实时反映**字典映射的结果。
* *示例:* 用户在左侧把 "是" 映射为 `true` (Boolean),右侧预览中原本的 "是" 应立即变为 `true`
#### 3. 核心数据结构 (Updated Interfaces)
请更新 TypeScript 接口以支持新的映射逻辑:
```typescript
// 字典映射项
interface ValueMapItem {
source: string | number; // Excel 里的原始值 (e.g., "是")
target: any; // JSON 里的目标值 (e.g., true, "Y", 1)
}
// 静态映射规则 (升级版)
interface StaticRule {
type: 'static';
source_column: string; // Excel 原表头
target_key: string; // JSON 目标 Key
data_type: 'string' | 'number' | 'boolean' | 'date' | 'array'; // 目标类型
// v2.1 新增: 字典映射配置
use_dictionary: boolean; // 是否启用字典映射
value_mapping?: ValueMapItem[];
mapping_fallback?: 'keep' | 'null' | any; // 没匹配到时的默认值
// v2.0 已有
format?: string; // 日期格式
separator?: string; // 数组分隔符
}
// 动态 API 规则 (保持 v2.0 不变)
interface ApiEnrichmentRule {
type: 'api_fetch';
target_key: string;
url_template: string; // "https://api.com/{{id}}"
method: 'GET' | 'POST';
headers?: Record<string, string>;
body_template?: string;
response_path: string; // "data.result"
}
// 提交配置 (保持 v2.0 不变)
interface SubmissionConfig {
target_url: string;
method: 'POST' | 'PUT';
batch_size: number;
}
// 最终导出的 Job Bundle
interface JobBundle {
meta: { version: string; generated_at: string };
config: {
// static_rules 仅用于前端回显Python 脚本其实只需要 enrichment 和 submission
// 但为了以后能在前端重新导入编辑,建议保留完整配置
static_rules: StaticRule[];
enrichment_rules: ApiEnrichmentRule[];
submission: SubmissionConfig;
};
// 注意: source_data 是前端已经应用了 "StaticRule" (包括字典映射) 后的干净数据
source_data: Record<string, any>[];
}
```
#### 4. 处理流程 (Processing Pipeline)
前端在生成 `source_data` 时,必须严格按照以下顺序处理每一单元格:
1. **Extract:** 读取 Excel 单元格原始值。
2. **Map (字典映射):**
* 如果启用了 `use_dictionary`:查找映射表。
* 找到 -> 替换为 Target Value。
* 没找到 -> 应用 `mapping_fallback` 策略。
3. **Cast (类型转换):**
* 将上一步的结果转换为 `data_type` 指定的类型 (e.g., String -> Boolean, String -> Number)。
* *注意:* 如果字典映射的目标值已经是正确的类型(如 `true`),则跳过此步或确保不会再次转为字符串 "true"。
4. **Format:** (如果是日期或数组) 应用格式化规则。
#### 5. 开发任务清单
1. **组件开发:**
* 修改 `ColumnConfigPanel.svelte` (或类似组件)。
* 新增 `DictionaryMapper` 子组件:包含“自动扫描”按钮和“键值对”编辑表格。
2. **逻辑核心:**
* 更新 `processRow` 函数,在类型转换前插入字典查找逻辑。
* 实现 `scanUniqueValues(columnData)` 函数,用于快速提取 Excel 列的去重值。
3. **预览同步:** 确保右侧 JSON 预览能实时响应字典配置的变更。
4. **导出验证:** 导出 `job_bundle.json`,检查 `source_data` 中的值是否已成功转换为映射后的值(例如 "Y" 而不是 "是")。
---
### 给 AI 的提示 (Prompt Tip)
* **性能注意:** 自动扫描 `scanUniqueValues` 时,如果 Excel 数据量极大(>10万行不要全量扫描。只扫描前 1000 行即可,并提示用户“仅扫描了前 1000 行,如有遗漏请手动添加”。
* **交互细节:** 字典映射的 Target Value 输入框,应该能智能识别类型。如果用户输入 `true`,应该被识别为 Boolean 而不是字符串 "true"。
---
### 执行步骤
请先基于上述文档,更新 **数据类型定义 (Interfaces)****核心处理逻辑 (`processRow` 函数)**

View File

@@ -106,9 +106,10 @@ npm run preview
-`excludeIfEmpty = true` 时,空值字段不会出现在输出 JSON 中
- 若目标字段包含 `.`,会按路径写入嵌套对象
## <20><><EFBFBD><EFBFBD><EFBFBD><EFBFBD>ͼ
![<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>ͼ 1](doc/images/image1.png)
## Screenshots
![<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>ͼ 2](doc/images/image2.png)
![Screenshot 1](doc/images/image1.png)
![Screenshot 2](doc/images/image2.png)

201
scripts/node/enricher.mjs Normal file
View File

@@ -0,0 +1,201 @@
/**
* enricher.mjs — API 数据补全脚本 (Node.js)
*
* 读取前端生成的 job_bundle.json执行 enrichment_rules 中的 API 请求,
* 将结果合并到 source_data 中,输出最终完整的 JSON 文件。
*
* 用法:
* node enricher.mjs job_bundle.json
* node enricher.mjs job_bundle.json -o enriched_data.json
* node enricher.mjs job_bundle.json --concurrency 10
*/
import { readFileSync, writeFileSync } from "fs";
import { basename, join, dirname } from "path";
// ── helpers ──
function resolvePath(obj, path) {
const keys = path.split(".");
let current = obj;
for (const key of keys) {
if (current == null) return null;
if (Array.isArray(current)) {
const idx = Number(key);
if (Number.isNaN(idx) || idx < 0 || idx >= current.length) return null;
current = current[idx];
} else if (typeof current === "object") {
current = current[key];
} else {
return null;
}
}
return current ?? null;
}
function renderTemplate(template, row) {
return template.replace(/\{\{(.+?)\}\}/g, (_, key) => {
const val = row[key];
return val != null ? String(val) : "";
});
}
// ── fetch one ──
async function fetchOne(rule, row, rowIndex, semaphore) {
const targetKey = rule.target_key;
const fallback = rule.fallback_value ?? null;
const url = renderTemplate(rule.url_template, row);
const method = (rule.method || "GET").toUpperCase();
const headers = {};
if (rule.headers) {
for (const [k, v] of Object.entries(rule.headers)) {
headers[k] = renderTemplate(v, row);
}
}
/** @type {RequestInit} */
const opts = { method, headers };
if (method === "POST" && rule.body_template) {
const body = renderTemplate(rule.body_template, row);
try {
JSON.parse(body);
headers["Content-Type"] = headers["Content-Type"] || "application/json";
opts.body = body;
} catch {
opts.body = body;
}
}
await semaphore.acquire();
try {
const resp = await fetch(url, opts);
if (!resp.ok) {
console.log(` [WARN] Row ${rowIndex} | ${targetKey} | HTTP ${resp.status} <- ${url}`);
return { rowIndex, targetKey, value: fallback };
}
const data = await resp.json();
const value = resolvePath(data, rule.response_path);
return { rowIndex, targetKey, value: value ?? fallback };
} catch (e) {
console.log(` [ERROR] Row ${rowIndex} | ${targetKey} | ${e.message}`);
return { rowIndex, targetKey, value: fallback };
} finally {
semaphore.release();
}
}
// ── semaphore ──
function createSemaphore(max) {
let current = 0;
/** @type {(() => void)[]} */
const queue = [];
return {
acquire() {
if (current < max) {
current++;
return Promise.resolve();
}
return new Promise((resolve) => queue.push(resolve));
},
release() {
current--;
if (queue.length > 0) {
current++;
queue.shift()();
}
},
};
}
// ── main ──
function parseArgs() {
const args = process.argv.slice(2);
const opts = { bundle: "", output: "", concurrency: 5 };
for (let i = 0; i < args.length; i++) {
if (args[i] === "-o" || args[i] === "--output") {
opts.output = args[++i];
} else if (args[i] === "--concurrency") {
opts.concurrency = Number(args[++i]) || 5;
} else if (args[i] === "--help" || args[i] === "-h") {
console.log("Usage: node enricher.mjs <job_bundle.json> [-o output.json] [--concurrency N]");
process.exit(0);
} else if (!opts.bundle) {
opts.bundle = args[i];
}
}
if (!opts.bundle) {
console.error("Error: Please provide a job_bundle.json path.");
console.error("Usage: node enricher.mjs <job_bundle.json>");
process.exit(1);
}
return opts;
}
async function main() {
const opts = parseArgs();
const raw = readFileSync(opts.bundle, "utf-8");
const bundle = JSON.parse(raw);
const meta = bundle.meta || {};
const config = bundle.config || {};
const sourceData = bundle.source_data || [];
const rules = config.enrichment_rules || [];
console.log(`=== Job Bundle v${meta.version || "?"} ===`);
console.log(`Generated: ${meta.generated_at || "?"}`);
console.log(`Rows: ${sourceData.length}`);
console.log(`Static rules: ${(config.static_rules || []).length}`);
console.log(`Enrichment rules: ${rules.length}`);
console.log();
if (rules.length === 0) {
console.log("No enrichment rules configured. Outputting source data as-is.");
} else {
const totalCalls = sourceData.length * rules.length;
console.log(`Enriching ${sourceData.length} rows x ${rules.length} rule(s) = ${totalCalls} API calls`);
console.log(`Concurrency: ${opts.concurrency}`);
console.log();
const semaphore = createSemaphore(opts.concurrency);
const tasks = [];
for (let rowIdx = 0; rowIdx < sourceData.length; rowIdx++) {
for (const rule of rules) {
tasks.push(fetchOne(rule, sourceData[rowIdx], rowIdx, semaphore));
}
}
const results = await Promise.all(tasks);
let errorCount = 0;
for (const r of results) {
if (r.value === undefined) errorCount++;
sourceData[r.rowIndex][r.targetKey] = r.value;
}
console.log();
console.log(`Done. ${totalCalls - errorCount}/${totalCalls} calls succeeded.`);
}
const output = {
meta,
submission: config.submission || {},
data: sourceData,
};
const outName = opts.output || opts.bundle.replace(/\.json$/i, "_enriched.json");
writeFileSync(outName, JSON.stringify(output, null, 2), "utf-8");
console.log(`\nEnriched data saved to: ${outName}`);
console.log(`Next step: node submitter.mjs ${outName}`);
}
main().catch((e) => {
console.error(e);
process.exit(1);
});

177
scripts/node/submitter.mjs Normal file
View File

@@ -0,0 +1,177 @@
/**
* submitter.mjs — 数据提交脚本 (Node.js)
*
* 读取 enricher.mjs 输出的 JSON 文件,按批次提交到目标接口。
* 记录提交成功和失败的记录到单独的日志文件。
*
* 用法:
* node submitter.mjs enriched_data.json
* node submitter.mjs enriched_data.json --batch-size 100
* node submitter.mjs enriched_data.json --url https://api.example.com/import
* node submitter.mjs enriched_data.json --dry-run
*/
import { readFileSync, writeFileSync } from "fs";
// ── args ──
function parseArgs() {
const args = process.argv.slice(2);
const opts = { input: "", url: "", method: "", batchSize: 0, dryRun: false };
for (let i = 0; i < args.length; i++) {
if (args[i] === "--url") {
opts.url = args[++i];
} else if (args[i] === "--method") {
opts.method = args[++i];
} else if (args[i] === "--batch-size") {
opts.batchSize = Number(args[++i]) || 0;
} else if (args[i] === "--dry-run") {
opts.dryRun = true;
} else if (args[i] === "--help" || args[i] === "-h") {
console.log("Usage: node submitter.mjs <enriched.json> [--url URL] [--method POST|PUT] [--batch-size N] [--dry-run]");
process.exit(0);
} else if (!opts.input) {
opts.input = args[i];
}
}
if (!opts.input) {
console.error("Error: Please provide an enriched JSON file path.");
console.error("Usage: node submitter.mjs <enriched.json>");
process.exit(1);
}
return opts;
}
// ── submit ──
async function submitBatch(url, method, batch, batchIndex, totalBatches) {
try {
const resp = await fetch(url, {
method,
headers: { "Content-Type": "application/json" },
body: JSON.stringify(batch),
});
const body = await resp.text();
const ok = resp.status < 400;
const tag = ok ? "OK" : "FAIL";
console.log(` Batch ${batchIndex}/${totalBatches}: HTTP ${resp.status} ${tag} (${batch.length} records)`);
return { ok, status: resp.status, body, data: batch };
} catch (e) {
const msg = e.message;
console.log(` Batch ${batchIndex}/${totalBatches}: ERROR - ${msg} (${batch.length} records)`);
return { ok: false, status: 0, body: msg, data: batch };
}
}
// ── main ──
async function main() {
const opts = parseArgs();
const raw = readFileSync(opts.input, "utf-8");
const payload = JSON.parse(raw);
const data = payload.data || [];
const submission = payload.submission || {};
const targetUrl = opts.url || submission.target_url || "";
const method = (opts.method || submission.method || "POST").toUpperCase();
const batchSize = opts.batchSize || submission.batch_size || 50;
if (!targetUrl) {
console.error("Error: No target URL configured. Use --url or set in job bundle.");
process.exit(1);
}
console.log("=== Submitter ===");
console.log(`Input: ${opts.input} (${data.length} records)`);
console.log(`Target: ${method} ${targetUrl}`);
console.log(`Batch size: ${batchSize}`);
console.log();
// 分批
const batches = [];
for (let i = 0; i < data.length; i += batchSize) {
batches.push(data.slice(i, i + batchSize));
}
const totalBatches = batches.length;
if (opts.dryRun) {
console.log(`[DRY RUN] Would submit ${data.length} records in ${totalBatches} batch(es)`);
for (let i = 0; i < batches.length; i++) {
console.log(` Batch ${i + 1}/${totalBatches}: ${batches[i].length} records`);
}
console.log("\nDry run complete. No data was sent.");
return;
}
/** @type {object[]} */
const successRecords = [];
/** @type {object[]} */
const failedRecords = [];
/** @type {object[]} */
const failedDetails = [];
console.log(`Submitting ${data.length} records in ${totalBatches} batch(es)...\n`);
for (let i = 0; i < batches.length; i++) {
const result = await submitBatch(targetUrl, method, batches[i], i + 1, totalBatches);
if (result.ok) {
successRecords.push(...result.data);
} else {
failedRecords.push(...result.data);
failedDetails.push({
batch_index: i + 1,
status: result.status,
response: result.body.slice(0, 500),
record_count: result.data.length,
});
}
}
// 结果统计
console.log();
console.log("=== Result ===");
console.log(`Success: ${successRecords.length} records`);
console.log(`Failed: ${failedRecords.length} records`);
const timestamp = new Date().toISOString().replace(/[-:T]/g, "").slice(0, 15);
const baseName = opts.input.replace(/\.json$/i, "");
if (successRecords.length > 0) {
const successFile = `${baseName}_success_${timestamp}.json`;
writeFileSync(successFile, JSON.stringify(successRecords, null, 2), "utf-8");
console.log(`\nSuccess log: ${successFile}`);
}
if (failedRecords.length > 0) {
const failedFile = `${baseName}_failed_${timestamp}.json`;
const report = {
summary: {
total_failed: failedRecords.length,
failed_batches: failedDetails.length,
target_url: targetUrl,
timestamp,
},
batch_errors: failedDetails,
failed_records: failedRecords,
};
writeFileSync(failedFile, JSON.stringify(report, null, 2), "utf-8");
console.log(`Failed log: ${failedFile}`);
const retryFile = `${baseName}_retry_${timestamp}.json`;
const retryPayload = { submission, data: failedRecords };
writeFileSync(retryFile, JSON.stringify(retryPayload, null, 2), "utf-8");
console.log(`\nTo retry failed records:`);
console.log(` node submitter.mjs ${retryFile}`);
}
if (successRecords.length === 0 && failedRecords.length === 0) {
console.log("\nNo records to submit.");
}
}
main().catch((e) => {
console.error(e);
process.exit(1);
});

186
scripts/py/enricher.py Normal file
View File

@@ -0,0 +1,186 @@
"""
enricher.py — API 数据补全脚本
读取前端生成的 job_bundle.json执行 enrichment_rules 中的 API 请求,
将结果合并到 source_data 中,输出最终完整的 JSON 文件。
用法:
python enricher.py job_bundle.json
python enricher.py job_bundle.json -o enriched_data.json
python enricher.py job_bundle.json --concurrency 10
"""
import argparse
import asyncio
import json
import re
import sys
from pathlib import Path
import aiohttp
def resolve_path(obj, path: str):
"""按点号路径从 dict/list 中取值,类似 lodash.get"""
keys = path.split(".")
current = obj
for key in keys:
if isinstance(current, dict) and key in current:
current = current[key]
elif isinstance(current, list):
try:
current = current[int(key)]
except (ValueError, IndexError):
return None
else:
return None
return current
def render_template(template: str, row: dict) -> str:
"""{{变量名}} 替换为行数据中的对应值"""
def replacer(match: re.Match) -> str:
key = match.group(1)
value = row.get(key, "")
return str(value) if value is not None else ""
return re.sub(r"\{\{(.+?)\}\}", replacer, template)
async def fetch_one(
session: aiohttp.ClientSession,
rule: dict,
row: dict,
row_index: int,
semaphore: asyncio.Semaphore,
) -> tuple[int, str, object]:
"""对单行数据执行一条 enrichment rule 的 API 请求"""
target_key = rule["target_key"]
fallback = rule.get("fallback_value")
url = render_template(rule["url_template"], row)
headers = {}
if rule.get("headers"):
for k, v in rule["headers"].items():
headers[k] = render_template(v, row)
method = rule.get("method", "GET").upper()
body = None
if method == "POST" and rule.get("body_template"):
body = render_template(rule["body_template"], row)
async with semaphore:
try:
kwargs: dict = {"headers": headers}
if body is not None:
try:
kwargs["json"] = json.loads(body)
except json.JSONDecodeError:
kwargs["data"] = body
async with session.request(method, url, **kwargs) as resp:
if resp.status >= 400:
print(f" [WARN] Row {row_index} | {target_key} | HTTP {resp.status} <- {url}")
return row_index, target_key, fallback
data = await resp.json(content_type=None)
value = resolve_path(data, rule["response_path"])
if value is None:
value = fallback
return row_index, target_key, value
except Exception as e:
print(f" [ERROR] Row {row_index} | {target_key} | {type(e).__name__}: {e}")
return row_index, target_key, fallback
async def run_enrichments(
source_data: list[dict],
rules: list[dict],
concurrency: int,
) -> list[dict]:
"""对所有行执行所有 enrichment rules"""
if not rules:
print("No enrichment rules configured. Outputting source data as-is.")
return source_data
semaphore = asyncio.Semaphore(concurrency)
tasks = []
total_calls = len(source_data) * len(rules)
print(f"Enriching {len(source_data)} rows x {len(rules)} rule(s) = {total_calls} API calls")
print(f"Concurrency: {concurrency}")
print()
async with aiohttp.ClientSession() as session:
for row_idx, row in enumerate(source_data):
for rule in rules:
tasks.append(fetch_one(session, rule, row, row_idx, semaphore))
results = await asyncio.gather(*tasks, return_exceptions=True)
error_count = 0
for result in results:
if isinstance(result, Exception):
print(f" [ERROR] Unexpected: {result}")
error_count += 1
continue
row_idx, target_key, value = result
source_data[row_idx][target_key] = value
print()
print(f"Done. {total_calls - error_count}/{total_calls} calls succeeded.")
return source_data
async def main():
parser = argparse.ArgumentParser(description="Enrich job_bundle.json with API data and output final JSON.")
parser.add_argument("bundle", help="Path to job_bundle.json")
parser.add_argument("-o", "--output", type=str, default=None, help="Output file path (default: <bundle>_enriched.json)")
parser.add_argument("--concurrency", type=int, default=5, help="Max concurrent API requests (default: 5)")
args = parser.parse_args()
bundle_path = Path(args.bundle)
if not bundle_path.exists():
print(f"Error: File not found: {bundle_path}")
sys.exit(1)
with open(bundle_path, "r", encoding="utf-8") as f:
bundle = json.load(f)
meta = bundle.get("meta", {})
config = bundle.get("config", {})
source_data = bundle.get("source_data", [])
print(f"=== Job Bundle v{meta.get('version', '?')} ===")
print(f"Generated: {meta.get('generated_at', '?')}")
print(f"Rows: {len(source_data)}")
print(f"Static rules: {len(config.get('static_rules', []))}")
print(f"Enrichment rules: {len(config.get('enrichment_rules', []))}")
print()
enriched = await run_enrichments(
source_data,
config.get("enrichment_rules", []),
args.concurrency,
)
# 构建输出,保留 submission 配置
output = {
"meta": meta,
"submission": config.get("submission", {}),
"data": enriched,
}
output_path = Path(args.output) if args.output else bundle_path.with_name(bundle_path.stem + "_enriched.json")
with open(output_path, "w", encoding="utf-8") as f:
json.dump(output, f, ensure_ascii=False, indent=2)
print(f"\nEnriched data saved to: {output_path}")
print(f"Next step: python submitter.py {output_path}")
if __name__ == "__main__":
asyncio.run(main())

161
scripts/py/submitter.py Normal file
View File

@@ -0,0 +1,161 @@
"""
submitter.py — 数据提交脚本
读取 enricher.py 输出的 JSON 文件,按批次提交到目标接口。
记录提交成功和失败的记录到单独的日志文件。
用法:
python submitter.py enriched_data.json
python submitter.py enriched_data.json --batch-size 100
python submitter.py enriched_data.json --url https://api.example.com/import # 覆盖 URL
python submitter.py enriched_data.json --dry-run # 只输出不提交
"""
import argparse
import asyncio
import json
import sys
from datetime import datetime
from pathlib import Path
import aiohttp
async def submit_batch(
session: aiohttp.ClientSession,
url: str,
method: str,
batch: list[dict],
batch_index: int,
total_batches: int,
) -> tuple[bool, int, str, list[dict]]:
"""提交一个批次,返回 (success, status_code, response_text, batch_data)"""
try:
async with session.request(method, url, json=batch) as resp:
status = resp.status
body = await resp.text()
ok = status < 400
tag = "OK" if ok else "FAIL"
print(f" Batch {batch_index}/{total_batches}: HTTP {status} {tag} ({len(batch)} records)")
return ok, status, body, batch
except Exception as e:
msg = f"{type(e).__name__}: {e}"
print(f" Batch {batch_index}/{total_batches}: ERROR - {msg} ({len(batch)} records)")
return False, 0, msg, batch
async def main():
parser = argparse.ArgumentParser(description="Submit enriched data to target API in batches.")
parser.add_argument("input", help="Path to enriched JSON file (from enricher.py)")
parser.add_argument("--url", type=str, default=None, help="Override target URL from config")
parser.add_argument("--method", type=str, default=None, choices=["POST", "PUT"], help="Override HTTP method")
parser.add_argument("--batch-size", type=int, default=None, help="Override batch size")
parser.add_argument("--dry-run", action="store_true", help="Print what would be submitted without sending")
args = parser.parse_args()
input_path = Path(args.input)
if not input_path.exists():
print(f"Error: File not found: {input_path}")
sys.exit(1)
with open(input_path, "r", encoding="utf-8") as f:
payload = json.load(f)
data = payload.get("data", [])
submission = payload.get("submission", {})
target_url = args.url or submission.get("target_url", "")
method = args.method or submission.get("method", "POST")
batch_size = args.batch_size or submission.get("batch_size", 50)
if not target_url:
print("Error: No target URL configured. Use --url or set in job bundle.")
sys.exit(1)
print(f"=== Submitter ===")
print(f"Input: {input_path} ({len(data)} records)")
print(f"Target: {method} {target_url}")
print(f"Batch size: {batch_size}")
print()
batches = [data[i : i + batch_size] for i in range(0, len(data), batch_size)]
total_batches = len(batches)
if args.dry_run:
print(f"[DRY RUN] Would submit {len(data)} records in {total_batches} batch(es)")
for i, batch in enumerate(batches, 1):
print(f" Batch {i}/{total_batches}: {len(batch)} records")
print("\nDry run complete. No data was sent.")
return
success_records: list[dict] = []
failed_records: list[dict] = []
failed_details: list[dict] = []
print(f"Submitting {len(data)} records in {total_batches} batch(es)...\n")
async with aiohttp.ClientSession() as session:
for i, batch in enumerate(batches, 1):
ok, status, body, batch_data = await submit_batch(
session, target_url, method, batch, i, total_batches
)
if ok:
success_records.extend(batch_data)
else:
failed_records.extend(batch_data)
failed_details.append({
"batch_index": i,
"status": status,
"response": body[:500],
"record_count": len(batch_data),
})
# 结果统计
print()
print(f"=== Result ===")
print(f"Success: {len(success_records)} records")
print(f"Failed: {len(failed_records)} records")
# 写入日志文件
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
base_name = input_path.stem
if success_records:
success_file = input_path.with_name(f"{base_name}_success_{timestamp}.json")
with open(success_file, "w", encoding="utf-8") as f:
json.dump(success_records, f, ensure_ascii=False, indent=2)
print(f"\nSuccess log: {success_file}")
if failed_records:
failed_file = input_path.with_name(f"{base_name}_failed_{timestamp}.json")
report = {
"summary": {
"total_failed": len(failed_records),
"failed_batches": len(failed_details),
"target_url": target_url,
"timestamp": timestamp,
},
"batch_errors": failed_details,
"failed_records": failed_records,
}
with open(failed_file, "w", encoding="utf-8") as f:
json.dump(report, f, ensure_ascii=False, indent=2)
print(f"Failed log: {failed_file}")
print(f"\nTo retry failed records:")
print(f' python submitter.py {failed_file} --url "{target_url}"')
# 生成一个可直接重试的文件
retry_file = input_path.with_name(f"{base_name}_retry_{timestamp}.json")
retry_payload = {
"submission": submission,
"data": failed_records,
}
with open(retry_file, "w", encoding="utf-8") as f:
json.dump(retry_payload, f, ensure_ascii=False, indent=2)
print(f" or: python submitter.py {retry_file}")
if not success_records and not failed_records:
print("\nNo records to submit.")
if __name__ == "__main__":
asyncio.run(main())

View File

@@ -0,0 +1,298 @@
<script lang="ts">
import type { ApiEnrichmentRule } from '$lib/types.js';
let {
rule,
headers,
onsave,
onclose
}: {
rule?: ApiEnrichmentRule;
headers: string[];
onsave: (rule: ApiEnrichmentRule) => void;
onclose: () => void;
} = $props();
// svelte-ignore state_referenced_locally — intentionally using initial values only
let targetKey = $state(rule?.target_key ?? '');
// svelte-ignore state_referenced_locally
let urlTemplate = $state(rule?.url_template ?? '');
// svelte-ignore state_referenced_locally
let method = $state<'GET' | 'POST'>(rule?.method ?? 'GET');
// svelte-ignore state_referenced_locally
let headerEntries = $state<{ key: string; value: string }[]>(
rule?.headers ? Object.entries(rule.headers).map(([key, value]) => ({ key, value })) : []
);
// svelte-ignore state_referenced_locally
let bodyTemplate = $state(rule?.body_template ?? '');
// svelte-ignore state_referenced_locally
let responsePath = $state(rule?.response_path ?? '');
// svelte-ignore state_referenced_locally
let fallbackValue = $state(rule?.fallback_value != null ? String(rule.fallback_value) : '');
let showUrlVars = $state(false);
let showBodyVars = $state(false);
let urlInput: HTMLInputElement | undefined = $state();
let bodyTextarea: HTMLTextAreaElement | undefined = $state();
function insertVariable(field: 'url' | 'body', header: string) {
const variable = `{{${header}}}`;
if (field === 'url') {
if (urlInput) {
const start = urlInput.selectionStart ?? urlTemplate.length;
urlTemplate = urlTemplate.slice(0, start) + variable + urlTemplate.slice(urlInput.selectionEnd ?? start);
} else {
urlTemplate += variable;
}
showUrlVars = false;
} else {
if (bodyTextarea) {
const start = bodyTextarea.selectionStart ?? bodyTemplate.length;
bodyTemplate = bodyTemplate.slice(0, start) + variable + bodyTemplate.slice(bodyTextarea.selectionEnd ?? start);
} else {
bodyTemplate += variable;
}
showBodyVars = false;
}
}
function addHeader() {
headerEntries = [...headerEntries, { key: '', value: '' }];
}
function removeHeader(index: number) {
headerEntries = headerEntries.filter((_, i) => i !== index);
}
function save() {
if (!targetKey.trim() || !urlTemplate.trim() || !responsePath.trim()) return;
const headersObj: Record<string, string> = {};
for (const entry of headerEntries) {
if (entry.key.trim()) {
headersObj[entry.key.trim()] = entry.value;
}
}
const newRule: ApiEnrichmentRule = {
type: 'api_fetch',
target_key: targetKey.trim(),
url_template: urlTemplate.trim(),
method,
response_path: responsePath.trim()
};
if (Object.keys(headersObj).length > 0) newRule.headers = headersObj;
if (method === 'POST' && bodyTemplate.trim()) newRule.body_template = bodyTemplate.trim();
if (fallbackValue !== '') newRule.fallback_value = fallbackValue;
onsave(newRule);
}
const isValid = $derived(targetKey.trim() !== '' && urlTemplate.trim() !== '' && responsePath.trim() !== '');
</script>
<!-- svelte-ignore a11y_no_static_element_interactions a11y_click_events_have_key_events -->
<!-- svelte-ignore a11y_click_events_have_key_events -->
<div class="fixed inset-0 z-50 flex items-center justify-center bg-black/40" onclick={onclose} role="presentation">
<!-- svelte-ignore a11y_no_static_element_interactions a11y_click_events_have_key_events -->
<!-- svelte-ignore a11y_click_events_have_key_events -->
<div class="max-h-[90vh] w-full max-w-lg overflow-y-auto rounded-lg bg-white p-6 shadow-xl" onclick={(e) => e.stopPropagation()}>
<div class="mb-4 flex items-center justify-between">
<h3 class="text-lg font-semibold text-gray-800">
{rule ? '编辑' : '添加'} API 字段
</h3>
<button onclick={onclose} aria-label="关闭" class="cursor-pointer text-gray-400 hover:text-gray-600">
<svg class="h-5 w-5" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M6 18L18 6M6 6l12 12" />
</svg>
</button>
</div>
<div class="space-y-4">
<!-- Target Key -->
<div>
<label for="api-target-key" class="mb-1 block text-sm font-medium text-gray-700">Target Key</label>
<input
id="api-target-key"
type="text"
bind:value={targetKey}
placeholder="user_balance"
class="w-full rounded-md border border-gray-300 px-3 py-2 text-sm focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
/>
<p class="mt-1 text-xs text-gray-400">最终 JSON 中的字段名</p>
</div>
<!-- Request URL -->
<div>
<label for="api-url" class="mb-1 block text-sm font-medium text-gray-700">Request URL</label>
<div class="flex gap-1">
<input
id="api-url"
type="text"
bind:this={urlInput}
bind:value={urlTemplate}
placeholder={"https://api.example.com/users/{{用户ID}}/detail"}
class="flex-1 rounded-md border border-gray-300 px-3 py-2 text-sm focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
/>
<div class="relative">
<button
onclick={() => { showUrlVars = !showUrlVars; showBodyVars = false; }}
class="cursor-pointer whitespace-nowrap rounded-md border border-gray-300 px-2 py-2 text-xs text-gray-600 hover:bg-gray-50"
title="插入变量"
>
&#123;&#123;x&#125;&#125;
</button>
{#if showUrlVars}
<div class="absolute right-0 z-10 mt-1 max-h-48 w-40 overflow-y-auto rounded-md border border-gray-200 bg-white py-1 shadow-lg">
{#each headers as h (h)}
<button
onclick={() => insertVariable('url', h)}
class="w-full cursor-pointer px-3 py-1.5 text-left text-xs text-gray-700 hover:bg-blue-50"
>
{h}
</button>
{/each}
</div>
{/if}
</div>
</div>
<p class="mt-1 text-xs text-gray-400">支持 &#123;&#123;列名&#125;&#125; 模板变量</p>
</div>
<!-- Request Method -->
<div>
<label for="api-method" class="mb-1 block text-sm font-medium text-gray-700">Request Method</label>
<select
id="api-method"
bind:value={method}
class="w-full rounded-md border border-gray-300 px-3 py-2 text-sm focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
>
<option value="GET">GET</option>
<option value="POST">POST</option>
</select>
</div>
<!-- Headers -->
<div>
<div class="mb-1 flex items-center justify-between">
<span class="text-sm font-medium text-gray-700">Headers</span>
<button onclick={addHeader} class="cursor-pointer text-xs text-blue-600 hover:text-blue-700">
+ 添加
</button>
</div>
{#if headerEntries.length > 0}
<div class="space-y-2">
{#each headerEntries as entry, i (i)}
<div class="flex items-center gap-1">
<input
type="text"
bind:value={entry.key}
placeholder="Key"
class="flex-1 rounded border border-gray-300 px-2 py-1.5 text-xs focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
/>
<input
type="text"
bind:value={entry.value}
placeholder="Value"
class="flex-1 rounded border border-gray-300 px-2 py-1.5 text-xs focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
/>
<button
onclick={() => removeHeader(i)}
class="cursor-pointer flex-shrink-0 text-gray-400 hover:text-red-500"
aria-label="删除此 Header"
>
<svg class="h-4 w-4" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M6 18L18 6M6 6l12 12" />
</svg>
</button>
</div>
{/each}
</div>
{:else}
<p class="text-xs text-gray-400">暂无 Header</p>
{/if}
</div>
<!-- Body (POST only) -->
{#if method === 'POST'}
<div>
<div class="mb-1 flex items-center justify-between">
<label for="api-body" class="text-sm font-medium text-gray-700">Request Body</label>
<div class="relative">
<button
onclick={() => { showBodyVars = !showBodyVars; showUrlVars = false; }}
class="cursor-pointer text-xs text-blue-600 hover:text-blue-700"
>
插入变量
</button>
{#if showBodyVars}
<div class="absolute right-0 z-10 mt-1 max-h-48 w-40 overflow-y-auto rounded-md border border-gray-200 bg-white py-1 shadow-lg">
{#each headers as h (h)}
<button
onclick={() => insertVariable('body', h)}
class="w-full cursor-pointer px-3 py-1.5 text-left text-xs text-gray-700 hover:bg-blue-50"
>
{h}
</button>
{/each}
</div>
{/if}
</div>
</div>
<textarea
id="api-body"
bind:this={bodyTextarea}
bind:value={bodyTemplate}
rows="4"
placeholder={'{"user_id": "{{用户ID}}"}'}
class="w-full rounded-md border border-gray-300 px-3 py-2 font-mono text-xs focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
></textarea>
</div>
{/if}
<!-- Response Path -->
<div>
<label for="api-response-path" class="mb-1 block text-sm font-medium text-gray-700">Response Extractor</label>
<input
id="api-response-path"
type="text"
bind:value={responsePath}
placeholder="data.balance"
class="w-full rounded-md border border-gray-300 px-3 py-2 text-sm focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
/>
<p class="mt-1 text-xs text-gray-400">从接口返回 JSON 中提取值的路径(如 data.result.value</p>
</div>
<!-- Fallback Value -->
<div>
<label for="api-fallback" class="mb-1 block text-sm font-medium text-gray-700">Fallback Value</label>
<input
id="api-fallback"
type="text"
bind:value={fallbackValue}
placeholder="null"
class="w-full rounded-md border border-gray-300 px-3 py-2 text-sm focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
/>
<p class="mt-1 text-xs text-gray-400">API 请求失败时的默认值</p>
</div>
</div>
<div class="mt-6 flex justify-end gap-2">
<button
onclick={onclose}
class="cursor-pointer rounded-md border border-gray-300 px-4 py-2 text-sm text-gray-700 hover:bg-gray-50"
>
取消
</button>
<button
onclick={save}
disabled={!isValid}
class="cursor-pointer rounded-md bg-blue-600 px-4 py-2 text-sm font-medium text-white hover:bg-blue-700 disabled:cursor-not-allowed disabled:opacity-50"
>
保存
</button>
</div>
</div>
</div>

View File

@@ -1,7 +1,16 @@
<script lang="ts">
import type { MappingConfig, DataType, DateFormat } from '$lib/types.js';
import type { MappingConfig, DataType, DateFormat, RowData } from '$lib/types.js';
import DictionaryMapper from './DictionaryMapper.svelte';
let { config = $bindable(), onclose }: { config: MappingConfig; onclose: () => void } = $props();
let {
config = $bindable(),
rows,
onclose
}: {
config: MappingConfig;
rows?: RowData[];
onclose: () => void;
} = $props();
const dataTypes: { value: DataType; label: string }[] = [
{ value: 'string', label: '字符串 (String)' },
@@ -19,6 +28,19 @@
{ value: 'timestamp', label: 'Unix Timestamp' }
];
// Initialize dictionary mapping config if not present
$effect(() => {
if (config.useDictionary === undefined) {
config.useDictionary = false;
}
if (!config.valueMapping) {
config.valueMapping = [];
}
if (!config.mappingFallback) {
config.mappingFallback = 'keep';
}
});
function onTypeChange(e: Event) {
const select = e.target as HTMLSelectElement;
config.type = select.value as DataType;
@@ -51,10 +73,17 @@
});
</script>
<div bind:this={panelEl} class="absolute top-full left-0 z-50 mt-1 w-72 rounded-lg border border-gray-200 bg-white p-4 shadow-xl">
<div
bind:this={panelEl}
class="absolute top-full left-0 z-50 mt-1 w-80 max-h-[80vh] overflow-y-auto rounded-lg border border-gray-200 bg-white p-4 shadow-xl"
>
<div class="mb-3 flex items-center justify-between">
<h4 class="text-sm font-semibold text-gray-700">列配置</h4>
<button onclick={onclose} aria-label="关闭配置" class="text-gray-400 hover:text-gray-600 cursor-pointer">
<button
onclick={onclose}
aria-label="关闭配置"
class="text-gray-400 hover:text-gray-600 cursor-pointer"
>
<svg class="h-4 w-4" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M6 18L18 6M6 6l12 12" />
</svg>
@@ -64,23 +93,42 @@
<div class="space-y-3">
<!-- Source header (read-only) -->
<div>
<label for="source-{config.source}" class="mb-1 block text-xs font-medium text-gray-500">源字段</label>
<input id="source-{config.source}" type="text" value={config.source} disabled
class="w-full rounded border border-gray-200 bg-gray-50 px-2 py-1.5 text-sm text-gray-500" />
<label for="source-{config.source}" class="mb-1 block text-xs font-medium text-gray-500"
>源字段</label
>
<input
id="source-{config.source}"
type="text"
value={config.source}
disabled
class="w-full rounded border border-gray-200 bg-gray-50 px-2 py-1.5 text-sm text-gray-500"
/>
</div>
<!-- Target key -->
<div>
<label for="target-{config.source}" class="mb-1 block text-xs font-medium text-gray-500">目标字段 (JSON Key)</label>
<input id="target-{config.source}" type="text" bind:value={config.target}
class="w-full rounded border border-gray-300 px-2 py-1.5 text-sm focus:border-blue-500 focus:ring-1 focus:ring-blue-500" />
<label for="target-{config.source}" class="mb-1 block text-xs font-medium text-gray-500"
>目标字段 (JSON Key)</label
>
<input
id="target-{config.source}"
type="text"
bind:value={config.target}
class="w-full rounded border border-gray-300 px-2 py-1.5 text-sm focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
/>
</div>
<!-- Data type -->
<div>
<label for="type-{config.source}" class="mb-1 block text-xs font-medium text-gray-500">数据类型</label>
<select id="type-{config.source}" value={config.type} onchange={onTypeChange}
class="w-full rounded border border-gray-300 px-2 py-1.5 text-sm focus:border-blue-500 focus:ring-1 focus:ring-blue-500">
<label for="type-{config.source}" class="mb-1 block text-xs font-medium text-gray-500"
>数据类型</label
>
<select
id="type-{config.source}"
value={config.type}
onchange={onTypeChange}
class="w-full rounded border border-gray-300 px-2 py-1.5 text-sm focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
>
{#each dataTypes as dt (dt.value)}
<option value={dt.value}>{dt.label}</option>
{/each}
@@ -90,9 +138,14 @@
<!-- Date format (only for date type) -->
{#if config.type === 'date'}
<div>
<label for="format-{config.source}" class="mb-1 block text-xs font-medium text-gray-500">日期格式</label>
<select id="format-{config.source}" bind:value={config.format}
class="w-full rounded border border-gray-300 px-2 py-1.5 text-sm focus:border-blue-500 focus:ring-1 focus:ring-blue-500">
<label for="format-{config.source}" class="mb-1 block text-xs font-medium text-gray-500"
>日期格式</label
>
<select
id="format-{config.source}"
bind:value={config.format}
class="w-full rounded border border-gray-300 px-2 py-1.5 text-sm focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
>
{#each dateFormats as df (df.value)}
<option value={df.value}>{df.label}</option>
{/each}
@@ -100,27 +153,77 @@
</div>
{/if}
<!-- Dictionary Mapping -->
{#if rows}
<DictionaryMapper
config={{
get useDictionary() {
return config.useDictionary ?? false;
},
set useDictionary(v) {
config.useDictionary = v;
},
get valueMapping() {
return config.valueMapping ?? [];
},
set valueMapping(v) {
config.valueMapping = v;
},
get mappingFallback() {
return config.mappingFallback ?? 'keep';
},
set mappingFallback(v) {
config.mappingFallback = v;
},
get mappingCustomValue() {
return config.mappingCustomValue;
},
set mappingCustomValue(v) {
config.mappingCustomValue = v;
}
}}
{rows}
sourceColumn={config.source}
/>
{/if}
<!-- Exclude if empty -->
<div class="flex items-center gap-2">
<input type="checkbox" id="exclude-empty-{config.source}" bind:checked={config.excludeIfEmpty}
class="rounded border-gray-300 text-blue-600 focus:ring-blue-500" />
<label for="exclude-empty-{config.source}" class="text-sm text-gray-600">空值时移除该字段</label>
<input
type="checkbox"
id="exclude-empty-{config.source}"
bind:checked={config.excludeIfEmpty}
class="rounded border-gray-300 text-blue-600 focus:ring-blue-500"
/>
<label for="exclude-empty-{config.source}" class="text-sm text-gray-600"
>空值时移除该字段</label
>
</div>
<!-- Default value -->
{#if !config.excludeIfEmpty}
<div>
<label for="default-{config.source}" class="mb-1 block text-xs font-medium text-gray-500">默认值 (空值时)</label>
<input id="default-{config.source}" type="text" bind:value={config.defaultValue}
<label for="default-{config.source}" class="mb-1 block text-xs font-medium text-gray-500"
>默认值 (空值时)</label
>
<input
id="default-{config.source}"
type="text"
bind:value={config.defaultValue}
class="w-full rounded border border-gray-300 px-2 py-1.5 text-sm focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
placeholder="留空则为 null" />
placeholder="留空则为 null"
/>
</div>
{/if}
<!-- Enable/disable column -->
<div class="flex items-center gap-2 border-t border-gray-100 pt-3">
<input type="checkbox" id="enabled-{config.source}" bind:checked={config.enabled}
class="rounded border-gray-300 text-blue-600 focus:ring-blue-500" />
<input
type="checkbox"
id="enabled-{config.source}"
bind:checked={config.enabled}
class="rounded border-gray-300 text-blue-600 focus:ring-blue-500"
/>
<label for="enabled-{config.source}" class="text-sm text-gray-600">包含此列到输出</label>
</div>
</div>

View File

@@ -0,0 +1,222 @@
<script lang="ts">
import type { ValueMapItem, MappingFallback, RowData } from '$lib/types.js';
import { scanUniqueValues } from '$lib/converter.js';
let {
config = $bindable(),
rows,
sourceColumn
}: {
config: {
useDictionary: boolean;
valueMapping: ValueMapItem[];
mappingFallback: MappingFallback;
mappingCustomValue?: string;
};
rows: RowData[];
sourceColumn: string;
} = $props();
const fallbackOptions: { value: MappingFallback; label: string }[] = [
{ value: 'keep', label: '保留原值' },
{ value: 'null', label: '设为 null' },
{ value: 'custom', label: '自定义值' }
];
let newSourceValue = $state('');
let newTargetValue = $state('');
function enableDictionary() {
config.useDictionary = true;
if (config.valueMapping.length === 0) {
scanColumnValues();
}
}
function disableDictionary() {
config.useDictionary = false;
}
function scanColumnValues() {
const uniqueValues = scanUniqueValues(sourceColumn, rows);
config.valueMapping = uniqueValues.map((v) => ({
source: v,
target: v
}));
}
function addMappingItem() {
if (!newSourceValue) return;
config.valueMapping = [
...config.valueMapping,
{ source: newSourceValue, target: newTargetValue || newSourceValue }
];
newSourceValue = '';
newTargetValue = '';
}
function removeMappingItem(index: number) {
config.valueMapping = config.valueMapping.filter((_, i) => i !== index);
}
function updateTargetValue(index: number, value: string) {
// Create a new array to trigger Svelte 5 reactivity
config.valueMapping = config.valueMapping.map((item, i) =>
i === index
? { ...item, target: parseTargetValue(value) }
: item
);
}
function parseTargetValue(value: string): unknown {
if (value === 'null' || value === '') return null;
if (value === 'true') return true;
if (value === 'false') return false;
if (value === 'undefined') return undefined;
const num = Number(value);
if (!isNaN(num) && value !== '') return num;
if ((value.startsWith('{') && value.endsWith('}')) || (value.startsWith('[') && value.endsWith(']'))) {
try {
return JSON.parse(value);
} catch {
return value;
}
}
return value;
}
function formatTargetValue(value: unknown): string {
if (value === null || value === undefined) return '';
if (typeof value === 'string') return value;
return JSON.stringify(value);
}
</script>
<div class="space-y-2">
<!-- Toggle -->
<div class="flex items-center justify-between border-b border-gray-100 pb-1.5">
<span class="text-xs font-semibold text-gray-700">字典映射</span>
{#if !config.useDictionary}
<button
onclick={enableDictionary}
class="rounded bg-blue-600 px-2 py-0.5 text-xs text-white hover:bg-blue-700"
>
启用
</button>
{:else}
<button
onclick={disableDictionary}
class="rounded border border-gray-300 px-2 py-0.5 text-xs text-gray-600 hover:bg-gray-50"
>
禁用
</button>
{/if}
</div>
{#if config.useDictionary}
<!-- Auto Scan Button -->
<button
onclick={scanColumnValues}
class="w-full rounded border border-blue-600 bg-blue-50 px-2 py-1 text-xs text-blue-600 hover:bg-blue-100"
>
🔄 自动扫描列值
</button>
<!-- Mapping Table -->
{#if config.valueMapping.length > 0}
<div class="max-h-32 overflow-y-auto rounded border border-gray-200">
<table class="w-full text-xs">
<thead class="bg-gray-50 sticky top-0">
<tr>
<th class="px-1.5 py-1 text-left font-medium text-gray-600 text-[10px]"></th>
<th class="px-1.5 py-1 text-left font-medium text-gray-600 text-[10px]">目标</th>
<th class="w-6"></th>
</tr>
</thead>
<tbody>
{#each config.valueMapping as item, index (index)}
<tr class="border-t border-gray-100">
<td class="px-1.5 py-1 text-gray-700 truncate max-w-24" title={String(item.source)}>{String(item.source)}</td>
<td class="px-1.5 py-1">
<input
type="text"
value={formatTargetValue(item.target)}
oninput={(e) => updateTargetValue(index, e.currentTarget.value)}
class="w-full rounded border border-gray-300 px-1 py-0.5 text-xs focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
/>
</td>
<td class="px-0.5 py-1">
<button
onclick={() => removeMappingItem(index)}
class="text-red-500 hover:text-red-700 p-0.5"
title="删除"
>
<svg class="h-3 w-3" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path
stroke-linecap="round"
stroke-linejoin="round"
stroke-width="2"
d="M6 18L18 6M6 6l12 12"
/>
</svg>
</button>
</td>
</tr>
{/each}
</tbody>
</table>
</div>
{:else}
<p class="text-[10px] text-gray-400 text-center">暂无映射项</p>
{/if}
<!-- Manual Add (compact) -->
<div class="flex gap-1">
<input
bind:value={newSourceValue}
type="text"
placeholder="源值"
class="w-20 rounded border border-gray-300 px-1.5 py-1 text-xs focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
/>
<input
bind:value={newTargetValue}
type="text"
placeholder="目标"
class="flex-1 rounded border border-gray-300 px-1.5 py-1 text-xs focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
/>
<button
onclick={addMappingItem}
disabled={!newSourceValue}
class="rounded border border-gray-300 px-2 py-1 text-xs text-gray-600 hover:bg-gray-50 disabled:opacity-50 whitespace-nowrap"
>
+
</button>
</div>
<!-- Fallback Strategy (compact) -->
<div class="flex items-center gap-2">
<span class="text-[10px] text-gray-500 whitespace-nowrap">未匹配:</span>
<select
bind:value={config.mappingFallback}
class="flex-1 rounded border border-gray-300 px-1.5 py-1 text-xs focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
>
{#each fallbackOptions as opt (opt.value)}
<option value={opt.value}>{opt.label}</option>
{/each}
</select>
</div>
<!-- Custom Fallback Value (compact) -->
{#if config.mappingFallback === 'custom'}
<input
bind:value={config.mappingCustomValue}
type="text"
placeholder="自定义默认值 (如: null, true, 0)"
class="w-full rounded border border-gray-300 px-2 py-1 text-xs focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
/>
{/if}
{/if}
</div>

View File

@@ -1,15 +1,23 @@
<script lang="ts">
import type { MappingConfig, RowData } from '$lib/types.js';
import type { MappingConfig, RowData, ApiEnrichmentRule } from '$lib/types.js';
import ColumnConfig from './ColumnConfig.svelte';
let {
headers,
rows,
mappings = $bindable()
mappings = $bindable(),
enrichmentRules = [],
onaddapi,
oneditapi,
ondeleteapi
}: {
headers: string[];
rows: RowData[];
mappings: MappingConfig[];
enrichmentRules?: ApiEnrichmentRule[];
onaddapi?: () => void;
oneditapi?: (index: number) => void;
ondeleteapi?: (index: number) => void;
} = $props();
let activeConfigIndex = $state<number | null>(null);
@@ -37,16 +45,67 @@
<div class="flex h-full flex-col overflow-hidden">
<div class="flex-shrink-0 border-b border-gray-200 bg-gray-50 px-4 py-2">
<div class="flex items-center justify-between">
<h3 class="text-sm font-semibold text-gray-700">
Excel 数据
<span class="ml-2 text-xs font-normal text-gray-400">
{rows.length}× {headers.length}
{#if enrichmentRules.length > 0}
+ {enrichmentRules.length} API 字段
{/if}
{#if rows.length > maxPreviewRows}
(显示前 {maxPreviewRows} 行)
{/if}
</span>
</h3>
{#if onaddapi}
<button
onclick={onaddapi}
class="inline-flex cursor-pointer items-center gap-1 rounded-md bg-purple-50 px-2 py-1 text-xs font-medium text-purple-700 hover:bg-purple-100"
>
<svg class="h-3.5 w-3.5" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M12 4v16m8-8H4" />
</svg>
添加 API 字段
</button>
{/if}
</div>
</div>
<!-- Enrichment rules bar -->
{#if enrichmentRules.length > 0}
<div class="flex flex-shrink-0 flex-wrap items-center gap-2 border-b border-gray-200 bg-purple-50/50 px-4 py-2">
<span class="text-xs font-medium text-purple-600">API 字段:</span>
{#each enrichmentRules as rule, i (i)}
<span class="inline-flex items-center gap-1 rounded-full bg-purple-100 px-2.5 py-0.5 text-xs font-medium text-purple-700">
{rule.target_key}
<span class="text-purple-400">({rule.method})</span>
{#if oneditapi}
<button
onclick={() => oneditapi?.(i)}
class="cursor-pointer text-purple-400 hover:text-purple-600"
title="编辑"
>
<svg class="h-3 w-3" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M11 5H6a2 2 0 00-2 2v11a2 2 0 002 2h11a2 2 0 002-2v-5m-1.414-9.414a2 2 0 112.828 2.828L11.828 15H9v-2.828l8.586-8.586z" />
</svg>
</button>
{/if}
{#if ondeleteapi}
<button
onclick={() => ondeleteapi?.(i)}
class="cursor-pointer text-purple-400 hover:text-red-500"
title="删除"
>
<svg class="h-3 w-3" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M6 18L18 6M6 6l12 12" />
</svg>
</button>
{/if}
</span>
{/each}
</div>
{/if}
<div class="flex-1 overflow-auto">
<table class="w-full text-sm">
@@ -72,10 +131,21 @@
{/if}
</div>
{#if activeConfigIndex === i && mappings[i]}
<ColumnConfig bind:config={mappings[i]} onclose={() => (activeConfigIndex = null)} />
<ColumnConfig bind:config={mappings[i]} {rows} onclose={() => (activeConfigIndex = null)} />
{/if}
</th>
{/each}
<!-- API enrichment columns (virtual) -->
{#each enrichmentRules as rule (rule.target_key)}
<th class="border-b border-r border-purple-200 bg-purple-50 px-3 py-2 text-left font-medium text-purple-600">
<div class="flex items-center gap-1">
<svg class="h-3 w-3 flex-shrink-0" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M13 10V3L4 14h7v7l9-11h-7z" />
</svg>
<span class="truncate" title="{rule.target_key} (API)">{rule.target_key}</span>
</div>
</th>
{/each}
</tr>
</thead>
<tbody>
@@ -86,6 +156,12 @@
<span class="block max-w-[200px] truncate">{displayValue(row[header])}</span>
</td>
{/each}
<!-- API placeholder cells -->
{#each enrichmentRules as _ (_.target_key)}
<td class="border-b border-r border-purple-100 bg-purple-50/30 px-3 py-1.5 text-xs italic text-purple-400">
[Pending API Fetch]
</td>
{/each}
</tr>
{/each}
</tbody>

View File

@@ -13,6 +13,7 @@
defaultOpen?: boolean;
} = $props();
// svelte-ignore state_referenced_locally — intentionally using initial value only
let open = $state(defaultOpen);
const valueType = $derived(

View File

@@ -0,0 +1,88 @@
<script lang="ts">
import type { SubmissionConfig } from '$lib/types.js';
let {
config = $bindable(),
onclose
}: {
config: SubmissionConfig;
onclose: () => void;
} = $props();
let localConfig = $state<SubmissionConfig>({ ...config });
function save() {
config.target_url = localConfig.target_url;
config.method = localConfig.method;
config.batch_size = localConfig.batch_size;
onclose();
}
</script>
<!-- svelte-ignore a11y_no_static_element_interactions a11y_click_events_have_key_events -->
<!-- svelte-ignore a11y_click_events_have_key_events -->
<div class="fixed inset-0 z-50 flex items-center justify-center bg-black/40" onclick={onclose} role="presentation">
<!-- svelte-ignore a11y_no_static_element_interactions a11y_click_events_have_key_events -->
<!-- svelte-ignore a11y_click_events_have_key_events -->
<div class="w-full max-w-md rounded-lg bg-white p-6 shadow-xl" onclick={(e) => e.stopPropagation()}>
<div class="mb-4 flex items-center justify-between">
<h3 class="text-lg font-semibold text-gray-800">提交设置</h3>
<button onclick={onclose} aria-label="关闭" class="cursor-pointer text-gray-400 hover:text-gray-600">
<svg class="h-5 w-5" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M6 18L18 6M6 6l12 12" />
</svg>
</button>
</div>
<div class="space-y-4">
<div>
<label for="target-url" class="mb-1 block text-sm font-medium text-gray-700">Target URL</label>
<input
id="target-url"
type="text"
bind:value={localConfig.target_url}
placeholder="https://api.db.com/bulk-insert"
class="w-full rounded-md border border-gray-300 px-3 py-2 text-sm focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
/>
</div>
<div>
<label for="method" class="mb-1 block text-sm font-medium text-gray-700">Method</label>
<select
id="method"
bind:value={localConfig.method}
class="w-full rounded-md border border-gray-300 px-3 py-2 text-sm focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
>
<option value="POST">POST</option>
<option value="PUT">PUT</option>
</select>
</div>
<div>
<label for="batch-size" class="mb-1 block text-sm font-medium text-gray-700">Batch Size</label>
<input
id="batch-size"
type="number"
bind:value={localConfig.batch_size}
min="1"
class="w-full rounded-md border border-gray-300 px-3 py-2 text-sm focus:border-blue-500 focus:ring-1 focus:ring-blue-500"
/>
</div>
</div>
<div class="mt-6 flex justify-end gap-2">
<button
onclick={onclose}
class="cursor-pointer rounded-md border border-gray-300 px-4 py-2 text-sm text-gray-700 hover:bg-gray-50"
>
取消
</button>
<button
onclick={save}
class="cursor-pointer rounded-md bg-blue-600 px-4 py-2 text-sm font-medium text-white hover:bg-blue-700"
>
保存
</button>
</div>
</div>
</div>

View File

@@ -1,5 +1,16 @@
import dayjs from 'dayjs';
import type { MappingConfig, RowData, DataType, MappingTemplate } from './types.js';
import type {
MappingConfig,
RowData,
DataType,
MappingTemplate,
ApiEnrichmentRule,
SubmissionConfig,
StaticRule,
JobBundle,
ValueMapItem,
MappingFallback
} from './types.js';
/**
* Check if a value is considered empty.
@@ -8,31 +19,107 @@ function isEmpty(value: unknown): boolean {
return value === undefined || value === null || value === '';
}
/**
* Parse a target value string into the appropriate type.
* Intelligently detects boolean, number, null, and JSON.
*/
function parseTargetValue(value: string): unknown {
if (value === 'null' || value === '') return null;
if (value === 'true') return true;
if (value === 'false') return false;
if (value === 'undefined') return undefined;
// Try parsing as number
const num = Number(value);
if (!isNaN(num) && value !== '') return num;
// Try parsing as JSON object/array
if ((value.startsWith('{') && value.endsWith('}')) || (value.startsWith('[') && value.endsWith(']'))) {
try {
return JSON.parse(value);
} catch {
// Not valid JSON, return as string
}
}
return value;
}
/**
* Apply dictionary mapping to a value.
* Returns the mapped value or applies the fallback strategy.
*/
function applyDictionaryMapping(
value: unknown,
valueMapping: ValueMapItem[],
fallback: MappingFallback,
customValue?: string
): unknown {
if (value === undefined || value === null) {
return fallback === 'null' ? null : value;
}
const normalizedValue = typeof value === 'object' ? JSON.stringify(value) : String(value);
// Try to find exact match in mapping
const mappedItem = valueMapping.find(
(item) => String(item.source) === normalizedValue
);
if (mappedItem) {
return mappedItem.target;
}
// Apply fallback strategy
switch (fallback) {
case 'keep':
return value;
case 'null':
return null;
case 'custom':
return customValue !== undefined ? parseTargetValue(customValue) : null;
default:
return value;
}
}
/**
* Convert a raw cell value to the specified data type.
*/
function convertValue(value: unknown, type: DataType, format?: string): unknown {
function convertValue(
value: unknown,
type: DataType,
format?: string,
mapping?: { enabled: boolean; items: ValueMapItem[]; fallback: MappingFallback; customValue?: string }
): unknown {
if (isEmpty(value)) return undefined;
// Apply dictionary mapping first (before type conversion)
let processedValue = value;
if (mapping?.enabled && mapping.items.length > 0) {
processedValue = applyDictionaryMapping(value, mapping.items, mapping.fallback, mapping.customValue);
if (processedValue === null && mapping.fallback === 'null') return null;
}
switch (type) {
case 'number': {
const num = Number(value);
return isNaN(num) ? value : num;
const num = Number(processedValue);
return isNaN(num) ? processedValue : num;
}
case 'boolean': {
if (typeof value === 'boolean') return value;
const str = String(value).toLowerCase().trim();
if (typeof processedValue === 'boolean') return processedValue;
const str = String(processedValue).toLowerCase().trim();
if (['true', '1', 'yes', '是'].includes(str)) return true;
if (['false', '0', 'no', '否'].includes(str)) return false;
return Boolean(value);
return Boolean(processedValue);
}
case 'date': {
return formatDate(value, format);
return formatDate(processedValue, format);
}
case 'string':
default:
if (value instanceof Date) return dayjs(value).format('YYYY-MM-DD HH:mm:ss');
return String(value);
if (processedValue instanceof Date) return dayjs(processedValue).format('YYYY-MM-DD HH:mm:ss');
return String(processedValue);
}
}
@@ -110,9 +197,29 @@ export function convertData(
const finalValue = isEmptyVal
? (mapping.defaultValue !== undefined && mapping.defaultValue !== ''
? convertValue(mapping.defaultValue, mapping.type, mapping.format)
? convertValue(
mapping.defaultValue,
mapping.type,
mapping.format,
mapping.useDictionary ? {
enabled: true,
items: mapping.valueMapping ?? [],
fallback: mapping.mappingFallback ?? 'keep',
customValue: mapping.mappingCustomValue
} : undefined
)
: null)
: convertValue(rawValue, mapping.type, mapping.format);
: convertValue(
rawValue,
mapping.type,
mapping.format,
mapping.useDictionary ? {
enabled: true,
items: mapping.valueMapping ?? [],
fallback: mapping.mappingFallback ?? 'keep',
customValue: mapping.mappingCustomValue
} : undefined
);
setNested(obj, mapping.target, finalValue);
}
@@ -143,6 +250,27 @@ export function createDefaultMappings(headers: string[], rows?: RowData[]): Mapp
/**
* Detect the data type of a column by sampling its values.
*/
/**
* Scan unique values from a column for auto-filling dictionary mapping.
* Limits to first 1000 rows for performance.
*/
export function scanUniqueValues(header: string, rows: RowData[]): (string | number)[] {
const MAX_ROWS = 1000;
const sample = rows.slice(0, MAX_ROWS);
const uniqueValues = new Set<string | number>();
for (const row of sample) {
const val = row[header];
if (val !== undefined && val !== null && val !== '') {
const normalized = typeof val === 'object' ? JSON.stringify(val) : (val as string | number);
uniqueValues.add(normalized);
}
}
return Array.from(uniqueValues).sort();
}
function detectColumnType(header: string, rows: RowData[]): DataType {
const sample = rows.slice(0, 20);
let dateCount = 0;
@@ -177,7 +305,12 @@ export function applyTemplate(
type: tmpl.type,
format: tmpl.format,
excludeIfEmpty: tmpl.excludeIfEmpty,
defaultValue: tmpl.defaultValue ?? ''
defaultValue: tmpl.defaultValue ?? '',
// v2.1: Apply dictionary mapping settings
useDictionary: tmpl.useDictionary,
valueMapping: tmpl.valueMapping,
mappingFallback: tmpl.mappingFallback,
mappingCustomValue: tmpl.mappingCustomValue
};
}
return mapping;
@@ -188,10 +321,68 @@ export function applyTemplate(
* Export current mappings as a template.
*/
export function exportTemplate(mappings: MappingConfig[]): MappingTemplate {
return mappings.map(({ source, target, type, format, excludeIfEmpty, defaultValue }) => {
return mappings.map(({ source, target, type, format, excludeIfEmpty, defaultValue, useDictionary, valueMapping, mappingFallback, mappingCustomValue }) => {
const entry: MappingTemplate[number] = { source, target, type, excludeIfEmpty };
if (type === 'date' && format) entry.format = format;
if (defaultValue) entry.defaultValue = defaultValue;
// v2.1: Export dictionary mapping settings
if (useDictionary) entry.useDictionary = true;
if (valueMapping && valueMapping.length > 0) entry.valueMapping = valueMapping;
if (mappingFallback) entry.mappingFallback = mappingFallback;
if (mappingCustomValue) entry.mappingCustomValue = mappingCustomValue;
return entry;
});
}
/**
* Convert enabled MappingConfigs to StaticRule format for Job Bundle.
*/
function toStaticRules(mappings: MappingConfig[]): StaticRule[] {
return mappings
.filter((m) => m.enabled)
.map((m) => {
const rule: StaticRule = {
type: 'static',
source: m.source,
target: m.target,
dataType: m.type
};
if (m.type === 'date' && m.format) rule.format = m.format;
// v2.1: Include dictionary mapping properties
if (m.useDictionary && m.valueMapping && m.valueMapping.length > 0) {
rule.useDictionary = true;
rule.valueMapping = m.valueMapping;
rule.mappingFallback = m.mappingFallback ?? 'keep';
if (m.mappingFallback === 'custom') {
rule.mappingCustomValue = m.mappingCustomValue;
}
}
return rule;
});
}
/**
* Generate a complete Job Bundle for export.
*/
export function generateJobBundle(
rows: RowData[],
mappings: MappingConfig[],
enrichmentRules: ApiEnrichmentRule[],
submissionConfig: SubmissionConfig
): JobBundle {
const sourceData = convertData(rows, mappings);
const staticRules = toStaticRules(mappings);
return {
meta: {
version: '1.0.0',
generated_at: new Date().toISOString()
},
config: {
static_rules: staticRules,
enrichment_rules: enrichmentRules,
submission: submissionConfig
},
source_data: sourceData
};
}

View File

@@ -1,5 +1,14 @@
export type DataType = 'string' | 'number' | 'boolean' | 'date';
/** Mapping fallback strategy when value is not found in dictionary */
export type MappingFallback = 'keep' | 'null' | 'custom';
/** Dictionary mapping item - maps source value to target value */
export interface ValueMapItem {
source: string | number;
target: unknown;
}
export type DateFormat =
| 'YYYY-MM-DD'
| 'YYYY/MM/DD'
@@ -23,6 +32,14 @@ export interface MappingConfig {
defaultValue?: string;
/** Whether this column is included in output */
enabled: boolean;
/** v2.1: Enable dictionary/value mapping for this column */
useDictionary?: boolean;
/** v2.1: Value mapping dictionary */
valueMapping?: ValueMapItem[];
/** v2.1: Fallback strategy when value not found in mapping */
mappingFallback?: MappingFallback;
/** v2.1: Custom fallback value (only used when mappingFallback is 'custom') */
mappingCustomValue?: string;
}
/** A single row of raw Excel data, keyed by original header */
@@ -31,5 +48,55 @@ export type RowData = Record<string, unknown>;
/** Template file structure for import/export */
export type MappingTemplate = Pick<
MappingConfig,
'source' | 'target' | 'type' | 'format' | 'excludeIfEmpty' | 'defaultValue'
'source' | 'target' | 'type' | 'format' | 'excludeIfEmpty' | 'defaultValue' | 'useDictionary' | 'valueMapping' | 'mappingFallback' | 'mappingCustomValue'
>[];
/** Static mapping rule in Job Bundle output */
export interface StaticRule {
type: 'static';
source: string;
target: string;
dataType: DataType;
format?: string;
/** v2.1: Enable dictionary/value mapping */
useDictionary?: boolean;
/** v2.1: Value mapping dictionary */
valueMapping?: ValueMapItem[];
/** v2.1: Fallback strategy when value not found in mapping */
mappingFallback?: MappingFallback;
/** v2.1: Custom fallback value */
mappingCustomValue?: string;
}
/** Dynamic API enrichment rule */
export interface ApiEnrichmentRule {
type: 'api_fetch';
target_key: string;
url_template: string;
method: 'GET' | 'POST';
headers?: Record<string, string>;
body_template?: string;
response_path: string;
fallback_value?: unknown;
}
/** Submission configuration for final data push */
export interface SubmissionConfig {
target_url: string;
method: 'POST' | 'PUT';
batch_size: number;
}
/** Final exported Job Bundle structure */
export interface JobBundle {
meta: {
version: string;
generated_at: string;
};
config: {
static_rules: StaticRule[];
enrichment_rules: ApiEnrichmentRule[];
submission: SubmissionConfig;
};
source_data: Record<string, unknown>[];
}

View File

@@ -1,9 +1,11 @@
<script lang="ts">
import type { MappingConfig, RowData, MappingTemplate } from '$lib/types.js';
import type { MappingConfig, RowData, MappingTemplate, ApiEnrichmentRule, SubmissionConfig } from '$lib/types.js';
import { parseExcelFile } from '$lib/excel.js';
import { convertData, createDefaultMappings, applyTemplate, exportTemplate } from '$lib/converter.js';
import { convertData, createDefaultMappings, applyTemplate, exportTemplate, generateJobBundle } from '$lib/converter.js';
import ExcelTable from '$lib/components/ExcelTable.svelte';
import JsonPreview from '$lib/components/JsonPreview.svelte';
import ApiConfigModal from '$lib/components/ApiConfigModal.svelte';
import SubmissionSettings from '$lib/components/SubmissionSettings.svelte';
// State
let headers = $state<string[]>([]);
@@ -14,6 +16,17 @@
let isDragOver = $state(false);
let copySuccess = $state(false);
// ETL state
let enrichmentRules = $state<ApiEnrichmentRule[]>([]);
let submissionConfig = $state<SubmissionConfig>({
target_url: '',
method: 'POST',
batch_size: 50
});
let showApiConfig = $state(false);
let showSubmissionSettings = $state(false);
let editingRuleIndex = $state<number | null>(null);
// Split panel
let splitPercent = $state(50);
let jsonCollapsed = $state(false);
@@ -50,28 +63,41 @@
let debounceTimer: ReturnType<typeof setTimeout>;
$effect(() => {
// $state.snapshot forces deep read of all nested properties,
// so changes to e.g. mappings[i].target will trigger this effect
const snapshotMappings = $state.snapshot(mappings);
const snapshotRules = $state.snapshot(enrichmentRules);
const currentRows = rows;
const dataReady = hasData;
clearTimeout(debounceTimer);
debounceTimer = setTimeout(() => {
convertedJson = dataReady ? convertData(currentRows, snapshotMappings) : [];
if (dataReady) {
const base = convertData(currentRows, snapshotMappings);
// Add API placeholders for preview
if (snapshotRules.length > 0) {
convertedJson = base.map((row) => {
const enriched = { ...row };
for (const rule of snapshotRules) {
enriched[rule.target_key] = '[Pending API Fetch]';
}
return enriched;
});
} else {
convertedJson = base;
}
} else {
convertedJson = [];
}
}, 150);
});
// Manual JSON editing — overrides converted output until next mapping/data change
// Manual JSON editing
let manualJson = $state<string | null>(null);
const jsonString = $derived(manualJson ?? JSON.stringify(convertedJson, null, 2));
function onJsonEdited(edited: string) {
manualJson = edited;
}
// Clear manual override when conversion changes
$effect(() => {
convertedJson;
manualJson = null;
@@ -179,6 +205,43 @@
errorMessage = '复制失败,请手动复制';
}
}
// Job Bundle export
function handleExportJobBundle() {
const bundle = generateJobBundle(rows, mappings, enrichmentRules, submissionConfig);
const blob = new Blob([JSON.stringify(bundle, null, 2)], { type: 'application/json' });
const url = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = url;
a.download = fileName ? fileName.replace(/\.[^.]+$/, '_job_bundle.json') : 'job_bundle.json';
a.click();
URL.revokeObjectURL(url);
}
// API enrichment rules management
function openAddApiConfig() {
editingRuleIndex = null;
showApiConfig = true;
}
function openEditApiConfig(index: number) {
editingRuleIndex = index;
showApiConfig = true;
}
function deleteApiRule(index: number) {
enrichmentRules = enrichmentRules.filter((_, i) => i !== index);
}
function handleSaveApiRule(rule: ApiEnrichmentRule) {
if (editingRuleIndex !== null) {
enrichmentRules = enrichmentRules.map((r, i) => (i === editingRuleIndex ? rule : r));
} else {
enrichmentRules = [...enrichmentRules, rule];
}
showApiConfig = false;
editingRuleIndex = null;
}
</script>
<div
@@ -231,8 +294,37 @@
导出配置
</button>
<div class="mx-2 h-6 w-px bg-gray-200"></div>
<!-- Submission Settings -->
<button
onclick={() => (showSubmissionSettings = true)}
disabled={!hasData}
class="inline-flex items-center gap-1 rounded-md border border-gray-300 bg-white px-3 py-1.5 text-sm text-gray-700 hover:bg-gray-50 disabled:cursor-not-allowed disabled:opacity-50 cursor-pointer"
>
<svg class="h-4 w-4" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M10.325 4.317c.426-1.756 2.924-1.756 3.35 0a1.724 1.724 0 002.573 1.066c1.543-.94 3.31.826 2.37 2.37a1.724 1.724 0 001.066 2.573c1.756.426 1.756 2.924 0 3.35a1.724 1.724 0 00-1.066 2.573c.94 1.543-.826 3.31-2.37 2.37a1.724 1.724 0 00-2.573 1.066c-.426 1.756-2.924 1.756-3.35 0a1.724 1.724 0 00-2.573-1.066c-1.543.94-3.31-.826-2.37-2.37a1.724 1.724 0 00-1.066-2.573c-1.756-.426-1.756-2.924 0-3.35a1.724 1.724 0 001.066-2.573c-.94-1.543.826-3.31 2.37-2.37.996.608 2.296.07 2.572-1.065z" />
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M15 12a3 3 0 11-6 0 3 3 0 016 0z" />
</svg>
提交设置
</button>
<div class="flex-1"></div>
<!-- Export Job Bundle -->
<button
onclick={handleExportJobBundle}
disabled={!hasData}
class="inline-flex items-center gap-1 rounded-md bg-emerald-600 px-3 py-1.5 text-sm font-medium text-white hover:bg-emerald-700 disabled:cursor-not-allowed disabled:opacity-50 cursor-pointer"
>
<svg class="h-4 w-4" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M20 7l-8-4-8 4m16 0l-8 4m8-4v10l-8 4m0-10L4 7m8 4v10M4 7v10l8 4" />
</svg>
导出任务包
</button>
<div class="mx-1 h-6 w-px bg-gray-200"></div>
<!-- JSON operations -->
<button
onclick={handleDownloadJson}
@@ -261,6 +353,19 @@
复制 JSON
{/if}
</button>
<!-- GitHub -->
<a
href="https://github.com/meowrain/Excel2JSON"
target="_blank"
rel="noopener noreferrer"
class="ml-1 inline-flex items-center rounded-md border border-gray-300 bg-white p-1.5 text-gray-600 hover:bg-gray-50 hover:text-gray-900"
title="GitHub"
>
<svg class="h-5 w-5" fill="currentColor" viewBox="0 0 24 24">
<path d="M12 .297c-6.63 0-12 5.373-12 12 0 5.303 3.438 9.8 8.205 11.385.6.113.82-.258.82-.577 0-.285-.01-1.04-.015-2.04-3.338.724-4.042-1.61-4.042-1.61C4.422 18.07 3.633 17.7 3.633 17.7c-1.087-.744.084-.729.084-.729 1.205.084 1.838 1.236 1.838 1.236 1.07 1.835 2.809 1.305 3.495.998.108-.776.417-1.305.76-1.605-2.665-.3-5.466-1.332-5.466-5.93 0-1.31.465-2.38 1.235-3.22-.135-.303-.54-1.523.105-3.176 0 0 1.005-.322 3.3 1.23.96-.267 1.98-.399 3-.405 1.02.006 2.04.138 3 .405 2.28-1.552 3.285-1.23 3.285-1.23.645 1.653.24 2.873.12 3.176.765.84 1.23 1.91 1.23 3.22 0 4.61-2.805 5.625-5.475 5.92.42.36.81 1.096.81 2.22 0 1.606-.015 2.896-.015 3.286 0 .315.21.69.825.57C20.565 22.092 24 17.592 24 12.297c0-6.627-5.373-12-12-12" />
</svg>
</a>
</header>
<!-- Error message -->
@@ -289,7 +394,15 @@
>
<!-- Left: Excel Table -->
<div class="overflow-hidden" style="width: {jsonCollapsed ? '100%' : `${splitPercent}%`}">
<ExcelTable {headers} {rows} bind:mappings />
<ExcelTable
{headers}
{rows}
bind:mappings
{enrichmentRules}
onaddapi={openAddApiConfig}
oneditapi={openEditApiConfig}
ondeleteapi={deleteApiRule}
/>
</div>
{#if !jsonCollapsed}
@@ -356,3 +469,20 @@
</div>
{/if}
</div>
<!-- Modals -->
{#if showApiConfig}
<ApiConfigModal
rule={editingRuleIndex !== null ? enrichmentRules[editingRuleIndex] : undefined}
{headers}
onsave={handleSaveApiRule}
onclose={() => { showApiConfig = false; editingRuleIndex = null; }}
/>
{/if}
{#if showSubmissionSettings}
<SubmissionSettings
bind:config={submissionConfig}
onclose={() => (showSubmissionSettings = false)}
/>
{/if}

View File

@@ -1,37 +1,8 @@
import devtoolsJson from 'vite-plugin-devtools-json';
import tailwindcss from '@tailwindcss/vite';
import { defineConfig } from 'vitest/config';
import { playwright } from '@vitest/browser-playwright';
import { defineConfig } from 'vite';
import { sveltekit } from '@sveltejs/kit/vite';
export default defineConfig({
plugins: [tailwindcss(), sveltekit(), devtoolsJson()],
test: {
expect: { requireAssertions: true },
projects: [
{
extends: './vite.config.ts',
test: {
name: 'client',
browser: {
enabled: true,
provider: playwright(),
instances: [{ browser: 'chromium', headless: true }]
},
include: ['src/**/*.svelte.{test,spec}.{js,ts}'],
exclude: ['src/lib/server/**']
}
},
{
extends: './vite.config.ts',
test: {
name: 'server',
environment: 'node',
include: ['src/**/*.{test,spec}.{js,ts}'],
exclude: ['src/**/*.svelte.{test,spec}.{js,ts}']
}
}
]
}
plugins: [tailwindcss(), sveltekit(), devtoolsJson()]
});

32
vitest.config.ts Normal file
View File

@@ -0,0 +1,32 @@
import { defineConfig } from 'vitest/config';
import { playwright } from '@vitest/browser-playwright';
export default defineConfig({
test: {
expect: { requireAssertions: true },
projects: [
{
extends: './vite.config.ts',
test: {
name: 'client',
browser: {
enabled: true,
provider: playwright(),
instances: [{ browser: 'chromium', headless: true }]
},
include: ['src/**/*.svelte.{test,spec}.{js,ts}'],
exclude: ['src/lib/server/**']
}
},
{
extends: './vite.config.ts',
test: {
name: 'server',
environment: 'node',
include: ['src/**/*.{test,spec}.{js,ts}'],
exclude: ['src/**/*.svelte.{test,spec}.{js,ts}']
}
}
]
}
});