后台自动核对REVIEW状态的项目

This commit is contained in:
lxp
2026-10-28 10:37:02 +08:00
parent c01ef63482
commit 4e7d11d586
23 changed files with 889 additions and 56 deletions
+2
View File
@@ -624,6 +624,8 @@ interface CommitInput {
版本冲突返回 409 `CONFLICT`。保留未保存的编辑内容,提示用户刷新并比对差异,不能直接换成最新 SHA 自动覆盖。
普通文件提交结果不明时由后台自动核对,核对成功后自动恢复写入,刷新工作区即可继续。请求与响应协议不变,不需要前端接入新的轮询接口。超时不代表提交失败,暂勿自动重传或自动发布。
### 6.4 多文件上传(保留目录)
`POST /admin-api/projects/{projectCode}/uploads/commit`,`multipart/form-data` → `CommitView`。
+30
View File
@@ -85,3 +85,33 @@ ALTER TABLE project MODIFY COLUMN publish_status VARCHAR(16) NOT NULL DEFAULT 'U
```
旧版七个字段及其索引可以暂留,代码已不再读写;确认备份及来源回填完成后再清理。特别是只有旧 Excel 批次、没有后台任务的历史项目,其来源需保留原批次清单或另行归档,不能直接删除唯一来源信息。本次不对实际数据库执行增删字段操作。
## 新建普通文件写入操作记录表(本次上线前执行)
只新增此表,不需要修改项目表。由运维手工执行;应用不自动迁库。时间全部为 UTC。
```sql
CREATE TABLE repository_write_operation (
operation_code CHAR(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL COMMENT '写入操作业务编码,同时作为幂等键',
project_code CHAR(32) NOT NULL COMMENT '项目业务编码',
repository_id BIGINT NOT NULL COMMENT '操作发起时Gitea仓库内部标识,仅核验使用',
repository_owner VARCHAR(255) NOT NULL COMMENT '操作发起时仓库所属账号',
repository_name VARCHAR(255) NOT NULL COMMENT '操作发起时仓库名称',
base_commit_sha VARCHAR(64) NOT NULL COMMENT '本次写入的基准提交SHA',
request_digest CHAR(64) NOT NULL COMMENT '请求内容SHA256摘要,不保存文件正文',
commit_message VARCHAR(1000) NOT NULL COMMENT '原始提交说明,不包含系统凭据尾注',
status VARCHAR(16) NOT NULL COMMENT '操作状态 PROCESSING处理中 NEEDS_REVIEW超时待确认 SUCCEEDED已成功 FAILED明确失败',
result_commit_sha VARCHAR(64) NULL COMMENT '确认成功的提交SHA',
failure_status INT NULL COMMENT '明确失败时原HTTP状态码',
last_error_code VARCHAR(64) NULL COMMENT '最近核对错误编码,不保存异常正文或凭证',
check_count INT NOT NULL DEFAULT 0 COMMENT '后台已执行核对次数',
next_check_at DATETIME(3) NOT NULL COMMENT '下次只读核对时间UTC',
created_at DATETIME(3) NOT NULL COMMENT '创建时间UTC',
updated_at DATETIME(3) NOT NULL COMMENT '最近更新时间UTC',
PRIMARY KEY (operation_code),
KEY idx_write_project_status (project_code, status, created_at),
KEY idx_write_next_check (status, next_check_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci COMMENT='普通文件写入幂等凭据及自动核对记录';
```
已完成的操作记录仍用于幂等重放,不要在客户端可能重试期间删除。历史 REVIEW 没有操作记录,不会凭最新 SHA 自动判成功;需核对实际结果后按原维护流程处理。迁移、Fork 继续使用原流程。
+10 -6
View File
@@ -4,7 +4,7 @@
## 部署与初始化
1. 备份数据库、暂停写入、排空旧进程。已有数据库手工执行 [增量 SQL](admin-database.md),新数据库先执行 [基础四表 SQL](database.md),再创建迁移任务表。应用不自动执行 SQL。社区端也会读取迁移任务,因此暂不启用管理端时仍需创建该表。
1. 备份数据库、暂停写入、排空旧进程。已有数据库手工执行 [增量 SQL](admin-database.md),新数据库先执行 [基础四表 SQL](database.md),再创建迁移任务表和普通文件写入操作记录表。应用不自动执行 SQL。社区端也会读取这两张表,因此暂不启用管理端时仍需创建。
2. 更新社区端、管理端和 Excel 导入工具,确保全部写入端使用数据库共享锁。
3. 在 `.env` 或部署环境中提供现有 `DB_*`、`GITEA_*`、`OSS_*`;后台默认关闭 AI,OSS 也需显式启用。另需提供 `REDIS_*` 连接变量,默认环境为 `test`,正式部署显式设置 `SPRING_PROFILES_ACTIVE=prod`。真实 `.env` 不提交,不用示例覆盖已有凭据。
4. 选择一个已有、正常的普通用户,执行初始化命令,交互输入两次管理员密码。密码至少 12 字符、UTF-8 最多 72 字节,不会写入命令参数或日志。该命令只初始化 USER,不用于重置已有管理员密码;成功后进程退出。
@@ -49,7 +49,7 @@ project 表只新增 moderation_reason 和 repo_write_status,来源标识、
全部写入端按项目编码取得同一 MySQL 命名锁,使用固定连接取得、持有和释放,支持内部重入。锁连接池每应用最多 8 个连接,另有普通业务连接池,需要预留数据库容量。远端请求期间不持有数据库长事务;写入前持久化状态,结果不明时重启后仍阻断写入。
普通文件上传的写入状态依次为 `IDLE → WRITING → IDLE`,最终恢复空闲后记录“仓库写入完成”日志。Gitea 已提交但响应丢失、状态回写失败等未知结果进入 `REVIEW`;即使 main 已有最新提交,发布仍会被阻断。上传提交前或发布前的 Gitea 读取失败应恢复 `IDLE`,异常传递保留远端结果确定性,避免把只读失败误标为未知写入。历史 `REVIEW` 仍需按实际操作结果人工核对恢复。
普通文件上传和编辑使用 repository_write_operation 持久化幂等凭据。正常为 `IDLE → WRITING → IDLE`;响应丢失先尝试只读匹配凭据,匹配成功仍返回 200。未知结果持久化 REVIEW,保持原有错误响应,由后台每 15 秒核对,10 分钟后改为每分钟继续核对。核对成功自动恢复 IDLE,不自动发布。提交前或发布前的 Gitea 读取失败保持 IDLE,不能误标为未知写入。操作编码由后端自动生成,前端不需要改造。
上线前确认目标 Gitea 已开启迁移、HTTP Git、LFS,技术账号具有组织仓库写权限。GitHub 预检优先读取 API;受限时严格核对公开网页身份、可见性和默认分支,无法确认时失败。网页回退拿不到的简介、许可证可能为空,需要管理员检查原仓库文件。
@@ -74,13 +74,17 @@ project 表只新增 moderation_reason 和 repo_write_status,来源标识、
逻辑删除、撤回和下架不会物理删除 Gitea 仓库或 OSS 对象;已公开 OSS 链接沿用现有生命周期。
### 普通项目人工核对后的写入恢复
### 普通项目后台自动核对
普通项目同时使用数据库 `repo_write_status` 和各写入进程内的待核对集合。只把数据库改为 `IDLE`,已有内存标记会在下一次提交或发布时重新写回 `REVIEW`;仅重启进程也无法解除数据库中保留的 `REVIEW` 或 `WRITING`。启动后首次请求若发现数据库未空闲,又会建立本进程的待核对标记。
上线顺序:手工执行 [操作表 SQL](admin-database.md#新建普通文件写入操作记录表本次上线前执行) → 排空旧写入进程 → 同时升级社区 API 与管理 API。接口沿用原有请求及成功/错误响应,不新增 202、前端结果查询或在线恢复接口。前端不需要接入新协议。
先按项目编码核对 Gitea 仓库身份、预期文件和提交历史,确认上次远端写入已结束。普通项目当前没有在线恢复接口;核对完成后停止社区 API、管理 API 及仍在运行的导入写入进程,再仅针对已核实的项目恢复数据库为 `IDLE`,最后启动上述服务。恢复后刷新目录并由管理员明确发布,恢复本身不更改发布状态或远端文件。多个进程需全部处理,避免仍持有旧标记的进程再次写回 `REVIEW`。
后台任务在两个 API 应用中默认启动,由数据库项目命名锁避免重复核对;在独立线程执行,避免阻塞迁移任务调度。`vibemake.repository-write.worker-enabled=false` 可停用本实例的核对消费,但必须至少保留一个启用的实例。操作记录在远端写入前落盘,进程重启后按 next_check_at 继续;结果未知只查不重发。每次扫描 main 可达的最近至多 1000 个提交,精确校验操作尾注、请求摘要、唯一父提交及仓库身份;超范围、读取失败或不匹配均保留未知,不自动判失败。
写入拦截日志同时记录 `persistedWriteStatus` 和 `memoryUncertain`。`persistedWriteStatus=IDLE memoryUncertain=true` 表示数据库已恢复、本进程仍保留待核对标记;`memoryUncertain=false` 且数据库非 `IDLE` 表示本次由持久化状态阻断。这两种拦截均发生在本次业务写入和发布 SHA 校验之前,不表示本次请求已调用 Gitea 写入。
前 10 分钟每 15 秒核对,超过 10 分钟改为每分钟继续核对,不强制解除保护。使用 projectCode、operationCode、baseCommitSha、commitSha、checkCount、errorCode 关联日志;原始请求正文和认证信息不进入操作表。操作状态 PROCESSING 为处理中,NEEDS_REVIEW 为超过 10 分钟仍未确认,SUCCEEDED 为确认成功,FAILED 为明确拒绝。操作终态与 IDLE 在同一事务确认;其他实例下次取得共享锁时按数据库结果清除旧内存标记,无需重启。
上传或编辑超时后如果立即查到凭据,原接口直接返回 200;暂时不能确认时仍返回原错误,不会把本次上传显示为成功。后台确认后自动恢复写入,管理员刷新工作区后即可继续。未确认期间禁止新的远端写入,重试不自动重发 Gitea POST。恢复不会自动发布。
历史 REVIEW/遗留 WRITING 没有操作记录时不做自动猜测,仍需核对仓库身份、文件、历史和上次操作是否已结束,再按原维护流程处理。迁移和 Fork 继续使用原专门核验流程。本阶段暂不提供管理员在线恢复入口。
## 验证
+1 -1
View File
@@ -1,6 +1,6 @@
# 数据库初始化
业务数据库名为 `vibemake`,只包含以下四张表。由人工执行;应用不运行迁移或自动建表。以下 SQL 面向 MySQL 8.0.13 及以上(URDF JSON 默认值使用表达式)。
业务数据库名为 `vibemake`,基础业务包含以下四张表;完整部署还需执行 [管理端增量 SQL](admin-database.md),新增迁移任务与仓库写入操作表。由人工执行;应用不运行迁移或自动建表。以下 SQL 面向 MySQL 8.0.13 及以上(URDF JSON 默认值使用表达式)。
```sql
CREATE DATABASE IF NOT EXISTS vibemake CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
+2
View File
@@ -841,6 +841,8 @@ entries 仅包含当前目录的直接子项,目录在前,名称升序。展
如果 expectedCommitSha 已不是最新 main,返回 409 CONFLICT。前端保留当前未提交输入,重新读取工作区版本,让用户处理差异后再提交;不要只替换 SHA 后自动覆盖内容。浏览历史版本时应使用只读模式,回到最新版本后再编辑。
普通文件提交结果不明时由后台自动核对,核对成功后自动恢复写入,刷新工作区即可继续。请求与响应协议不变,不需要前端接入新的轮询接口。超时不代表提交失败,暂勿自动重传或自动发布。
### 上传并提交
`POST /api/workspace/projects/{projectCode}/uploads/commit`,需要登录,使用 FormData。
@@ -42,6 +42,8 @@ import java.util.regex.*;
"spring.datasource.driver-class-name=org.h2.Driver",
"vibemake.admin.worker-enabled=false",
"vibemake.oss.enabled=false",
"vibemake.gitea.owner=vibemake",
"vibemake.repository-write.worker-enabled=false",
"logging.level.root=WARN"
})
@ActiveProfiles("test")
@@ -95,8 +97,13 @@ class AdminWorkflowTest {
@Autowired ObjectMapper json;
@Autowired FakeGiteaClient gitea;
@Autowired ProjectService projectService;
@Autowired FileService fileService;
@Autowired UserService userService;
@Autowired ProjectLocks locks;
@Autowired com.linkerbot.vibemake.service.write.RepositoryWrites writes;
@Autowired RepositoryWriteOperationMapper writeOperations;
@Autowired javax.sql.DataSource dataSource;
@Autowired org.springframework.transaction.support.TransactionTemplate transactions;
@Autowired OssProperties oss;
@Autowired FakeOssObjectStore ossObjects;
private User admin;
@@ -209,47 +216,136 @@ class AdminWorkflowTest {
assertThat(projectService.published(code).getPublishedCommitSha()).isEqualTo(current);
}
private String initialSha(String code) throws Exception {
return request(signed(get("/admin-api/projects/" + code + "/tree")), 200)
.path("data").path("commitSha").asText();
}
private void due(String operationCode, boolean overdue) {
java.time.LocalDateTime now = ProjectSupport.now();
jdbc.update("UPDATE repository_write_operation SET next_check_at=?, created_at=? WHERE operation_code=?",
now.minusSeconds(1), now.minusMinutes(overdue ? 11 : 1), operationCode);
}
private String operationState(String operationCode) {
return writeOperations.selectById(operationCode).getStatus();
}
private String operationCode(String projectCode) {
return jdbc.queryForObject("SELECT operation_code FROM repository_write_operation WHERE project_code=?",
String.class, projectCode);
}
private com.linkerbot.vibemake.bean.Contracts.CommitView replayUpload(String code, String base, String key,
String directory) {
try (ProjectActors.Scope scope = ProjectActors.enter(new ProjectActors.AdminActor(admin.getUserCode(), admin.getNickname()))) {
return fileService.upload(0, code, base, "测试上传", directory,
List.of(new MockMultipartFile("files", "sample.txt", "text/plain", "hello".getBytes())),
List.of("models/sample.txt"), key);
}
}
@Test
void uploadedCommitWithLostResponseStillBlocksPublicationAtLatestSha() throws Exception {
void lostUploadResponseIsConfirmedAndSameOperationNeverRepeatsWrite() throws Exception {
String code = createProject();
String initial = request(signed(get("/admin-api/projects/" + code + "/tree")), 200)
.path("data").path("commitSha").asText();
String initial = initialSha(code);
gitea.timeoutNextCommit = true;
JsonNode failure = request(uploadRequest(code, initial), 504);
assertThat(failure.path("code").asText()).isEqualTo("REMOTE_RESULT_UNKNOWN");
assertThat(failure.path("data").path("projectCode").asText()).isEqualTo(code);
assertThat(writeState(code)).isEqualTo("REVIEW");
String current = request(signed(get("/admin-api/projects/" + code + "/tree")), 200)
.path("data").path("commitSha").asText();
JsonNode result = request(uploadRequest(code, initial), 200);
String key = operationCode(code);
String current = result.path("data").path("commitSha").asText();
assertThat(current).isNotEqualTo(initial);
assertThat(request(signed(get("/admin-api/projects/" + code + "/files")
.param("path", "models/sample.txt")), 200)
.path("data").path("content").asText()).isEqualTo("hello");
JsonNode blocked = request(signed(body(
put("/admin-api/projects/" + code + "/publication"), Map.of("commitSha", current))), 409);
assertThat(blocked.path("code").asText()).isEqualTo("REMOTE_RESULT_UNKNOWN");
assertThat(writeState(code)).isEqualTo("REVIEW");
assertThat(projects.selectOne(new com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper<Project>()
.eq(Project::getProjectCode, code)).getPublishedCommitSha()).isNull();
// 只改数据库不能清除当前进程的待核对标记,下一次写入会重新持久化 REVIEW。
jdbc.update("UPDATE project SET repo_write_status='IDLE' WHERE project_code=?", code);
assertThat(result.path("data").path("message").asText()).isEqualTo("测试上传");
assertThat(writeState(code)).isEqualTo("IDLE");
JsonNode blockedAgain = request(signed(body(
put("/admin-api/projects/" + code + "/publication"), Map.of("commitSha", current))), 409);
assertThat(blockedAgain.path("code").asText()).isEqualTo("REMOTE_RESULT_UNKNOWN");
assertThat(writeState(code)).isEqualTo("REVIEW");
assertThat(operationState(key)).isEqualTo("SUCCEEDED");
int calls = gitea.commitCalls;
assertThat(replayUpload(code, initial, key, "").commitSha()).isEqualTo(current);
assertThat(gitea.commitCalls).isEqualTo(calls);
assertThatThrownBy(() -> replayUpload(code, initial, key, "other"))
.isInstanceOfSatisfying(BusinessException.class,
error -> assertThat(error.code()).isEqualTo("IDEMPOTENCY_KEY_REUSED"));
assertThat(request(signed(get("/admin-api/projects/" + code + "/commits")), 200)
.path("data").path("items").get(0).path("message").asText()).isEqualTo("测试上传");
assertThat(jdbc.queryForObject("SELECT published_commit_sha FROM project WHERE project_code=?",
String.class, code)).isNull();
}
// 模拟核对远端结果后持锁恢复,同时清除数据库和本进程标记。
locks.serial(code, () -> {
locks.confirmKnownResult(code);
return null;
});
@Test
void pendingWriteKeepsOriginalErrorAndRecoversAfterRestartWithoutFrontendChanges() throws Exception {
String code = createProject();
String initial = initialSha(code);
gitea.timeoutNextCommit = true;
gitea.hideReceipts = true;
JsonNode error = request(uploadRequest(code, initial), 504);
assertThat(error.path("code").asText()).isEqualTo("REMOTE_RESULT_UNKNOWN");
assertThat(error.path("data").path("projectCode").asText()).isEqualTo(code);
String key = operationCode(code);
assertThat(operationState(key)).isEqualTo("PROCESSING");
assertThat(writeState(code)).isEqualTo("REVIEW");
int calls = gitea.commitCalls;
assertThat(request(uploadRequest(code, initial), 409).path("code").asText())
.isEqualTo("REMOTE_RESULT_UNKNOWN");
assertThatThrownBy(() -> replayUpload(code, initial, key, ""))
.isInstanceOfSatisfying(BusinessException.class, business -> assertThat(business.status()).isEqualTo(409));
assertThat(gitea.commitCalls).isEqualTo(calls);
String current = initialSha(code);
request(signed(body(put("/admin-api/projects/" + code + "/publication"),
Map.of("commitSha", current))), 409);
due(key, false);
gitea.hideReceipts = false;
try (ProjectLocks restarted = new ProjectLocks(dataSource, projects)) {
com.linkerbot.vibemake.service.write.RepositoryWrites worker =
new com.linkerbot.vibemake.service.write.RepositoryWrites(writeOperations, projects,
projectService, restarted, gitea, transactions);
worker.reconcileDue();
}
assertThat(operationState(key)).isEqualTo("SUCCEEDED");
assertThat(writeState(code)).isEqualTo("IDLE");
assertThat(writeOperations.selectById(key).getResultCommitSha()).isEqualTo(current);
assertThat(gitea.commitCalls).isEqualTo(calls);
assertThat(jdbc.queryForObject("SELECT published_commit_sha FROM project WHERE project_code=?",
String.class, code)).isNull();
// 原实例仍保留 uncertain,下一次正常写入必须按数据库 IDLE 清理,无需重启。
request(signed(body(put("/admin-api/projects/" + code + "/publication"),
Map.of("commitSha", current))), 200);
assertThat(writeState(code)).isEqualTo("IDLE");
assertThat(projectService.published(code).getPublishedCommitSha()).isEqualTo(current);
}
@ParameterizedTest
@ValueSource(strings = {"missing", "parent", "identity", "read"})
void missingOrInvalidReceiptNeverBecomesSuccessOrResendsWrite(String fault) throws Exception {
String code = createProject();
String initial = initialSha(code);
gitea.hideReceipts = fault.equals("missing");
gitea.wrongReceiptParent = fault.equals("parent");
gitea.wrongRepositoryIdentity = fault.equals("identity");
gitea.failReceiptReads = fault.equals("read");
gitea.timeoutNextCommit = true;
request(uploadRequest(code, initial), 504);
String key = operationCode(code);
int calls = gitea.commitCalls;
due(key, false);
writes.reconcileDue();
assertThat(operationState(key)).isEqualTo("PROCESSING");
assertThat(writeState(code)).isEqualTo("REVIEW");
assertThat(writeOperations.selectById(key).getNextCheckAt())
.isAfter(ProjectSupport.now().plusSeconds(10));
due(key, true);
writes.reconcileDue();
assertThat(operationState(key)).isEqualTo("NEEDS_REVIEW");
assertThat(writeOperations.selectById(key).getNextCheckAt())
.isAfter(ProjectSupport.now().plusSeconds(50));
assertThat(gitea.commitCalls).isEqualTo(calls);
assertThat(writeState(code)).isEqualTo("REVIEW");
}
@Test
void legacyReviewWithoutOperationIsNeverGuessedSuccessful() throws Exception {
String code = createProject();
locks.serial(code, () -> { locks.markUncertain(code); return null; });
int calls = gitea.commitCalls;
writes.reconcileDue();
assertThat(writeState(code)).isEqualTo("REVIEW");
assertThat(gitea.commitCalls).isEqualTo(calls);
}
private JsonNode createImport() throws Exception {
@@ -12,6 +12,7 @@ final class FakeGiteaClient implements GiteaClient {
private static final class RepoData {
Repository repository;
String head;
Map<String, String> parents = new HashMap<>();
Map<String, Map<String, byte[]>> snapshots = new HashMap<>();
List<HistoryEntry> history = new ArrayList<>();
}
@@ -27,8 +28,14 @@ final class FakeGiteaClient implements GiteaClient {
int deletions;
boolean mainConflict;
private final Map<String, RemoteRepository> migrated = new HashMap<>();
private final Set<String> migrationNames = new HashSet<>();
final Map<String, String> migrationDescriptions = new HashMap<>();
private final Set<String> mainReady = new HashSet<>();
boolean hideReceipts;
boolean failReceiptReads;
boolean wrongReceiptParent;
boolean wrongRepositoryIdentity;
int commitCalls;
boolean timeoutNextCommit;
boolean timeoutNextHead;
boolean timeoutNextCreate;
@@ -36,7 +43,13 @@ final class FakeGiteaClient implements GiteaClient {
void reset() {
repositories.clear();
hideReceipts = false;
failReceiptReads = false;
wrongReceiptParent = false;
wrongRepositoryIdentity = false;
commitCalls = 0;
migrated.clear();
migrationNames.clear();
mainReady.clear();
migrations = 0;
deletions = 0;
@@ -151,6 +164,7 @@ final class FakeGiteaClient implements GiteaClient {
String authorName,
String authorCode) {
lastAuthorCode = authorCode;
commitCalls++;
RepoData repo = repo(repository);
Map<String, byte[]> snapshot =
repo.head == null ? new TreeMap<>() : new TreeMap<>(repo.snapshots.get(repo.head));
@@ -164,7 +178,9 @@ final class FakeGiteaClient implements GiteaClient {
if (change.operation().equals("delete")) snapshot.remove(change.path());
else snapshot.put(change.path(), change.bytes().clone());
}
String parent = repo.head;
repo.head = sha();
repo.parents.put(repo.head, parent);
repo.snapshots.put(repo.head, snapshot);
repo.history.addFirst(
new HistoryEntry(repo.head, message, authorName, "2026-10-13T00:00:00Z"));
@@ -181,7 +197,9 @@ final class FakeGiteaClient implements GiteaClient {
List<HistoryEntry> history = repo(repository).history;
int from = (int) Math.min(history.size(), (page - 1) * size);
int to = (int) Math.min(history.size(), from + size);
return new PageResponse<>(history.subList(from, to), page, size, history.size());
return new PageResponse<>(history.subList(from, to).stream().map(item ->
new HistoryEntry(item.commitSha(), WriteReceipt.displayMessage(item.message()),
item.authorName(), item.authorDate())).toList(), page, size, history.size());
}
@Override
@@ -197,6 +215,7 @@ final class FakeGiteaClient implements GiteaClient {
public synchronized RemoteRepository migrate(
String source, String name, java.time.Duration timeout, String description) {
migrations++;
migrationNames.add(name);
Repository repository = createRepository(name, true);
RemoteRepository remote =
new RemoteRepository(
@@ -232,7 +251,16 @@ final class FakeGiteaClient implements GiteaClient {
@Override
public synchronized RemoteRepository inspect(String name) {
RemoteRepository remote = migrated.get(name);
if (remote == null) throw new GiteaFailure(404, "NOT_FOUND", "仓库不存在", false);
if (remote == null && migrationNames.contains(name))
throw new GiteaFailure(404, "NOT_FOUND", "仓库不存在", false);
if (remote == null) {
for (RepoData data : repositories.values()) {
if (data.repository.name().equals(name))
return new RemoteRepository(data.repository.id() + (wrongRepositoryIdentity ? 1 : 0),
data.repository.owner(), name, true, false, false, null, "main");
}
throw new GiteaFailure(404, "NOT_FOUND", "仓库不存在", false);
}
return remote;
}
@@ -264,4 +292,20 @@ final class FakeGiteaClient implements GiteaClient {
remote.originalUrl(),
branch));
}
@Override
public synchronized String findWriteReceipt(Repository repository, String baseSha,
String operationCode, String digest) {
if (failReceiptReads) throw new GiteaFailure(504, "READ_TIMEOUT", "核对读取超时", false);
if (hideReceipts) return null;
RepoData data = repo(repository);
for (HistoryEntry entry : data.history) {
if (entry.message().endsWith(WriteReceipt.suffix(operationCode, digest))) {
if (wrongReceiptParent || !baseSha.equals(data.parents.get(entry.commitSha())))
throw new GiteaFailure(409, "WRITE_RECEIPT_PARENT_MISMATCH", "父提交不匹配", false);
return entry.commitSha();
}
}
return null;
}
}
@@ -12,18 +12,29 @@ final class FakeGiteaClient implements GiteaClient {
private static final class RepoData {
Repository repository;
String head;
Map<String, String> parents = new HashMap<>();
Map<String, Map<String, byte[]>> snapshots = new HashMap<>();
List<HistoryEntry> history = new ArrayList<>();
}
private final Map<Long, RepoData> repositories = new HashMap<>();
private final AtomicLong sequence = new AtomicLong();
boolean hideReceipts;
boolean failReceiptReads;
boolean wrongReceiptParent;
boolean wrongRepositoryIdentity;
int commitCalls;
boolean timeoutNextCommit;
boolean timeoutNextCreate;
boolean failNextCreate;
void reset() {
repositories.clear();
hideReceipts = false;
failReceiptReads = false;
wrongReceiptParent = false;
wrongRepositoryIdentity = false;
commitCalls = 0;
timeoutNextCommit = false;
timeoutNextCreate = false;
failNextCreate = false;
@@ -123,6 +134,7 @@ final class FakeGiteaClient implements GiteaClient {
String message,
String authorName,
String authorCode) {
commitCalls++;
RepoData repo = repo(repository);
Map<String, byte[]> snapshot =
repo.head == null ? new TreeMap<>() : new TreeMap<>(repo.snapshots.get(repo.head));
@@ -136,7 +148,9 @@ final class FakeGiteaClient implements GiteaClient {
if (change.operation().equals("delete")) snapshot.remove(change.path());
else snapshot.put(change.path(), change.bytes().clone());
}
String parent = repo.head;
repo.head = sha();
repo.parents.put(repo.head, parent);
repo.snapshots.put(repo.head, snapshot);
repo.history.addFirst(
new HistoryEntry(repo.head, message, authorName, "2026-10-13T00:00:00Z"));
@@ -153,6 +167,34 @@ final class FakeGiteaClient implements GiteaClient {
List<HistoryEntry> history = repo(repository).history;
int from = (int) Math.min(history.size(), (page - 1) * size);
int to = (int) Math.min(history.size(), from + size);
return new PageResponse<>(history.subList(from, to), page, size, history.size());
return new PageResponse<>(history.subList(from, to).stream().map(item ->
new HistoryEntry(item.commitSha(), WriteReceipt.displayMessage(item.message()),
item.authorName(), item.authorDate())).toList(), page, size, history.size());
}
@Override
public synchronized String findWriteReceipt(Repository repository, String baseSha,
String operationCode, String digest) {
if (failReceiptReads) throw new GiteaFailure(504, "READ_TIMEOUT", "核对读取超时", false);
if (hideReceipts) return null;
RepoData data = repo(repository);
for (HistoryEntry entry : data.history) {
if (entry.message().endsWith(WriteReceipt.suffix(operationCode, digest))) {
if (wrongReceiptParent || !baseSha.equals(data.parents.get(entry.commitSha())))
throw new GiteaFailure(409, "WRITE_RECEIPT_PARENT_MISMATCH", "父提交不匹配", false);
return entry.commitSha();
}
}
return null;
}
@Override
public synchronized RemoteRepository inspect(String name) {
for (RepoData data : repositories.values()) {
if (data.repository.name().equals(name))
return new RemoteRepository(data.repository.id() + (wrongRepositoryIdentity ? 1 : 0),
data.repository.owner(), name, true, false, false, null, "main");
}
throw new GiteaFailure(404, "NOT_FOUND", "仓库不存在", false);
}
}
@@ -98,6 +98,7 @@ class QueryEfficiencyTest {
}
for (String table :
List.of(
"repository_write_operation",
"project_import_task",
"project_like",
"project_comment",
@@ -31,6 +31,6 @@ final class SchemaSupport {
jdbc.execute(ddl);
tables++;
}
if (tables != 5) throw new IllegalStateException("必须包含四张基础表和迁移任务表");
if (tables != 6) throw new IllegalStateException("必须包含四张基础表、迁移任务表和仓库写入操作表");
}
}
@@ -38,6 +38,8 @@ import java.util.concurrent.*;
"spring.datasource.password=",
"spring.datasource.driver-class-name=org.h2.Driver",
"logging.level.root=INFO",
"vibemake.gitea.owner=vibemake",
"vibemake.repository-write.worker-enabled=false",
"logging.level.org.springframework=WARN"
})
@ActiveProfiles("test")
@@ -82,6 +84,8 @@ class WorkflowTest {
@Autowired ObjectMapper json;
@Autowired JdbcTemplate jdbc;
@Autowired FakeGiteaClient gitea;
@Autowired com.linkerbot.vibemake.service.FileService fileService;
@Autowired com.linkerbot.vibemake.service.write.RepositoryWrites writes;
@Autowired OssProperties oss;
@Autowired FakeOssObjectStore ossObjects;
@Autowired UrdfProperties urdfLimits;
@@ -1107,7 +1111,7 @@ class WorkflowTest {
"PROJECT",
"PROJECT_COMMENT",
"PROJECT_LIKE",
"PROJECT_IMPORT_TASK");
"PROJECT_IMPORT_TASK", "REPOSITORY_WRITE_OPERATION");
}
@Test
@@ -1487,6 +1491,58 @@ class WorkflowTest {
413);
}
@Test
void editorWriteRecoversInBackgroundWithoutChangingHttpContract() throws Exception {
String email = "pending-editor@example.invalid";
String owner = login(email);
String code = create(owner, "编辑处理中");
String sha = head(owner, code);
Map<String, Object> input = Map.of("expectedCommitSha", sha, "message", "编辑说明",
"changes", List.of(Map.of("action", "UPDATE_FILE", "path", "README.md", "content", "新说明")));
gitea.hideReceipts = true;
gitea.timeoutNextCommit = true;
JsonNode failure = perform(body(auth(post("/api/workspace/projects/" + code + "/commits"), owner), input), 504);
assertThat(failure.path("code").asText()).isEqualTo("REMOTE_RESULT_UNKNOWN");
String operation = jdbc.queryForObject("SELECT operation_code FROM repository_write_operation WHERE project_code=?",
String.class, code);
int calls = gitea.commitCalls;
perform(body(auth(post("/api/workspace/projects/" + code + "/commits"), owner), input), 409);
assertThat(gitea.commitCalls).isEqualTo(calls);
gitea.hideReceipts = false;
jdbc.update("UPDATE repository_write_operation SET next_check_at=? WHERE operation_code=?",
com.linkerbot.vibemake.service.support.ProjectSupport.now().minusSeconds(1), operation);
writes.reconcileDue();
String confirmed = head(owner, code);
assertThat(confirmed).isNotEqualTo(sha);
long userId = jdbc.queryForObject("SELECT id FROM sys_user WHERE email=?", Long.class, email);
assertThat(fileService.commit(userId, code,
json.convertValue(input, com.linkerbot.vibemake.bean.Contracts.CommitInput.class), operation).commitSha())
.isEqualTo(confirmed);
assertThat(gitea.commitCalls).isEqualTo(calls);
}
@Test
void rejectedEditorWriteReplaysFailureWithoutCallingGitea() throws Exception {
String email = "rejected-editor@example.invalid";
String owner = login(email);
String code = create(owner, "拒绝重复创建");
Map<String, Object> input = Map.of("expectedCommitSha", head(owner, code), "message", "不能重复创建",
"changes", List.of(Map.of("action", "CREATE_FILE", "path", "README.md", "content", "重复")));
perform(body(auth(post("/api/workspace/projects/" + code + "/commits"), owner), input), 409);
String key = jdbc.queryForObject("SELECT operation_code FROM repository_write_operation WHERE project_code=?",
String.class, code);
int calls = gitea.commitCalls;
long userId = jdbc.queryForObject("SELECT id FROM sys_user WHERE email=?", Long.class, email);
assertThatThrownBy(() -> fileService.commit(userId, code,
json.convertValue(input, com.linkerbot.vibemake.bean.Contracts.CommitInput.class), key))
.isInstanceOfSatisfying(BusinessException.class, error -> assertThat(error.status()).isEqualTo(409));
assertThat(gitea.commitCalls).isEqualTo(calls);
assertThat(jdbc.queryForObject("SELECT status FROM repository_write_operation WHERE operation_code=?",
String.class, key)).isEqualTo("FAILED");
assertThat(jdbc.queryForObject("SELECT repo_write_status FROM project WHERE project_code=?",
String.class, code)).isEqualTo("IDLE");
}
@Test
void unknownRemoteWritesAreNotRepeatedAndFailedCreatesRemainVisible() throws Exception {
String owner = login("failure@example.com");
@@ -1545,7 +1601,7 @@ class WorkflowTest {
"实际成功")));
perform(
body(auth(post("/api/workspace/projects/" + code + "/commits"), owner), request),
504);
200);
perform(
body(auth(post("/api/workspace/projects/" + code + "/commits"), owner), request),
409);
@@ -0,0 +1,30 @@
package com.linkerbot.vibemake.bean.entity;
import com.baomidou.mybatisplus.annotation.IdType;
import com.baomidou.mybatisplus.annotation.TableId;
import lombok.Getter;
import lombok.Setter;
import java.time.LocalDateTime;
/** 仓库写入凭据。不保存文件正文或认证信息。 */
@Getter
@Setter
public class RepositoryWriteOperation {
@TableId(type = IdType.INPUT)
private String operationCode;
private String projectCode;
private Long repositoryId;
private String repositoryOwner;
private String repositoryName;
private String baseCommitSha;
private String requestDigest;
private String commitMessage;
private String status;
private String resultCommitSha;
private Integer failureStatus;
private String lastErrorCode;
private Integer checkCount;
private LocalDateTime nextCheckAt;
private LocalDateTime createdAt;
private LocalDateTime updatedAt;
}
@@ -28,6 +28,11 @@ public interface FileService {
CommitView commit(long userId, String code, CommitInput input);
CommitView commit(long userId, String code, CommitInput input, String operationCode);
CommitView upload(long userId, String code, String expectedSha, String message, String directory,
List<MultipartFile> files, List<String> paths, String operationCode);
CommitView upload(
long userId,
String code,
@@ -15,6 +15,12 @@ public interface GiteaClient {
record Change(String operation, String path, String blobSha, byte[] bytes) {}
/** 只从 main 可达的提交中查找,并校验完整尾注及唯一父提交;空结果不是失败证明。 */
default String findWriteReceipt(Repository repository, String baseSha, String operationCode,
String requestDigest) {
throw new UnsupportedOperationException("未实现仓库写入凭据核对");
}
record RemoteRepository(long id, String owner, String name, boolean privateRepo,
boolean mirror, boolean empty, String originalUrl, String defaultBranch) {
public Repository repository() { return new Repository(id, owner, name); }
@@ -425,7 +425,7 @@ public class RestGiteaClient implements GiteaClient {
items.add(
new HistoryEntry(
node.path("sha").asText(),
commit.path("message").asText(),
WriteReceipt.displayMessage(commit.path("message").asText()),
commit.path("author").path("name").asText(),
commit.path("author").path("date").asText()));
}
@@ -436,4 +436,35 @@ public class RestGiteaClient implements GiteaClient {
.orElse((long) items.size());
return new PageResponse<>(items, page, size, total);
}
@Override
public String findWriteReceipt(Repository repo, String baseSha, String operationCode,
String requestDigest) {
// 普通文件写入被项目锁串行化,未知期间禁止下一次写入。限制扫描量防止异常仓库拖垮任务。
// 超出扫描范围不推断失败,留在处理中并交给管理员兜底。
String suffix = WriteReceipt.suffix(operationCode, requestDigest);
String head = head(repo);
String found = null;
for (int page = 1; page <= 20; page++) {
JsonNode entries = parse(request("GET", repoPath(repo)
+ "/commits?sha=" + head + "&page=" + page + "&limit=50", null).body(), false);
if (!entries.isArray())
throw new GiteaFailure(502, "GITEA_INVALID_RESPONSE", "提交记录格式异常", false);
for (JsonNode entry : entries) {
if (entry.path("commit").path("message").asText().stripTrailing().endsWith(suffix)) {
JsonNode parents = entry.path("parents");
if (parents.size() != 1 || !baseSha.equals(parents.path(0).path("sha").asText()))
throw new GiteaFailure(409, "WRITE_RECEIPT_PARENT_MISMATCH", "提交父版本不匹配", false);
String sha = FilePaths.sha(entry.path("sha").asText());
if (found != null && !found.equals(sha))
throw new GiteaFailure(409, "WRITE_RECEIPT_AMBIGUOUS", "存在多个操作凭据", false);
found = sha;
}
}
if (entries.size() < 50) break;
}
if (!head.equals(head(repo)))
throw new GiteaFailure(409, "WRITE_RECEIPT_BRANCH_CHANGED", "核对期间分支版本已变化", false);
return found;
}
}
@@ -0,0 +1,14 @@
package com.linkerbot.vibemake.service.gitea;
/** 固定尾注同时绑定操作编码和请求摘要,普通提交信息不会被当作凭据。 */
public final class WriteReceipt {
private WriteReceipt() {}
public static String suffix(String code, String digest) {
return "\n\nVibemake-Operation: " + code + "\nVibemake-Request: " + digest;
}
public static String displayMessage(String message) {
return message.replaceFirst("\\n\\nVibemake-Operation: [a-f0-9]{32}\\nVibemake-Request: [a-f0-9]{64}\\s*$", "");
}
}
@@ -11,6 +11,8 @@ import com.linkerbot.vibemake.service.gitea.*;
import com.linkerbot.vibemake.service.state.RateLimits;
import com.linkerbot.vibemake.service.support.ProjectActors;
import com.linkerbot.vibemake.service.support.ProjectSupport;
import com.linkerbot.vibemake.service.write.RepositoryWrites;
import com.linkerbot.vibemake.service.write.WriteRequestDigest;
import lombok.RequiredArgsConstructor;
@@ -33,6 +35,8 @@ public class FileServiceImpl implements FileService {
private final UserService users;
private final GiteaClient gitea;
private final VibemakeProperties config;
private final RepositoryWrites writes;
private final WriteRequestDigest digests;
@Override
public TreeView ownerTree(long userId, String code, String path, String commitSha) {
@@ -137,8 +141,14 @@ public class FileServiceImpl implements FileService {
@Override
public CommitView commit(long userId, String code, CommitInput input) {
return commit(userId, code, input, null);
}
@Override
public CommitView commit(long userId, String code, CommitInput input, String operationCode) {
projects.readyOwner(userId, code);
rateLimits.operation("file-write", userId);
return projects.write(
return writes.execute(userId, code, operationCode, digests.commit(input), request -> projects.write(
userId,
code,
project -> {
@@ -191,19 +201,18 @@ public class FileServiceImpl implements FileService {
size(bytes, config.getMaxRequestBytes());
ProjectActors.Author author = ProjectActors.author(users, userId);
CommitView result =
gitea.commit(
repo,
writes.commit(
request, project, sha,
changes,
input.message(),
author.nickname(),
author.code());
author);
log.info(
"文件提交 projectCode={} commitSha={} fileCount={}",
code,
result.commitSha(),
changes.size());
return result;
});
}));
}
@Override
@@ -215,6 +224,14 @@ public class FileServiceImpl implements FileService {
String directory,
List<MultipartFile> files,
List<String> paths) {
return upload(userId, code, expectedSha, message, directory, files, paths, null);
}
@Override
public CommitView upload(long userId, String code, String expectedSha, String message,
String directory, List<MultipartFile> files, List<String> paths,
String operationCode) {
projects.readyOwner(userId, code);
rateLimits.operation("file-write", userId);
if (message == null || message.isBlank() || message.length() > 1000)
throw BusinessException.badRequest("提交说明不合法");
@@ -224,7 +241,8 @@ public class FileServiceImpl implements FileService {
|| (paths != null && paths.size() != files.size()))
throw BusinessException.badRequest("文件和路径数量不合法");
String prefix = FilePaths.directory(directory);
return projects.write(
return writes.execute(userId, code, operationCode,
digests.upload(expectedSha, message, directory, files, paths), request -> projects.write(
userId,
code,
project -> {
@@ -264,14 +282,14 @@ public class FileServiceImpl implements FileService {
}
ProjectActors.Author author = ProjectActors.author(users, userId);
CommitView result =
gitea.commit(repo, changes, message, author.nickname(), author.code());
writes.commit(request, project, sha, changes, message, author);
log.info(
"文件上传 projectCode={} commitSha={} fileCount={}",
code,
result.commitSha(),
changes.size());
return result;
});
}));
}
@Override
@@ -0,0 +1,8 @@
package com.linkerbot.vibemake.service.mapper;
import com.baomidou.mybatisplus.core.mapper.BaseMapper;
import com.linkerbot.vibemake.bean.entity.RepositoryWriteOperation;
import org.apache.ibatis.annotations.Mapper;
@Mapper
public interface RepositoryWriteOperationMapper extends BaseMapper<RepositoryWriteOperation> {}
@@ -28,7 +28,7 @@ import javax.sql.DataSource;
/** 数据库命名锁不持有业务事务;专用连接在整个项目操作期间保持占用。 */
@Component
public class ProjectLocks {
public class ProjectLocks implements AutoCloseable {
private static final org.slf4j.Logger log =
org.slf4j.LoggerFactory.getLogger(ProjectLocks.class);
private final ReentrantLock[] locks = new ReentrantLock[128];
@@ -83,6 +83,10 @@ public class ProjectLocks {
() -> {
if (writing.get().contains(code)) return action.get();
Project guard = guard(code);
// 已持有共享项目锁,以数据库为准同步其他进程确认的恢复结果。
if (guard != null && "IDLE".equals(guard.getRepoWriteStatus())
&& uncertain.remove(code))
log.info("同步已恢复的仓库写入状态 projectCode={} writeStatus=IDLE", code);
boolean memoryUncertain = uncertain.contains(code);
if (memoryUncertain
|| guard != null && !"IDLE".equals(guard.getRepoWriteStatus())) {
@@ -212,7 +216,7 @@ public class ProjectLocks {
}
private BusinessException blocked() {
return new BusinessException(409, "REMOTE_RESULT_UNKNOWN", "上次仓库写入结果待人工核对,暂不能继续写入");
return new BusinessException(409, "REMOTE_RESULT_UNKNOWN", "仓库写入结果正在核对,请稍后查询处理结果");
}
@PreDestroy
@@ -0,0 +1,45 @@
package com.linkerbot.vibemake.service.write;
import lombok.RequiredArgsConstructor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
@Configuration
@EnableScheduling
@RequiredArgsConstructor
@ConditionalOnProperty(name = "vibemake.repository-write.worker-enabled", matchIfMissing = true)
public class RepositoryWriteWorker {
private static final Logger log = LoggerFactory.getLogger(RepositoryWriteWorker.class);
private final RepositoryWrites writes;
private final java.util.concurrent.ExecutorService executor = java.util.concurrent.Executors
.newSingleThreadExecutor(Thread.ofPlatform().name("repository-write-check").factory());
private final java.util.concurrent.atomic.AtomicBoolean running = new java.util.concurrent.atomic.AtomicBoolean();
@Scheduled(fixedDelay = 1000, initialDelay = 15000)
public void reconcile() {
if (executor.isShutdown() || !running.compareAndSet(false, true)) return;
try {
executor.execute(() -> {
try {
writes.reconcileDue();
} catch (RuntimeException error) {
com.linkerbot.vibemake.common.SafeExceptionLog.write(log, "仓库写入核对任务失败", error);
} finally {
running.set(false);
}
});
} catch (java.util.concurrent.RejectedExecutionException shutdown) {
running.set(false);
}
}
@jakarta.annotation.PreDestroy
public void close() throws InterruptedException {
executor.shutdown();
if (!executor.awaitTermination(5, java.util.concurrent.TimeUnit.SECONDS)) executor.shutdownNow();
}
}
@@ -0,0 +1,239 @@
package com.linkerbot.vibemake.service.write;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.linkerbot.vibemake.bean.Contracts.CommitView;
import com.linkerbot.vibemake.bean.entity.Project;
import com.linkerbot.vibemake.bean.entity.RepositoryWriteOperation;
import com.linkerbot.vibemake.common.*;
import com.linkerbot.vibemake.service.ProjectService;
import com.linkerbot.vibemake.service.gitea.*;
import com.linkerbot.vibemake.service.mapper.ProjectMapper;
import com.linkerbot.vibemake.service.mapper.RepositoryWriteOperationMapper;
import com.linkerbot.vibemake.service.support.ProjectActors;
import com.linkerbot.vibemake.service.support.ProjectLocks;
import com.linkerbot.vibemake.service.support.ProjectSupport;
import lombok.RequiredArgsConstructor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Service;
import org.springframework.transaction.support.TransactionTemplate;
import java.time.LocalDateTime;
import java.util.List;
import java.util.Objects;
import java.util.function.Function;
/** 数据库凭据先于远端写入落盘。未知结果只读取核对,绝不自动重发写请求。 */
@Service
@RequiredArgsConstructor
public class RepositoryWrites {
private static final Logger log = LoggerFactory.getLogger(RepositoryWrites.class);
private static final List<String> ACTIVE = List.of("PROCESSING", "NEEDS_REVIEW");
private final RepositoryWriteOperationMapper operations;
private final ProjectMapper projectMapper;
private final ProjectService projects;
private final ProjectLocks locks;
private final GiteaClient gitea;
private final TransactionTemplate transactions;
public record Request(String code, String digest) {}
public CommitView execute(long userId, String projectCode, String key, String digest,
Function<Request, CommitView> action) {
String operationCode = key == null ? Codes.next() : key.replace("-", "").toLowerCase(java.util.Locale.ROOT);
if (!operationCode.matches("[a-f0-9]{32}"))
throw BusinessException.badRequest("操作编码必须为 UUID 或 32 位十六进制编码");
return locks.serial(projectCode, () -> {
projects.readyOwner(userId, projectCode); // 重放也必须重新鉴权。
RepositoryWriteOperation previous = operations.selectById(operationCode);
if (previous != null) {
if (!projectCode.equals(previous.getProjectCode()) || !digest.equals(previous.getRequestDigest()))
throw new BusinessException(409, "IDEMPOTENCY_KEY_REUSED", "操作编码已用于不同请求,请勿复用");
return replay(previous);
}
RepositoryWriteOperation pending = active(projectCode);
if (pending != null)
throw pending(pending);
try {
CommitView result = action.apply(new Request(operationCode, digest));
RepositoryWriteOperation operation = operations.selectById(operationCode);
if (operation == null) throw new IllegalStateException("仓库写入缺少持久化凭据");
complete(operation, result.commitSha(), "SUCCEEDED");
return result;
} catch (RuntimeException error) {
RepositoryWriteOperation operation = operations.selectById(operationCode);
if (operation == null) throw error; // 参数校验、读取失败发生在任何远端写入之前。
if (!projectCode.equals(operation.getProjectCode()) || !digest.equals(operation.getRequestDigest()))
throw new BusinessException(409, "IDEMPOTENCY_KEY_REUSED", "操作编码已用于不同请求,请勿复用");
boolean definite = error instanceof GiteaFailure remote && !remote.uncertain();
if (definite) {
BusinessException business = (BusinessException) error;
operation.setStatus("FAILED");
operation.setFailureStatus(business.status());
operation.setLastErrorCode(business.code());
save(operation);
throw error;
}
locks.markUncertain(projectCode);
operation.setLastErrorCode(error instanceof BusinessException business
? business.code() : "WRITE_RESULT_UNKNOWN");
save(operation);
log.warn("仓库写入转后台核对 projectCode={} operationCode={} errorCode={} errorType={}",
projectCode, operationCode, operation.getLastErrorCode(), error.getClass().getSimpleName());
if (!(error instanceof BusinessException)) SafeExceptionLog.write(log, "仓库提交结果未知", error);
throw error;
}
});
}
/** 必须在 ProjectLocks.write 内调用;此处是唯一一次发起远端写请求的位置。 */
public CommitView commit(Request request, Project project, String baseSha,
List<GiteaClient.Change> changes, String message,
ProjectActors.Author author) {
GiteaClient.Repository repository = projects.repository(project);
RepositoryWriteOperation operation = new RepositoryWriteOperation();
operation.setOperationCode(request.code());
operation.setProjectCode(project.getProjectCode());
operation.setRepositoryId(repository.id());
operation.setRepositoryOwner(repository.owner());
operation.setRepositoryName(repository.name());
operation.setBaseCommitSha(baseSha);
operation.setRequestDigest(request.digest());
operation.setCommitMessage(message);
operation.setStatus("PROCESSING");
operation.setCheckCount(0);
operation.setCreatedAt(ProjectSupport.now());
operation.setUpdatedAt(operation.getCreatedAt());
operation.setNextCheckAt(operation.getCreatedAt().plusSeconds(15));
try {
operations.insert(operation);
} catch (org.springframework.dao.DuplicateKeyException collision) {
throw new BusinessException(409, "IDEMPOTENCY_KEY_REUSED", "操作编码已用于其他请求,请勿复用");
}
log.info("仓库写入开始 projectCode={} operationCode={} baseCommitSha={} fileCount={}",
project.getProjectCode(), request.code(), baseSha, changes.size());
try {
CommitView result = gitea.commit(repository, changes,
message + WriteReceipt.suffix(request.code(), request.digest()), author.nickname(), author.code());
return new CommitView(result.commitSha(), message);
} catch (GiteaFailure error) {
if (!error.uncertain()) throw error;
// 尝试立即消除响应丢失的影响,查询失败依旧保留未知,不覆盖原始错误确定性。
String receipt = receipt(operation, project);
if (receipt != null) return new CommitView(receipt, message);
throw error;
}
}
private CommitView replay(RepositoryWriteOperation operation) {
return switch (operation.getStatus()) {
case "SUCCEEDED" -> new CommitView(operation.getResultCommitSha(), operation.getCommitMessage());
case "FAILED" -> throw new BusinessException(operation.getFailureStatus(),
operation.getLastErrorCode(), "该操作已被仓库拒绝,请刷新后使用新的操作编码提交");
default -> throw pending(operation);
};
}
private BusinessException pending(RepositoryWriteOperation operation) {
return new BusinessException(409, "REMOTE_RESULT_UNKNOWN", "仓库提交结果正在后台核对,请稍后刷新工作区,暂勿重复提交",
java.util.Map.of("projectCode", operation.getProjectCode()));
}
public RepositoryWriteOperation active(String code) {
return operations.selectOne(new LambdaQueryWrapper<RepositoryWriteOperation>()
.eq(RepositoryWriteOperation::getProjectCode, code)
.in(RepositoryWriteOperation::getStatus, ACTIVE)
.orderByDesc(RepositoryWriteOperation::getCreatedAt).last("LIMIT 1"));
}
private boolean slowCheckDue(RepositoryWriteOperation operation) {
return !ProjectSupport.now().isBefore(operation.getCreatedAt().plusMinutes(10));
}
/** 重启后直接扫描数据库,无内存任务队列;多实例由相同项目命名锁串行核对。 */
public void reconcileDue() {
List<RepositoryWriteOperation> due = operations.selectList(new LambdaQueryWrapper<RepositoryWriteOperation>()
.in(RepositoryWriteOperation::getStatus, ACTIVE)
.le(RepositoryWriteOperation::getNextCheckAt, ProjectSupport.now())
.orderByAsc(RepositoryWriteOperation::getNextCheckAt).last("LIMIT 20"));
for (RepositoryWriteOperation candidate : due) {
try {
locks.serial(candidate.getProjectCode(), () -> {
reconcile(candidate.getOperationCode());
return null;
});
} catch (RuntimeException error) {
log.warn("仓库后台核对未完成 projectCode={} operationCode={} errorCode={} errorType={}",
candidate.getProjectCode(), candidate.getOperationCode(),
error instanceof BusinessException business ? business.code() : "CHECK_FAILED",
error.getClass().getSimpleName());
}
}
}
private void reconcile(String operationCode) {
RepositoryWriteOperation operation = operations.selectById(operationCode);
if (operation == null || !ACTIVE.contains(operation.getStatus())
|| operation.getNextCheckAt().isAfter(ProjectSupport.now())) return;
Project project = projectMapper.selectOne(new LambdaQueryWrapper<Project>()
.eq(Project::getProjectCode, operation.getProjectCode()));
String receipt = project == null ? null : receipt(operation, project);
operation.setCheckCount(operation.getCheckCount() + 1);
if (receipt != null) {
complete(operation, receipt, "SUCCEEDED");
return;
}
boolean overdue = slowCheckDue(operation);
operation.setStatus(overdue ? "NEEDS_REVIEW" : "PROCESSING");
operation.setNextCheckAt(ProjectSupport.now().plusSeconds(overdue ? 60 : 15));
save(operation);
log.info("仓库写入继续核对 projectCode={} operationCode={} status={} checkCount={} errorCode={}",
operation.getProjectCode(), operationCode, operation.getStatus(),
operation.getCheckCount(), operation.getLastErrorCode());
}
private String receipt(RepositoryWriteOperation operation, Project project) {
try {
if (!"READY".equals(project.getRepoStatus())
|| !Objects.equals(project.getGiteaRepoId(), operation.getRepositoryId())
|| !Objects.equals(project.getGiteaOwner(), operation.getRepositoryOwner())
|| !Objects.equals(project.getGiteaRepoName(), operation.getRepositoryName()))
throw new GiteaFailure(409, "WRITE_REPOSITORY_MISMATCH", "项目仓库身份发生变化", false);
GiteaClient.RemoteRepository remote = gitea.inspect(operation.getRepositoryName());
if (remote.id() != operation.getRepositoryId()
|| !remote.owner().equals(operation.getRepositoryOwner())
|| !remote.name().equals(operation.getRepositoryName())
|| !remote.privateRepo() || remote.mirror() || remote.empty()
|| !"main".equals(remote.defaultBranch()))
throw new GiteaFailure(409, "WRITE_REPOSITORY_MISMATCH", "远端仓库身份发生变化", false);
String sha = gitea.findWriteReceipt(remote.repository(), operation.getBaseCommitSha(),
operation.getOperationCode(), operation.getRequestDigest());
if (sha == null) operation.setLastErrorCode("WRITE_RECEIPT_NOT_FOUND");
return sha;
} catch (RuntimeException error) {
operation.setLastErrorCode(error instanceof BusinessException business
? business.code() : "WRITE_RECEIPT_READ_FAILED");
log.warn("仓库写入凭据读取未确认 projectCode={} operationCode={} errorCode={} errorType={}",
operation.getProjectCode(), operation.getOperationCode(), operation.getLastErrorCode(),
error.getClass().getSimpleName());
if (!(error instanceof BusinessException)) SafeExceptionLog.write(log, "仓库凭据核对异常", error);
return null;
}
}
private void complete(RepositoryWriteOperation operation, String sha, String status) {
// 项目恢复与操作终态在同一事务提交;任务获得锁后重读终态,旧任务不会清除新操作的保护。
transactions.executeWithoutResult(transaction -> {
operation.setStatus(status);
operation.setResultCommitSha(sha);
save(operation);
locks.confirmKnownResult(operation.getProjectCode());
});
log.info("仓库写入结果已落盘 projectCode={} operationCode={} status={} commitSha={}",
operation.getProjectCode(), operation.getOperationCode(), status, sha);
}
private void save(RepositoryWriteOperation operation) {
operation.setUpdatedAt(ProjectSupport.now());
if (operations.updateById(operation) != 1) throw new IllegalStateException("仓库写入记录更新失败");
}
}
@@ -0,0 +1,78 @@
package com.linkerbot.vibemake.service.write;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.linkerbot.vibemake.bean.Contracts.CommitInput;
import com.linkerbot.vibemake.common.BusinessException;
import com.linkerbot.vibemake.common.FilePaths;
import com.linkerbot.vibemake.service.config.VibemakeProperties;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Component;
import org.springframework.web.multipart.MultipartFile;
import java.io.IOException;
import java.io.InputStream;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.ArrayList;
import java.util.HexFormat;
import java.util.List;
@Component
@RequiredArgsConstructor
public class WriteRequestDigest {
private final ObjectMapper json;
private final VibemakeProperties config;
public String commit(CommitInput input) {
return digest(List.of("commit-v1", input));
}
public String upload(String expected, String message, String directory,
List<MultipartFile> files, List<String> paths) {
if (message == null || files == null || files.isEmpty() || files.size() > 500
|| paths != null && paths.size() != files.size())
throw BusinessException.badRequest("文件和路径数量不合法");
List<Object> items = new ArrayList<>();
long total = 0;
for (int i = 0; i < files.size(); i++) {
MultipartFile file = files.get(i);
total += file.getSize();
if (file.getSize() > config.getMaxFileBytes() || total > config.getMaxRequestBytes())
throw new BusinessException(413, "FILE_TOO_LARGE", "文件或请求超过大小限制");
MessageDigest hash = sha256();
long bytes = 0;
try (InputStream stream = file.getInputStream()) {
byte[] buffer = new byte[8192];
int count;
while ((count = stream.read(buffer)) != -1) {
bytes += count;
if (bytes > config.getMaxFileBytes())
throw new BusinessException(413, "FILE_TOO_LARGE", "文件超过大小限制");
hash.update(buffer, 0, count);
}
} catch (IOException error) {
throw BusinessException.badRequest("读取上传文件失败");
}
items.add(List.of(FilePaths.file(paths == null ? file.getOriginalFilename() : paths.get(i)),
bytes, HexFormat.of().formatHex(hash.digest())));
}
return digest(List.of("upload-v1", FilePaths.sha(expected), message,
FilePaths.directory(directory), items));
}
private String digest(Object value) {
try {
return HexFormat.of().formatHex(sha256().digest(json.writeValueAsBytes(value)));
} catch (JsonProcessingException error) {
throw new IllegalStateException("无法生成仓库请求摘要", error);
}
}
private MessageDigest sha256() {
try {
return MessageDigest.getInstance("SHA-256");
} catch (NoSuchAlgorithmException error) {
throw new IllegalStateException(error);
}
}
}
@@ -0,0 +1,78 @@
package com.linkerbot.vibemake.service.gitea;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.linkerbot.vibemake.service.config.VibemakeProperties;
import com.sun.net.httpserver.HttpServer;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import java.net.InetSocketAddress;
import java.nio.charset.StandardCharsets;
import java.util.List;
import java.util.Map;
import java.util.concurrent.atomic.AtomicInteger;
import static org.assertj.core.api.Assertions.*;
class GiteaWriteReceiptTest {
@ParameterizedTest
@ValueSource(strings = {"valid", "wrongOperation", "wrongDigest", "wrongParent", "duplicate", "branchChanged", "readFailure"})
void receiptRequiresExactIdentityAndParentOnStableMainHistory(String mode) throws Exception {
String code = "a".repeat(32);
String digest = "b".repeat(64);
String base = "1".repeat(40);
String sha = "2".repeat(40);
String head = "3".repeat(40);
ObjectMapper json = new ObjectMapper();
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
AtomicInteger heads = new AtomicInteger();
server.createContext("/api/v1/repos/vibemake/repo/branches/main", exchange -> {
String value = mode.equals("branchChanged") && heads.incrementAndGet() > 1 ? "4".repeat(40) : head;
byte[] body = json.writeValueAsBytes(Map.of("commit", Map.of("id", value)));
exchange.sendResponseHeaders(200, body.length);
exchange.getResponseBody().write(body);
exchange.close();
});
server.createContext("/api/v1/repos/vibemake/repo/commits", exchange -> {
assertThat(exchange.getRequestMethod()).isEqualTo("GET");
assertThat(exchange.getRequestURI().getQuery()).contains("sha=" + head);
String message = "用户提交" + WriteReceipt.suffix(mode.equals("wrongOperation") ? "c".repeat(32) : code,
mode.equals("wrongDigest") ? "c".repeat(64) : digest);
Map<String, Object> match = Map.of("sha", sha, "commit", Map.of("message", message),
"parents", List.of(Map.of("sha", mode.equals("wrongParent") ? head : base)));
Map<String, Object> unrelated = Map.of("sha", head, "commit", Map.of("message", "另一次提交"),
"parents", List.of(Map.of("sha", sha)));
List<Object> entries = new java.util.ArrayList<>(List.of(unrelated, match));
if (mode.equals("duplicate")) entries.add(Map.of("sha", "5".repeat(40), "commit", Map.of("message", message),
"parents", List.of(Map.of("sha", base))));
byte[] body = mode.equals("readFailure") ? "failure".getBytes(StandardCharsets.UTF_8)
: json.writeValueAsBytes(entries);
exchange.sendResponseHeaders(mode.equals("readFailure") ? 503 : 200, body.length);
exchange.getResponseBody().write(body);
exchange.close();
});
server.start();
try {
VibemakeProperties config = new VibemakeProperties();
config.getGitea().setBaseUrl("http://127.0.0.1:" + server.getAddress().getPort());
config.getGitea().setToken("unit-test-placeholder");
RestGiteaClient client = new RestGiteaClient(config, json);
GiteaClient.Repository repo = new GiteaClient.Repository(1, "vibemake", "repo");
if (List.of("wrongParent", "duplicate", "branchChanged", "readFailure").contains(mode)) {
assertThatThrownBy(() -> client.findWriteReceipt(repo, base, code, digest))
.isInstanceOfSatisfying(GiteaFailure.class, error -> {
assertThat(error.uncertain()).isFalse();
assertThat(error.code()).isEqualTo(switch (mode) {
case "wrongParent" -> "WRITE_RECEIPT_PARENT_MISMATCH";
case "duplicate" -> "WRITE_RECEIPT_AMBIGUOUS";
case "branchChanged" -> "WRITE_RECEIPT_BRANCH_CHANGED";
default -> "GITEA_ERROR";
});
});
} else {
assertThat(client.findWriteReceipt(repo, base, code, digest))
.isEqualTo(mode.equals("valid") ? sha : null);
}
} finally {
server.stop(0);
}
}
}