项目编码缩短,github开源项目导入社区

This commit is contained in:
lxp
2026-10-15 16:48:30 +08:00
parent a1076d1098
commit 211f6c96f9
30 changed files with 2567 additions and 25 deletions
+2
View File
@@ -5,6 +5,8 @@ DB_PASSWORD='replace-with-your-database-password'
GITEA_BASE_URL=http://localhost:3000
GITEA_ORGANIZATION=community
GITEA_TOKEN='replace-with-your-gitea-token'
# 可选,仅供批量导入读取 GitHub 元数据;不能填写 Gitea Token。
GITHUB_TOKEN=''
# 默认 test,部署生产环境时改为 prod;连接信息也应对应生产环境。
SPRING_PROFILES_ACTIVE=test
# 与 cadworld 一致,默认日志目录为 logs,可改为部署环境中的绝对路径。
+34 -1
View File
@@ -15,6 +15,7 @@ Spring Boot 社区后端,工程分层参考 cadworld。使用 Java 21、Spring
| community-service | 业务 Service 接口在 service 根包,实现类在 service.impl;包含 mapper、config、gitea、ai、support |
| community-security | 平台登录会话、验证码、身份上下文、认证过滤器和安全配置 |
| community-common | 通用响应、分页、异常处理、requestId 和基础工具 |
| community-import | 固定 Excel 的 GitHub 项目运维导入命令,不提供 HTTP 接口 |
UserService、ProjectService、FileService、InteractionService、CoverUploadService 采用接口加实现类,Controller、安全模块和服务之间通过接口注入。GiteaClient 与 RestGiteaClient 定义仓库调用边界;项目锁、UTC 时间和分页检查归入 support。安全模块内的 TokenService、CaptchaService 保留具体类,首期沿用本机缓存。
@@ -75,7 +76,7 @@ SPRING_PROFILES_ACTIVE=prod java -Djava.awt.headless=true -jar community-api/tar
mvn clean verify
```
普通测试固定使用 test 配置,覆盖连接为临时 H2 MySQL 模式及模拟 Gitea/AI,验证登录、权限、提交冲突、发布快照、Fork、评论、点赞,以及 AI 筛选、引用校验、SSE、限流、超时和取消,不访问真实数据库、仓库或模型。完整安装验证可执行 `mvn clean install`。
普通测试固定使用 test 配置,覆盖连接为临时 H2 MySQL 模式及模拟 Gitea/AI,验证登录、权限、提交冲突、发布快照、Fork、评论、点赞,以及 AI 筛选、引用校验、SSE、限流、超时和取消,不访问真实数据库、仓库或模型。未加载外部服务凭据时,执行 `AI_ENABLED=false OSS_ENABLED=false mvn test`;AI 流程测试自行启用模拟模型。完整安装验证可执行 `AI_ENABLED=false OSS_ENABLED=false mvn clean install`。
真实 Gitea 联调:
@@ -125,3 +126,35 @@ python3 scripts/test-ai.py
创建或 Fork 前先写入 CREATING 项目,成功后标记 READY。结果不明或数据库回写失败需要人工核对;不提供后台自动恢复、无条件自动重试或跨重启完整幂等。具体恢复方式见接口文档。
## 项目短编码与 GitHub 初始化导入
普通创建、Fork 和导入的新项目使用 12 位小写项目编码,字符集为 `0123456789abcdefghjkmnpqrstvwxyz`。普通项目用安全随机数生成,项目编码唯一约束冲突最多重试 5 次;其他约束错误不重试。导入编码由 `github-import:v1:<GitHub仓库ID>` 的 SHA-256 前 60 位确定,避免重跑或仓库更名后重复创建。Gitea 仓库名称为 `p_{projectCode}`。历史 32 位编码、仓库及链接继续使用,数据库无需改表。
导入工具仅接受以下固定 Sheet:`1_轮式机器人`、`2_四足机器人`、`3_小人形机器人`、`4_大人形机器人`、`5_机械臂`、`6_其他类型`、`7_开源硬件`。只选 I 列精确为“可访问”、L 列有合法 HTTPS 封面、B 列为具体 GitHub 仓库的行。其他 Sheet 不参与导入;K 列图片公式不会执行。原始 A~H 数据列与表头存在错位,工具按实际位置读取,忽略 H 列的无意义备注 `100`,不将“待查”等信息改写成已确认。
封面保留原 URL 和 GIF,不转存 OSS。描述由原中文简介和技术资料组合,不调用 AI。类型沿用现有配置:轮式、四足、人形分别映射对应类型,机械臂、其他和硬件映射 GENERAL,原始分类保留在描述里。
脚本加载本项目 `.env`,依赖 Java 21、Maven,以及正式执行时可用的 MySQL 和 Gitea。独立导入模块不启动 Web 服务,也不加载登录、AI 或 OSS 组件。Gitea 技术账号需具备组织所有者及仓库写权限,实例需开启迁移、HTTP Git 和 LFS。`GITHUB_TOKEN` 可选,仅用于 GitHub 元数据 API;API 限流时读取同一公开仓库的 GitHub 网页,交叉核验仓库 ID、规范地址、公开状态和默认分支,缺失或不一致则预检失败。网页和图片请求均不携带凭据。`GITEA_TOKEN` 仅访问配置的 Gitea,不能代替 `GITHUB_TOKEN`;切勿把 Token 放进 URL 或命令参数。
```bash
IMPORT_EXCEL='/绝对路径/项目.xlsx'
# 仅验证静态筛选,不访问网络、业务库或 Gitea。
bash scripts/import-github-projects.sh --excel "$IMPORT_EXCEL" --offline
# 默认 dry-run:在线检查仓库和图片,并生成清单;不写业务库或 Gitea。
bash scripts/import-github-projects.sh --excel "$IMPORT_EXCEL" --dry-run
# 先试导前三个尚未完成的候选。必须使用已有有效用户的 userCode。
bash scripts/import-github-projects.sh --excel "$IMPORT_EXCEL" --execute --owner-user-code '<用户编码>' --limit 3
# 试导检查通过后,用相同状态目录续跑其余项目。
bash scripts/import-github-projects.sh --excel "$IMPORT_EXCEL" --execute --owner-user-code '<用户编码>'
```
`--owner-email` 可替代 `--owner-user-code` 查找已有账号,二者互斥,工具不会创建用户;状态中始终保存查到的用户编码。`--state-dir` 默认 `logs/github-import`,包含原子写入的 `manifest.json` 和 `report.json`;此目录必须保留并备份,不能在迁移进行中删除。清单绑定 Excel 指纹、归属用户和 Gitea 目标,不能复用于另一文件或环境。同一目录有进程锁,`--concurrency` 默认为 2,可选 1~4;同一批次只运行一个进程。`--limit` 限制尚未完成候选的处理数量;已完成项目、迁移结果未知或不具备收据核对条件的 REVIEW 项目不计入。
工具先创建 CREATING 项目记录,直接调用 Gitea `/repos/migrate`,不先创建空仓库。迁移使用私有普通仓库、开启 LFS,保留 Git 内容、历史、分支和标签;不迁移工单、PR、Wiki 或 Release 附件。迁移成功收据先落盘,再关联项目并自动发布。独立迁移请求默认超时 1800 秒,可用 `--migration-timeout-seconds` 调整,普通 Gitea 请求超时不变。原默认分支不是 main 时补建 main 并设为默认,保留原分支;同名 main 内容冲突时停止该项目,不覆盖源分支或已冻结版本。
进程在 MIGRATING 阶段中断,或迁移超时、结果不明确时,项目进入 REVIEW,禁止自动重发或仅凭仓库存在判定成功。需要核对 Gitea 迁移及 LFS 完成情况、来源和提交后由维护人员处理;不要直接把 REVIEW 改为 READY 或删除仓库再重跑。已确认的迁移收据会用于数据库回写或发布失败后的续跑,避免重复迁移。对于已持久化成功收据、仅在主分支适配时出现不明写入结果的项目,续跑先只读核对仓库身份、来源、私有属性、默认分支和两条分支的冻结 SHA;全部一致才继续关联和发布,否则保持 REVIEW,不重发分支写入。已完成项目不会因后续人工撤回、修改或删除而被批次重新发布。
报告逐项包含 Sheet、行号、短编码、阶段、错误码和兼容性提醒。退出码 0 表示本次选定工作完成,1 表示命令、配置或持久化等致命错误,2 表示存在预检未通过、失败或待核对项目。在线仓库或图片失效会减少实际导入量;报告中的候选数仍与静态筛选对账。
本次只完成仓库迁移、关联和发布。社区仍有单文件读取限制,LFS 实体读取、子模块及符号链接的浏览/Fork 能力没有扩展;当前发布快照检测到的限制会写入报告。子模块只保留引用,不递归迁移独立仓库,也不下载网盘或其他外链资源。保留所有分支不代表社区提供分支切换界面。
@@ -12,6 +12,7 @@ import com.linkerbot.community.bean.entity.Project;
import com.linkerbot.community.bean.entity.User;
import com.linkerbot.community.common.BusinessException;
import com.linkerbot.community.common.Codes;
import com.linkerbot.community.common.ProjectCodes;
import com.linkerbot.community.security.TokenService;
import com.linkerbot.community.service.ai.AiModelClient;
import com.linkerbot.community.service.config.AiProperties;
@@ -99,7 +100,7 @@ class AiSearchTest {
private Project project(String name, String type, String description) {
Project project = new Project();
project.setProjectCode(Codes.next());
project.setProjectCode(ProjectCodes.next());
project.setOwnerUserId(user.getId());
project.setName(name);
project.setDescription(description);
@@ -165,6 +166,25 @@ class AiSearchTest {
return events.getLast();
}
@Test void mixedLegacyAndShortCodesSupportFollowupReferences() throws Exception {
Project current = project("短编码机器人", "GENERAL", "支持教学");
Project legacy = project("旧编码机器人", "GENERAL", "支持教学");
legacy.setProjectCode(Codes.next());
legacy.setGiteaRepoName("p_" + legacy.getProjectCode());
projects.updateById(legacy);
model.handler = (name, input) -> {
if (!name.endsWith("intent")) return rank(input);
assertThat(input.path("previousProjects").size()).isEqualTo(2);
ObjectNode intent = intent(null, List.of());
intent.withArray("focusProjectCodes").add(legacy.getProjectCode());
return intent;
};
List<JsonNode> events = finish(start(Map.of("message", "第二个项目支持什么", "previousProjectCodes",
List.of(current.getProjectCode(), legacy.getProjectCode()))));
assertThat(events.getLast().path("data").path("projects").get(0).path("project").path("projectCode").asText())
.isEqualTo(legacy.getProjectCode()).hasSize(32);
}
@ParameterizedTest
@ValueSource(strings = {"GENERAL", "HUMANOID", "QUADRUPED", "WHEELED"})
void filtersConfiguredTypesAndKeepsProjectContract(String type) throws Exception {
@@ -70,7 +70,8 @@ class GiteaLiveTest {
void removeOnlyRepositoriesCreatedByThisTest() throws Exception {
HttpClient http = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
for (String repo : jdbc.queryForList("SELECT gitea_repo_name FROM project", String.class)) {
if (!repo.matches("p_[0-9a-f]{32}")) throw new IllegalStateException("拒绝清理非测试仓库");
if (!repo.matches("p_([0-9a-f]{32}|[0-9abcdefghjkmnpqrstvwxyz]{12})"))
throw new IllegalStateException("拒绝清理非测试仓库");
HttpRequest request =
HttpRequest.newBuilder(
URI.create(
@@ -171,6 +171,27 @@ class WorkflowTest {
200);
}
@Test
void shortProjectsAndLegacyProjectsRemainUsableTogether() throws Exception {
String owner = login("short-owner@example.com");
String code = create(owner, "旧项目");
assertThat(code).matches("[0-9abcdefghjkmnpqrstvwxyz]{12}");
assertThat(jdbc.queryForObject("SELECT project_code FROM project WHERE project_code = ?", String.class, code)).isEqualTo(code);
String legacy = "abcdef0123456789abcdef0123456789";
jdbc.update("UPDATE project SET project_code = ?, gitea_repo_name = ? WHERE project_code = ?", legacy, "p_" + legacy, code);
String sha = head(owner, legacy);
publish(owner, legacy, sha);
assertThat(perform(get("/api/community/projects/" + legacy), 200).path("data").path("projectCode").asText()).isEqualTo(legacy);
JsonNode copy = perform(body(auth(post("/api/community/projects/" + legacy + "/forks"), owner),
Map.of("name", "新 Fork", "publishedCommitSha", sha)), 200).path("data");
assertThat(copy.path("projectCode").asText()).matches("[0-9abcdefghjkmnpqrstvwxyz]{12}");
assertThat(copy.path("sourceProjectCode").asText()).isEqualTo(legacy);
perform(auth(put("/api/community/projects/" + legacy + "/like"), owner), 200);
JsonNode comment = perform(body(auth(post("/api/community/projects/" + legacy + "/comments"), owner), Map.of("content", "兼容测试")), 200).path("data");
assertThat(comment.path("commentCode").asText()).hasSize(32);
assertThat(copy.path("owner").path("userCode").asText()).hasSize(32);
}
@Test
void coverRoundTripsThroughMetadataPublicationAndForkWithoutFileCommit() throws Exception {
String owner = login("cover-owner@example.com");
@@ -0,0 +1,41 @@
package com.linkerbot.community.common;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.security.SecureRandom;
/** 仅用于项目编码;旧的 UUID 编码继续有效。 */
public final class ProjectCodes {
public static final String ALPHABET = "0123456789abcdefghjkmnpqrstvwxyz";
private static final SecureRandom RANDOM = new SecureRandom();
private ProjectCodes() {}
public static String next() {
StringBuilder code = new StringBuilder(12);
for (int i = 0; i < 12; i++) code.append(ALPHABET.charAt(RANDOM.nextInt(32)));
return code.toString();
}
public static String github(long repositoryId) {
if (repositoryId <= 0) throw new IllegalArgumentException("GitHub 仓库 ID 必须为正数");
try {
byte[] hash =
MessageDigest.getInstance("SHA-256")
.digest(
("github-import:v1:" + repositoryId)
.getBytes(StandardCharsets.UTF_8));
StringBuilder code = new StringBuilder(12);
for (int i = 0; i < 12; i++) {
int offset = i * 5;
int pair = (hash[offset / 8] & 255) << 8;
if (offset / 8 + 1 < hash.length) pair |= hash[offset / 8 + 1] & 255;
code.append(ALPHABET.charAt((pair >>> (11 - offset % 8)) & 31));
}
return code.toString();
} catch (NoSuchAlgorithmException error) {
throw new IllegalStateException("SHA-256 不可用", error);
}
}
}
@@ -0,0 +1,40 @@
package com.linkerbot.community.common;
import static org.assertj.core.api.Assertions.*;
import org.junit.jupiter.api.Test;
import java.math.BigInteger;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.util.HashSet;
import java.util.Set;
class ProjectCodesTest {
@Test
void randomCodesAndOtherIdentifiersStaySeparate() {
Set<String> codes = new HashSet<>();
for (int i = 0; i < 10000; i++) {
String code = ProjectCodes.next();
assertThat(code).matches("[0-9abcdefghjkmnpqrstvwxyz]{12}");
assertThat(codes.add(code)).isTrue();
}
assertThat(Codes.next()).matches("[a-f0-9]{32}");
}
@Test
void importUsesExactlyFirstSixtySha256Bits() throws Exception {
byte[] hash =
MessageDigest.getInstance("SHA-256")
.digest("github-import:v1:12345".getBytes(StandardCharsets.UTF_8));
BigInteger firstSixty = new BigInteger(1, hash).shiftRight(196);
StringBuilder expected = new StringBuilder();
for (int i = 11; i >= 0; i--)
expected.append(
ProjectCodes.ALPHABET.charAt(firstSixty.shiftRight(i * 5).intValue() & 31));
assertThat(ProjectCodes.github(12345)).isEqualTo(expected.toString());
assertThat(ProjectCodes.github(12345)).isNotEqualTo(ProjectCodes.github(12346));
assertThatThrownBy(() -> ProjectCodes.github(0))
.isInstanceOf(IllegalArgumentException.class);
}
}
+12
View File
@@ -0,0 +1,12 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent><groupId>com.linkerbot</groupId><artifactId>community</artifactId><version>0.0.1-SNAPSHOT</version></parent>
<artifactId>community-import</artifactId>
<dependencies>
<dependency><groupId>com.linkerbot</groupId><artifactId>community-service</artifactId><version>${project.version}</version></dependency>
<dependency><groupId>com.mysql</groupId><artifactId>mysql-connector-j</artifactId><scope>runtime</scope></dependency>
<dependency><groupId>com.h2database</groupId><artifactId>h2</artifactId><scope>test</scope></dependency>
</dependencies>
<build><plugins><plugin><groupId>org.springframework.boot</groupId><artifactId>spring-boot-maven-plugin</artifactId></plugin></plugins></build>
</project>
@@ -0,0 +1,27 @@
package com.linkerbot.community.importer;
import com.linkerbot.community.service.config.CommunityProperties;
import com.linkerbot.community.service.config.MybatisConfiguration;
import com.linkerbot.community.service.gitea.RestGiteaClient;
import com.linkerbot.community.service.impl.ProjectServiceImpl;
import com.linkerbot.community.service.impl.UserServiceImpl;
import com.linkerbot.community.service.support.ProjectLocks;
import org.mybatis.spring.annotation.MapperScan;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Import;
/** 显式加载所需组件,避免扫描在线应用的安全、AI 和 OSS 模块。 */
@Configuration(proxyBeanMethods = false)
@EnableAutoConfiguration
@MapperScan("com.linkerbot.community.service.mapper")
@Import({
CommunityProperties.class,
MybatisConfiguration.class,
RestGiteaClient.class,
ProjectServiceImpl.class,
UserServiceImpl.class,
ProjectLocks.class
})
public class DatabaseConfiguration {}
@@ -0,0 +1,304 @@
package com.linkerbot.community.importer;
import static com.linkerbot.community.importer.ImportModels.*;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.linkerbot.community.bean.entity.User;
import com.linkerbot.community.common.BusinessException;
import com.linkerbot.community.common.ProjectCodes;
import com.linkerbot.community.service.ProjectService;
import com.linkerbot.community.service.config.CommunityProperties;
import com.linkerbot.community.service.gitea.*;
import com.linkerbot.community.service.mapper.ProjectMapper;
import com.linkerbot.community.service.mapper.UserMapper;
import org.springframework.boot.WebApplicationType;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.context.ConfigurableApplicationContext;
import java.io.IOException;
import java.nio.file.Path;
import java.time.Duration;
import java.util.*;
import java.util.concurrent.*;
public final class ImportApplication {
public record Options(
Path excel,
Path state,
String owner,
String ownerEmail,
boolean execute,
boolean offline,
int limit,
int concurrency,
Duration migrationTimeout) {
public static Options parse(String[] args) {
Map<String, String> values = new HashMap<>();
Set<String> flags = new HashSet<>();
Set<String> booleanOptions = Set.of("--execute", "--dry-run", "--offline");
Set<String> valuedOptions =
Set.of(
"--excel",
"--state-dir",
"--owner-user-code",
"--owner-email",
"--limit",
"--concurrency",
"--migration-timeout-seconds");
for (int i = 0; i < args.length; i++) {
String key = args[i];
if (values.containsKey(key) || flags.contains(key))
throw new IllegalArgumentException("参数重复:" + key);
if (booleanOptions.contains(key)) flags.add(key);
else if (valuedOptions.contains(key)
&& i + 1 < args.length
&& !args[i + 1].startsWith("--")) values.put(key, args[++i]);
else throw new IllegalArgumentException("不支持的参数或缺少参数值");
}
if (!values.containsKey("--excel")) throw new IllegalArgumentException("必须提供 --excel");
boolean execute = flags.contains("--execute");
if (execute && (flags.contains("--dry-run") || flags.contains("--offline")))
throw new IllegalArgumentException("--execute 不能与 --dry-run 或 --offline 同时使用");
String owner = values.getOrDefault("--owner-user-code", "");
String ownerEmail =
values.getOrDefault("--owner-email", "").trim().toLowerCase(Locale.ROOT);
if (!owner.isBlank() && !ownerEmail.isBlank())
throw new IllegalArgumentException("用户编码和邮箱只能提供一个");
if (execute && owner.isBlank() && ownerEmail.isBlank())
throw new IllegalArgumentException("正式执行必须提供 --owner-user-code 或 --owner-email");
int limit =
Integer.parseInt(
values.getOrDefault("--limit", String.valueOf(Integer.MAX_VALUE)));
int concurrency = Integer.parseInt(values.getOrDefault("--concurrency", "2"));
long seconds =
Long.parseLong(values.getOrDefault("--migration-timeout-seconds", "1800"));
if (limit < 1 || concurrency < 1 || concurrency > 4 || seconds < 1)
throw new IllegalArgumentException("数量、并发或超时参数不合法(并发范围 1~4)");
return new Options(
Path.of(values.get("--excel")),
Path.of(values.getOrDefault("--state-dir", "logs/github-import")),
owner,
ownerEmail,
execute,
flags.contains("--offline"),
limit,
concurrency,
Duration.ofSeconds(seconds));
}
}
public static void main(String[] args) {
if (args.length == 0 || Arrays.asList(args).contains("--help")) {
System.out.println(
"用法:--excel <文件> [--state-dir <目录>] [--dry-run | --execute (--owner-user-code"
+ " <编码> 或 --owner-email <已有邮箱>)] [--limit N] [--concurrency 2] [--offline]"
+ " [--migration-timeout-seconds 1800]");
return;
}
int status;
try {
status = run(Options.parse(args));
} catch (Exception error) {
String code =
error instanceof ImportFailure failure
? failure.code()
: error instanceof BusinessException business
? business.code()
: error.getClass().getSimpleName();
String detail =
error instanceof ImportFailure failure
? ":" + failure.getMessage()
: ";检查参数、配置及运行报告";
System.err.println("导入命令终止 code=" + code + detail);
status = 1;
}
if (status != 0) System.exit(status);
}
public static int run(Options options) throws Exception {
WorkbookData workbook = new WorkbookReader().read(options.excel());
try (StateStore store = new StateStore(options.state(), workbook)) {
System.out.printf(
"静态筛选 candidates=%d excluded=%d fingerprint=%s%n",
workbook.candidates().size(),
workbook.excluded().size(),
workbook.fingerprint());
if (options.offline()) {
store.report();
return 0;
}
CommunityProperties config = environmentConfig();
if (store.manifest.giteaBaseUrl != null
&& (!store.manifest.giteaBaseUrl.equals(config.getGitea().getBaseUrl())
|| !store.manifest.giteaOrganization.equals(
config.getGitea().getOwner())))
throw new ImportFailure("TARGET_CHANGED", "状态目录绑定的 Gitea 目标已变化");
GiteaClient remote = new RestGiteaClient(config, new ObjectMapper());
remote.checkMigrationSettings();
SourceChecks source = new SourceChecks(System.getenv("GITHUB_TOKEN"));
Set<Long> ids = new HashSet<>();
int selected = 0;
List<Entry> work = new ArrayList<>();
for (Entry entry : store.manifest.entries) {
if (entry.source != null && !ids.add(entry.source.id()))
throw new ImportFailure("DUPLICATE_SOURCE", "清单中 GitHub 仓库重复");
if (entry.phase == Phase.MIGRATING)
store.update(
entry,
e -> {
e.phase = Phase.REVIEW;
e.errorCode = "MIGRATION_RESULT_UNKNOWN";
e.message = "进程中断后的迁移结果未知,禁止自动重发";
});
if (entry.phase == Phase.PUBLISHED
|| entry.phase == Phase.REVIEW
&& !ImportCoordinator.canVerifyNormalization(entry)
|| selected >= options.limit()) continue;
selected++;
GitHubRepository found;
try {
found = source.repository(entry.candidate.repositoryUrl());
if (entry.source != null && found.id() != entry.source.id())
throw new ImportFailure("SOURCE_CHANGED", "来源仓库身份变化,需人工核对");
if (entry.source == null && !ids.add(found.id()))
throw new ImportFailure("DUPLICATE_SOURCE", "不同表格行指向同一个仓库");
source.cover(entry.candidate.coverUrl());
} catch (IOException | InterruptedException error) {
if (error instanceof InterruptedException) {
Thread.currentThread().interrupt();
throw error;
}
preflightFailure(store, entry, "SOURCE_NETWORK_ERROR");
store.report();
continue;
} catch (ImportFailure error) {
preflightFailure(store, entry, error.code());
store.report();
continue;
}
GitHubRepository verified = found;
store.update(
entry,
e -> {
if (e.source == null) e.source = verified;
e.projectCode = ProjectCodes.github(verified.id());
e.preflightPassed = true;
e.errorCode = null;
e.message = null;
if (e.phase == Phase.PENDING || e.phase == Phase.PREFLIGHT_FAILED)
e.phase = Phase.READY;
});
work.add(entry);
System.out.printf(
"预检通过 sheet=%s row=%d projectCode=%s%n",
entry.candidate.sheet(), entry.candidate.row(), entry.projectCode);
store.report();
}
store.report();
if (!options.execute()) return work.size() == selected ? 0 : 2;
try (ConfigurableApplicationContext context =
new SpringApplicationBuilder(DatabaseConfiguration.class)
.web(WebApplicationType.NONE)
.logStartupInfo(false)
.run()) {
UserMapper users = context.getBean(UserMapper.class);
LambdaQueryWrapper<User> ownerQuery = new LambdaQueryWrapper<>();
if (!options.owner().isBlank()) ownerQuery.eq(User::getUserCode, options.owner());
else ownerQuery.eq(User::getEmail, options.ownerEmail());
User owner = users.selectOne(ownerQuery);
if (owner == null || !"ACTIVE".equals(owner.getStatus()))
throw new ImportFailure("INVALID_OWNER", "归属用户不存在或不可用");
System.out.println("导入归属已核验 userCode=" + owner.getUserCode());
if (store.manifest.ownerUserCode != null
&& !store.manifest.ownerUserCode.equals(owner.getUserCode()))
throw new ImportFailure("OWNER_CHANGED", "批次归属用户已变化");
store.manifest.ownerUserCode = owner.getUserCode();
store.manifest.giteaBaseUrl = config.getGitea().getBaseUrl();
store.manifest.giteaOrganization = config.getGitea().getOwner();
store.save();
ImportCoordinator coordinator =
new ImportCoordinator(
context.getBean(ProjectMapper.class),
context.getBean(ProjectService.class),
context.getBean(GiteaClient.class),
context.getBean(CommunityProperties.class),
store,
owner,
options.migrationTimeout());
try (ExecutorService executor =
Executors.newFixedThreadPool(options.concurrency())) {
try {
CompletionService<Void> jobs = new ExecutorCompletionService<>(executor);
for (Entry entry : work)
jobs.submit(
() -> {
try {
coordinator.run(entry);
store.report();
} catch (IOException error) {
throw new java.io.UncheckedIOException(error);
}
return null;
});
for (int completed = 0; completed < work.size(); completed++)
jobs.take().get();
} finally {
executor.shutdownNow();
}
}
} finally {
store.report();
}
return store.manifest.entries.stream()
.anyMatch(
e ->
e.phase == Phase.FAILED
|| e.phase == Phase.REVIEW
|| e.phase == Phase.PREFLIGHT_FAILED)
? 2
: 0;
}
}
private static void preflightFailure(StateStore store, Entry entry, String code)
throws IOException {
store.update(
entry,
e -> {
boolean reviewing = e.phase == Phase.REVIEW;
e.preflightPassed = false;
e.errorCode = code;
e.message = "在线预检失败,请核对网络或来源后重跑";
e.phase =
reviewing
|| code.equals("SOURCE_CHANGED")
|| code.equals("DUPLICATE_SOURCE")
? Phase.REVIEW
: Phase.PREFLIGHT_FAILED;
});
System.out.printf(
"预检未通过 sheet=%s row=%d code=%s%n",
entry.candidate.sheet(), entry.candidate.row(), code);
}
private static CommunityProperties environmentConfig() {
CommunityProperties config = new CommunityProperties();
String baseUrl =
System.getenv()
.getOrDefault("GITEA_BASE_URL", "http://localhost:3000")
.replaceAll("/+$", "");
java.net.URI base = java.net.URI.create(baseUrl);
if (base.getUserInfo() != null
|| base.getQuery() != null
|| base.getFragment() != null
|| base.getHost() == null
|| !("http".equals(base.getScheme()) || "https".equals(base.getScheme())))
throw new ImportFailure("INVALID_GITEA_URL", "Gitea 基础地址不能包含凭据、查询或片段");
config.getGitea().setBaseUrl(baseUrl);
config.getGitea().setOwner(System.getenv().getOrDefault("GITEA_ORGANIZATION", "community"));
config.getGitea().setToken(System.getenv().getOrDefault("GITEA_TOKEN", ""));
return config;
}
}
@@ -0,0 +1,350 @@
package com.linkerbot.community.importer;
import static com.linkerbot.community.importer.ImportModels.*;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.linkerbot.community.bean.entity.Project;
import com.linkerbot.community.bean.entity.User;
import com.linkerbot.community.common.BusinessException;
import com.linkerbot.community.service.ProjectService;
import com.linkerbot.community.service.config.CommunityProperties;
import com.linkerbot.community.service.gitea.*;
import com.linkerbot.community.service.gitea.GiteaClient.RemoteRepository;
import com.linkerbot.community.service.mapper.ProjectMapper;
import com.linkerbot.community.service.support.ProjectInputs;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.time.*;
import java.time.temporal.ChronoUnit;
import java.util.*;
public final class ImportCoordinator {
private static final Logger log = LoggerFactory.getLogger(ImportCoordinator.class);
private final ProjectMapper projects;
private final ProjectService service;
private final GiteaClient gitea;
private final CommunityProperties config;
private final StateStore store;
private final User owner;
private final Duration timeout;
private final String batch;
public static boolean canVerifyNormalization(Entry entry) {
return entry.phase == Phase.REVIEW
&& entry.receipt != null
&& entry.targetSha != null
&& "NORMALIZING".equals(entry.failedPhase)
&& !"SOURCE_CHANGED".equals(entry.errorCode)
&& !"DUPLICATE_SOURCE".equals(entry.errorCode);
}
public ImportCoordinator(
ProjectMapper projects,
ProjectService service,
GiteaClient gitea,
CommunityProperties config,
StateStore store,
User owner,
Duration timeout) {
this.projects = projects;
this.service = service;
this.gitea = gitea;
this.config = config;
this.store = store;
this.owner = owner;
this.timeout = timeout;
this.batch =
store.manifest.fingerprint.substring(
0, Math.min(12, store.manifest.fingerprint.length()));
}
public void run(Entry entry) throws IOException {
long started = System.nanoTime();
Project project = null;
boolean owned = false;
try {
if (!entry.preflightPassed
|| entry.phase == Phase.REVIEW && !canVerifyNormalization(entry)
|| entry.phase == Phase.PUBLISHED) return;
ProjectInputs.validate(entry.candidate.input(), config);
String repoName = "p_" + entry.projectCode;
project =
projects.selectOne(
new LambdaQueryWrapper<Project>()
.eq(Project::getProjectCode, entry.projectCode));
if (project == null) {
if (entry.receipt != null
|| entry.reservedAt != null
&& entry.phase != Phase.RESERVING
&& !"RESERVING".equals(entry.failedPhase))
throw new ImportFailure("PROJECT_MISSING", "已有进度记录对应的项目丢失,需人工核对");
ensureAbsent(repoName);
store.update(
entry,
e -> {
if (e.reservedAt == null)
e.reservedAt =
LocalDateTime.now(Clock.systemUTC())
.truncatedTo(ChronoUnit.MILLIS)
.toString();
e.phase = Phase.RESERVING;
});
project = new Project();
project.setProjectCode(entry.projectCode);
project.setOwnerUserId(owner.getId());
project.setName(entry.candidate.name().trim());
project.setDescription(entry.candidate.description());
project.setCoverUrl(ProjectInputs.coverUrl(entry.candidate.coverUrl()));
project.setProjectType(entry.candidate.projectType());
project.setGiteaOwner(config.getGitea().getOwner());
project.setGiteaRepoName(repoName);
project.setRepoStatus("CREATING");
project.setPublishStatus("UNPUBLISHED");
project.setDeleted(false);
project.setCreatedAt(LocalDateTime.parse(entry.reservedAt));
project.setUpdatedAt(project.getCreatedAt());
projects.insert(project);
owned = true;
store.phase(entry, Phase.RESERVED);
log.info(
"导入项目记录创建 batch={} projectCode={} sheet={} row={}",
batch,
entry.projectCode,
entry.candidate.sheet(),
entry.candidate.row());
} else {
if (entry.reservedAt == null
|| Boolean.TRUE.equals(project.getDeleted())
|| !Objects.equals(project.getOwnerUserId(), owner.getId())
|| !Objects.equals(
project.getCreatedAt(), LocalDateTime.parse(entry.reservedAt))
|| !Objects.equals(project.getGiteaOwner(), config.getGitea().getOwner())
|| !Objects.equals(project.getGiteaRepoName(), repoName))
throw new ImportFailure("PROJECT_CONFLICT", "编码已被其他项目占用或项目归属变化,禁止接管");
owned = true;
if ("PUBLISHED".equals(project.getPublishStatus())) {
validateRemote(entry, gitea.inspect(repoName));
if (!Objects.equals(
project.getGiteaRepoId(),
entry.receipt == null ? null : entry.receipt.id())
|| !Objects.equals(project.getPublishedCommitSha(), entry.targetSha))
throw new ImportFailure("PUBLICATION_CONFLICT", "已发布项目与导入收据不一致");
store.phase(entry, Phase.PUBLISHED);
return;
}
}
if (entry.phase == Phase.REVIEW) {
RemoteRepository confirmed = gitea.inspect(repoName);
validateRemote(entry, confirmed);
boolean applied =
confirmed.id() == entry.receipt.id()
&& "main".equals(confirmed.defaultBranch());
try {
applied =
applied
&& entry.targetSha.equals(
gitea.branch(
confirmed.repository(),
entry.source.defaultBranch()))
&& entry.targetSha.equals(
gitea.branch(confirmed.repository(), "main"));
} catch (GiteaFailure missing) {
if (missing.status() != 404) throw missing;
applied = false;
}
if (!applied) {
store.update(
entry,
e -> {
e.errorCode = "NORMALIZATION_NOT_CONFIRMED";
e.message = "迁移收据已确认,但目标 main、默认分支或 SHA 尚未核对通过;不重发远端写入";
});
return;
}
store.phase(entry, Phase.BRANCH_READY);
log.info("分支适配结果已核对 batch={} projectCode={}", batch, entry.projectCode);
}
if (entry.receipt == null) {
ensureAbsent(repoName);
store.phase(entry, Phase.MIGRATING);
log.info(
"原生迁移开始 batch={} projectCode={} source={}",
batch,
entry.projectCode,
entry.source.url());
RemoteRepository receipt =
gitea.migrate(entry.source.url() + ".git", repoName, timeout);
// 收据必须先持久化;之后数据库或主分支处理失败可安全续跑。
store.update(
entry,
e -> {
e.receipt = receipt;
e.phase = Phase.MIGRATED;
});
log.info("原生迁移完成 batch={} projectCode={}", batch, entry.projectCode);
}
RemoteRepository remote = gitea.inspect(repoName);
validateRemote(entry, remote);
if (remote.id() != entry.receipt.id())
throw new ImportFailure("REPOSITORY_REPLACED", "仓库标识与迁移收据不一致");
store.phase(entry, Phase.NORMALIZING);
String sourceSha = gitea.branch(remote.repository(), entry.source.defaultBranch());
if (entry.targetSha != null && !entry.targetSha.equals(sourceSha))
throw new ImportFailure("SOURCE_BRANCH_CHANGED", "目标仓库的源分支已变化,禁止自动改写冻结版本");
if (entry.targetSha == null) store.update(entry, e -> e.targetSha = sourceSha);
String mainSha;
try {
mainSha = gitea.branch(remote.repository(), "main");
} catch (GiteaFailure missing) {
if (missing.status() != 404) throw missing;
gitea.createBranch(remote.repository(), "main", sourceSha);
mainSha = gitea.branch(remote.repository(), "main");
}
if (!sourceSha.equals(mainSha))
throw new ImportFailure("MAIN_BRANCH_CONFLICT", "main 与源默认分支内容不同,禁止覆盖");
if (!"main".equals(remote.defaultBranch()))
gitea.defaultBranch(remote.repository(), "main");
store.update(
entry,
e -> {
e.targetSha = sourceSha;
e.phase = Phase.BRANCH_READY;
});
project.setGiteaRepoId(remote.id());
project.setRepoStatus("READY");
project.setUpdatedAt(LocalDateTime.now(Clock.systemUTC()));
if (projects.updateById(project) != 1)
throw new ImportFailure("DATABASE_UPDATE_FAILED", "项目仓库关联写入失败");
store.phase(entry, Phase.LINKED);
List<String> warnings;
try {
warnings =
gitea.migrationWarnings(
remote.repository(), sourceSha, config.getMaxFileBytes());
} catch (RuntimeException warningFailure) {
warnings = List.of("兼容性扫描未完成;请人工检查大文件、LFS、子模块和符号链接");
}
List<String> checkedWarnings = warnings;
store.update(entry, e -> e.warnings = new ArrayList<>(checkedWarnings));
store.phase(entry, Phase.PUBLISHING);
service.publish(owner.getId(), entry.projectCode, sourceSha);
store.phase(entry, Phase.PUBLISHED);
log.info(
"导入发布完成 batch={} projectCode={} durationMs={}",
batch,
entry.projectCode,
(System.nanoTime() - started) / 1_000_000);
} catch (IOException persistenceFailure) {
// 状态落盘失败必须终止本次执行,不能接着发出新的远端写入。
log.error(
"导入状态落盘失败 batch={} projectCode={} stage={} errorType={}",
batch,
entry.projectCode,
entry.phase,
persistenceFailure.getClass().getSimpleName());
throw persistenceFailure;
} catch (RuntimeException error) {
String failedPhase =
entry.phase == Phase.REVIEW && entry.failedPhase != null
? entry.failedPhase
: entry.phase.name();
boolean review =
entry.phase == Phase.MIGRATING
|| entry.phase == Phase.REVIEW
|| error instanceof GiteaFailure failure && failure.uncertain();
String code =
error instanceof ImportFailure failure
? failure.code()
: error instanceof BusinessException business
? business.code()
: "IMPORT_INTERNAL_ERROR";
if (code.equals("REMOTE_ALREADY_EXISTS") || code.equals("PROJECT_MISSING"))
review = true;
if (entry.phase == Phase.MIGRATING
&& error instanceof GiteaFailure failure
&& !failure.uncertain()) {
try {
gitea.inspect("p_" + entry.projectCode);
} catch (GiteaFailure missing) {
if (missing.status() == 404) review = false;
}
}
if (owned && project != null && !"READY".equals(project.getRepoStatus())) {
project.setRepoStatus(review ? "CREATING" : "FAILED");
project.setUpdatedAt(LocalDateTime.now(Clock.systemUTC()));
try {
projects.updateById(project);
} catch (RuntimeException databaseFailure) {
log.error(
"导入失败状态回写异常 batch={} projectCode={} errorType={}",
batch,
entry.projectCode,
databaseFailure.getClass().getSimpleName());
}
}
String message =
error instanceof ImportFailure failure
? failure.getMessage()
: "导入阶段失败,按错误码和服务端日志核对";
boolean requiresReview = review;
store.update(
entry,
e -> {
e.phase = requiresReview ? Phase.REVIEW : Phase.FAILED;
e.errorCode = code;
e.failedPhase = failedPhase;
e.message = message;
});
String location =
error.getStackTrace().length == 0
? "unknown"
: error.getStackTrace()[0].toString();
log.error(
"导入异常 batch={} projectCode={} source={} stage={} code={} review={} errorType={}"
+ " location={} durationMs={}",
batch,
entry.projectCode,
entry.source.url(),
failedPhase,
code,
review,
error.getClass().getSimpleName(),
location,
(System.nanoTime() - started) / 1_000_000);
}
}
private void ensureAbsent(String name) {
try {
gitea.inspect(name);
} catch (GiteaFailure missing) {
if (missing.status() == 404) return;
throw missing;
}
throw new ImportFailure("REMOTE_ALREADY_EXISTS", "目标仓库已存在但没有已确认迁移收据,需人工核对");
}
private void validateRemote(Entry entry, RemoteRepository remote) {
String original;
try {
original = WorkbookReader.repositoryKey(remote.originalUrl());
} catch (RuntimeException invalid) {
throw new ImportFailure("SOURCE_CONFLICT", "目标仓库缺少合法来源信息");
}
if (!remote.owner().equals(config.getGitea().getOwner())
|| !remote.name().equals("p_" + entry.projectCode)
|| !remote.privateRepo()
|| remote.mirror()
|| remote.empty()
|| !(original.equals(WorkbookReader.repositoryKey(entry.source.url()))
|| original.equals(
WorkbookReader.repositoryKey(entry.candidate.repositoryUrl()))))
throw new ImportFailure("REPOSITORY_CONFLICT", "目标仓库归属、来源、可见性或内容不符合迁移要求");
}
}
@@ -0,0 +1,15 @@
package com.linkerbot.community.importer;
/** 仅携带允许输出的诊断信息,不保留远端响应或认证信息。 */
public final class ImportFailure extends RuntimeException {
private final String code;
public ImportFailure(String code, String message) {
super(message);
this.code = code;
}
public String code() {
return code;
}
}
@@ -0,0 +1,79 @@
package com.linkerbot.community.importer;
import com.linkerbot.community.bean.Contracts.ProjectInput;
import com.linkerbot.community.service.gitea.GiteaClient.RemoteRepository;
import java.util.ArrayList;
import java.util.List;
public final class ImportModels {
private ImportModels() {}
public record Candidate(
String sheet,
int row,
String name,
String repositoryUrl,
String description,
String projectType,
String coverUrl) {
public ProjectInput input() {
return new ProjectInput(name, description, projectType, coverUrl);
}
}
public record Excluded(String sheet, int row, String name, String reason) {}
public record WorkbookData(
String fingerprint, List<Candidate> candidates, List<Excluded> excluded) {}
public record GitHubRepository(long id, String url, String defaultBranch) {}
public enum Phase {
PENDING,
PREFLIGHT_FAILED,
READY,
RESERVING,
RESERVED,
MIGRATING,
MIGRATED,
NORMALIZING,
BRANCH_READY,
LINKED,
PUBLISHING,
PUBLISHED,
FAILED,
REVIEW
}
public static final class Entry {
public Candidate candidate;
public GitHubRepository source;
public String projectCode;
public Phase phase = Phase.PENDING;
public boolean preflightPassed;
public String reservedAt;
public RemoteRepository receipt;
public String targetSha;
public String errorCode;
public String failedPhase;
public String message;
public List<String> warnings = new ArrayList<>();
public Entry() {}
public Entry(Candidate candidate) {
this.candidate = candidate;
}
}
public static final class Manifest {
public int version = 1;
public String fingerprint;
public String ownerUserCode;
public String giteaBaseUrl;
public String giteaOrganization;
public List<Entry> entries = new ArrayList<>();
public List<Excluded> excluded = new ArrayList<>();
}
}
@@ -0,0 +1,206 @@
package com.linkerbot.community.importer;
import static com.linkerbot.community.importer.ImportModels.*;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.InputStream;
import java.net.URI;
import java.net.http.*;
import java.time.Duration;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public class SourceChecks {
private final HttpClient http =
HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(10))
.followRedirects(HttpClient.Redirect.NEVER)
.build();
private final ObjectMapper json = new ObjectMapper();
private final String token;
private boolean apiLimited;
public SourceChecks(String token) {
this.token = token == null ? "" : token;
}
public GitHubRepository repository(String url) throws Exception {
if (apiLimited) return publicPage(url);
URI uri = URI.create("https://api.github.com/repos/" + WorkbookReader.repositoryKey(url));
for (int redirects = 0; redirects <= 5; redirects++) {
if (!"https".equalsIgnoreCase(uri.getScheme())
|| !"api.github.com".equalsIgnoreCase(uri.getHost())
|| uri.getUserInfo() != null
|| (uri.getPort() != -1 && uri.getPort() != 443))
throw new ImportFailure("GITHUB_REDIRECT", "GitHub API 跳转地址不符合要求");
HttpRequest.Builder request =
HttpRequest.newBuilder(uri)
.timeout(Duration.ofSeconds(30))
.header("Accept", "application/vnd.github+json")
.header("User-Agent", "community-import/1");
if (!token.isBlank()) request.header("Authorization", "Bearer " + token);
HttpResponse<String> response =
http.send(request.GET().build(), HttpResponse.BodyHandlers.ofString());
int status = response.statusCode();
if (status >= 300 && status < 400) {
uri = redirect(uri, response);
continue;
}
if (status == 403 || status == 429) {
apiLimited = true;
return publicPage(url);
}
if (status == 404) throw new ImportFailure("GITHUB_NOT_FOUND", "GitHub 仓库不可访问");
if (status != 200) throw new ImportFailure("GITHUB_HTTP_" + status, "GitHub 返回异常状态");
JsonNode data = json.readTree(response.body());
if (data.path("id").asLong() <= 0
|| data.path("private").asBoolean()
|| data.path("default_branch").asText().isBlank()
|| data.path("size").asLong(-1) < 0)
throw new ImportFailure("GITHUB_INVALID_REPOSITORY", "仓库元数据无效或不是公开仓库");
String canonical =
"https://github.com/"
+ WorkbookReader.repositoryKey(data.path("html_url").asText());
return new GitHubRepository(
data.path("id").asLong(), canonical, data.path("default_branch").asText());
}
throw new ImportFailure("GITHUB_REDIRECT", "GitHub 跳转次数过多");
}
private GitHubRepository publicPage(String url) throws Exception {
URI uri = URI.create("https://github.com/" + WorkbookReader.repositoryKey(url));
for (int redirects = 0; redirects <= 5; redirects++) {
if (!"https".equalsIgnoreCase(uri.getScheme())
|| !"github.com".equalsIgnoreCase(uri.getHost())
|| uri.getUserInfo() != null
|| (uri.getPort() != -1 && uri.getPort() != 443))
throw new ImportFailure("GITHUB_REDIRECT", "GitHub 网页跳转地址不符合要求");
HttpRequest request =
HttpRequest.newBuilder(uri)
.timeout(Duration.ofSeconds(30))
.header("Accept", "text/html")
.header("User-Agent", "community-import/1")
.GET()
.build();
// 网页请求不携带任何 GitHub 或 Gitea 凭据。
HttpResponse<String> response =
http.send(request, HttpResponse.BodyHandlers.ofString());
int status = response.statusCode();
if (status >= 300 && status < 400) {
uri = redirect(uri, response);
continue;
}
if (status == 403 || status == 429)
throw new ImportFailure("GITHUB_ACCESS_OR_RATE_LIMIT", "GitHub API 及公开网页均不可访问");
if (status != 200)
throw new ImportFailure("GITHUB_PAGE_HTTP_" + status, "GitHub 公开仓库网页不可访问");
return parsePublicPage(response.body(), uri.toString());
}
throw new ImportFailure("GITHUB_REDIRECT", "GitHub 网页跳转次数过多");
}
static GitHubRepository parsePublicPage(String html, String url) throws Exception {
String key = WorkbookReader.repositoryKey(url);
String id = pageMeta(html, "repository_id");
if (!id.matches("[1-9][0-9]*")
|| !key.equalsIgnoreCase(pageMeta(html, "repository_nwo"))
|| !"true".equals(pageMeta(html, "repository_public")))
throw new ImportFailure("GITHUB_PAGE_METADATA", "GitHub 网页仓库身份或公开状态无法核验");
Matcher scripts =
Pattern.compile(
"<script[^>]*data-target=\"react-app.embeddedData\"[^>]*>(.*?)</script>",
Pattern.DOTALL)
.matcher(html);
while (scripts.find()) {
JsonNode repo =
new ObjectMapper()
.readTree(scripts.group(1))
.path("payload")
.path("codeViewLayoutRoute")
.path("repo");
if (repo.path("id").asLong() != Long.parseLong(id)) continue;
String name = repo.path("ownerLogin").asText() + "/" + repo.path("name").asText();
if (!name.equalsIgnoreCase(key)
|| !repo.path("public").asBoolean()
|| !repo.path("private").isBoolean()
|| repo.path("private").asBoolean()
|| !repo.path("isEmpty").isBoolean()
|| repo.path("isEmpty").asBoolean()
|| repo.path("defaultBranch").asText().isBlank()) break;
return new GitHubRepository(
Long.parseLong(id),
"https://github.com/" + name,
repo.path("defaultBranch").asText());
}
throw new ImportFailure("GITHUB_PAGE_METADATA", "GitHub 网页缺少一致的仓库 ID、公开状态或默认分支");
}
private static String pageMeta(String html, String field) {
Matcher meta =
Pattern.compile(
"<meta name=\"octolytics-dimension-"
+ field
+ "\" content=\"([^\"]*)\"[^>]*>")
.matcher(html);
return meta.find() ? meta.group(1) : "";
}
public void cover(String url) throws Exception {
URI uri = URI.create(url);
for (int redirects = 0; redirects <= 5; redirects++) {
if (!"https".equalsIgnoreCase(uri.getScheme())
|| uri.getHost() == null
|| uri.getUserInfo() != null)
throw new ImportFailure("COVER_REDIRECT", "封面跳转地址必须为 HTTPS");
HttpRequest request =
HttpRequest.newBuilder(uri)
.timeout(Duration.ofSeconds(30))
.header("User-Agent", "community-import/1")
.header("Range", "bytes=0-511")
.GET()
.build();
HttpResponse<InputStream> response =
http.send(request, HttpResponse.BodyHandlers.ofInputStream());
try (InputStream body = response.body()) {
int status = response.statusCode();
if (status >= 300 && status < 400) {
uri = redirect(uri, response);
continue;
}
if (status != 200 && status != 206)
throw new ImportFailure("COVER_HTTP_" + status, "封面链接无法读取");
byte[] signature = body.readNBytes(16);
boolean image =
signature.length >= 3
&& ((signature[0] & 255) == 255 && (signature[1] & 255) == 216
|| signature.length >= 8
&& (signature[0] & 255) == 137
&& signature[1] == 'P'
&& signature[2] == 'N'
&& signature[3] == 'G'
|| signature[0] == 'G'
&& signature[1] == 'I'
&& signature[2] == 'F'
|| signature.length >= 12
&& signature[0] == 'R'
&& signature[1] == 'I'
&& signature[8] == 'W'
&& signature[9] == 'E');
if (!image)
throw new ImportFailure("COVER_NOT_IMAGE", "封面未返回 JPG、PNG、GIF 或 WebP 图片内容");
return;
}
}
throw new ImportFailure("COVER_REDIRECT", "封面跳转次数过多");
}
private URI redirect(URI uri, HttpResponse<?> response) {
return uri.resolve(
response.headers()
.firstValue("Location")
.orElseThrow(
() -> new ImportFailure("INVALID_REDIRECT", "跳转响应缺少 Location")));
}
}
@@ -0,0 +1,130 @@
package com.linkerbot.community.importer;
import static com.linkerbot.community.importer.ImportModels.*;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.channels.FileLock;
import java.nio.file.*;
import java.util.*;
import java.util.function.Consumer;
public final class StateStore implements AutoCloseable {
private final Path directory;
private final ObjectMapper json = new ObjectMapper().findAndRegisterModules();
private final FileChannel channel;
private final FileLock lock;
public final Manifest manifest;
public StateStore(Path directory, WorkbookData data) throws IOException {
this.directory = directory;
Files.createDirectories(directory);
channel =
FileChannel.open(
directory.resolve(".lock"),
StandardOpenOption.CREATE,
StandardOpenOption.WRITE);
FileLock acquired;
try {
acquired = channel.tryLock();
} catch (RuntimeException error) {
channel.close();
throw error;
}
if (acquired == null) {
channel.close();
throw new ImportFailure("BATCH_LOCKED", "同一状态目录已有导入进程");
}
lock = acquired;
try {
Path file = directory.resolve("manifest.json");
if (Files.exists(file)) {
manifest = json.readValue(file.toFile(), Manifest.class);
if (manifest.version != 1
|| !data.fingerprint().equals(manifest.fingerprint)
|| !data.candidates()
.equals(manifest.entries.stream().map(e -> e.candidate).toList())
|| !data.excluded().equals(manifest.excluded))
throw new ImportFailure("BATCH_CHANGED", "Excel 或清单内容已变化,请使用新的状态目录");
} else {
manifest = new Manifest();
manifest.fingerprint = data.fingerprint();
manifest.excluded.addAll(data.excluded());
data.candidates().forEach(c -> manifest.entries.add(new Entry(c)));
save();
}
} catch (IOException | RuntimeException error) {
lock.release();
channel.close();
throw error;
}
}
public synchronized void update(Entry entry, Consumer<Entry> change) throws IOException {
change.accept(entry);
save();
}
public synchronized void phase(Entry entry, Phase phase) throws IOException {
update(
entry,
e -> {
e.phase = phase;
e.errorCode = null;
e.message = null;
e.failedPhase = null;
});
}
public synchronized void save() throws IOException {
atomic("manifest.json", manifest);
}
public synchronized void report() throws IOException {
Map<String, Long> counts = new LinkedHashMap<>();
for (Phase phase : Phase.values())
counts.put(
phase.name(), manifest.entries.stream().filter(e -> e.phase == phase).count());
Map<String, Object> report = new LinkedHashMap<>();
report.put("fingerprint", manifest.fingerprint);
report.put("candidateCount", manifest.entries.size());
report.put("excludedCount", manifest.excluded.size());
report.put("counts", counts);
report.put("entries", manifest.entries);
report.put("excluded", manifest.excluded);
atomic("report.json", report);
}
private void atomic(String name, Object value) throws IOException {
Path temporary = directory.resolve(name + ".tmp");
byte[] bytes = json.writerWithDefaultPrettyPrinter().writeValueAsBytes(value);
try (FileChannel file =
FileChannel.open(
temporary,
StandardOpenOption.CREATE,
StandardOpenOption.TRUNCATE_EXISTING,
StandardOpenOption.WRITE)) {
ByteBuffer buffer = ByteBuffer.wrap(bytes);
while (buffer.hasRemaining()) file.write(buffer);
file.force(true);
}
// 不支持原子替换的状态目录必须停止执行,不能降级为可能损坏进度的普通覆盖。
Files.move(
temporary,
directory.resolve(name),
StandardCopyOption.ATOMIC_MOVE,
StandardCopyOption.REPLACE_EXISTING);
}
@Override
public void close() throws IOException {
try {
lock.release();
} finally {
channel.close();
}
}
}
@@ -0,0 +1,261 @@
package com.linkerbot.community.importer;
import static com.linkerbot.community.importer.ImportModels.*;
import com.linkerbot.community.service.support.ProjectInputs;
import java.io.InputStream;
import java.net.URI;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.MessageDigest;
import java.util.*;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import javax.xml.stream.XMLInputFactory;
import javax.xml.stream.XMLStreamConstants;
import javax.xml.stream.XMLStreamReader;
/** 此处只支持经确认的固定工作簿布局;不执行公式或文档中的指令。 */
public final class WorkbookReader {
public static final Map<String, String> SHEETS;
static {
LinkedHashMap<String, String> types = new LinkedHashMap<>();
types.put("1_轮式机器人", "WHEELED");
types.put("2_四足机器人", "QUADRUPED");
types.put("3_小人形机器人", "HUMANOID");
types.put("4_大人形机器人", "HUMANOID");
types.put("5_机械臂", "GENERAL");
types.put("6_其他类型", "GENERAL");
types.put("7_开源硬件", "GENERAL");
SHEETS = Collections.unmodifiableMap(types);
}
private final XMLInputFactory xml = XMLInputFactory.newFactory();
public WorkbookReader() {
xml.setProperty(XMLInputFactory.SUPPORT_DTD, false);
xml.setProperty("javax.xml.stream.isSupportingExternalEntities", false);
}
private XMLStreamReader open(ZipFile zip, String name) throws Exception {
ZipEntry entry = zip.getEntry(name);
if (entry == null || entry.getSize() > 32L * 1024 * 1024)
throw new IllegalArgumentException("Excel XML 缺失或过大:" + name);
return xml.createXMLStreamReader(zip.getInputStream(entry));
}
public WorkbookData read(Path path) throws Exception {
String fingerprint;
try (InputStream input = Files.newInputStream(path)) {
MessageDigest digest = MessageDigest.getInstance("SHA-256");
byte[] buffer = new byte[8192];
int count;
while ((count = input.read(buffer)) != -1) digest.update(buffer, 0, count);
fingerprint = HexFormat.of().formatHex(digest.digest());
}
List<Candidate> candidates = new ArrayList<>();
List<Excluded> excluded = new ArrayList<>();
try (ZipFile zip = new ZipFile(path.toFile())) {
List<String> strings = sharedStrings(zip);
Map<String, String> relationships = new HashMap<>();
XMLStreamReader rel = open(zip, "xl/_rels/workbook.xml.rels");
while (rel.hasNext()) {
if (rel.next() == XMLStreamConstants.START_ELEMENT
&& rel.getLocalName().equals("Relationship"))
relationships.put(
rel.getAttributeValue(null, "Id"),
rel.getAttributeValue(null, "Target"));
}
rel.close();
Map<String, String> sheets = new HashMap<>();
XMLStreamReader book = open(zip, "xl/workbook.xml");
while (book.hasNext()) {
if (book.next() == XMLStreamConstants.START_ELEMENT
&& book.getLocalName().equals("sheet")) {
String name = book.getAttributeValue(null, "name");
if (!SHEETS.containsKey(name)) continue;
String id =
book.getAttributeValue(
"http://schemas.openxmlformats.org/officeDocument/2006/relationships",
"id");
String target = relationships.get(id);
if (target == null) throw new IllegalArgumentException("Sheet 关联缺失:" + name);
String resolved =
target.startsWith("/")
? target.substring(1)
: Path.of("xl")
.resolve(target)
.normalize()
.toString()
.replace('\\', '/');
if (!resolved.startsWith("xl/"))
throw new IllegalArgumentException("非法 Sheet 路径");
if (sheets.put(name, resolved) != null)
throw new IllegalArgumentException("重复 Sheet:" + name);
}
}
book.close();
if (!sheets.keySet().equals(SHEETS.keySet()))
throw new IllegalArgumentException("缺少固定的七个 Sheet,停止导入");
for (String sheet : SHEETS.keySet())
readSheet(zip, sheets.get(sheet), sheet, strings, candidates, excluded);
}
return new WorkbookData(fingerprint, List.copyOf(candidates), List.copyOf(excluded));
}
private List<String> sharedStrings(ZipFile zip) throws Exception {
List<String> strings = new ArrayList<>();
if (zip.getEntry("xl/sharedStrings.xml") == null) return strings;
XMLStreamReader reader = open(zip, "xl/sharedStrings.xml");
StringBuilder text = new StringBuilder();
while (reader.hasNext()) {
int event = reader.next();
if (event == XMLStreamConstants.START_ELEMENT && reader.getLocalName().equals("si"))
text.setLength(0);
else if (event == XMLStreamConstants.START_ELEMENT && reader.getLocalName().equals("t"))
text.append(reader.getElementText());
else if (event == XMLStreamConstants.END_ELEMENT && reader.getLocalName().equals("si"))
strings.add(text.toString());
}
reader.close();
return strings;
}
private void readSheet(
ZipFile zip,
String path,
String sheet,
List<String> strings,
List<Candidate> candidates,
List<Excluded> excluded)
throws Exception {
XMLStreamReader reader = open(zip, path);
Map<String, String> cells = new HashMap<>();
int row = 0;
String column = "";
String type = "";
StringBuilder value = new StringBuilder();
boolean headersValid = false;
while (reader.hasNext()) {
int event = reader.next();
String tag =
event == XMLStreamConstants.START_ELEMENT
|| event == XMLStreamConstants.END_ELEMENT
? reader.getLocalName()
: "";
if (event == XMLStreamConstants.START_ELEMENT) {
switch (tag) {
case "row" -> {
row = Integer.parseInt(reader.getAttributeValue(null, "r"));
cells.clear();
}
case "c" -> {
column = reader.getAttributeValue(null, "r").replaceAll("[0-9]", "");
type = reader.getAttributeValue(null, "t");
value.setLength(0);
}
case "v", "t" -> value.append(reader.getElementText());
default -> {}
}
} else if (event == XMLStreamConstants.END_ELEMENT && tag.equals("c")) {
String text = value.toString();
if ("s".equals(type) && !text.isEmpty()) text = strings.get(Integer.parseInt(text));
cells.put(column, text.trim());
} else if (event == XMLStreamConstants.END_ELEMENT && tag.equals("row")) {
if (row == 2) {
headersValid =
"GitHub链接状态".equals(cells.get("I"))
&& "封面图片 URL".equals(cells.get("L"));
if (!headersValid) throw new IllegalArgumentException("Sheet 布局不符合约定:" + sheet);
} else if (row >= 3 && !cells.values().stream().allMatch(String::isBlank)) {
if (!headersValid)
throw new IllegalArgumentException("Sheet 缺少第 2 行表头:" + sheet);
addRow(sheet, row, cells, candidates, excluded);
}
}
}
reader.close();
if (!headersValid) throw new IllegalArgumentException("Sheet 没有有效表头:" + sheet);
}
private void addRow(
String sheet,
int row,
Map<String, String> cells,
List<Candidate> candidates,
List<Excluded> excluded) {
String name = cells.getOrDefault("A", "");
String reason = null;
if (!"可访问".equals(cells.get("I"))) reason = "LINK_STATUS_NOT_ACCESSIBLE";
else if (cells.getOrDefault("L", "").isBlank()) reason = "NO_COVER";
else {
try {
ProjectInputs.coverUrl(cells.get("L"));
repositoryKey(cells.get("B"));
} catch (RuntimeException invalid) {
reason = "INVALID_URL";
}
}
if (name.isBlank() || name.length() > 100) reason = "INVALID_NAME";
if (reason != null) {
excluded.add(new Excluded(sheet, row, name, reason));
return;
}
StringBuilder description =
new StringBuilder("原始分类:")
.append(sheet.substring(2))
.append("\n简介:")
.append(cells.getOrDefault("C", ""));
for (Map.Entry<String, String> field :
List.of(
Map.entry("D", "CAD 文件类型"),
Map.entry("E", "类型/尺寸"),
Map.entry("F", "URDF 情况"),
Map.entry("G", "资料验证状态")))
description
.append('\n')
.append(field.getValue())
.append(":")
.append(cells.getOrDefault(field.getKey(), "未提供"));
String remark = cells.getOrDefault("H", "");
if (!remark.isBlank() && !remark.equals("100")) description.append("\n备注:").append(remark);
String source = "\n来源:" + cells.get("B");
String body = description.toString();
if (body.length() + source.length() > 2000)
body = body.substring(0, 2000 - source.length() - 1) + "…";
candidates.add(
new Candidate(
sheet,
row,
name,
cells.get("B"),
body + source,
SHEETS.get(sheet),
cells.get("L")));
}
public static String repositoryKey(String url) {
if (url == null) throw new IllegalArgumentException("仓库地址为空");
URI uri = URI.create(url.trim());
String path = uri.getPath() == null ? "" : uri.getPath().replaceAll("/$", "");
if (!"https".equalsIgnoreCase(uri.getScheme())
|| !"github.com".equalsIgnoreCase(uri.getHost())
|| uri.getUserInfo() != null
|| (uri.getPort() != -1 && uri.getPort() != 443)
|| uri.getQuery() != null
|| uri.getFragment() != null
|| !path.matches("/[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+"))
throw new IllegalArgumentException("必须是 GitHub 仓库根地址");
if (path.endsWith(".git")) path = path.substring(0, path.length() - 4);
String[] parts = path.substring(1).split("/");
if (parts.length != 2
|| parts[0].equals(".")
|| parts[0].equals("..")
|| parts[1].equals(".")
|| parts[1].equals("..")) throw new IllegalArgumentException("仓库地址不合法");
return path.substring(1).toLowerCase(Locale.ROOT);
}
}
@@ -0,0 +1,28 @@
spring:
main:
web-application-type: none
banner-mode: off
sql:
init:
mode: never
datasource:
url: ${DB_URL:jdbc:mysql://localhost:3306/community?useUnicode=true&characterEncoding=utf8&serverTimezone=UTC}
username: ${DB_USERNAME:root}
password: ${DB_PASSWORD:}
driver-class-name: com.mysql.cj.jdbc.Driver
hikari:
maximum-pool-size: 4
connection-timeout: 5000
mybatis-plus:
configuration:
map-underscore-to-camel-case: true
community:
gitea:
base-url: ${GITEA_BASE_URL:http://localhost:3000}
owner: ${GITEA_ORGANIZATION:community}
token: ${GITEA_TOKEN:}
logging:
level:
root: WARN
com.linkerbot.community: INFO
com.linkerbot.community.service.mapper: WARN
@@ -0,0 +1,332 @@
package com.linkerbot.community.importer;
import static com.linkerbot.community.importer.ImportModels.*;
import static org.assertj.core.api.Assertions.*;
import static org.mockito.ArgumentMatchers.*;
import static org.mockito.Mockito.*;
import com.linkerbot.community.bean.entity.Project;
import com.linkerbot.community.bean.entity.User;
import com.linkerbot.community.common.*;
import com.linkerbot.community.service.ProjectService;
import com.linkerbot.community.service.config.CommunityProperties;
import com.linkerbot.community.service.gitea.*;
import com.linkerbot.community.service.mapper.*;
import org.junit.jupiter.api.*;
import org.junit.jupiter.api.io.TempDir;
import org.mockito.AdditionalAnswers;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.context.annotation.*;
import org.springframework.jdbc.core.JdbcTemplate;
import java.nio.file.*;
import java.time.*;
import java.util.*;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.regex.*;
@SpringBootTest(
classes = {DatabaseConfiguration.class, ImportCoordinatorTest.Fakes.class},
properties = {
"spring.datasource.url=jdbc:h2:mem:importer;MODE=MySQL;DB_CLOSE_DELAY=-1",
"spring.datasource.username=sa",
"spring.datasource.password=",
"spring.datasource.driver-class-name=org.h2.Driver",
"spring.main.web-application-type=none",
"logging.level.root=WARN"
})
class ImportCoordinatorTest {
@Configuration(proxyBeanMethods = false)
static class Fakes {
@Bean
@Primary
GiteaClient fake() {
return mock(GiteaClient.class);
}
}
@Autowired ProjectMapper projects;
@Autowired UserMapper users;
@Autowired ProjectService service;
@Autowired GiteaClient gitea;
@Autowired CommunityProperties config;
@Autowired JdbcTemplate jdbc;
@TempDir Path directory;
private StateStore store;
private Entry entry;
private User owner;
private GiteaClient.RemoteRepository remote;
private AtomicBoolean exists;
private AtomicBoolean main;
private static final String SHA = "a".repeat(40);
@BeforeEach
void prepare() throws Exception {
Path schema = Path.of("../docs/database.md");
Matcher tables =
Pattern.compile("CREATE TABLE [\\s\\S]*?;").matcher(Files.readString(schema));
jdbc.execute("DROP ALL OBJECTS");
while (tables.find())
jdbc.execute(
tables.group()
.replaceAll(" COMMENT '[^']*'", "")
.replaceAll("\\) ENGINE=InnoDB[^;]*;", ");"));
owner = new User();
owner.setUserCode(Codes.next());
owner.setEmail("owner@example.invalid");
owner.setNickname("整理账号");
owner.setStatus("ACTIVE");
owner.setCreatedAt(LocalDateTime.now());
owner.setUpdatedAt(owner.getCreatedAt());
users.insert(owner);
Candidate candidate =
new Candidate(
"1_轮式机器人",
3,
"迁移项目",
"https://github.com/example/robot",
"中文简介;URDF:待查",
"WHEELED",
"https://images.example.com/a.gif");
store =
new StateStore(
directory.resolve("state"),
new WorkbookData("fingerprint", List.of(candidate), List.of()));
entry = store.manifest.entries.getFirst();
entry.source = new GitHubRepository(12345, candidate.repositoryUrl(), "master");
entry.projectCode = ProjectCodes.github(12345);
entry.preflightPassed = true;
entry.phase = Phase.READY;
store.save();
remote =
new GiteaClient.RemoteRepository(
77,
"community",
"p_" + entry.projectCode,
true,
false,
false,
candidate.repositoryUrl() + ".git",
"master");
reset(gitea);
exists = new AtomicBoolean();
main = new AtomicBoolean();
when(gitea.inspect(anyString()))
.thenAnswer(
call -> {
if (!exists.get())
throw new GiteaFailure(404, "NOT_FOUND", "不存在", false);
return remote;
});
when(gitea.migrate(anyString(), anyString(), any(Duration.class)))
.thenAnswer(
call -> {
exists.set(true);
return remote;
});
when(gitea.branch(any(), anyString()))
.thenAnswer(
call -> {
if (call.getArgument(1).equals("main") && !main.get())
throw new GiteaFailure(404, "NOT_FOUND", "无 main", false);
return SHA;
});
doAnswer(
call -> {
main.set(true);
return null;
})
.when(gitea)
.createBranch(any(), eq("main"), eq(SHA));
when(gitea.head(any())).thenReturn(SHA);
when(gitea.migrationWarnings(any(), anyString(), anyLong()))
.thenReturn(List.of("LFS 社区读取限制"));
}
@AfterEach
void close() throws Exception {
store.close();
}
private ImportCoordinator worker(ProjectMapper mapper, ProjectService publisher) {
return new ImportCoordinator(
mapper, publisher, gitea, config, store, owner, Duration.ofMinutes(30));
}
@Test
void migrationBindsShortCodeAdaptsMasterAndPublishesWithoutEmptyRepository() throws Exception {
worker(projects, service).run(entry);
assertThat(entry.phase).isEqualTo(Phase.PUBLISHED);
Project project = service.published(entry.projectCode);
assertThat(project.getProjectCode()).isEqualTo(entry.projectCode).hasSize(12);
assertThat(project.getGiteaRepoId()).isEqualTo(77L);
assertThat(project.getPublishedCommitSha()).isEqualTo(SHA);
assertThat(project.getSourceProjectId()).isNull();
assertThat(project.getCoverUrl()).endsWith("a.gif");
assertThat(service.view(project, owner.getId()).projectCode()).isEqualTo(entry.projectCode);
verify(gitea, never()).createRepository(anyString(), anyBoolean());
verify(gitea).createBranch(remote.repository(), "main", SHA);
worker(projects, service).run(entry);
verify(gitea, times(1)).migrate(anyString(), anyString(), any(Duration.class));
}
@Test
void normalizationErrorResumesOnlyAfterReadOnlyConfirmationOfAppliedState() throws Exception {
doThrow(new GiteaFailure(502, "GITEA_ERROR", "写入后响应失败", true))
.when(gitea)
.defaultBranch(any(), eq("main"));
worker(projects, service).run(entry);
assertThat(entry.phase).isEqualTo(Phase.REVIEW);
assertThat(ImportCoordinator.canVerifyNormalization(entry)).isTrue();
// 默认分支仍未生效时,只读核对,不重发写入。
worker(projects, service).run(entry);
assertThat(entry.phase).isEqualTo(Phase.REVIEW);
assertThat(entry.errorCode).isEqualTo("NORMALIZATION_NOT_CONFIRMED");
verify(gitea, times(1)).defaultBranch(any(), anyString());
remote =
new GiteaClient.RemoteRepository(
remote.id(),
remote.owner(),
remote.name(),
true,
false,
false,
remote.originalUrl(),
"main");
worker(projects, service).run(entry);
assertThat(entry.phase).isEqualTo(Phase.PUBLISHED);
verify(gitea, times(1)).migrate(anyString(), anyString(), any(Duration.class));
verify(gitea, times(1)).createBranch(any(), anyString(), anyString());
verify(gitea, times(1)).defaultBranch(any(), anyString());
}
@Test
void unknownMigrationIsNeverAutomaticallyRepeated() throws Exception {
when(gitea.migrate(anyString(), anyString(), any(Duration.class)))
.thenAnswer(
call -> {
exists.set(true);
throw new GiteaFailure(504, "REMOTE_RESULT_UNKNOWN", "超时", true);
});
worker(projects, service).run(entry);
assertThat(entry.phase).isEqualTo(Phase.REVIEW);
assertThat(
jdbc.queryForObject(
"SELECT repo_status FROM project WHERE project_code = ?",
String.class,
entry.projectCode))
.isEqualTo("CREATING");
worker(projects, service).run(entry);
verify(gitea, times(1)).migrate(anyString(), anyString(), any(Duration.class));
}
@Test
void databaseReservationFailureCanResumeWhenBothProjectAndRemoteAreAbsent() throws Exception {
ProjectMapper failing = mock(ProjectMapper.class, AdditionalAnswers.delegatesTo(projects));
doThrow(new IllegalStateException("模拟写入前数据库故障")).when(failing).insert(any(Project.class));
worker(failing, service).run(entry);
assertThat(entry.phase).isEqualTo(Phase.FAILED);
assertThat(entry.failedPhase).isEqualTo("RESERVING");
String reservedAt = entry.reservedAt;
verify(gitea, never()).migrate(anyString(), anyString(), any(Duration.class));
worker(projects, service).run(entry);
assertThat(entry.phase).isEqualTo(Phase.PUBLISHED);
assertThat(entry.reservedAt).isEqualTo(reservedAt);
verify(gitea, times(1)).migrate(anyString(), anyString(), any(Duration.class));
}
@Test
void persistenceFailureBeforeReservationStopsRemoteWrites() throws Exception {
Files.createDirectory(directory.resolve("state/manifest.json.tmp"));
assertThatThrownBy(() -> worker(projects, service).run(entry))
.isInstanceOf(java.io.IOException.class);
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM project", Integer.class)).isZero();
verify(gitea, never()).migrate(anyString(), anyString(), any(Duration.class));
}
@Test
void confirmedReceiptSurvivesDatabaseWriteFailureAndRestart() throws Exception {
ProjectMapper failing = mock(ProjectMapper.class, AdditionalAnswers.delegatesTo(projects));
AtomicBoolean fail = new AtomicBoolean(true);
doAnswer(
call -> {
if (fail.getAndSet(false)) throw new IllegalStateException("模拟数据库回写失败");
return projects.updateById((Project) call.getArgument(0));
})
.when(failing)
.updateById(any(Project.class));
worker(failing, service).run(entry);
assertThat(entry.phase).isEqualTo(Phase.FAILED);
assertThat(entry.receipt).isNotNull();
store.close();
store =
new StateStore(
directory.resolve("state"),
new WorkbookData("fingerprint", List.of(entry.candidate), List.of()));
entry = store.manifest.entries.getFirst();
worker(projects, service).run(entry);
assertThat(entry.phase).isEqualTo(Phase.PUBLISHED);
verify(gitea, times(1)).migrate(anyString(), anyString(), any(Duration.class));
}
@Test
void publicationFailureResumesWithoutMigratingAgain() throws Exception {
ProjectService publisher = mock(ProjectService.class);
when(publisher.publish(owner.getId(), entry.projectCode, SHA))
.thenThrow(BusinessException.conflict("模拟发布失败"))
.thenAnswer(call -> service.publish(owner.getId(), entry.projectCode, SHA));
worker(projects, publisher).run(entry);
assertThat(entry.phase).isEqualTo(Phase.FAILED);
worker(projects, publisher).run(entry);
assertThat(entry.phase).isEqualTo(Phase.PUBLISHED);
verify(gitea, times(1)).migrate(anyString(), anyString(), any(Duration.class));
}
@Test
void conflictingMainIsPreservedAndNotPublished() throws Exception {
when(gitea.branch(any(), eq("main"))).thenReturn("b".repeat(40));
worker(projects, service).run(entry);
assertThat(entry.errorCode).isEqualTo("MAIN_BRANCH_CONFLICT");
assertThat(
jdbc.queryForObject(
"SELECT publish_status FROM project WHERE project_code = ?",
String.class,
entry.projectCode))
.isEqualTo("UNPUBLISHED");
verify(gitea, never()).defaultBranch(any(), anyString());
}
@Test
void unrelatedProjectAndRemoteRepositoryAreNeverAdopted() throws Exception {
exists.set(true);
worker(projects, service).run(entry);
assertThat(entry.phase).isEqualTo(Phase.REVIEW);
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM project", Integer.class)).isZero();
verify(gitea, never()).migrate(anyString(), anyString(), any(Duration.class));
exists.set(false);
entry.phase = Phase.READY;
when(gitea.createRepository(anyString(), eq(true)))
.thenReturn(new GiteaClient.Repository(88, "community", "temporary"));
String existing = service.create(owner.getId(), entry.candidate.input()).projectCode();
jdbc.update(
"UPDATE project SET project_code = ? WHERE project_code = ?",
entry.projectCode,
existing);
String before =
jdbc.queryForObject(
"SELECT repo_status FROM project WHERE project_code = ?",
String.class,
entry.projectCode);
worker(projects, service).run(entry);
assertThat(entry.errorCode).isEqualTo("PROJECT_CONFLICT");
assertThat(
jdbc.queryForObject(
"SELECT repo_status FROM project WHERE project_code = ?",
String.class,
entry.projectCode))
.isEqualTo(before);
}
}
@@ -0,0 +1,45 @@
package com.linkerbot.community.importer;
import static org.junit.jupiter.api.Assertions.*;
import org.junit.jupiter.api.Test;
class SourceChecksTest {
private static final String PAGE =
"""
<meta name="octolytics-dimension-repository_id" content="482055059" />
<meta name="octolytics-dimension-repository_nwo" content="neurobionics/opensourceleg" />
<meta name="octolytics-dimension-repository_public" content="true" />
<script type="application/json" data-target="react-app.embeddedData">
{"payload":{"codeViewLayoutRoute":{"repo":{"id":482055059,
"defaultBranch":"main","name":"opensourceleg","ownerLogin":"neurobionics",
"public":true,"private":false,"isEmpty":false}}}}
</script>
""";
@Test
void publicPageRequiresMatchingIdentityVisibilityAndBranch() throws Exception {
ImportModels.GitHubRepository repo =
SourceChecks.parsePublicPage(PAGE, "https://github.com/neurobionics/opensourceleg");
assertEquals(482055059L, repo.id());
assertEquals("main", repo.defaultBranch());
for (String changed :
new String[] {
PAGE.replace("content=\"482055059\"", "content=\"42\""),
PAGE.replace("\"public\":true", "\"public\":false"),
PAGE.replace("\"private\":false", "\"private\":true"),
PAGE.replace("\"isEmpty\":false", "\"isEmpty\":true"),
PAGE.replace("\"defaultBranch\":\"main\"", "\"defaultBranch\":\"\""),
PAGE.replace("react-app.embeddedData", "other-data")
}) {
assertThrows(
ImportFailure.class,
() ->
SourceChecks.parsePublicPage(
changed, "https://github.com/neurobionics/opensourceleg"));
}
assertThrows(
ImportFailure.class,
() -> SourceChecks.parsePublicPage(PAGE, "https://github.com/someone/other"));
}
}
@@ -0,0 +1,195 @@
package com.linkerbot.community.importer;
import static com.linkerbot.community.importer.ImportModels.*;
import static org.assertj.core.api.Assertions.*;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.charset.StandardCharsets;
import java.nio.file.*;
import java.util.*;
import java.util.zip.*;
class WorkbookReaderTest {
@TempDir Path directory;
static Path fixture(Path path) throws Exception {
try (ZipOutputStream zip = new ZipOutputStream(Files.newOutputStream(path))) {
List<String> names = new ArrayList<>(WorkbookReader.SHEETS.keySet());
names.add("10_项目封面图鉴");
names.add("0_汇总总表");
StringBuilder book =
new StringBuilder(
"<workbook"
+ " xmlns=\"http://schemas.openxmlformats.org/spreadsheetml/2006/main\""
+ " xmlns:r=\"http://schemas.openxmlformats.org/officeDocument/2006/relationships\"><sheets>");
StringBuilder rel = new StringBuilder("<Relationships>");
for (int i = 0; i < names.size(); i++) {
book.append("<sheet name=\"")
.append(names.get(i))
.append("\" r:id=\"r")
.append(i)
.append("\"/>");
rel.append("<Relationship Id=\"r")
.append(i)
.append("\" Target=\"worksheets/s")
.append(i)
.append(".xml\"/>");
String header =
"<row r=\"2\">"
+ cell("I2", "GitHub链接状态")
+ cell("L2", "封面图片 URL")
+ "</row>";
String rows =
row(
3,
"可访问",
"https://images.example.com/a.gif",
"https://github.com/example/repo" + i);
if (i == 0)
rows +=
row(
4,
"可访问(已重定向)",
"https://images.example.com/a.png",
"https://github.com/example/redirect")
+ row(5, "可访问", "", "https://github.com/example/no-cover")
+ row(
6,
"可访问",
"https://images.example.com/a.png",
"https://github.com/example")
+ row(
7,
"可访问",
"https://images.example.com/a.png",
"https://github.com/example/repo/tree/main")
+ row(
8,
"可访问",
"http://images.example.com/a.png",
"https://github.com/example/plain")
+ row(
9,
"可访问",
"https://images.example.com/a.png",
"https://gitlab.com/example/repo");
entry(
zip,
"xl/worksheets/s" + i + ".xml",
"<worksheet><sheetData>" + header + rows + "</sheetData></worksheet>");
}
entry(zip, "xl/workbook.xml", book + "</sheets></workbook>");
entry(zip, "xl/_rels/workbook.xml.rels", rel + "</Relationships>");
}
return path;
}
private static String row(int row, String status, String cover, String url) {
return "<row r=\""
+ row
+ "\">"
+ cell("A" + row, "测试项目")
+ cell("B" + row, url)
+ cell("C" + row, "中文简介")
+ cell("D" + row, "STEP")
+ cell("E" + row, "小型")
+ cell("F" + row, "待查")
+ cell("G" + row, "待验证")
+ cell("H" + row, "100")
+ cell("I" + row, status)
+ "<c r=\"K"
+ row
+ "\"><f>IMAGE(L"
+ row
+ ")</f><v>点开图片URL</v></c>"
+ cell("L" + row, cover)
+ "</row>";
}
private static String cell(String ref, String value) {
return "<c r=\"" + ref + "\" t=\"inlineStr\"><is><t>" + value + "</t></is></c>";
}
private static void entry(ZipOutputStream zip, String name, String value) throws Exception {
zip.putNextEntry(new ZipEntry(name));
zip.write(value.getBytes(StandardCharsets.UTF_8));
zip.closeEntry();
}
@Test
void fixedSheetsAndActualColumnsAreUsedWithoutExecutingCoverFormula() throws Exception {
WorkbookData data = new WorkbookReader().read(fixture(directory.resolve("source.xlsx")));
assertThat(data.candidates()).hasSize(7);
assertThat(data.excluded()).hasSize(6);
assertThat(data.candidates().stream().map(Candidate::sheet))
.containsExactlyElementsOf(WorkbookReader.SHEETS.keySet());
Candidate first = data.candidates().getFirst();
assertThat(first.name()).isEqualTo("测试项目");
assertThat(first.projectType()).isEqualTo("WHEELED");
assertThat(first.description())
.contains("中文简介", "CAD 文件类型:STEP", "URDF 情况:待查", "资料验证状态:待验证")
.doesNotContain("100");
assertThat(first.coverUrl()).endsWith("a.gif");
assertThat(new WorkbookReader().read(directory.resolve("source.xlsx"))).isEqualTo(data);
}
@Test
void stateLockFingerprintAndReceiptsSurviveRestart() throws Exception {
WorkbookData data = new WorkbookReader().read(fixture(directory.resolve("source.xlsx")));
Path state = directory.resolve("state");
try (StateStore store = new StateStore(state, data)) {
store.update(
store.manifest.entries.getFirst(),
e -> {
e.projectCode = "111111111111";
e.phase = Phase.MIGRATING;
});
assertThatThrownBy(() -> new StateStore(state, data))
.isInstanceOf(RuntimeException.class);
store.report();
}
try (StateStore store = new StateStore(state, data)) {
assertThat(store.manifest.entries.getFirst().phase).isEqualTo(Phase.MIGRATING);
assertThat(store.manifest.entries.getFirst().projectCode).isEqualTo("111111111111");
}
assertThatThrownBy(
() ->
new StateStore(
state,
new WorkbookData(
"other", data.candidates(), data.excluded())))
.isInstanceOf(ImportFailure.class);
try (StateStore ignored = new StateStore(state, data)) {
assertThat(Files.exists(state.resolve("report.json"))).isTrue();
}
}
@Test
void cliIsNonMutatingUnlessExplicitlyExecuting() {
ImportApplication.Options options =
ImportApplication.Options.parse(new String[] {"--excel", "source.xlsx"});
assertThat(options.execute()).isFalse();
assertThat(options.concurrency()).isEqualTo(2);
assertThat(options.migrationTimeout().toMinutes()).isEqualTo(30);
assertThatThrownBy(
() ->
ImportApplication.Options.parse(
new String[] {"--excel", "s", "--execute"}))
.isInstanceOf(IllegalArgumentException.class);
assertThatThrownBy(
() ->
ImportApplication.Options.parse(
new String[] {
"--excel",
"s",
"--execute",
"--offline",
"--owner-user-code",
"u"
}))
.isInstanceOf(IllegalArgumentException.class);
}
}
@@ -4,6 +4,7 @@ import com.linkerbot.community.bean.Contracts.*;
import com.linkerbot.community.common.PageResponse;
import java.util.List;
import java.time.Duration;
public interface GiteaClient {
record Repository(long id, String owner, String name) {}
@@ -12,6 +13,29 @@ public interface GiteaClient {
record Change(String operation, String path, String blobSha, byte[] bytes) {}
record RemoteRepository(long id, String owner, String name, boolean privateRepo,
boolean mirror, boolean empty, String originalUrl, String defaultBranch) {
public Repository repository() { return new Repository(id, owner, name); }
}
default RemoteRepository migrate(String cloneUrl, String name, Duration timeout) {
throw new UnsupportedOperationException("未实现迁移");
}
default RemoteRepository inspect(String name) { throw new UnsupportedOperationException("未实现仓库检查"); }
default String branch(Repository repository, String branch) { throw new UnsupportedOperationException("未实现分支检查"); }
default void createBranch(Repository repository, String name, String sha) { throw new UnsupportedOperationException("未实现分支创建"); }
default void defaultBranch(Repository repository, String name) { throw new UnsupportedOperationException("未实现默认分支设置"); }
default void checkMigrationSettings() { throw new UnsupportedOperationException("未实现迁移能力检查"); }
default List<String> migrationWarnings(Repository repository, String sha, long maxFileBytes) {
return List.of("社区下载仍受单文件限制,LFS 实体读取及特殊 Git 文件的 Fork 能力未扩展");
}
Repository createRepository(String name, boolean initialize);
String head(Repository repository);
@@ -18,6 +18,7 @@ import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.util.*;
import java.time.Duration;
@Component
public class RestGiteaClient implements GiteaClient {
@@ -51,13 +52,17 @@ public class RestGiteaClient implements GiteaClient {
}
private HttpResponse<byte[]> request(String method, String path, Object body) {
return request(method, path, body, properties.getGitea().getRequestTimeout());
}
private HttpResponse<byte[]> request(String method, String path, Object body, Duration timeout) {
if (properties.getGitea().getToken().isBlank())
throw new GiteaFailure(503, "GITEA_NOT_CONFIGURED", "请配置 Gitea Token", false);
try {
String base = properties.getGitea().getBaseUrl().replaceAll("/+$", "");
HttpRequest.Builder builder =
HttpRequest.newBuilder(URI.create(base + "/api/v1" + path))
.timeout(properties.getGitea().getRequestTimeout())
.timeout(timeout)
.header("Accept", "application/json")
.header("Authorization", "token " + properties.getGitea().getToken());
if (body == null) builder.method(method, HttpRequest.BodyPublishers.noBody());
@@ -72,6 +77,8 @@ public class RestGiteaClient implements GiteaClient {
int status = response.statusCode();
if (status < 200 || status >= 300) {
if (status == 404) throw new GiteaFailure(404, "NOT_FOUND", "仓库文件或版本不存在", false);
if (status == 422 && path.equals("/repos/migrate"))
throw new GiteaFailure(422, "GITEA_MIGRATION_REJECTED", "Gitea 迁移失败,需核对源站访问和服务端迁移日志", false);
if (status == 409 || status == 422)
throw new GiteaFailure(409, "GITEA_CONFLICT", "仓库内容冲突,请刷新后重试", false);
throw new GiteaFailure(
@@ -117,11 +124,93 @@ public class RestGiteaClient implements GiteaClient {
@Override
public String head(Repository repo) {
return branch(repo, "main");
}
@Override
public String branch(Repository repo, String branchName) {
JsonNode branch =
parse(request("GET", repoPath(repo) + "/branches/main", null).body(), false);
parse(request("GET", repoPath(repo) + "/branches/" + encode(branchName), null).body(), false);
return FilePaths.sha(branch.path("commit").path("id").asText());
}
private RemoteRepository remote(JsonNode data) {
if (data.path("id").asLong() <= 0)
throw new GiteaFailure(502, "GITEA_INVALID_RESPONSE", "仓库响应缺少标识", true);
return new RemoteRepository(data.path("id").asLong(), data.path("owner").path("login").asText(),
data.path("name").asText(), data.path("private").asBoolean(), data.path("mirror").asBoolean(),
data.path("empty").asBoolean(), data.path("original_url").asText(), data.path("default_branch").asText());
}
@Override
public RemoteRepository migrate(String cloneUrl, String name, Duration timeout) {
Map<String, Object> body = new LinkedHashMap<>();
body.put("clone_addr", cloneUrl);
body.put("repo_owner", properties.getGitea().getOwner());
body.put("repo_name", name);
body.put("service", "git");
body.put("private", true);
body.put("mirror", false);
body.put("lfs", true);
for (String option : List.of("issues", "labels", "milestones", "pull_requests", "releases", "wiki"))
body.put(option, false);
HttpResponse<byte[]> response = request("POST", "/repos/migrate", body, timeout);
if (response.statusCode() != 201)
throw new GiteaFailure(502, "MIGRATION_NOT_CONFIRMED", "迁移没有返回已完成的 201 收据,需人工核对", true);
return remote(parse(response.body(), true));
}
@Override
public RemoteRepository inspect(String name) {
return remote(parse(request("GET", "/repos/" + encode(properties.getGitea().getOwner()) + "/" + encode(name), null).body(), false));
}
@Override
public void createBranch(Repository repo, String name, String sha) {
request("POST", repoPath(repo) + "/branches", Map.of("new_branch_name", name, "old_ref_name", FilePaths.sha(sha)));
}
@Override
public void defaultBranch(Repository repo, String name) {
request("PATCH", repoPath(repo), Map.of("default_branch", name));
}
@Override
public void checkMigrationSettings() {
JsonNode settings = parse(request("GET", "/settings/repository", null).body(), false);
if (!settings.has("lfs_disabled") || settings.path("lfs_disabled").asBoolean()
|| settings.path("migrations_disabled").asBoolean() || settings.path("http_git_disabled").asBoolean())
throw BusinessException.badRequest("Gitea 必须启用迁移、HTTP Git 和 LFS");
}
@Override
public List<String> migrationWarnings(Repository repo, String sha, long maxFileBytes) {
List<String> warnings = new ArrayList<>();
warnings.add("已请求迁移 LFS;社区 API 的 LFS 实体读取能力未扩展");
long large = 0;
long links = 0;
long modules = 0;
long seen = 0;
for (int page = 1; page <= 10000; page++) {
JsonNode tree = parse(request("GET", repoPath(repo) + "/git/trees/" + FilePaths.sha(sha)
+ "?recursive=true&per_page=1000&page=" + page, null).body(), false);
JsonNode entries = tree.path("tree");
if (!entries.isArray()) throw new GiteaFailure(502, "GITEA_INVALID_RESPONSE", "Git 文件树响应格式异常", false);
for (JsonNode entry : entries) {
if (entry.path("size").asLong() > maxFileBytes) large++;
if (entry.path("mode").asText().equals("120000")) links++;
if (entry.path("mode").asText().equals("160000")) modules++;
}
seen += entries.size();
if (seen >= tree.path("total_count").asLong(seen) && !tree.path("truncated").asBoolean()) break;
if (entries.isEmpty() || page == 10000) { warnings.add("文件树扫描不完整,需要人工核对剩余文件"); break; }
}
if (large > 0) warnings.add("发布快照有 " + large + " 个文件超过社区单文件读取限制 " + maxFileBytes + " 字节");
if (links > 0) warnings.add("发布快照有 " + links + " 个符号链接,社区普通文件读取和 Fork 不支持");
if (modules > 0) warnings.add("发布快照有 " + modules + " 个子模块引用,未递归迁移且社区 Fork 不支持");
return List.copyOf(warnings);
}
private JsonNode contents(Repository repo, String sha, String path) {
String suffix = path.isEmpty() ? "" : "/" + filePath(path);
return parse(
@@ -15,6 +15,8 @@ import com.linkerbot.community.service.config.CommunityProperties;
import com.linkerbot.community.service.gitea.*;
import com.linkerbot.community.service.mapper.*;
import com.linkerbot.community.service.support.ProjectLocks;
import com.linkerbot.community.service.support.ProjectInputs;
import org.springframework.dao.DuplicateKeyException;
import lombok.RequiredArgsConstructor;
@@ -22,7 +24,6 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Service;
import java.net.URI;
import java.util.*;
import java.util.function.Function;
@@ -46,25 +47,11 @@ public class ProjectServiceImpl implements ProjectService {
}
private void validateInput(ProjectInput input) {
if (!config.getTypes().containsKey(input.projectType()))
throw BusinessException.badRequest("项目类型不在可选列表中");
ProjectInputs.validate(input, config);
}
private String normalizeCoverUrl(String value) {
if (value == null || value.isBlank()) return null;
String normalized = value.trim();
try {
URI uri = URI.create(normalized);
if (normalized.length() > 2048
|| !"https".equalsIgnoreCase(uri.getScheme())
|| uri.getHost() == null
|| uri.getRawUserInfo() != null
|| (uri.getPort() != -1 && (uri.getPort() < 1 || uri.getPort() > 65535)))
throw new IllegalArgumentException();
} catch (IllegalArgumentException error) {
throw BusinessException.badRequest("封面地址必须是最多 2048 字符且不含用户凭据的 HTTPS 地址");
}
return normalized;
return ProjectInputs.coverUrl(value);
}
@Override
@@ -185,7 +172,7 @@ public class ProjectServiceImpl implements ProjectService {
private Project initialize(long userId, ProjectInput input, Project source) {
validateInput(input);
Project project = new Project();
project.setProjectCode(Codes.next());
project.setProjectCode(ProjectCodes.next());
project.setOwnerUserId(userId);
project.setName(input.name().trim());
project.setDescription(input.description() == null ? "" : input.description());
@@ -202,8 +189,27 @@ public class ProjectServiceImpl implements ProjectService {
project.setSourceProjectId(source.getId());
project.setSourceCommitSha(source.getPublishedCommitSha());
}
projects.insert(project);
return project;
for (int attempt = 1; attempt <= 5; attempt++) {
try {
projects.insert(project);
return project;
} catch (DuplicateKeyException duplicate) {
boolean codeConflict = false;
for (Throwable cause = duplicate; cause != null; cause = cause.getCause()) {
if (cause.getMessage() != null
&& cause.getMessage().toLowerCase(Locale.ROOT).contains("uk_project_code"))
codeConflict = true;
}
if (!codeConflict) throw duplicate;
log.warn("项目编码冲突 projectCode={} attempt={}", project.getProjectCode(), attempt);
if (attempt == 5)
throw new BusinessException(503, "PROJECT_CODE_EXHAUSTED", "项目编码分配失败,请稍后重试");
project.setId(null);
project.setProjectCode(ProjectCodes.next());
project.setGiteaRepoName("p_" + project.getProjectCode());
}
}
throw new IllegalStateException("项目编码分配未完成");
}
@Override
@@ -0,0 +1,38 @@
package com.linkerbot.community.service.support;
import com.linkerbot.community.bean.Contracts.ProjectInput;
import com.linkerbot.community.common.BusinessException;
import com.linkerbot.community.service.config.CommunityProperties;
import java.net.URI;
public final class ProjectInputs {
private ProjectInputs() {}
public static void validate(ProjectInput input, CommunityProperties config) {
if (input.name() == null || input.name().isBlank() || input.name().length() > 100)
throw BusinessException.badRequest("项目名称必须为 1~100 字符");
if (input.description() != null && input.description().length() > 2000)
throw BusinessException.badRequest("项目描述不能超过 2000 字符");
if (!config.getTypes().containsKey(input.projectType()))
throw BusinessException.badRequest("项目类型不在可选列表中");
coverUrl(input.coverUrl());
}
public static String coverUrl(String value) {
if (value == null || value.isBlank()) return null;
String normalized = value.trim();
try {
URI uri = URI.create(normalized);
if (normalized.length() > 2048
|| !"https".equalsIgnoreCase(uri.getScheme())
|| uri.getHost() == null
|| uri.getRawUserInfo() != null
|| (uri.getPort() != -1 && (uri.getPort() < 1 || uri.getPort() > 65535)))
throw new IllegalArgumentException();
} catch (IllegalArgumentException error) {
throw BusinessException.badRequest("封面地址必须是最多 2048 字符且不含用户凭据的 HTTPS 地址");
}
return normalized;
}
}
@@ -0,0 +1,89 @@
package com.linkerbot.community.service;
import static org.assertj.core.api.Assertions.*;
import static org.mockito.ArgumentMatchers.*;
import static org.mockito.Mockito.*;
import com.linkerbot.community.bean.Contracts.ProjectInput;
import com.linkerbot.community.bean.entity.Project;
import com.linkerbot.community.bean.entity.User;
import com.linkerbot.community.common.BusinessException;
import com.linkerbot.community.common.ProjectCodes;
import com.linkerbot.community.service.config.CommunityProperties;
import com.linkerbot.community.service.gitea.GiteaClient;
import com.linkerbot.community.service.impl.ProjectServiceImpl;
import com.linkerbot.community.service.mapper.*;
import com.linkerbot.community.service.support.ProjectLocks;
import org.junit.jupiter.api.Test;
import org.mockito.MockedStatic;
import org.springframework.dao.DuplicateKeyException;
class ProjectCodeAllocationTest {
@Test
void retriesOnlyProjectCodeConstraintBeforeAnyRemoteCreation() {
ProjectMapper mapper = mock(ProjectMapper.class);
GiteaClient gitea = mock(GiteaClient.class);
when(mapper.insert(any(Project.class)))
.thenThrow(new DuplicateKeyException("UK_PROJECT_CODE_INDEX"))
.thenAnswer(
call -> {
Project project = call.getArgument(0);
project.setId(1L);
return 1;
});
when(gitea.createRepository(anyString(), eq(true)))
.thenReturn(new GiteaClient.Repository(1, "community", "p_222222222222"));
try (MockedStatic<ProjectCodes> codes = mockStatic(ProjectCodes.class)) {
codes.when(ProjectCodes::next).thenReturn("111111111111", "222222222222");
assertThat(
service(mapper, gitea)
.create(1, new ProjectInput("测试", "", "GENERAL", null))
.projectCode())
.isEqualTo("222222222222");
}
verify(mapper, times(2)).insert(any(Project.class));
verify(gitea).createRepository("p_222222222222", true);
verify(gitea, never()).createRepository("p_111111111111", true);
}
@Test
void stopsAfterFiveCollisionsAndDoesNotRetryOtherConstraints() {
ProjectMapper mapper = mock(ProjectMapper.class);
GiteaClient gitea = mock(GiteaClient.class);
when(mapper.insert(any(Project.class)))
.thenThrow(new DuplicateKeyException("uk_project_code"));
assertThatThrownBy(
() ->
service(mapper, gitea)
.create(1, new ProjectInput("测试", "", "GENERAL", null)))
.isInstanceOfSatisfying(
BusinessException.class,
e -> assertThat(e.code()).isEqualTo("PROJECT_CODE_EXHAUSTED"));
verify(mapper, times(5)).insert(any(Project.class));
verifyNoInteractions(gitea);
reset(mapper);
when(mapper.insert(any(Project.class)))
.thenThrow(new DuplicateKeyException("uk_project_repo"));
assertThatThrownBy(
() ->
service(mapper, gitea)
.create(1, new ProjectInput("测试", "", "GENERAL", null)))
.isInstanceOf(DuplicateKeyException.class);
verify(mapper).insert(any(Project.class));
verifyNoInteractions(gitea);
}
private ProjectServiceImpl service(ProjectMapper projects, GiteaClient gitea) {
UserService users = mock(UserService.class);
when(users.require(1)).thenReturn(new User());
return new ProjectServiceImpl(
projects,
mock(LikeMapper.class),
mock(CommentMapper.class),
users,
gitea,
new CommunityProperties(),
new ProjectLocks());
}
}
@@ -0,0 +1,138 @@
package com.linkerbot.community.service.gitea;
import static org.assertj.core.api.Assertions.*;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.linkerbot.community.service.config.CommunityProperties;
import com.sun.net.httpserver.HttpServer;
import org.junit.jupiter.api.Test;
import java.net.InetSocketAddress;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.util.concurrent.atomic.AtomicReference;
class GiteaMigrationTest {
@Test
void nativeMigrationUsesConfirmedSchemaAndKeepsLfsAndPrivateEnabled() throws Exception {
ObjectMapper json = new ObjectMapper();
AtomicReference<JsonNode> body = new AtomicReference<>();
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext(
"/api/v1/repos/migrate",
exchange -> {
body.set(json.readTree(exchange.getRequestBody()));
assertThat(exchange.getRequestHeaders().getFirst("Authorization"))
.isEqualTo("token fixture-token");
byte[] response =
"{\"id\":77,\"owner\":{\"login\":\"community\"},\"name\":\"p_111111111111\",\"private\":true,\"mirror\":false,\"empty\":false,\"default_branch\":\"master\",\"original_url\":\"https://github.com/example/robot.git\"}"
.getBytes(StandardCharsets.UTF_8);
exchange.sendResponseHeaders(201, response.length);
exchange.getResponseBody().write(response);
exchange.close();
});
server.start();
try {
CommunityProperties config = new CommunityProperties();
config.getGitea().setBaseUrl("http://127.0.0.1:" + server.getAddress().getPort());
config.getGitea().setToken("fixture-token");
GiteaClient client = new RestGiteaClient(config, json);
GiteaClient.RemoteRepository result =
client.migrate(
"https://github.com/example/robot.git",
"p_111111111111",
Duration.ofSeconds(2));
assertThat(result.id()).isEqualTo(77);
assertThat(result.defaultBranch()).isEqualTo("master");
assertThat(body.get().path("repo_owner").asText()).isEqualTo("community");
assertThat(body.get().path("repo_name").asText()).isEqualTo("p_111111111111");
assertThat(body.get().path("service").asText()).isEqualTo("git");
assertThat(body.get().path("private").asBoolean()).isTrue();
assertThat(body.get().path("lfs").asBoolean()).isTrue();
for (String key :
new String[] {
"mirror",
"issues",
"labels",
"milestones",
"pull_requests",
"releases",
"wiki"
}) assertThat(body.get().path(key).asBoolean()).as(key).isFalse();
} finally {
server.stop(0);
}
}
@Test
void migrationValidationOrCloneFailureIsNotReportedAsARepositoryNameConflict()
throws Exception {
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext(
"/api/v1/repos/migrate",
exchange -> {
byte[] response =
"{\"message\":\"Migration failed: fatal: connection failed\"}"
.getBytes(StandardCharsets.UTF_8);
exchange.sendResponseHeaders(422, response.length);
exchange.getResponseBody().write(response);
exchange.close();
});
server.start();
try {
CommunityProperties config = new CommunityProperties();
config.getGitea().setBaseUrl("http://127.0.0.1:" + server.getAddress().getPort());
config.getGitea().setToken("fixture-token");
GiteaClient client = new RestGiteaClient(config, new ObjectMapper());
assertThatThrownBy(
() ->
client.migrate(
"https://github.com/example/robot.git",
"p_111111111111",
Duration.ofSeconds(2)))
.isInstanceOfSatisfying(
GiteaFailure.class,
error -> {
assertThat(error.status()).isEqualTo(422);
assertThat(error.code()).isEqualTo("GITEA_MIGRATION_REJECTED");
assertThat(error.uncertain()).isFalse();
});
} finally {
server.stop(0);
}
}
@Test
void migrationTimeoutIsAnUnknownRemoteWriteResult() throws Exception {
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext(
"/api/v1/repos/migrate",
exchange -> {
try {
Thread.sleep(300);
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
}
exchange.close();
});
server.start();
try {
CommunityProperties config = new CommunityProperties();
config.getGitea().setBaseUrl("http://127.0.0.1:" + server.getAddress().getPort());
config.getGitea().setToken("fixture-token");
assertThatThrownBy(
() ->
new RestGiteaClient(config, new ObjectMapper())
.migrate(
"https://github.com/example/robot.git",
"p_111111111111",
Duration.ofMillis(50)))
.isInstanceOfSatisfying(
GiteaFailure.class, error -> assertThat(error.uncertain()).isTrue());
} finally {
server.stop(0);
}
}
}
+2
View File
@@ -89,6 +89,8 @@ Git SHA 使用完整的 40 或 64 位十六进制字符串,不使用缩写。
| publishedCommitSha | 项目详情 | 社区当前发布版本;Fork 时原样提交此值 |
| blobSha | 文件或目录条目 | 文件或目录对应的 Git 对象标识,前端不用它提交文件 |
新建、Fork 和 GitHub 初始化导入的 `projectCode` 为 12 位小写字母和数字,例如 `7m4k9x2p8q6r`。已有 32 位项目编码及链接继续有效。前端将编码作为不透明字符串传递,不截取、不补齐,也不固定要求 32 位;`sourceProjectCode`、AI 的 `previousProjectCodes` 及结果编码同样兼容两种长度,最大长度仍为 32。用户编码、评论编码及 Commit SHA 的规则不变。
项目详情不包含工作区当前 main 的 SHA。进入编辑器、提交或发布前,从工作区目录或文件接口获取 `commitSha`。
## 接口总览
+2
View File
@@ -80,6 +80,8 @@ CREATE TABLE project_like (
内部 ID 仅用于数据库关联,对外使用业务编码。时间均由后端写入 UTC。`CREATING` 项目表示仓库初始化未完成,不代表编辑草稿。Gitea 数据库与业务库分离,不修改 Gitea 内部表。
新增项目使用 12 位项目编码,历史 32 位编码继续保留。现有 `project_code CHAR(32)` 可容纳两者,不需要修改表结构或历史记录;仓库名称仍为 `p_{projectCode}`。用户编码和评论编码保持 32 位。GitHub 运维导入的来源和进度保存于批次状态目录,GitHub 来源不占用社区内部 Fork 来源字段。
## 已有数据库增加项目封面
已使用旧版结构的环境,先在对应业务库执行以下 SQL,再部署新版本应用;新建数据库已包含此字段,不重复执行。应用不会自动迁移数据库。
+1
View File
@@ -17,6 +17,7 @@
<module>community-service</module>
<module>community-security</module>
<module>community-api</module>
<module>community-import</module>
</modules>
<properties>
<java.version>21</java.version>
+11
View File
@@ -0,0 +1,11 @@
#!/usr/bin/env bash
set -euo pipefail
COMMUNITY_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
if [[ -f "$COMMUNITY_ROOT/.env" ]]; then
set -a
source "$COMMUNITY_ROOT/.env"
set +a
fi
cd "$COMMUNITY_ROOT"
mvn -q -pl community-import -am -DskipTests package
exec java -Djava.awt.headless=true -Dstdout.encoding=UTF-8 -Dstderr.encoding=UTF-8 -jar community-import/target/community-import-0.0.1-SNAPSHOT.jar "$@"