cover_url、urdf_main_object_key、urdf_prefix只存相对路径,不包含域名+环境根目录
This commit is contained in:
@@ -21,6 +21,8 @@ OSS_ACCESS_KEY_ID=''
|
||||
OSS_ACCESS_KEY_SECRET=''
|
||||
# 可选,固定公开 CDN/OSS 基础地址;留空使用 Bucket 的标准 HTTPS 域名。
|
||||
OSS_PUBLIC_BASE_URL=''
|
||||
# 可选,历史公开基础地址,多个使用英文逗号分隔;不含环境根目录或查询参数。
|
||||
OSS_LEGACY_PUBLIC_BASE_URLS=''
|
||||
# URDF 直传限制:单文件 50 MiB、整包 200 MiB、最多 500 个文件,不受仓库 multipart 限制影响。
|
||||
URDF_MAX_FILE_BYTES=52428800
|
||||
URDF_MAX_TOTAL_BYTES=209715200
|
||||
|
||||
@@ -92,7 +92,7 @@ python3 scripts/test-gitea.py
|
||||
|
||||
## 项目封面与 OSS 直传
|
||||
|
||||
项目通过 `coverUrl` 保存固定公开图片地址,支持 OSS/CDN 或外部 HTTPS 图片。无封面时返回 null;创建时可省略,修改时省略或 null 保留原值,传空字符串清除。Fork 继承同一地址,替换、清除或删除项目不自动删除 OSS 图片。
|
||||
项目 API 的 `coverUrl` 接收和返回完整 HTTPS 地址,平台封面在数据库中保存 `/project-covers/...` 环境相对路径,外部图片保留完整 URL。无封面时返回 null;创建时可省略,修改时省略或 null 保留原值,传空字符串清除。Fork 继承同一资源引用,替换、清除或删除项目不自动删除 OSS 图片。URDF 同样保存 `/urdf/...` 环境相对路径,上传和 OSS 操作仍使用含环境根目录的真实 key。人工迁移和回退见 [数据库说明](docs/database.md#资源路径历史数据迁移)。
|
||||
|
||||
已有数据库必须先执行 [封面字段升级 SQL](docs/database.md#已有数据库增加项目封面),再部署新应用;新建数据库使用更新后的建表 SQL。应用不会自动变更表结构。
|
||||
|
||||
@@ -159,13 +159,38 @@ Fork 不受普通在线文件读取的10 MiB限制,文件浏览、下载和 LF
|
||||
|
||||
## 项目短编码与 GitHub 初始化导入
|
||||
|
||||
### 历史外链封面转存 OSS
|
||||
|
||||
`scripts/migrate-project-covers.sh` 读取本项目 `.env`,将数据库现有外链图片下载并转存至当前环境 OSS,上传并核验成功后才更新 `cover_url`。配置 `DB_URL`、`DB_USERNAME`、`DB_PASSWORD`、`OSS_REGION`、`OSS_BUCKET`、`OSS_ACCESS_KEY_ID`、`OSS_ACCESS_KEY_SECRET`;带 `&` 的 JDBC 地址必须加引号。环境根目录沿用 test 的 `vibemake_dev` 或 prod 的 `vibemake`。
|
||||
|
||||
```bash
|
||||
# 默认只读预览,不下载、不上传、不更新数据库。
|
||||
bash scripts/migrate-project-covers.sh --dry-run
|
||||
# 先迁移三个项目,核对新封面的访问效果。
|
||||
bash scripts/migrate-project-covers.sh --execute --limit 3
|
||||
# 迁移其余外链,默认并发 2。
|
||||
bash scripts/migrate-project-covers.sh --execute
|
||||
```
|
||||
|
||||
支持 JPG、PNG、GIF、WebP,按实际文件头识别类型,原图不压缩、不转码。默认单图最多 5 MiB,可用 `--max-bytes` 调整,上限 50 MiB;`--concurrency` 范围为 1~4。下载只允许公开 HTTPS 地址,每次跳转重新校验,不携带任何平台或 GitHub 凭据,不访问内网地址。失效、超时、超限及非图片资源保留原值并进入失败报告。
|
||||
|
||||
实际对象位于 `{环境根目录}/project-covers/migrated/{图片SHA256}.{扩展名}`;数据库只保存 `/project-covers/migrated/...`。相同外链只下载一次,相同图片内容复用对象,已转存记录在再次运行时自动跳过;已属于本平台的完整 URL 也跳过,可按数据库迁移说明归一化。扫描包含已逻辑删除项目,以完成历史数据统一。未来新提交外链的行为不由此批处理修改。
|
||||
|
||||
每次运行在 `logs/cover-migration/` 生成独立报告目录(可用 `--report-dir` 修改)。`preview.json`、`report.json` 只包含项目编码、来源主机、状态及错误码,不记录完整外链或查询参数。`backup.jsonl` 保存修改前原值和拟写入路径,更新前强制落盘;目录和备份文件分别仅允许当前用户访问。原值可能包含签名参数,应作为受限数据库备份保存,不上传仓库或发送聊天。回写通过项目编码和原始 URL 字节比较,跳过迁移期间被修改的封面,不改 `updated_at`,不覆盖其他字段。
|
||||
|
||||
失败时可以重跑;上传成功但回写未完成的对象也按内容哈希复用,不会自动删除可能共享的对象。退出码 0 表示成功或仅预览,1 表示配置、连接等整体错误,2 表示存在逐项失败。人工恢复时仅将仍等于 `newCoverPath` 的记录按项目编码还原为 `originalCoverUrl`,避免覆盖之后的编辑。
|
||||
|
||||
对已经核验过的失效链接,可使用 `--source-overrides /绝对路径/备用源.json` 指定备用下载地址,JSON 格式为 `{"项目编码":"https://可信图片地址"}`。下载走同一套 HTTPS、大小和图片类型校验;数据库条件比较及原值备份仍使用原来的外链。采用的备用源另存于受限权限的 `source-overrides-backup.json`,便于审计。已成功迁移的项目在重跑时照常跳过。网络中断最多重试三次,证书校验失败不会绕过验证。
|
||||
|
||||
### 项目编码与导入流程
|
||||
|
||||
普通创建、Fork 和导入的新项目使用 12 位小写项目编码,字符集为 `0123456789abcdefghjkmnpqrstvwxyz`。普通项目用安全随机数生成,项目编码唯一约束冲突最多重试 5 次;其他约束错误不重试。导入编码由 `github-import:v1:<GitHub仓库ID>` 的 SHA-256 前 60 位确定,避免重跑或仓库更名后重复创建。Gitea 仓库名称为 `p_{projectCode}`。历史 32 位编码、仓库及链接继续使用,数据库无需改表。
|
||||
|
||||
导入工具仅接受以下固定 Sheet:`1_轮式机器人`、`2_四足机器人`、`3_小人形机器人`、`4_大人形机器人`、`5_机械臂`、`6_其他类型`、`7_开源硬件`。只选 I 列精确为“可访问”、L 列有合法 HTTPS 封面、B 列为具体 GitHub 仓库的行。其他 Sheet 不参与导入;K 列图片公式不会执行。原始 A~H 数据列与表头存在错位,工具按实际位置读取,忽略 H 列的无意义备注 `100`,不将“待查”等信息改写成已确认。
|
||||
|
||||
封面保留原 URL 和 GIF,不转存 OSS。描述由原中文简介和技术资料组合,不调用 AI。轮式、四足、人形分别映射对应类型,机械臂映射 ROBOTIC_ARM,其他和硬件映射 GENERAL,原始分类保留在描述里。
|
||||
外部封面保留原 URL 和 GIF,不转存 OSS;当前环境的平台封面统一转换为相对路径。描述由原中文简介和技术资料组合,不调用 AI。轮式、四足、人形分别映射对应类型,机械臂映射 ROBOTIC_ARM,其他和硬件映射 GENERAL,原始分类保留在描述里。
|
||||
|
||||
脚本加载本项目 `.env`,依赖 Java 21、Maven,以及正式执行时可用的 MySQL 和 Gitea。独立导入模块不启动 Web 服务,也不加载登录、AI 或 OSS 组件。Gitea 技术账号需具备组织所有者及仓库写权限,实例需开启迁移、HTTP Git 和 LFS。`GITHUB_TOKEN` 可选,仅用于 GitHub 元数据 API;API 限流时读取同一公开仓库的 GitHub 网页,交叉核验仓库 ID、规范地址、公开状态和默认分支,缺失或不一致则预检失败。网页和图片请求均不携带凭据。`GITEA_TOKEN` 仅访问配置的 Gitea,不能代替 `GITHUB_TOKEN`;切勿把 Token 放进 URL 或命令参数。
|
||||
脚本加载本项目 `.env`,依赖 Java 21、Maven,以及正式执行时可用的 MySQL 和 Gitea。独立导入模块不启动 Web 服务,也不加载登录、AI 或 OSS 客户端;只读取资源地址配置,不需要 OSS 凭据。test/prod 根目录与在线应用一致,历史公开地址使用 `OSS_LEGACY_PUBLIC_BASE_URLS`。Gitea 技术账号需具备组织所有者及仓库写权限,实例需开启迁移、HTTP Git 和 LFS。`GITHUB_TOKEN` 可选,仅用于 GitHub 元数据 API;API 限流时读取同一公开仓库的 GitHub 网页,交叉核验仓库 ID、规范地址、公开状态和默认分支,缺失或不一致则预检失败。网页和图片请求均不携带凭据。`GITEA_TOKEN` 仅访问配置的 Gitea,不能代替 `GITHUB_TOKEN`;切勿把 Token 放进 URL 或命令参数。
|
||||
|
||||
```bash
|
||||
IMPORT_EXCEL='/绝对路径/项目.xlsx'
|
||||
|
||||
@@ -27,6 +27,7 @@ community:
|
||||
access-key-id: ${OSS_ACCESS_KEY_ID:}
|
||||
access-key-secret: ${OSS_ACCESS_KEY_SECRET:}
|
||||
public-base-url: ${OSS_PUBLIC_BASE_URL:https://linkerbot.oss-cn-beijing.aliyuncs.com}
|
||||
legacy-public-base-urls: ${OSS_LEGACY_PUBLIC_BASE_URLS:}
|
||||
urdf:
|
||||
max-file-bytes: ${URDF_MAX_FILE_BYTES:52428800}
|
||||
max-total-bytes: ${URDF_MAX_TOTAL_BYTES:209715200}
|
||||
|
||||
@@ -21,6 +21,7 @@ import com.linkerbot.community.service.ProjectService;
|
||||
import com.linkerbot.community.service.ai.AiModelClient;
|
||||
import com.linkerbot.community.service.ai.AiProjectAssistant;
|
||||
import com.linkerbot.community.service.config.AiProperties;
|
||||
import com.linkerbot.community.service.config.OssProperties;
|
||||
import com.linkerbot.community.service.config.ForkProperties;
|
||||
import com.linkerbot.community.service.gitea.GiteaClient.Change;
|
||||
import com.linkerbot.community.service.mapper.ProjectMapper;
|
||||
@@ -91,6 +92,7 @@ class AiSearchTest {
|
||||
@Autowired AiProjectAssistant assistant;
|
||||
@Autowired FakeGitSnapshotCopier snapshots;
|
||||
@Autowired ForkProperties forkProperties;
|
||||
@Autowired OssProperties oss;
|
||||
private User user;
|
||||
private String token;
|
||||
|
||||
@@ -462,6 +464,9 @@ class AiSearchTest {
|
||||
|
||||
@Test void forksExplicitCodeWithCustomNameAndCopiesPublishedSnapshot() throws Exception {
|
||||
Project source = publishedRepository("轮式教学");
|
||||
String cover = "/project-covers/111111111111/20261019/a.png";
|
||||
source.setCoverUrl(cover);
|
||||
projects.updateById(source);
|
||||
model.handler = (name, input) -> forkIntent(source.getProjectCode(), "我的学习副本");
|
||||
List<JsonNode> events = finish(start(Map.of("message", "Fork " + source.getProjectCode() + ",命名为我的学习副本")));
|
||||
JsonNode data = events.getLast().path("data");
|
||||
@@ -474,6 +479,8 @@ class AiSearchTest {
|
||||
assertThat(fork.path("publishStatus").asText()).isEqualTo("UNPUBLISHED");
|
||||
assertThat(fork.has("id")).isFalse();
|
||||
Project copied = projectService.owner(user.getId(), fork.path("projectCode").asText());
|
||||
assertThat(copied.getCoverUrl()).isEqualTo(cover);
|
||||
assertThat(fork.path("coverUrl").asText()).isEqualTo(oss.publicBaseUrl() + "/" + oss.getRootDirectory() + cover);
|
||||
assertThat(gitea.file(projectService.repository(copied), gitea.head(projectService.repository(copied)), "README.md").bytes())
|
||||
.isEqualTo(gitea.file(projectService.repository(source), source.getPublishedCommitSha(), "README.md").bytes());
|
||||
assertThat(projects.selectCount(null)).isEqualTo(2L);
|
||||
@@ -483,6 +490,23 @@ class AiSearchTest {
|
||||
.containsExactly("understanding", "forking");
|
||||
}
|
||||
|
||||
@Test void createsHostedCoverAndSearchReturnsPublicUrlForStoredPath() throws Exception {
|
||||
String cover = "/project-covers/111111111111/20261019/a.png";
|
||||
String fullUrl = oss.publicBaseUrl() + "/" + oss.getRootDirectory() + cover;
|
||||
model.handler = (name, input) -> createIntent("WHEELED", "路径测试机器人").put("coverUrl", fullUrl);
|
||||
JsonNode created = finish(start(Map.of("message", "创建路径测试机器人,使用给定封面")))
|
||||
.getLast().path("data").path("project");
|
||||
assertThat(created.path("coverUrl").asText()).isEqualTo(fullUrl);
|
||||
Project project = projectService.owner(user.getId(), created.path("projectCode").asText());
|
||||
assertThat(project.getCoverUrl()).isEqualTo(cover);
|
||||
projectService.publish(user.getId(), project.getProjectCode(), gitea.head(projectService.repository(project)));
|
||||
model.handler = (name, input) -> name.endsWith("intent") ? intent("WHEELED", List.of("路径测试")) : rank(input);
|
||||
JsonNode card = finish(start(Map.of("message", "寻找路径测试机器人")))
|
||||
.getLast().path("data").path("projects").get(0).path("project");
|
||||
assertThat(card.path("coverUrl").asText()).isEqualTo(fullUrl);
|
||||
assertThat(projectService.owner(user.getId(), project.getProjectCode()).getCoverUrl()).isEqualTo(cover);
|
||||
}
|
||||
|
||||
@Test void forkGetsItsOwnDeadlineAfterUnderstandingRatherThanTheSearchTimeout() throws Exception {
|
||||
Project source = publishedRepository("教学项目");
|
||||
properties.setSearchTimeout(Duration.ofMillis(150));
|
||||
|
||||
+150
@@ -0,0 +1,150 @@
|
||||
package com.linkerbot.community.api;
|
||||
|
||||
import static org.assertj.core.api.Assertions.*;
|
||||
|
||||
import com.linkerbot.community.service.config.OssProperties;
|
||||
import com.linkerbot.community.service.oss.OssResourcePaths;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.Map;
|
||||
import java.util.regex.Matcher;
|
||||
import java.util.regex.Pattern;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
import org.springframework.jdbc.datasource.SingleConnectionDataSource;
|
||||
import org.springframework.jdbc.datasource.DriverManagerDataSource;
|
||||
|
||||
/** 执行数据库文档中的 SQL;仅将 MySQL JOIN UPDATE、排序规则和正则语法适配到 H2。 */
|
||||
class ResourcePathMigrationTest {
|
||||
private static final String ROOT = "vibemake_dev";
|
||||
private static final String BASE = "https://linkerbot.oss-cn-beijing.aliyuncs.com";
|
||||
private static final String CODE = "1".repeat(12);
|
||||
private static final String COVER = "/project-covers/" + "1".repeat(32) + "/20261019/" + "a".repeat(32) + ".png";
|
||||
private static final String PREFIX = "/urdf/" + CODE + "/" + "a".repeat(32) + "/";
|
||||
private static final String MAIN = PREFIX + "urdf/机械 臂 #?%+.URDF";
|
||||
private SingleConnectionDataSource source;
|
||||
private JdbcTemplate jdbc;
|
||||
private String document;
|
||||
private OssResourcePaths paths;
|
||||
|
||||
@BeforeEach
|
||||
void prepare() throws Exception {
|
||||
source = new SingleConnectionDataSource("jdbc:h2:mem:migration;MODE=MySQL", "sa", "", true);
|
||||
jdbc = new JdbcTemplate(source);
|
||||
jdbc.execute("CREATE TABLE project (project_code VARCHAR(32) PRIMARY KEY, cover_url VARCHAR(2048),"
|
||||
+ " urdf_main_object_key VARCHAR(512) NOT NULL, urdf_prefix VARCHAR(512) NOT NULL)");
|
||||
document = Files.readString(Path.of("../docs/database.md"));
|
||||
OssProperties oss = new OssProperties();
|
||||
oss.setRootDirectory(ROOT);
|
||||
oss.setPublicBaseUrl(BASE);
|
||||
paths = new OssResourcePaths(oss);
|
||||
}
|
||||
|
||||
@AfterEach
|
||||
void close() {
|
||||
source.destroy();
|
||||
}
|
||||
|
||||
@Test
|
||||
void migrationIsIdempotentAndPreservesExternalUrlsAndObjectLocations() {
|
||||
insert(CODE, BASE + "/" + ROOT + COVER, ROOT + MAIN, ROOT + PREFIX);
|
||||
insert("2".repeat(12), "https://external.example.com" + COVER, "", "");
|
||||
insert("3".repeat(12), BASE + "/" + ROOT + COVER + "?version=2", "", "");
|
||||
insert("4".repeat(12), BASE + "/" + ROOT + COVER + "#preview", "", "");
|
||||
insert("5".repeat(12), COVER, "/urdf/already/a.urdf", "/urdf/already/");
|
||||
insert("6".repeat(12), BASE + "/vibemake" + COVER, "vibemake" + MAIN, "vibemake" + PREFIX);
|
||||
insert("7".repeat(12), null, ROOT + "/urdf/wrong/a.urdf", ROOT + PREFIX);
|
||||
insert("8".repeat(12), BASE + ".evil/" + ROOT + COVER, "", "");
|
||||
insert("9".repeat(12), BASE + "/VIBEMAKE_DEV" + COVER, "", "");
|
||||
Map<String, Object> externalBefore = row("2".repeat(12));
|
||||
String publicCover = paths.coverUrl(row(CODE).get("COVER_URL").toString());
|
||||
String publicMain = paths.publicUrl(ROOT + MAIN);
|
||||
execute("migration-prepare");
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM resource_path_changes", Integer.class)).isEqualTo(1);
|
||||
execute("migration-apply");
|
||||
assertThat(row(CODE)).containsEntry("COVER_URL", COVER)
|
||||
.containsEntry("URDF_MAIN_OBJECT_KEY", MAIN).containsEntry("URDF_PREFIX", PREFIX);
|
||||
assertThat(paths.coverUrl(row(CODE).get("COVER_URL").toString())).isEqualTo(publicCover);
|
||||
assertThat(paths.publicUrl(row(CODE).get("URDF_MAIN_OBJECT_KEY").toString())).isEqualTo(publicMain);
|
||||
assertThat(row("2".repeat(12))).isEqualTo(externalBefore);
|
||||
assertThat(row("3".repeat(12)).get("COVER_URL")).asString().endsWith("?version=2");
|
||||
assertThat(row("4".repeat(12)).get("COVER_URL")).asString().endsWith("#preview");
|
||||
assertThat(row("6".repeat(12)).get("URDF_PREFIX")).isEqualTo("vibemake" + PREFIX);
|
||||
assertThat(row("7".repeat(12)).get("URDF_PREFIX")).isEqualTo(ROOT + PREFIX);
|
||||
assertThat(row("8".repeat(12)).get("COVER_URL")).asString().contains(".evil/");
|
||||
assertThat(row("9".repeat(12)).get("COVER_URL")).asString().contains("/VIBEMAKE_DEV/");
|
||||
execute("migration-prepare");
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM resource_path_changes", Integer.class)).isZero();
|
||||
execute("migration-apply");
|
||||
assertThat(row(CODE).get("COVER_URL")).isEqualTo(COVER);
|
||||
}
|
||||
|
||||
@Test
|
||||
void concurrentChangesAreNotOverwrittenAndNewPathsCanBeRolledBack() {
|
||||
insert(CODE, BASE + "/" + ROOT + COVER, ROOT + MAIN, ROOT + PREFIX);
|
||||
execute("migration-prepare");
|
||||
jdbc.update("UPDATE project SET cover_url = ? WHERE project_code = ?", "https://external.example.com/new.png", CODE);
|
||||
execute("migration-apply");
|
||||
assertThat(row(CODE)).containsEntry("COVER_URL", "https://external.example.com/new.png")
|
||||
.containsEntry("URDF_MAIN_OBJECT_KEY", ROOT + MAIN);
|
||||
jdbc.update("UPDATE project SET cover_url = ? WHERE project_code = ?", BASE + "/" + ROOT + COVER, CODE);
|
||||
execute("migration-prepare");
|
||||
execute("migration-apply");
|
||||
execute("rollback-prepare");
|
||||
jdbc.update("INSERT INTO resource_cover_rollback VALUES (?, ?, ?)", CODE, COVER, paths.coverUrl(COVER));
|
||||
execute("rollback-apply");
|
||||
assertThat(row(CODE)).containsEntry("COVER_URL", BASE + "/" + ROOT + COVER)
|
||||
.containsEntry("URDF_MAIN_OBJECT_KEY", ROOT + MAIN).containsEntry("URDF_PREFIX", ROOT + PREFIX);
|
||||
execute("rollback-apply");
|
||||
assertThat(row(CODE).get("URDF_PREFIX")).isEqualTo(ROOT + PREFIX);
|
||||
}
|
||||
|
||||
private void insert(String code, String cover, String main, String prefix) {
|
||||
jdbc.update("INSERT INTO project VALUES (?, ?, ?, ?)", code, cover, main, prefix);
|
||||
}
|
||||
|
||||
@Test
|
||||
void urdfResumeCommitsEarlierCoverChangeWithoutStartingAnotherTransaction() throws Exception {
|
||||
insert(CODE, BASE + "/" + ROOT + COVER, ROOT + MAIN, ROOT + PREFIX);
|
||||
source.getConnection().setAutoCommit(false);
|
||||
jdbc.update("UPDATE project SET cover_url = ? WHERE project_code = ?", COVER, CODE);
|
||||
execute("migration-resume-urdf");
|
||||
JdbcTemplate observer = new JdbcTemplate(
|
||||
new DriverManagerDataSource("jdbc:h2:mem:migration;MODE=MySQL", "sa", ""));
|
||||
assertThat(observer.queryForMap("SELECT * FROM project WHERE project_code = ?", CODE))
|
||||
.containsEntry("COVER_URL", COVER).containsEntry("URDF_MAIN_OBJECT_KEY", MAIN)
|
||||
.containsEntry("URDF_PREFIX", PREFIX);
|
||||
execute("migration-resume-urdf");
|
||||
assertThat(row(CODE).get("URDF_PREFIX")).isEqualTo(PREFIX);
|
||||
}
|
||||
|
||||
private Map<String, Object> row(String code) {
|
||||
return jdbc.queryForMap("SELECT * FROM project WHERE project_code = ?", code);
|
||||
}
|
||||
|
||||
private void execute(String name) {
|
||||
Matcher block = Pattern.compile("```sql\\s*-- resource-path-" + name + "\\s+([\\s\\S]*?)```").matcher(document);
|
||||
assertThat(block.find()).as("文档 SQL 块 %s", name).isTrue();
|
||||
String sql = block.group(1).replaceAll("(?m)^--.*$", "")
|
||||
.replace(" COLLATE utf8mb4_bin", "")
|
||||
.replace(" CHARACTER SET utf8mb4", "")
|
||||
.replace("DROP TEMPORARY TABLE", "DROP TABLE")
|
||||
.replace("START TRANSACTION", "SET AUTOCOMMIT FALSE")
|
||||
.replace("SET NAMES utf8mb4", "")
|
||||
.replace("<=>", "IS NOT DISTINCT FROM")
|
||||
.replace("[[:cntrl:]:]", "[\\p{Cntrl}:]");
|
||||
for (String statement : sql.split(";")) {
|
||||
if (statement.isBlank()) continue;
|
||||
Matcher update = Pattern.compile("UPDATE project p JOIN (\\w+) c ON c.project_code = p.project_code\\s+SET ([\\s\\S]*?)\\s+WHERE ([\\s\\S]*)")
|
||||
.matcher(statement.trim());
|
||||
if (update.matches()) {
|
||||
statement = "MERGE INTO project p USING " + update.group(1)
|
||||
+ " c ON c.project_code = p.project_code WHEN MATCHED AND " + update.group(3)
|
||||
+ " THEN UPDATE SET " + update.group(2).replace("p.", "");
|
||||
}
|
||||
jdbc.execute(statement);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -14,6 +14,8 @@ import com.linkerbot.community.service.config.UrdfProperties;
|
||||
import com.linkerbot.community.common.BusinessException;
|
||||
|
||||
import org.junit.jupiter.api.*;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.ValueSource;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc;
|
||||
import org.springframework.boot.test.autoconfigure.web.servlet.MockMvcPrint;
|
||||
@@ -84,6 +86,7 @@ class WorkflowTest {
|
||||
snapshots.available = true;
|
||||
snapshots.beforePush = null;
|
||||
oss.setEnabled(false);
|
||||
oss.setLegacyPublicBaseUrls(List.of());
|
||||
ossObjects.reset();
|
||||
urdfLimits.setMaxFileBytes(52428800);
|
||||
urdfLimits.setMaxTotalBytes(209715200);
|
||||
@@ -358,6 +361,41 @@ class WorkflowTest {
|
||||
.path("data").path("coverUrl").asText()).isEqualTo(max);
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@ValueSource(booleans = {false, true})
|
||||
void hostedCoversStoreRelativePathsAndLegacyUrlsRemainUsableByFork(boolean legacy) throws Exception {
|
||||
enableUrdfOss();
|
||||
String owner = login("hosted-cover-owner@example.com");
|
||||
String other = login("hosted-cover-other@example.com");
|
||||
String stored = "/project-covers/111111111111/20261019/a.png";
|
||||
String uploaded = oss.uploadUrl() + "/vibemake_dev" + stored;
|
||||
String expected = "https://images.example.com/assets/vibemake_dev" + stored;
|
||||
Map<String, Object> input = new LinkedHashMap<>(Map.of(
|
||||
"name", "平台封面", "projectType", "GENERAL", "coverUrl", uploaded));
|
||||
JsonNode created = perform(body(auth(post("/api/workspace/projects"), owner), input), 200).path("data");
|
||||
String code = created.path("projectCode").asText();
|
||||
String endpoint = "/api/workspace/projects/" + code;
|
||||
assertThat(created.path("coverUrl").asText()).isEqualTo(expected);
|
||||
assertThat(jdbc.queryForObject("SELECT cover_url FROM project WHERE project_code = ?", String.class, code)).isEqualTo(stored);
|
||||
if (legacy) jdbc.update("UPDATE project SET cover_url = ? WHERE project_code = ?", uploaded, code);
|
||||
assertThat(perform(auth(get(endpoint), owner), 200).path("data").path("coverUrl").asText()).isEqualTo(expected);
|
||||
assertThat(jdbc.queryForObject("SELECT cover_url FROM project WHERE project_code = ?", String.class, code))
|
||||
.isEqualTo(legacy ? uploaded : stored); // 读取不会隐式迁移。
|
||||
String sha = head(owner, code);
|
||||
publish(owner, code, sha);
|
||||
JsonNode fork = perform(body(auth(post("/api/community/projects/" + code + "/forks"), other),
|
||||
Map.of("name", "封面副本", "publishedCommitSha", sha)), 200).path("data");
|
||||
assertThat(fork.path("coverUrl").asText()).isEqualTo(expected);
|
||||
assertThat(jdbc.queryForObject("SELECT cover_url FROM project WHERE project_code = ?", String.class,
|
||||
fork.path("projectCode").asText())).isEqualTo(stored);
|
||||
input.put("coverUrl", expected);
|
||||
perform(body(auth(patch(endpoint), owner), input), 200);
|
||||
assertThat(jdbc.queryForObject("SELECT cover_url FROM project WHERE project_code = ?", String.class, code)).isEqualTo(stored);
|
||||
oss.setPublicBaseUrl("https://new-cdn.example.com/base");
|
||||
assertThat(perform(get("/api/community/projects/" + code), 200).path("data").path("coverUrl").asText())
|
||||
.isEqualTo("https://new-cdn.example.com/base/vibemake_dev" + stored);
|
||||
}
|
||||
|
||||
private void enableUrdfOss() {
|
||||
oss.setRegion("cn-hangzhou");
|
||||
oss.setEndpoint("");
|
||||
@@ -463,7 +501,9 @@ class WorkflowTest {
|
||||
assertThat(bound.path("urdfMainUrl").asText()).isEqualTo(baseUrl + "urdf/%E6%9C%BA%E6%A2%B0%20%E8%87%82.URDF");
|
||||
assertThat(bound.has("id")).isFalse();
|
||||
assertThat(jdbc.queryForObject("SELECT urdf_main_object_key FROM project WHERE project_code = ?", String.class, code))
|
||||
.isEqualTo(expectedPrefix + "urdf/机械 臂.URDF");
|
||||
.isEqualTo(expectedPrefix.substring("vibemake_dev".length()) + "urdf/机械 臂.URDF");
|
||||
assertThat(jdbc.queryForObject("SELECT urdf_prefix FROM project WHERE project_code = ?", String.class, code))
|
||||
.isEqualTo(expectedPrefix.substring("vibemake_dev".length()));
|
||||
assertThat(json.readTree(jdbc.queryForObject("SELECT urdf_files FROM project WHERE project_code = ?", String.class, code)))
|
||||
.isEqualTo(json.valueToTree(files));
|
||||
assertThat(perform(get("/api/community/projects/" + code), 200).path("data").path("urdfMainUrl")).isEqualTo(bound.path("urdfMainUrl"));
|
||||
@@ -510,8 +550,9 @@ class WorkflowTest {
|
||||
.contains(first.path("batchCode").asText());
|
||||
}
|
||||
|
||||
@Test
|
||||
void urdfForkCopiesIndependentObjectsAndCopyFailureDoesNotReturnReady() throws Exception {
|
||||
@ParameterizedTest
|
||||
@ValueSource(booleans = {false, true})
|
||||
void urdfForkCopiesIndependentObjectsAndCopyFailureDoesNotReturnReady(boolean legacy) throws Exception {
|
||||
String owner = login("urdf-fork-source@example.com");
|
||||
String other = login("urdf-fork-target@example.com");
|
||||
String code = create(owner, "URDF 来源");
|
||||
@@ -519,6 +560,8 @@ class WorkflowTest {
|
||||
List<Map<String, Object>> files = urdfFiles();
|
||||
JsonNode uploaded = uploadUrdf(owner, code, files);
|
||||
JsonNode source = confirmUrdf(owner, code, uploaded, files, 200).path("data");
|
||||
if (legacy) jdbc.update("UPDATE project SET urdf_prefix = CONCAT('vibemake_dev', urdf_prefix),"
|
||||
+ " urdf_main_object_key = CONCAT('vibemake_dev', urdf_main_object_key) WHERE project_code = ?", code);
|
||||
String sha = head(owner, code);
|
||||
publish(owner, code, sha);
|
||||
String forkEndpoint = "/api/community/projects/" + code + "/forks";
|
||||
@@ -528,8 +571,9 @@ class WorkflowTest {
|
||||
assertThat(fork.path("repoStatus").asText()).isEqualTo("READY");
|
||||
assertThat(fork.path("urdfMainUrl").asText()).contains("/urdf/" + targetCode + "/").doesNotContain("/urdf/" + code + "/");
|
||||
String targetPrefix = jdbc.queryForObject("SELECT urdf_prefix FROM project WHERE project_code = ?", String.class, targetCode);
|
||||
assertThat(targetPrefix).startsWith("/urdf/" + targetCode + "/");
|
||||
for (int index = 0; index < files.size(); index++)
|
||||
assertThat(ossObjects.objects.get(targetPrefix + files.get(index).get("relativePath")))
|
||||
assertThat(ossObjects.objects.get("vibemake_dev" + targetPrefix + files.get(index).get("relativePath")))
|
||||
.containsExactly(ossObjects.objects.get(uploaded.path("files").get(index).path("objectKey").asText()));
|
||||
confirmUrdf(owner, code, uploadUrdf(owner, code, files), files, 200);
|
||||
perform(auth(delete("/api/workspace/projects/" + code + "/urdf"), owner), 200);
|
||||
|
||||
+6
-2
@@ -2,6 +2,8 @@ package com.linkerbot.community.importer;
|
||||
|
||||
import com.linkerbot.community.service.config.CommunityProperties;
|
||||
import com.linkerbot.community.service.config.MybatisConfiguration;
|
||||
import com.linkerbot.community.service.config.OssProperties;
|
||||
import com.linkerbot.community.service.oss.OssResourcePaths;
|
||||
import com.linkerbot.community.service.gitea.RestGiteaClient;
|
||||
import com.linkerbot.community.service.impl.ProjectServiceImpl;
|
||||
import com.linkerbot.community.service.impl.UserServiceImpl;
|
||||
@@ -12,7 +14,7 @@ import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
import org.springframework.context.annotation.Import;
|
||||
|
||||
/** 显式加载所需组件,避免扫描在线应用的安全、AI 和 OSS 模块。 */
|
||||
/** 仅加载资源路径配置,不初始化在线安全、AI 或 OSS 客户端。 */
|
||||
@Configuration(proxyBeanMethods = false)
|
||||
@EnableAutoConfiguration
|
||||
@MapperScan("com.linkerbot.community.service.mapper")
|
||||
@@ -22,6 +24,8 @@ import org.springframework.context.annotation.Import;
|
||||
RestGiteaClient.class,
|
||||
ProjectServiceImpl.class,
|
||||
UserServiceImpl.class,
|
||||
ProjectLocks.class
|
||||
ProjectLocks.class,
|
||||
OssProperties.class,
|
||||
OssResourcePaths.class
|
||||
})
|
||||
public class DatabaseConfiguration {}
|
||||
|
||||
@@ -8,6 +8,7 @@ import com.linkerbot.community.bean.entity.User;
|
||||
import com.linkerbot.community.common.BusinessException;
|
||||
import com.linkerbot.community.common.ProjectCodes;
|
||||
import com.linkerbot.community.service.ProjectService;
|
||||
import com.linkerbot.community.service.oss.OssResourcePaths;
|
||||
import com.linkerbot.community.service.config.CommunityProperties;
|
||||
import com.linkerbot.community.service.gitea.*;
|
||||
import com.linkerbot.community.service.mapper.ProjectMapper;
|
||||
@@ -90,6 +91,12 @@ public final class ImportApplication {
|
||||
}
|
||||
|
||||
public static void main(String[] args) {
|
||||
if (args.length > 0 && "covers".equals(args[0])) {
|
||||
int status = com.linkerbot.community.importer.cover.CoverMigrationApplication.runCli(
|
||||
Arrays.copyOfRange(args, 1, args.length));
|
||||
if (status != 0) System.exit(status);
|
||||
return;
|
||||
}
|
||||
if (args.length == 0 || Arrays.asList(args).contains("--help")) {
|
||||
System.out.println(
|
||||
"用法:--excel <文件> [--state-dir <目录>] [--dry-run | --execute (--owner-user-code"
|
||||
@@ -224,6 +231,7 @@ public final class ImportApplication {
|
||||
context.getBean(ProjectService.class),
|
||||
context.getBean(GiteaClient.class),
|
||||
context.getBean(CommunityProperties.class),
|
||||
context.getBean(OssResourcePaths.class),
|
||||
store,
|
||||
owner,
|
||||
options.migrationTimeout());
|
||||
|
||||
+5
-1
@@ -8,6 +8,7 @@ import com.linkerbot.community.bean.entity.User;
|
||||
import com.linkerbot.community.common.BusinessException;
|
||||
import com.linkerbot.community.service.ProjectService;
|
||||
import com.linkerbot.community.service.config.CommunityProperties;
|
||||
import com.linkerbot.community.service.oss.OssResourcePaths;
|
||||
import com.linkerbot.community.service.gitea.*;
|
||||
import com.linkerbot.community.service.gitea.GiteaClient.RemoteRepository;
|
||||
import com.linkerbot.community.service.mapper.ProjectMapper;
|
||||
@@ -27,6 +28,7 @@ public final class ImportCoordinator {
|
||||
private final ProjectService service;
|
||||
private final GiteaClient gitea;
|
||||
private final CommunityProperties config;
|
||||
private final OssResourcePaths resourcePaths;
|
||||
private final StateStore store;
|
||||
private final User owner;
|
||||
private final Duration timeout;
|
||||
@@ -46,6 +48,7 @@ public final class ImportCoordinator {
|
||||
ProjectService service,
|
||||
GiteaClient gitea,
|
||||
CommunityProperties config,
|
||||
OssResourcePaths resourcePaths,
|
||||
StateStore store,
|
||||
User owner,
|
||||
Duration timeout) {
|
||||
@@ -53,6 +56,7 @@ public final class ImportCoordinator {
|
||||
this.service = service;
|
||||
this.gitea = gitea;
|
||||
this.config = config;
|
||||
this.resourcePaths = resourcePaths;
|
||||
this.store = store;
|
||||
this.owner = owner;
|
||||
this.timeout = timeout;
|
||||
@@ -97,7 +101,7 @@ public final class ImportCoordinator {
|
||||
project.setOwnerUserId(owner.getId());
|
||||
project.setName(entry.candidate.name().trim());
|
||||
project.setDescription(entry.candidate.description());
|
||||
project.setCoverUrl(ProjectInputs.coverUrl(entry.candidate.coverUrl()));
|
||||
project.setCoverUrl(resourcePaths.coverStorage(entry.candidate.coverUrl()));
|
||||
project.setProjectType(entry.candidate.projectType());
|
||||
project.setGiteaOwner(config.getGitea().getOwner());
|
||||
project.setGiteaRepoName(repoName);
|
||||
|
||||
+172
@@ -0,0 +1,172 @@
|
||||
package com.linkerbot.community.importer.cover;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.fasterxml.jackson.core.type.TypeReference;
|
||||
import com.linkerbot.community.service.config.OssProperties;
|
||||
import com.linkerbot.community.service.oss.OssResourcePaths;
|
||||
import java.nio.ByteBuffer;
|
||||
import java.nio.channels.FileChannel;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.nio.file.StandardOpenOption;
|
||||
import java.nio.file.attribute.PosixFilePermissions;
|
||||
import java.time.Instant;
|
||||
import java.sql.SQLException;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.UUID;
|
||||
import org.springframework.boot.WebApplicationType;
|
||||
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
|
||||
import org.springframework.boot.builder.SpringApplicationBuilder;
|
||||
import org.springframework.context.ConfigurableApplicationContext;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
import org.springframework.context.annotation.Import;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
|
||||
public final class CoverMigrationApplication {
|
||||
@Configuration(proxyBeanMethods = false)
|
||||
@EnableAutoConfiguration
|
||||
@Import({OssProperties.class, OssResourcePaths.class})
|
||||
public static class Config {}
|
||||
|
||||
public record Options(boolean execute, int limit, int concurrency, int maxBytes, Path reportDirectory, Path sourceOverrides) {
|
||||
public static Options parse(String[] args) {
|
||||
boolean execute = false;
|
||||
boolean dryRun = false;
|
||||
int limit = Integer.MAX_VALUE;
|
||||
int concurrency = 2;
|
||||
int maxBytes = 5 * 1024 * 1024;
|
||||
Path directory = Path.of("logs/cover-migration");
|
||||
Path overrides = null;
|
||||
List<String> seen = new ArrayList<>();
|
||||
for (int index = 0; index < args.length; index++) {
|
||||
String key = args[index];
|
||||
if (seen.contains(key)) throw new IllegalArgumentException("参数重复");
|
||||
seen.add(key);
|
||||
if ("--execute".equals(key)) execute = true;
|
||||
else if ("--dry-run".equals(key)) dryRun = true;
|
||||
else {
|
||||
if (index + 1 == args.length) throw new IllegalArgumentException("缺少参数值");
|
||||
String value = args[++index];
|
||||
switch (key) {
|
||||
case "--limit" -> limit = Integer.parseInt(value);
|
||||
case "--concurrency" -> concurrency = Integer.parseInt(value);
|
||||
case "--max-bytes" -> maxBytes = Integer.parseInt(value);
|
||||
case "--report-dir" -> directory = Path.of(value);
|
||||
case "--source-overrides" -> overrides = Path.of(value);
|
||||
default -> throw new IllegalArgumentException("不支持的参数");
|
||||
}
|
||||
}
|
||||
}
|
||||
if (execute && dryRun || limit < 1 || concurrency < 1 || concurrency > 4 || maxBytes < 1 || maxBytes > 50 * 1024 * 1024)
|
||||
throw new IllegalArgumentException("参数范围不合法");
|
||||
return new Options(execute, limit, concurrency, maxBytes, directory, overrides);
|
||||
}
|
||||
}
|
||||
|
||||
public static int runCli(String[] args) {
|
||||
if (Arrays.asList(args).contains("--help")) {
|
||||
System.out.println("用法:covers [--dry-run | --execute] [--limit N] [--concurrency 2]"
|
||||
+ " [--max-bytes 5242880] [--report-dir logs/cover-migration] [--source-overrides <JSON文件>]");
|
||||
return 0;
|
||||
}
|
||||
try {
|
||||
Options options = Options.parse(args);
|
||||
try (ConfigurableApplicationContext context = new SpringApplicationBuilder(Config.class)
|
||||
.web(WebApplicationType.NONE).logStartupInfo(false).run()) {
|
||||
return run(options, context.getBean(JdbcTemplate.class), context.getBean(OssProperties.class),
|
||||
context.getBean(OssResourcePaths.class));
|
||||
}
|
||||
} catch (Exception error) {
|
||||
String code = error instanceof ExternalCoverDownloader.Failure failure ? failure.code() : error.getClass().getSimpleName();
|
||||
System.err.println("封面迁移终止 code=" + code + ";检查本地配置和运行报告,不输出凭据或原始异常文本");
|
||||
for (Throwable cause = error; cause != null; cause = cause.getCause()) {
|
||||
if (cause instanceof SQLException sql)
|
||||
System.err.printf("数据库异常 sqlState=%s vendorCode=%d type=%s%n", sql.getSQLState(), sql.getErrorCode(), sql.getClass().getSimpleName());
|
||||
}
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
static int run(Options options, JdbcTemplate jdbc, OssProperties oss, OssResourcePaths paths) throws Exception {
|
||||
CoverMigrator.Repository repository = new CoverMigrator.JdbcRepository(jdbc);
|
||||
ExternalCoverDownloader downloader = new ExternalCoverDownloader(options.maxBytes());
|
||||
CoverMigrator scanner = new CoverMigrator(repository, paths, downloader::download, (key, image, sha) -> {});
|
||||
List<CoverMigrator.Candidate> candidates = scanner.candidates(options.limit());
|
||||
long unique = candidates.stream().map(candidate -> candidate.originalUrl().trim()).distinct().count();
|
||||
System.out.printf("封面迁移预览 environmentRoot=%s candidateProjects=%d uniqueUrls=%d execute=%s%n",
|
||||
oss.getRootDirectory(), candidates.size(), unique, options.execute());
|
||||
Path run = options.reportDirectory().resolve(Instant.now().toString().replace(':', '-') + "-" + UUID.randomUUID().toString().substring(0, 8));
|
||||
Files.createDirectories(run);
|
||||
restrict(run, "rwx------");
|
||||
ObjectMapper json = new ObjectMapper();
|
||||
Map<String, String> overrides = options.sourceOverrides() == null ? Map.of()
|
||||
: json.readValue(Files.readString(options.sourceOverrides()), new TypeReference<Map<String, String>>() {});
|
||||
Map<String, String> sources = downloadSources(candidates, overrides);
|
||||
List<Map<String, String>> preview = candidates.stream().map(candidate -> Map.of(
|
||||
"projectCode", candidate.projectCode(), "sourceHost", String.valueOf(CoverMigrator.host(candidate.originalUrl())))).toList();
|
||||
writeReport(json, run.resolve("preview.json"), preview);
|
||||
System.out.println("迁移报告目录=" + run.toAbsolutePath());
|
||||
if (!options.execute() || candidates.isEmpty()) return 0;
|
||||
if (!sources.isEmpty()) writeReport(json, run.resolve("source-overrides-backup.json"), overrides);
|
||||
try (CoverOssUploader uploader = new CoverOssUploader(oss);
|
||||
DurableBackup backup = new DurableBackup(run.resolve("backup.jsonl"), json)) {
|
||||
CoverMigrator migrator = new CoverMigrator(repository, paths,
|
||||
url -> downloader.download(sources.getOrDefault(url, url)), uploader);
|
||||
List<CoverMigrator.Result> results = migrator.migrate(candidates, options.concurrency(), backup::record);
|
||||
writeReport(json, run.resolve("report.json"), results);
|
||||
long updated = results.stream().filter(result -> "UPDATED".equals(result.status())).count();
|
||||
long failed = results.stream().filter(result -> "FAILED".equals(result.status())).count();
|
||||
long changed = results.size() - updated - failed;
|
||||
System.out.printf("封面迁移结束 updated=%d failed=%d changedSkipped=%d%n", updated, failed, changed);
|
||||
return failed == 0 ? 0 : 2;
|
||||
}
|
||||
}
|
||||
|
||||
static Map<String, String> downloadSources(List<CoverMigrator.Candidate> candidates, Map<String, String> overrides) {
|
||||
Map<String, String> sources = new LinkedHashMap<>();
|
||||
for (CoverMigrator.Candidate candidate : candidates) {
|
||||
String source = overrides.get(candidate.projectCode());
|
||||
if (source == null) continue;
|
||||
source = com.linkerbot.community.service.support.ProjectInputs.coverUrl(source);
|
||||
if (source == null) throw new IllegalArgumentException("备用下载地址不能为空");
|
||||
String previous = sources.putIfAbsent(candidate.originalUrl().trim(), source);
|
||||
if (previous != null && !previous.equals(source))
|
||||
throw new IllegalArgumentException("相同原始封面不能配置不同下载源");
|
||||
}
|
||||
return Map.copyOf(sources);
|
||||
}
|
||||
|
||||
private static void writeReport(ObjectMapper json, Path path, Object value) throws Exception {
|
||||
Files.write(path, json.writerWithDefaultPrettyPrinter().writeValueAsBytes(value), StandardOpenOption.CREATE_NEW);
|
||||
restrict(path, "rw-------");
|
||||
}
|
||||
|
||||
static void restrict(Path path, String mode) throws Exception {
|
||||
if (Files.getFileStore(path).supportsFileAttributeView("posix"))
|
||||
Files.setPosixFilePermissions(path, PosixFilePermissions.fromString(mode));
|
||||
}
|
||||
|
||||
static final class DurableBackup implements AutoCloseable {
|
||||
private final FileChannel channel;
|
||||
private final ObjectMapper json;
|
||||
DurableBackup(Path path, ObjectMapper json) throws Exception {
|
||||
this.json = json;
|
||||
channel = FileChannel.open(path, StandardOpenOption.CREATE_NEW, StandardOpenOption.WRITE);
|
||||
restrict(path, "rw-------");
|
||||
}
|
||||
synchronized void record(CoverMigrator.Candidate candidate, String path) throws Exception {
|
||||
// 原始地址仅保存于受限备份,不进入普通日志;先持久化再修改数据库。
|
||||
byte[] bytes = (json.writeValueAsString(Map.of("projectCode", candidate.projectCode(),
|
||||
"originalCoverUrl", candidate.originalUrl(), "newCoverPath", path)) + "\n").getBytes(StandardCharsets.UTF_8);
|
||||
ByteBuffer buffer = ByteBuffer.wrap(bytes);
|
||||
while (buffer.hasRemaining()) channel.write(buffer);
|
||||
channel.force(true);
|
||||
}
|
||||
@Override public void close() throws Exception { channel.close(); }
|
||||
}
|
||||
}
|
||||
+137
@@ -0,0 +1,137 @@
|
||||
package com.linkerbot.community.importer.cover;
|
||||
|
||||
import com.linkerbot.community.common.BusinessException;
|
||||
import com.linkerbot.community.service.oss.OssResourcePaths;
|
||||
import java.net.URI;
|
||||
import java.security.MessageDigest;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collections;
|
||||
import java.util.Comparator;
|
||||
import java.util.HexFormat;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.ExecutorService;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.Future;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
|
||||
public class CoverMigrator {
|
||||
public record Candidate(String projectCode, String originalUrl) {}
|
||||
public record Result(String projectCode, String sourceHost, String status, String code, String coverPath) {}
|
||||
public interface Repository {
|
||||
List<Candidate> list();
|
||||
boolean replace(Candidate candidate, String coverPath);
|
||||
}
|
||||
@FunctionalInterface public interface Downloader { ExternalCoverDownloader.Image download(String url) throws Exception; }
|
||||
@FunctionalInterface public interface Uploader { void upload(String key, ExternalCoverDownloader.Image image, String sha256) throws Exception; }
|
||||
@FunctionalInterface public interface Backup { void record(Candidate candidate, String path) throws Exception; }
|
||||
|
||||
public static class JdbcRepository implements Repository {
|
||||
private final JdbcTemplate jdbc;
|
||||
public JdbcRepository(JdbcTemplate jdbc) { this.jdbc = jdbc; }
|
||||
@Override public List<Candidate> list() {
|
||||
return jdbc.query("SELECT project_code, cover_url FROM project WHERE cover_url IS NOT NULL"
|
||||
+ " AND cover_url <> '' ORDER BY project_code",
|
||||
(row, index) -> new Candidate(row.getString("project_code"), row.getString("cover_url")));
|
||||
}
|
||||
@Override public boolean replace(Candidate candidate, String path) {
|
||||
// HEX 比较原始 UTF-8 字节,避免默认不区分大小写的排序规则覆盖并发修改。
|
||||
return jdbc.update("UPDATE project SET cover_url = ? WHERE project_code = ? AND HEX(cover_url) = HEX(?)",
|
||||
path, candidate.projectCode(), candidate.originalUrl()) == 1;
|
||||
}
|
||||
}
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(CoverMigrator.class);
|
||||
private final Repository repository;
|
||||
private final OssResourcePaths paths;
|
||||
private final Downloader downloader;
|
||||
private final Uploader uploader;
|
||||
|
||||
public CoverMigrator(Repository repository, OssResourcePaths paths, Downloader downloader, Uploader uploader) {
|
||||
this.repository = repository;
|
||||
this.paths = paths;
|
||||
this.downloader = downloader;
|
||||
this.uploader = uploader;
|
||||
}
|
||||
|
||||
public List<Candidate> candidates(int limit) {
|
||||
List<Candidate> candidates = new ArrayList<>();
|
||||
for (Candidate candidate : repository.list()) {
|
||||
String value = candidate.originalUrl();
|
||||
if (value == null || value.isBlank() || value.startsWith("/")) continue;
|
||||
try {
|
||||
String normalized = paths.coverStorage(value);
|
||||
if (normalized != null && normalized.startsWith("/project-covers/")) continue;
|
||||
} catch (BusinessException | IllegalArgumentException error) {
|
||||
// 非法历史外链进入失败报告,不因单条坏数据终止整个批次。
|
||||
}
|
||||
candidates.add(candidate);
|
||||
if (candidates.size() == limit) break;
|
||||
}
|
||||
return List.copyOf(candidates);
|
||||
}
|
||||
|
||||
public List<Result> migrate(List<Candidate> candidates, int concurrency, Backup backup) throws Exception {
|
||||
Map<String, List<Candidate>> groups = new LinkedHashMap<>();
|
||||
for (Candidate candidate : candidates)
|
||||
groups.computeIfAbsent(candidate.originalUrl().trim(), ignored -> new ArrayList<>()).add(candidate);
|
||||
List<Result> results = Collections.synchronizedList(new ArrayList<>());
|
||||
try (ExecutorService executor = Executors.newFixedThreadPool(concurrency)) {
|
||||
List<Future<?>> jobs = new ArrayList<>();
|
||||
for (Map.Entry<String, List<Candidate>> group : groups.entrySet())
|
||||
jobs.add(executor.submit(() -> migrateGroup(group.getKey(), group.getValue(), backup, results)));
|
||||
for (Future<?> job : jobs) job.get();
|
||||
}
|
||||
results.sort(Comparator.comparing(Result::projectCode));
|
||||
return List.copyOf(results);
|
||||
}
|
||||
|
||||
private void migrateGroup(String url, List<Candidate> group, Backup backup, List<Result> results) {
|
||||
String phase = "DOWNLOAD";
|
||||
String path;
|
||||
try {
|
||||
ExternalCoverDownloader.Image image = downloader.download(url);
|
||||
String digest = HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(image.bytes()));
|
||||
path = "/project-covers/migrated/" + digest + "." + image.extension();
|
||||
phase = "UPLOAD";
|
||||
uploader.upload(paths.objectKey(path), image, digest);
|
||||
} catch (Exception error) {
|
||||
for (Candidate candidate : group) failed(candidate, phase, error, results);
|
||||
return;
|
||||
}
|
||||
for (Candidate candidate : group) {
|
||||
phase = "BACKUP";
|
||||
try {
|
||||
backup.record(candidate, path);
|
||||
phase = "UPDATE";
|
||||
boolean updated = repository.replace(candidate, path);
|
||||
String status = updated ? "UPDATED" : "CHANGED";
|
||||
results.add(new Result(candidate.projectCode(), host(url), status, "", path));
|
||||
log.info("封面转存完成 projectCode={} status={} coverPath={}", candidate.projectCode(), status, path);
|
||||
} catch (Exception error) {
|
||||
failed(candidate, phase, error, results);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private void failed(Candidate candidate, String phase, Exception error, List<Result> results) {
|
||||
String code = error instanceof ExternalCoverDownloader.Failure failure ? failure.code()
|
||||
: error instanceof BusinessException business ? business.code() : error.getClass().getSimpleName();
|
||||
if (error.getCause() != null) code += "/" + error.getCause().getClass().getSimpleName();
|
||||
results.add(new Result(candidate.projectCode(), host(candidate.originalUrl()), "FAILED", phase + ":" + code, null));
|
||||
// 异常文本可能包含带参数的原始 URL,不打印原始异常及查询参数。
|
||||
log.warn("封面转存失败 projectCode={} sourceHost={} phase={} code={}",
|
||||
candidate.projectCode(), host(candidate.originalUrl()), phase, code);
|
||||
}
|
||||
|
||||
public static String host(String url) {
|
||||
try {
|
||||
String host = URI.create(url.trim()).getHost();
|
||||
return host == null ? "invalid" : host;
|
||||
}
|
||||
catch (IllegalArgumentException error) { return "invalid"; }
|
||||
}
|
||||
}
|
||||
+72
@@ -0,0 +1,72 @@
|
||||
package com.linkerbot.community.importer.cover;
|
||||
|
||||
import com.aliyun.oss.ClientBuilderConfiguration;
|
||||
import com.aliyun.oss.OSS;
|
||||
import com.aliyun.oss.OSSClientBuilder;
|
||||
import com.aliyun.oss.OSSException;
|
||||
import com.aliyun.oss.common.auth.DefaultCredentialProvider;
|
||||
import com.aliyun.oss.common.comm.SignVersion;
|
||||
import com.aliyun.oss.model.ObjectMetadata;
|
||||
import com.aliyun.oss.model.PutObjectRequest;
|
||||
import com.linkerbot.community.service.config.OssProperties;
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.util.Map;
|
||||
|
||||
final class CoverOssUploader implements AutoCloseable, CoverMigrator.Uploader {
|
||||
private final OSS client;
|
||||
private final String bucket;
|
||||
|
||||
CoverOssUploader(OssProperties properties) {
|
||||
properties.setEnabled(true);
|
||||
if (!properties.isConfigurationValid()) throw new ExternalCoverDownloader.Failure("OSS_CONFIGURATION_MISSING");
|
||||
ClientBuilderConfiguration configuration = new ClientBuilderConfiguration();
|
||||
configuration.setSignatureVersion(SignVersion.V4);
|
||||
configuration.setConnectionTimeout(10000);
|
||||
configuration.setSocketTimeout(20000);
|
||||
configuration.setRequestTimeoutEnabled(true);
|
||||
configuration.setRequestTimeout(30000);
|
||||
configuration.setMaxErrorRetry(1);
|
||||
client = OSSClientBuilder.create().endpoint(properties.resolvedEndpoint()).region(properties.getRegion())
|
||||
.credentialsProvider(new DefaultCredentialProvider(properties.getAccessKeyId(), properties.getAccessKeySecret()))
|
||||
.clientConfiguration(configuration).build();
|
||||
bucket = properties.getBucket();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void upload(String key, ExternalCoverDownloader.Image image, String sha256) {
|
||||
ObjectMetadata existing = metadata(key);
|
||||
if (existing != null) {
|
||||
verify(existing, image, sha256);
|
||||
return;
|
||||
}
|
||||
ObjectMetadata metadata = new ObjectMetadata();
|
||||
metadata.setContentLength(image.bytes().length);
|
||||
metadata.setContentType(image.contentType());
|
||||
metadata.setUserMetadata(Map.of("sha256", sha256));
|
||||
metadata.setHeader("x-oss-forbid-overwrite", "true");
|
||||
try {
|
||||
client.putObject(new PutObjectRequest(bucket, key, new ByteArrayInputStream(image.bytes()), metadata));
|
||||
} catch (OSSException error) {
|
||||
if (!"FileAlreadyExists".equals(error.getErrorCode()))
|
||||
throw new ExternalCoverDownloader.Failure("OSS_PUT_" + error.getErrorCode());
|
||||
}
|
||||
ObjectMetadata uploaded = metadata(key);
|
||||
if (uploaded == null) throw new ExternalCoverDownloader.Failure("OSS_VERIFY_MISSING");
|
||||
verify(uploaded, image, sha256);
|
||||
}
|
||||
|
||||
private ObjectMetadata metadata(String key) {
|
||||
try { return client.getObjectMetadata(bucket, key); }
|
||||
catch (OSSException error) {
|
||||
if ("NoSuchKey".equals(error.getErrorCode())) return null;
|
||||
throw new ExternalCoverDownloader.Failure("OSS_HEAD_" + error.getErrorCode());
|
||||
}
|
||||
}
|
||||
|
||||
private void verify(ObjectMetadata metadata, ExternalCoverDownloader.Image image, String sha256) {
|
||||
if (metadata.getContentLength() != image.bytes().length || !sha256.equals(metadata.getUserMetadata().get("sha256")))
|
||||
throw new ExternalCoverDownloader.Failure("OSS_VERIFY_MISMATCH");
|
||||
}
|
||||
|
||||
@Override public void close() { client.shutdown(); }
|
||||
}
|
||||
+132
@@ -0,0 +1,132 @@
|
||||
package com.linkerbot.community.importer.cover;
|
||||
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.io.InputStream;
|
||||
import java.io.IOException;
|
||||
import java.net.HttpURLConnection;
|
||||
import java.net.InetAddress;
|
||||
import java.net.URI;
|
||||
import java.time.Duration;
|
||||
import java.util.Arrays;
|
||||
|
||||
/** 仅下载公开 HTTPS 图片,不附加任何 GitHub、Gitea 或 OSS 凭据。 */
|
||||
public class ExternalCoverDownloader {
|
||||
public record Image(byte[] bytes, String contentType, String extension) {}
|
||||
|
||||
public static class Failure extends RuntimeException {
|
||||
private final String code;
|
||||
|
||||
public Failure(String code) {
|
||||
super(code);
|
||||
this.code = code;
|
||||
}
|
||||
|
||||
public String code() { return code; }
|
||||
}
|
||||
|
||||
private final int maxBytes;
|
||||
|
||||
public ExternalCoverDownloader(int maxBytes) { this.maxBytes = maxBytes; }
|
||||
|
||||
public Image download(String url) throws Exception {
|
||||
for (int attempt = 1; attempt <= 3; attempt++) {
|
||||
try { return fetch(url); }
|
||||
catch (IOException error) {
|
||||
if (attempt == 3 || !retryable(error) || Thread.currentThread().isInterrupted()) throw error;
|
||||
Thread.sleep(200L * attempt);
|
||||
}
|
||||
}
|
||||
throw new IllegalStateException("封面下载未完成");
|
||||
}
|
||||
|
||||
private boolean retryable(IOException error) {
|
||||
for (Throwable cause = error; cause != null; cause = cause.getCause()) {
|
||||
if (cause instanceof java.security.cert.CertificateException
|
||||
|| cause instanceof java.security.cert.CertPathValidatorException
|
||||
|| cause instanceof java.security.cert.CertPathBuilderException) return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
Image fetch(String url) throws Exception {
|
||||
URI uri = URI.create(url.trim());
|
||||
long deadline = System.nanoTime() + Duration.ofSeconds(45).toNanos();
|
||||
for (int redirects = 0; redirects <= 5; redirects++) {
|
||||
validate(uri);
|
||||
HttpURLConnection connection = (HttpURLConnection) uri.toURL().openConnection();
|
||||
connection.setInstanceFollowRedirects(false);
|
||||
connection.setConnectTimeout(10000);
|
||||
connection.setReadTimeout(15000);
|
||||
connection.setRequestProperty("User-Agent", "community-cover-migration/1");
|
||||
connection.setRequestProperty("Accept", "image/*");
|
||||
try {
|
||||
int status = connection.getResponseCode();
|
||||
if (status >= 300 && status < 400) {
|
||||
String location = connection.getHeaderField("Location");
|
||||
if (location == null) throw new Failure("COVER_REDIRECT_INVALID");
|
||||
uri = uri.resolve(location);
|
||||
if (System.nanoTime() > deadline) throw new Failure("COVER_DOWNLOAD_TIMEOUT");
|
||||
continue;
|
||||
}
|
||||
if (status != 200) throw new Failure("COVER_HTTP_" + status);
|
||||
long length = connection.getContentLengthLong();
|
||||
if (length > maxBytes) throw new Failure("COVER_TOO_LARGE");
|
||||
byte[] bytes;
|
||||
try (InputStream input = connection.getInputStream()) {
|
||||
bytes = read(input, maxBytes, deadline);
|
||||
}
|
||||
if (length >= 0 && length != bytes.length) throw new Failure("COVER_INCOMPLETE");
|
||||
return image(bytes);
|
||||
} finally {
|
||||
connection.disconnect();
|
||||
}
|
||||
}
|
||||
throw new Failure("COVER_REDIRECT_LIMIT");
|
||||
}
|
||||
|
||||
static void validate(URI uri) throws Exception {
|
||||
if (!"https".equalsIgnoreCase(uri.getScheme()) || uri.getHost() == null
|
||||
|| uri.getRawUserInfo() != null
|
||||
|| (uri.getPort() != -1 && uri.getPort() != 443))
|
||||
throw new Failure("COVER_URL_INVALID");
|
||||
for (InetAddress address : InetAddress.getAllByName(uri.getHost())) {
|
||||
byte[] bytes = address.getAddress();
|
||||
boolean reservedIpv4 = bytes.length == 4 && ((bytes[0] & 255) == 0
|
||||
|| (bytes[0] & 255) >= 224
|
||||
|| (bytes[0] & 255) == 100 && ((bytes[1] & 255) & 192) == 64
|
||||
|| (bytes[0] & 255) == 198 && ((bytes[1] & 255) == 18 || (bytes[1] & 255) == 19));
|
||||
boolean privateIpv6 = bytes.length == 16 && (bytes[0] & 254) == 252;
|
||||
if (address.isAnyLocalAddress() || address.isLoopbackAddress() || address.isLinkLocalAddress()
|
||||
|| address.isSiteLocalAddress() || address.isMulticastAddress() || reservedIpv4 || privateIpv6)
|
||||
throw new Failure("COVER_PRIVATE_ADDRESS");
|
||||
}
|
||||
}
|
||||
|
||||
static byte[] read(InputStream input, int maxBytes, long deadline) throws Exception {
|
||||
ByteArrayOutputStream output = new ByteArrayOutputStream();
|
||||
byte[] buffer = new byte[8192];
|
||||
int count;
|
||||
while ((count = input.read(buffer)) != -1) {
|
||||
if (System.nanoTime() > deadline || Thread.currentThread().isInterrupted())
|
||||
throw new Failure("COVER_DOWNLOAD_TIMEOUT");
|
||||
if (count > maxBytes - output.size()) throw new Failure("COVER_TOO_LARGE");
|
||||
output.write(buffer, 0, count);
|
||||
}
|
||||
return output.toByteArray();
|
||||
}
|
||||
|
||||
static Image image(byte[] bytes) {
|
||||
if (bytes.length >= 3 && (bytes[0] & 255) == 255 && (bytes[1] & 255) == 216 && (bytes[2] & 255) == 255)
|
||||
return new Image(bytes, "image/jpeg", "jpg");
|
||||
if (bytes.length >= 8 && Arrays.equals(Arrays.copyOf(bytes, 8),
|
||||
new byte[] {(byte) 137, 80, 78, 71, 13, 10, 26, 10}))
|
||||
return new Image(bytes, "image/png", "png");
|
||||
if (bytes.length >= 6 && bytes[0] == 'G' && bytes[1] == 'I' && bytes[2] == 'F'
|
||||
&& bytes[3] == '8' && (bytes[4] == '7' || bytes[4] == '9') && bytes[5] == 'a')
|
||||
return new Image(bytes, "image/gif", "gif");
|
||||
if (bytes.length >= 12 && bytes[0] == 'R' && bytes[1] == 'I' && bytes[2] == 'F' && bytes[3] == 'F'
|
||||
&& bytes[8] == 'W' && bytes[9] == 'E' && bytes[10] == 'B' && bytes[11] == 'P')
|
||||
return new Image(bytes, "image/webp", "webp");
|
||||
throw new Failure("COVER_FORMAT_UNSUPPORTED");
|
||||
}
|
||||
}
|
||||
@@ -17,6 +17,16 @@ mybatis-plus:
|
||||
configuration:
|
||||
map-underscore-to-camel-case: true
|
||||
community:
|
||||
oss:
|
||||
# 导入只转换资源地址,永不启用 OSS SDK 或校验上传凭据。
|
||||
enabled: false
|
||||
region: ${OSS_REGION:cn-beijing}
|
||||
bucket: ${OSS_BUCKET:linkerbot}
|
||||
access-key-id: ${OSS_ACCESS_KEY_ID:}
|
||||
access-key-secret: ${OSS_ACCESS_KEY_SECRET:}
|
||||
public-base-url: ${OSS_PUBLIC_BASE_URL:https://linkerbot.oss-cn-beijing.aliyuncs.com}
|
||||
legacy-public-base-urls: ${OSS_LEGACY_PUBLIC_BASE_URLS:}
|
||||
root-directory: vibemake_dev
|
||||
gitea:
|
||||
base-url: ${GITEA_BASE_URL:http://localhost:3000}
|
||||
owner: ${GITEA_ORGANIZATION:community}
|
||||
@@ -26,3 +36,11 @@ logging:
|
||||
root: WARN
|
||||
com.linkerbot.community: INFO
|
||||
com.linkerbot.community.service.mapper: WARN
|
||||
---
|
||||
spring:
|
||||
config:
|
||||
activate:
|
||||
on-profile: prod
|
||||
community:
|
||||
oss:
|
||||
root-directory: vibemake
|
||||
|
||||
+26
-1
@@ -11,6 +11,9 @@ import com.linkerbot.community.bean.entity.User;
|
||||
import com.linkerbot.community.common.*;
|
||||
import com.linkerbot.community.service.ProjectService;
|
||||
import com.linkerbot.community.service.config.CommunityProperties;
|
||||
import com.linkerbot.community.service.oss.OssResourcePaths;
|
||||
import com.linkerbot.community.service.oss.OssObjectStore;
|
||||
import com.linkerbot.community.service.config.OssProperties;
|
||||
import com.linkerbot.community.service.gitea.*;
|
||||
import com.linkerbot.community.service.mapper.*;
|
||||
|
||||
@@ -53,6 +56,9 @@ class ImportCoordinatorTest {
|
||||
@Autowired ProjectService service;
|
||||
@Autowired GiteaClient gitea;
|
||||
@Autowired CommunityProperties config;
|
||||
@Autowired OssResourcePaths resourcePaths;
|
||||
@Autowired OssProperties oss;
|
||||
@Autowired org.springframework.context.ApplicationContext context;
|
||||
@Autowired JdbcTemplate jdbc;
|
||||
@TempDir Path directory;
|
||||
private StateStore store;
|
||||
@@ -155,7 +161,7 @@ class ImportCoordinatorTest {
|
||||
|
||||
private ImportCoordinator worker(ProjectMapper mapper, ProjectService publisher) {
|
||||
return new ImportCoordinator(
|
||||
mapper, publisher, gitea, config, store, owner, Duration.ofMinutes(30));
|
||||
mapper, publisher, gitea, config, resourcePaths, store, owner, Duration.ofMinutes(30));
|
||||
}
|
||||
|
||||
@Test
|
||||
@@ -168,6 +174,9 @@ class ImportCoordinatorTest {
|
||||
assertThat(project.getPublishedCommitSha()).isEqualTo(SHA);
|
||||
assertThat(project.getSourceProjectId()).isNull();
|
||||
assertThat(project.getCoverUrl()).endsWith("a.gif");
|
||||
assertThat(oss.isEnabled()).isFalse();
|
||||
assertThat(oss.getAccessKeySecret()).isEmpty();
|
||||
assertThat(context.getBeansOfType(OssObjectStore.class)).isEmpty();
|
||||
assertThat(service.view(project, owner.getId()).projectCode()).isEqualTo(entry.projectCode);
|
||||
verify(gitea, never()).createRepository(anyString(), anyBoolean());
|
||||
verify(gitea).createBranch(remote.repository(), "main", SHA);
|
||||
@@ -175,6 +184,22 @@ class ImportCoordinatorTest {
|
||||
verify(gitea, times(1)).migrate(anyString(), anyString(), any(Duration.class));
|
||||
}
|
||||
|
||||
@Test
|
||||
void hostedImportCoversNormalizeWithoutOssCredentials() throws Exception {
|
||||
String path = "/project-covers/111111111111/20261019/a.png";
|
||||
Candidate original = entry.candidate;
|
||||
entry.candidate = new Candidate(original.sheet(), original.row(), original.name(), original.repositoryUrl(),
|
||||
original.description(), original.projectType(), oss.publicBaseUrl() + "/" + oss.getRootDirectory() + path);
|
||||
worker(projects, service).run(entry);
|
||||
assertThat(entry.phase).isEqualTo(Phase.PUBLISHED);
|
||||
Project project = service.published(entry.projectCode);
|
||||
assertThat(project.getCoverUrl()).isEqualTo(path);
|
||||
assertThat(service.view(project, owner.getId()).coverUrl())
|
||||
.isEqualTo(oss.publicBaseUrl() + "/" + oss.getRootDirectory() + path);
|
||||
assertThat(oss.isEnabled()).isFalse();
|
||||
assertThat(oss.getAccessKeyId()).isEmpty();
|
||||
}
|
||||
|
||||
@Test
|
||||
void normalizationErrorResumesOnlyAfterReadOnlyConfirmationOfAppliedState() throws Exception {
|
||||
doThrow(new GiteaFailure(502, "GITEA_ERROR", "写入后响应失败", true))
|
||||
|
||||
+134
@@ -0,0 +1,134 @@
|
||||
package com.linkerbot.community.importer.cover;
|
||||
|
||||
import static org.assertj.core.api.Assertions.*;
|
||||
import static org.mockito.ArgumentMatchers.*;
|
||||
import static org.mockito.Mockito.*;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.linkerbot.community.service.config.OssProperties;
|
||||
import com.linkerbot.community.service.oss.OssResourcePaths;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.HexFormat;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
import org.mockito.InOrder;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
import org.springframework.jdbc.datasource.DriverManagerDataSource;
|
||||
|
||||
public class CoverMigratorTest {
|
||||
@TempDir Path directory;
|
||||
private final CoverMigrator.Repository repository = mock(CoverMigrator.Repository.class);
|
||||
private final CoverMigrator.Downloader downloader = mock(CoverMigrator.Downloader.class);
|
||||
private final CoverMigrator.Uploader uploader = mock(CoverMigrator.Uploader.class);
|
||||
private final CoverMigrator.Backup backup = mock(CoverMigrator.Backup.class);
|
||||
private final OssProperties oss = properties();
|
||||
private final OssResourcePaths paths = new OssResourcePaths(oss);
|
||||
private final ExternalCoverDownloader.Image image = new ExternalCoverDownloader.Image(new byte[] {1, 2, 3}, "image/gif", "gif");
|
||||
private static final CoverMigrator.Candidate FIRST = new CoverMigrator.Candidate("1".repeat(12), "https://raw.example.com/a.gif");
|
||||
|
||||
private static OssProperties properties() {
|
||||
OssProperties properties = new OssProperties();
|
||||
properties.setRootDirectory("vibemake_dev");
|
||||
properties.setPublicBaseUrl("https://cdn.example.com");
|
||||
return properties;
|
||||
}
|
||||
|
||||
@Test
|
||||
void previewSkipsManagedReferencesWithoutAnyDownloadOrWrite() {
|
||||
when(repository.list()).thenReturn(List.of(
|
||||
new CoverMigrator.Candidate("1".repeat(12), "/project-covers/a.png"),
|
||||
new CoverMigrator.Candidate("2".repeat(12), "https://cdn.example.com/vibemake_dev/project-covers/a.png"), FIRST));
|
||||
CoverMigrator migrator = new CoverMigrator(repository, paths, downloader, uploader);
|
||||
assertThat(migrator.candidates(10)).containsExactly(FIRST);
|
||||
verify(repository, never()).replace(any(), anyString());
|
||||
verifyNoInteractions(downloader, uploader);
|
||||
}
|
||||
|
||||
@Test
|
||||
void duplicatesDownloadAndUploadOnceAndBackupPrecedesConditionalUpdate() throws Exception {
|
||||
CoverMigrator.Candidate second = new CoverMigrator.Candidate("2".repeat(12), FIRST.originalUrl());
|
||||
when(downloader.download(FIRST.originalUrl())).thenReturn(image);
|
||||
when(repository.replace(any(), anyString())).thenReturn(true);
|
||||
List<CoverMigrator.Result> results = new CoverMigrator(repository, paths, downloader, uploader)
|
||||
.migrate(List.of(FIRST, second), 2, backup);
|
||||
assertThat(results).allSatisfy(result -> {
|
||||
assertThat(result.status()).isEqualTo("UPDATED");
|
||||
assertThat(result.coverPath()).matches("/project-covers/migrated/[a-f0-9]{64}[.]gif");
|
||||
});
|
||||
verify(downloader).download(FIRST.originalUrl());
|
||||
verify(uploader).upload(startsWith("vibemake_dev/project-covers/migrated/"), same(image), anyString());
|
||||
InOrder order = inOrder(backup, repository);
|
||||
order.verify(backup).record(eq(FIRST), anyString());
|
||||
order.verify(repository).replace(eq(FIRST), anyString());
|
||||
order.verify(backup).record(eq(second), anyString());
|
||||
order.verify(repository).replace(eq(second), anyString());
|
||||
}
|
||||
|
||||
@Test
|
||||
void downloadUploadAndBackupFailuresNeverReplaceDatabaseValues() throws Exception {
|
||||
when(downloader.download(anyString())).thenThrow(new ExternalCoverDownloader.Failure("COVER_HTTP_404"));
|
||||
CoverMigrator migrator = new CoverMigrator(repository, paths, downloader, uploader);
|
||||
assertThat(migrator.migrate(List.of(FIRST), 1, backup).getFirst().code()).isEqualTo("DOWNLOAD:COVER_HTTP_404");
|
||||
verifyNoInteractions(uploader, backup);
|
||||
doReturn(image).when(downloader).download(anyString());
|
||||
doThrow(new IllegalStateException()).when(uploader).upload(anyString(), any(), anyString());
|
||||
assertThat(migrator.migrate(List.of(FIRST), 1, backup).getFirst().code()).startsWith("UPLOAD:");
|
||||
verifyNoInteractions(backup);
|
||||
doNothing().when(uploader).upload(anyString(), any(), anyString());
|
||||
doThrow(new IllegalStateException()).when(backup).record(any(), anyString());
|
||||
assertThat(migrator.migrate(List.of(FIRST), 1, backup).getFirst().code()).startsWith("BACKUP:");
|
||||
verify(repository, never()).replace(any(), anyString());
|
||||
}
|
||||
|
||||
@Test
|
||||
void conditionalUpdateProtectsCaseOnlyEditsAndIsIdempotent() throws Exception {
|
||||
JdbcTemplate jdbc = new JdbcTemplate(new DriverManagerDataSource("jdbc:h2:mem:cover_migrate;MODE=MySQL;DB_CLOSE_DELAY=-1", "sa", ""));
|
||||
jdbc.execute("DROP ALL OBJECTS");
|
||||
jdbc.execute("CREATE ALIAS HEX FOR 'com.linkerbot.community.importer.cover.CoverMigratorTest.hex'");
|
||||
jdbc.execute("CREATE TABLE project (project_code VARCHAR(32) PRIMARY KEY, cover_url VARCHAR(2048))");
|
||||
jdbc.update("INSERT INTO project VALUES (?, ?)", FIRST.projectCode(), FIRST.originalUrl());
|
||||
CoverMigrator.JdbcRepository database = new CoverMigrator.JdbcRepository(jdbc);
|
||||
when(downloader.download(anyString())).thenReturn(image);
|
||||
CoverMigrator migrator = new CoverMigrator(database, paths, downloader, uploader);
|
||||
List<CoverMigrator.Candidate> candidates = migrator.candidates(10);
|
||||
jdbc.update("UPDATE project SET cover_url = ?", "https://raw.example.com/A.gif");
|
||||
assertThat(migrator.migrate(candidates, 1, backup).getFirst().status()).isEqualTo("CHANGED");
|
||||
assertThat(jdbc.queryForObject("SELECT cover_url FROM project", String.class)).endsWith("A.gif");
|
||||
assertThat(migrator.migrate(migrator.candidates(10), 1, backup).getFirst().status()).isEqualTo("UPDATED");
|
||||
assertThat(migrator.candidates(10)).isEmpty();
|
||||
}
|
||||
|
||||
@Test
|
||||
void durableBackupPreservesOriginalValueBeforeWriteAndOptionsRejectConflicts() throws Exception {
|
||||
Path path = directory.resolve("backup.jsonl");
|
||||
try (CoverMigrationApplication.DurableBackup backupFile = new CoverMigrationApplication.DurableBackup(path, new ObjectMapper())) {
|
||||
backupFile.record(FIRST, "/project-covers/migrated/a.gif");
|
||||
assertThat(new ObjectMapper().readTree(Files.readString(path)).path("originalCoverUrl").asText())
|
||||
.isEqualTo(FIRST.originalUrl());
|
||||
}
|
||||
assertThat(CoverMigrationApplication.Options.parse(new String[0]).execute()).isFalse();
|
||||
assertThatThrownBy(() -> CoverMigrationApplication.Options.parse(new String[] {"--dry-run", "--execute"}))
|
||||
.isInstanceOf(IllegalArgumentException.class);
|
||||
}
|
||||
|
||||
public static String hex(String value) {
|
||||
return value == null ? null : HexFormat.of().formatHex(value.getBytes(StandardCharsets.UTF_8));
|
||||
}
|
||||
|
||||
@Test
|
||||
void explicitRecoverySourcesDoNotReplaceOriginalValuesUsedForBackupAndCompare() {
|
||||
String alternate = "https://raw.example.com/commit/a.gif";
|
||||
assertThat(CoverMigrationApplication.downloadSources(List.of(FIRST), Map.of(FIRST.projectCode(), alternate)))
|
||||
.containsEntry(FIRST.originalUrl(), alternate);
|
||||
assertThat(FIRST.originalUrl()).isEqualTo("https://raw.example.com/a.gif");
|
||||
assertThat(CoverMigrationApplication.downloadSources(List.of(), Map.of(FIRST.projectCode(), alternate))).isEmpty();
|
||||
CoverMigrator.Candidate second = new CoverMigrator.Candidate("2".repeat(12), FIRST.originalUrl());
|
||||
assertThatThrownBy(() -> CoverMigrationApplication.downloadSources(List.of(FIRST, second),
|
||||
Map.of(FIRST.projectCode(), alternate, second.projectCode(), "https://raw.example.com/other.gif")))
|
||||
.isInstanceOf(IllegalArgumentException.class);
|
||||
}
|
||||
}
|
||||
+61
@@ -0,0 +1,61 @@
|
||||
package com.linkerbot.community.importer.cover;
|
||||
|
||||
import static org.assertj.core.api.Assertions.*;
|
||||
import static org.mockito.Mockito.*;
|
||||
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.net.URI;
|
||||
import java.time.Duration;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
class ExternalCoverDownloaderTest {
|
||||
@Test
|
||||
void detectsActualImageTypesAndRejectsHtmlDisguisedAsImage() {
|
||||
assertThat(ExternalCoverDownloader.image(new byte[] {(byte) 255, (byte) 216, (byte) 255}).contentType())
|
||||
.isEqualTo("image/jpeg");
|
||||
assertThat(ExternalCoverDownloader.image(new byte[] {(byte) 137, 80, 78, 71, 13, 10, 26, 10}).extension())
|
||||
.isEqualTo("png");
|
||||
assertThat(ExternalCoverDownloader.image("GIF89a-data".getBytes()).extension()).isEqualTo("gif");
|
||||
assertThat(ExternalCoverDownloader.image("RIFF1234WEBP-data".getBytes()).extension()).isEqualTo("webp");
|
||||
assertThatThrownBy(() -> ExternalCoverDownloader.image("<html>denied</html>".getBytes()))
|
||||
.isInstanceOfSatisfying(ExternalCoverDownloader.Failure.class,
|
||||
error -> assertThat(error.code()).isEqualTo("COVER_FORMAT_UNSUPPORTED"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void boundedReadRejectsOverflowAndExpiredDownloads() throws Exception {
|
||||
long deadline = System.nanoTime() + Duration.ofSeconds(5).toNanos();
|
||||
assertThat(ExternalCoverDownloader.read(new ByteArrayInputStream(new byte[10]), 10, deadline)).hasSize(10);
|
||||
assertThatThrownBy(() -> ExternalCoverDownloader.read(new ByteArrayInputStream(new byte[11]), 10, deadline))
|
||||
.isInstanceOfSatisfying(ExternalCoverDownloader.Failure.class,
|
||||
error -> assertThat(error.code()).isEqualTo("COVER_TOO_LARGE"));
|
||||
assertThatThrownBy(() -> ExternalCoverDownloader.read(new ByteArrayInputStream(new byte[1]), 10, 0))
|
||||
.isInstanceOf(ExternalCoverDownloader.Failure.class);
|
||||
}
|
||||
|
||||
@Test
|
||||
void rejectsPrivateNetworksCredentialsAndUnsafeRedirectTargets() {
|
||||
for (String value : new String[] {"https://127.0.0.1/a", "https://10.0.0.1/a", "https://169.254.169.254/a",
|
||||
"https://[::1]/a", "https://[fd00::1]/a", "https://100.64.0.1/a", "https://198.18.0.1/a",
|
||||
"http://example.com/a", "https://user:pass@example.com/a", "https://example.com:8443/a"})
|
||||
assertThatThrownBy(() -> ExternalCoverDownloader.validate(URI.create(value)))
|
||||
.isInstanceOf(ExternalCoverDownloader.Failure.class);
|
||||
}
|
||||
|
||||
@Test
|
||||
void retriesTransientHandshakesButKeepsCertificateVerificationFailures() throws Exception {
|
||||
ExternalCoverDownloader downloader = spy(new ExternalCoverDownloader(1024));
|
||||
String url = "https://raw.example.com/a.png";
|
||||
ExternalCoverDownloader.Image image = new ExternalCoverDownloader.Image(new byte[1], "image/png", "png");
|
||||
doThrow(new javax.net.ssl.SSLHandshakeException("transient"))
|
||||
.doReturn(image).when(downloader).fetch(url);
|
||||
assertThat(downloader.download(url)).isSameAs(image);
|
||||
verify(downloader, times(2)).fetch(url);
|
||||
clearInvocations(downloader);
|
||||
javax.net.ssl.SSLHandshakeException invalidCertificate = new javax.net.ssl.SSLHandshakeException("untrusted");
|
||||
invalidCertificate.initCause(new java.security.cert.CertificateException());
|
||||
doThrow(invalidCertificate).when(downloader).fetch(url);
|
||||
assertThatThrownBy(() -> downloader.download(url)).isSameAs(invalidCertificate);
|
||||
verify(downloader).fetch(url);
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -148,7 +148,7 @@ public class AiProjectAssistant {
|
||||
Project source = projects.published(sourceCode);
|
||||
String name = intent.name() == null ? source.getName() : intent.name();
|
||||
// 来源的名称可能被其他调用方更新,执行前仍需按现有输入约束校验。
|
||||
ProjectInputs.validate(new ProjectInput(name, source.getDescription(), source.getProjectType(), source.getCoverUrl()), properties);
|
||||
ProjectInputs.validateMetadata(new ProjectInput(name, source.getDescription(), source.getProjectType(), null), properties);
|
||||
ForkInput input = new ForkInput(source.getPublishedCommitSha(), name);
|
||||
progress.accept(new Progress("forking", "正在 Fork 项目到你的工作区"));
|
||||
checkCancelled();
|
||||
|
||||
+9
@@ -2,6 +2,8 @@ package com.linkerbot.community.service.config;
|
||||
|
||||
import jakarta.validation.constraints.AssertTrue;
|
||||
import java.net.URI;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import lombok.Getter;
|
||||
import lombok.Setter;
|
||||
import org.springframework.boot.context.properties.ConfigurationProperties;
|
||||
@@ -22,6 +24,13 @@ public class OssProperties {
|
||||
private String accessKeySecret = "";
|
||||
private String publicBaseUrl = "";
|
||||
private String rootDirectory = "";
|
||||
private List<String> legacyPublicBaseUrls = new ArrayList<>();
|
||||
|
||||
@AssertTrue(message = "历史 OSS/CDN 基础地址必须是无凭据、查询参数及片段的 HTTPS 地址")
|
||||
public boolean isLegacyPublicBaseUrlsValid() {
|
||||
return legacyPublicBaseUrls != null
|
||||
&& legacyPublicBaseUrls.stream().allMatch(value -> validBaseUrl(value, true));
|
||||
}
|
||||
|
||||
@AssertTrue(message = "开启 OSS 时必须配置合法的地域 ID(如 cn-beijing)、Bucket、凭据及根目录;Endpoint 为可选 HTTPS 地址")
|
||||
public boolean isConfigurationValid() {
|
||||
|
||||
+23
-8
@@ -19,6 +19,7 @@ import com.linkerbot.community.service.fork.ForkExecutor;
|
||||
import com.linkerbot.community.service.fork.GitSnapshotCopier;
|
||||
import com.linkerbot.community.service.mapper.*;
|
||||
import com.linkerbot.community.service.oss.UrdfResources;
|
||||
import com.linkerbot.community.service.oss.OssResourcePaths;
|
||||
import com.linkerbot.community.service.support.ProjectLocks;
|
||||
import com.linkerbot.community.service.support.ProjectInputs;
|
||||
import org.springframework.dao.DuplicateKeyException;
|
||||
@@ -44,7 +45,8 @@ public class ProjectServiceImpl implements ProjectService {
|
||||
private final GiteaClient gitea;
|
||||
private final CommunityProperties config;
|
||||
private final ProjectLocks locks;
|
||||
// 离线导入只加载仓库和数据库组件,没有 URDF 时无需初始化 OSS 模块。
|
||||
private final OssResourcePaths resourcePaths;
|
||||
// 离线导入只需地址转换配置,没有 URDF 时无需初始化 OSS 客户端。
|
||||
private final ObjectProvider<UrdfResources> urdf;
|
||||
private final ObjectProvider<ForkExecutor> forks;
|
||||
private final ObjectProvider<GitSnapshotCopier> snapshots;
|
||||
@@ -61,7 +63,7 @@ public class ProjectServiceImpl implements ProjectService {
|
||||
}
|
||||
|
||||
private String normalizeCoverUrl(String value) {
|
||||
return ProjectInputs.coverUrl(value);
|
||||
return resourcePaths.coverStorage(value);
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -130,13 +132,24 @@ public class ProjectServiceImpl implements ProjectService {
|
||||
.eq(ProjectLike::getProjectId, project.getId())
|
||||
.eq(ProjectLike::getUserId, currentUser))
|
||||
> 0;
|
||||
String coverUrl;
|
||||
String urdfMainUrl;
|
||||
String urdfBaseUrl;
|
||||
try {
|
||||
coverUrl = resourcePaths.coverUrl(project.getCoverUrl());
|
||||
urdfMainUrl = urdfUrl(project.getUrdfMainObjectKey());
|
||||
urdfBaseUrl = urdfUrl(project.getUrdfPrefix());
|
||||
} catch (BusinessException error) {
|
||||
log.warn("项目资源地址生成失败 projectCode={} code={}", project.getProjectCode(), error.code());
|
||||
throw error;
|
||||
}
|
||||
return new ProjectView(
|
||||
project.getProjectCode(),
|
||||
project.getName(),
|
||||
project.getDescription(),
|
||||
project.getCoverUrl(),
|
||||
urdfUrl(project.getUrdfMainObjectKey()),
|
||||
urdfUrl(project.getUrdfPrefix()),
|
||||
coverUrl,
|
||||
urdfMainUrl,
|
||||
urdfBaseUrl,
|
||||
project.getProjectType(),
|
||||
users.publicView(project.getOwnerUserId()),
|
||||
project.getRepoStatus(),
|
||||
@@ -196,13 +209,15 @@ public class ProjectServiceImpl implements ProjectService {
|
||||
}
|
||||
|
||||
private Project initialize(long userId, ProjectInput input, Project source, ForkDeadline deadline) {
|
||||
validateInput(input);
|
||||
if (source == null) validateInput(input);
|
||||
else ProjectInputs.validateMetadata(input, config);
|
||||
Project project = new Project();
|
||||
project.setProjectCode(ProjectCodes.next());
|
||||
project.setOwnerUserId(userId);
|
||||
project.setName(input.name().trim());
|
||||
project.setDescription(input.description() == null ? "" : input.description());
|
||||
project.setCoverUrl(normalizeCoverUrl(input.coverUrl()));
|
||||
project.setCoverUrl(source == null ? normalizeCoverUrl(input.coverUrl())
|
||||
: resourcePaths.coverReference(source.getCoverUrl()));
|
||||
project.setProjectType(input.projectType());
|
||||
project.setGiteaOwner(config.getGitea().getOwner());
|
||||
project.setGiteaRepoName("p_" + project.getProjectCode());
|
||||
@@ -258,7 +273,7 @@ public class ProjectServiceImpl implements ProjectService {
|
||||
ProjectInput data =
|
||||
new ProjectInput(
|
||||
input.name(), source.getDescription(), source.getProjectType(), source.getCoverUrl());
|
||||
validateInput(data);
|
||||
ProjectInputs.validateMetadata(data, config);
|
||||
ForkExecutor executor = forks.getIfAvailable();
|
||||
if (executor == null) throw new BusinessException(503, "FORK_UNAVAILABLE", "当前进程未开启 Fork");
|
||||
return executor.execute(deadline -> {
|
||||
|
||||
+3
-1
@@ -33,6 +33,7 @@ public class OssPostSigner {
|
||||
private final OssProperties properties;
|
||||
private final ObjectMapper json;
|
||||
private final Clock clock;
|
||||
private final OssResourcePaths resourcePaths;
|
||||
|
||||
@Autowired
|
||||
public OssPostSigner(OssProperties properties, ObjectMapper json) {
|
||||
@@ -43,6 +44,7 @@ public class OssPostSigner {
|
||||
this.properties = properties;
|
||||
this.json = json;
|
||||
this.clock = clock;
|
||||
this.resourcePaths = new OssResourcePaths(properties);
|
||||
}
|
||||
|
||||
public CoverUploadSignature sign(String userCode, String contentType, String extension) {
|
||||
@@ -50,7 +52,7 @@ public class OssPostSigner {
|
||||
+ DATE.format(clock.instant()) + "/" + Codes.next() + "." + extension;
|
||||
SignedPost post = post(key, contentType, 1, MAX_FILE_BYTES, VALIDITY, false);
|
||||
return new CoverUploadSignature(properties.uploadUrl(), key,
|
||||
properties.publicBaseUrl() + "/" + key, post.expiresAt(), post.formFields());
|
||||
resourcePaths.publicUrl(key), post.expiresAt(), post.formFields());
|
||||
}
|
||||
|
||||
public record SignedPost(LocalDateTime expiresAt, Map<String, String> formFields) {}
|
||||
|
||||
+121
@@ -0,0 +1,121 @@
|
||||
package com.linkerbot.community.service.oss;
|
||||
|
||||
import com.linkerbot.community.common.BusinessException;
|
||||
import com.linkerbot.community.common.FilePaths;
|
||||
import com.linkerbot.community.service.config.OssProperties;
|
||||
import com.linkerbot.community.service.support.ProjectInputs;
|
||||
import java.net.URI;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import lombok.RequiredArgsConstructor;
|
||||
import org.springframework.stereotype.Component;
|
||||
import org.springframework.web.util.UriUtils;
|
||||
|
||||
/** 仅转换资源地址,不访问 OSS,也不依赖上传凭据。存储路径为未编码的环境相对路径。 */
|
||||
@Component
|
||||
@RequiredArgsConstructor
|
||||
public class OssResourcePaths {
|
||||
private final OssProperties oss;
|
||||
|
||||
/** 外部请求仍只接受 HTTPS URL;仅转换明确属于当前环境的托管封面。 */
|
||||
public String coverStorage(String value) {
|
||||
String url = ProjectInputs.coverUrl(value);
|
||||
if (url == null) return null;
|
||||
URI candidate = URI.create(url);
|
||||
if (candidate.getRawQuery() != null || candidate.getRawFragment() != null) return url;
|
||||
for (String baseValue : managedBases()) {
|
||||
URI base = URI.create(baseValue);
|
||||
if (!sameOrigin(candidate, base)) continue;
|
||||
String prefix = base.getRawPath().replaceAll("/+$", "")
|
||||
+ "/" + oss.getRootDirectory() + "/project-covers/";
|
||||
if (!candidate.getRawPath().startsWith(prefix)) continue;
|
||||
try {
|
||||
String[] segments = candidate.getRawPath().substring(prefix.length()).split("/", -1);
|
||||
for (int index = 0; index < segments.length; index++) {
|
||||
segments[index] = UriUtils.decode(segments[index], StandardCharsets.UTF_8);
|
||||
// 保留含歧义分隔符或 UTF-8 替换字符的原地址,避免引用到不同对象。
|
||||
if (segments[index].contains("/") || segments[index].indexOf('\uFFFD') >= 0) return url;
|
||||
}
|
||||
return storagePath("/project-covers/" + String.join("/", segments));
|
||||
} catch (IllegalArgumentException | BusinessException error) {
|
||||
// 非规范地址保留原语义,不进行路径折叠或二次解码。
|
||||
return url;
|
||||
}
|
||||
}
|
||||
return url;
|
||||
}
|
||||
|
||||
/** 内部 Fork 可以复用已持久化的路径,不走客户端 URL 校验。 */
|
||||
public String coverReference(String value) {
|
||||
if (value != null && value.startsWith("/project-covers/")) return storagePath(value);
|
||||
return coverStorage(value);
|
||||
}
|
||||
|
||||
public String coverUrl(String value) {
|
||||
String reference = coverReference(value);
|
||||
return reference != null && reference.startsWith("/project-covers/")
|
||||
? publicUrl(reference) : reference;
|
||||
}
|
||||
|
||||
/** 兼容当前环境的旧 key;绝不把其他环境的 key 重定位至当前环境。 */
|
||||
public String storagePath(String value) {
|
||||
if (value == null || value.isEmpty()) throw invalidPath();
|
||||
String path = value;
|
||||
String root = oss.getRootDirectory();
|
||||
if (root != null && !root.isBlank() && path.startsWith(root + "/"))
|
||||
path = path.substring(root.length());
|
||||
if (!path.startsWith("/project-covers/") && !path.startsWith("/urdf/")) throw invalidPath();
|
||||
String relative = path.substring(1);
|
||||
if (relative.endsWith("/")) relative = relative.substring(0, relative.length() - 1);
|
||||
try {
|
||||
FilePaths.file(relative);
|
||||
} catch (BusinessException error) {
|
||||
throw invalidPath();
|
||||
}
|
||||
return path;
|
||||
}
|
||||
|
||||
public String objectKey(String value) {
|
||||
String path = storagePath(value);
|
||||
String root = oss.getRootDirectory();
|
||||
if (root == null || !root.matches("[A-Za-z0-9_-]+"))
|
||||
throw new BusinessException(503, "OSS_UNAVAILABLE", "OSS 资源根目录未正确配置");
|
||||
return root + path;
|
||||
}
|
||||
|
||||
public String publicUrl(String value) {
|
||||
if (value == null || value.isEmpty()) return null;
|
||||
String[] segments = objectKey(value).split("/", -1);
|
||||
for (int index = 0; index < segments.length; index++)
|
||||
segments[index] = UriUtils.encodePathSegment(segments[index], StandardCharsets.UTF_8);
|
||||
return oss.publicBaseUrl() + "/" + String.join("/", segments);
|
||||
}
|
||||
|
||||
private List<String> managedBases() {
|
||||
List<String> bases = new ArrayList<>(oss.getLegacyPublicBaseUrls());
|
||||
if (oss.getPublicBaseUrl() != null && !oss.getPublicBaseUrl().isBlank())
|
||||
bases.add(oss.getPublicBaseUrl());
|
||||
if (oss.getBucket() != null && !oss.getBucket().isBlank()
|
||||
&& oss.getRegion() != null && !oss.getRegion().isBlank()) {
|
||||
bases.add("https://" + oss.getBucket() + ".oss-" + oss.getRegion() + ".aliyuncs.com");
|
||||
bases.add(oss.uploadUrl());
|
||||
}
|
||||
return bases;
|
||||
}
|
||||
|
||||
private boolean sameOrigin(URI first, URI second) {
|
||||
return second.getHost() != null
|
||||
&& first.getScheme().equalsIgnoreCase(second.getScheme())
|
||||
&& first.getHost().equalsIgnoreCase(second.getHost())
|
||||
&& port(first) == port(second);
|
||||
}
|
||||
|
||||
private int port(URI uri) {
|
||||
return uri.getPort() == -1 ? 443 : uri.getPort();
|
||||
}
|
||||
|
||||
private BusinessException invalidPath() {
|
||||
return new BusinessException(500, "RESOURCE_PATH_INVALID", "资源存储路径异常");
|
||||
}
|
||||
}
|
||||
+24
-17
@@ -25,7 +25,6 @@ import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.http.MediaTypeFactory;
|
||||
import org.springframework.stereotype.Component;
|
||||
import org.springframework.web.util.UriUtils;
|
||||
|
||||
@Component
|
||||
@RequiredArgsConstructor
|
||||
@@ -37,6 +36,7 @@ public class UrdfResources {
|
||||
private final OssPostSigner signer;
|
||||
private final OssObjectStore store;
|
||||
private final ObjectMapper json;
|
||||
private final OssResourcePaths paths;
|
||||
|
||||
public void requireEnabled() {
|
||||
if (!oss.isEnabled())
|
||||
@@ -47,7 +47,7 @@ public class UrdfResources {
|
||||
public String validate(String prefix, List<UrdfFile> files) {
|
||||
if (files == null) throw BusinessException.badRequest("必须提供 URDF 文件清单");
|
||||
if (files.size() > limits.getMaxFiles()) throw tooLarge();
|
||||
Set<String> paths = new HashSet<>();
|
||||
Set<String> seenPaths = new HashSet<>();
|
||||
long total = 0;
|
||||
String main = null;
|
||||
int mainCount = 0;
|
||||
@@ -55,8 +55,8 @@ public class UrdfResources {
|
||||
if (file == null || file.sizeBytes() == null || file.sizeBytes() < 0)
|
||||
throw BusinessException.badRequest("文件大小不合法");
|
||||
String path = FilePaths.file(file.relativePath());
|
||||
if (!paths.add(path)) throw BusinessException.badRequest("上传路径不能重复");
|
||||
String key = prefix + path;
|
||||
if (!seenPaths.add(path)) throw BusinessException.badRequest("上传路径不能重复");
|
||||
String key = paths.objectKey(prefix + path);
|
||||
if (key.length() > 512 || key.getBytes(StandardCharsets.UTF_8).length > 1023)
|
||||
throw BusinessException.badRequest("URDF 文件对象键过长");
|
||||
if (file.sizeBytes() > limits.getMaxFileBytes()
|
||||
@@ -83,7 +83,7 @@ public class UrdfResources {
|
||||
if (projectCode == null || !projectCode.matches("[a-z0-9]{12}|[a-z0-9]{32}")
|
||||
|| batchCode == null || !batchCode.matches("[a-f0-9]{32}"))
|
||||
throw BusinessException.badRequest("URDF 项目或批次编码不合法");
|
||||
return oss.getRootDirectory() + "/urdf/" + projectCode + "/" + batchCode + "/";
|
||||
return "/urdf/" + projectCode + "/" + batchCode + "/";
|
||||
}
|
||||
|
||||
public UrdfUploadSignature sign(String projectCode, List<UrdfFile> files) {
|
||||
@@ -97,7 +97,7 @@ public class UrdfResources {
|
||||
String type = file.relativePath().equals(main) ? "application/xml"
|
||||
: MediaTypeFactory.getMediaType(file.relativePath().toLowerCase(Locale.ROOT))
|
||||
.map(Object::toString).orElse("application/octet-stream");
|
||||
String key = prefix + file.relativePath();
|
||||
String key = paths.objectKey(prefix + file.relativePath());
|
||||
OssPostSigner.SignedPost post = signer.signObject(key, type, file.sizeBytes());
|
||||
if (expiration == null || post.expiresAt().isBefore(expiration)) expiration = post.expiresAt();
|
||||
signatures.add(new UrdfFileSignature(file.relativePath(), key, post.formFields()));
|
||||
@@ -116,10 +116,11 @@ public class UrdfResources {
|
||||
|
||||
private OssObjectStore.Metadata checkedMetadata(String prefix, UrdfFile file) {
|
||||
ForkDeadline.checkpoint();
|
||||
OssObjectStore.Metadata metadata = store.head(prefix + file.relativePath());
|
||||
String key = paths.objectKey(prefix + file.relativePath());
|
||||
OssObjectStore.Metadata metadata = store.head(key);
|
||||
if (metadata.sizeBytes() != file.sizeBytes()) {
|
||||
log.warn("URDF 文件大小不符 objectKey={} expectedBytes={} actualBytes={}",
|
||||
prefix + file.relativePath(), file.sizeBytes(), metadata.sizeBytes());
|
||||
key, file.sizeBytes(), metadata.sizeBytes());
|
||||
throw new BusinessException(400, "URDF_FILE_SIZE_MISMATCH", "URDF 文件大小不符,请重新上传");
|
||||
}
|
||||
return metadata;
|
||||
@@ -144,13 +145,22 @@ public class UrdfResources {
|
||||
source.getProjectCode(), error.getClass().getSimpleName());
|
||||
throw new BusinessException(500, "URDF_MANIFEST_INVALID", "URDF 文件清单异常");
|
||||
}
|
||||
String sourcePrefix = source.getUrdfPrefix();
|
||||
String root = oss.getRootDirectory() + "/urdf/" + source.getProjectCode() + "/";
|
||||
String sourcePrefix;
|
||||
String sourceMain;
|
||||
try {
|
||||
sourcePrefix = paths.storagePath(source.getUrdfPrefix());
|
||||
sourceMain = paths.storagePath(source.getUrdfMainObjectKey());
|
||||
} catch (BusinessException error) {
|
||||
log.warn("Fork URDF 路径异常 sourceProjectCode={} projectCode={} code={}",
|
||||
source.getProjectCode(), target.getProjectCode(), error.code());
|
||||
throw new BusinessException(500, "URDF_MANIFEST_INVALID", "URDF 资源路径异常");
|
||||
}
|
||||
String root = "/urdf/" + source.getProjectCode() + "/";
|
||||
if (sourcePrefix == null || !sourcePrefix.startsWith(root)
|
||||
|| !sourcePrefix.substring(root.length()).matches("[a-f0-9]{32}/"))
|
||||
throw new BusinessException(500, "URDF_MANIFEST_INVALID", "URDF 资源目录异常");
|
||||
String main = validate(sourcePrefix, files);
|
||||
if (!source.getUrdfMainObjectKey().equals(sourcePrefix + main))
|
||||
if (!sourceMain.equals(sourcePrefix + main))
|
||||
throw new BusinessException(500, "URDF_MANIFEST_INVALID", "URDF 主文件记录异常");
|
||||
String targetPrefix = prefix(target.getProjectCode(), Codes.next());
|
||||
validate(targetPrefix, files);
|
||||
@@ -159,7 +169,8 @@ public class UrdfResources {
|
||||
for (UrdfFile file : files) {
|
||||
OssObjectStore.Metadata metadata = checkedMetadata(sourcePrefix, file);
|
||||
ForkDeadline.checkpoint();
|
||||
store.copy(sourcePrefix + file.relativePath(), targetPrefix + file.relativePath(), metadata.etag());
|
||||
store.copy(paths.objectKey(sourcePrefix + file.relativePath()),
|
||||
paths.objectKey(targetPrefix + file.relativePath()), metadata.etag());
|
||||
}
|
||||
ForkDeadline.checkpoint();
|
||||
target.setUrdfPrefix(targetPrefix);
|
||||
@@ -170,11 +181,7 @@ public class UrdfResources {
|
||||
}
|
||||
|
||||
public String url(String objectKey) {
|
||||
if (objectKey == null || objectKey.isEmpty()) return null;
|
||||
String[] segments = objectKey.split("/", -1);
|
||||
for (int index = 0; index < segments.length; index++)
|
||||
segments[index] = UriUtils.encodePathSegment(segments[index], StandardCharsets.UTF_8);
|
||||
return oss.publicBaseUrl() + "/" + String.join("/", segments);
|
||||
return paths.publicUrl(objectKey);
|
||||
}
|
||||
|
||||
private long totalBytes(List<UrdfFile> files) {
|
||||
|
||||
+5
-1
@@ -10,13 +10,17 @@ public final class ProjectInputs {
|
||||
private ProjectInputs() {}
|
||||
|
||||
public static void validate(ProjectInput input, CommunityProperties config) {
|
||||
validateMetadata(input, config);
|
||||
coverUrl(input.coverUrl());
|
||||
}
|
||||
|
||||
public static void validateMetadata(ProjectInput input, CommunityProperties config) {
|
||||
if (input.name() == null || input.name().isBlank() || input.name().length() > 100)
|
||||
throw BusinessException.badRequest("项目名称必须为 1~100 字符");
|
||||
if (input.description() != null && input.description().length() > 2000)
|
||||
throw BusinessException.badRequest("项目描述不能超过 2000 字符");
|
||||
if (!config.getTypes().containsKey(input.projectType()))
|
||||
throw BusinessException.badRequest("项目类型不在可选列表中");
|
||||
coverUrl(input.coverUrl());
|
||||
}
|
||||
|
||||
public static String coverUrl(String value) {
|
||||
|
||||
+3
@@ -16,6 +16,8 @@ import com.linkerbot.community.service.fork.GitSnapshotCopier;
|
||||
import com.linkerbot.community.service.impl.ProjectServiceImpl;
|
||||
import com.linkerbot.community.service.mapper.*;
|
||||
import com.linkerbot.community.service.oss.UrdfResources;
|
||||
import com.linkerbot.community.service.oss.OssResourcePaths;
|
||||
import com.linkerbot.community.service.config.OssProperties;
|
||||
import com.linkerbot.community.service.support.ProjectLocks;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
@@ -89,6 +91,7 @@ class ProjectCodeAllocationTest {
|
||||
gitea,
|
||||
new CommunityProperties(),
|
||||
new ProjectLocks(),
|
||||
new OssResourcePaths(new OssProperties()),
|
||||
new StaticListableBeanFactory().getBeanProvider(UrdfResources.class),
|
||||
new StaticListableBeanFactory().getBeanProvider(ForkExecutor.class),
|
||||
new StaticListableBeanFactory().getBeanProvider(GitSnapshotCopier.class));
|
||||
|
||||
+3
-1
@@ -15,6 +15,8 @@ import com.linkerbot.community.service.gitea.GiteaClient;
|
||||
import com.linkerbot.community.service.impl.ProjectServiceImpl;
|
||||
import com.linkerbot.community.service.mapper.*;
|
||||
import com.linkerbot.community.service.oss.UrdfResources;
|
||||
import com.linkerbot.community.service.oss.OssResourcePaths;
|
||||
import com.linkerbot.community.service.config.OssProperties;
|
||||
import com.linkerbot.community.service.support.ProjectLocks;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.atomic.AtomicReference;
|
||||
@@ -73,7 +75,7 @@ class ProjectForkStateTest {
|
||||
factory.addBean("snapshots", copier);
|
||||
ProjectLocks locks = new ProjectLocks();
|
||||
ProjectServiceImpl service = new ProjectServiceImpl(projects, mock(LikeMapper.class), mock(CommentMapper.class), users,
|
||||
gitea, new CommunityProperties(), locks, factory.getBeanProvider(UrdfResources.class),
|
||||
gitea, new CommunityProperties(), locks, new OssResourcePaths(new OssProperties()), factory.getBeanProvider(UrdfResources.class),
|
||||
factory.getBeanProvider(ForkExecutor.class), factory.getBeanProvider(GitSnapshotCopier.class));
|
||||
try {
|
||||
assertThatThrownBy(() -> service.fork(1, source.getProjectCode(), new ForkInput(source.getPublishedCommitSha(), "副本")))
|
||||
|
||||
+17
@@ -74,4 +74,21 @@ class OssPropertiesTest {
|
||||
enabled().withPropertyValues("community.oss." + property)
|
||||
.run(context -> assertThat(context).hasFailed());
|
||||
}
|
||||
|
||||
@Test
|
||||
void bindsLegacyBasesAndValidatesThemEvenWhenUploadsAreDisabled() {
|
||||
runner.withPropertyValues("community.oss.legacy-public-base-urls=")
|
||||
.run(context -> {
|
||||
assertThat(context).hasNotFailed();
|
||||
assertThat(context.getBean(OssProperties.class).getLegacyPublicBaseUrls()).isEmpty();
|
||||
});
|
||||
runner.withPropertyValues("community.oss.legacy-public-base-urls=https://old.example.com/assets,https://old-bucket.example.com")
|
||||
.run(context -> {
|
||||
assertThat(context).hasNotFailed();
|
||||
assertThat(context.getBean(OssProperties.class).getLegacyPublicBaseUrls()).hasSize(2);
|
||||
});
|
||||
for (String invalid : new String[] {"http://old.example.com", "https://old.example.com?token=example", "https://user:pass@old.example.com"})
|
||||
runner.withPropertyValues("community.oss.legacy-public-base-urls=" + invalid)
|
||||
.run(context -> assertThat(context).hasFailed());
|
||||
}
|
||||
}
|
||||
|
||||
+89
@@ -0,0 +1,89 @@
|
||||
package com.linkerbot.community.service.oss;
|
||||
|
||||
import static org.assertj.core.api.Assertions.*;
|
||||
|
||||
import com.linkerbot.community.common.BusinessException;
|
||||
import com.linkerbot.community.service.config.OssProperties;
|
||||
import java.util.List;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.ValueSource;
|
||||
|
||||
class OssResourcePathsTest {
|
||||
private final OssProperties oss = new OssProperties();
|
||||
private final OssResourcePaths paths = new OssResourcePaths(oss);
|
||||
private static final String COVER = "/project-covers/111111111111/20261019/a.png";
|
||||
|
||||
@BeforeEach
|
||||
void configure() {
|
||||
oss.setRootDirectory("vibemake_dev");
|
||||
oss.setRegion("cn-beijing");
|
||||
oss.setBucket("examplebucket");
|
||||
oss.setPublicBaseUrl("https://cdn.example.com/assets/");
|
||||
oss.setLegacyPublicBaseUrls(List.of("https://old.example.com/previous/"));
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@ValueSource(strings = {"https://cdn.example.com/assets", "https://CDN.EXAMPLE.COM:443/assets",
|
||||
"https://examplebucket.oss-cn-beijing.aliyuncs.com", "https://old.example.com/previous"})
|
||||
void normalizesOnlyManagedCoversAndRendersWithCurrentCdn(String base) {
|
||||
String original = base + "/vibemake_dev" + COVER;
|
||||
assertThat(paths.coverStorage(" " + original + " ")).isEqualTo(COVER);
|
||||
assertThat(paths.coverUrl(original)).isEqualTo("https://cdn.example.com/assets/vibemake_dev" + COVER);
|
||||
assertThat(paths.coverReference(COVER)).isEqualTo(COVER);
|
||||
assertThat(paths.coverUrl(COVER)).isEqualTo(paths.coverUrl(original));
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@ValueSource(strings = {"https://external.example.com/vibemake_dev", "https://cdn.example.com/assets/vibemake",
|
||||
"https://cdn.example.com.evil/assets/vibemake_dev", "https://cdn.example.com:444/assets/vibemake_dev",
|
||||
"https://cdn.example.com/assets-other/vibemake_dev", "https://cdn.example.com/assets/vibemake_dev-extra"})
|
||||
void keepsExternalOriginsOtherEnvironmentsAndWrongPathBoundaries(String base) {
|
||||
String url = base + COVER;
|
||||
assertThat(paths.coverStorage(url)).isEqualTo(url);
|
||||
assertThat(paths.coverUrl(url)).isEqualTo(url);
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@ValueSource(strings = {"?version=2", "#preview", "?", "#"})
|
||||
void preservesQueryAndFragmentIncludingEmptyValues(String suffix) {
|
||||
String url = "https://cdn.example.com/assets/vibemake_dev" + COVER + suffix;
|
||||
assertThat(paths.coverStorage(url)).isEqualTo(url);
|
||||
assertThat(paths.coverUrl(url)).isEqualTo(url);
|
||||
}
|
||||
|
||||
@Test
|
||||
void encodesOnlyOncePreservesPlusAndNeverDropsCdnBasePath() {
|
||||
String stored = "/project-covers/user/日期/a #?%+.png";
|
||||
String url = "https://cdn.example.com/assets/vibemake_dev/project-covers/user/"
|
||||
+ "%E6%97%A5%E6%9C%9F/a%20%23%3F%25+.png";
|
||||
assertThat(paths.coverUrl(stored)).isEqualTo(url);
|
||||
assertThat(paths.coverStorage(url)).isEqualTo(stored);
|
||||
assertThat(paths.coverStorage("https://cdn.example.com/assets/vibemake_dev/project-covers/a%252Fb.png"))
|
||||
.isEqualTo("/project-covers/a%2Fb.png");
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@ValueSource(strings = {"../a.png", "%2e%2e/a.png", "user%2Fa.png", "user//a.png", "user/%5Ca.png", "user/%FF.png"})
|
||||
void doesNotRewriteAmbiguousOrUnsafeUrlPaths(String suffix) {
|
||||
String url = "https://cdn.example.com/assets/vibemake_dev/project-covers/" + suffix;
|
||||
assertThat(paths.coverStorage(url)).isEqualTo(url);
|
||||
}
|
||||
|
||||
@Test
|
||||
void supportsLegacyUrdfKeysAndDirectoryUrlsWithoutReinterpretingOtherEnvironments() {
|
||||
String relative = "/urdf/7m4k9x2p8q6r/" + "a".repeat(32) + "/";
|
||||
assertThat(paths.storagePath("vibemake_dev" + relative)).isEqualTo(relative);
|
||||
assertThat(paths.objectKey(relative)).isEqualTo("vibemake_dev" + relative);
|
||||
assertThat(paths.publicUrl(relative)).isEqualTo(paths.publicUrl("vibemake_dev" + relative)).endsWith("/");
|
||||
assertThatThrownBy(() -> paths.publicUrl("vibemake" + relative)).isInstanceOf(BusinessException.class);
|
||||
oss.setRootDirectory("vibemake");
|
||||
assertThat(paths.publicUrl(relative)).isEqualTo("https://cdn.example.com/assets/vibemake" + relative);
|
||||
assertThatThrownBy(() -> paths.publicUrl("vibemake_dev" + relative)).isInstanceOf(BusinessException.class);
|
||||
assertThat(paths.publicUrl("")).isNull();
|
||||
assertThat(paths.coverStorage(null)).isNull();
|
||||
assertThatThrownBy(() -> paths.coverStorage(COVER)).isInstanceOf(BusinessException.class);
|
||||
assertThatThrownBy(() -> paths.objectKey("/urdf/../other/a.urdf")).isInstanceOf(BusinessException.class);
|
||||
}
|
||||
}
|
||||
+46
-8
@@ -11,13 +11,15 @@ import com.linkerbot.community.service.config.OssProperties;
|
||||
import com.linkerbot.community.service.config.UrdfProperties;
|
||||
import java.util.List;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.ValueSource;
|
||||
|
||||
class UrdfResourcesTest {
|
||||
private final OssProperties oss = new OssProperties();
|
||||
private final UrdfProperties limits = new UrdfProperties();
|
||||
private final OssObjectStore store = mock(OssObjectStore.class);
|
||||
private final UrdfResources resources = new UrdfResources(
|
||||
oss, limits, mock(OssPostSigner.class), store, new ObjectMapper());
|
||||
oss, limits, mock(OssPostSigner.class), store, new ObjectMapper(), new OssResourcePaths(oss));
|
||||
|
||||
@Test
|
||||
void acceptsExactLimitsAndRejectsOverflowCountsAndCrossProjectBatches() {
|
||||
@@ -39,6 +41,7 @@ class UrdfResourcesTest {
|
||||
@Test
|
||||
void encodesUrlSegmentsWithoutEncodingSlashesAndKeepsEmptyUrlsNull() {
|
||||
oss.setPublicBaseUrl("https://cdn.example.com/assets/");
|
||||
oss.setRootDirectory("vibemake");
|
||||
assertThat(resources.url("vibemake/urdf/目录/a #?%+.urdf"))
|
||||
.isEqualTo("https://cdn.example.com/assets/vibemake/urdf/%E7%9B%AE%E5%BD%95/a%20%23%3F%25+.urdf");
|
||||
assertThat(resources.url("vibemake/urdf/a/")).endsWith("/a/");
|
||||
@@ -46,28 +49,63 @@ class UrdfResourcesTest {
|
||||
assertThat(resources.url(null)).isNull();
|
||||
}
|
||||
|
||||
@Test
|
||||
void failedCopyLeavesTargetEmptyAndSuccessfulCopyUsesSourceEtag() {
|
||||
@ParameterizedTest
|
||||
@ValueSource(booleans = {false, true})
|
||||
void failedCopyLeavesTargetEmptyAndSuccessfulCopyUsesSourceEtag(boolean legacy) {
|
||||
oss.setEnabled(true);
|
||||
oss.setRootDirectory("vibemake_dev");
|
||||
Project source = new Project();
|
||||
source.setProjectCode("7m4k9x2p8q6r");
|
||||
String sourcePrefix = resources.prefix(source.getProjectCode(), "a".repeat(32));
|
||||
String relativePrefix = resources.prefix(source.getProjectCode(), "a".repeat(32));
|
||||
String sourcePrefix = legacy ? "vibemake_dev" + relativePrefix : relativePrefix;
|
||||
String sourceKeyPrefix = "vibemake_dev" + relativePrefix;
|
||||
source.setUrdfPrefix(sourcePrefix);
|
||||
source.setUrdfMainObjectKey(sourcePrefix + "a.urdf");
|
||||
source.setUrdfFiles(resources.manifest(List.of(new UrdfFile("a.urdf", 12L))));
|
||||
Project target = new Project();
|
||||
target.setProjectCode("222222222222");
|
||||
when(store.head(sourcePrefix + "a.urdf")).thenReturn(new OssObjectStore.Metadata(12, "source-etag"));
|
||||
when(store.head(sourceKeyPrefix + "a.urdf")).thenReturn(new OssObjectStore.Metadata(12, "source-etag"));
|
||||
doThrow(new BusinessException(502, "OSS_ERROR", "copy failed")).when(store).copy(anyString(), anyString(), anyString());
|
||||
assertThatThrownBy(() -> resources.copy(source, target)).isInstanceOf(BusinessException.class);
|
||||
assertThat(target.getUrdfPrefix()).isEmpty();
|
||||
assertThat(target.getUrdfMainObjectKey()).isEmpty();
|
||||
reset(store);
|
||||
when(store.head(sourcePrefix + "a.urdf")).thenReturn(new OssObjectStore.Metadata(12, "source-etag"));
|
||||
when(store.head(sourceKeyPrefix + "a.urdf")).thenReturn(new OssObjectStore.Metadata(12, "source-etag"));
|
||||
resources.copy(source, target);
|
||||
verify(store).copy(sourcePrefix + "a.urdf", target.getUrdfPrefix() + "a.urdf", "source-etag");
|
||||
assertThat(target.getUrdfPrefix()).startsWith("vibemake_dev/urdf/222222222222/");
|
||||
verify(store).copy(sourceKeyPrefix + "a.urdf", "vibemake_dev" + target.getUrdfPrefix() + "a.urdf", "source-etag");
|
||||
assertThat(target.getUrdfPrefix()).startsWith("/urdf/222222222222/");
|
||||
assertThat(target.getUrdfMainObjectKey()).isEqualTo(target.getUrdfPrefix() + "a.urdf");
|
||||
}
|
||||
|
||||
@Test
|
||||
void rejectsForeignEnvironmentBeforeAnyOssRequest() {
|
||||
oss.setEnabled(true);
|
||||
oss.setRootDirectory("vibemake_dev");
|
||||
Project source = new Project();
|
||||
source.setProjectCode("7m4k9x2p8q6r");
|
||||
source.setUrdfPrefix("vibemake/urdf/7m4k9x2p8q6r/" + "a".repeat(32) + "/");
|
||||
source.setUrdfMainObjectKey(source.getUrdfPrefix() + "a.urdf");
|
||||
source.setUrdfFiles(resources.manifest(List.of(new UrdfFile("a.urdf", 12L))));
|
||||
Project target = new Project();
|
||||
target.setProjectCode("222222222222");
|
||||
assertThatThrownBy(() -> resources.copy(source, target))
|
||||
.isInstanceOfSatisfying(BusinessException.class,
|
||||
error -> assertThat(error.code()).isEqualTo("URDF_MANIFEST_INVALID"));
|
||||
verifyNoInteractions(store);
|
||||
}
|
||||
|
||||
@Test
|
||||
void lengthLimitIncludesEnvironmentRootAndUsesUtf8Bytes() {
|
||||
oss.setRootDirectory("vibemake_dev");
|
||||
String prefix = resources.prefix("7m4k9x2p8q6r", "a".repeat(32));
|
||||
int remaining = 512 - ("vibemake_dev" + prefix).length() - ".urdf".length();
|
||||
assertThat(resources.validate(prefix, List.of(new UrdfFile("a".repeat(remaining) + ".urdf", 1L))))
|
||||
.endsWith(".urdf");
|
||||
assertThatThrownBy(() -> resources.validate(prefix,
|
||||
List.of(new UrdfFile("a".repeat(remaining + 1) + ".urdf", 1L))))
|
||||
.isInstanceOf(BusinessException.class);
|
||||
assertThatThrownBy(() -> resources.validate(prefix,
|
||||
List.of(new UrdfFile("中".repeat(340) + ".urdf", 1L))))
|
||||
.isInstanceOf(BusinessException.class);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -568,7 +568,9 @@ async function replaceProjectCover(project: Project, file: File, token: string)
|
||||
|
||||
服务端启用 `OSS_ENABLED=true`,配置 `OSS_REGION`、`OSS_BUCKET`、`OSS_ACCESS_KEY_ID`、`OSS_ACCESS_KEY_SECRET`;地域 ID 使用 `cn-beijing` 等编码,不带 `oss-` 前缀。Endpoint 自动生成为 `https://oss-{region}.aliyuncs.com`,例如北京为 `https://oss-cn-beijing.aliyuncs.com`,无需配置 Endpoint 环境变量。特殊域名场景可显式设置 `community.oss.endpoint`,优先使用配置值,要求为公网 HTTPS 地址,不带 Bucket、路径、凭据或查询参数。`OSS_PUBLIC_BASE_URL` 可选。启用时配置缺失或不合法会停止启动。密钥不提交仓库、不返回浏览器;浏览器只收到 AccessKeyId 派生的 credential、Policy 和签名。
|
||||
|
||||
公开基础地址可以是 OSS 或 CDN 域名,可带基础路径;最终地址为 `{公开基础地址}/{objectKey}`,CDN 路径需正确映射到同一对象。留空时为 `https://{Bucket}.{Endpoint 主机名}/{objectKey}`。数据库保存固定地址,不保存临时签名下载链接。
|
||||
公开基础地址可以是 OSS 或 CDN 域名,可带基础路径;最终地址为 `{公开基础地址}/{objectKey}`,CDN 路径需正确映射到同一对象。留空时为 `https://{Bucket}.{Endpoint 主机名}/{objectKey}`。API 的 `coverUrl` 继续接收和返回完整 HTTPS URL;数据库仅对本平台当前环境资源保存 `/project-covers/...` 未编码路径,域名和环境根目录由后端配置补齐,外链原样保存。创建、PATCH、清除封面及 Fork 的调用方式不变,不允许客户端改为提交相对路径。
|
||||
|
||||
历史完整平台 URL 及当前环境的 URDF key 均兼容读取,读取不自动迁移。更换域名时,在 `community.oss.legacy-public-base-urls` 配置历史公开基础地址(可带 CDN 基础路径,不含环境根目录、凭据、查询参数及片段),或通过 `OSS_LEGACY_PUBLIC_BASE_URLS` 使用英文逗号分隔。后端按域名、端口和完整路径边界匹配当前环境;带查询参数或片段的封面继续按外链保留。迁移 SQL 与回退步骤见 [资源路径迁移](database.md#资源路径历史数据迁移)。
|
||||
|
||||
OSS 控制台配置步骤:
|
||||
|
||||
@@ -619,7 +621,7 @@ URDF 为可选项目资源,不进入 Gitea,也不生成 Commit。先创建
|
||||
}
|
||||
```
|
||||
|
||||
提交签名响应的真实 `batchCode` 和最初的完整清单,不提交 URL、目录前缀或主文件对象键。后端重新识别唯一主文件,生成当前环境、本项目的路径,通过 OSS 核验所有文件存在且实际大小一致后,一次性保存资源记录。成功 data 为更新后的 Project,包含 `urdfMainUrl`、`urdfBaseUrl`;URL 已编码,前端不要再次整体编码。未确认、上传失败、取消或核验失败均不会替换原绑定。
|
||||
提交签名响应的真实 `batchCode` 和最初的完整清单,不提交 URL、目录前缀或主文件对象键。后端重新识别唯一主文件,生成当前环境、本项目的路径,通过 OSS 核验所有文件存在且实际大小一致后,一次性保存资源记录。数据库的 `urdf_prefix` 保存 `/urdf/{projectCode}/{batchCode}/`,`urdf_main_object_key` 为此前缀加主文件相对路径,均不含环境根目录和域名;真实 OSS key 仍包含环境根目录。成功 data 为更新后的 Project,包含 `urdfMainUrl`、`urdfBaseUrl` 完整地址;URL 已编码,前端不要再次整体编码。未确认、上传失败、取消或核验失败均不会替换原绑定。
|
||||
|
||||
替换重复执行“签名 → 直传 → 确认”,并发以最后一次成功确认生效。签名过期时重新申请新批次并重新上传,不修改旧签名。没有上传会话或后台任务查询接口。
|
||||
|
||||
@@ -1257,6 +1259,7 @@ if (result.status !== 'CHAT') {
|
||||
| 500 | CAPTCHA_ERROR | 验证码生成失败,可重新获取并记录 requestId |
|
||||
| 500 | PROJECT_INITIALIZATION_ERROR | 项目初始化或回写未完成,查看 data.projectCode 和工作区,请后端核对 |
|
||||
| 500 | INTERNAL_ERROR | 展示服务异常并保留 requestId |
|
||||
| 500 | RESOURCE_PATH_INVALID | 历史资源路径格式或所属环境异常,保留 requestId 请后端核对 |
|
||||
| 502 | GITEA_ERROR | 仓库服务异常;写请求先核对结果,避免自动重试 |
|
||||
| 502 | GITEA_INVALID_RESPONSE | 仓库响应无法确认;写请求先核对结果,避免自动重试 |
|
||||
| 503 | GITEA_NOT_CONFIGURED | 后端缺少仓库配置,请后端处理 |
|
||||
|
||||
+187
-8
@@ -25,10 +25,10 @@ CREATE TABLE project (
|
||||
owner_user_id BIGINT NOT NULL COMMENT '所属平台用户',
|
||||
name VARCHAR(100) NOT NULL COMMENT '项目名称',
|
||||
description TEXT NOT NULL COMMENT '项目描述',
|
||||
cover_url VARCHAR(2048) NULL COMMENT '封面图片 HTTPS 地址',
|
||||
urdf_main_object_key VARCHAR(512) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL DEFAULT '' COMMENT 'URDF主文件OSS对象键,不保存完整URL',
|
||||
cover_url VARCHAR(2048) NULL COMMENT '项目封面,平台资源存环境相对路径,外部图片存完整HTTPS地址',
|
||||
urdf_main_object_key VARCHAR(512) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL DEFAULT '' COMMENT 'URDF主文件环境相对路径,以斜杠开头,不含域名和环境根目录',
|
||||
urdf_files JSON NOT NULL DEFAULT (JSON_ARRAY()) COMMENT 'URDF文件清单数组,元素包含relativePath和sizeBytes,相对路径以urdf_prefix为基准',
|
||||
urdf_prefix VARCHAR(512) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL DEFAULT '' COMMENT 'URDF资源OSS目录前缀,包含环境根目录和上传批次,以斜杠结尾',
|
||||
urdf_prefix VARCHAR(512) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL DEFAULT '' COMMENT 'URDF资源环境相对目录,包含上传批次,以斜杠开头和结尾',
|
||||
project_type VARCHAR(64) NOT NULL COMMENT '配置中的项目类型编码',
|
||||
gitea_repo_id BIGINT NULL COMMENT 'Gitea 仓库标识',
|
||||
gitea_owner VARCHAR(100) NOT NULL COMMENT 'Gitea 仓库所属组织',
|
||||
@@ -91,10 +91,10 @@ CREATE TABLE project_like (
|
||||
|
||||
```sql
|
||||
ALTER TABLE project
|
||||
ADD COLUMN cover_url VARCHAR(2048) NULL COMMENT '封面图片 HTTPS 地址' AFTER description;
|
||||
ADD COLUMN cover_url VARCHAR(2048) NULL COMMENT '项目封面,平台资源存环境相对路径,外部图片存完整HTTPS地址' AFTER description;
|
||||
```
|
||||
|
||||
历史项目的封面为 NULL。字段保存固定公开地址,可以来自 OSS、CDN 或外部 HTTPS 图片;不存储临时签名下载地址。替换、清除封面和删除项目不会自动删除 OSS 对象,Fork 可能共用同一地址。
|
||||
历史项目的封面为 NULL。平台封面保存 `/project-covers/...` 环境相对路径;外部 HTTPS 图片保留完整地址。历史完整平台 URL 仍可读取,不保存临时签名下载地址。替换、清除封面和删除项目不会自动删除 OSS 对象,Fork 可能共用同一资源。
|
||||
|
||||
## 已有数据库增加项目 URDF
|
||||
|
||||
@@ -104,14 +104,193 @@ ALTER TABLE project
|
||||
ALTER TABLE project
|
||||
ADD COLUMN urdf_main_object_key VARCHAR(512)
|
||||
CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL DEFAULT ''
|
||||
COMMENT 'URDF主文件OSS对象键,不保存完整URL',
|
||||
COMMENT 'URDF主文件环境相对路径,以斜杠开头,不含域名和环境根目录',
|
||||
ADD COLUMN urdf_files JSON NOT NULL DEFAULT (JSON_ARRAY())
|
||||
COMMENT 'URDF文件清单数组,元素包含relativePath和sizeBytes,相对路径以urdf_prefix为基准',
|
||||
ADD COLUMN urdf_prefix VARCHAR(512)
|
||||
CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL DEFAULT ''
|
||||
COMMENT 'URDF资源OSS目录前缀,包含环境根目录和上传批次,以斜杠结尾';
|
||||
COMMENT 'URDF资源环境相对目录,包含上传批次,以斜杠开头和结尾';
|
||||
```
|
||||
|
||||
未上传或已清除时,两个字符串均为 `''`,文件清单为 `[]`。每个项目只绑定一个当前资源包,包内必须恰好有一个 `.urdf`(忽略扩展名大小写)。清单包含全部文件,例如 `[{"relativePath":"urdf/robot.urdf","sizeBytes":1234},{"relativePath":"meshes/base.stl","sizeBytes":5678}]`。
|
||||
|
||||
前缀为 `{环境根目录}/urdf/{projectCode}/{batchCode}/`,主文件对象键为此前缀加主文件相对路径。数据库不保存完整 URL;接口根据 OSS/CDN 配置生成并编码 URL。替换、清除、删除项目不物理删除对象,公开旧链接仍可读取;Fork 复制到新项目的独立目录,不共用源前缀。URDF 不进入 Gitea、不生成 Commit,更新立即影响社区展示。
|
||||
存储前缀为 `/urdf/{projectCode}/{batchCode}/`,主文件路径为此前缀加主文件相对路径,不含域名及环境根目录。实际 OSS key 为 `{环境根目录}{存储路径}`,接口根据 OSS/CDN 配置生成并编码 URL。历史带当前环境根目录的 key 兼容读取;其他环境的 key 不会被自动重定位。替换、清除、删除项目不物理删除对象,公开旧链接仍可读取;Fork 复制到新项目的独立目录,不共用源前缀。URDF 不进入 Gitea、不生成 Commit,更新立即影响社区展示。
|
||||
|
||||
## 资源路径历史数据迁移
|
||||
|
||||
先升级所有在线及导入写入进程,新版本兼容新旧格式,读取不修改记录。按环境分别执行以下操作;test 根目录为 `vibemake_dev`,prod 为 `vibemake`。OSS 对象不移动,`urdf_files`、`updated_at` 不变。不要在不同环境共用同一业务数据库。
|
||||
|
||||
迁移前备份数据库,并将候选记录的原值导出至备份目录。以下临时表仅属于当前数据库连接,不能代替持久备份。封面按每个受控公开基础地址分别执行一遍,包含 Bucket 标准域名、当前 CDN 基础地址和 `community.oss.legacy-public-base-urls` 中的历史地址;基础地址不含环境根目录。不使用全局 REPLACE,不转换外链、带参数/片段地址或其他环境路径。
|
||||
|
||||
### 预览与准备
|
||||
|
||||
以下命令在同一个 MySQL 8 连接内执行。仅自动转换上传流程生成的规范 ASCII 封面路径,编码形式、非标准目录和异常 URDF 记录列出后跳过,由维护人员核验。数据库比较使用二进制排序规则,避免默认不区分大小写导致误匹配。所有正则的输入和模式均显式指定 `utf8mb4_bin`,避免字段、会话变量和 CONCAT 结果的排序规则推导冲突(1267);仅设置 SET NAMES 不足以消除表达式冲突。这里是字符排序规则,不使用 BINARY 强制转换作为正则参数。
|
||||
|
||||
```sql
|
||||
-- resource-path-migration-prepare
|
||||
SET NAMES utf8mb4 COLLATE utf8mb4_bin;
|
||||
SET @resource_root = 'vibemake_dev';
|
||||
SET @resource_public_base = 'https://linkerbot.oss-cn-beijing.aliyuncs.com';
|
||||
SET @cover_prefix = CONCAT(TRIM(TRAILING '/' FROM @resource_public_base), '/', @resource_root, '/project-covers/');
|
||||
DROP TEMPORARY TABLE IF EXISTS resource_path_audit;
|
||||
CREATE TEMPORARY TABLE resource_path_audit AS
|
||||
SELECT project_code, cover_url AS old_cover_url,
|
||||
urdf_main_object_key AS old_main, urdf_prefix AS old_prefix,
|
||||
LEFT(cover_url, CHAR_LENGTH(@cover_prefix)) COLLATE utf8mb4_bin = @cover_prefix COLLATE utf8mb4_bin
|
||||
AND REGEXP_LIKE(SUBSTRING(cover_url, CHAR_LENGTH(@cover_prefix) + 1) COLLATE utf8mb4_bin,
|
||||
'^[a-z0-9]{12}([a-z0-9]{20})?/[0-9]{8}/[a-f0-9]{32}[.](jpg|png|webp)$' COLLATE utf8mb4_bin, 'c') AS cover_ok,
|
||||
REGEXP_LIKE(urdf_prefix COLLATE utf8mb4_bin,
|
||||
CONCAT('^', @resource_root, '/urdf/', TRIM(project_code), '/[a-f0-9]{32}/$') COLLATE utf8mb4_bin, 'c')
|
||||
AND LEFT(urdf_main_object_key, CHAR_LENGTH(urdf_prefix)) COLLATE utf8mb4_bin = urdf_prefix COLLATE utf8mb4_bin
|
||||
AND CHAR_LENGTH(urdf_main_object_key) > CHAR_LENGTH(urdf_prefix)
|
||||
AND REGEXP_LIKE(urdf_main_object_key COLLATE utf8mb4_bin, '[.]urdf$' COLLATE utf8mb4_bin, 'i')
|
||||
AND LOCATE('//', urdf_main_object_key) = 0
|
||||
AND LOCATE(CHAR(92), urdf_main_object_key) = 0
|
||||
AND NOT REGEXP_LIKE(urdf_main_object_key COLLATE utf8mb4_bin,
|
||||
'(^|/)[.]{1,2}(/|$)|(^|/)[.]git(/|$)|[[:cntrl:]:]' COLLATE utf8mb4_bin, 'i') AS urdf_ok
|
||||
FROM project;
|
||||
|
||||
DROP TEMPORARY TABLE IF EXISTS resource_path_changes;
|
||||
CREATE TEMPORARY TABLE resource_path_changes AS
|
||||
SELECT project_code, old_cover_url, old_main, old_prefix,
|
||||
CASE WHEN cover_ok THEN CONCAT('/project-covers/', SUBSTRING(old_cover_url, CHAR_LENGTH(@cover_prefix) + 1)) ELSE old_cover_url END AS new_cover_url,
|
||||
CASE WHEN urdf_ok THEN SUBSTRING(old_main, CHAR_LENGTH(@resource_root) + 1) ELSE old_main END AS new_main,
|
||||
CASE WHEN urdf_ok THEN SUBSTRING(old_prefix, CHAR_LENGTH(@resource_root) + 1) ELSE old_prefix END AS new_prefix
|
||||
FROM resource_path_audit WHERE cover_ok OR urdf_ok;
|
||||
|
||||
SELECT COUNT(*) AS candidate_count FROM resource_path_changes;
|
||||
-- 将下列结果导出并保存;包含迁移前后值,供审计及精确恢复使用。
|
||||
SELECT * FROM resource_path_changes ORDER BY project_code;
|
||||
-- 必须人工核验的封面及 URDF,结果不包含完整外链。
|
||||
SELECT project_code, '封面路径非标准或包含参数,请人工核验' AS reason
|
||||
FROM resource_path_audit
|
||||
WHERE LEFT(old_cover_url, CHAR_LENGTH(@cover_prefix)) COLLATE utf8mb4_bin = @cover_prefix COLLATE utf8mb4_bin AND NOT cover_ok
|
||||
UNION ALL
|
||||
SELECT project_code, 'URDF路径或主文件关系异常,或属于其他环境' AS reason
|
||||
FROM resource_path_audit
|
||||
WHERE (old_main <> '' OR old_prefix <> '') AND NOT urdf_ok
|
||||
AND NOT (LEFT(old_main, 6) = '/urdf/' AND LEFT(old_prefix, 6) = '/urdf/');
|
||||
```
|
||||
|
||||
### 人工执行转换
|
||||
|
||||
确认备份、预览和异常清单后,在原连接内执行。更新时比较三列原值,避免覆盖预览后的并发修改;数量不足时重新预览,不能删除原值比较条件强制更新。再次运行准备及转换流程时,已转换行不会再次命中。
|
||||
|
||||
```sql
|
||||
-- resource-path-migration-apply
|
||||
START TRANSACTION;
|
||||
UPDATE project p JOIN resource_path_changes c ON c.project_code = p.project_code
|
||||
SET p.cover_url = c.new_cover_url,
|
||||
p.urdf_main_object_key = c.new_main,
|
||||
p.urdf_prefix = c.new_prefix
|
||||
WHERE p.cover_url COLLATE utf8mb4_bin <=> c.old_cover_url COLLATE utf8mb4_bin
|
||||
AND p.urdf_main_object_key COLLATE utf8mb4_bin = c.old_main COLLATE utf8mb4_bin
|
||||
AND p.urdf_prefix COLLATE utf8mb4_bin = c.old_prefix COLLATE utf8mb4_bin;
|
||||
COMMIT;
|
||||
```
|
||||
|
||||
验证项目列表、详情、AI 卡片及 Fork 的公开 URL 与迁移前指向相同对象。迁移不触发项目重新发布,也不删除任何 OSS 对象。
|
||||
|
||||
### URDF 更新遇到 1267 时续跑
|
||||
|
||||
旧脚本如果在 URDF 更新处报 `Illegal mix of collations ... regexp_like`,先前成功的封面更新可能仍未提交。保留原连接,在同一连接执行下面的续跑 SQL;不要重新发起 START TRANSACTION 或修改字段排序规则。末尾 COMMIT 会同时提交该连接中先前成功的更新。prod 将根目录改为 `vibemake`。该脚本不再修改封面,且只匹配未转换的当前环境 URDF。
|
||||
|
||||
```sql
|
||||
-- resource-path-migration-resume-urdf
|
||||
SET NAMES utf8mb4 COLLATE utf8mb4_bin;
|
||||
SET @resource_root = 'vibemake_dev';
|
||||
UPDATE project
|
||||
SET urdf_main_object_key = SUBSTRING(urdf_main_object_key, CHAR_LENGTH(@resource_root) + 1),
|
||||
urdf_prefix = SUBSTRING(urdf_prefix, CHAR_LENGTH(@resource_root) + 1)
|
||||
WHERE REGEXP_LIKE(
|
||||
urdf_prefix COLLATE utf8mb4_bin,
|
||||
CONCAT('^', @resource_root, '/urdf/', TRIM(project_code), '/[a-f0-9]{32}/$') COLLATE utf8mb4_bin,
|
||||
'c'
|
||||
)
|
||||
AND LEFT(urdf_main_object_key, CHAR_LENGTH(urdf_prefix)) COLLATE utf8mb4_bin = urdf_prefix COLLATE utf8mb4_bin
|
||||
AND CHAR_LENGTH(urdf_main_object_key) > CHAR_LENGTH(urdf_prefix)
|
||||
AND REGEXP_LIKE(urdf_main_object_key COLLATE utf8mb4_bin, '[.]urdf$' COLLATE utf8mb4_bin, 'i')
|
||||
AND LOCATE('//', urdf_main_object_key) = 0
|
||||
AND LOCATE(CHAR(92), urdf_main_object_key) = 0
|
||||
AND NOT REGEXP_LIKE(
|
||||
urdf_main_object_key COLLATE utf8mb4_bin,
|
||||
'(^|/)[.]{1,2}(/|$)|(^|/)[.]git(/|$)|[[:cntrl:]:]' COLLATE utf8mb4_bin,
|
||||
'i'
|
||||
);
|
||||
COMMIT;
|
||||
```
|
||||
|
||||
### 更新字段中文注释
|
||||
|
||||
以下 DDL 独立执行,不与数据转换事务混用。字段名、类型、长度及空值约定不变;应用不会自动执行。
|
||||
|
||||
```sql
|
||||
ALTER TABLE project
|
||||
MODIFY COLUMN cover_url VARCHAR(2048) NULL
|
||||
COMMENT '项目封面,平台资源存环境相对路径,外部图片存完整HTTPS地址',
|
||||
MODIFY COLUMN urdf_main_object_key VARCHAR(512) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL DEFAULT ''
|
||||
COMMENT 'URDF主文件环境相对路径,以斜杠开头,不含域名和环境根目录',
|
||||
MODIFY COLUMN urdf_prefix VARCHAR(512) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL DEFAULT ''
|
||||
COMMENT 'URDF资源环境相对目录,包含上传批次,以斜杠开头和结尾';
|
||||
```
|
||||
|
||||
### 回退旧版本
|
||||
|
||||
先暂停所有资源写入并备份数据库。必须转换当前全部新格式记录,不能仅恢复迁移前备份,否则会丢失升级后的有效写入。回退封面时需要 UTF-8 URL 编码,使用离线生成器准备 SQL 值;工具不连接数据库,也不执行 SQL。
|
||||
|
||||
```sql
|
||||
-- 导出查询结果为 JSON Lines(仅结果值,无列头);保持资源写入暂停。
|
||||
SELECT JSON_OBJECT('projectCode', project_code, 'coverPath', cover_url)
|
||||
FROM project WHERE LEFT(cover_url, 16) = '/project-covers/';
|
||||
```
|
||||
|
||||
```bash
|
||||
python3 scripts/resource-path-rollback.py --root vibemake_dev \
|
||||
--public-base-url 'https://linkerbot.oss-cn-beijing.aliyuncs.com' \
|
||||
--input /backup/resource-covers.jsonl --output /backup/resource-cover-values.txt
|
||||
```
|
||||
|
||||
生成失败时不要继续。新开连接,设置与当前环境一致的根目录,执行以下准备 SQL,再人工执行生成的 `.txt` 中 INSERT 语句。生成器将中文、空格、百分号等正确编码;超出旧字段 2048 字符上限时会拒绝生成,需先修复该项目封面或保留兼容版本。
|
||||
|
||||
```sql
|
||||
-- resource-path-rollback-prepare
|
||||
SET NAMES utf8mb4 COLLATE utf8mb4_bin;
|
||||
SET @resource_root = 'vibemake_dev';
|
||||
DROP TEMPORARY TABLE IF EXISTS resource_cover_rollback;
|
||||
CREATE TEMPORARY TABLE resource_cover_rollback (
|
||||
project_code CHAR(32) NOT NULL PRIMARY KEY,
|
||||
cover_path VARCHAR(2048) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
|
||||
full_url VARCHAR(2048) NOT NULL
|
||||
);
|
||||
-- 在此执行生成器输出的 INSERT;为空文件表示没有待回退的平台封面。
|
||||
```
|
||||
|
||||
确认映射覆盖所有 `/project-covers/` 记录,再执行以下 SQL。URDF 两列仅在属于本项目且互相匹配时成对恢复前缀;异常记录先修复,不能直接回退代码。
|
||||
|
||||
```sql
|
||||
-- resource-path-rollback-apply
|
||||
START TRANSACTION;
|
||||
UPDATE project p JOIN resource_cover_rollback c ON c.project_code = p.project_code
|
||||
SET p.cover_url = c.full_url
|
||||
WHERE p.cover_url COLLATE utf8mb4_bin = c.cover_path COLLATE utf8mb4_bin;
|
||||
UPDATE project
|
||||
SET urdf_main_object_key = CONCAT(@resource_root, urdf_main_object_key),
|
||||
urdf_prefix = CONCAT(@resource_root, urdf_prefix)
|
||||
WHERE REGEXP_LIKE(urdf_prefix COLLATE utf8mb4_bin,
|
||||
CONCAT('^/urdf/', TRIM(project_code), '/[a-f0-9]{32}/$') COLLATE utf8mb4_bin, 'c')
|
||||
AND LEFT(urdf_main_object_key, CHAR_LENGTH(urdf_prefix)) COLLATE utf8mb4_bin = urdf_prefix COLLATE utf8mb4_bin
|
||||
AND CHAR_LENGTH(urdf_main_object_key) > CHAR_LENGTH(urdf_prefix)
|
||||
AND REGEXP_LIKE(urdf_main_object_key COLLATE utf8mb4_bin, '[.]urdf$' COLLATE utf8mb4_bin, 'i')
|
||||
AND LOCATE('//', urdf_main_object_key) = 0
|
||||
AND LOCATE(CHAR(92), urdf_main_object_key) = 0
|
||||
AND NOT REGEXP_LIKE(urdf_main_object_key COLLATE utf8mb4_bin,
|
||||
'(^|/)[.]{1,2}(/|$)|(^|/)[.]git(/|$)|[[:cntrl:]:]' COLLATE utf8mb4_bin, 'i')
|
||||
AND CHAR_LENGTH(CONCAT(@resource_root, urdf_main_object_key)) <= 512
|
||||
AND CHAR_LENGTH(CONCAT(@resource_root, urdf_prefix)) <= 512;
|
||||
COMMIT;
|
||||
SELECT project_code FROM project
|
||||
WHERE LEFT(cover_url, 16) = '/project-covers/'
|
||||
OR LEFT(urdf_main_object_key, 6) = '/urdf/' OR LEFT(urdf_prefix, 6) = '/urdf/';
|
||||
```
|
||||
|
||||
最后查询必须为空,随后验证旧格式地址并回退应用,再恢复写入。旧版不识别相对路径,禁止只回退代码而跳过数据转换。
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
COMMUNITY_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
if [[ -f "$COMMUNITY_ROOT/.env" ]]; then
|
||||
set -a
|
||||
source "$COMMUNITY_ROOT/.env"
|
||||
set +a
|
||||
fi
|
||||
cd "$COMMUNITY_ROOT"
|
||||
mvn -q -pl community-import -am -DskipTests package
|
||||
exec java -Djava.awt.headless=true -Dstdout.encoding=UTF-8 -Dstderr.encoding=UTF-8 \
|
||||
-Dlogging.level.root=WARN -Dlogging.level.org.springframework.jdbc=INFO \
|
||||
-Dlogging.level.org.apache.http=WARN -Ddebug=false \
|
||||
-jar community-import/target/community-import-0.0.1-SNAPSHOT.jar covers "$@"
|
||||
@@ -0,0 +1,67 @@
|
||||
#!/usr/bin/env python3
|
||||
"""将封面路径导出数据转换为供人工执行的回退 SQL;不连接数据库。"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
from urllib.parse import quote, urlsplit
|
||||
|
||||
|
||||
def sql_text(value: str) -> str:
|
||||
return "CONVERT(0x" + value.encode("utf-8").hex() + " USING utf8mb4)"
|
||||
|
||||
|
||||
def generate(rows: list[dict], root: str, public_base: str) -> str:
|
||||
if not re.fullmatch(r"[A-Za-z0-9_-]+", root):
|
||||
raise ValueError("环境根目录不合法")
|
||||
base = urlsplit(public_base)
|
||||
if (base.scheme != "https" or not base.hostname or base.username is not None
|
||||
or any(char.isspace() or ord(char) < 32 for char in public_base)
|
||||
or base.password is not None or "?" in public_base or "#" in public_base
|
||||
or (base.port is not None and not 1 <= base.port <= 65535)):
|
||||
raise ValueError("公开基础地址必须是无凭据和参数的 HTTPS 地址")
|
||||
statements = []
|
||||
seen = set()
|
||||
for row in rows:
|
||||
code = row.get("projectCode", "")
|
||||
path = row.get("coverPath", "")
|
||||
if not isinstance(code, str) or not re.fullmatch(r"[a-z0-9]{12}|[a-z0-9]{32}", code):
|
||||
raise ValueError("项目编码不合法")
|
||||
if code in seen:
|
||||
raise ValueError(f"项目编码重复 projectCode={code}")
|
||||
seen.add(code)
|
||||
if (not isinstance(path, str) or not path.startswith("/project-covers/")
|
||||
or any(ord(char) < 32 or ord(char) == 127 for char in path)
|
||||
or "\\" in path or ":" in path
|
||||
or any(part in ("", ".", "..") or part.lower() == ".git" for part in path[1:].split("/"))):
|
||||
raise ValueError(f"封面存储路径异常 projectCode={code}")
|
||||
full_url = public_base.rstrip("/") + "/" + root + quote(path, safe="/!$&'()*+,;=@-._~")
|
||||
if len(full_url) > 2048:
|
||||
raise ValueError(f"编码后的封面地址超过2048字符,无法回退 projectCode={code}")
|
||||
statements.append("INSERT INTO resource_cover_rollback (project_code, cover_path, full_url) VALUES ("
|
||||
+ ", ".join(sql_text(value) for value in (code, path, full_url)) + ");")
|
||||
return "\n".join(statements) + ("\n" if statements else "")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--root", required=True)
|
||||
parser.add_argument("--public-base-url", required=True)
|
||||
parser.add_argument("--input", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
try:
|
||||
rows = [json.loads(line) for line in args.input.read_text(encoding="utf-8").splitlines() if line.strip()]
|
||||
result = generate(rows, args.root, args.public_base_url)
|
||||
# 全部校验成功后才落盘,不输出原始地址或路径。
|
||||
args.output.write_text(result, encoding="utf-8")
|
||||
print(f"已生成回退SQL,记录数={len(rows)};请按数据库文档人工执行")
|
||||
except (ValueError, TypeError, AttributeError, OSError) as error:
|
||||
if isinstance(error, json.JSONDecodeError):
|
||||
parser.exit(1, "输入不是合法的JSON Lines,请使用无列头导出\n")
|
||||
parser.exit(1, f"回退SQL生成失败:{error.__class__.__name__};检查输入文件、地址、路径及字段长度\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,59 @@
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
|
||||
spec = importlib.util.spec_from_file_location("rollback", Path(__file__).parents[1] / "resource-path-rollback.py")
|
||||
rollback = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(rollback)
|
||||
|
||||
|
||||
class ResourcePathRollbackTest(unittest.TestCase):
|
||||
def test_preserves_unicode_percent_plus_and_sql_quotes_without_changing_keys(self):
|
||||
path = "/project-covers/用户/a #?%+'图片.png"
|
||||
sql = rollback.generate([{"projectCode": "1" * 12, "coverPath": path}],
|
||||
"vibemake_dev", "https://cdn.example.com/assets/")
|
||||
values = [bytes.fromhex(value).decode("utf-8") for value in re.findall(r"CONVERT\(0x([0-9a-f]+) USING utf8mb4\)", sql)]
|
||||
self.assertEqual(values, ["1" * 12, path,
|
||||
"https://cdn.example.com/assets/vibemake_dev/project-covers/%E7%94%A8%E6%88%B7/"
|
||||
"a%20%23%3F%25+'%E5%9B%BE%E7%89%87.png"])
|
||||
self.assertNotIn(path, sql)
|
||||
|
||||
def test_rejects_duplicates_invalid_paths_and_invalid_origins(self):
|
||||
valid = {"projectCode": "1" * 12, "coverPath": "/project-covers/a.png"}
|
||||
with self.assertRaises(ValueError):
|
||||
rollback.generate([valid, valid], "vibemake", "https://cdn.example.com")
|
||||
for path in ("/urdf/a.png", "/project-covers/../a.png", "/project-covers/a\\b.png", "/project-covers/a\n.png"):
|
||||
with self.subTest(path=path), self.assertRaises(ValueError):
|
||||
rollback.generate([{**valid, "coverPath": path}], "vibemake", "https://cdn.example.com")
|
||||
for base in ("http://cdn.example.com", "https://user:pass@cdn.example.com", "https://cdn.example.com?token=example"):
|
||||
with self.subTest(base=base), self.assertRaises(ValueError):
|
||||
rollback.generate([valid], "vibemake", base)
|
||||
|
||||
def test_rejects_urls_that_would_overflow_old_schema(self):
|
||||
with self.assertRaises(ValueError):
|
||||
rollback.generate([{"projectCode": "1" * 12, "coverPath": "/project-covers/" + "中" * 300}],
|
||||
"vibemake", "https://cdn.example.com")
|
||||
self.assertEqual(rollback.generate([], "vibemake", "https://cdn.example.com"), "")
|
||||
|
||||
def test_cli_does_not_overwrite_output_or_log_input_on_failure(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
input_path = Path(directory) / "input.jsonl"
|
||||
output_path = Path(directory) / "output.txt"
|
||||
input_path.write_text('{"projectCode":"invalid-secret-token","coverPath":"/bad"}', encoding="utf-8")
|
||||
output_path.write_text("existing output", encoding="utf-8")
|
||||
result = subprocess.run([sys.executable, str(Path(__file__).parents[1] / "resource-path-rollback.py"),
|
||||
"--root", "vibemake", "--public-base-url", "https://cdn.example.com",
|
||||
"--input", str(input_path), "--output", str(output_path)],
|
||||
capture_output=True, text=True, check=False)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertEqual(output_path.read_text(encoding="utf-8"), "existing output")
|
||||
self.assertNotIn("invalid-secret-token", result.stdout + result.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user