跨域配置和fork修改为git原生fork
This commit is contained in:
@@ -34,3 +34,11 @@ AI_REQUEST_TIMEOUT=30s
|
||||
AI_SEARCH_TIMEOUT=65s
|
||||
AI_MAX_CONCURRENT=4
|
||||
AI_REQUESTS_PER_MINUTE=10
|
||||
|
||||
# Fork 依赖 Git 和 git-lfs;普通文件在线读取限制不影响 Git 快照复制。
|
||||
FORK_GIT_EXECUTABLE=git
|
||||
FORK_TIMEOUT=5m
|
||||
FORK_MAX_CONCURRENT=2
|
||||
FORK_MAX_TEMPORARY_BYTES=5368709120
|
||||
# 可选:独立临时磁盘目录,默认系统临时目录下 community-fork。
|
||||
# FORK_TEMPORARY_DIRECTORY=/var/tmp/community-fork
|
||||
|
||||
@@ -45,7 +45,7 @@ bash scripts/run-local.sh
|
||||
|
||||
## 环境与日志
|
||||
|
||||
配置与日志方式参考 cadworld:application.yml 保存公共配置,application-test.yml 和 application-prod.yml 分别保存环境相关的数据库、Gitea 和日志配置。不指定环境时默认 test,通过 SPRING_PROFILES_ACTIVE 或启动参数切换。
|
||||
配置与日志方式参考 cadworld:application.yml 保存公共配置,application-test.yml 和 application-prod.yml 分别保存环境相关的数据库、Gitea、跨域白名单和日志配置。不指定环境时默认 test,通过 SPRING_PROFILES_ACTIVE 或启动参数切换。
|
||||
|
||||
prod 启动时检查数据库连接、用户名、密码、Gitea 地址和 Token 是否已配置且非空;缺失时停止启动,错误信息只包含配置项名称。
|
||||
|
||||
@@ -53,6 +53,7 @@ prod 启动时检查数据库连接、用户名、密码、Gitea 地址和 Token
|
||||
| --- | --- | --- |
|
||||
| 数据库 | 默认本机 community,可用 DB_* 覆盖 | 必须配置 DB_URL、DB_USERNAME、DB_PASSWORD |
|
||||
| Gitea | 默认本机地址,使用 GITEA_* 覆盖 | 必须配置 GITEA_BASE_URL、GITEA_TOKEN;组织默认 community |
|
||||
| API 跨域来源 | `https://dev-vibemake.robotquan.com` | `https://vibemake.robotquan.com` |
|
||||
| OSS 封面目录 | vibemake_dev/project-covers/ | vibemake/project-covers/ |
|
||||
| OSS URDF 目录 | vibemake_dev/urdf/{projectCode}/{batchCode}/ | vibemake/urdf/{projectCode}/{batchCode}/ |
|
||||
| 连接池 | 最少 1、最多 10 个连接 | 最少 1、最多 20 个连接 |
|
||||
@@ -67,6 +68,8 @@ SPRING_PROFILES_ACTIVE=prod java -Djava.awt.headless=true -jar community-api/tar
|
||||
|
||||
本机脚本支持传递启动参数,例如 `bash scripts/run-local.sh --spring.profiles.active=prod`,此时加载 .env 中的连接信息并使用 prod 配置。测试和生产数据库、Gitea 组织需要各自隔离时,分别设置对应的 DB_*、GITEA_* 变量。
|
||||
|
||||
跨域白名单由 `community.cors.allowed-origins` 指定,仅作用于 `/api/**`。支持 Bearer Token、JSON、文件上传下载和 AI SSE,预检缓存 3600 秒,暴露 `X-Request-Id`、`Content-Disposition`,不启用跨域 Cookie 凭据。启动日志会记录生效的白名单。默认不开放 HTTP、localhost、其他端口或跨环境来源;本地继续使用前端开发代理,配置见[开发代理示例](docs/community-api.md#开发代理示例)。部署代理需透传预检请求、Origin 和后端 CORS 响应头,避免重复添加跨域头;OSS Bucket 的 CORS 仍需独立配置。
|
||||
|
||||
日志配置沿用 cadworld-common 的 logback-spring.xml 模板,放在 community-common;应用名为 community-api。测试和生产默认都写入 logs,可用 LOG_PATH 覆盖目录;目录相对于应用启动时的工作目录。本机启动脚本从项目根目录运行。
|
||||
|
||||
同时输出控制台、community-api.log 和 community-api-error.log,日志时间使用 Asia/Shanghai,格式包含线程、级别、requestId 和类名。应用日志按天或每 100MB 归档,保留 30 天、总归档上限 10GB;错误日志仅接收 ERROR,按天或每 50MB 归档,保留 60 天、总归档上限 5GB。归档写入 archive 子目录并压缩为 gzip。SQL Mapper 和 Spring Web 保持 INFO,不记录凭据、邮箱或代码内容。
|
||||
@@ -113,7 +116,7 @@ Bucket 需要配置前端实际域名的 CORS,允许 POST;图片通过公开
|
||||
|
||||
查找时仅检索已发布且仓库就绪项目的名称、描述、类型:先理解需求,再查询最多20个候选,最后排序并校验最多5个项目卡片。项目编码、引用原文和当前发布状态由后端校验;缺少依据的明确要求不算匹配,未证实的偏好单独列出。
|
||||
|
||||
每次模型调用默认30秒,整个处理65秒(含仓库操作);`AI_REQUEST_TIMEOUT`、`AI_SEARCH_TIMEOUT` 可调整。`AI_MAX_CONCURRENT` 默认4,`AI_REQUESTS_PER_MINUTE` 默认每用户10次,均为单实例内存控制,超限不排队。连接断开时尽力取消上游调用,实际工作结束后释放并发名额。写入开始后取消或超时不保证回滚;错误事件保留初始化异常中的 projectCode,先核对工作区,不自动重发创建或 Fork。AI 错误不影响普通项目搜索。前端请求和 SSE 解析示例见接口文档的“AI 项目助手”。
|
||||
每次模型调用默认30秒,模型/搜索阶段65秒;进入 Fork 阶段后采用独立5分钟预算并保留15秒入口及终态余量,前端和代理读取超时至少400秒;`AI_REQUEST_TIMEOUT`、`AI_SEARCH_TIMEOUT` 可调整。`AI_MAX_CONCURRENT` 默认4,`AI_REQUESTS_PER_MINUTE` 默认每用户10次,均为单实例内存控制,超限不排队。连接断开时尽力取消上游调用,实际工作结束后释放并发名额。写入开始后取消或超时不保证回滚;错误事件保留初始化异常中的 projectCode,先核对工作区,不自动重发创建或 Fork。AI 错误不影响普通项目搜索。前端请求和 SSE 解析示例见接口文档的“AI 项目助手”。
|
||||
|
||||
本期不读取 README、代码或评论,不建立向量索引,不修改四表结构。检索效果依赖项目描述,建议写清用途、技术栈、主要硬件和适用场景。描述修改即时生效;工作区文件提交仍须重新发布才更新社区代码。
|
||||
|
||||
@@ -125,6 +128,26 @@ python3 scripts/test-ai.py
|
||||
|
||||
脚本在内存 H2 中建立公开示例项目,验证四足检索及多轮改为轮式,默认 Maven 测试跳过该联调。密钥从当前项目 `.env` 或环境变量读取,脚本不输出密钥、不写真实数据库、不访问 Gitea。
|
||||
|
||||
## Git 快照 Fork
|
||||
|
||||
后端保留鉴权、项目编码、来源记录和 OSS 复制,仓库数据通过 Git 对象级传输。按发布 SHA 的根 tree 生成一个无父提交,只推送新 main;保留普通大文件、文件模式、符号链接和子模块引用,不继承上游历史、其他分支或标签。子模块依赖仓库不递归复制。LFS 仅传输发布快照所需实体,推送后从新副本独立下载校验;不会只有指针却缺少实际对象。
|
||||
|
||||
应用运行用户的 PATH 需要包含 Git 和 git-lfs(本机验证 Git 2.52 / git-lfs 3.8)。CI 同样安装这两个程序,执行本地 Git 仓库及模拟 LFS 协议测试;普通业务测试替换仓库传输边界,不访问外部服务。启动时检查依赖,缺失仅使 Fork 返回 FORK_UNAVAILABLE,不阻止普通创建或其他接口。
|
||||
|
||||
开发环境可通过 `brew install git git-lfs` 安装;Debian/Ubuntu 的 CI 或部署镜像安装 `git`、`git-lfs` 软件包。以应用运行用户执行 `git --version` 和 `git lfs version` 确认 PATH 生效,无需执行全局 `git lfs install`。LFS 传输通过任务内的本地端点校验转发,拒绝 HTTP 重定向及指向其他地址的 action URL;临时目录禁止写入 git-lfs 原始错误报告,防止环境凭据落盘。
|
||||
|
||||
| 配置 | 默认值 | 用途 |
|
||||
| --- | --- | --- |
|
||||
| FORK_GIT_EXECUTABLE | git | Git 可执行文件路径,git-lfs 从子进程 PATH 查找 |
|
||||
| FORK_TIMEOUT | 5m | Git、LFS、核验和 OSS 共用的总时间预算 |
|
||||
| FORK_MAX_CONCURRENT | 2 | 单实例并发上限,实际工作结束后释放名额 |
|
||||
| FORK_MAX_TEMPORARY_BYTES | 5368709120 | 每次临时磁盘上限,含 LFS 源缓存与副本校验缓存 |
|
||||
| FORK_TEMPORARY_DIRECTORY | 系统临时目录/community-fork | 建议使用容量充足的独立临时磁盘 |
|
||||
|
||||
服务账号的 GITEA_TOKEN 需同时允许 REST 仓库创建、HTTP Git 读写及 LFS 读写。凭据通过进程环境注入,只用于配置的 Gitea 地址;不写入 URL、命令参数、Git 配置或日志。Fork 接口继续同步返回,前端和反向代理读取超时至少400秒;AI 进入 forking 后切换专用预算并继续发送心跳。
|
||||
|
||||
Fork 不受普通在线文件读取的10 MiB限制,文件浏览、下载和 LFS 在线展开能力仍维持原规则。失败时先核对响应 data.projectCode:明确失败为 FAILED,远端写入或数据库状态待核对为 CREATING;不会自动重发、删除远端仓库或 OSS 对象。完成后清理本地临时目录。使用 scripts/test-gitea.py 可执行显式真实 Gitea 联调,仅创建并清理本次测试专用仓库,不修改已有社区项目。
|
||||
|
||||
## 运行约定
|
||||
|
||||
项目名称、描述、类型、封面修改直接更新展示;工作区代码提交后,社区仍读取已发布 SHA,需要手动更新。Fork 复制已发布文件快照,不继承上游提交历史,也不自动同步。项目删除采用逻辑删除,保留仓库和已有 Fork。
|
||||
@@ -162,4 +185,4 @@ bash scripts/import-github-projects.sh --excel "$IMPORT_EXCEL" --execute --owner
|
||||
|
||||
报告逐项包含 Sheet、行号、短编码、阶段、错误码和兼容性提醒。退出码 0 表示本次选定工作完成,1 表示命令、配置或持久化等致命错误,2 表示存在预检未通过、失败或待核对项目。在线仓库或图片失效会减少实际导入量;报告中的候选数仍与静态筛选对账。
|
||||
|
||||
本次只完成仓库迁移、关联和发布。社区仍有单文件读取限制,LFS 实体读取、子模块及符号链接的浏览/Fork 能力没有扩展;当前发布快照检测到的限制会写入报告。子模块只保留引用,不递归迁移独立仓库,也不下载网盘或其他外链资源。保留所有分支不代表社区提供分支切换界面。
|
||||
本次只完成仓库迁移、关联和发布。在线浏览和下载仍有单文件限制,LFS 实体、子模块及符号链接的在线浏览能力未扩展;Git 快照 Fork 已支持大文件、LFS 实体、符号链接及子模块引用,依赖仓库仍不递归复制。当前发布快照检测到的限制会写入报告。子模块只保留引用,不递归迁移独立仓库,也不下载网盘或其他外链资源。保留所有分支不代表社区提供分支切换界面。
|
||||
|
||||
@@ -11,6 +11,9 @@ spring:
|
||||
connection-timeout: 5000
|
||||
|
||||
community:
|
||||
cors:
|
||||
allowed-origins:
|
||||
- https://vibemake.robotquan.com
|
||||
oss:
|
||||
root-directory: vibemake
|
||||
gitea:
|
||||
|
||||
@@ -11,6 +11,9 @@ spring:
|
||||
connection-timeout: 5000
|
||||
|
||||
community:
|
||||
cors:
|
||||
allowed-origins:
|
||||
- https://dev-vibemake.robotquan.com
|
||||
oss:
|
||||
root-directory: vibemake_dev
|
||||
gitea:
|
||||
|
||||
@@ -55,6 +55,12 @@ community:
|
||||
gitea:
|
||||
connect-timeout: 5s
|
||||
request-timeout: 30s
|
||||
fork:
|
||||
git-executable: ${FORK_GIT_EXECUTABLE:git}
|
||||
timeout: ${FORK_TIMEOUT:5m}
|
||||
max-concurrent: ${FORK_MAX_CONCURRENT:2}
|
||||
max-temporary-bytes: ${FORK_MAX_TEMPORARY_BYTES:5368709120}
|
||||
temporary-directory: ${FORK_TEMPORARY_DIRECTORY:${java.io.tmpdir}/community-fork}
|
||||
springdoc:
|
||||
swagger-ui:
|
||||
path: /swagger-ui.html
|
||||
|
||||
@@ -21,6 +21,7 @@ import com.linkerbot.community.service.ProjectService;
|
||||
import com.linkerbot.community.service.ai.AiModelClient;
|
||||
import com.linkerbot.community.service.ai.AiProjectAssistant;
|
||||
import com.linkerbot.community.service.config.AiProperties;
|
||||
import com.linkerbot.community.service.config.ForkProperties;
|
||||
import com.linkerbot.community.service.gitea.GiteaClient.Change;
|
||||
import com.linkerbot.community.service.mapper.ProjectMapper;
|
||||
import com.linkerbot.community.service.mapper.UserMapper;
|
||||
@@ -61,6 +62,8 @@ import org.springframework.test.web.servlet.*;
|
||||
class AiSearchTest {
|
||||
@TestConfiguration
|
||||
static class Config {
|
||||
@Bean @Primary FakeGitSnapshotCopier fakeSnapshots(FakeGiteaClient gitea) { return new FakeGitSnapshotCopier(gitea); }
|
||||
|
||||
@Bean @Primary FakeModel fakeModel() { return new FakeModel(); }
|
||||
@Bean @Primary FakeGiteaClient fakeGitea() { return new FakeGiteaClient(); }
|
||||
}
|
||||
@@ -86,12 +89,17 @@ class AiSearchTest {
|
||||
@Autowired FakeGiteaClient gitea;
|
||||
@Autowired ProjectService projectService;
|
||||
@Autowired AiProjectAssistant assistant;
|
||||
@Autowired FakeGitSnapshotCopier snapshots;
|
||||
@Autowired ForkProperties forkProperties;
|
||||
private User user;
|
||||
private String token;
|
||||
|
||||
@BeforeEach void reset() throws Exception {
|
||||
SchemaSupport.reset(jdbc);
|
||||
gitea.reset();
|
||||
snapshots.available = true;
|
||||
snapshots.beforePush = null;
|
||||
forkProperties.setTimeout(Duration.ofMinutes(5));
|
||||
user = new User();
|
||||
user.setId(USER_SEQUENCE.incrementAndGet());
|
||||
user.setUserCode(Codes.next());
|
||||
@@ -178,6 +186,28 @@ class AiSearchTest {
|
||||
return events.getLast();
|
||||
}
|
||||
|
||||
@Test
|
||||
void crossOriginSseKeepsCorsHeadersAcrossAsyncDispatch() throws Exception {
|
||||
String origin = "https://dev-vibemake.robotquan.com";
|
||||
MvcResult initial = mvc.perform(post("/api/community/ai/chat")
|
||||
.header("Origin", origin)
|
||||
.header("Authorization", "Bearer " + token)
|
||||
.accept("text/event-stream")
|
||||
.contentType("application/json")
|
||||
.content(json.writeValueAsBytes(Map.of("message", "推荐机器人项目"))))
|
||||
.andExpect(status().isOk())
|
||||
.andExpect(request().asyncStarted())
|
||||
.andExpect(header().string("Access-Control-Allow-Origin", origin))
|
||||
.andReturn();
|
||||
initial.getAsyncResult(6000);
|
||||
mvc.perform(asyncDispatch(initial))
|
||||
.andExpect(status().isOk())
|
||||
.andExpect(content().contentTypeCompatibleWith("text/event-stream"))
|
||||
.andExpect(header().string("Access-Control-Allow-Origin", origin))
|
||||
.andExpect(header().exists("Access-Control-Expose-Headers"))
|
||||
.andExpect(header().exists("X-Request-Id"));
|
||||
}
|
||||
|
||||
private ObjectNode createIntent(String type, String name) {
|
||||
return intent(type, List.of()).put("action", "CREATE").put("projectName", name)
|
||||
.put("description", "计划用于机器人教学实验");
|
||||
@@ -293,6 +323,45 @@ class AiSearchTest {
|
||||
.containsExactly("understanding", "forking");
|
||||
}
|
||||
|
||||
@Test void forkGetsItsOwnDeadlineAfterUnderstandingRatherThanTheSearchTimeout() throws Exception {
|
||||
Project source = publishedRepository("教学项目");
|
||||
properties.setSearchTimeout(Duration.ofMillis(150));
|
||||
forkProperties.setTimeout(Duration.ofSeconds(2));
|
||||
model.handler = (name, input) -> forkIntent(source.getProjectCode(), null);
|
||||
snapshots.beforePush = () -> {
|
||||
try { Thread.sleep(350); }
|
||||
catch (InterruptedException error) { Thread.currentThread().interrupt(); throw new BusinessException(499, "FORK_CANCELLED", "已取消"); }
|
||||
};
|
||||
JsonNode result = finish(start(Map.of("message", "Fork " + source.getProjectCode()))).getLast();
|
||||
assertThat(result.path("code").asText()).isEqualTo("OK");
|
||||
assertThat(result.path("data").path("status").asText()).isEqualTo("FORKED");
|
||||
}
|
||||
|
||||
@Test void disconnectDuringForkInterruptsItsWorkerAndFinishesTheProjectState() throws Exception {
|
||||
Project source = publishedRepository("取消 Fork 测试");
|
||||
model.handler = (name, input) -> forkIntent(source.getProjectCode(), null);
|
||||
CountDownLatch entered = new CountDownLatch(1);
|
||||
CountDownLatch interrupted = new CountDownLatch(1);
|
||||
snapshots.beforePush = () -> {
|
||||
entered.countDown();
|
||||
try { new CountDownLatch(1).await(); }
|
||||
catch (InterruptedException error) { interrupted.countDown(); Thread.currentThread().interrupt(); }
|
||||
throw new BusinessException(499, "FORK_CANCELLED", "已取消");
|
||||
};
|
||||
MvcResult active = start(Map.of("message", "Fork " + source.getProjectCode()));
|
||||
assertThat(entered.await(2, TimeUnit.SECONDS)).isTrue();
|
||||
MockAsyncContext context = (MockAsyncContext) active.getRequest().getAsyncContext();
|
||||
for (AsyncListener listener : context.getListeners()) listener.onError(new AsyncEvent(context, new IOException("closed")));
|
||||
assertThat(interrupted.await(2, TimeUnit.SECONDS)).isTrue();
|
||||
await().atMost(Duration.ofSeconds(2)).untilAsserted(() -> {
|
||||
List<Project> copies = projects.selectList(null).stream().filter(project -> !project.getProjectCode().equals(source.getProjectCode())).toList();
|
||||
assertThat(copies).singleElement().satisfies(copy -> assertThat(copy.getRepoStatus()).isEqualTo("FAILED"));
|
||||
});
|
||||
snapshots.beforePush = null;
|
||||
assertThat(finish(start(Map.of("message", "Fork " + source.getProjectCode()))).getLast()
|
||||
.path("data").path("status").asText()).isEqualTo("FORKED");
|
||||
}
|
||||
|
||||
@Test void forksCurrentPublicationRatherThanTheVersionShownPreviously() throws Exception {
|
||||
Project source = publishedRepository("教学项目");
|
||||
String latest = gitea.commit(projectService.repository(source),
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
package com.linkerbot.community.api;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import com.linkerbot.community.security.CorsProperties;
|
||||
import com.linkerbot.community.security.SecurityConfiguration;
|
||||
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.CsvSource;
|
||||
import org.springframework.boot.context.properties.bind.Bindable;
|
||||
import org.springframework.boot.context.properties.bind.Binder;
|
||||
import org.springframework.boot.env.YamlPropertySourceLoader;
|
||||
import org.springframework.core.env.PropertySource;
|
||||
import org.springframework.core.env.StandardEnvironment;
|
||||
import org.springframework.core.io.ClassPathResource;
|
||||
import org.springframework.mock.web.MockHttpServletRequest;
|
||||
import org.springframework.web.cors.CorsConfiguration;
|
||||
import org.springframework.web.cors.UrlBasedCorsConfigurationSource;
|
||||
|
||||
class CorsProfileTest {
|
||||
@ParameterizedTest
|
||||
@CsvSource({
|
||||
"test, https://dev-vibemake.robotquan.com, https://vibemake.robotquan.com",
|
||||
"prod, https://vibemake.robotquan.com, https://dev-vibemake.robotquan.com"
|
||||
})
|
||||
void profilesAllowOnlyTheirOwnHttpsFrontend(String profile, String allowedOrigin, String otherOrigin)
|
||||
throws Exception {
|
||||
StandardEnvironment environment = new StandardEnvironment();
|
||||
YamlPropertySourceLoader loader = new YamlPropertySourceLoader();
|
||||
for (PropertySource<?> source : loader.load(profile,
|
||||
new ClassPathResource("application-" + profile + ".yml"))) {
|
||||
environment.getPropertySources().addFirst(source);
|
||||
}
|
||||
CorsProperties properties = Binder.get(environment)
|
||||
.bind("community.cors", Bindable.of(CorsProperties.class)).get();
|
||||
assertThat(properties.getAllowedOrigins()).containsExactly(allowedOrigin);
|
||||
|
||||
UrlBasedCorsConfigurationSource source =
|
||||
new SecurityConfiguration().corsConfigurationSource(properties);
|
||||
CorsConfiguration configuration = source.getCorsConfiguration(
|
||||
new MockHttpServletRequest("OPTIONS", "/api/workspace/projects"));
|
||||
assertThat(configuration).isNotNull();
|
||||
assertThat(configuration.checkOrigin(allowedOrigin)).isEqualTo(allowedOrigin);
|
||||
assertThat(configuration.checkOrigin(otherOrigin)).isNull();
|
||||
assertThat(configuration.checkOrigin(allowedOrigin.replace("https://", "http://"))).isNull();
|
||||
assertThat(configuration.checkOrigin(allowedOrigin + ":8443")).isNull();
|
||||
assertThat(configuration.checkOrigin("http://localhost:5173")).isNull();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,165 @@
|
||||
package com.linkerbot.community.api;
|
||||
|
||||
import static org.hamcrest.Matchers.containsString;
|
||||
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.*;
|
||||
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.*;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.CsvSource;
|
||||
import org.junit.jupiter.params.provider.ValueSource;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc;
|
||||
import org.springframework.boot.test.autoconfigure.web.servlet.MockMvcPrint;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
import org.springframework.http.HttpHeaders;
|
||||
import org.springframework.http.MediaType;
|
||||
import org.springframework.test.context.ActiveProfiles;
|
||||
import org.springframework.test.web.servlet.MockMvc;
|
||||
|
||||
@SpringBootTest(properties = {
|
||||
"spring.datasource.url=jdbc:h2:mem:cors;MODE=MySQL;DB_CLOSE_DELAY=-1",
|
||||
"spring.datasource.username=sa",
|
||||
"spring.datasource.password=",
|
||||
"spring.datasource.driver-class-name=org.h2.Driver",
|
||||
"community.ai.enabled=false",
|
||||
"community.oss.enabled=false",
|
||||
"logging.level.org.springframework=WARN"
|
||||
})
|
||||
@ActiveProfiles("test")
|
||||
@AutoConfigureMockMvc(print = MockMvcPrint.NONE)
|
||||
class CorsTest {
|
||||
private static final String FRONTEND_ORIGIN = "https://dev-vibemake.robotquan.com";
|
||||
|
||||
@Autowired MockMvc mvc;
|
||||
|
||||
@ParameterizedTest
|
||||
@CsvSource({
|
||||
"GET, /api/users/me",
|
||||
"HEAD, /api/workspace/projects",
|
||||
"POST, /api/auth/login",
|
||||
"POST, /api/community/ai/chat",
|
||||
"PUT, /api/workspace/projects/example/publication",
|
||||
"PATCH, /api/workspace/projects/example",
|
||||
"DELETE, /api/workspace/projects/example",
|
||||
"OPTIONS, /api/workspace/projects"
|
||||
})
|
||||
void preflightAllowsSupportedMethodsAndHeadersWithoutAuthentication(String method, String path)
|
||||
throws Exception {
|
||||
mvc.perform(options(path)
|
||||
.header(HttpHeaders.ORIGIN, FRONTEND_ORIGIN)
|
||||
.header(HttpHeaders.ACCESS_CONTROL_REQUEST_METHOD, method)
|
||||
.header(HttpHeaders.ACCESS_CONTROL_REQUEST_HEADERS,
|
||||
"authorization, content-type, accept"))
|
||||
.andExpect(status().isOk())
|
||||
.andExpect(header().string(HttpHeaders.ACCESS_CONTROL_ALLOW_ORIGIN, FRONTEND_ORIGIN))
|
||||
.andExpect(header().string(HttpHeaders.ACCESS_CONTROL_ALLOW_METHODS, containsString(method)))
|
||||
.andExpect(header().string(HttpHeaders.ACCESS_CONTROL_ALLOW_HEADERS, containsString("authorization")))
|
||||
.andExpect(header().string(HttpHeaders.ACCESS_CONTROL_ALLOW_HEADERS, containsString("content-type")))
|
||||
.andExpect(header().string(HttpHeaders.ACCESS_CONTROL_ALLOW_HEADERS, containsString("accept")))
|
||||
.andExpect(header().string(HttpHeaders.ACCESS_CONTROL_MAX_AGE, "3600"))
|
||||
.andExpect(header().doesNotExist(HttpHeaders.ACCESS_CONTROL_ALLOW_CREDENTIALS));
|
||||
}
|
||||
|
||||
@Test
|
||||
void preflightRunsBeforeBearerTokenValidation() throws Exception {
|
||||
mvc.perform(options("/api/users/me")
|
||||
.header(HttpHeaders.ORIGIN, FRONTEND_ORIGIN)
|
||||
.header(HttpHeaders.ACCESS_CONTROL_REQUEST_METHOD, "GET")
|
||||
.header(HttpHeaders.AUTHORIZATION, "Bearer invalid-test-token"))
|
||||
.andExpect(status().isOk())
|
||||
.andExpect(header().string(HttpHeaders.ACCESS_CONTROL_ALLOW_ORIGIN, FRONTEND_ORIGIN));
|
||||
}
|
||||
|
||||
@Test
|
||||
void publicResponsesExposeRequestIdAndDownloadHeaders() throws Exception {
|
||||
mvc.perform(get("/api/project-types").header(HttpHeaders.ORIGIN, FRONTEND_ORIGIN))
|
||||
.andExpect(status().isOk())
|
||||
.andExpect(jsonPath("$.code").value("OK"))
|
||||
.andExpect(header().string(HttpHeaders.ACCESS_CONTROL_ALLOW_ORIGIN, FRONTEND_ORIGIN))
|
||||
.andExpect(header().string(HttpHeaders.ACCESS_CONTROL_EXPOSE_HEADERS, containsString("X-Request-Id")))
|
||||
.andExpect(header().string(HttpHeaders.ACCESS_CONTROL_EXPOSE_HEADERS, containsString("Content-Disposition")))
|
||||
.andExpect(header().exists("X-Request-Id"))
|
||||
.andExpect(header().doesNotExist(HttpHeaders.ACCESS_CONTROL_ALLOW_CREDENTIALS));
|
||||
}
|
||||
|
||||
@Test
|
||||
void unauthenticatedAndInvalidTokenResponsesKeepCorsHeaders() throws Exception {
|
||||
mvc.perform(get("/api/users/me").header(HttpHeaders.ORIGIN, FRONTEND_ORIGIN))
|
||||
.andExpect(status().isUnauthorized())
|
||||
.andExpect(jsonPath("$.code").value("UNAUTHORIZED"))
|
||||
.andExpect(header().string(HttpHeaders.ACCESS_CONTROL_ALLOW_ORIGIN, FRONTEND_ORIGIN));
|
||||
|
||||
mvc.perform(get("/api/project-types")
|
||||
.header(HttpHeaders.ORIGIN, FRONTEND_ORIGIN)
|
||||
.header(HttpHeaders.AUTHORIZATION, "Bearer invalid-test-token"))
|
||||
.andExpect(status().isUnauthorized())
|
||||
.andExpect(jsonPath("$.code").value("UNAUTHORIZED"))
|
||||
.andExpect(header().string(HttpHeaders.ACCESS_CONTROL_ALLOW_ORIGIN, FRONTEND_ORIGIN));
|
||||
}
|
||||
|
||||
@Test
|
||||
void validationErrorsKeepCorsHeaders() throws Exception {
|
||||
mvc.perform(post("/api/auth/login")
|
||||
.header(HttpHeaders.ORIGIN, FRONTEND_ORIGIN)
|
||||
.contentType(MediaType.APPLICATION_JSON)
|
||||
.content("{}"))
|
||||
.andExpect(status().isBadRequest())
|
||||
.andExpect(jsonPath("$.code").value("BAD_REQUEST"))
|
||||
.andExpect(header().string(HttpHeaders.ACCESS_CONTROL_ALLOW_ORIGIN, FRONTEND_ORIGIN));
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@ValueSource(strings = {
|
||||
"https://untrusted.example",
|
||||
"http://dev-vibemake.robotquan.com",
|
||||
"https://vibemake.robotquan.com",
|
||||
"http://localhost:5173",
|
||||
"https://dev-vibemake.robotquan.com:8443",
|
||||
"https://dev-vibemake.robotquan.com.untrusted.example",
|
||||
"null"
|
||||
})
|
||||
void unlistedOriginsAreRejectedForPreflightAndActualRequests(String origin) throws Exception {
|
||||
mvc.perform(options("/api/workspace/projects")
|
||||
.header(HttpHeaders.ORIGIN, origin)
|
||||
.header(HttpHeaders.ACCESS_CONTROL_REQUEST_METHOD, "POST"))
|
||||
.andExpect(status().isForbidden())
|
||||
.andExpect(header().doesNotExist(HttpHeaders.ACCESS_CONTROL_ALLOW_ORIGIN));
|
||||
|
||||
mvc.perform(get("/api/project-types").header(HttpHeaders.ORIGIN, origin))
|
||||
.andExpect(status().isForbidden())
|
||||
.andExpect(header().doesNotExist(HttpHeaders.ACCESS_CONTROL_ALLOW_ORIGIN));
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@CsvSource({"TRACE, Authorization", "POST, X-Unapproved-Header"})
|
||||
void unsupportedMethodsOrHeadersAreRejected(String method, String requestedHeader)
|
||||
throws Exception {
|
||||
mvc.perform(options("/api/workspace/projects")
|
||||
.header(HttpHeaders.ORIGIN, FRONTEND_ORIGIN)
|
||||
.header(HttpHeaders.ACCESS_CONTROL_REQUEST_METHOD, method)
|
||||
.header(HttpHeaders.ACCESS_CONTROL_REQUEST_HEADERS, requestedHeader))
|
||||
.andExpect(status().isForbidden())
|
||||
.andExpect(header().doesNotExist(HttpHeaders.ACCESS_CONTROL_ALLOW_ORIGIN));
|
||||
}
|
||||
|
||||
@Test
|
||||
void requestsWithoutOriginKeepExistingAuthenticationBehavior() throws Exception {
|
||||
mvc.perform(get("/api/project-types"))
|
||||
.andExpect(status().isOk())
|
||||
.andExpect(jsonPath("$.code").value("OK"))
|
||||
.andExpect(header().doesNotExist(HttpHeaders.ACCESS_CONTROL_ALLOW_ORIGIN));
|
||||
|
||||
mvc.perform(get("/api/users/me"))
|
||||
.andExpect(status().isUnauthorized())
|
||||
.andExpect(jsonPath("$.code").value("UNAUTHORIZED"))
|
||||
.andExpect(header().doesNotExist(HttpHeaders.ACCESS_CONTROL_ALLOW_ORIGIN));
|
||||
}
|
||||
|
||||
@Test
|
||||
void corsDoesNotExposeNonApiRoutes() throws Exception {
|
||||
mvc.perform(get("/v3/api-docs").header(HttpHeaders.ORIGIN, FRONTEND_ORIGIN))
|
||||
.andExpect(status().isOk())
|
||||
.andExpect(header().doesNotExist(HttpHeaders.ACCESS_CONTROL_ALLOW_ORIGIN));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
package com.linkerbot.community.api;
|
||||
|
||||
import com.linkerbot.community.common.BusinessException;
|
||||
import com.linkerbot.community.service.fork.ForkDeadline;
|
||||
import com.linkerbot.community.service.fork.GitSnapshotCopier;
|
||||
import com.linkerbot.community.service.gitea.GiteaClient;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
/** 业务测试只替换仓库传输,真实 Git/LFS 由独立协议测试覆盖。 */
|
||||
final class FakeGitSnapshotCopier implements GitSnapshotCopier {
|
||||
private final FakeGiteaClient gitea;
|
||||
volatile boolean available = true;
|
||||
volatile Runnable beforePush;
|
||||
|
||||
FakeGitSnapshotCopier(FakeGiteaClient gitea) { this.gitea = gitea; }
|
||||
|
||||
@Override public void checkAvailable() {
|
||||
if (!available) throw new BusinessException(503, "FORK_UNAVAILABLE", "模拟 Git 依赖不可用");
|
||||
}
|
||||
|
||||
@Override
|
||||
public PreparedSnapshot prepare(GiteaClient.Repository source, String sha, String authorName, String authorCode, ForkDeadline deadline) {
|
||||
deadline.check();
|
||||
Map<String, byte[]> files = gitea.snapshot(source, sha);
|
||||
return new PreparedSnapshot() {
|
||||
@Override public String push(GiteaClient.Repository target) {
|
||||
if (beforePush != null) beforePush.run();
|
||||
deadline.beginWrite();
|
||||
List<GiteaClient.Change> changes = files.entrySet().stream()
|
||||
.map(file -> new GiteaClient.Change("create", file.getKey(), null, file.getValue())).toList();
|
||||
String commit = gitea.commit(target, changes, "Fork community snapshot", authorName, authorCode).commitSha();
|
||||
deadline.confirmWrite();
|
||||
return commit;
|
||||
}
|
||||
@Override public void close() {}
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -45,6 +45,15 @@ final class FakeGiteaClient implements GiteaClient {
|
||||
return repositories.get(repository.id());
|
||||
}
|
||||
|
||||
synchronized Map<String, byte[]> snapshot(Repository repository, String sha) {
|
||||
RepoData data = repo(repository);
|
||||
Map<String, byte[]> snapshot = data == null ? null : data.snapshots.get(sha);
|
||||
if (snapshot == null) throw new GiteaFailure(404, "NOT_FOUND", "发布版本不存在", false);
|
||||
Map<String, byte[]> copy = new TreeMap<>();
|
||||
snapshot.forEach((name, bytes) -> copy.put(name, bytes.clone()));
|
||||
return copy;
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized Repository createRepository(String name, boolean initialize) {
|
||||
if (failNextCreate) {
|
||||
|
||||
@@ -8,9 +8,13 @@ import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.linkerbot.community.security.CaptchaService;
|
||||
import com.linkerbot.community.service.config.CommunityProperties;
|
||||
import com.linkerbot.community.service.fork.ForkDeadline;
|
||||
import com.linkerbot.community.service.fork.GitCommandRunner;
|
||||
import com.linkerbot.community.service.gitea.GiteaClient;
|
||||
|
||||
import org.junit.jupiter.api.*;
|
||||
import org.junit.jupiter.api.condition.EnabledIfEnvironmentVariable;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc;
|
||||
import org.springframework.boot.test.autoconfigure.web.servlet.MockMvcPrint;
|
||||
@@ -26,6 +30,10 @@ import org.springframework.test.web.servlet.request.MockHttpServletRequestBuilde
|
||||
import java.net.URI;
|
||||
import java.net.http.*;
|
||||
import java.time.Duration;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.security.MessageDigest;
|
||||
import java.util.*;
|
||||
|
||||
@SpringBootTest(
|
||||
@@ -34,6 +42,8 @@ import java.util.*;
|
||||
"spring.datasource.username=sa",
|
||||
"spring.datasource.password=",
|
||||
"spring.datasource.driver-class-name=org.h2.Driver",
|
||||
"community.ai.enabled=false",
|
||||
"community.oss.enabled=false",
|
||||
"logging.level.root=INFO",
|
||||
"logging.level.org.springframework=WARN"
|
||||
})
|
||||
@@ -60,6 +70,9 @@ class GiteaLiveTest {
|
||||
@Autowired ObjectMapper json;
|
||||
@Autowired JdbcTemplate jdbc;
|
||||
@Autowired CommunityProperties config;
|
||||
@Autowired GitCommandRunner nativeGit;
|
||||
@Autowired GiteaClient remote;
|
||||
@TempDir Path gitFixtures;
|
||||
|
||||
@BeforeEach
|
||||
void schema() throws Exception {
|
||||
@@ -128,6 +141,122 @@ class GiteaLiveTest {
|
||||
.asText();
|
||||
}
|
||||
|
||||
@Test
|
||||
void realSnapshotForkPreservesLargeCadGitlinksSymlinksAndLfsWithoutUnpublishedHistory() throws Exception {
|
||||
String owner = login("live-snapshot-owner@example.invalid");
|
||||
String other = login("live-snapshot-fork@example.invalid");
|
||||
String source = perform(body(auth(post("/api/workspace/projects"), owner),
|
||||
Map.of("name", "HyperDog OpenFish 类型快照联调", "projectType", "GENERAL")), 200).path("projectCode").asText();
|
||||
String original = perform(auth(get("/api/workspace/projects/" + source + "/tree"), owner), 200).path("commitSha").asText();
|
||||
Path staging = Files.createDirectory(gitFixtures.resolve("fixture.git"));
|
||||
String base = config.getGitea().getBaseUrl().replaceAll("/+$", "");
|
||||
String url = base + "/" + config.getGitea().getOwner() + "/p_" + source + ".git";
|
||||
String basic = Base64.getEncoder().encodeToString((remote.authenticatedUsername() + ":" + config.getGitea().getToken())
|
||||
.getBytes(StandardCharsets.UTF_8));
|
||||
Map<String, String> settings = new LinkedHashMap<>();
|
||||
settings.put("http." + base + "/.extraHeader", "Authorization: Basic " + basic);
|
||||
settings.put("lfs.url", url + "/info/lfs");
|
||||
settings.put("lfs.pushurl", url + "/info/lfs");
|
||||
settings.put("lfs.basictransfersonly", "true");
|
||||
fixtureGit(staging, settings, "init", "--bare", "--quiet");
|
||||
fixtureGit(staging, settings, "remote", "add", "origin", url);
|
||||
fixtureGit(staging, settings, "fetch", "--quiet", "origin", "refs/heads/main");
|
||||
fixtureGit(staging, settings, "read-tree", original);
|
||||
String ordinary = fixtureBlob(staging, settings, "published".getBytes(StandardCharsets.UTF_8));
|
||||
String cad = fixtureBlob(staging, settings, new byte[13 * 1024 * 1024]);
|
||||
String link = fixtureBlob(staging, settings, "README.md".getBytes(StandardCharsets.UTF_8));
|
||||
byte[] lfsBytes = new byte[256 * 1024];
|
||||
new Random(42).nextBytes(lfsBytes);
|
||||
String oid = HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(lfsBytes));
|
||||
Path object = staging.resolve("lfs/objects/" + oid.substring(0, 2) + "/" + oid.substring(2, 4) + "/" + oid);
|
||||
Files.createDirectories(object.getParent());
|
||||
Files.write(object, lfsBytes);
|
||||
String pointer = fixtureBlob(staging, settings, ("version https://git-lfs.github.com/spec/v1\noid sha256:" + oid
|
||||
+ "\nsize " + lfsBytes.length + "\n").getBytes(StandardCharsets.UTF_8));
|
||||
String attributes = fixtureBlob(staging, settings, "model.bin filter=lfs diff=lfs merge=lfs -text\n".getBytes(StandardCharsets.UTF_8));
|
||||
Map<String, String> files = new LinkedHashMap<>();
|
||||
files.put("README.md", "100644," + ordinary);
|
||||
files.put("说明/示例.txt", "100644," + ordinary);
|
||||
files.put("run.sh", "100755," + ordinary);
|
||||
files.put("SOLIDWORKS_FILES/Assembly_#1.SLDASM", "100644," + cad);
|
||||
files.put("readme-link", "120000," + link);
|
||||
files.put("model.bin", "100644," + pointer);
|
||||
files.put(".gitattributes", "100644," + attributes);
|
||||
for (String path : List.of("uros/drive_base", "uros/micro-ROS-Agent", "uros/micro_ros_msgs", "uros_pkg/src/micro_ros_setup"))
|
||||
files.put(path, "160000," + "a".repeat(40));
|
||||
for (Map.Entry<String, String> file : files.entrySet()) {
|
||||
String[] modeAndHash = file.getValue().split(",");
|
||||
fixtureGit(staging, settings, "update-index", "--add", "--cacheinfo", modeAndHash[0], modeAndHash[1], file.getKey());
|
||||
}
|
||||
String tree = fixtureGit(staging, settings, "write-tree");
|
||||
String publication = fixtureGit(staging, settings, "commit-tree", tree, "-p", original, "-m", "Special Git/LFS fixture");
|
||||
fixtureGit(staging, settings, "update-ref", "refs/heads/fixture", publication);
|
||||
fixtureGit(staging, settings, "lfs", "push", "--all", "origin", "refs/heads/fixture");
|
||||
fixtureGit(staging, settings, "push", "--quiet", "origin", publication + ":refs/heads/main");
|
||||
perform(body(auth(put("/api/workspace/projects/" + source + "/publication"), owner), Map.of("commitSha", publication)), 200);
|
||||
perform(body(auth(post("/api/workspace/projects/" + source + "/commits"), owner), Map.of(
|
||||
"expectedCommitSha", publication, "message", "unpublished change", "changes", List.of(Map.of(
|
||||
"action", "UPDATE_FILE", "path", "README.md", "content", "private unpublished")))), 200);
|
||||
Set<String> copies = new HashSet<>();
|
||||
for (String token : List.of(owner, other)) {
|
||||
JsonNode fork = perform(body(auth(post("/api/community/projects/" + source + "/forks"), token),
|
||||
Map.of("name", "快照联调副本", "publishedCommitSha", publication)), 200);
|
||||
String code = fork.path("projectCode").asText();
|
||||
assertThat(copies.add(code)).isTrue();
|
||||
assertThat(fork.path("repoStatus").asText()).isEqualTo("READY");
|
||||
assertThat(fork.path("sourceCommitSha").asText()).isEqualTo(publication);
|
||||
GiteaClient.Repository repository = new GiteaClient.Repository(0, config.getGitea().getOwner(), "p_" + code);
|
||||
JsonNode metadata = readRepository("p_" + code, "");
|
||||
assertThat(metadata.path("empty").asBoolean()).as("Gitea metadata empty; defaultBranch=%s", metadata.path("default_branch").asText()).isFalse();
|
||||
String head = readRepository("p_" + code, "/branches/main").path("commit").path("id").asText();
|
||||
JsonNode commit = readRepository("p_" + code, "/git/commits/" + head);
|
||||
assertThat(commit.path("parents").isArray()).isTrue();
|
||||
assertThat(commit.path("parents").isEmpty()).isTrue();
|
||||
String copyRemote = "copy" + copies.size();
|
||||
fixtureGit(staging, settings, "remote", "add", copyRemote, base + "/" + config.getGitea().getOwner() + "/p_" + code + ".git");
|
||||
fixtureGit(staging, settings, "fetch", "--quiet", "--no-tags", copyRemote, "refs/heads/main");
|
||||
assertThat(fixtureGit(staging, settings, "rev-parse", head + "^{tree}")).isEqualTo(tree);
|
||||
assertThat(remote.history(repository, 1, 20).total()).isEqualTo(1);
|
||||
JsonNode entries = readRepository("p_" + code, "/git/trees/" + head + "?recursive=true");
|
||||
assertThat(entries.path("tree").findValuesAsText("mode")).contains("100755", "120000", "160000");
|
||||
assertThat(perform(auth(get("/api/workspace/projects/" + code + "/files").param("path", "README.md"), token), 200)
|
||||
.path("content").asText()).isEqualTo("published");
|
||||
mvc.perform(auth(get("/api/workspace/projects/" + code + "/files")
|
||||
.param("path", "SOLIDWORKS_FILES/Assembly_#1.SLDASM").param("download", "true"), token)).andExpect(status().is(413));
|
||||
}
|
||||
}
|
||||
|
||||
private String fixtureGit(Path staging, Map<String, String> settings, String... args) {
|
||||
GitCommandRunner.Result result = nativeGit.run(staging, new ForkDeadline(Duration.ofMinutes(2)), "live-fixture", settings,
|
||||
Map.of("GIT_AUTHOR_NAME", "Fixture", "GIT_AUTHOR_EMAIL", "fixture@users.community.invalid",
|
||||
"GIT_COMMITTER_NAME", "Fixture", "GIT_COMMITTER_EMAIL", "fixture@users.community.invalid"), args);
|
||||
assertThat(result.exitCode()).as("live Git fixture command completed").isZero();
|
||||
return result.output();
|
||||
}
|
||||
|
||||
private String fixtureBlob(Path staging, Map<String, String> settings, byte[] bytes) throws Exception {
|
||||
Path input = Files.createTempFile(gitFixtures, "blob-", ".bin");
|
||||
Files.write(input, bytes);
|
||||
return fixtureGit(staging, settings, "hash-object", "-w", input.toString());
|
||||
}
|
||||
|
||||
private JsonNode readRepository(String repository, String suffix) throws Exception {
|
||||
String url = config.getGitea().getBaseUrl().replaceAll("/+$", "") + "/api/v1/repos/"
|
||||
+ config.getGitea().getOwner() + "/" + repository + suffix;
|
||||
HttpRequest request = HttpRequest.newBuilder(URI.create(url)).timeout(Duration.ofSeconds(30))
|
||||
.header("Authorization", "token " + config.getGitea().getToken()).GET().build();
|
||||
HttpResponse<byte[]> response = HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofByteArray());
|
||||
if (response.statusCode() != 200) {
|
||||
String diagnostic = new String(response.body(), StandardCharsets.UTF_8)
|
||||
.replace(config.getGitea().getToken(), "[redacted]")
|
||||
.replaceAll("(?im)(Authorization|Cookie|Set-Cookie):[^\\r\\n]*", "$1: [redacted]");
|
||||
if (diagnostic.length() > 18000) diagnostic = diagnostic.substring(0, 18000);
|
||||
assertThat(response.statusCode()).withFailMessage("Fixture Gitea response: %s", diagnostic).isEqualTo(200);
|
||||
}
|
||||
assertThat(response.statusCode()).isEqualTo(200);
|
||||
return json.readTree(response.body());
|
||||
}
|
||||
|
||||
@Test
|
||||
void realOrganizationRepositoryCommitPublishUploadAndSnapshotFork() throws Exception {
|
||||
String owner = login("live-owner@example.invalid"),
|
||||
@@ -228,6 +357,8 @@ class GiteaLiveTest {
|
||||
c.path("commitSha").asText())),
|
||||
200);
|
||||
String copy = fork.path("projectCode").asText();
|
||||
readRepository("p_" + copy, "/contents/%E6%BA%90%E7%A0%81/hello%20world.txt?ref="
|
||||
+ readRepository("p_" + copy, "/branches/main").path("commit").path("id").asText());
|
||||
assertThat(
|
||||
perform(
|
||||
auth(
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package com.linkerbot.community.api;
|
||||
|
||||
import static org.assertj.core.api.Assertions.*;
|
||||
import static org.hamcrest.Matchers.containsString;
|
||||
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.*;
|
||||
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.*;
|
||||
|
||||
@@ -10,6 +11,7 @@ import com.linkerbot.community.security.CaptchaService;
|
||||
import com.linkerbot.community.service.config.CommunityProperties;
|
||||
import com.linkerbot.community.service.config.OssProperties;
|
||||
import com.linkerbot.community.service.config.UrdfProperties;
|
||||
import com.linkerbot.community.common.BusinessException;
|
||||
|
||||
import org.junit.jupiter.api.*;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
@@ -42,6 +44,8 @@ import java.util.concurrent.*;
|
||||
class WorkflowTest {
|
||||
@TestConfiguration
|
||||
static class Config {
|
||||
@Bean @Primary FakeGitSnapshotCopier fakeSnapshots(FakeGiteaClient gitea) { return new FakeGitSnapshotCopier(gitea); }
|
||||
|
||||
@Bean
|
||||
@Primary
|
||||
FakeOssObjectStore fakeOss() { return new FakeOssObjectStore(); }
|
||||
@@ -71,11 +75,14 @@ class WorkflowTest {
|
||||
@Autowired OssProperties oss;
|
||||
@Autowired FakeOssObjectStore ossObjects;
|
||||
@Autowired UrdfProperties urdfLimits;
|
||||
@Autowired FakeGitSnapshotCopier snapshots;
|
||||
|
||||
@BeforeEach
|
||||
void reset() throws Exception {
|
||||
SchemaSupport.reset(jdbc);
|
||||
gitea.reset();
|
||||
snapshots.available = true;
|
||||
snapshots.beforePush = null;
|
||||
oss.setEnabled(false);
|
||||
ossObjects.reset();
|
||||
urdfLimits.setMaxFileBytes(52428800);
|
||||
@@ -182,6 +189,54 @@ class WorkflowTest {
|
||||
200);
|
||||
}
|
||||
|
||||
@Test
|
||||
void crossOriginLoginDownloadsAndBusinessErrorsKeepCorsHeaders() throws Exception {
|
||||
String origin = "https://dev-vibemake.robotquan.com";
|
||||
String captchaCode = perform(get("/api/auth/captcha").header("Origin", origin), 200)
|
||||
.path("data").path("captchaCode").asText();
|
||||
MvcResult loginResult = mvc.perform(body(post("/api/auth/login").header("Origin", origin),
|
||||
Map.of("email", "cors-owner@example.invalid", "captchaCode", captchaCode,
|
||||
"captchaAnswer", "ABCDE")))
|
||||
.andExpect(status().isOk())
|
||||
.andExpect(header().string("Access-Control-Allow-Origin", origin))
|
||||
.andExpect(jsonPath("$.code").value("OK"))
|
||||
.andReturn();
|
||||
String owner = json.readTree(loginResult.getResponse().getContentAsString())
|
||||
.path("data").path("token").asText();
|
||||
String code = create(owner, "跨域下载项目");
|
||||
|
||||
mvc.perform(auth(get("/api/workspace/projects/" + code + "/files"), owner)
|
||||
.header("Origin", origin)
|
||||
.param("path", "README.md").param("download", "true"))
|
||||
.andExpect(status().isOk())
|
||||
.andExpect(content().contentType(MediaType.APPLICATION_OCTET_STREAM))
|
||||
.andExpect(header().string("Content-Disposition", containsString("attachment")))
|
||||
.andExpect(header().string("Access-Control-Allow-Origin", origin))
|
||||
.andExpect(header().string("Access-Control-Expose-Headers", containsString("Content-Disposition")))
|
||||
.andExpect(header().string("Access-Control-Expose-Headers", containsString("X-Request-Id")));
|
||||
|
||||
String other = login("cors-other@example.invalid");
|
||||
mvc.perform(auth(get("/api/workspace/projects/" + code), other).header("Origin", origin))
|
||||
.andExpect(status().isNotFound())
|
||||
.andExpect(jsonPath("$.code").value("NOT_FOUND"))
|
||||
.andExpect(header().string("Access-Control-Allow-Origin", origin));
|
||||
|
||||
mvc.perform(auth(get("/api/workspace/projects/missing-project"), owner).header("Origin", origin))
|
||||
.andExpect(status().isNotFound())
|
||||
.andExpect(jsonPath("$.code").value("NOT_FOUND"))
|
||||
.andExpect(header().string("Access-Control-Allow-Origin", origin));
|
||||
|
||||
jdbc.update("UPDATE sys_user SET status = 'DISABLED' WHERE email = ?", "cors-other@example.invalid");
|
||||
String disabledCaptchaCode = perform(get("/api/auth/captcha").header("Origin", origin), 200)
|
||||
.path("data").path("captchaCode").asText();
|
||||
mvc.perform(body(post("/api/auth/login").header("Origin", origin),
|
||||
Map.of("email", "cors-other@example.invalid", "captchaCode", disabledCaptchaCode,
|
||||
"captchaAnswer", "ABCDE")))
|
||||
.andExpect(status().isForbidden())
|
||||
.andExpect(jsonPath("$.code").value("ACCOUNT_DISABLED"))
|
||||
.andExpect(header().string("Access-Control-Allow-Origin", origin));
|
||||
}
|
||||
|
||||
@Test
|
||||
void roboticArmTypeSupportsDiscoveryCreationEditingAndCommunityFiltering() throws Exception {
|
||||
JsonNode types = perform(get("/api/project-types"), 200).path("data");
|
||||
@@ -583,6 +638,43 @@ class WorkflowTest {
|
||||
"SYS_USER", "PROJECT", "PROJECT_COMMENT", "PROJECT_LIKE");
|
||||
}
|
||||
|
||||
@Test
|
||||
void unavailableForkRejectsBeforeCreatingAnyProjectRecord() throws Exception {
|
||||
String token = login("fork-runtime@example.invalid");
|
||||
String source = create(token, "来源");
|
||||
String sha = head(token, source);
|
||||
publish(token, source, sha);
|
||||
snapshots.available = false;
|
||||
JsonNode error = perform(body(auth(post("/api/community/projects/" + source + "/forks"), token),
|
||||
Map.of("name", "副本", "publishedCommitSha", sha)), 503);
|
||||
assertThat(error.path("code").asText()).isEqualTo("FORK_UNAVAILABLE");
|
||||
assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM project", Integer.class)).isEqualTo(1);
|
||||
}
|
||||
|
||||
@Test
|
||||
void failedAndUncertainGitCopiesPreserveProjectCodeAndRepositoryAssociation() throws Exception {
|
||||
String token = login("fork-status@example.invalid");
|
||||
String source = create(token, "来源");
|
||||
String sha = head(token, source);
|
||||
publish(token, source, sha);
|
||||
snapshots.beforePush = () -> { throw new BusinessException(502, "FORK_COPY_FAILED", "模拟复制失败"); };
|
||||
JsonNode failed = perform(body(auth(post("/api/community/projects/" + source + "/forks"), token),
|
||||
Map.of("name", "失败副本", "publishedCommitSha", sha)), 502);
|
||||
String failedCode = failed.path("data").path("projectCode").asText();
|
||||
assertThat(failed.path("code").asText()).isEqualTo("FORK_COPY_FAILED");
|
||||
assertThat(jdbc.queryForObject("SELECT repo_status FROM project WHERE project_code = ?", String.class, failedCode)).isEqualTo("FAILED");
|
||||
assertThat(jdbc.queryForObject("SELECT gitea_repo_id FROM project WHERE project_code = ?", Long.class, failedCode)).isPositive();
|
||||
snapshots.beforePush = null;
|
||||
gitea.timeoutNextCommit = true;
|
||||
JsonNode unknown = perform(body(auth(post("/api/community/projects/" + source + "/forks"), token),
|
||||
Map.of("name", "待核对副本", "publishedCommitSha", sha)), 504);
|
||||
String unknownCode = unknown.path("data").path("projectCode").asText();
|
||||
assertThat(unknown.path("code").asText()).isEqualTo("REMOTE_RESULT_UNKNOWN");
|
||||
assertThat(jdbc.queryForObject("SELECT repo_status FROM project WHERE project_code = ?", String.class, unknownCode)).isEqualTo("CREATING");
|
||||
assertThat(jdbc.queryForObject("SELECT gitea_repo_id FROM project WHERE project_code = ?", Long.class, unknownCode)).isPositive();
|
||||
assertThat(jdbc.queryForObject("SELECT repo_status FROM project WHERE project_code = ?", String.class, source)).isEqualTo("READY");
|
||||
}
|
||||
|
||||
@Test
|
||||
void publishPinsFilesAndForkCopiesOnlyPublishedSnapshot() throws Exception {
|
||||
String owner = login("owner@example.com");
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
package com.linkerbot.community.security;
|
||||
|
||||
import lombok.Getter;
|
||||
import lombok.Setter;
|
||||
|
||||
import org.springframework.boot.context.properties.ConfigurationProperties;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
@Getter
|
||||
@Setter
|
||||
@Component
|
||||
@ConfigurationProperties(prefix = "community.cors")
|
||||
public class CorsProperties {
|
||||
private List<String> allowedOrigins = List.of();
|
||||
}
|
||||
+32
-2
@@ -3,8 +3,11 @@ package com.linkerbot.community.security;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.linkerbot.community.common.*;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.boot.web.servlet.FilterRegistrationBean;
|
||||
import org.springframework.context.annotation.*;
|
||||
import org.springframework.http.HttpHeaders;
|
||||
import org.springframework.http.HttpMethod;
|
||||
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
|
||||
import org.springframework.security.config.http.SessionCreationPolicy;
|
||||
@@ -12,9 +15,33 @@ import org.springframework.security.web.SecurityFilterChain;
|
||||
import org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter;
|
||||
import org.springframework.security.web.context.RequestAttributeSecurityContextRepository;
|
||||
import org.springframework.security.web.context.SecurityContextRepository;
|
||||
import org.springframework.web.cors.CorsConfiguration;
|
||||
import org.springframework.web.cors.UrlBasedCorsConfigurationSource;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
@Configuration
|
||||
public class SecurityConfiguration {
|
||||
private static final Logger log = LoggerFactory.getLogger(SecurityConfiguration.class);
|
||||
|
||||
@Bean
|
||||
public UrlBasedCorsConfigurationSource corsConfigurationSource(CorsProperties properties) {
|
||||
CorsConfiguration configuration = new CorsConfiguration();
|
||||
configuration.setAllowedOrigins(properties.getAllowedOrigins());
|
||||
configuration.setAllowedMethods(
|
||||
List.of("GET", "HEAD", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"));
|
||||
configuration.setAllowedHeaders(
|
||||
List.of(HttpHeaders.AUTHORIZATION, HttpHeaders.CONTENT_TYPE, HttpHeaders.ACCEPT));
|
||||
configuration.setExposedHeaders(List.of("X-Request-Id", HttpHeaders.CONTENT_DISPOSITION));
|
||||
configuration.setAllowCredentials(false);
|
||||
configuration.setMaxAge(3600L);
|
||||
|
||||
UrlBasedCorsConfigurationSource source = new UrlBasedCorsConfigurationSource();
|
||||
source.registerCorsConfiguration("/api/**", configuration);
|
||||
log.info("API 跨域配置已加载 path=/api/** allowedOrigins={}", properties.getAllowedOrigins());
|
||||
return source;
|
||||
}
|
||||
|
||||
@Bean
|
||||
public SecurityContextRepository requestSecurityContextRepository() {
|
||||
// 仅在本次请求及其异步分派内保留身份,不创建 HttpSession。
|
||||
@@ -33,8 +60,11 @@ public class SecurityConfiguration {
|
||||
@Bean
|
||||
public SecurityFilterChain securityFilterChain(
|
||||
HttpSecurity http, BearerTokenFilter tokenFilter, ObjectMapper json,
|
||||
SecurityContextRepository contextRepository) throws Exception {
|
||||
http.csrf(csrf -> csrf.disable())
|
||||
SecurityContextRepository contextRepository,
|
||||
UrlBasedCorsConfigurationSource corsConfigurationSource) throws Exception {
|
||||
// 在 Token 鉴权前处理预检,并让鉴权失败响应同样携带跨域头。
|
||||
http.cors(cors -> cors.configurationSource(corsConfigurationSource))
|
||||
.csrf(csrf -> csrf.disable())
|
||||
.sessionManagement(
|
||||
session -> session.sessionCreationPolicy(SessionCreationPolicy.STATELESS))
|
||||
.securityContext(context -> context.securityContextRepository(contextRepository))
|
||||
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
package com.linkerbot.community.service.config;
|
||||
|
||||
import jakarta.validation.constraints.AssertTrue;
|
||||
import jakarta.validation.constraints.Max;
|
||||
import jakarta.validation.constraints.Min;
|
||||
import jakarta.validation.constraints.NotBlank;
|
||||
import jakarta.validation.constraints.Positive;
|
||||
import java.nio.file.Path;
|
||||
import java.time.Duration;
|
||||
import lombok.Getter;
|
||||
import lombok.Setter;
|
||||
import org.springframework.boot.context.properties.ConfigurationProperties;
|
||||
import org.springframework.stereotype.Component;
|
||||
import org.springframework.validation.annotation.Validated;
|
||||
|
||||
@Getter
|
||||
@Setter
|
||||
@Validated
|
||||
@Component
|
||||
@ConfigurationProperties(prefix = "community.fork")
|
||||
public class ForkProperties {
|
||||
@NotBlank private String gitExecutable = "git";
|
||||
private Duration timeout = Duration.ofMinutes(5);
|
||||
@Min(1) @Max(32) private int maxConcurrent = 2;
|
||||
@Positive private long maxTemporaryBytes = 5L * 1024 * 1024 * 1024;
|
||||
@NotBlank private String temporaryDirectory = Path.of(System.getProperty("java.io.tmpdir"), "community-fork").toString();
|
||||
|
||||
@AssertTrue(message = "Fork 超时必须在 100 毫秒至 1 小时之间")
|
||||
public boolean isTimeoutValid() {
|
||||
return timeout != null && timeout.compareTo(Duration.ofMillis(100)) >= 0
|
||||
&& timeout.compareTo(Duration.ofHours(1)) <= 0;
|
||||
}
|
||||
}
|
||||
+268
@@ -0,0 +1,268 @@
|
||||
package com.linkerbot.community.service.fork;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.linkerbot.community.common.BusinessException;
|
||||
import com.linkerbot.community.common.FilePaths;
|
||||
import com.linkerbot.community.service.config.CommunityProperties;
|
||||
import com.linkerbot.community.service.config.ForkProperties;
|
||||
import com.linkerbot.community.service.gitea.GiteaClient;
|
||||
import com.linkerbot.community.service.gitea.GiteaClient.Repository;
|
||||
import java.io.IOException;
|
||||
import java.net.URI;
|
||||
import java.net.URLEncoder;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.*;
|
||||
import java.nio.file.attribute.BasicFileAttributes;
|
||||
import java.nio.file.attribute.PosixFilePermissions;
|
||||
import java.time.Duration;
|
||||
import java.util.*;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
/** 使用原始 Git tree 生成根提交,不检出文件、不递归子模块、不复制上游历史。 */
|
||||
@Component
|
||||
public class CliGitSnapshotCopier implements GitSnapshotCopier {
|
||||
private static final Logger log = LoggerFactory.getLogger(CliGitSnapshotCopier.class);
|
||||
private final ForkProperties properties;
|
||||
private final CommunityProperties community;
|
||||
private final GitCommandRunner commands;
|
||||
private final GiteaClient gitea;
|
||||
private final ObjectMapper json;
|
||||
|
||||
public CliGitSnapshotCopier(ForkProperties properties, CommunityProperties community, GitCommandRunner commands,
|
||||
GiteaClient gitea, ObjectMapper json) {
|
||||
this.properties = properties;
|
||||
this.community = community;
|
||||
this.commands = commands;
|
||||
this.gitea = gitea;
|
||||
this.json = json;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void checkAvailable() {
|
||||
try {
|
||||
trustedBase();
|
||||
if (community.getGitea().getToken().isBlank()) throw unavailable();
|
||||
ForkDeadline deadline = new ForkDeadline(Duration.ofSeconds(5));
|
||||
if (commands.run(null, deadline, "git-runtime", Map.of(), Map.of(), "--version").exitCode() != 0
|
||||
|| commands.run(null, deadline, "lfs-runtime", Map.of(), Map.of(), "lfs", "version").exitCode() != 0)
|
||||
throw unavailable();
|
||||
Path root = Files.createDirectories(Path.of(properties.getTemporaryDirectory()));
|
||||
Path probe = Files.createTempFile(root, ".fork-probe-", ".tmp");
|
||||
Files.delete(probe);
|
||||
} catch (IOException | RuntimeException error) {
|
||||
throw unavailable();
|
||||
}
|
||||
}
|
||||
|
||||
private BusinessException unavailable() {
|
||||
return new BusinessException(503, "FORK_UNAVAILABLE", "Fork 不可用,请检查 Git、git-lfs、Gitea 配置和临时目录");
|
||||
}
|
||||
|
||||
protected URI trustedBase() {
|
||||
URI base = URI.create(community.getGitea().getBaseUrl().replaceAll("/+$", ""));
|
||||
if (!("http".equals(base.getScheme()) || "https".equals(base.getScheme())) || base.getHost() == null
|
||||
|| base.getRawUserInfo() != null || base.getRawQuery() != null || base.getRawFragment() != null)
|
||||
throw unavailable();
|
||||
return base;
|
||||
}
|
||||
|
||||
protected String repositoryUrl(Repository repository) {
|
||||
String owner = URLEncoder.encode(repository.owner(), StandardCharsets.UTF_8).replace("+", "%20");
|
||||
String name = URLEncoder.encode(repository.name(), StandardCharsets.UTF_8).replace("+", "%20");
|
||||
return trustedBase() + "/" + owner + "/" + name + ".git";
|
||||
}
|
||||
|
||||
protected Map<String, String> configuration(String lfsEndpoint) {
|
||||
String username = gitea.authenticatedUsername();
|
||||
if (username == null || username.isBlank()) throw unavailable();
|
||||
String basic = Base64.getEncoder().encodeToString((username + ":" + community.getGitea().getToken()).getBytes(StandardCharsets.UTF_8));
|
||||
Map<String, String> settings = new LinkedHashMap<>();
|
||||
settings.put("http." + trustedBase() + "/.extraHeader", "Authorization: Basic " + basic);
|
||||
settings.put("lfs.url", lfsEndpoint);
|
||||
settings.put("lfs.pushurl", lfsEndpoint);
|
||||
settings.put("lfs.fetchinclude", "");
|
||||
settings.put("lfs.fetchexclude", "");
|
||||
settings.put("lfs.fetchrecentalways", "false");
|
||||
settings.put("lfs.fetchrecentrefsdays", "0");
|
||||
settings.put("lfs.fetchrecentcommitsdays", "0");
|
||||
settings.put("lfs.basictransfersonly", "true");
|
||||
settings.put("lfs.standalonetransferagent", "");
|
||||
settings.put("lfs.transfer.maxretries", "1");
|
||||
settings.put("lfs.transfer.maxverifies", "1");
|
||||
settings.put("lfs.transfer.maxretrydelay", "0");
|
||||
settings.put("lfs.concurrenttransfers", "2");
|
||||
return settings;
|
||||
}
|
||||
|
||||
@Override
|
||||
public PreparedSnapshot prepare(Repository source, String publishedSha, String authorName, String authorCode,
|
||||
ForkDeadline deadline) {
|
||||
String sha = FilePaths.sha(publishedSha);
|
||||
Path workspace = null;
|
||||
Snapshot snapshot = null;
|
||||
try {
|
||||
deadline.check();
|
||||
workspace = Files.createTempDirectory(Path.of(properties.getTemporaryDirectory()), "fork-" + deadline.projectCode() + "-");
|
||||
Files.setPosixFilePermissions(workspace, PosixFilePermissions.fromString("rwx------"));
|
||||
String url = repositoryUrl(source);
|
||||
Map<String, String> settings = configuration(url + "/info/lfs");
|
||||
snapshot = new Snapshot(workspace, deadline, settings);
|
||||
snapshot.require("init", "init", "--bare", "--quiet");
|
||||
blockLfsLogs(workspace.resolve("lfs"));
|
||||
snapshot.require("source-remote", "remote", "add", "source", url);
|
||||
GitCommandRunner.Result fetched = snapshot.run("fetch-sha", "fetch", "--quiet", "--depth=1", "--no-tags", "source", sha);
|
||||
if (fetched.exitCode() != 0) {
|
||||
boolean shallow = snapshot.require("shallow-check", "rev-parse", "--is-shallow-repository").equals("true");
|
||||
if (shallow) snapshot.require("fetch-main", "fetch", "--quiet", "--no-tags", "--unshallow", "source", "refs/heads/main");
|
||||
else snapshot.require("fetch-main", "fetch", "--quiet", "--no-tags", "source", "refs/heads/main");
|
||||
}
|
||||
snapshot.require("source-check", "cat-file", "-e", sha + "^{commit}");
|
||||
snapshot.tree = FilePaths.sha(snapshot.require("source-tree", "rev-parse", sha + "^{tree}"));
|
||||
String author = authorName == null ? "Community user" : authorName.replaceAll("[\\r\\n<>\\x00]", " ").trim();
|
||||
if (author.isEmpty()) author = "Community user";
|
||||
Map<String, String> identity = Map.of("GIT_AUTHOR_NAME", author, "GIT_COMMITTER_NAME", author,
|
||||
"GIT_AUTHOR_EMAIL", authorCode + "@users.community.invalid", "GIT_COMMITTER_EMAIL", authorCode + "@users.community.invalid");
|
||||
GitCommandRunner.Result commit = commands.run(workspace, deadline, "root-commit", settings, identity,
|
||||
"commit-tree", snapshot.tree, "-m", "Fork community snapshot");
|
||||
if (commit.exitCode() != 0) throw GitCommandRunner.failed(deadline);
|
||||
snapshot.commit = FilePaths.sha(commit.output());
|
||||
snapshot.require("snapshot-ref", "update-ref", "refs/heads/snapshot", snapshot.commit);
|
||||
snapshot.require("snapshot-head", "symbolic-ref", "HEAD", "refs/heads/snapshot");
|
||||
String parents = snapshot.require("root-check", "rev-list", "--parents", "-n", "1", snapshot.commit);
|
||||
if (!parents.equals(snapshot.commit)) throw GitCommandRunner.failed(deadline);
|
||||
JsonNode listing = json.readTree(snapshot.require("lfs-list", "lfs", "ls-files", "--json", sha));
|
||||
JsonNode objects = listing == null ? null : listing.get("files");
|
||||
if (objects == null || !(objects.isArray() || objects.isNull())) throw GitCommandRunner.failed(deadline);
|
||||
snapshot.hasLfs = objects.isArray() && !objects.isEmpty();
|
||||
if (snapshot.hasLfs) {
|
||||
snapshot.protectLfs();
|
||||
snapshot.require("lfs-fetch", "lfs", "fetch", "--include=", "--exclude=", "source", sha);
|
||||
snapshot.require("lfs-fsck", "lfs", "fsck", "--objects", sha);
|
||||
}
|
||||
log.info("Fork 快照准备完成 projectCode={} sourceRepository={} sourceCommitSha={} snapshotCommitSha={} hasLfs={}",
|
||||
deadline.projectCode(), source.name(), sha, snapshot.commit, snapshot.hasLfs);
|
||||
return snapshot;
|
||||
} catch (IOException error) {
|
||||
if (snapshot != null) snapshot.close();
|
||||
else if (workspace != null) clean(workspace, deadline.projectCode());
|
||||
throw GitCommandRunner.failed(deadline);
|
||||
} catch (RuntimeException error) {
|
||||
if (snapshot != null) snapshot.close();
|
||||
else if (workspace != null) clean(workspace, deadline.projectCode());
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
private final class Snapshot implements PreparedSnapshot {
|
||||
private final Path workspace;
|
||||
private final ForkDeadline deadline;
|
||||
private Map<String, String> settings;
|
||||
private String tree;
|
||||
private String commit;
|
||||
private boolean hasLfs;
|
||||
private LfsEndpointGuard lfsGuard;
|
||||
|
||||
private Snapshot(Path workspace, ForkDeadline deadline, Map<String, String> settings) {
|
||||
this.workspace = workspace;
|
||||
this.deadline = deadline;
|
||||
this.settings = settings;
|
||||
}
|
||||
|
||||
private GitCommandRunner.Result run(String stage, String... args) {
|
||||
return commands.run(workspace, deadline, stage, settings, Map.of(), args);
|
||||
}
|
||||
|
||||
private String require(String stage, String... args) {
|
||||
GitCommandRunner.Result result = run(stage, args);
|
||||
if (result.exitCode() != 0) {
|
||||
if (result.rejected()) deadline.confirmWrite();
|
||||
throw GitCommandRunner.failed(deadline);
|
||||
}
|
||||
return result.output();
|
||||
}
|
||||
|
||||
private void protectLfs() {
|
||||
if (lfsGuard != null) lfsGuard.close();
|
||||
try {
|
||||
String header = settings.get("http." + trustedBase() + "/.extraHeader");
|
||||
lfsGuard = new LfsEndpointGuard(trustedBase(), URI.create(settings.get("lfs.url")),
|
||||
header.substring("Authorization: ".length()), deadline, json);
|
||||
settings.put("lfs.url", lfsGuard.endpoint());
|
||||
settings.put("lfs.pushurl", lfsGuard.endpoint());
|
||||
settings.put("http." + lfsGuard.endpoint() + "/.extraHeader", "Authorization: " + lfsGuard.localAuthorization());
|
||||
} catch (IOException error) { throw GitCommandRunner.failed(deadline); }
|
||||
}
|
||||
|
||||
@Override
|
||||
public String push(Repository target) {
|
||||
String url = repositoryUrl(target);
|
||||
settings = configuration(url + "/info/lfs");
|
||||
if (hasLfs) protectLfs();
|
||||
require("target-remote", "remote", "add", "target", url);
|
||||
if (!require("snapshot-tree", "rev-parse", commit + "^{tree}").equals(tree)) throw GitCommandRunner.failed(deadline);
|
||||
if (hasLfs) {
|
||||
deadline.beginWrite();
|
||||
require("lfs-push", "lfs", "push", "--all", "target", "refs/heads/snapshot");
|
||||
deadline.confirmWrite();
|
||||
}
|
||||
deadline.beginWrite();
|
||||
require("git-push", "push", "--quiet", "target", commit + ":refs/heads/main");
|
||||
String remote = require("target-check", "ls-remote", "target", "refs/heads/main");
|
||||
if (!remote.equals(commit + "\trefs/heads/main")) throw GitCommandRunner.failed(deadline);
|
||||
deadline.confirmWrite();
|
||||
verifyRemote(target, commit, deadline);
|
||||
if (hasLfs) {
|
||||
// 用独立缓存从副本下载,确认不依赖来源的本地缓存且实体完整。
|
||||
try { blockLfsLogs(workspace.resolve("verified-lfs")); }
|
||||
catch (IOException error) { throw GitCommandRunner.failed(deadline); }
|
||||
settings.put("lfs.storage", workspace.resolve("verified-lfs").toString());
|
||||
require("lfs-verify-fetch", "lfs", "fetch", "--include=", "--exclude=", "target", commit);
|
||||
require("lfs-verify-fsck", "lfs", "fsck", "--objects", commit);
|
||||
}
|
||||
deadline.check();
|
||||
return commit;
|
||||
}
|
||||
|
||||
@Override public void close() {
|
||||
if (lfsGuard != null) lfsGuard.close();
|
||||
clean(workspace, deadline.projectCode());
|
||||
}
|
||||
}
|
||||
|
||||
protected void verifyRemote(Repository target, String commit, ForkDeadline deadline) {
|
||||
// Gitea 首次推送的元数据由服务端同步;在 empty 清除之前访问分支 API 可能失败。
|
||||
while (gitea.inspect(target.name()).empty()) {
|
||||
deadline.check();
|
||||
try { Thread.sleep(Math.min(100, Math.max(1, deadline.remaining().toMillis()))); }
|
||||
catch (InterruptedException error) { Thread.currentThread().interrupt(); throw deadline.failure(); }
|
||||
}
|
||||
if (!gitea.head(target).equals(commit)) throw GitCommandRunner.failed(deadline);
|
||||
}
|
||||
|
||||
private void blockLfsLogs(Path storage) throws IOException {
|
||||
// git-lfs 错误报告会写入全部 GIT_* 环境,必须在任何 LFS 命令前阻止落盘。
|
||||
Files.createDirectories(storage);
|
||||
Files.createFile(storage.resolve("logs"));
|
||||
}
|
||||
|
||||
private void clean(Path directory, String projectCode) {
|
||||
try {
|
||||
Files.walkFileTree(directory, new SimpleFileVisitor<>() {
|
||||
@Override public FileVisitResult visitFile(Path file, BasicFileAttributes attributes) throws IOException {
|
||||
Files.deleteIfExists(file);
|
||||
return FileVisitResult.CONTINUE;
|
||||
}
|
||||
@Override public FileVisitResult postVisitDirectory(Path folder, IOException error) throws IOException {
|
||||
if (error != null) throw error;
|
||||
Files.deleteIfExists(folder);
|
||||
return FileVisitResult.CONTINUE;
|
||||
}
|
||||
});
|
||||
} catch (IOException error) {
|
||||
log.warn("Fork 临时目录清理失败 projectCode={} errorType={}", projectCode, error.getClass().getSimpleName());
|
||||
}
|
||||
}
|
||||
}
|
||||
+73
@@ -0,0 +1,73 @@
|
||||
package com.linkerbot.community.service.fork;
|
||||
|
||||
import com.linkerbot.community.common.BusinessException;
|
||||
import com.linkerbot.community.service.gitea.GiteaFailure;
|
||||
import java.time.Duration;
|
||||
import java.util.Map;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
/** 一次 Fork 的截止时间和远端写入状态;仅在其独立工作线程内绑定。 */
|
||||
public final class ForkDeadline {
|
||||
private static final Logger log = LoggerFactory.getLogger(ForkDeadline.class);
|
||||
private static final ThreadLocal<ForkDeadline> CURRENT = new ThreadLocal<>();
|
||||
private final long expires;
|
||||
private volatile String projectCode;
|
||||
private volatile boolean writing;
|
||||
private volatile String cancellation;
|
||||
private final List<Runnable> cleanup = new ArrayList<>();
|
||||
|
||||
public ForkDeadline(Duration timeout) { expires = System.nanoTime() + timeout.toNanos(); }
|
||||
|
||||
static void bind(ForkDeadline deadline) { CURRENT.set(deadline); }
|
||||
static void unbind() { CURRENT.remove(); }
|
||||
public static ForkDeadline current() { return CURRENT.get(); }
|
||||
public void projectCode(String code) { projectCode = code; }
|
||||
public String projectCode() { return projectCode; }
|
||||
public boolean writing() { return writing; }
|
||||
public void onClose(Runnable action) { cleanup.add(action); }
|
||||
void closeResources() {
|
||||
for (Runnable action : cleanup) {
|
||||
try { action.run(); }
|
||||
catch (RuntimeException error) { log.warn("Fork 资源关闭失败 projectCode={} errorType={}", projectCode, error.getClass().getSimpleName()); }
|
||||
}
|
||||
cleanup.clear();
|
||||
}
|
||||
|
||||
public static void checkpoint() {
|
||||
ForkDeadline deadline = current();
|
||||
if (deadline != null) deadline.check();
|
||||
}
|
||||
|
||||
public void check() {
|
||||
if (cancellation != null || Thread.currentThread().isInterrupted() || System.nanoTime() >= expires)
|
||||
throw failure();
|
||||
}
|
||||
|
||||
public Duration remaining() {
|
||||
check();
|
||||
return Duration.ofNanos(Math.max(1, expires - System.nanoTime()));
|
||||
}
|
||||
|
||||
public static Duration bounded(Duration maximum) {
|
||||
ForkDeadline deadline = current();
|
||||
if (deadline == null) return maximum;
|
||||
Duration remaining = deadline.remaining();
|
||||
return remaining.compareTo(maximum) < 0 ? remaining : maximum;
|
||||
}
|
||||
|
||||
public synchronized void beginWrite() { check(); writing = true; }
|
||||
public synchronized void confirmWrite() { writing = false; }
|
||||
synchronized void cancel(String code) { cancellation = code; }
|
||||
|
||||
public BusinessException failure() {
|
||||
String code = cancellation == null ? "FORK_TIMEOUT" : cancellation;
|
||||
String message = "Fork 已超时或取消,请先核对工作区项目状态";
|
||||
if (writing) return new GiteaFailure(504, "REMOTE_RESULT_UNKNOWN", message, true);
|
||||
return new BusinessException(code.equals("FORK_CANCELLED") ? 499 : 504, code, message, details());
|
||||
}
|
||||
|
||||
public Map<String, String> details() { return projectCode == null ? null : Map.of("projectCode", projectCode); }
|
||||
}
|
||||
+90
@@ -0,0 +1,90 @@
|
||||
package com.linkerbot.community.service.fork;
|
||||
|
||||
import com.linkerbot.community.common.BusinessException;
|
||||
import com.linkerbot.community.service.config.ForkProperties;
|
||||
import jakarta.annotation.PostConstruct;
|
||||
import jakarta.annotation.PreDestroy;
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.*;
|
||||
import java.util.function.Function;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.slf4j.MDC;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
/** 同步等待结果,但工作线程及并发名额的生命周期不依赖 HTTP 连接。 */
|
||||
@Component
|
||||
public class ForkExecutor {
|
||||
private static final Logger log = LoggerFactory.getLogger(ForkExecutor.class);
|
||||
private final ForkProperties properties;
|
||||
private final GitSnapshotCopier copier;
|
||||
private final Semaphore permits;
|
||||
private final ExecutorService workers = Executors.newThreadPerTaskExecutor(Thread.ofVirtual().name("project-fork-", 0).factory());
|
||||
private final Set<FutureTask<?>> active = ConcurrentHashMap.newKeySet();
|
||||
|
||||
public ForkExecutor(ForkProperties properties, GitSnapshotCopier copier) {
|
||||
this.properties = properties;
|
||||
this.copier = copier;
|
||||
permits = new Semaphore(properties.getMaxConcurrent());
|
||||
}
|
||||
|
||||
@PostConstruct
|
||||
public void inspectRuntime() {
|
||||
try { copier.checkAvailable(); }
|
||||
catch (BusinessException error) { log.warn("Fork 运行依赖不可用 code={}", error.code()); }
|
||||
}
|
||||
|
||||
public <T> T execute(Function<ForkDeadline, T> action) {
|
||||
ForkDeadline deadline = new ForkDeadline(properties.getTimeout());
|
||||
if (!permits.tryAcquire()) throw new BusinessException(429, "FORK_BUSY", "Fork 繁忙,请稍后手动重试");
|
||||
try { copier.checkAvailable(); }
|
||||
catch (RuntimeException error) { permits.release(); throw error; }
|
||||
String requestId = MDC.get("requestId");
|
||||
FutureTask<T> task = new FutureTask<>(() -> {
|
||||
ForkDeadline.bind(deadline);
|
||||
if (requestId != null) MDC.put("requestId", requestId);
|
||||
try { deadline.check(); return action.apply(deadline); }
|
||||
finally { deadline.closeResources(); ForkDeadline.unbind(); MDC.remove("requestId"); }
|
||||
});
|
||||
active.add(task);
|
||||
try {
|
||||
workers.execute(() -> {
|
||||
try { task.run(); }
|
||||
finally { active.remove(task); permits.release(); }
|
||||
});
|
||||
} catch (RejectedExecutionException error) {
|
||||
active.remove(task);
|
||||
permits.release();
|
||||
throw new BusinessException(503, "FORK_UNAVAILABLE", "Fork 正在停止,请稍后重试");
|
||||
}
|
||||
try {
|
||||
return task.get(deadline.remaining().toNanos(), TimeUnit.NANOSECONDS);
|
||||
} catch (TimeoutException error) {
|
||||
deadline.cancel("FORK_TIMEOUT");
|
||||
task.cancel(true);
|
||||
throw failure(deadline);
|
||||
} catch (InterruptedException error) {
|
||||
deadline.cancel("FORK_CANCELLED");
|
||||
task.cancel(true);
|
||||
Thread.currentThread().interrupt();
|
||||
throw failure(deadline);
|
||||
} catch (CancellationException error) {
|
||||
throw failure(deadline);
|
||||
} catch (BusinessException error) {
|
||||
deadline.cancel("FORK_TIMEOUT");
|
||||
task.cancel(true);
|
||||
throw failure(deadline);
|
||||
} catch (ExecutionException error) {
|
||||
if (error.getCause() instanceof RuntimeException cause) throw cause;
|
||||
throw new BusinessException(500, "FORK_COPY_FAILED", "Fork 复制失败", deadline.details());
|
||||
}
|
||||
}
|
||||
|
||||
private BusinessException failure(ForkDeadline deadline) {
|
||||
BusinessException error = deadline.failure();
|
||||
return new BusinessException(error.status(), error.code(), error.getMessage(), deadline.details());
|
||||
}
|
||||
|
||||
@PreDestroy
|
||||
public void close() { active.forEach(task -> task.cancel(true)); workers.shutdownNow(); }
|
||||
}
|
||||
+188
@@ -0,0 +1,188 @@
|
||||
package com.linkerbot.community.service.fork;
|
||||
|
||||
import com.linkerbot.community.common.BusinessException;
|
||||
import com.linkerbot.community.service.config.ForkProperties;
|
||||
import com.linkerbot.community.service.gitea.GiteaFailure;
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.*;
|
||||
import java.nio.file.attribute.BasicFileAttributes;
|
||||
import java.util.*;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.atomic.AtomicBoolean;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
/** 不执行 Shell,不记录参数或原始输出;输出与磁盘占用均有上限。 */
|
||||
@Component
|
||||
public class GitCommandRunner {
|
||||
private static final Logger log = LoggerFactory.getLogger(GitCommandRunner.class);
|
||||
private static final int MAX_OUTPUT_BYTES = 8 * 1024 * 1024;
|
||||
private final ForkProperties properties;
|
||||
|
||||
public GitCommandRunner(ForkProperties properties) { this.properties = properties; }
|
||||
|
||||
public record Result(int exitCode, String output, boolean rejected) {}
|
||||
|
||||
public Result run(Path workspace, ForkDeadline deadline, String stage, Map<String, String> configuration,
|
||||
Map<String, String> identity, String... arguments) {
|
||||
deadline.check();
|
||||
List<String> command = new ArrayList<>();
|
||||
command.add(properties.getGitExecutable());
|
||||
if (workspace != null) command.add("--git-dir=" + workspace);
|
||||
command.addAll(List.of(arguments));
|
||||
ProcessBuilder builder = new ProcessBuilder(command);
|
||||
Map<String, String> environment = builder.environment();
|
||||
Map<String, String> inherited = new HashMap<>(environment);
|
||||
environment.clear();
|
||||
for (String key : List.of("PATH", "SystemRoot", "TMPDIR", "TEMP", "TMP"))
|
||||
if (inherited.containsKey(key)) environment.put(key, inherited.get(key));
|
||||
environment.put("LC_ALL", "C.UTF-8");
|
||||
environment.put("GIT_CONFIG_NOSYSTEM", "1");
|
||||
environment.put("GIT_CONFIG_GLOBAL", "/dev/null");
|
||||
environment.put("GIT_ATTR_NOSYSTEM", "1");
|
||||
environment.put("GIT_TERMINAL_PROMPT", "0");
|
||||
environment.put("GIT_LFS_SKIP_SMUDGE", "1");
|
||||
Map<String, String> settings = new LinkedHashMap<>();
|
||||
settings.put("core.hooksPath", "/dev/null");
|
||||
settings.put("credential.helper", "");
|
||||
settings.put("http.followRedirects", "false");
|
||||
settings.put("protocol.allow", "never");
|
||||
settings.put("protocol.http.allow", "always");
|
||||
settings.put("protocol.https.allow", "always");
|
||||
settings.putAll(configuration);
|
||||
environment.put("GIT_CONFIG_COUNT", Integer.toString(settings.size()));
|
||||
int index = 0;
|
||||
for (Map.Entry<String, String> setting : settings.entrySet()) {
|
||||
environment.put("GIT_CONFIG_KEY_" + index, setting.getKey());
|
||||
environment.put("GIT_CONFIG_VALUE_" + index, setting.getValue());
|
||||
index++;
|
||||
}
|
||||
environment.putAll(identity);
|
||||
Process process = null;
|
||||
Set<ProcessHandle> children = new HashSet<>();
|
||||
Thread reader = null;
|
||||
Thread errorReader = null;
|
||||
AtomicBoolean overflow = new AtomicBoolean();
|
||||
ByteArrayOutputStream output = new ByteArrayOutputStream();
|
||||
ByteArrayOutputStream errors = new ByteArrayOutputStream();
|
||||
long started = System.nanoTime();
|
||||
try {
|
||||
process = builder.start();
|
||||
process.getOutputStream().close();
|
||||
InputStream stdout = process.getInputStream();
|
||||
reader = Thread.ofVirtual().start(() -> drain(stdout, output, overflow, MAX_OUTPUT_BYTES));
|
||||
InputStream stderr = process.getErrorStream();
|
||||
errorReader = Thread.ofVirtual().start(() -> drain(stderr, errors, new AtomicBoolean(), 32768));
|
||||
long checked = 0;
|
||||
while (true) {
|
||||
process.descendants().forEach(children::add);
|
||||
deadline.check();
|
||||
if (overflow.get()) throw resourceLimit();
|
||||
if (workspace != null && System.nanoTime() - checked >= TimeUnit.MILLISECONDS.toNanos(250)) {
|
||||
checkDisk(workspace);
|
||||
checked = System.nanoTime();
|
||||
}
|
||||
if (process.waitFor(50, TimeUnit.MILLISECONDS)) break;
|
||||
}
|
||||
while (reader.isAlive()) { reader.join(50); deadline.check(); }
|
||||
while (errorReader.isAlive()) { errorReader.join(50); deadline.check(); }
|
||||
deadline.check();
|
||||
if (overflow.get()) throw resourceLimit();
|
||||
if (workspace != null) checkDisk(workspace);
|
||||
int exitCode = process.exitValue();
|
||||
String diagnostic = errors.toString(StandardCharsets.UTF_8).toLowerCase(Locale.ROOT);
|
||||
boolean rejected = exitCode != 0 && (diagnostic.contains("authentication failed")
|
||||
|| diagnostic.contains("could not read username") || diagnostic.contains("[remote rejected]")
|
||||
|| diagnostic.contains("requested url returned error: 401") || diagnostic.contains("requested url returned error: 403")
|
||||
|| diagnostic.contains("requested url returned error: 404"));
|
||||
log.info("Fork Git 阶段完成 projectCode={} stage={} exitCode={} failureType={} durationMs={}",
|
||||
deadline.projectCode(), stage, exitCode, rejected ? "REMOTE_REJECTED" : exitCode == 0 ? "NONE" : "EXIT_FAILURE",
|
||||
TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - started));
|
||||
return new Result(exitCode, output.toString(StandardCharsets.UTF_8).trim(), rejected);
|
||||
} catch (InterruptedException error) {
|
||||
Thread.currentThread().interrupt();
|
||||
deadline.check();
|
||||
throw deadline.failure();
|
||||
} catch (IOException error) {
|
||||
log.warn("Fork Git 进程异常 projectCode={} stage={} errorType={}", deadline.projectCode(), stage, error.getClass().getSimpleName());
|
||||
throw failed(deadline);
|
||||
} finally {
|
||||
if (process != null) stop(process, children);
|
||||
boolean interrupted = Thread.interrupted();
|
||||
for (Thread drain : new Thread[] {reader, errorReader}) {
|
||||
while (drain != null && drain.isAlive()) {
|
||||
try { drain.join(); }
|
||||
catch (InterruptedException error) { interrupted = true; }
|
||||
}
|
||||
}
|
||||
if (interrupted) Thread.currentThread().interrupt();
|
||||
}
|
||||
}
|
||||
|
||||
private void drain(InputStream source, ByteArrayOutputStream target, AtomicBoolean overflow, int limit) {
|
||||
try (InputStream input = source) {
|
||||
byte[] buffer = new byte[8192];
|
||||
int count;
|
||||
while ((count = input.read(buffer)) != -1) {
|
||||
if (count > limit - target.size()) overflow.set(true);
|
||||
if (!overflow.get()) target.write(buffer, 0, count);
|
||||
}
|
||||
} catch (IOException ignored) {
|
||||
// 进程被终止时管道会关闭;最终由退出状态和截止时间判断结果。
|
||||
}
|
||||
}
|
||||
|
||||
private void stop(Process process, Set<ProcessHandle> descendants) {
|
||||
boolean interrupted = Thread.interrupted();
|
||||
process.descendants().forEach(descendants::add);
|
||||
descendants.forEach(child -> { if (child.isAlive()) child.destroyForcibly(); });
|
||||
try {
|
||||
// 先让父进程回收子进程,再终止仍未退出的父进程,避免遗留僵尸。
|
||||
try { if (!descendants.isEmpty()) process.waitFor(1, TimeUnit.SECONDS); }
|
||||
catch (InterruptedException error) { interrupted = true; }
|
||||
if (process.isAlive()) process.destroyForcibly();
|
||||
while (process.isAlive()) {
|
||||
try { process.waitFor(100, TimeUnit.MILLISECONDS); }
|
||||
catch (InterruptedException error) { interrupted = true; }
|
||||
}
|
||||
for (ProcessHandle child : descendants) {
|
||||
while (child.isAlive()) {
|
||||
try { child.onExit().get(100, TimeUnit.MILLISECONDS); }
|
||||
catch (InterruptedException error) { interrupted = true; }
|
||||
catch (java.util.concurrent.ExecutionException | java.util.concurrent.TimeoutException ignored) { child.destroyForcibly(); }
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
if (interrupted) Thread.currentThread().interrupt();
|
||||
}
|
||||
}
|
||||
|
||||
public void checkDisk(Path workspace) throws IOException {
|
||||
long[] bytes = {0};
|
||||
Files.walkFileTree(workspace, new SimpleFileVisitor<>() {
|
||||
@Override public FileVisitResult visitFile(Path file, BasicFileAttributes attributes) {
|
||||
long size = attributes.size();
|
||||
if (size > properties.getMaxTemporaryBytes() - bytes[0]) throw resourceLimit();
|
||||
bytes[0] += size;
|
||||
return FileVisitResult.CONTINUE;
|
||||
}
|
||||
@Override public FileVisitResult visitFileFailed(Path file, IOException error) throws IOException {
|
||||
if (error instanceof NoSuchFileException) return FileVisitResult.CONTINUE;
|
||||
throw error;
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
private BusinessException resourceLimit() {
|
||||
return new BusinessException(413, "FORK_RESOURCE_LIMIT", "Fork 临时磁盘或元数据超过限制");
|
||||
}
|
||||
|
||||
static BusinessException failed(ForkDeadline deadline) {
|
||||
if (deadline.writing()) return new GiteaFailure(504, "REMOTE_RESULT_UNKNOWN", "Fork 写入结果待核对,请先查看工作区", true);
|
||||
return new BusinessException(502, "FORK_COPY_FAILED", "Fork 快照或 LFS 复制失败,请通过 requestId 排查");
|
||||
}
|
||||
}
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
package com.linkerbot.community.service.fork;
|
||||
|
||||
import com.linkerbot.community.service.gitea.GiteaClient.Repository;
|
||||
|
||||
/** 仓库传输边界;准备阶段不创建远端仓库,业务测试可使用内存替身。 */
|
||||
public interface GitSnapshotCopier {
|
||||
void checkAvailable();
|
||||
|
||||
PreparedSnapshot prepare(Repository source, String publishedSha, String authorName, String authorCode,
|
||||
ForkDeadline deadline);
|
||||
|
||||
interface PreparedSnapshot extends AutoCloseable {
|
||||
String push(Repository target);
|
||||
@Override void close();
|
||||
}
|
||||
}
|
||||
+165
@@ -0,0 +1,165 @@
|
||||
package com.linkerbot.community.service.fork;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.fasterxml.jackson.databind.node.ObjectNode;
|
||||
import com.sun.net.httpserver.HttpExchange;
|
||||
import com.sun.net.httpserver.HttpServer;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.net.InetSocketAddress;
|
||||
import java.net.URI;
|
||||
import java.net.http.HttpClient;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.security.MessageDigest;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.UUID;
|
||||
import java.util.concurrent.*;
|
||||
import java.util.concurrent.atomic.AtomicBoolean;
|
||||
|
||||
/** git-lfs 自身不支持禁止重定向;在任务内约束所有 batch/action 请求到配置中的 Gitea。 */
|
||||
final class LfsEndpointGuard implements AutoCloseable {
|
||||
private static final int MAX_METADATA = 8 * 1024 * 1024;
|
||||
private final URI trustedBase;
|
||||
private final URI endpoint;
|
||||
private final String authorization;
|
||||
private final String localAuthorization = "Bearer " + UUID.randomUUID();
|
||||
private final ForkDeadline deadline;
|
||||
private final ObjectMapper json;
|
||||
private final HttpServer server;
|
||||
private final HttpClient http;
|
||||
private final ExecutorService workers = Executors.newVirtualThreadPerTaskExecutor();
|
||||
private final ScheduledExecutorService timer = Executors.newSingleThreadScheduledExecutor(Thread.ofVirtual().factory());
|
||||
private final Map<String, Action> actions = new ConcurrentHashMap<>();
|
||||
private final Set<HttpExchange> exchanges = ConcurrentHashMap.newKeySet();
|
||||
private final Set<InputStream> responses = ConcurrentHashMap.newKeySet();
|
||||
private final AtomicBoolean closed = new AtomicBoolean();
|
||||
private record Action(URI url, Map<String, String> headers) {}
|
||||
|
||||
LfsEndpointGuard(URI trustedBase, URI endpoint, String authorization, ForkDeadline deadline, ObjectMapper json) throws IOException {
|
||||
this.trustedBase = trustedBase;
|
||||
this.endpoint = endpoint;
|
||||
this.authorization = authorization;
|
||||
this.deadline = deadline;
|
||||
this.json = json;
|
||||
if (!allowed(endpoint)) throw new IOException("LFS endpoint rejected");
|
||||
http = HttpClient.newBuilder().followRedirects(HttpClient.Redirect.NEVER).connectTimeout(deadline.remaining()).build();
|
||||
try { server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0); }
|
||||
catch (IOException error) { http.shutdownNow(); throw error; }
|
||||
try {
|
||||
server.setExecutor(workers);
|
||||
server.createContext("/lfs", this::handle);
|
||||
server.start();
|
||||
timer.schedule(this::close, deadline.remaining().toNanos(), TimeUnit.NANOSECONDS);
|
||||
} catch (RuntimeException error) { close(); throw error; }
|
||||
}
|
||||
|
||||
String endpoint() { return "http://127.0.0.1:" + server.getAddress().getPort() + "/lfs"; }
|
||||
String localAuthorization() { return localAuthorization; }
|
||||
|
||||
private boolean allowed(URI uri) {
|
||||
return uri.getRawUserInfo() == null && uri.getRawFragment() == null
|
||||
&& trustedBase.getScheme().equalsIgnoreCase(uri.getScheme())
|
||||
&& trustedBase.getRawAuthority().equalsIgnoreCase(uri.getRawAuthority())
|
||||
&& uri.normalize().getRawPath().startsWith(trustedBase.getRawPath().replaceAll("/+$", "") + "/");
|
||||
}
|
||||
|
||||
private void handle(HttpExchange exchange) {
|
||||
exchanges.add(exchange);
|
||||
try (HttpExchange request = exchange) {
|
||||
deadline.check();
|
||||
String supplied = request.getRequestHeaders().getFirst("Authorization");
|
||||
if (supplied == null || !MessageDigest.isEqual(supplied.getBytes(StandardCharsets.UTF_8), localAuthorization.getBytes(StandardCharsets.UTF_8))) {
|
||||
request.sendResponseHeaders(403, -1);
|
||||
return;
|
||||
}
|
||||
String path = request.getRequestURI().getPath();
|
||||
boolean batch = path.equals("/lfs/objects/batch") && request.getRequestMethod().equals("POST");
|
||||
Action action = batch ? new Action(URI.create(endpoint + "/objects/batch"), Map.of()) : actions.get(path);
|
||||
if (action == null || !Set.of("GET", "POST", "PUT").contains(request.getRequestMethod())) {
|
||||
request.sendResponseHeaders(400, -1);
|
||||
return;
|
||||
}
|
||||
HttpRequest.Builder outbound = HttpRequest.newBuilder(action.url()).timeout(deadline.remaining())
|
||||
.header("Authorization", authorization);
|
||||
if (batch) outbound.header("Accept", "application/vnd.git-lfs+json");
|
||||
action.headers().forEach(outbound::setHeader);
|
||||
String type = request.getRequestHeaders().getFirst("Content-Type");
|
||||
if (type != null) outbound.setHeader("Content-Type", type);
|
||||
HttpRequest.BodyPublisher body = batch ? HttpRequest.BodyPublishers.ofByteArray(metadata(request.getRequestBody()))
|
||||
: request.getRequestMethod().equals("GET") ? HttpRequest.BodyPublishers.noBody()
|
||||
: HttpRequest.BodyPublishers.ofInputStream(request::getRequestBody);
|
||||
outbound.method(request.getRequestMethod(), body);
|
||||
HttpResponse<InputStream> response = http.send(outbound.build(), HttpResponse.BodyHandlers.ofInputStream());
|
||||
InputStream stream = response.body();
|
||||
responses.add(stream);
|
||||
try (InputStream input = stream) {
|
||||
// 不把 Location 传给 git-lfs,禁止它自行追随重定向。
|
||||
if (response.statusCode() < 200 || response.statusCode() >= 300) {
|
||||
request.sendResponseHeaders(response.statusCode() >= 300 && response.statusCode() < 400 ? 502 : response.statusCode(), -1);
|
||||
return;
|
||||
}
|
||||
if (batch) {
|
||||
JsonNode result = json.readTree(metadata(input));
|
||||
for (JsonNode object : result.path("objects")) {
|
||||
JsonNode entries = object.path("actions");
|
||||
for (JsonNode value : entries) {
|
||||
if (!(value instanceof ObjectNode item)) throw new IOException("LFS action rejected");
|
||||
URI destination = URI.create(item.path("href").asText());
|
||||
if (!allowed(destination)) throw new IOException("LFS action origin rejected");
|
||||
Map<String, String> headers = new java.util.HashMap<>();
|
||||
JsonNode header = item.path("header");
|
||||
header.fields().forEachRemaining(field -> {
|
||||
if (Set.of("authorization", "content-type", "accept").contains(field.getKey().toLowerCase(java.util.Locale.ROOT)))
|
||||
headers.put(field.getKey(), field.getValue().asText());
|
||||
});
|
||||
String route = "/lfs/action/" + UUID.randomUUID();
|
||||
actions.put(route, new Action(destination, headers));
|
||||
item.put("href", endpoint().replace("/lfs", "") + route);
|
||||
item.putObject("header").put("Authorization", localAuthorization);
|
||||
}
|
||||
}
|
||||
byte[] bytes = json.writeValueAsBytes(result);
|
||||
request.getResponseHeaders().set("Content-Type", "application/vnd.git-lfs+json");
|
||||
request.sendResponseHeaders(response.statusCode(), bytes.length);
|
||||
request.getResponseBody().write(bytes);
|
||||
} else {
|
||||
request.sendResponseHeaders(response.statusCode(), 0);
|
||||
byte[] buffer = new byte[65536];
|
||||
int count;
|
||||
while ((count = input.read(buffer)) != -1) {
|
||||
deadline.check();
|
||||
request.getResponseBody().write(buffer, 0, count);
|
||||
}
|
||||
}
|
||||
} finally { responses.remove(stream); }
|
||||
} catch (IOException | InterruptedException | RuntimeException error) {
|
||||
if (error instanceof InterruptedException) Thread.currentThread().interrupt();
|
||||
// 不输出服务端原始响应、地址、头或凭据;git-lfs 会以非零状态结束该阶段。
|
||||
exchange.close();
|
||||
} finally { exchanges.remove(exchange); }
|
||||
}
|
||||
|
||||
private byte[] metadata(InputStream input) throws IOException {
|
||||
byte[] bytes = input.readNBytes(MAX_METADATA + 1);
|
||||
if (bytes.length > MAX_METADATA) throw new IOException("LFS metadata limit exceeded");
|
||||
return bytes;
|
||||
}
|
||||
|
||||
@Override public void close() {
|
||||
if (closed.compareAndSet(false, true)) {
|
||||
server.stop(0);
|
||||
exchanges.forEach(HttpExchange::close);
|
||||
responses.forEach(stream -> { try { stream.close(); } catch (IOException ignored) {} });
|
||||
http.shutdownNow();
|
||||
workers.shutdownNow();
|
||||
timer.shutdownNow();
|
||||
actions.clear();
|
||||
}
|
||||
// 并发名额只能在转发流也实际退出后释放;不能只停止命令行进程。
|
||||
workers.close();
|
||||
}
|
||||
}
|
||||
+3
-1
@@ -7,6 +7,8 @@ import java.util.List;
|
||||
import java.time.Duration;
|
||||
|
||||
public interface GiteaClient {
|
||||
default String authenticatedUsername() { throw new UnsupportedOperationException("未实现服务账号查询"); }
|
||||
|
||||
record Repository(long id, String owner, String name) {}
|
||||
|
||||
record StoredFile(String path, String blobSha, byte[] bytes) {}
|
||||
@@ -33,7 +35,7 @@ public interface GiteaClient {
|
||||
default void checkMigrationSettings() { throw new UnsupportedOperationException("未实现迁移能力检查"); }
|
||||
|
||||
default List<String> migrationWarnings(Repository repository, String sha, long maxFileBytes) {
|
||||
return List.of("社区下载仍受单文件限制,LFS 实体读取及特殊 Git 文件的 Fork 能力未扩展");
|
||||
return List.of("在线下载仍受单文件限制,LFS 实体在线读取能力未扩展;Git 快照 Fork 保留子模块引用,不递归复制依赖");
|
||||
}
|
||||
|
||||
Repository createRepository(String name, boolean initialize);
|
||||
|
||||
+13
-4
@@ -7,6 +7,7 @@ import com.linkerbot.community.common.BusinessException;
|
||||
import com.linkerbot.community.common.FilePaths;
|
||||
import com.linkerbot.community.common.PageResponse;
|
||||
import com.linkerbot.community.service.config.CommunityProperties;
|
||||
import com.linkerbot.community.service.fork.ForkDeadline;
|
||||
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
@@ -62,7 +63,7 @@ public class RestGiteaClient implements GiteaClient {
|
||||
String base = properties.getGitea().getBaseUrl().replaceAll("/+$", "");
|
||||
HttpRequest.Builder builder =
|
||||
HttpRequest.newBuilder(URI.create(base + "/api/v1" + path))
|
||||
.timeout(timeout)
|
||||
.timeout(ForkDeadline.bounded(timeout))
|
||||
.header("Accept", "application/json")
|
||||
.header("Authorization", "token " + properties.getGitea().getToken());
|
||||
if (body == null) builder.method(method, HttpRequest.BodyPublishers.noBody());
|
||||
@@ -106,6 +107,14 @@ public class RestGiteaClient implements GiteaClient {
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public String authenticatedUsername() {
|
||||
JsonNode user = parse(request("GET", "/user", null).body(), false);
|
||||
if (!user.path("login").isTextual() || user.path("login").asText().isBlank())
|
||||
throw new GiteaFailure(502, "FORK_COPY_FAILED", "无法确认 Gitea 服务账号身份", false);
|
||||
return user.path("login").asText();
|
||||
}
|
||||
|
||||
@Override
|
||||
public Repository createRepository(String name, boolean initialize) {
|
||||
Map<String, Object> body = new LinkedHashMap<>();
|
||||
@@ -186,7 +195,7 @@ public class RestGiteaClient implements GiteaClient {
|
||||
@Override
|
||||
public List<String> migrationWarnings(Repository repo, String sha, long maxFileBytes) {
|
||||
List<String> warnings = new ArrayList<>();
|
||||
warnings.add("已请求迁移 LFS;社区 API 的 LFS 实体读取能力未扩展");
|
||||
warnings.add("已请求迁移 LFS;在线 API 的 LFS 实体读取能力未扩展,Git 快照 Fork 会复制发布版本引用的实体");
|
||||
long large = 0;
|
||||
long links = 0;
|
||||
long modules = 0;
|
||||
@@ -206,8 +215,8 @@ public class RestGiteaClient implements GiteaClient {
|
||||
if (entries.isEmpty() || page == 10000) { warnings.add("文件树扫描不完整,需要人工核对剩余文件"); break; }
|
||||
}
|
||||
if (large > 0) warnings.add("发布快照有 " + large + " 个文件超过社区单文件读取限制 " + maxFileBytes + " 字节");
|
||||
if (links > 0) warnings.add("发布快照有 " + links + " 个符号链接,社区普通文件读取和 Fork 不支持");
|
||||
if (modules > 0) warnings.add("发布快照有 " + modules + " 个子模块引用,未递归迁移且社区 Fork 不支持");
|
||||
if (links > 0) warnings.add("发布快照有 " + links + " 个符号链接,在线普通文件读取不支持,Git 快照 Fork 保留链接");
|
||||
if (modules > 0) warnings.add("发布快照有 " + modules + " 个子模块引用,Git 快照 Fork 保留引用,不递归复制依赖仓库");
|
||||
return List.copyOf(warnings);
|
||||
}
|
||||
|
||||
|
||||
+10
-2
@@ -8,6 +8,7 @@ import com.linkerbot.community.common.BusinessException;
|
||||
import com.linkerbot.community.service.AiSearchService;
|
||||
import com.linkerbot.community.service.ai.AiProjectAssistant;
|
||||
import com.linkerbot.community.service.config.AiProperties;
|
||||
import com.linkerbot.community.service.config.ForkProperties;
|
||||
import jakarta.annotation.PreDestroy;
|
||||
import java.io.IOException;
|
||||
import java.time.Duration;
|
||||
@@ -26,6 +27,7 @@ public class AiSearchServiceImpl implements AiSearchService {
|
||||
private static final Logger log = LoggerFactory.getLogger(AiSearchServiceImpl.class);
|
||||
private final AiProperties properties;
|
||||
private final AiProjectAssistant assistant;
|
||||
private final ForkProperties forkProperties;
|
||||
private final Semaphore permits;
|
||||
private final ExecutorService workers = Executors.newThreadPerTaskExecutor(
|
||||
Thread.ofVirtual().name("ai-search-", 0).factory());
|
||||
@@ -35,9 +37,10 @@ public class AiSearchServiceImpl implements AiSearchService {
|
||||
.expireAfterAccess(Duration.ofMinutes(1)).maximumSize(10000).build();
|
||||
private final Set<Job> active = ConcurrentHashMap.newKeySet();
|
||||
|
||||
public AiSearchServiceImpl(AiProperties properties, AiProjectAssistant assistant) {
|
||||
public AiSearchServiceImpl(AiProperties properties, AiProjectAssistant assistant, ForkProperties forkProperties) {
|
||||
this.properties = properties;
|
||||
this.assistant = assistant;
|
||||
this.forkProperties = forkProperties;
|
||||
permits = new Semaphore(properties.getMaxConcurrent());
|
||||
}
|
||||
|
||||
@@ -97,7 +100,7 @@ public class AiSearchServiceImpl implements AiSearchService {
|
||||
private final ChatRequest request;
|
||||
private final String requestId;
|
||||
private final long started = System.nanoTime();
|
||||
private final SseEmitter emitter = new SseEmitter(properties.getSearchTimeout().toMillis());
|
||||
private final SseEmitter emitter = new SseEmitter(properties.getSearchTimeout().plus(forkProperties.getTimeout()).plusSeconds(15).toMillis());
|
||||
private final AtomicBoolean finished = new AtomicBoolean();
|
||||
private final AtomicBoolean released = new AtomicBoolean();
|
||||
private final FutureTask<Void> task;
|
||||
@@ -152,6 +155,11 @@ public class AiSearchServiceImpl implements AiSearchService {
|
||||
|
||||
private void send(String event, Object data) {
|
||||
if (finished.get() || Thread.currentThread().isInterrupted()) throw cancelled();
|
||||
if (data instanceof Progress progress && "forking".equals(progress.stage())) {
|
||||
if (deadline != null) deadline.cancel(false);
|
||||
deadline = timer.schedule(() -> fail(new BusinessException(504, "FORK_TIMEOUT", "Fork 超时,请先核对工作区项目状态")),
|
||||
forkProperties.getTimeout().plusSeconds(15).toMillis(), TimeUnit.MILLISECONDS);
|
||||
}
|
||||
if (!write(event, success(data))) throw cancelled();
|
||||
}
|
||||
|
||||
|
||||
+96
-61
@@ -13,6 +13,9 @@ import com.linkerbot.community.service.ProjectService;
|
||||
import com.linkerbot.community.service.UserService;
|
||||
import com.linkerbot.community.service.config.CommunityProperties;
|
||||
import com.linkerbot.community.service.gitea.*;
|
||||
import com.linkerbot.community.service.fork.ForkDeadline;
|
||||
import com.linkerbot.community.service.fork.ForkExecutor;
|
||||
import com.linkerbot.community.service.fork.GitSnapshotCopier;
|
||||
import com.linkerbot.community.service.mapper.*;
|
||||
import com.linkerbot.community.service.oss.UrdfResources;
|
||||
import com.linkerbot.community.service.support.ProjectLocks;
|
||||
@@ -42,6 +45,8 @@ public class ProjectServiceImpl implements ProjectService {
|
||||
private final ProjectLocks locks;
|
||||
// 离线导入只加载仓库和数据库组件,没有 URDF 时无需初始化 OSS 模块。
|
||||
private final ObjectProvider<UrdfResources> urdf;
|
||||
private final ObjectProvider<ForkExecutor> forks;
|
||||
private final ObjectProvider<GitSnapshotCopier> snapshots;
|
||||
|
||||
@Override
|
||||
public List<TypeView> types() {
|
||||
@@ -180,6 +185,10 @@ public class ProjectServiceImpl implements ProjectService {
|
||||
}
|
||||
|
||||
private Project initialize(long userId, ProjectInput input, Project source) {
|
||||
return initialize(userId, input, source, null);
|
||||
}
|
||||
|
||||
private Project initialize(long userId, ProjectInput input, Project source, ForkDeadline deadline) {
|
||||
validateInput(input);
|
||||
Project project = new Project();
|
||||
project.setProjectCode(ProjectCodes.next());
|
||||
@@ -201,6 +210,10 @@ public class ProjectServiceImpl implements ProjectService {
|
||||
}
|
||||
for (int attempt = 1; attempt <= 5; attempt++) {
|
||||
try {
|
||||
if (deadline != null) {
|
||||
deadline.check();
|
||||
deadline.projectCode(project.getProjectCode());
|
||||
}
|
||||
projects.insert(project);
|
||||
return project;
|
||||
} catch (DuplicateKeyException duplicate) {
|
||||
@@ -226,7 +239,7 @@ public class ProjectServiceImpl implements ProjectService {
|
||||
public ProjectView create(long userId, ProjectInput input) {
|
||||
users.require(userId);
|
||||
Project project = initialize(userId, input, null);
|
||||
return locks.write(project.getProjectCode(), () -> createRemote(project, null));
|
||||
return locks.write(project.getProjectCode(), () -> createRemote(project));
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -238,33 +251,97 @@ public class ProjectServiceImpl implements ProjectService {
|
||||
ProjectInput data =
|
||||
new ProjectInput(
|
||||
input.name(), source.getDescription(), source.getProjectType(), source.getCoverUrl());
|
||||
Project project = initialize(userId, data, source);
|
||||
return locks.write(
|
||||
project.getProjectCode(),
|
||||
() -> createRemote(project, new ForkContext(source, user)));
|
||||
validateInput(data);
|
||||
ForkExecutor executor = forks.getIfAvailable();
|
||||
if (executor == null) throw new BusinessException(503, "FORK_UNAVAILABLE", "当前进程未开启 Fork");
|
||||
return executor.execute(deadline -> {
|
||||
deadline.check();
|
||||
Project project;
|
||||
try { project = initialize(userId, data, source, deadline); }
|
||||
catch (BusinessException error) { throw error; }
|
||||
catch (RuntimeException error) {
|
||||
if (deadline.projectCode() != null) locks.markUncertain(deadline.projectCode());
|
||||
log.error("Fork 项目记录写入异常 projectCode={} sourceProjectCode={} errorType={}",
|
||||
deadline.projectCode(), sourceCode, error.getClass().getSimpleName());
|
||||
throw new BusinessException(500, "PROJECT_INITIALIZATION_ERROR", "Fork 项目记录待核对,请按项目编码人工检查", deadline.details());
|
||||
}
|
||||
return locks.write(project.getProjectCode(), () -> forkRemote(project, source, user, deadline));
|
||||
});
|
||||
}
|
||||
|
||||
private record ForkContext(Project source, User user) {}
|
||||
private ProjectView forkRemote(Project project, Project source, User user, ForkDeadline deadline) {
|
||||
long started = System.nanoTime();
|
||||
log.info("Fork 开始 projectCode={} sourceProjectCode={} sourceCommitSha={}",
|
||||
project.getProjectCode(), source.getProjectCode(), project.getSourceCommitSha());
|
||||
try (GitSnapshotCopier.PreparedSnapshot snapshot = snapshots.getObject().prepare(
|
||||
repository(source), project.getSourceCommitSha(), user.getNickname(), user.getUserCode(), deadline)) {
|
||||
deadline.beginWrite();
|
||||
GiteaClient.Repository repo;
|
||||
try {
|
||||
repo = gitea.createRepository(project.getGiteaRepoName(), false);
|
||||
} catch (GiteaFailure error) {
|
||||
if (!error.uncertain()) deadline.confirmWrite();
|
||||
throw error;
|
||||
}
|
||||
project.setGiteaRepoId(repo.id());
|
||||
// 先保存仓库关联,再执行 Git/LFS 推送,便于中断后按项目编码核对。
|
||||
if (projects.updateById(project) != 1) throw new IllegalStateException("Fork 仓库关联回写失败");
|
||||
deadline.confirmWrite();
|
||||
deadline.check();
|
||||
String commit = snapshot.push(repo);
|
||||
deadline.check();
|
||||
if (source.getUrdfMainObjectKey() != null && !source.getUrdfMainObjectKey().isEmpty())
|
||||
urdf.getObject().copy(source, project);
|
||||
deadline.check();
|
||||
project.setRepoStatus("READY");
|
||||
project.setUpdatedAt(now());
|
||||
if (projects.updateById(project) != 1) throw new IllegalStateException("Fork 完成状态回写失败");
|
||||
log.info("Fork 完成 projectCode={} sourceProjectCode={} sourceCommitSha={} snapshotCommitSha={} durationMs={}",
|
||||
project.getProjectCode(), source.getProjectCode(), project.getSourceCommitSha(), commit,
|
||||
java.util.concurrent.TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - started));
|
||||
return view(project, project.getOwnerUserId());
|
||||
} catch (RuntimeException error) {
|
||||
boolean uncertain = deadline.writing() || !(error instanceof BusinessException)
|
||||
|| error instanceof GiteaFailure remote && remote.uncertain();
|
||||
String code = error instanceof BusinessException business ? business.code() : "PROJECT_INITIALIZATION_ERROR";
|
||||
int status = error instanceof BusinessException business ? business.status() : 500;
|
||||
String message = error instanceof BusinessException business ? business.getMessage() : "Fork 初始化未完成,请按项目编码核对";
|
||||
if (deadline.writing() && error instanceof BusinessException) {
|
||||
code = "REMOTE_RESULT_UNKNOWN";
|
||||
status = 504;
|
||||
message = "Fork 远端写入结果待核对,请先查看工作区项目状态";
|
||||
}
|
||||
boolean interrupted = Thread.interrupted();
|
||||
try {
|
||||
project.setRepoStatus(uncertain ? "CREATING" : "FAILED");
|
||||
project.setUpdatedAt(now());
|
||||
if (uncertain) locks.markUncertain(project.getProjectCode());
|
||||
if (projects.updateById(project) != 1) throw new IllegalStateException("Fork 失败状态回写失败");
|
||||
} catch (RuntimeException persistence) {
|
||||
locks.markUncertain(project.getProjectCode());
|
||||
log.error("Fork 状态回写异常 projectCode={} errorType={}", project.getProjectCode(), persistence.getClass().getSimpleName());
|
||||
code = "PROJECT_INITIALIZATION_ERROR";
|
||||
status = 500;
|
||||
message = "Fork 状态未确认,请按项目编码人工核对";
|
||||
} finally {
|
||||
if (interrupted) Thread.currentThread().interrupt();
|
||||
}
|
||||
log.warn("Fork 失败 projectCode={} sourceProjectCode={} sourceCommitSha={} code={} uncertain={} errorType={} durationMs={}",
|
||||
project.getProjectCode(), source.getProjectCode(), project.getSourceCommitSha(), code, uncertain,
|
||||
error.getClass().getSimpleName(), java.util.concurrent.TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - started));
|
||||
throw new BusinessException(status, code, message, Map.of("projectCode", project.getProjectCode()));
|
||||
}
|
||||
}
|
||||
|
||||
private ProjectView createRemote(Project project, ForkContext fork) {
|
||||
private ProjectView createRemote(Project project) {
|
||||
try {
|
||||
GiteaClient.Repository repo =
|
||||
gitea.createRepository(project.getGiteaRepoName(), fork == null);
|
||||
gitea.createRepository(project.getGiteaRepoName(), true);
|
||||
project.setGiteaRepoId(repo.id());
|
||||
if (fork != null) {
|
||||
copySnapshot(
|
||||
repository(fork.source()),
|
||||
fork.source().getPublishedCommitSha(),
|
||||
repo,
|
||||
fork.user());
|
||||
if (fork.source().getUrdfMainObjectKey() != null
|
||||
&& !fork.source().getUrdfMainObjectKey().isEmpty())
|
||||
urdf.getObject().copy(fork.source(), project);
|
||||
}
|
||||
project.setRepoStatus("READY");
|
||||
project.setUpdatedAt(now());
|
||||
projects.updateById(project);
|
||||
log.info("仓库创建完成 projectCode={} fork={}", project.getProjectCode(), fork != null);
|
||||
log.info("仓库创建完成 projectCode={} fork={}", project.getProjectCode(), false);
|
||||
return view(project, project.getOwnerUserId());
|
||||
} catch (GiteaFailure error) {
|
||||
project.setRepoStatus(error.uncertain() ? "CREATING" : "FAILED");
|
||||
@@ -304,48 +381,6 @@ public class ProjectServiceImpl implements ProjectService {
|
||||
}
|
||||
}
|
||||
|
||||
private void copySnapshot(
|
||||
GiteaClient.Repository source, String sha, GiteaClient.Repository target, User user) {
|
||||
Deque<String> directories = new ArrayDeque<>();
|
||||
directories.add("");
|
||||
List<GiteaClient.Change> batch = new ArrayList<>();
|
||||
long batchBytes = 0;
|
||||
while (!directories.isEmpty()) {
|
||||
for (TreeEntry entry : gitea.tree(source, sha, directories.removeFirst())) {
|
||||
FilePaths.file(entry.path());
|
||||
if (entry.type().equals("dir")) {
|
||||
directories.add(entry.path());
|
||||
continue;
|
||||
}
|
||||
if (!entry.type().equals("file"))
|
||||
throw BusinessException.badRequest("Fork 暂不支持符号链接或子模块");
|
||||
GiteaClient.StoredFile file = gitea.file(source, sha, entry.path());
|
||||
if (!batch.isEmpty()
|
||||
&& (batchBytes + file.bytes().length > config.getMaxRequestBytes()
|
||||
|| batch.size() >= 100)) {
|
||||
gitea.commit(
|
||||
target,
|
||||
batch,
|
||||
"Fork community snapshot",
|
||||
user.getNickname(),
|
||||
user.getUserCode());
|
||||
batch.clear();
|
||||
batchBytes = 0;
|
||||
}
|
||||
batch.add(new GiteaClient.Change("create", file.path(), null, file.bytes()));
|
||||
batchBytes += file.bytes().length;
|
||||
}
|
||||
}
|
||||
if (!batch.isEmpty())
|
||||
gitea.commit(
|
||||
target,
|
||||
batch,
|
||||
"Fork community snapshot",
|
||||
user.getNickname(),
|
||||
user.getUserCode());
|
||||
else if (batchBytes == 0) throw BusinessException.badRequest("不能 Fork 空仓库");
|
||||
}
|
||||
|
||||
@Override
|
||||
public <T> T write(long userId, String code, Function<Project, T> action) {
|
||||
return locks.write(
|
||||
|
||||
+49
-8
@@ -4,6 +4,7 @@ import com.aliyun.oss.ClientBuilderConfiguration;
|
||||
import com.aliyun.oss.ClientException;
|
||||
import com.aliyun.oss.OSS;
|
||||
import com.aliyun.oss.OSSClientBuilder;
|
||||
import com.aliyun.oss.OSSClient;
|
||||
import com.aliyun.oss.OSSException;
|
||||
import com.aliyun.oss.common.comm.SignVersion;
|
||||
import com.aliyun.oss.common.auth.DefaultCredentialProvider;
|
||||
@@ -11,6 +12,8 @@ import com.aliyun.oss.model.CopyObjectRequest;
|
||||
import com.aliyun.oss.model.ObjectMetadata;
|
||||
import com.linkerbot.community.common.BusinessException;
|
||||
import com.linkerbot.community.service.config.OssProperties;
|
||||
import com.linkerbot.community.service.fork.ForkDeadline;
|
||||
import com.linkerbot.community.service.gitea.GiteaFailure;
|
||||
import jakarta.annotation.PreDestroy;
|
||||
import java.util.List;
|
||||
import lombok.RequiredArgsConstructor;
|
||||
@@ -24,25 +27,54 @@ public class AliyunOssObjectStore implements OssObjectStore {
|
||||
private static final Logger log = LoggerFactory.getLogger(AliyunOssObjectStore.class);
|
||||
private final OssProperties properties;
|
||||
private OSS client;
|
||||
private final ThreadLocal<OSSClient> forkClients = new ThreadLocal<>();
|
||||
|
||||
private synchronized OSS client() {
|
||||
if (!properties.isEnabled())
|
||||
throw new BusinessException(503, "OSS_UNAVAILABLE", "OSS URDF 上传暂未开启");
|
||||
ForkDeadline deadline = ForkDeadline.current();
|
||||
if (deadline != null) {
|
||||
deadline.check();
|
||||
OSSClient bounded = forkClients.get();
|
||||
if (bounded == null) {
|
||||
ClientBuilderConfiguration configuration = configuration();
|
||||
configuration.setRequestTimeoutEnabled(true);
|
||||
configuration.setMaxErrorRetry(0);
|
||||
bounded = (OSSClient) build(configuration);
|
||||
forkClients.set(bounded);
|
||||
OSSClient owned = bounded;
|
||||
deadline.onClose(() -> { forkClients.remove(); owned.shutdown(); });
|
||||
}
|
||||
int remaining = (int) Math.min(30000, Math.max(1, deadline.remaining().toMillis()));
|
||||
bounded.getClientConfiguration().setRequestTimeout(remaining);
|
||||
bounded.getClientConfiguration().setSocketTimeout(remaining);
|
||||
bounded.getClientConfiguration().setConnectionTimeout(Math.min(5000, remaining));
|
||||
bounded.getClientConfiguration().setConnectionRequestTimeout(Math.min(5000, remaining));
|
||||
return bounded;
|
||||
}
|
||||
if (client == null) {
|
||||
ClientBuilderConfiguration configuration = new ClientBuilderConfiguration();
|
||||
configuration.setSignatureVersion(SignVersion.V4);
|
||||
configuration.setConnectionTimeout(5000);
|
||||
configuration.setSocketTimeout(30000);
|
||||
configuration.setMaxErrorRetry(1);
|
||||
client = OSSClientBuilder.create()
|
||||
client = build(configuration());
|
||||
}
|
||||
return client;
|
||||
}
|
||||
|
||||
private ClientBuilderConfiguration configuration() {
|
||||
ClientBuilderConfiguration configuration = new ClientBuilderConfiguration();
|
||||
configuration.setSignatureVersion(SignVersion.V4);
|
||||
configuration.setConnectionTimeout(5000);
|
||||
configuration.setSocketTimeout(30000);
|
||||
configuration.setMaxErrorRetry(1);
|
||||
return configuration;
|
||||
}
|
||||
|
||||
private OSS build(ClientBuilderConfiguration configuration) {
|
||||
return OSSClientBuilder.create()
|
||||
.endpoint(properties.resolvedEndpoint())
|
||||
.region(properties.getRegion())
|
||||
.credentialsProvider(new DefaultCredentialProvider(
|
||||
properties.getAccessKeyId(), properties.getAccessKeySecret()))
|
||||
.clientConfiguration(configuration)
|
||||
.build();
|
||||
}
|
||||
return client;
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -53,20 +85,29 @@ public class AliyunOssObjectStore implements OssObjectStore {
|
||||
} catch (OSSException error) {
|
||||
throw failure("head", objectKey, error);
|
||||
} catch (ClientException error) {
|
||||
ForkDeadline.checkpoint();
|
||||
throw transportFailure("head", objectKey, error);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public void copy(String sourceKey, String targetKey, String sourceEtag) {
|
||||
ForkDeadline deadline = ForkDeadline.current();
|
||||
if (deadline != null) deadline.beginWrite();
|
||||
try {
|
||||
CopyObjectRequest request = new CopyObjectRequest(
|
||||
properties.getBucket(), sourceKey, properties.getBucket(), targetKey);
|
||||
request.setMatchingETagConstraints(List.of(sourceEtag));
|
||||
client().copyObject(request);
|
||||
if (deadline != null) deadline.confirmWrite();
|
||||
} catch (OSSException error) {
|
||||
if (deadline != null) deadline.confirmWrite();
|
||||
throw failure("copy", targetKey, error);
|
||||
} catch (ClientException error) {
|
||||
if (deadline != null) {
|
||||
deadline.check();
|
||||
throw new GiteaFailure(504, "REMOTE_RESULT_UNKNOWN", "Fork OSS 复制结果待核对", true);
|
||||
}
|
||||
throw transportFailure("copy", targetKey, error);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -12,6 +12,7 @@ import com.linkerbot.community.common.Codes;
|
||||
import com.linkerbot.community.common.FilePaths;
|
||||
import com.linkerbot.community.service.config.OssProperties;
|
||||
import com.linkerbot.community.service.config.UrdfProperties;
|
||||
import com.linkerbot.community.service.fork.ForkDeadline;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.LocalDateTime;
|
||||
import java.util.ArrayList;
|
||||
@@ -114,6 +115,7 @@ public class UrdfResources {
|
||||
}
|
||||
|
||||
private OssObjectStore.Metadata checkedMetadata(String prefix, UrdfFile file) {
|
||||
ForkDeadline.checkpoint();
|
||||
OssObjectStore.Metadata metadata = store.head(prefix + file.relativePath());
|
||||
if (metadata.sizeBytes() != file.sizeBytes()) {
|
||||
log.warn("URDF 文件大小不符 objectKey={} expectedBytes={} actualBytes={}",
|
||||
@@ -156,8 +158,10 @@ public class UrdfResources {
|
||||
source.getProjectCode(), target.getProjectCode(), files.size());
|
||||
for (UrdfFile file : files) {
|
||||
OssObjectStore.Metadata metadata = checkedMetadata(sourcePrefix, file);
|
||||
ForkDeadline.checkpoint();
|
||||
store.copy(sourcePrefix + file.relativePath(), targetPrefix + file.relativePath(), metadata.etag());
|
||||
}
|
||||
ForkDeadline.checkpoint();
|
||||
target.setUrdfPrefix(targetPrefix);
|
||||
target.setUrdfMainObjectKey(targetPrefix + main);
|
||||
target.setUrdfFiles(manifest(files));
|
||||
|
||||
+5
-1
@@ -11,6 +11,8 @@ import com.linkerbot.community.common.BusinessException;
|
||||
import com.linkerbot.community.common.ProjectCodes;
|
||||
import com.linkerbot.community.service.config.CommunityProperties;
|
||||
import com.linkerbot.community.service.gitea.GiteaClient;
|
||||
import com.linkerbot.community.service.fork.ForkExecutor;
|
||||
import com.linkerbot.community.service.fork.GitSnapshotCopier;
|
||||
import com.linkerbot.community.service.impl.ProjectServiceImpl;
|
||||
import com.linkerbot.community.service.mapper.*;
|
||||
import com.linkerbot.community.service.oss.UrdfResources;
|
||||
@@ -87,6 +89,8 @@ class ProjectCodeAllocationTest {
|
||||
gitea,
|
||||
new CommunityProperties(),
|
||||
new ProjectLocks(),
|
||||
new StaticListableBeanFactory().getBeanProvider(UrdfResources.class));
|
||||
new StaticListableBeanFactory().getBeanProvider(UrdfResources.class),
|
||||
new StaticListableBeanFactory().getBeanProvider(ForkExecutor.class),
|
||||
new StaticListableBeanFactory().getBeanProvider(GitSnapshotCopier.class));
|
||||
}
|
||||
}
|
||||
|
||||
+333
@@ -0,0 +1,333 @@
|
||||
package com.linkerbot.community.service.fork;
|
||||
|
||||
import static org.assertj.core.api.Assertions.*;
|
||||
import static org.mockito.Mockito.*;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.fasterxml.jackson.databind.node.ArrayNode;
|
||||
import com.fasterxml.jackson.databind.node.ObjectNode;
|
||||
import com.linkerbot.community.common.BusinessException;
|
||||
import com.linkerbot.community.service.config.CommunityProperties;
|
||||
import com.linkerbot.community.service.config.ForkProperties;
|
||||
import com.linkerbot.community.service.gitea.GiteaClient;
|
||||
import com.linkerbot.community.service.gitea.GiteaClient.Repository;
|
||||
import com.sun.net.httpserver.HttpExchange;
|
||||
import com.sun.net.httpserver.HttpServer;
|
||||
import java.io.IOException;
|
||||
import java.net.InetSocketAddress;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.*;
|
||||
import java.security.MessageDigest;
|
||||
import java.time.Duration;
|
||||
import java.util.*;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
import org.junit.jupiter.api.*;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.ValueSource;
|
||||
|
||||
class CliGitSnapshotCopierTest {
|
||||
@TempDir Path root;
|
||||
private final ObjectMapper json = new ObjectMapper();
|
||||
private final Map<String, Map<String, byte[]>> objects = new ConcurrentHashMap<>();
|
||||
private final AtomicInteger unexpectedRequests = new AtomicInteger();
|
||||
private HttpServer lfs;
|
||||
private String origin;
|
||||
private ForkProperties properties;
|
||||
private CommunityProperties community;
|
||||
private GitCommandRunner commands;
|
||||
private GiteaClient gitea;
|
||||
private CliGitSnapshotCopier copier;
|
||||
private volatile boolean rejectTargetDownload;
|
||||
private volatile boolean redirectLfs;
|
||||
private volatile boolean externalAction;
|
||||
private volatile boolean rejectGitPush;
|
||||
|
||||
@BeforeEach void setup() throws Exception {
|
||||
lfs = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
|
||||
lfs.setExecutor(Executors.newVirtualThreadPerTaskExecutor());
|
||||
origin = "http://127.0.0.1:" + lfs.getAddress().getPort();
|
||||
lfs.createContext("/", this::serveLfs);
|
||||
lfs.start();
|
||||
properties = new ForkProperties();
|
||||
properties.setTemporaryDirectory(root.resolve("jobs").toString());
|
||||
community = new CommunityProperties();
|
||||
community.getGitea().setBaseUrl(origin);
|
||||
community.getGitea().setToken("fixture-token");
|
||||
commands = new GitCommandRunner(properties);
|
||||
gitea = mock(GiteaClient.class);
|
||||
when(gitea.authenticatedUsername()).thenReturn("fixture");
|
||||
copier = new CliGitSnapshotCopier(properties, community, commands, gitea, json) {
|
||||
@Override protected void verifyRemote(Repository target, String commit, ForkDeadline deadline) {}
|
||||
@Override protected String repositoryUrl(Repository repo) {
|
||||
return rejectGitPush && repo.name().equals("target") ? origin + "/denied.git" : repository(repo.name()).toUri().toString();
|
||||
}
|
||||
@Override protected Map<String, String> configuration(String endpoint) {
|
||||
String name = endpoint.contains("source.git") ? "source" : "target";
|
||||
Map<String, String> settings = super.configuration(origin + "/" + name + "/info/lfs");
|
||||
settings.put("protocol.file.allow", "always");
|
||||
return settings;
|
||||
}
|
||||
};
|
||||
copier.checkAvailable();
|
||||
git(repository("source"), null, "init", "--bare", "--quiet");
|
||||
git(repository("target"), null, "init", "--bare", "--quiet");
|
||||
}
|
||||
|
||||
@AfterEach void stop() { if (lfs != null) lfs.stop(0); }
|
||||
|
||||
private Path repository(String name) { return root.resolve(name + ".git"); }
|
||||
private ForkDeadline deadline() {
|
||||
ForkDeadline deadline = new ForkDeadline(Duration.ofSeconds(25));
|
||||
deadline.projectCode("111111111111");
|
||||
return deadline;
|
||||
}
|
||||
|
||||
private String git(Path repository, byte[] input, String... args) throws Exception {
|
||||
List<String> command = new ArrayList<>(List.of("git", "--git-dir=" + repository));
|
||||
command.addAll(List.of(args));
|
||||
ProcessBuilder builder = new ProcessBuilder(command);
|
||||
builder.environment().putAll(Map.of("GIT_CONFIG_NOSYSTEM", "1", "GIT_CONFIG_GLOBAL", "/dev/null",
|
||||
"GIT_AUTHOR_NAME", "Fixture", "GIT_AUTHOR_EMAIL", "fixture@users.community.invalid",
|
||||
"GIT_COMMITTER_NAME", "Fixture", "GIT_COMMITTER_EMAIL", "fixture@users.community.invalid"));
|
||||
Process process = builder.start();
|
||||
if (input != null) process.getOutputStream().write(input);
|
||||
process.getOutputStream().close();
|
||||
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8).trim();
|
||||
String error = new String(process.getErrorStream().readAllBytes(), StandardCharsets.UTF_8);
|
||||
assertThat(process.waitFor(15, TimeUnit.SECONDS)).as("fixture Git completed").isTrue();
|
||||
assertThat(process.exitValue()).as("fixture Git: %s", error).isZero();
|
||||
return output;
|
||||
}
|
||||
|
||||
private String blob(byte[] bytes) throws Exception { return git(repository("source"), bytes, "hash-object", "-w", "--stdin"); }
|
||||
private String tree(String entries) throws Exception { return git(repository("source"), entries.getBytes(StandardCharsets.UTF_8), "mktree"); }
|
||||
private String commit(String tree, String parent, String message) throws Exception {
|
||||
return parent == null ? git(repository("source"), null, "commit-tree", tree, "-m", message)
|
||||
: git(repository("source"), null, "commit-tree", tree, "-p", parent, "-m", message);
|
||||
}
|
||||
private String simplePublication() throws Exception {
|
||||
String tree = tree("100644 blob " + blob("published".getBytes(StandardCharsets.UTF_8)) + "\tREADME.md\n");
|
||||
String sha = commit(tree, null, "published");
|
||||
git(repository("source"), null, "update-ref", "refs/heads/main", sha);
|
||||
return sha;
|
||||
}
|
||||
private void assertClean() throws IOException {
|
||||
try (java.util.stream.Stream<Path> paths = Files.list(root.resolve("jobs"))) { assertThat(paths.toList()).isEmpty(); }
|
||||
}
|
||||
|
||||
@Test void preservesPublishedTreeLargeFilesModesAndGitlinksWithoutHistory() throws Exception {
|
||||
String initial = simplePublication();
|
||||
String normal = blob("published".getBytes(StandardCharsets.UTF_8));
|
||||
String large = blob(new byte[11 * 1024 * 1024]);
|
||||
String link = blob("README.md".getBytes(StandardCharsets.UTF_8));
|
||||
String subtree = tree("160000 commit " + "a".repeat(40) + "\tdrive_base\n");
|
||||
String publishedTree = tree("100644 blob " + normal + "\tREADME.md\n100644 blob " + normal
|
||||
+ "\t说明.txt\n100755 blob " + normal + "\trun.sh\n100644 blob " + large
|
||||
+ "\tassembly.SLDASM\n120000 blob " + link + "\treadme-link\n040000 tree " + subtree + "\turos\n");
|
||||
String published = commit(publishedTree, initial, "published special files");
|
||||
String changedTree = tree("100644 blob " + blob("unpublished private content".getBytes(StandardCharsets.UTF_8)) + "\tprivate.txt\n");
|
||||
String unpublished = commit(changedTree, published, "not published");
|
||||
git(repository("source"), null, "update-ref", "refs/heads/main", unpublished);
|
||||
git(repository("source"), null, "update-ref", "refs/tags/old", initial);
|
||||
String result;
|
||||
try (GitSnapshotCopier.PreparedSnapshot snapshot = copier.prepare(new Repository(1, "community", "source"), published,
|
||||
"测试用户", "public-code", deadline())) {
|
||||
try (java.util.stream.Stream<Path> paths = Files.list(root.resolve("jobs"))) {
|
||||
String config = Files.readString(paths.findFirst().orElseThrow().resolve("config"));
|
||||
assertThat(config).doesNotContain("fixture-token", "Authorization", "Basic ");
|
||||
}
|
||||
result = snapshot.push(new Repository(2, "community", "target"));
|
||||
}
|
||||
assertThat(git(repository("target"), null, "rev-parse", "main^{tree}")).isEqualTo(publishedTree);
|
||||
assertThat(repository("target").resolve("shallow")).doesNotExist();
|
||||
assertThat(git(repository("target"), null, "rev-list", "--parents", "main")).isEqualTo(result);
|
||||
assertThat(git(repository("target"), null, "for-each-ref", "--format=%(refname)")).isEqualTo("refs/heads/main");
|
||||
assertThat(git(repository("target"), null, "ls-tree", "-r", "main")).contains("100755 blob", "120000 blob", "160000 commit");
|
||||
assertThat(git(repository("target"), null, "log", "--format=%ae", "main")).isEqualTo("public-code@users.community.invalid");
|
||||
for (String original : List.of(initial, published, unpublished)) {
|
||||
Process check = new ProcessBuilder("git", "--git-dir=" + repository("target"), "cat-file", "-e", original)
|
||||
.redirectError(ProcessBuilder.Redirect.DISCARD).start();
|
||||
assertThat(check.waitFor()).isNotZero();
|
||||
}
|
||||
assertClean();
|
||||
}
|
||||
|
||||
@Test void supportsIndependentRepeatedCopiesOfOneSource() throws Exception {
|
||||
String sha = simplePublication();
|
||||
for (String target : List.of("target", "another")) {
|
||||
git(repository(target), null, "init", "--bare", "--quiet");
|
||||
try (GitSnapshotCopier.PreparedSnapshot snapshot = copier.prepare(new Repository(1, "community", "source"), sha,
|
||||
target, target, deadline())) {
|
||||
snapshot.push(new Repository(2, "community", target));
|
||||
}
|
||||
assertThat(git(repository(target), null, "rev-list", "--count", "main")).isEqualTo("1");
|
||||
}
|
||||
assertClean();
|
||||
}
|
||||
|
||||
private String lfsPublication(byte[] bytes, boolean storeSource) throws Exception {
|
||||
String oid = HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(bytes));
|
||||
if (storeSource) objects.computeIfAbsent("source", ignored -> new ConcurrentHashMap<>()).put(oid, bytes);
|
||||
String pointer = "version https://git-lfs.github.com/spec/v1\noid sha256:" + oid + "\nsize " + bytes.length + "\n";
|
||||
String attributes = blob("big.bin filter=lfs diff=lfs merge=lfs -text\n".getBytes(StandardCharsets.UTF_8));
|
||||
String malicious = blob(("[lfs]\nurl = " + origin + "/forbidden\nfetchinclude = no-such-file\nfetchexclude = *\n").getBytes(StandardCharsets.UTF_8));
|
||||
String tree = tree("100644 blob " + attributes + "\t.gitattributes\n100644 blob " + malicious + "\t.lfsconfig\n100644 blob "
|
||||
+ blob(pointer.getBytes(StandardCharsets.UTF_8)) + "\tbig.bin\n");
|
||||
String sha = commit(tree, null, "LFS publication");
|
||||
git(repository("source"), null, "update-ref", "refs/heads/main", sha);
|
||||
return sha;
|
||||
}
|
||||
|
||||
@Test void transfersLfsEntitiesAndVerifiesDownloadFromTargetIgnoringSourceLfsconfig() throws Exception {
|
||||
byte[] bytes = new byte[128 * 1024];
|
||||
new Random(42).nextBytes(bytes);
|
||||
String sha = lfsPublication(bytes, true);
|
||||
try (GitSnapshotCopier.PreparedSnapshot snapshot = copier.prepare(new Repository(1, "community", "source"), sha,
|
||||
"测试用户", "public-code", deadline())) {
|
||||
snapshot.push(new Repository(2, "community", "target"));
|
||||
}
|
||||
assertThat(objects.get("target").values()).singleElement().satisfies(actual -> assertThat(actual).isEqualTo(bytes));
|
||||
assertThat(unexpectedRequests).hasValue(0);
|
||||
assertThat(git(repository("target"), null, "rev-parse", "main^{tree}"))
|
||||
.isEqualTo(git(repository("source"), null, "rev-parse", sha + "^{tree}"));
|
||||
assertClean();
|
||||
}
|
||||
|
||||
@Test void refusesMissingLfsObjectBeforeAnyTargetPush() throws Exception {
|
||||
String sha = lfsPublication("missing entity".getBytes(StandardCharsets.UTF_8), false);
|
||||
assertThatThrownBy(() -> copier.prepare(new Repository(1, "community", "source"), sha, "Fixture", "code", deadline()))
|
||||
.isInstanceOfSatisfying(BusinessException.class, error -> assertThat(error.code()).isEqualTo("FORK_COPY_FAILED"));
|
||||
assertThat(objects.get("target")).isNull();
|
||||
assertClean();
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@ValueSource(booleans = {false, true})
|
||||
void refusesLfsRedirectsAndActionsOutsideConfiguredGitea(boolean redirect) throws Exception {
|
||||
String sha = lfsPublication("restricted entity".getBytes(StandardCharsets.UTF_8), true);
|
||||
redirectLfs = redirect;
|
||||
externalAction = !redirect;
|
||||
assertThatThrownBy(() -> copier.prepare(new Repository(1, "community", "source"), sha, "Fixture", "code", deadline()))
|
||||
.isInstanceOfSatisfying(BusinessException.class, error -> assertThat(error.code()).isEqualTo("FORK_COPY_FAILED"));
|
||||
assertThat(unexpectedRequests).hasValue(0);
|
||||
assertClean();
|
||||
}
|
||||
|
||||
@Test void waitsForGiteaMetadataBeforeReadingTheFirstBranchAndChecksExactCommit() {
|
||||
String sha = "a".repeat(40);
|
||||
when(gitea.inspect("target")).thenReturn(
|
||||
new GiteaClient.RemoteRepository(2, "community", "target", true, false, true, "", "main"),
|
||||
new GiteaClient.RemoteRepository(2, "community", "target", true, false, false, "", "main"));
|
||||
when(gitea.head(any())).thenReturn(sha);
|
||||
CliGitSnapshotCopier real = new CliGitSnapshotCopier(properties, community, commands, gitea, json);
|
||||
real.verifyRemote(new Repository(2, "community", "target"), sha, deadline());
|
||||
verify(gitea, times(2)).inspect("target");
|
||||
verify(gitea).head(any());
|
||||
when(gitea.head(any())).thenReturn("b".repeat(40));
|
||||
assertThatThrownBy(() -> real.verifyRemote(new Repository(2, "community", "target"), sha, deadline()))
|
||||
.isInstanceOfSatisfying(BusinessException.class, error -> assertThat(error.code()).isEqualTo("FORK_COPY_FAILED"));
|
||||
}
|
||||
|
||||
@Test void refusesSuccessIfTargetLfsCannotBeDownloaded() throws Exception {
|
||||
String sha = lfsPublication("target download must work".getBytes(StandardCharsets.UTF_8), true);
|
||||
rejectTargetDownload = true;
|
||||
ForkDeadline deadline = deadline();
|
||||
try (GitSnapshotCopier.PreparedSnapshot snapshot = copier.prepare(new Repository(1, "community", "source"), sha,
|
||||
"Fixture", "code", deadline)) {
|
||||
assertThatThrownBy(() -> snapshot.push(new Repository(2, "community", "target")))
|
||||
.isInstanceOfSatisfying(BusinessException.class, error -> assertThat(error.code()).isEqualTo("FORK_COPY_FAILED"));
|
||||
assertThat(deadline.writing()).isFalse();
|
||||
}
|
||||
assertClean();
|
||||
}
|
||||
|
||||
@Test void rejectsUnknownShaAndDiskQuotaAndCleansTemporaryData() throws Exception {
|
||||
String sha = simplePublication();
|
||||
assertThatThrownBy(() -> copier.prepare(new Repository(1, "community", "source"), "b".repeat(40), "Fixture", "code", deadline()))
|
||||
.isInstanceOf(BusinessException.class);
|
||||
assertClean();
|
||||
properties.setMaxTemporaryBytes(1);
|
||||
assertThatThrownBy(() -> copier.prepare(new Repository(1, "community", "source"), sha, "Fixture", "code", deadline()))
|
||||
.isInstanceOfSatisfying(BusinessException.class, error -> assertThat(error.code()).isEqualTo("FORK_RESOURCE_LIMIT"));
|
||||
assertClean();
|
||||
}
|
||||
|
||||
@Test void rejectsMissingBinaryAndAuthenticationFailure() throws Exception {
|
||||
properties.setGitExecutable(root.resolve("missing-git").toString());
|
||||
assertThatThrownBy(copier::checkAvailable).isInstanceOfSatisfying(BusinessException.class,
|
||||
error -> assertThat(error.code()).isEqualTo("FORK_UNAVAILABLE"));
|
||||
properties.setGitExecutable("git");
|
||||
CliGitSnapshotCopier http = new CliGitSnapshotCopier(properties, community, commands, gitea, json);
|
||||
assertThatThrownBy(() -> http.prepare(new Repository(1, "community", "source"), "c".repeat(40), "Fixture", "code", deadline()))
|
||||
.isInstanceOfSatisfying(BusinessException.class, error -> assertThat(error.code()).isEqualTo("FORK_COPY_FAILED"));
|
||||
assertClean();
|
||||
}
|
||||
|
||||
@Test void marksExplicitPushAuthenticationRejectionAsDefiniteFailure() throws Exception {
|
||||
String sha = simplePublication();
|
||||
rejectGitPush = true;
|
||||
ForkDeadline deadline = deadline();
|
||||
try (GitSnapshotCopier.PreparedSnapshot snapshot = copier.prepare(new Repository(1, "community", "source"), sha,
|
||||
"Fixture", "code", deadline)) {
|
||||
assertThatThrownBy(() -> snapshot.push(new Repository(2, "community", "target")))
|
||||
.isInstanceOfSatisfying(BusinessException.class, error -> assertThat(error.code()).isEqualTo("FORK_COPY_FAILED"));
|
||||
assertThat(deadline.writing()).isFalse();
|
||||
}
|
||||
assertClean();
|
||||
}
|
||||
|
||||
private void serveLfs(HttpExchange exchange) throws IOException {
|
||||
try (HttpExchange request = exchange) {
|
||||
String path = request.getRequestURI().getPath();
|
||||
String[] parts = path.split("/");
|
||||
if (parts.length < 3 || !Set.of("source", "target").contains(parts[1])) {
|
||||
unexpectedRequests.incrementAndGet();
|
||||
request.sendResponseHeaders(401, -1);
|
||||
return;
|
||||
}
|
||||
String repository = parts[1];
|
||||
Map<String, byte[]> stored = objects.computeIfAbsent(repository, ignored -> new ConcurrentHashMap<>());
|
||||
if (path.endsWith("/objects/batch")) {
|
||||
if (redirectLfs) {
|
||||
request.getResponseHeaders().set("Location", origin + "/forbidden");
|
||||
request.sendResponseHeaders(307, -1);
|
||||
return;
|
||||
}
|
||||
JsonNode input = json.readTree(request.getRequestBody());
|
||||
ObjectNode response = json.createObjectNode().put("transfer", "basic");
|
||||
ArrayNode entries = response.putArray("objects");
|
||||
for (JsonNode object : input.path("objects")) {
|
||||
String oid = object.path("oid").asText();
|
||||
ObjectNode entry = entries.addObject().put("oid", oid).put("size", object.path("size").asLong());
|
||||
boolean download = input.path("operation").asText().equals("download");
|
||||
if (download && (!stored.containsKey(oid) || repository.equals("target") && rejectTargetDownload)) {
|
||||
entry.putObject("error").put("code", 404).put("message", "fixture missing object");
|
||||
} else if (download || !stored.containsKey(oid)) {
|
||||
entry.putObject("actions").putObject(download ? "download" : "upload")
|
||||
.put("href", externalAction ? origin.replace("127.0.0.1", "localhost") + "/forbidden"
|
||||
: origin + "/" + repository + "/objects/" + oid);
|
||||
}
|
||||
}
|
||||
byte[] body = json.writeValueAsBytes(response);
|
||||
request.getResponseHeaders().set("Content-Type", "application/vnd.git-lfs+json");
|
||||
request.sendResponseHeaders(200, body.length);
|
||||
request.getResponseBody().write(body);
|
||||
} else if (path.contains("/objects/")) {
|
||||
String oid = parts[parts.length - 1];
|
||||
if (request.getRequestMethod().equals("PUT")) {
|
||||
stored.put(oid, request.getRequestBody().readAllBytes());
|
||||
request.sendResponseHeaders(200, -1);
|
||||
} else if (stored.containsKey(oid)) {
|
||||
byte[] bytes = stored.get(oid);
|
||||
request.sendResponseHeaders(200, bytes.length);
|
||||
request.getResponseBody().write(bytes);
|
||||
} else request.sendResponseHeaders(404, -1);
|
||||
} else request.sendResponseHeaders(404, -1);
|
||||
}
|
||||
}
|
||||
}
|
||||
+84
@@ -0,0 +1,84 @@
|
||||
package com.linkerbot.community.service.fork;
|
||||
|
||||
import static org.assertj.core.api.Assertions.*;
|
||||
import static org.awaitility.Awaitility.await;
|
||||
import static org.mockito.Mockito.*;
|
||||
|
||||
import com.linkerbot.community.common.BusinessException;
|
||||
import com.linkerbot.community.service.config.ForkProperties;
|
||||
import java.time.Duration;
|
||||
import java.util.concurrent.*;
|
||||
import java.util.concurrent.atomic.AtomicBoolean;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.ValueSource;
|
||||
import org.slf4j.MDC;
|
||||
|
||||
class ForkExecutorTest {
|
||||
@Test void unavailableRuntimeRejectsBeforeExecutingBusinessWork() {
|
||||
GitSnapshotCopier copier = mock(GitSnapshotCopier.class);
|
||||
doThrow(new BusinessException(503, "FORK_UNAVAILABLE", "unavailable")).when(copier).checkAvailable();
|
||||
ForkExecutor executor = new ForkExecutor(new ForkProperties(), copier);
|
||||
AtomicBoolean called = new AtomicBoolean();
|
||||
try {
|
||||
executor.inspectRuntime();
|
||||
assertThatThrownBy(() -> executor.execute(deadline -> { called.set(true); return 1; }))
|
||||
.isInstanceOfSatisfying(BusinessException.class, error -> assertThat(error.code()).isEqualTo("FORK_UNAVAILABLE"));
|
||||
assertThat(called).isFalse();
|
||||
} finally { executor.close(); }
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@ValueSource(booleans = {false, true})
|
||||
void timeoutRetainsCapacityUntilActualWorkAndCleanupEnd(boolean writing) throws Exception {
|
||||
ForkProperties properties = new ForkProperties();
|
||||
properties.setTimeout(Duration.ofMillis(150));
|
||||
properties.setMaxConcurrent(1);
|
||||
ForkExecutor executor = new ForkExecutor(properties, mock(GitSnapshotCopier.class));
|
||||
CountDownLatch entered = new CountDownLatch(1);
|
||||
CountDownLatch interrupted = new CountDownLatch(1);
|
||||
CountDownLatch release = new CountDownLatch(1);
|
||||
AtomicBoolean cleaned = new AtomicBoolean();
|
||||
try (ExecutorService callers = Executors.newVirtualThreadPerTaskExecutor()) {
|
||||
Future<BusinessException> result = callers.submit(() -> {
|
||||
try {
|
||||
executor.execute(deadline -> {
|
||||
deadline.projectCode("111111111111");
|
||||
deadline.onClose(() -> cleaned.set(true));
|
||||
if (writing) deadline.beginWrite();
|
||||
entered.countDown();
|
||||
while (release.getCount() != 0) {
|
||||
try { release.await(); }
|
||||
catch (InterruptedException error) { interrupted.countDown(); }
|
||||
}
|
||||
return 1;
|
||||
});
|
||||
throw new AssertionError("timeout expected");
|
||||
} catch (BusinessException error) { return error; }
|
||||
});
|
||||
assertThat(entered.await(2, TimeUnit.SECONDS)).isTrue();
|
||||
BusinessException error = result.get(2, TimeUnit.SECONDS);
|
||||
assertThat(error.code()).isEqualTo(writing ? "REMOTE_RESULT_UNKNOWN" : "FORK_TIMEOUT");
|
||||
assertThat(error.details()).isEqualTo(java.util.Map.of("projectCode", "111111111111"));
|
||||
assertThat(interrupted.await(2, TimeUnit.SECONDS)).isTrue();
|
||||
assertThat(cleaned).isFalse();
|
||||
assertThatThrownBy(() -> executor.execute(deadline -> 2))
|
||||
.isInstanceOfSatisfying(BusinessException.class, busy -> assertThat(busy.code()).isEqualTo("FORK_BUSY"));
|
||||
release.countDown();
|
||||
await().atMost(Duration.ofSeconds(2)).untilAsserted(() -> assertThat(executor.<Integer>execute(deadline -> 3)).isEqualTo(3));
|
||||
assertThat(cleaned).isTrue();
|
||||
} finally { release.countDown(); executor.close(); }
|
||||
}
|
||||
|
||||
@Test void propagatesRequestContextAndDoesNotBindDeadlineToCaller() {
|
||||
ForkExecutor executor = new ForkExecutor(new ForkProperties(), mock(GitSnapshotCopier.class));
|
||||
MDC.put("requestId", "fixture-request");
|
||||
try {
|
||||
assertThat(executor.<String>execute(deadline -> {
|
||||
assertThat(ForkDeadline.current()).isSameAs(deadline);
|
||||
return MDC.get("requestId");
|
||||
})).isEqualTo("fixture-request");
|
||||
assertThat(ForkDeadline.current()).isNull();
|
||||
} finally { MDC.remove("requestId"); executor.close(); }
|
||||
}
|
||||
}
|
||||
+62
@@ -0,0 +1,62 @@
|
||||
package com.linkerbot.community.service.fork;
|
||||
|
||||
import static org.assertj.core.api.Assertions.*;
|
||||
import static org.awaitility.Awaitility.await;
|
||||
|
||||
import com.linkerbot.community.common.BusinessException;
|
||||
import com.linkerbot.community.service.config.ForkProperties;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.nio.file.attribute.PosixFilePermissions;
|
||||
import java.time.Duration;
|
||||
import java.util.Map;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
|
||||
class GitCommandRunnerTest {
|
||||
@TempDir Path root;
|
||||
|
||||
private GitCommandRunner script(String content) throws Exception {
|
||||
Path executable = root.resolve("fixture-git");
|
||||
Files.writeString(executable, "#!/bin/sh\n" + content);
|
||||
Files.setPosixFilePermissions(executable, PosixFilePermissions.fromString("rwx------"));
|
||||
ForkProperties properties = new ForkProperties();
|
||||
properties.setGitExecutable(executable.toString());
|
||||
return new GitCommandRunner(properties);
|
||||
}
|
||||
|
||||
@Test void timeoutTerminatesTheProcessAndItsChild() throws Exception {
|
||||
Path pid = root.resolve("child.pid");
|
||||
GitCommandRunner runner = script("sleep 60 &\nchild=$!\nprintf '%s' \"$child\" > '" + pid + "'\nwait \"$child\"\n");
|
||||
ForkDeadline deadline = new ForkDeadline(Duration.ofSeconds(10));
|
||||
try (java.util.concurrent.ExecutorService worker = java.util.concurrent.Executors.newVirtualThreadPerTaskExecutor()) {
|
||||
java.util.concurrent.Future<?> task = worker.submit(() -> runner.run(null, deadline, "blocking", Map.of(), Map.of(), "--version"));
|
||||
await().atMost(Duration.ofSeconds(5)).until(() -> Files.exists(pid) && Files.size(pid) > 0);
|
||||
deadline.cancel("FORK_TIMEOUT");
|
||||
assertThatThrownBy(() -> task.get(3, java.util.concurrent.TimeUnit.SECONDS))
|
||||
.isInstanceOf(java.util.concurrent.ExecutionException.class)
|
||||
.hasCauseInstanceOf(BusinessException.class)
|
||||
.satisfies(error -> assertThat(((BusinessException) error.getCause()).code()).isEqualTo("FORK_TIMEOUT"));
|
||||
}
|
||||
long child = Long.parseLong(Files.readString(pid));
|
||||
await().atMost(Duration.ofSeconds(2)).untilAsserted(() ->
|
||||
assertThat(ProcessHandle.of(child).map(ProcessHandle::isAlive).orElse(false)).isFalse());
|
||||
}
|
||||
|
||||
@Test void limitsUntrustedProcessOutput() throws Exception {
|
||||
GitCommandRunner runner = script("exec head -c 16000000 /dev/zero\n");
|
||||
assertThatThrownBy(() -> runner.run(null, new ForkDeadline(Duration.ofSeconds(5)), "verbose", Map.of(), Map.of(), "--version"))
|
||||
.isInstanceOfSatisfying(BusinessException.class, error -> assertThat(error.code()).isEqualTo("FORK_RESOURCE_LIMIT"));
|
||||
}
|
||||
|
||||
@Test void putsCredentialsInEnvironmentWithoutArgumentsOrInheritedGitConfiguration() throws Exception {
|
||||
GitCommandRunner runner = script("printf '%s|%s|%s' \"$*\" \"$GIT_CONFIG_VALUE_6\" \"$GIT_CONFIG_GLOBAL\"\n");
|
||||
GitCommandRunner.Result result = runner.run(null, new ForkDeadline(Duration.ofSeconds(3)), "environment",
|
||||
Map.of("http.http://fixture.invalid/.extraHeader", "Authorization: Basic fixture-only"), Map.of(), "ls-remote", "http://fixture.invalid/repo.git");
|
||||
assertThat(result.exitCode()).isZero();
|
||||
String[] parts = result.output().split("\\|", -1);
|
||||
assertThat(parts[0]).doesNotContain("Authorization", "fixture-only");
|
||||
assertThat(parts[1]).isEqualTo("Authorization: Basic fixture-only");
|
||||
assertThat(parts[2]).isEqualTo("/dev/null");
|
||||
}
|
||||
}
|
||||
+96
@@ -0,0 +1,96 @@
|
||||
package com.linkerbot.community.service.fork;
|
||||
|
||||
import static org.assertj.core.api.Assertions.*;
|
||||
import static org.mockito.ArgumentMatchers.*;
|
||||
import static org.mockito.Mockito.*;
|
||||
|
||||
import com.linkerbot.community.bean.Contracts.ForkInput;
|
||||
import com.linkerbot.community.bean.entity.Project;
|
||||
import com.linkerbot.community.bean.entity.User;
|
||||
import com.linkerbot.community.common.BusinessException;
|
||||
import com.linkerbot.community.service.UserService;
|
||||
import com.linkerbot.community.service.config.CommunityProperties;
|
||||
import com.linkerbot.community.service.config.ForkProperties;
|
||||
import com.linkerbot.community.service.gitea.GiteaClient;
|
||||
import com.linkerbot.community.service.impl.ProjectServiceImpl;
|
||||
import com.linkerbot.community.service.mapper.*;
|
||||
import com.linkerbot.community.service.oss.UrdfResources;
|
||||
import com.linkerbot.community.service.support.ProjectLocks;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.atomic.AtomicReference;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.ValueSource;
|
||||
import org.springframework.beans.factory.support.StaticListableBeanFactory;
|
||||
|
||||
class ProjectForkStateTest {
|
||||
@ParameterizedTest
|
||||
@ValueSource(strings = {"initial", "association", "completion"})
|
||||
void databaseWriteFailuresRemainUncertainWithCode(String stage) {
|
||||
boolean completion = stage.equals("completion");
|
||||
Project source = new Project();
|
||||
source.setId(1L);
|
||||
source.setProjectCode("111111111111");
|
||||
source.setPublishedCommitSha("a".repeat(40));
|
||||
source.setProjectType("GENERAL");
|
||||
source.setName("来源");
|
||||
source.setDescription("");
|
||||
source.setGiteaRepoId(1L);
|
||||
source.setGiteaOwner("community");
|
||||
source.setGiteaRepoName("p_111111111111");
|
||||
ProjectMapper projects = mock(ProjectMapper.class);
|
||||
when(projects.selectOne(any())).thenReturn(source);
|
||||
AtomicReference<Project> target = new AtomicReference<>();
|
||||
when(projects.insert(any(Project.class))).thenAnswer(call -> {
|
||||
Project project = call.getArgument(0);
|
||||
project.setId(2L);
|
||||
target.set(project);
|
||||
if (stage.equals("initial")) throw new IllegalStateException("simulated insertion receipt lost");
|
||||
return 1;
|
||||
});
|
||||
AtomicReference<Boolean> failed = new AtomicReference<>(false);
|
||||
when(projects.updateById(any(Project.class))).thenAnswer(call -> {
|
||||
Project project = call.getArgument(0);
|
||||
if (!failed.get() && (completion == "READY".equals(project.getRepoStatus()))) {
|
||||
failed.set(true);
|
||||
throw new IllegalStateException("simulated database write failure");
|
||||
}
|
||||
return 1;
|
||||
});
|
||||
User user = new User();
|
||||
user.setNickname("测试用户");
|
||||
user.setUserCode("user-code");
|
||||
UserService users = mock(UserService.class);
|
||||
when(users.require(1)).thenReturn(user);
|
||||
GiteaClient gitea = mock(GiteaClient.class);
|
||||
when(gitea.createRepository(anyString(), eq(false))).thenReturn(new GiteaClient.Repository(2, "community", "target"));
|
||||
GitSnapshotCopier copier = mock(GitSnapshotCopier.class);
|
||||
GitSnapshotCopier.PreparedSnapshot snapshot = mock(GitSnapshotCopier.PreparedSnapshot.class);
|
||||
when(copier.prepare(any(), anyString(), anyString(), anyString(), any())).thenReturn(snapshot);
|
||||
when(snapshot.push(any())).thenReturn("b".repeat(40));
|
||||
ForkExecutor executor = new ForkExecutor(new ForkProperties(), copier);
|
||||
StaticListableBeanFactory factory = new StaticListableBeanFactory();
|
||||
factory.addBean("forks", executor);
|
||||
factory.addBean("snapshots", copier);
|
||||
ProjectLocks locks = new ProjectLocks();
|
||||
ProjectServiceImpl service = new ProjectServiceImpl(projects, mock(LikeMapper.class), mock(CommentMapper.class), users,
|
||||
gitea, new CommunityProperties(), locks, factory.getBeanProvider(UrdfResources.class),
|
||||
factory.getBeanProvider(ForkExecutor.class), factory.getBeanProvider(GitSnapshotCopier.class));
|
||||
try {
|
||||
assertThatThrownBy(() -> service.fork(1, source.getProjectCode(), new ForkInput(source.getPublishedCommitSha(), "副本")))
|
||||
.isInstanceOfSatisfying(BusinessException.class, error -> {
|
||||
assertThat(error.code()).isEqualTo("PROJECT_INITIALIZATION_ERROR");
|
||||
assertThat(error.details()).isEqualTo(Map.of("projectCode", target.get().getProjectCode()));
|
||||
});
|
||||
assertThat(target.get().getRepoStatus()).isEqualTo("CREATING");
|
||||
assertThat(target.get().getGiteaRepoId()).isEqualTo(stage.equals("initial") ? null : 2L);
|
||||
assertThatThrownBy(() -> locks.write(target.get().getProjectCode(), () -> true))
|
||||
.isInstanceOfSatisfying(BusinessException.class, error -> assertThat(error.code()).isEqualTo("REMOTE_RESULT_UNKNOWN"));
|
||||
if (completion) verify(snapshot).push(any());
|
||||
else verify(snapshot, never()).push(any());
|
||||
if (stage.equals("initial")) {
|
||||
verify(copier, never()).prepare(any(), anyString(), anyString(), anyString(), any());
|
||||
verify(snapshot, never()).close();
|
||||
} else verify(snapshot).close();
|
||||
} finally { executor.close(); }
|
||||
}
|
||||
}
|
||||
+27
@@ -4,6 +4,12 @@ import static org.assertj.core.api.Assertions.*;
|
||||
|
||||
import com.linkerbot.community.common.BusinessException;
|
||||
import com.linkerbot.community.service.config.OssProperties;
|
||||
import com.linkerbot.community.service.config.ForkProperties;
|
||||
import com.linkerbot.community.service.fork.ForkExecutor;
|
||||
import com.linkerbot.community.service.fork.GitSnapshotCopier;
|
||||
import java.time.Duration;
|
||||
import static org.awaitility.Awaitility.await;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import com.sun.net.httpserver.HttpExchange;
|
||||
import com.sun.net.httpserver.HttpServer;
|
||||
import java.io.IOException;
|
||||
@@ -16,6 +22,7 @@ import org.junit.jupiter.api.Test;
|
||||
|
||||
/** 使用本机 HTTP 替身验证真实 SDK 的 V4 请求和 XML 解析,不访问阿里云。 */
|
||||
class AliyunOssObjectStoreTest {
|
||||
private volatile long responseDelayMs;
|
||||
private HttpServer server;
|
||||
private AliyunOssObjectStore store;
|
||||
private final AtomicReference<String> copySource = new AtomicReference<>();
|
||||
@@ -39,6 +46,10 @@ class AliyunOssObjectStoreTest {
|
||||
|
||||
private void respond(HttpExchange exchange) throws IOException {
|
||||
try (HttpExchange request = exchange) {
|
||||
if (responseDelayMs > 0) {
|
||||
try { Thread.sleep(responseDelayMs); }
|
||||
catch (InterruptedException error) { Thread.currentThread().interrupt(); }
|
||||
}
|
||||
authorization.set(request.getRequestHeaders().getFirst("Authorization"));
|
||||
request.getResponseHeaders().set("x-oss-request-id", "fake-request-id");
|
||||
if (request.getRequestMethod().equals("HEAD")) {
|
||||
@@ -68,6 +79,22 @@ class AliyunOssObjectStoreTest {
|
||||
if (server != null) server.stop(0);
|
||||
}
|
||||
|
||||
@Test
|
||||
void forkBudgetBoundsSdkRequestsAndClosesItsDedicatedClient() {
|
||||
store.head("vibemake_dev/urdf/warm/main.urdf");
|
||||
ForkProperties fork = new ForkProperties();
|
||||
fork.setTimeout(Duration.ofMillis(400));
|
||||
fork.setMaxConcurrent(1);
|
||||
ForkExecutor executor = new ForkExecutor(fork, mock(GitSnapshotCopier.class));
|
||||
responseDelayMs = 2000;
|
||||
try {
|
||||
assertThatThrownBy(() -> executor.execute(deadline -> store.head("vibemake_dev/urdf/slow/main.urdf")))
|
||||
.isInstanceOfSatisfying(BusinessException.class, error -> assertThat(error.code()).isEqualTo("FORK_TIMEOUT"));
|
||||
await().atMost(Duration.ofSeconds(1)).untilAsserted(() ->
|
||||
assertThat(executor.<Integer>execute(deadline -> 1)).isEqualTo(1));
|
||||
} finally { executor.close(); }
|
||||
}
|
||||
|
||||
@Test
|
||||
void realSdkReadsMetadataAndCopiesWithV4AuthenticationAndSourceEtag() {
|
||||
OssObjectStore.Metadata metadata = store.head("vibemake_dev/urdf/a/机械 臂.urdf");
|
||||
|
||||
+38
-6
@@ -40,7 +40,16 @@
|
||||
| 接口调试 | `/swagger-ui.html` |
|
||||
| OpenAPI | `/v3/api-docs` |
|
||||
|
||||
当前后端没有配置跨域白名单。开发时通过前端开发服务器代理 `/api`,部署时通过同域反向代理访问;仅允许 OPTIONS 请求不代表已经支持浏览器跨域访问。
|
||||
后端按环境为 `/api/**` 配置跨域白名单,配置项为 `community.cors.allowed-origins`:
|
||||
|
||||
| 后端环境 | 允许的前端 Origin |
|
||||
| --- | --- |
|
||||
| test(默认) | `https://dev-vibemake.robotquan.com` |
|
||||
| prod | `https://vibemake.robotquan.com` |
|
||||
|
||||
允许 `GET、HEAD、POST、PUT、PATCH、DELETE、OPTIONS`,请求头允许 `Authorization、Content-Type、Accept`,预检缓存 3600 秒。前端继续通过 `Authorization: Bearer <平台 token>` 鉴权,无需设置 `credentials: 'include'`;后端不启用跨域 Cookie 凭据。成功及鉴权、业务错误响应均支持上述来源,并暴露 `X-Request-Id`、`Content-Disposition` 响应头。
|
||||
|
||||
其他域名、HTTP 来源、非默认端口和另一环境的前端来源均不在默认白名单内。本地开发继续通过前端开发服务器代理 `/api`;部署代理需透传 `Origin`、预检请求及后端 CORS 响应头,避免重复添加跨域头。OSS 直传的跨域规则仍需单独在 Bucket 配置。
|
||||
|
||||
### 统一响应
|
||||
|
||||
@@ -565,7 +574,7 @@ OSS 控制台配置步骤:
|
||||
|
||||
1. 准备指定地域的 Bucket,确认 test/prod 的根目录分别为 `vibemake_dev/`、`vibemake/`。目录是对象前缀,不是 Bucket 名。
|
||||
2. 给服务端 RAM 用户授予对应环境目录的 `oss:PutObject` 权限,例如资源 `acs:oss:*:*:YOUR_BUCKET/vibemake_dev/project-covers/*`;正式环境改为 `vibemake/project-covers/*`,两个环境可使用各自凭据。
|
||||
3. 配置 Bucket CORS:AllowedOrigins 填真实前端开发、部署 Origin(含协议和端口),AllowedMethods 包含 POST,若使用 fetch 读图则增加 GET、HEAD;AllowedHeaders 允许 Content-Type,可按需要暴露 ETag、x-oss-request-id。调用签名接口仍走原有 `/api` 代理,不能用后端 OPTIONS 放行替代 OSS CORS。
|
||||
3. 配置 Bucket CORS:AllowedOrigins 填真实前端开发、部署 Origin(含协议和端口),AllowedMethods 包含 POST,若使用 fetch 读图则增加 GET、HEAD;AllowedHeaders 允许 Content-Type,可按需要暴露 ETag、x-oss-request-id。调用签名接口可按上述 API 跨域约定直接访问后端,或通过 `/api` 代理访问;后端 API 的 CORS 不能替代 OSS CORS。
|
||||
4. 对封面目录配置公开 GetObject 读取权限,或使用能够公开返回图片的 CDN;私有源站通过 CDN 时需配置回源鉴权。对象沿用 Bucket 的访问规则,应用不修改整个 Bucket 的 ACL,也不生成临时下载链接。
|
||||
5. 测试环境完成“获取签名 → 浏览器直传 → 保存项目 → 匿名读取图片”,核对对象 key 和浏览器网络请求;生产环境重复验收 `vibemake/` 路径。
|
||||
|
||||
@@ -922,9 +931,24 @@ commitSha 必须为当前 main 的完整 SHA,不能发布历史提交。成功
|
||||
|
||||
成功 data 为当前用户的新 Project,状态 READY、UNPUBLISHED;使用返回的新 projectCode 跳转工作区。类型和描述继承源项目,sourceProjectCode、sourceCommitSha 记录来源。
|
||||
|
||||
Fork 复制发布版本的所有普通文件,不复制源仓库提交历史,不自动同步上游。允许同一用户多次 Fork。所有文件复制完成才返回 READY;不支持符号链接或子模块。
|
||||
Fork 使用 Git 仓库级快照复制:冻结校验通过的 publishedCommitSha,保留该提交完整的文件树,在新私有仓库 main 上创建一个无父提交的初始版本,不继承源仓库的历史、其他分支或标签,不自动同步上游。允许同一用户及多个用户多次 Fork 同一来源。新副本的提交 SHA 与 sourceCommitSha 不同,应通过工作区 tree 接口获取新版本。
|
||||
|
||||
源项目发布版本已变化时返回 409 CONFLICT,刷新社区详情后由用户重新确认。Fork 是同步请求,可能耗时较长;提交期间显示加载状态并防止重复点击。超时、网络断开或失败后先查询工作区,不要自动重发。
|
||||
复制保留普通文件、执行权限、符号链接、`.gitmodules` 和子模块引用;不递归下载子模块依赖。Git LFS 指针及发布快照引用的实体会完整复制,并从副本独立下载校验。缺失的 LFS 实体会导致 Fork 失败。复制不使用普通文件接口的 10 MiB 读取上限,在线文件浏览、下载及 LFS 展开能力仍受原限制。URDF 按受理时绑定的当前清单独立复制到新目录,复制时校验 ETag,封面继承原 URL;Git、LFS 和 OSS 全部完成后才返回 READY。
|
||||
|
||||
后端部署需要 Git 和 git-lfs。默认 Fork 总预算5分钟、单实例最多同时2次,每次临时磁盘上限5 GiB(含 Git 对象、LFS 源缓存和副本校验缓存)。配置分别为 FORK_TIMEOUT、FORK_MAX_CONCURRENT、FORK_MAX_TEMPORARY_BYTES;临时目录可通过 FORK_TEMPORARY_DIRECTORY 指定。依赖不可用或并发超限时,在创建项目记录前拒绝。普通创建和其他 API 不依赖 Fork 运行环境。
|
||||
|
||||
| 错误码 | 行为 |
|
||||
| --- | --- |
|
||||
| FORK_UNAVAILABLE(503) | Git、git-lfs、临时目录或 Gitea 配置不可用,联系后端处理 |
|
||||
| FORK_BUSY(429) | 达到 Fork 并发上限,稍后手动重试 |
|
||||
| FORK_RESOURCE_LIMIT(413) | 临时磁盘或元数据超过限制,核对项目状态及服务端配额 |
|
||||
| FORK_COPY_FAILED(502) | 快照、认证或 LFS 复制/校验失败,使用 requestId 排查 |
|
||||
| FORK_TIMEOUT(504)/ FORK_CANCELLED(499) | 已超时或取消,先核对工作区再决定下一步 |
|
||||
| REMOTE_RESULT_UNKNOWN(504) | 远端写入结果未确认,保留 CREATING,禁止自动重试 |
|
||||
|
||||
创建记录后发生异常时,错误 data.projectCode 指向本次新副本。明确失败标记 FAILED;远端写入结果不明或回写异常保留 CREATING 并阻止后续写入。后端终止相关子进程、清理本地临时目录,不自动删除远端仓库或 OSS 对象,也不自动修复历史失败副本。
|
||||
|
||||
源项目发布版本已变化时返回 409 CONFLICT,刷新社区详情后由用户重新确认。Fork 是同步请求,可能耗时较长;前端和代理读取超时默认至少400秒,提交期间显示加载状态并防止重复点击。超时、网络断开或失败后先查询工作区,不要自动重发。
|
||||
|
||||
## 评论和点赞
|
||||
|
||||
@@ -1078,9 +1102,10 @@ data:{"code":"OK","message":"成功","data":{"status":"NO_MATCH","reply":"暂未
|
||||
| AI_INVALID_RESPONSE | 模型结果的操作、参数、编码或引用未通过校验,不执行该操作 |
|
||||
| AI_SEARCH_FAILURE | 服务内部异常,携带 requestId 联系后端;涉及写入时先核对工作区 |
|
||||
| NOT_FOUND / CONFLICT | Fork 来源不可访问或发布版本变化,不自动重新选择或换版本执行 |
|
||||
| FORK_UNAVAILABLE / FORK_BUSY / FORK_RESOURCE_LIMIT / FORK_COPY_FAILED / FORK_TIMEOUT / FORK_CANCELLED | 按上文 Fork 错误码处理;error 事件不会改变已建立 SSE 的 HTTP 200 状态 |
|
||||
| GITEA_ERROR / REMOTE_RESULT_UNKNOWN / PROJECT_INITIALIZATION_ERROR | 项目初始化失败或结果待核对,优先使用 data.projectCode 查询工作区详情 |
|
||||
|
||||
默认每次模型调用30秒,整次处理65秒(包括仓库操作),全局并发4个、每用户滑动一分钟最多10次,超限不排队。主动取消通过 AbortController;服务端在检测连接断开后尽力取消上游,实际工作结束后释放并发名额。写入前检测取消,写入开始后超时或断开不保证回滚;即使没有收到项目编码,也应先查询工作区列表核对,不自动重发。不要在页面上同时启动多个 AI 请求;代理应关闭 SSE 缓冲,读取超时至少90秒。
|
||||
默认每次模型调用30秒,模型/搜索阶段总预算65秒;进入 forking 后切换为独立 Fork 预算(默认5分钟,另留15秒入口及终态余量),不再受搜索阶段65秒截止时间影响。SSE 容器超时覆盖搜索与 Fork 两阶段,心跳持续发送。全局 AI 并发4个、每用户滑动一分钟最多10次,Fork 另受单实例2次并发限制,超限不排队。主动取消通过 AbortController;服务端在检测连接断开后尽力取消上游,实际工作结束后释放并发名额。写入前检测取消,写入开始后超时或断开不保证回滚;即使没有收到项目编码,也应先查询工作区列表核对,不自动重发。不要在页面上同时启动多个 AI 请求;代理应关闭 SSE 缓冲,读取超时至少400秒。
|
||||
|
||||
### fetch 接收 SSE 示例
|
||||
|
||||
@@ -1247,7 +1272,7 @@ const result = await chatAi(
|
||||
|
||||
## 前端请求示例
|
||||
|
||||
以下示例使用浏览器 fetch 和 TypeScript,类型声明见前面的响应结构。API_BASE 使用 `/api`,配合同域反向代理或开发代理。
|
||||
以下示例使用浏览器 fetch 和 TypeScript,类型声明见前面的响应结构。API_BASE 默认使用 `/api`,配合同域反向代理或开发代理;从白名单前端直接跨域访问时,改为实际后端的 HTTPS 地址加 `/api`,继续传递 Bearer Token,无需设置 `credentials: 'include'`。
|
||||
|
||||
### JSON 与 FormData 请求封装
|
||||
|
||||
@@ -1404,6 +1429,8 @@ function parseUtcTime(value: UtcDateTime): Date {
|
||||
|
||||
使用 Vite 时可在现有配置中加入以下 server 配置,保留 `/api` 路径,不进行 rewrite:
|
||||
|
||||
本地代理的 `changeOrigin` 会修改上游 Host,但可能保留浏览器的 localhost Origin。为避免本地写请求被后端白名单拒绝,示例通过 [Vite 的 configure 回调](https://vite.dev/config/server-options#server-proxy)在本地代理转发时移除 Origin,Authorization 保持原样;线上部署代理仍需透传真实 Origin。
|
||||
|
||||
```typescript
|
||||
export default {
|
||||
server: {
|
||||
@@ -1411,6 +1438,11 @@ export default {
|
||||
'/api': {
|
||||
target: 'http://localhost:8993',
|
||||
changeOrigin: true,
|
||||
configure(proxy) {
|
||||
proxy.on('proxyReq', (proxyReq) => {
|
||||
proxyReq.removeHeader('origin');
|
||||
});
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
|
||||
@@ -19,6 +19,9 @@ if config.exists():
|
||||
if not environment.get("GITEA_TOKEN"):
|
||||
raise SystemExit("请先配置 GITEA_TOKEN 和 GITEA_ORGANIZATION")
|
||||
environment["COMMUNITY_LIVE_GITEA"] = "true"
|
||||
# 本脚本只验证仓库协议;OSS/AI 有独立测试,避免本机其他服务配置阻止启动。
|
||||
environment["AI_ENABLED"] = "false"
|
||||
environment["OSS_ENABLED"] = "false"
|
||||
raise SystemExit(subprocess.call([
|
||||
"mvn", "-q", "-Dtest=GiteaLiveTest", "-Dsurefire.failIfNoSpecifiedTests=false", "test"
|
||||
], cwd=root, env=environment))
|
||||
|
||||
Reference in New Issue
Block a user