diff --git a/CONTEXT.md b/CONTEXT.md index 5c7bbe45..b85797b5 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -30,6 +30,20 @@ allowed proxy CIDRs, and whether concurrent reuse is allowed. It does not own credential identity, key material, lifetime, persistence, or runtime ownership. Each credential intake path normalizes the grant before installing it. +## Peer Relay advertisement + +A platform peer may prefer an eligible directly connected credential relay by +omitting covered credential-leaf edges from only its own advertised OSPF +connection row. Its local route calculation still uses the complete physical +adjacency so direct-destination fallback remains available. Other peers' +source-owned rows and versions are never rewritten, cached for promotion, or +otherwise changed by this projection. + +Relay eligibility comes from the transport-authenticated credential identity +and grant, not self-reported route metadata. The advertisement Module does not +support changing a credential's relay permission in place; such a permission +change is a credential revocation and new authenticated Session. + ## Attached peer An attached peer is an ordinary `PeerManagerCore` connected to another diff --git a/Cargo.lock b/Cargo.lock index 534dae33..1ff1e5f2 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2541,6 +2541,18 @@ dependencies = [ "windows 0.52.0", ] +[[package]] +name = "easytier-ios" +version = "0.1.0" +dependencies = [ + "easytier-core", + "easytier-ffi", + "serde_json", + "tokio", + "tracing-subscriber", + "uuid", +] + [[package]] name = "easytier-mini" version = "2.6.4" diff --git a/Cargo.toml b/Cargo.toml index 0e712a08..5492867f 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -10,6 +10,7 @@ members = [ "easytier-contrib/easytier-ffi", "easytier-contrib/easytier-uptime", "easytier-contrib/easytier-android-jni", + "easytier-contrib/easytier-ios", ] default-members = ["easytier", "easytier-web"] exclude = [ diff --git a/docs/easytier-web-managed-config-sync-plan.md b/docs/easytier-web-managed-config-sync-plan.md new file mode 100644 index 00000000..c2ae168c --- /dev/null +++ b/docs/easytier-web-managed-config-sync-plan.md @@ -0,0 +1,563 @@ +# EasyTier Web Managed Config Incremental Sync Plan + +## Status + +- 状态:Implemented(核心协议、持久化与 Session 增量收敛) +- 实施范围:EasyTier Web 的 HTTP 接收、校验、SQLite 持久化和 Session 运行态收敛 +- 上游依赖:后续由 Console 计算并发送 Patch +- 兼容要求:保留现有 Full PUT + +本文记录当前接收端方案。Session 在能够证明 Patch base 与已应用 revision 连续时 +只收敛 touched instances;重启、通知丢失、revision 断链或并发积压时沿用 Full +reconcile。 + +## 1. 背景与结论 + +当前 `/validate-token` webhook 已经只交换 token、机器信息和 revision,不再 +携带完整 managed config 集合。剩余的大集合位于独立的配置发布路径: + +```text +PUT /api/internal/users/:user-id/machines/:machine-id/networks +``` + +Console 每次发布都会向该路径发送完整 Exact Set。实例很多时,请求体、JSON +解析、现有配置扫描和逐条 SQLite 写入都随实例总数增长。 + +第一阶段采用以下方案: + +1. 保留 PUT,作为完整发布、首次同步和冲突恢复路径。 +2. 在同一路径增加 PATCH;普通变更只发送完整的单实例 upsert 和删除 ID。 +3. PATCH 使用 `expected_config_revision` 做 compare-and-swap(CAS)。 +4. Full/Patch 的配置变更与 revision 更新在一个 SQLite transaction 中提交。 +5. Patch 只查询和写入 touched instances,不扫描完整 Target。 +6. 写入成功后通知 Session 本次 base、target 和 touched instance IDs。 +7. Session 仅在 applied revision 精确匹配 base 时增量收敛,否则安全回退 Full。 + +普通变更的接收端成本由: + +```text +O(total instances) +``` + +降为: + +```text +wire / JSON / persistence transaction / runtime config apply = O(changed instances) +``` + +冷启动或 revision 冲突仍需要 `O(total)` 的 Full。这是没有可用基线时传递完整 +目标状态所必需的成本;如果 Full 超过安全的单请求上限,需另行设计 staged +snapshot,而不是直接分页写入 live rows。 + +## 2. 目标与非目标 + +### 2.1 目标 + +1. 普通新增、更新和删除只传输、解析、查询并写入变化实例。 +2. Config rows 与 persisted revision 原子提交。 +3. Patch 可安全重试,并能确定性处理并发或乱序请求。 +4. 保持 user-owned 与 web-owned 配置的 ownership 规则。 +5. 保持 Full Exact Set 的删除和显式空集合语义。 +6. 为 Full 和 Patch 设置显式且可测试的容量限制。 +7. 先部署接收端,再允许 Console 使用 Patch。 + +### 2.2 非目标 + +1. 修改 `/validate-token` request/response。 +2. 在本阶段实现 Console 的 diff/cache 逻辑。 +3. 优化 Core heartbeat 中的完整运行实例上报。 +4. 实现 Full 分页、上传会话或持久化 delivery FSM。 +5. 让冷启动 Full 的成本低于 `O(total)`。 + +## 3. 必须保持的语义 + +### 3.1 Full Exact Set + +Full 表示一个 `(user_id, machine_id)` 下全部期望的 web-owned configs: + +- 请求中存在的实例应被创建或更新; +- 已存在但请求中缺失的 web-owned 实例应被删除; +- 空集合应删除该 Target 下全部 web-owned 实例; +- user-owned 实例不能被覆盖或删除。 + +### 3.2 Patch + +Patch 只描述从一个已知 revision 到另一个 revision 的变化: + +- `upserts`:新增或变化实例的完整 config; +- `delete_instance_ids`:从目标集合中删除的实例 ID; +- `expected_config_revision`:receiver 必须已经处于的 base revision; +- `config_revision`:提交完成后的 target revision。 + +Patch 不是独立的完整目标。当前 revision 与 expected revision 不一致时,必须 +返回冲突且不做任何写入。 + +### 3.3 Revision invariants + +1. 一个 persisted revision 只对应与其一起提交的 web-owned projection。 +2. Config mutation 和 revision advancement 必须位于同一 transaction。 +3. Patch 只能应用在完全匹配的 expected revision 上。 +4. 当前 revision 已等于 target revision 时,返回幂等成功且不重复写入。 +5. Publisher 不得为不同目标状态复用同一个 target revision。 +6. 任何其他写路径只要改变 web-owned row,就必须在同一 transaction 中清除 + managed revision;否则未来 Patch 会基于错误的 base。 +7. Persisted revision 与 Session applied revision 保持为两个不同事实。HTTP + 成功只代表本地持久化完成,不代表 Core 已经应用。 + +## 4. HTTP contract + +### 4.1 保留 Full PUT + +路径不变: + +```text +PUT /api/internal/users/:user-id/machines/:machine-id/networks +``` + +现有 JSON shape 保持兼容: + +```json +{ + "managed_network_configs": [ + { + "instance_id": "11111111-1111-1111-1111-111111111111", + "network_config": {} + } + ], + "config_revision": "target-revision", + "expected_config_revision": "base-revision" +} +``` + +`expected_config_revision` 保持当前含义: + +- 字段缺失:兼容旧调用者,不检查 base; +- 空字符串:要求当前 persisted revision 不存在; +- 非空字符串:要求当前 revision 与该值相等。 + +新 Console 必须发送 expected revision。省略 expected 的形式只用于旧版本兼容 +和明确的运维修复。 + +`config_revision` 的处理: + +- 非空:配置与 target revision 原子提交; +- 缺失:保留旧 Full 请求兼容,但清除已有 managed revision,因此该结果不能 + 作为后续 Patch base; +- 空字符串:拒绝为 400。 + +Revisioned Full 遇到 user-owned instance ID 冲突时整体失败。Legacy +unrevisioned Full 保持当前兼容行为:跳过 user-owned row,且绝不覆盖它。 + +### 4.2 新增 Patch + +同一资源增加: + +```text +PATCH /api/internal/users/:user-id/machines/:machine-id/networks +``` + +请求格式: + +```json +{ + "upserts": [ + { + "instance_id": "11111111-1111-1111-1111-111111111111", + "network_config": {} + } + ], + "delete_instance_ids": [ + "22222222-2222-2222-2222-222222222222" + ], + "config_revision": "target-revision", + "expected_config_revision": "base-revision" +} +``` + +Patch contract: + +1. 两个 revision 字段均必填、非空且不能相同。 +2. `upserts` 中的 instance ID 不得重复。 +3. `delete_instance_ids` 中的 ID 不得重复。 +4. 同一个 ID 不得同时出现在 upsert 和 delete 中。 +5. 每个 upsert 必须携带该实例的完整 `NetworkConfig`,不支持字段级 JSON + Patch。 +6. `network_config` 内部的 instance ID 不受信任,receiver 使用 envelope 中的 + `instance_id` 进行归一化。 +7. 删除不存在的 ID 是幂等 no-op。 +8. Upsert 或 delete 碰到 user-owned row 时,整个 Patch 返回冲突且不写入。 +9. 不允许从“receiver revision 不存在”的未知状态直接 Patch;使用 Full 建立 + Exact Set 和首个 revision。 +10. 空 Patch 不能把 revision 改成另一个值;这通常表示 publisher revision + 计算错误,因此返回 400。 + +### 4.3 HTTP outcomes + +| 条件 | HTTP | 语义 | +| --- | ---: | --- | +| Full/Patch 新提交成功 | 204 | Config 和 revision 已持久化 | +| Target revision 已经存在 | 204 | 幂等成功,无 row mutation | +| Expected revision 不匹配 | 409 | 零写入,调用者重新观察或发送 Full | +| User-owned ownership 冲突 | 409 | 零写入,不能自动覆盖 | +| 非法 ID、重复、交集或非法 config | 400 | 调用 contract 错误 | +| 请求超过 byte limit | 413 | 未进入 reconciliation | +| 条目数或单 config 超过限制 | 422 | 超出接收端容量 contract | +| SQLite 错误 | 500 | Transaction rollback | + +409 返回机器可读字段:revision 冲突为 +`code=managed_config_revision_conflict` 并在已知时带 +`current_config_revision`;ownership 冲突为 +`code=managed_config_ownership_conflict`。响应不得返回配置内容。日志不得记录 +token、secret 或完整 config JSON。 + +## 5. Receiver architecture + +### 5.1 Module responsibilities + +| Module | 本阶段职责 | +| --- | --- | +| Internal HTTP Adapter | 内部鉴权、body/count limit、DTO 解析、HTTP 状态映射 | +| `ClientManager` | 解析 Target,调用 managed-config Interface,成功后通知 Session | +| `client_manager::managed_config` | Full/Patch 规则、归一化、typed outcome | +| `Db` Adapter | CAS、ownership fence、批量 mutation、revision transaction | +| Session runtime reconciliation | 校验 applied/base/target fence,增量收敛 touched instances;断链时 Full | + +HTTP Adapter 不实现 ownership、diff 或 transaction 逻辑。PUT 和 PATCH 共用 +managed-config Module,避免两套规则逐渐分叉。 + +### 5.2 Internal Interface + +Module 接收两种 intent: + +```text +Full { + desired_configs, + target_revision: Option, + expected_revision: Any | Exact(Option) +} + +Patch { + upserts, + delete_instance_ids, + target_revision: Revision, + expected_revision: Revision +} +``` + +返回 typed outcome: + +```text +Applied { + previous_revision, + target_revision +} + +AlreadyApplied { + target_revision +} + +RevisionConflict { + expected_revision, + current_revision +} + +OwnershipConflict { + instance_id +} +``` + +Validation error 与 database error 保持独立类型。HTTP handler 只负责将这些结果 +映射到 section 4.3 的状态码。 + +## 6. Receiver implementation + +### 6.1 Validation and normalization + +在打开 SQLite write transaction 之前完成: + +- request byte/count/per-entry limit; +- UUID、重复 ID 和 upsert/delete 交集校验; +- config key 拼写归一化; +- envelope instance ID 覆盖 nested identity; +- `NetworkConfig` 反序列化。 + +这样非法大请求不会长时间占用 SQLite writer lock。Ownership 必须在 transaction +内重新查询,因为 transaction 外的结果可能已过期。 + +当 request 带 target revision 时,可以先做一次 O(1) revision read;如果当前值 +已经等于 target,可直接返回 `AlreadyApplied`,避免完整 config 归一化。任何可能 +写入的请求仍必须在 transaction 内再次检查 revision。 + +### 6.2 Full transaction + +在同一个 SQLite connection 上执行: + +1. `BEGIN IMMEDIATE`。 +2. 读取 `(user_id, machine_id)` 当前 persisted revision。 +3. 若 supplied target 已经是 current,返回 `AlreadyApplied`。 +4. 检查 optional expected revision。 +5. 只读取现有 row 的 `(instance_id, source)`;不加载无关 config JSON。 +6. 执行 user-owned ownership fence。 +7. 批量 upsert 全部 desired web-owned rows。 +8. 计算并批量删除 `existing_web_ids - desired_ids`。 +9. 最后写入 supplied target revision;legacy unrevisioned Full 则删除旧 revision。 +10. Commit。 + +任一步骤失败都 rollback。Full 仍是 `O(total)`,但不会再逐条独立提交,也不会 +出现“部分 rows 已更新、revision 仍是旧值”的中间持久状态。 + +### 6.3 Patch transaction + +在同一个 SQLite connection 上执行: + +1. `BEGIN IMMEDIATE`。 +2. 读取 current revision。 +3. 如果 current 等于 target,返回 `AlreadyApplied`。 +4. 如果 current 不等于 expected,返回 `RevisionConflict`。 +5. 只查询 upsert/delete IDs 的 source。 +6. 任一 touched ID 属于 user 时,返回 `OwnershipConflict`。 +7. 批量 upsert changed configs。 +8. 批量删除 requested web-owned IDs。 +9. 最后写入 target revision。 +10. Commit。 + +Patch 禁止: + +- list 全部 Target rows; +- 重算完整 Target digest; +- 根据 touched IDs 之外的数据做 stale-row scan。 + +因此其数据库工作量只随 `upserts + deletes` 增长。 + +### 6.4 Bounded batch SQL + +批量操作不构造无限长 SQL。根据 SQLite bind-variable limit 选取固定 batch size, +并在同一个 transaction 内分批执行: + +- multi-row `INSERT ... ON CONFLICT DO UPDATE`; +- 带 `source = web` 条件的 batch delete; +- 只返回 instance ID/source 的 ownership query。 + +Patch statement 数量应为 `O(ceil(delta / batch_size))`;Full 为 +`O(ceil(total / batch_size))`。每个 accepted request 只有一个 transaction 和 +一次 revision 写入。 + +### 6.5 Alternate-write revision invalidation + +现有其他路径可能 save、delete、disable 或改变 web-owned row。若这些路径修改 +rows 后仍保留旧 managed revision,Patch CAS 会把错误状态当作正确 base。 + +因此所有 config mutation Adapter 必须遵守: + +1. 判断 mutation 是否改变 web-owned row; +2. 在一个 transaction 中执行 mutation; +3. 在 commit 前删除该 Target 的 managed revision。 + +Managed Full/Patch 在同一 transaction 内先完成 mutation,最后写入新的 target +revision。只影响 user-owned rows 的操作不清除 managed revision。 + +本方案不在 `/validate-token` 读取 revision 时重算完整 digest,否则周期性验证会 +重新变成 `O(total)`。Revision 完整性由所有写入 Adapter 局部维护。 + +### 6.6 Locking, cancellation and notification + +现有 per-target process-local lock 可以保留,用于减少同进程的重复工作,但它不 +承担正确性。正确性由 SQLite transaction 和 CAS 提供。 + +- Transaction 内不执行 Session RPC、网络请求或无关 async 工作。 +- HTTP future 在 commit 前取消时,transaction drop 必须 rollback。 +- Commit 后即使 response 或 notification 丢失,persisted state 仍然有效;调用者 + 用同一 target retry 会得到幂等成功。 +- 只有带 target revision 的 `Applied` 才通知匹配的 live Session; + `AlreadyApplied`、legacy unrevisioned Full、conflict 和失败不重复通知。 +- Notification 必须发生在 commit 之后。 +- Full notification 清除任何 pending delta,触发完整收敛。 +- Patch notification 携带 expected revision、target revision、upsert IDs 和本次 + transaction 实际接受删除的 web-owned IDs。请求删除但数据库原本不存在的 ID + 仍是 no-op,不能借机删除 Core 中同 ID 的 user-owned 实例。只有 Session applied + revision 精确等于 expected revision,且没有更早的 Patch 等待处理时,才保留该 + delta。 +- 两次 Patch 在前一次完成前积压时不合并 delta;Session 清除 pending delta,并在 + 最新 heartbeat/revision 上执行一次 Full。这避免引入 Patch queue 或 delivery FSM。 +- 增量 round 只读取 upsert rows,只删除本次 delete IDs,只对 touched running + instances 执行 runtime Patch/Run。完成前再次校验 persisted target revision;只有 + 全部 touched instances 成功且 target 仍相同,才推进 applied revision。 +- 任何通过 EasyTier Web mutation route 直接 Run、Save、Delete 或切换实例状态的 + 操作在执行前和结束后(包括部分 side effect 后返回错误)都清除 Session applied + revision 与 pending delta、增加运行配置 cache epoch,并唤醒一次 Full + reconcile。旧 round 只有 epoch 仍匹配时才能推进 applied revision;新一轮不得 + 信任 mutation 前缓存的 runtime config。否则 runtime-only mutation 或 Core 成功、 + SQLite 失败的复合 mutation 可能在 persisted revision 不变时破坏 Patch base 的 + 完整性。 + +## 7. Capacity contract + +当前 route 没有显式 body limit,Axum `Json` 使用依赖版本的默认 2 MiB 限制。 +生产容量不应依赖框架隐式默认值。 + +本阶段定义并测试四个独立限制: + +- decoded request 最大 bytes; +- Full entries / Patch upserts 最大数量; +- Patch deletes 最大数量; +- 单个 `network_config` 最大 bytes。 + +限制只应用于 internal managed-config route,不提高其他 public route 的 limit。 +具体默认值不能拍脑袋确定:先采集 1k/10k representative configs 的 encoded +size 和 peak memory,再选择有明确 headroom 的默认值及硬上限。 + +提高 Full limit 只是确保 fallback 覆盖已支持的生产规模,不是稳态优化。请求压缩 +同样只能降低 wire bytes,不能降低 JSON materialization 和 SQLite 工作量,因此 +不作为 Patch 的前置条件。 + +## 8. Failure and recovery + +| Failure | Receiver state | Caller action | +| --- | --- | --- | +| Invalid payload | Unchanged | 修复请求,不重试相同 payload | +| Capacity exceeded | Unchanged | 使用较小 Patch;Full 需检查支持规模 | +| Revision conflict | Unchanged | 重新观察;有 base 时重算 Patch,否则 Full | +| Ownership conflict | Unchanged | 解决 ownership,不能自动覆盖 | +| SQLite error before commit | Rolled back | 从相同 observed revision 重试 | +| Response lost after commit | Target committed | 同一 target retry,幂等成功 | +| Process exits before revisioned Session notify | Target committed | 现有 revision reconciliation 恢复 | +| Alternate web-row mutation | Revision atomically cleared | 下一次观察触发 Full 修复 | +| Console cache loss | Receiver unchanged | Console 发布 Full | + +Receiver 不保存 Patch delivery ledger。Publisher 根据自己的完整目标和 receiver +当前 revision 重算 Patch 或选择 Full。 + +## 9. Rollout and rollback + +### 9.1 Receiver-first rollout + +1. 为现有 Full 行为增加 characterization tests。 +2. 将 Full rows/revision 改为一个 atomic transaction。 +3. 为 alternate web-row mutation 增加 revision invalidation。 +4. 增加 PATCH、typed conflict、capacity limits 和 metrics。 +5. 在 Console 仍只发送 PUT 时部署到全部 EasyTier Web 实例。 +6. 完成旧 Console PUT、新 Console PUT/PATCH contract 测试。 +7. 最后启用 Console Patch 发布。 + +Patch capability 不通过 `/validate-token` 协商。部署顺序就是 compatibility gate; +这样不会把配置能力重新耦合回鉴权 Interface。 + +Console 遇到 409 可以 re-observe 后发送 Full。它不能把 404、401 或 malformed +response 当作旧 receiver 并静默换一种 mutation contract;出现 404 表示接收端 +部署门禁未满足。 + +### 9.2 Rollback + +- Console 尚未发送 Patch 时,EasyTier Web 可正常回滚。 +- Console 已发送 Patch 后,先回滚 Console,使调用恢复为 PUT,再回滚 Web。 +- PUT 在整个发布周期保持兼容。 +- Patch 和 Full 写入相同 rows/revision,不需要格式级数据迁移。 + +本方案不新增 persistent table。现有 Target/instance unique index 应覆盖 touched-ID +查询;若实现时需要新 index,必须先用实际 SQLite query plan 证明。 + +## 10. Verification + +### 10.1 Contract tests + +- 现有 Full JSON 继续接受。 +- 空 Full 删除所有 web-owned rows,保留 user-owned rows。 +- Patch add/update/delete 与等价 Full 得到相同最终 projection。 +- Duplicate/overlap/invalid config 返回 400 且零写入。 +- Patch 缺少 revision 返回 400。 +- Revision conflict 返回 409 和 current revision,不返回 config。 +- Byte/count/per-entry limits 分别有确定性测试。 + +### 10.2 Transaction and ownership tests + +- 在 upsert 后、delete 后、revision write 前注入错误,rows/revision 全部 rollback。 +- Revisioned Full/Patch 的 user-owned collision 整体 rollback。 +- 删除不存在的 ID 幂等成功。 +- 两个 target 从同一 base 并发时,一个成功、一个 409。 +- 相同 target retry 只有第一次写入,第二次为 no-op success。 +- Alternate save/delete/disable web row 与 revision invalidation 原子提交。 +- User-owned-only mutation 不清除 managed revision。 +- 数据库重连后,任一 persisted revision 都对应完整一致的 rows。 + +### 10.3 Scale tests + +至少使用 1k 和 10k representative entries: + +- 单实例 Patch 的 decoded bytes、row reads、writes 和 statement count 不随 Target + 总实例数增长; +- Patch 不执行 list-all query; +- Full 使用 bounded batches 和一个 transaction; +- Revision read 保持 O(1); +- 超限 Full 稳定返回 413/422,而不是耗尽进程内存; +- 并发请求无 deadlock,且 CAS 结果确定。 + +Session 测试还必须验证:精确 base/target 使用 touched-instance reconcile;base +不匹配、目标 revision 已变化、Full notification 和 Patch backlog 都使用 Full; +touched runtime apply 失败不推进 applied revision;删除只作用于本次 delete IDs。 +运行态 Config Get/Patch/Run/Delete 数量应随 touched instances 增长。为确认运行实例 +身份而进行的一次 list/meta RPC 可以保留,它不发送或重写所有实例配置。 + +## 11. Observability + +每个请求记录结构化字段,但不记录 config 内容: + +- mode:`full` / `patch`; +- user/machine scope; +- request bytes; +- desired/upsert/delete count; +- normalization、target-lock wait、transaction duration; +- SQL statement/batch count; +- result:applied、already-applied、revision-conflict、ownership-conflict、 + invalid、oversized、database-error; +- Session notification 是否发送。 + +Rollout acceptance: + +- Console 启用后 Patch 占普通变更的绝大多数; +- 单实例变化的 request size 与 SQLite cost 与单实例成比例; +- conflict rate 可解释且稳定; +- 支持规模内的 Full 没有 413/422; +- validate-token latency 不随 Target 实例数增长。 + +## 12. 后续优化 + +### 12.1 Session runtime delta apply(已实现) + +Patch commit outcome 已携带 touched IDs。Session 只在 applied revision 正好等于 +Patch base 时执行 touched-instance reconcile;重启、revision 断链、通知丢失或 +并发 Patch backlog 都退回 Full。接收端不保存 Patch queue,也不合并 delta。 + +### 12.2 Chunked Full + +不能把 Full Exact Set 直接分页写入 live rows:接收端无法在中间页判断哪些旧 +实例最终应删除,crash 也会暴露半套目标。 + +如果测量证明单请求 Full 无法覆盖必须支持的冷恢复规模,需要单独设计带 +snapshot ID、staging rows、expiry、finalize 和 atomic swap 的协议。在出现数据 +证明前不新增该状态机。 + +## 13. Implementation files and checklist + +主要涉及: + +- `easytier-web/src/restful/network.rs` +- `easytier-web/src/client_manager/mod.rs` +- `easytier-web/src/client_manager/managed_config.rs` +- `easytier-web/src/db/mod.rs` +- 对应 contract、database 和 managed-config tests + +完成条件: + +- [x] 现有 Full compatibility tests 固定。 +- [x] Full config rows 与 revision 原子提交。 +- [x] Alternate web-owned mutations 原子清除 revision。 +- [x] PATCH contract 和 typed 409 实现。 +- [x] Patch 只查询、写入 touched IDs。 +- [ ] Bulk SQL 遵守 tested bind-count bound。 +- [ ] Route byte/count/per-entry limits 有文档和测试。 +- [x] User-owned rows 不能被 Full/Patch 覆盖或删除。 +- [x] Empty Full 语义保持。 +- [x] Applied/AlreadyApplied/conflict 的通知行为符合设计。 +- [x] Session 在 revision 连续时只收敛 touched instances,断链时使用 Full。 +- [ ] 1k/10k scale 与 concurrent CAS tests 通过。 +- [ ] Receiver-first compatibility matrix 通过。 diff --git a/easytier-contrib/easytier-android-jni/kotlin/com/easytier/jni/EasyTierJNI.kt b/easytier-contrib/easytier-android-jni/kotlin/com/easytier/jni/EasyTierJNI.kt index db07d54f..ccc2410d 100644 --- a/easytier-contrib/easytier-android-jni/kotlin/com/easytier/jni/EasyTierJNI.kt +++ b/easytier-contrib/easytier-android-jni/kotlin/com/easytier/jni/EasyTierJNI.kt @@ -73,6 +73,14 @@ object EasyTierJNI { */ @JvmStatic external fun retainNetworkInstance(instanceNames: Array?): Int + /** + * 停止指定的网络实例,其他实例不受影响 + * @param instanceName 要停止的实例名称,不存在时为 no-op + * @return 0 表示成功,-1 表示失败 + * @throws RuntimeException 当操作失败时抛出异常 + */ + @JvmStatic external fun deleteNetworkInstance(instanceName: String): Int + /** * 收集网络信息 * @param maxLength 最大返回条目数 diff --git a/easytier-contrib/easytier-android-jni/src/lib.rs b/easytier-contrib/easytier-android-jni/src/lib.rs index e70180d8..1b9e0097 100644 --- a/easytier-contrib/easytier-android-jni/src/lib.rs +++ b/easytier-contrib/easytier-android-jni/src/lib.rs @@ -9,6 +9,7 @@ //! - `parseConfig(config)`: validate TOML config text. //! - `runNetworkInstance(config)`: start a local network instance. //! - `retainNetworkInstance(instanceNames)`: retain named instances and stop the rest. +//! - `deleteNetworkInstance(instanceName)`: stop exactly one named instance. //! - `listInstances()`: return running instance names and IDs as JSON. //! - `collectNetworkInfos()`: return running instance info as a JSON string. //! - `callJsonRpc(...)`: call an exposed EasyTier RPC service with JSON payload. @@ -106,6 +107,23 @@ pub extern "system" fn Java_com_easytier_jni_EasyTierJNI_retainNetworkInstance( network_api::retain_network_instance_jni(env, class, instance_names) } +/// Stop exactly one named network instance without affecting other instances. +/// +/// Java signature: +/// `EasyTierJNI.deleteNetworkInstance(instanceName: String): Int` +/// +/// An unknown name is a no-op. On failure this returns `-1` and throws +/// `RuntimeException`. +#[unsafe(no_mangle)] +pub extern "system" fn Java_com_easytier_jni_EasyTierJNI_deleteNetworkInstance( + env: JNIEnv, + class: JClass, + instance_name: JString, +) -> jint { + logger::init(); + network_api::delete_network_instance_jni(env, class, instance_name) +} + /// Collect running network instance information. /// /// Java signature: diff --git a/easytier-contrib/easytier-android-jni/src/network_api.rs b/easytier-contrib/easytier-android-jni/src/network_api.rs index 1ae76102..7d11b106 100644 --- a/easytier-contrib/easytier-android-jni/src/network_api.rs +++ b/easytier-contrib/easytier-android-jni/src/network_api.rs @@ -2,8 +2,8 @@ use std::{ffi::CStr, ptr}; use easytier::proto::api::manage::{NetworkInstanceRunningInfo, NetworkInstanceRunningInfoMap}; use easytier_ffi::{ - KeyValuePair, collect_network_infos, free_string, list_instance, parse_config, - retain_network_instance, run_network_instance, set_tun_fd, + KeyValuePair, collect_network_infos, delete_network_instance, free_string, list_instance, + parse_config, retain_network_instance, run_network_instance, set_tun_fd, }; use jni::JNIEnv; use jni::objects::{JClass, JObjectArray, JString}; @@ -76,6 +76,30 @@ pub(crate) fn run_network_instance_jni(mut env: JNIEnv, _class: JClass, config: } } +pub(crate) fn delete_network_instance_jni( + mut env: JNIEnv, + _class: JClass, + instance_name: JString, +) -> jint { + let instance_name = match jstring_to_cstring(&mut env, &instance_name) { + Ok(name) => name, + Err(error) => { + throw_exception(&mut env, &format!("Invalid instance name: {error}")); + return -1; + } + }; + let instance_names = [instance_name.as_ptr()]; + unsafe { + let result = delete_network_instance(instance_names.as_ptr(), instance_names.len()); + if result != 0 + && let Some(error) = get_last_error() + { + throw_exception(&mut env, &error); + } + result + } +} + pub(crate) fn retain_network_instance_jni( mut env: JNIEnv, _class: JClass, diff --git a/easytier-contrib/easytier-ios/Cargo.toml b/easytier-contrib/easytier-ios/Cargo.toml new file mode 100644 index 00000000..d984e613 --- /dev/null +++ b/easytier-contrib/easytier-ios/Cargo.toml @@ -0,0 +1,23 @@ +[package] +name = "easytier-ios" +version = "0.1.0" +edition.workspace = true + +[lib] +crate-type = ["staticlib", "rlib"] + +[dependencies] +serde_json = "1.0" +tracing-subscriber = { version = "0.3", features = ["env-filter"] } +easytier-ffi = { path = "../easytier-ffi", default-features = false, features = [ + "c-abi", +] } + +[dev-dependencies] +uuid = "1" +tokio = { version = "1", features = ["io-util"] } +easytier-core = { path = "../../easytier-core" } +easytier-ffi = { path = "../easytier-ffi", default-features = false, features = [ + "c-abi", + "ffi-dataplane", +] } diff --git a/easytier-contrib/easytier-ios/build-xcframework.sh b/easytier-contrib/easytier-ios/build-xcframework.sh new file mode 100755 index 00000000..7ca70f05 --- /dev/null +++ b/easytier-contrib/easytier-ios/build-xcframework.sh @@ -0,0 +1,70 @@ +#!/usr/bin/env bash +# +# Build the easytier-ios static library slices for the Flutter iOS client. +# +# This script only runs on macOS: it needs the Apple SDK (aarch64-apple-ios*, +# x86_64-apple-ios targets) plus `lipo`. Run it from the EasyTier repository +# root or from this crate directory. +# +# rustup target add aarch64-apple-ios aarch64-apple-ios-sim x86_64-apple-ios +# ./build-xcframework.sh +# +# Output (workspace target directory + ./xcframework/sim): +# target/aarch64-apple-ios/release/libeasytier_ios.a (device) +# xcframework/sim/libeasytier_ios.a (simulator, lipo merged) + +set -euo pipefail + +CRATE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# The crate lives in a workspace; build artifacts land in the workspace root +# target directory regardless of the current directory. +WORKSPACE_ROOT="$(cd "${CRATE_DIR}/../.." && pwd)" +TARGET_DIR="${WORKSPACE_ROOT}/target" +OUT_DIR="${CRATE_DIR}/xcframework" + +if [[ "$(uname)" != "Darwin" ]]; then + echo "error: build-xcframework.sh must run on macOS (needs Apple SDK, lipo)" >&2 + exit 1 +fi + +cd "${WORKSPACE_ROOT}" + +# The Rust iOS targets emit a `___chkstk_darwin` stack-probe call but do not +# link the compiler-rt archive that provides it. Point the linker at the +# matching device or simulator archive shipped inside the Xcode toolchain. +CLANG_BIN="$(xcrun --find clang)" # .../Toolchains/XcodeDefault.xctoolchain/usr/bin/clang +TOOLCHAIN_USR="${CLANG_BIN%/bin/clang}" # .../XcodeDefault.xctoolchain/usr +CLANG_RT_DIR="$(cd "${TOOLCHAIN_USR}/lib/clang" && cd "$(ls | sort -V | tail -1)/lib/darwin" && pwd)" +CLANG_RT_RUSTFLAGS="${RUSTFLAGS:-} -C link-arg=-L${CLANG_RT_DIR}" +echo "==> using libclang_rt from ${CLANG_RT_DIR}" + +# kcp-sys's bindgen rejects the `-sim` in the aarch64-apple-ios-sim target +# triple; give bindgen an explicit simulator target so the C bindings build. +SIM_SDK="$(xcrun --sdk iphonesimulator --show-sdk-path)" + +echo "==> building aarch64-apple-ios (device)" +RUSTFLAGS="${CLANG_RT_RUSTFLAGS} -C link-arg=-lclang_rt.ios" \ + cargo build -p easytier-ios --release --target aarch64-apple-ios + +echo "==> building aarch64-apple-ios-sim (Apple Silicon simulator)" +BINDGEN_EXTRA_CLANG_ARGS="--target=arm64-apple-ios17.0-simulator -isysroot ${SIM_SDK}" \ + RUSTFLAGS="${CLANG_RT_RUSTFLAGS} -C link-arg=-lclang_rt.iossim" \ + cargo build -p easytier-ios --release --target aarch64-apple-ios-sim + +echo "==> building x86_64-apple-ios (Intel simulator)" +BINDGEN_EXTRA_CLANG_ARGS="--target=x86_64-apple-ios17.0-simulator -isysroot ${SIM_SDK}" \ + RUSTFLAGS="${CLANG_RT_RUSTFLAGS} -C link-arg=-lclang_rt.iossim" \ + cargo build -p easytier-ios --release --target x86_64-apple-ios + +rm -rf "${OUT_DIR}" +mkdir -p "${OUT_DIR}/sim" + +echo "==> lipo: merge simulator slices" +lipo -create \ + "${TARGET_DIR}/aarch64-apple-ios-sim/release/libeasytier_ios.a" \ + "${TARGET_DIR}/x86_64-apple-ios/release/libeasytier_ios.a" \ + -output "${OUT_DIR}/sim/libeasytier_ios.a" + +echo "==> done:" +echo " device: ${TARGET_DIR}/aarch64-apple-ios/release/libeasytier_ios.a" +echo " simulator: ${OUT_DIR}/sim/libeasytier_ios.a" diff --git a/easytier-contrib/easytier-ios/easytier-ios.h b/easytier-contrib/easytier-ios/easytier-ios.h new file mode 100644 index 00000000..87c283c8 --- /dev/null +++ b/easytier-contrib/easytier-ios/easytier-ios.h @@ -0,0 +1,138 @@ +/** + * @file easytier-ios.h + * @brief iOS-facing C ABI for EasyTier. + * + * This library embeds EasyTier into an iOS app without a TUN device or + * NEPacketTunnel: it manages EasyTier instances and bridges to the EasyTier + * management RPC surface. Loopback port forwarding into the virtual network + * is configured through easytier_ios_call_json_rpc() with + * api.config.ConfigRpcService/PatchConfig port-forward patches; there is no + * built-in forwarder. + * + * Error handling: functions returning `int` return 0 on success and -1 on + * failure; functions returning `char *` return NULL on failure. Call + * easytier_ios_last_error() on the same thread to retrieve details. + * + * Threading: all functions are safe to call from any thread. The last-error + * buffer is thread-local, so query it on the thread that received the + * failure. + */ + +#ifndef EASYTIER_IOS_H +#define EASYTIER_IOS_H + +#include + +#ifdef __cplusplus +extern "C" { +#endif + +/** + * @brief Enable diagnostic EasyTier logging to stderr. + * + * Installs a narrow global tracing subscriber that records port-forward + * lifecycle events. Repeated calls are idempotent. + * + * @return 0 on success, -1 if another tracing subscriber was installed first. + */ +int easytier_ios_enable_diagnostic_logging(void); + +/** + * @brief Start one EasyTier network instance from a TOML config string. + * + * The config's `instance_name` must be unique among instances started + * through this library. + * + * @param toml Non-null pointer to a NUL-terminated UTF-8 TOML config string. + * @return 0 on success, -1 on failure. + */ +int easytier_ios_run_instance(const char *toml); + +/** + * @brief Keep the named instances and stop all others. + * + * @param names_json Null, empty, or a NUL-terminated JSON array of instance + * name strings. Null / empty / `[]` stops every running + * instance. + * @return 0 on success, -1 on failure. + */ +int easytier_ios_retain_instances(const char *names_json); + +/** + * @brief Stop exactly one named instance without affecting other instances. + * + * An unknown name is a no-op. + * + * @param instance_name Non-null NUL-terminated instance name. + * @return 0 on success, -1 on failure. + */ +int easytier_ios_delete_instance(const char *instance_name); + +/** + * @brief Collect running instance information as a JSON object. + * + * The result maps each instance name to its running info JSON object. + * + * @param max_length Maximum number of instances to report. + * @return A newly allocated NUL-terminated JSON string on success, NULL on + * failure. + * + * @ownership The caller owns the returned string and must release it with + * easytier_ios_free_string(). + */ +char *easytier_ios_collect_network_infos(int max_length); + +/** + * @brief Call an exposed EasyTier management RPC method using protobuf JSON. + * + * `service_name` is the protobuf service name (e.g. + * "api.config.ConfigRpcService"), `method_name` the RPC method name (e.g. + * "PatchConfig"). `payload_json` must contain the protobuf JSON request, + * including any `instance` selector required by the target RPC. + * + * Port forwarding into the virtual network is driven through this bridge + * with api.config.ConfigRpcService/PatchConfig port-forward patches. + * + * @param service_name Non-null NUL-terminated RPC service name. + * @param method_name Non-null NUL-terminated RPC method name. + * @param payload_json Non-null NUL-terminated protobuf JSON request body. + * @return A newly allocated NUL-terminated JSON response string on success, + * NULL on failure. + * + * @ownership The caller owns the returned string and must release it with + * easytier_ios_free_string(). + */ +char *easytier_ios_call_json_rpc(const char *service_name, + const char *method_name, + const char *payload_json); + +/** + * @brief Return the last error message on this thread. + * + * Combines wrapper-side errors recorded by this library with the + * easytier-ffi last FFI error. + * + * @return A newly allocated NUL-terminated string, or NULL when there is no + * recorded error. + * + * @ownership The caller owns the returned string and must release it with + * easytier_ios_free_string(). + */ +char *easytier_ios_last_error(void); + +/** + * @brief Release a string returned by this library. + * + * Use this for strings returned by easytier_ios_collect_network_infos(), + * easytier_ios_call_json_rpc() and easytier_ios_last_error(). Passing NULL + * is a no-op. The string must not be used after this call. + * + * @param s NULL, or a string previously returned by this library. + */ +void easytier_ios_free_string(char *s); + +#ifdef __cplusplus +} +#endif + +#endif /* EASYTIER_IOS_H */ diff --git a/easytier-contrib/easytier-ios/src/error.rs b/easytier-contrib/easytier-ios/src/error.rs new file mode 100644 index 00000000..9b4b7b03 --- /dev/null +++ b/easytier-contrib/easytier-ios/src/error.rs @@ -0,0 +1,70 @@ +use std::{ + cell::RefCell, + ffi::{CStr, CString, c_char}, + ptr, +}; + +thread_local! { + // Thread-local last error for the easytier-ios C ABI. Wrapper-side + // argument/JSON failures are recorded here; easytier-ffi records + // instance/RPC failures in its own buffer. `last_error` merges both. + static LAST_ERROR: RefCell> = const { RefCell::new(Vec::new()) }; +} + +pub(crate) fn set_error(message: &str) { + LAST_ERROR.with(|cell| { + let mut buffer = cell.borrow_mut(); + buffer.clear(); + buffer.extend_from_slice(message.as_bytes()); + }); +} + +pub(crate) fn clear_error() { + LAST_ERROR.with(|cell| cell.borrow_mut().clear()); +} + +fn thread_local_error() -> Option { + LAST_ERROR.with(|cell| { + let buffer = cell.borrow(); + if buffer.is_empty() { + None + } else { + Some(String::from_utf8_lossy(&buffer).into_owned()) + } + }) +} + +fn ffi_error() -> Option { + unsafe { + let mut error_ptr: *const c_char = ptr::null(); + easytier_ffi::get_error_msg(&mut error_ptr); + if error_ptr.is_null() { + None + } else { + let error_str = CStr::from_ptr(error_ptr).to_string_lossy().into_owned(); + easytier_ffi::free_string(error_ptr); + Some(error_str) + } + } +} + +/// Merge both error layers: this wrapper's own thread-local buffer and +/// easytier-ffi's last FFI error. +pub(crate) fn last_error() -> Option { + match (ffi_error(), thread_local_error()) { + (Some(ffi_error), Some(local_error)) => Some(format!("{local_error}; {ffi_error}")), + (Some(ffi_error), None) => Some(ffi_error), + (None, Some(local_error)) => Some(local_error), + (None, None) => None, + } +} + +/// Copy the merged last error into a newly allocated C string (null when +/// there is no error). The caller owns the result and must release it with +/// `easytier_ios_free_string`. +pub(crate) fn last_error_raw() -> *mut c_char { + match last_error().and_then(|message| CString::new(message).ok()) { + Some(message) => message.into_raw(), + None => ptr::null_mut(), + } +} diff --git a/easytier-contrib/easytier-ios/src/lib.rs b/easytier-contrib/easytier-ios/src/lib.rs new file mode 100644 index 00000000..c238b24c --- /dev/null +++ b/easytier-contrib/easytier-ios/src/lib.rs @@ -0,0 +1,1066 @@ +//! iOS-facing C ABI for EasyTier. +//! +//! This crate is the iOS counterpart of `easytier-android-jni`: a thin +//! wrapper over the easytier-ffi C ABI (instance management + the generic +//! `call_json_rpc` management bridge) for apps that embed EasyTier without +//! a TUN device or NEPacketTunnel. Loopback port forwarding into the +//! virtual network is configured through `api.config.ConfigRpcService/PatchConfig` +//! port-forward patches; there is no forwarder in this crate. It builds as +//! a static library for the Apple toolchain; `easytier-ios.h` declares the +//! callable surface. +//! +//! All exported functions are panic-safe: panics are caught at the FFI +//! boundary and reported through `easytier_ios_last_error`. + +mod error; +mod strings; + +use std::{ + ffi::{CStr, c_char, c_int}, + panic::{AssertUnwindSafe, catch_unwind}, + ptr, + sync::OnceLock, +}; + +use strings::cstring_for; + +/// Install a narrow tracing subscriber for an embedding app that needs to +/// diagnose EasyTier transport failures. The subscriber writes port-forward +/// lifecycle events to stderr. +/// +/// Returns 0 on success, -1 when another global subscriber was installed +/// first (see `easytier_ios_last_error`). Repeated calls are idempotent. +#[unsafe(no_mangle)] +pub extern "C" fn easytier_ios_enable_diagnostic_logging() -> c_int { + guarded(-1, || { + error::clear_error(); + static RESULT: OnceLock> = OnceLock::new(); + let result = RESULT.get_or_init(|| { + let filter = + tracing_subscriber::EnvFilter::new("easytier_core::gateway::port_forward=info"); + tracing_subscriber::fmt() + .with_ansi(false) + .with_env_filter(filter) + .with_writer(std::io::stderr) + .try_init() + .map_err(|error| error.to_string()) + }); + match result { + Ok(()) => 0, + Err(message) => { + error::set_error(&format!("failed to enable diagnostic logging: {message}")); + -1 + } + } + }) +} + +/// Run `body` catching any panic; panics are recorded in the last-error +/// buffer so they never unwind across the FFI boundary. +fn guarded(default: R, body: impl FnOnce() -> R) -> R { + match catch_unwind(AssertUnwindSafe(body)) { + Ok(result) => result, + Err(_) => { + error::set_error("internal panic in easytier-ios"); + default + } + } +} + +/// Read a required non-null C string argument. +/// +/// # Safety +/// When non-null, `value` must point to a NUL-terminated string. +unsafe fn cstr_arg<'a>(value: *const c_char, what: &str) -> Result<&'a str, String> { + if value.is_null() { + return Err(format!("{what} must not be null")); + } + // SAFETY: caller guarantees `value` is NUL-terminated. + let text = unsafe { CStr::from_ptr(value) }; + text.to_str() + .map_err(|_| format!("{what} is not valid UTF-8")) +} + +/// Start one EasyTier network instance from a TOML config string. +/// +/// Returns 0 on success, -1 on failure (see `easytier_ios_last_error`). +/// +/// # Safety +/// `toml` must be a non-null pointer to a NUL-terminated UTF-8 string. +#[unsafe(no_mangle)] +pub unsafe extern "C" fn easytier_ios_run_instance(toml: *const c_char) -> c_int { + unsafe { + guarded(-1, || { + error::clear_error(); + let toml = match cstr_arg(toml, "config string") { + Ok(toml) => toml, + Err(message) => { + error::set_error(&message); + return -1; + } + }; + easytier_ffi::run_network_instance(toml.as_ptr() as *const c_char) + }) + } +} + +/// Keep the named instances and stop all others. +/// +/// `names_json` is a JSON array of instance name strings; null, empty or `[]` +/// stops every running instance. Returns 0 on success, -1 on failure. +/// +/// # Safety +/// `names_json` may be null; when non-null it must point to a NUL-terminated +/// UTF-8 string containing a JSON array of strings. +#[unsafe(no_mangle)] +pub unsafe extern "C" fn easytier_ios_retain_instances(names_json: *const c_char) -> c_int { + unsafe { + guarded(-1, || { + error::clear_error(); + let names: Vec = if names_json.is_null() { + Vec::new() + } else { + let text = match cstr_arg(names_json, "instance names JSON") { + Ok(text) => text, + Err(message) => { + error::set_error(&message); + return -1; + } + }; + if text.trim().is_empty() { + Vec::new() + } else { + match serde_json::from_str(text) { + Ok(names) => names, + Err(parse_error) => { + error::set_error(&format!( + "invalid instance names JSON: {parse_error}" + )); + return -1; + } + } + } + }; + let c_names: Vec = match names + .iter() + .map(|name| cstring_for(name, "instance name")) + .collect::>>() + { + Ok(c_names) => c_names, + Err(invalid) => { + error::set_error(&invalid.to_string()); + return -1; + } + }; + let pointers: Vec<*const c_char> = c_names.iter().map(|name| name.as_ptr()).collect(); + easytier_ffi::retain_network_instance(pointers.as_ptr(), pointers.len()) + }) + } +} + +/// Stop exactly one named instance without affecting other instances. +/// +/// Unknown names are ignored. Returns 0 on success, -1 on failure. +/// +/// # Safety +/// `instance_name` must be a non-null pointer to a NUL-terminated UTF-8 string. +#[unsafe(no_mangle)] +pub unsafe extern "C" fn easytier_ios_delete_instance(instance_name: *const c_char) -> c_int { + unsafe { + guarded(-1, || { + error::clear_error(); + if let Err(message) = cstr_arg(instance_name, "instance name") { + error::set_error(&message); + return -1; + } + let instance_names = [instance_name]; + easytier_ffi::delete_network_instance(instance_names.as_ptr(), instance_names.len()) + }) + } +} + +/// Collect running instance information as a JSON object mapping each +/// instance name to its running info JSON. +/// +/// Returns a newly allocated string the caller must release with +/// `easytier_ios_free_string`, or null on failure (see +/// `easytier_ios_last_error`). +#[unsafe(no_mangle)] +pub extern "C" fn easytier_ios_collect_network_infos(max_length: c_int) -> *mut c_char { + guarded(ptr::null_mut(), || { + error::clear_error(); + let max_length = max_length.max(0) as usize; + let mut infos = vec![ + easytier_ffi::KeyValuePair { + key: ptr::null(), + value: ptr::null(), + }; + max_length + ]; + // SAFETY: `infos` is writable storage for `max_length` entries; every + // returned key/value string is released below with + // `easytier_ffi::free_string`. + let count = unsafe { easytier_ffi::collect_network_infos(infos.as_mut_ptr(), max_length) }; + if count < 0 { + return ptr::null_mut(); + } + let mut map = serde_json::Map::new(); + for info in infos.iter().take(count as usize) { + if info.key.is_null() || info.value.is_null() { + break; + } + // SAFETY: non-null entries are NUL-terminated strings allocated by + // easytier-ffi, each released exactly once after copying. + let (key, value) = unsafe { + let key = CStr::from_ptr(info.key).to_string_lossy().into_owned(); + let value = CStr::from_ptr(info.value).to_string_lossy().into_owned(); + easytier_ffi::free_string(info.key); + easytier_ffi::free_string(info.value); + (key, value) + }; + let value = serde_json::from_str(&value).unwrap_or(serde_json::Value::String(value)); + map.insert(key, value); + } + let json = serde_json::Value::Object(map).to_string(); + match std::ffi::CString::new(json) { + Ok(json) => json.into_raw(), + Err(_) => { + error::set_error("network info JSON contains a null byte"); + ptr::null_mut() + } + } + }) +} + +/// Call an exposed EasyTier management RPC method using protobuf JSON. +/// +/// Thin wrapper over easytier-ffi's `call_json_rpc` with a null domain (the +/// generic management registry). `payload_json` must contain the protobuf +/// JSON request, including any `instance` selector required by the target +/// RPC. Port forwarding into the virtual network is driven through this +/// bridge with `api.config.ConfigRpcService/PatchConfig` port-forward patches. +/// +/// Returns a newly allocated JSON response string the caller must release +/// with `easytier_ios_free_string`, or null on failure (see +/// `easytier_ios_last_error`). +/// +/// # Safety +/// `service_name`, `method_name` and `payload_json` must be non-null +/// pointers to NUL-terminated UTF-8 strings. +#[unsafe(no_mangle)] +pub unsafe extern "C" fn easytier_ios_call_json_rpc( + service_name: *const c_char, + method_name: *const c_char, + payload_json: *const c_char, +) -> *mut c_char { + unsafe { + guarded(ptr::null_mut(), || { + error::clear_error(); + if let Err(message) = cstr_arg(service_name, "service name") { + error::set_error(&message); + return ptr::null_mut(); + } + if let Err(message) = cstr_arg(method_name, "method name") { + error::set_error(&message); + return ptr::null_mut(); + } + let payload = match cstr_arg(payload_json, "payload JSON") { + Ok(payload) => payload, + Err(message) => { + error::set_error(&message); + return ptr::null_mut(); + } + }; + let mut out: *const c_char = ptr::null(); + let rc = easytier_ffi::call_json_rpc( + service_name, + method_name, + ptr::null(), + payload.as_ptr() as *const c_char, + &mut out, + ); + if rc != 0 { + // easytier-ffi already recorded the failure in its own + // last-error buffer. + return ptr::null_mut(); + } + out as *mut c_char + }) + } +} + +/// Return the last error message on this thread, or null when there is none. +/// +/// Combines wrapper-side errors recorded by this library with the +/// easytier-ffi last FFI error. The returned string is newly allocated and +/// must be released with `easytier_ios_free_string`. +#[unsafe(no_mangle)] +pub extern "C" fn easytier_ios_last_error() -> *mut c_char { + guarded(ptr::null_mut(), error::last_error_raw) +} + +/// Release a string returned by `easytier_ios_collect_network_infos`, +/// `easytier_ios_call_json_rpc` or `easytier_ios_last_error`. Passing null +/// is a no-op. +/// +/// # Safety +/// `s` must be null or a string previously returned by this library, not yet +/// freed. +#[unsafe(no_mangle)] +pub unsafe extern "C" fn easytier_ios_free_string(s: *mut c_char) { + let _ = catch_unwind(AssertUnwindSafe(|| { + if !s.is_null() { + // SAFETY: caller guarantees `s` came from this library. + drop(unsafe { std::ffi::CString::from_raw(s) }); + } + })); +} + +#[cfg(test)] +mod tests { + //! Host-target smoke tests for the C ABI surface. They exercise the + //! wrapper logic (argument validation, error propagation, string + //! ownership) plus the full port-forward patch path against real `no_tun` + //! instances; two of them link instances over the process-wide ring + //! registry (through the always-on `ring://` listener and a + //! TOML ring connector) to verify a patched TCP port forward end to end. + + use super::*; + use std::{ + ffi::{CStr, CString}, + io::{Read, Write}, + net::TcpStream, + sync::{ + Arc, Mutex, MutexGuard, + atomic::{AtomicBool, Ordering}, + }, + }; + + /// Tests share the process-wide easytier-ffi state; run them serially. + static TEST_LOCK: Mutex<()> = Mutex::new(()); + + fn init_logs() { + use std::sync::Once; + static LOG_INIT: Once = Once::new(); + LOG_INIT.call_once(|| { + let filter = tracing_subscriber::EnvFilter::new( + std::env::var("RUST_LOG") + .unwrap_or_else(|_| "easytier_core=debug,easytier=debug,info".to_owned()), + ); + let _ = tracing_subscriber::fmt() + .with_env_filter(filter) + .with_writer(std::io::stderr) + .try_init(); + }); + } + + fn acquire() -> MutexGuard<'static, ()> { + init_logs(); + TEST_LOCK.lock().unwrap_or_else(|error| error.into_inner()) + } + + fn unique_instance(tag: &str) -> String { + format!( + "ios-{tag}-{}", + &uuid::Uuid::new_v4().simple().to_string()[..12] + ) + } + + fn minimal_config(instance: &str) -> CString { + CString::new(format!( + "instance_name = \"{instance}\"\n\ + instance_id = \"{}\"\n\ + ipv4 = \"10.126.126.1\"\n\ + listeners = []\n\ + flags.no_tun = true\n", + uuid::Uuid::new_v4() + )) + .unwrap() + } + + fn retain_all() { + let empty = CString::new("[]").unwrap(); + // SAFETY: `empty` is a valid NUL-terminated string. + assert_eq!(unsafe { easytier_ios_retain_instances(empty.as_ptr()) }, 0); + } + + struct InstanceGuard; + + impl InstanceGuard { + fn run(instance: &str) -> Self { + let config = minimal_config(instance); + // SAFETY: `config` is a valid NUL-terminated string. + let result = unsafe { easytier_ios_run_instance(config.as_ptr()) }; + assert_eq!(result, 0, "run_instance failed: {:?}", take_last_error()); + Self + } + } + + impl Drop for InstanceGuard { + fn drop(&mut self) { + retain_all(); + } + } + + fn take_last_error() -> Option { + let error_ptr = easytier_ios_last_error(); + if error_ptr.is_null() { + return None; + } + // SAFETY: `error_ptr` was returned by `easytier_ios_last_error` and is + // freed exactly once below. + let message = unsafe { CStr::from_ptr(error_ptr) } + .to_string_lossy() + .into_owned(); + unsafe { easytier_ios_free_string(error_ptr) }; + Some(message) + } + + fn collect_infos_json() -> String { + let json_ptr = easytier_ios_collect_network_infos(32); + assert!( + !json_ptr.is_null(), + "collect failed: {:?}", + take_last_error() + ); + // SAFETY: `json_ptr` was returned by + // `easytier_ios_collect_network_infos` and is freed exactly once below. + let json = unsafe { CStr::from_ptr(json_ptr) } + .to_string_lossy() + .into_owned(); + unsafe { easytier_ios_free_string(json_ptr) }; + json + } + + /// Call a management RPC through the wrapper and return the response JSON. + fn call_json_rpc(service: &str, method: &str, payload: &serde_json::Value) -> String { + let service_c = CString::new(service).unwrap(); + let method_c = CString::new(method).unwrap(); + let payload_c = CString::new(payload.to_string()).unwrap(); + // SAFETY: all three pointers are valid NUL-terminated strings. + let response_ptr = unsafe { + easytier_ios_call_json_rpc(service_c.as_ptr(), method_c.as_ptr(), payload_c.as_ptr()) + }; + assert!( + !response_ptr.is_null(), + "{service}/{method} failed: {:?}", + take_last_error() + ); + // SAFETY: `response_ptr` was returned by `easytier_ios_call_json_rpc` + // and is freed exactly once below. + let response = unsafe { CStr::from_ptr(response_ptr) } + .to_string_lossy() + .into_owned(); + unsafe { easytier_ios_free_string(response_ptr) }; + response + } + + /// The port-forward patch JSON used by the app: one ADD/REMOVE entry over + /// `api.config.ConfigRpcService/PatchConfig`, IPv4 addresses encoded as + /// network-order u32. + fn port_forward_patch(instance: &str, action: &str, bind_port: u16) -> serde_json::Value { + serde_json::json!({ + "patch": { + "port_forwards": [{ + "action": action, + "cfg": { + "bind_addr": {"ipv4": {"addr": 0x7F000001}, "port": bind_port}, + "dst_addr": {"ipv4": {"addr": 0x0A7E7E63}, "port": 80}, + "socket_type": "TCP", + }, + }], + }, + "instance": {"instance_selector": {"name": instance}}, + }) + } + + #[test] + fn run_collect_retain_roundtrip() { + let _guard = acquire(); + let instance = unique_instance("smoke"); + let _instance = InstanceGuard::run(&instance); + + let json = collect_infos_json(); + let infos: serde_json::Map = + serde_json::from_str(&json).expect("collect_network_infos must return a JSON object"); + assert!( + infos.contains_key(&instance), + "instance {instance} missing from collected infos: {json}" + ); + + retain_all(); + let infos = collect_infos_json(); + assert!( + !infos.contains(&instance), + "instance {instance} still listed after retain([]): {infos}" + ); + } + + #[test] + fn delete_instance_stops_only_the_named_instance() { + let _guard = acquire(); + let deleted = unique_instance("delete"); + let retained = unique_instance("retain"); + let _deleted_instance = InstanceGuard::run(&deleted); + let _retained_instance = InstanceGuard::run(&retained); + + let deleted_name = CString::new(deleted.clone()).unwrap(); + // SAFETY: `deleted_name` is a valid NUL-terminated string. + assert_eq!( + unsafe { easytier_ios_delete_instance(deleted_name.as_ptr()) }, + 0, + "delete failed: {:?}", + take_last_error() + ); + + let infos = collect_infos_json(); + assert!( + !infos.contains(&deleted), + "deleted instance remains: {infos}" + ); + assert!( + infos.contains(&retained), + "unrelated instance stopped: {infos}" + ); + } + + #[test] + fn retain_instances_rejects_invalid_json() { + let _guard = acquire(); + let garbage = CString::new("not json").unwrap(); + // SAFETY: `garbage` is a valid NUL-terminated string. + let result = unsafe { easytier_ios_retain_instances(garbage.as_ptr()) }; + assert_eq!(result, -1); + let message = take_last_error().expect("last error must be set"); + assert!( + message.contains("invalid instance names JSON"), + "unexpected error: {message}" + ); + } + + #[test] + fn call_json_rpc_rejects_null_payload() { + let _guard = acquire(); + let service = CString::new("api.config.ConfigRpcService").unwrap(); + let method = CString::new("PatchConfig").unwrap(); + // SAFETY: service/method are valid; payload null must be rejected. + let response = + unsafe { easytier_ios_call_json_rpc(service.as_ptr(), method.as_ptr(), ptr::null()) }; + assert!(response.is_null()); + let message = take_last_error().expect("last error must be set"); + assert!( + message.contains("payload JSON"), + "unexpected error: {message}" + ); + } + + #[test] + fn call_json_rpc_rejects_null_service_and_method() { + let _guard = acquire(); + let service = CString::new("api.config.ConfigRpcService").unwrap(); + let method = CString::new("PatchConfig").unwrap(); + let payload = CString::new("{}").unwrap(); + + // SAFETY: method and payload are valid; service null must be rejected. + let response = + unsafe { easytier_ios_call_json_rpc(ptr::null(), method.as_ptr(), payload.as_ptr()) }; + assert!(response.is_null()); + let message = take_last_error().expect("last error must be set"); + assert!( + message.contains("service name"), + "unexpected error: {message}" + ); + + // SAFETY: service and payload are valid; method null must be rejected. + let response = + unsafe { easytier_ios_call_json_rpc(service.as_ptr(), ptr::null(), payload.as_ptr()) }; + assert!(response.is_null()); + let message = take_last_error().expect("last error must be set"); + assert!( + message.contains("method name"), + "unexpected error: {message}" + ); + } + + #[test] + fn port_forward_patch_roundtrip() { + let _guard = acquire(); + let instance = unique_instance("patch"); + let _instance = InstanceGuard::run(&instance); + + // ADD a loopback TCP forward to an unreachable virtual target; the + // patch itself only installs the listener and must succeed. + call_json_rpc( + "api.config.ConfigRpcService", + "PatchConfig", + &port_forward_patch(&instance, "ADD", 18787), + ); + + // The forward is now visible in the instance config dump. + let config = call_json_rpc( + "api.config.ConfigRpcService", + "GetConfig", + &serde_json::json!({ + "instance": {"instance_selector": {"name": instance}}, + }), + ); + assert!( + config.contains("18787"), + "patched forward missing from config dump: {config}" + ); + + // REMOVE matches the whole PortForwardConfig entry (patch_vec uses + // PartialEq), so the dst fields must repeat the ADD values. + call_json_rpc( + "api.config.ConfigRpcService", + "PatchConfig", + &port_forward_patch(&instance, "REMOVE", 18787), + ); + let config = call_json_rpc( + "api.config.ConfigRpcService", + "GetConfig", + &serde_json::json!({ + "instance": {"instance_selector": {"name": instance}}, + }), + ); + assert!( + !config.contains("18787"), + "removed forward still in config dump: {config}" + ); + } + + /// Ownership of an easytier-ffi allocated string: freed exactly once on + /// drop, in a thread that itself calls back into easytier-ffi so the + /// easytier-ffi free runs on a thread the runtime has seen. + struct FfiString(*const c_char); + + impl Drop for FfiString { + fn drop(&mut self) { + if !self.0.is_null() { + // `self.0` was allocated by easytier-ffi and is freed exactly + // once here. + easytier_ffi::free_string(self.0); + } + } + } + + /// Wait for the next data-plane completion on `session`. + fn next_completion( + session: u64, + timeout: std::time::Duration, + ) -> easytier_ffi::DataPlaneCompletion { + let deadline = std::time::Instant::now() + timeout; + loop { + let ready = easytier_ffi::data_plane_completion_wait(session, 1_000); + assert!(ready >= 0, "completion wait failed: {ready}"); + if ready == 1 { + let mut completions = [easytier_ffi::DataPlaneCompletion::default(); 1]; + // SAFETY: `completions` is writable storage for one value. + let drained = unsafe { + easytier_ffi::data_plane_completion_drain(session, completions.as_mut_ptr(), 1) + }; + assert!(drained >= 0, "completion drain failed: {drained}"); + if drained == 1 { + return completions[0]; + } + } + assert!(std::time::Instant::now() < deadline, "completion timed out"); + } + } + + /// Run a data-plane echo server on `instance`: bind `port` on the + /// instance's virtual stack, then loop accepting a stream, reading one + /// payload and replying verbatim, until `stop` is set. Holds the + /// instance's single native data-plane session for its lifetime. + fn data_plane_echo_server(instance: &str, port: u16, stop: Arc) { + let instance = CString::new(instance).unwrap(); + let mut session = 0u64; + // SAFETY: `instance` is a valid NUL-terminated string and `session` + // points to writable storage for one `u64`. + let result = + unsafe { easytier_ffi::data_plane_session_open(instance.as_ptr(), &mut session) }; + assert_eq!(result, 0, "session open failed: {result}"); + let session = session; + + // The data-plane runtime starts after the instance; retry the bind + // until it is up. + let mut bind_operation = 0u64; + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(15); + loop { + // SAFETY: `bind_operation` points to writable storage for one `u64`. + let result = unsafe { + easytier_ffi::data_plane_tcp_bind_submit(session, port, 15_000, &mut bind_operation) + }; + if result == 0 { + break; + } + assert!( + std::time::Instant::now() < deadline, + "bind submit failed: {:?}", + take_last_error() + ); + std::thread::sleep(std::time::Duration::from_millis(50)); + } + let bind_completion = next_completion(session, std::time::Duration::from_secs(15)); + assert_eq!(bind_completion.operation_id, bind_operation); + assert_eq!( + bind_completion.status, 0, + "bind failed: status={} kind={}", + bind_completion.status, bind_completion.operation_kind + ); + let mut listener = 0u64; + let mut local = easytier_ffi::DataPlaneSocketAddr::default(); + // SAFETY: outputs reference writable, non-overlapping storage. + let result = unsafe { + easytier_ffi::data_plane_tcp_bind_result_take( + session, + bind_operation, + &mut listener, + &mut local, + ) + }; + assert_eq!(result, 0, "bind result take failed: {result}"); + + while !stop.load(Ordering::Acquire) { + let mut accept_operation = 0u64; + // SAFETY: `accept_operation` points to writable storage for one `u64`. + let result = unsafe { + easytier_ffi::data_plane_tcp_accept_submit( + session, + listener, + 5_000, + &mut accept_operation, + ) + }; + assert_eq!(result, 0, "accept submit failed: {result}"); + let accept_completion = next_completion(session, std::time::Duration::from_secs(10)); + assert_eq!(accept_completion.operation_id, accept_operation); + if accept_completion.status != 0 { + // Accept timed out or the listener was closed; re-check stop + // and either loop for the next connection or exit. + easytier_ffi::data_plane_operation_free(session, accept_operation); + if stop.load(Ordering::Acquire) { + break; + } + continue; + } + let mut stream = 0u64; + let mut peer = easytier_ffi::DataPlaneSocketAddr::default(); + // SAFETY: outputs reference writable, non-overlapping storage. + let result = unsafe { + easytier_ffi::data_plane_tcp_accept_result_take( + session, + accept_operation, + &mut stream, + &mut local, + &mut peer, + ) + }; + assert_eq!(result, 0, "accept result take failed: {result}"); + + let mut read_operation = 0u64; + // SAFETY: `read_operation` points to writable storage for one `u64`. + let result = unsafe { + easytier_ffi::data_plane_tcp_read_submit(session, stream, 4096, &mut read_operation) + }; + assert_eq!(result, 0, "read submit failed: {result}"); + let read_completion = next_completion(session, std::time::Duration::from_secs(15)); + assert_eq!(read_completion.operation_id, read_operation); + assert_eq!( + read_completion.status, 0, + "read failed: {}", + read_completion.status + ); + let mut data = [0u8; 4096]; + let mut len = 0u32; + let mut eof = false; + // SAFETY: `data` is writable for its length and the scalar outputs + // reference writable, non-overlapping storage. + let result = unsafe { + easytier_ffi::data_plane_tcp_read_result_take( + session, + read_operation, + data.as_mut_ptr(), + data.len() as u32, + &mut len, + &mut eof, + ) + }; + assert_eq!(result, 0, "read result take failed: {result}"); + assert!(!eof, "unexpected eof before payload"); + + // Echo the received payload back verbatim. + let payload = &data[..len as usize]; + let mut write_operation = 0u64; + // SAFETY: the ABI copies `payload` before returning. + let result = unsafe { + easytier_ffi::data_plane_tcp_write_submit( + session, + stream, + payload.as_ptr(), + payload.len() as u32, + &mut write_operation, + ) + }; + assert_eq!(result, 0, "write submit failed: {result}"); + let write_completion = next_completion(session, std::time::Duration::from_secs(15)); + assert_eq!(write_completion.operation_id, write_operation); + assert_eq!( + write_completion.status, 0, + "write failed: {}", + write_completion.status + ); + + // `stream` was accepted above and is closed exactly once. + easytier_ffi::data_plane_resource_close(session, stream); + } + + // `listener` and `session` were opened above and are closed exactly + // once here. + easytier_ffi::data_plane_resource_close(session, listener); + easytier_ffi::data_plane_session_close(session); + } + + #[test] + fn ring_connector_pair_sees_peer() { + let _guard = acquire(); + let server = unique_instance("srv"); + let client = unique_instance("cli"); + let server_id = uuid::Uuid::new_v4(); + let client_id = uuid::Uuid::new_v4(); + + let server_config = CString::new(format!( + "instance_name = \"{server}\"\n\ + instance_id = \"{server_id}\"\n\ + ipv4 = \"10.126.126.99\"\n\ + listeners = []\n\ + flags.no_tun = true\n" + )) + .unwrap(); + // SAFETY: `server_config` is a valid NUL-terminated string. + assert_eq!( + unsafe { easytier_ios_run_instance(server_config.as_ptr()) }, + 0, + "run server failed: {:?}", + take_last_error() + ); + let _cleanup = InstanceGuard; + + let client_config = CString::new(format!( + "instance_name = \"{client}\"\n\ + instance_id = \"{client_id}\"\n\ + ipv4 = \"10.126.126.100\"\n\ + listeners = []\n\ + flags.no_tun = true\n\ + [[peer]]\n\ + uri = \"ring://{server_id}\"\n", + )) + .unwrap(); + // SAFETY: `client_config` is a valid NUL-terminated string. + assert_eq!( + unsafe { easytier_ios_run_instance(client_config.as_ptr()) }, + 0, + "run client failed: {:?}", + take_last_error() + ); + + // Peer setup over the ring registry is asynchronous; wait until the + // client reports a connected route to the server instance before + // relying on overlay connectivity. + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(15); + loop { + let peers = call_json_rpc( + "api.instance.PeerManageRpcService", + "ListPeer", + &serde_json::json!({ + "instance": {"instance_selector": {"name": client}}, + }), + ); + eprintln!("[dbg] ListPeer raw: {peers}"); + let peers: serde_json::Value = serde_json::from_str(&peers).unwrap(); + let connected = peers["peer_infos"] + .as_array() + .map(|infos| { + infos.iter().any(|info| { + info["directly_connected_conns"] + .as_array() + .is_some_and(|conns| !conns.is_empty()) + }) + }) + .unwrap_or(false); + if connected { + break; + } + assert!( + std::time::Instant::now() < deadline, + "client never connected to server: {peers}" + ); + std::thread::sleep(std::time::Duration::from_millis(200)); + } + } + + /// End to end: two instances linked over the ring registry, a TCP echo + /// listener on the far end's virtual IP, and a patched loopback port + /// forward on the near end carrying a full byte round trip. + #[test] + fn port_forward_echo_roundtrip_over_ring() { + let _guard = acquire(); + let server = unique_instance("srv"); + let client = unique_instance("cli"); + let server_id = uuid::Uuid::new_v4(); + let client_id = uuid::Uuid::new_v4(); + let server_ipv4 = "10.126.126.99"; + + let server_config = CString::new(format!( + "instance_name = \"{server}\"\n\ + instance_id = \"{server_id}\"\n\ + ipv4 = \"{server_ipv4}\"\n\ + listeners = []\n\ + flags.no_tun = true\n" + )) + .unwrap(); + // SAFETY: `server_config` is a valid NUL-terminated string. + assert_eq!( + unsafe { easytier_ios_run_instance(server_config.as_ptr()) }, + 0, + "run server failed: {:?}", + take_last_error() + ); + let _cleanup = InstanceGuard; + + let client_config = CString::new(format!( + "instance_name = \"{client}\"\n\ + instance_id = \"{client_id}\"\n\ + ipv4 = \"10.126.126.100\"\n\ + listeners = []\n\ + flags.no_tun = true\n\ + [[peer]]\n\ + uri = \"ring://{server_id}\"\n" + )) + .unwrap(); + // SAFETY: `client_config` is a valid NUL-terminated string. + assert_eq!( + unsafe { easytier_ios_run_instance(client_config.as_ptr()) }, + 0, + "run client failed: {:?}", + take_last_error() + ); + + // Peer setup over the ring registry is asynchronous; wait until the + // client reports a connected route to the server instance before + // relying on overlay connectivity. + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(15); + loop { + let peers = call_json_rpc( + "api.instance.PeerManageRpcService", + "ListPeer", + &serde_json::json!({ + "instance": {"instance_selector": {"name": client}}, + }), + ); + let peers: serde_json::Value = serde_json::from_str(&peers).unwrap(); + let connected = peers["peer_infos"] + .as_array() + .map(|infos| { + infos.iter().any(|info| { + info["directly_connected_conns"] + .as_array() + .is_some_and(|conns| !conns.is_empty()) + }) + }) + .unwrap_or(false); + if connected { + break; + } + assert!( + std::time::Instant::now() < deadline, + "client never connected to server: {peers}" + ); + std::thread::sleep(std::time::Duration::from_millis(200)); + } + + // The forward target must be a TCP listener on the server instance's + // virtual IP. In `no_tun` mode the kernel has no such address, so the + // server side opens one through its own data plane: an + // easytier-ffi data-plane session bound on the virtual stack, with a + // blocking accept/read/write echo loop. + let echo_port: u16 = 23456; + let echo_stop = Arc::new(AtomicBool::new(false)); + let echo = std::thread::spawn({ + let echo_stop = echo_stop.clone(); + let server = server.clone(); + move || data_plane_echo_server(&server, echo_port, echo_stop) + }); + + // Patch in the loopback forward on the client instance. Peer setup is + // asynchronous; the patch itself installs the listener regardless, so + // retry only while the RPC itself fails (e.g. instance not ready). + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(15); + let bind_port = 18788; + let patch = serde_json::json!({ + "patch": { + "port_forwards": [{ + "action": "ADD", + "cfg": { + "bind_addr": {"ipv4": {"addr": 0x7F000001}, "port": bind_port}, + "dst_addr": {"ipv4": {"addr": 0x0A7E7E63}, "port": echo_port}, + "socket_type": "TCP", + }, + }], + }, + "instance": {"instance_selector": {"name": client}}, + }); + loop { + let service = CString::new("api.config.ConfigRpcService").unwrap(); + let method = CString::new("PatchConfig").unwrap(); + let payload = CString::new(patch.to_string()).unwrap(); + let mut out: *const c_char = ptr::null(); + // SAFETY: all pointers are valid NUL-terminated strings; `out` is + // owned by easytier-ffi and released by the FfiString guard below. + let rc = unsafe { + easytier_ffi::call_json_rpc( + service.as_ptr(), + method.as_ptr(), + ptr::null(), + payload.as_ptr(), + &mut out, + ) + }; + let _response = FfiString(out); + if rc == 0 { + break; + } + assert!( + std::time::Instant::now() < deadline, + "patch_config never succeeded: {:?}", + take_last_error() + ); + std::thread::sleep(std::time::Duration::from_millis(100)); + } + + // Dial the patched loopback forward until the data-plane path to the + // server virtual IP comes up, then verify an echo round trip. + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(15); + loop { + if let Ok(mut stream) = TcpStream::connect(("127.0.0.1", bind_port)) { + stream + .set_read_timeout(Some(std::time::Duration::from_secs(2))) + .unwrap(); + if stream.write_all(b"ping").is_ok() { + let mut buf = [0u8; 4]; + if stream.read_exact(&mut buf).is_ok() && &buf == b"ping" { + break; + } + } + } + assert!( + std::time::Instant::now() < deadline, + "no echo through patched port forward" + ); + std::thread::sleep(std::time::Duration::from_millis(100)); + } + + echo_stop.store(true, Ordering::Release); + echo.join().expect("echo server thread panicked"); + } +} diff --git a/easytier-contrib/easytier-ios/src/strings.rs b/easytier-contrib/easytier-ios/src/strings.rs new file mode 100644 index 00000000..9a1fc97d --- /dev/null +++ b/easytier-contrib/easytier-ios/src/strings.rs @@ -0,0 +1,11 @@ +use std::ffi::CString; + +/// Build a NUL-terminated C string from a Rust string for FFI calls. +pub(crate) fn cstring_for(value: &str, what: &str) -> std::io::Result { + CString::new(value).map_err(|_| { + std::io::Error::new( + std::io::ErrorKind::InvalidInput, + format!("{what} contains a null byte"), + ) + }) +} diff --git a/easytier-core/src/config/api.rs b/easytier-core/src/config/api.rs index 69bc9ada..03b98d9a 100644 --- a/easytier-core/src/config/api.rs +++ b/easytier-core/src/config/api.rs @@ -153,6 +153,7 @@ pub fn network_config_from_toml(config: &TomlConfig) -> NetworkConfig { result.disable_udp_hole_punching = Some(flags.disable_udp_hole_punching); result.disable_upnp = Some(flags.disable_upnp); result.disable_relay_data = Some(flags.disable_relay_data); + result.prefer_peer_relay = Some(flags.prefer_peer_relay); result.enable_udp_broadcast_relay = Some(flags.enable_udp_broadcast_relay); result.disable_sym_hole_punching = Some(flags.disable_sym_hole_punching); result.enable_magic_dns = Some(flags.accept_dns); diff --git a/easytier-core/src/config/api_input.rs b/easytier-core/src/config/api_input.rs index 5764f92b..174ede6e 100644 --- a/easytier-core/src/config/api_input.rs +++ b/easytier-core/src/config/api_input.rs @@ -473,6 +473,10 @@ impl NetworkConfigExt for NetworkConfig { flags.disable_relay_data = disable_relay_data; } + if let Some(prefer_peer_relay) = self.prefer_peer_relay { + flags.prefer_peer_relay = prefer_peer_relay; + } + if let Some(enable_udp_broadcast_relay) = self.enable_udp_broadcast_relay { flags.enable_udp_broadcast_relay = enable_udp_broadcast_relay; } @@ -672,6 +676,7 @@ impl NetworkConfigExt for NetworkConfig { result.disable_udp_hole_punching = Some(flags.disable_udp_hole_punching); result.disable_upnp = Some(flags.disable_upnp); result.disable_relay_data = Some(flags.disable_relay_data); + result.prefer_peer_relay = Some(flags.prefer_peer_relay); result.enable_udp_broadcast_relay = Some(flags.enable_udp_broadcast_relay); result.disable_sym_hole_punching = Some(flags.disable_sym_hole_punching); result.enable_magic_dns = Some(flags.accept_dns); @@ -775,6 +780,24 @@ mod tests { assert_eq!(output.managed_credentials[0].reusable, Some(true)); } + #[test] + fn peer_relay_preference_round_trips_independently() { + let input = NetworkConfig { + disable_relay_data: Some(false), + prefer_peer_relay: Some(true), + ..standalone_config() + }; + + let config = input.gen_config().unwrap(); + let flags = config.get_flags(); + assert!(!flags.disable_relay_data); + assert!(flags.prefer_peer_relay); + + let output = NetworkConfig::new_from_config(&config).unwrap(); + assert_eq!(output.disable_relay_data, Some(false)); + assert_eq!(output.prefer_peer_relay, Some(true)); + } + #[test] fn legacy_enabled_vpn_portal_config_reports_migration_error() { let error = NetworkConfig { diff --git a/easytier-core/src/config/toml.rs b/easytier-core/src/config/toml.rs index c56ed849..db2ecced 100644 --- a/easytier-core/src/config/toml.rs +++ b/easytier-core/src/config/toml.rs @@ -69,6 +69,7 @@ pub fn gen_default_flags() -> Flags { instance_recv_bps_limit: u64::MAX, disable_upnp: false, disable_relay_data: false, + prefer_peer_relay: false, enable_udp_broadcast_relay: false, socket_mark: None, } @@ -162,6 +163,7 @@ define_flags_diff! { need_p2p, disable_upnp, disable_relay_data, + prefer_peer_relay, enable_udp_broadcast_relay, socket_mark, ], @@ -1488,6 +1490,7 @@ socket_mark = 66 flags.bind_device = false; flags.enable_ipv6 = false; flags.relay_network_whitelist = "".to_string(); + flags.prefer_peer_relay = true; flags.mtu = 0; flags.foreign_relay_bps_limit = u64::MAX - 1; flags.instance_recv_bps_limit = u64::MAX - 2; @@ -1521,6 +1524,7 @@ socket_mark = 66 assert!(!reloaded_flags.bind_device); assert!(!reloaded_flags.enable_ipv6); assert_eq!(reloaded_flags.relay_network_whitelist, ""); + assert!(reloaded_flags.prefer_peer_relay); assert_eq!(reloaded_flags.mtu, 0); assert_eq!(reloaded_flags.foreign_relay_bps_limit, u64::MAX - 1); assert_eq!(reloaded_flags.instance_recv_bps_limit, u64::MAX - 2); diff --git a/easytier-core/src/connectivity/protocol/raw.rs b/easytier-core/src/connectivity/protocol/raw.rs index 8c05ab8e..bec69bd9 100644 --- a/easytier-core/src/connectivity/protocol/raw.rs +++ b/easytier-core/src/connectivity/protocol/raw.rs @@ -16,6 +16,7 @@ use crate::{ tcp::{ TcpBindOptions, TcpListenOptions, TcpSocketListener, TcpSocketPurpose, VirtualTcpListenerFactory, VirtualTcpSocket, VirtualTcpSocketFactory, + is_retryable_tcp_io_error, }, udp::{ UdpBindOptions, UdpSession, UdpSessionAcceptKind, UdpSessionListenRequest, @@ -165,9 +166,20 @@ where } async fn accept(&mut self) -> anyhow::Result { - let local_url = self.inner.local_url(); - let socket = self.inner.accept().await?; - Ok(upgrade_accepted_tcp_with_local_url(socket, local_url)?) + loop { + let local_url = self.inner.local_url(); + let socket = self.inner.accept().await?; + match upgrade_accepted_tcp_with_local_url(socket, local_url) { + Ok(tunnel) => return Ok(tunnel), + Err(error) if is_retryable_accepted_tcp_error(&error) => { + tracing::warn!( + ?error, + "accepted tcp connection failed with retryable error" + ); + } + Err(error) => return Err(error.into()), + } + } } fn local_url(&self) -> Url { @@ -406,6 +418,14 @@ where TcpTunnelUpgrader::new(info).upgrade(socket) } +fn is_retryable_accepted_tcp_error(error: &TunnelError) -> bool { + matches!( + error, + TunnelError::IOError(error) + if is_retryable_tcp_io_error(error) + ) +} + pub(crate) fn upgrade_accepted_byte_stream( socket: S, local_url: Url, @@ -493,8 +513,10 @@ fn socket_url(scheme: &str, addr: SocketAddr) -> Url { #[cfg(test)] pub(crate) mod tests { use std::{ + collections::VecDeque, io, pin::Pin, + sync::Mutex, task::{Context, Poll}, }; @@ -503,6 +525,7 @@ pub(crate) mod tests { use crate::{ packet::ZCPacket, + socket::tcp::VirtualTcpListener, socket::udp::{UdpSessionKind, VirtualUdpSocket}, }; @@ -527,6 +550,7 @@ pub(crate) mod tests { stream: DuplexStream, local_addr: SocketAddr, peer_addr: SocketAddr, + peer_addr_error: Option, transport_label: Option<&'static str>, } @@ -545,6 +569,7 @@ pub(crate) mod tests { stream, local_addr, peer_addr, + peer_addr_error: None, transport_label: None, } } @@ -553,6 +578,11 @@ pub(crate) mod tests { self.transport_label = Some(transport_label); self } + + fn with_peer_addr_error(mut self, kind: io::ErrorKind) -> Self { + self.peer_addr_error = Some(kind); + self + } } impl AsyncRead for MockTcpSocket { @@ -589,6 +619,9 @@ pub(crate) mod tests { } fn peer_addr(&self) -> io::Result { + if let Some(kind) = self.peer_addr_error { + return Err(io::Error::new(kind, "mock peer address failure")); + } Ok(self.peer_addr) } @@ -597,6 +630,56 @@ pub(crate) mod tests { } } + struct MockTcpListener { + local_addr: SocketAddr, + accepts: Mutex>>, + } + + impl MockTcpListener { + fn new( + local_addr: SocketAddr, + accepts: Vec>, + ) -> Self { + Self { + local_addr, + accepts: Mutex::new(accepts.into_iter().collect()), + } + } + } + + #[async_trait] + impl VirtualTcpListener for MockTcpListener { + type Socket = MockTcpSocket; + + fn local_addr(&self) -> io::Result { + Ok(self.local_addr) + } + + async fn accept(&self) -> io::Result<(Self::Socket, SocketAddr)> { + let result = { self.accepts.lock().unwrap().pop_front() }; + match result { + Some(result) => result, + None => std::future::pending().await, + } + } + } + + struct MockTcpListenerFactory { + listener: Arc, + } + + #[async_trait] + impl VirtualTcpListenerFactory for MockTcpListenerFactory { + type Listener = MockTcpListener; + + async fn bind_tcp( + &self, + _options: TcpListenOptions, + ) -> anyhow::Result> { + Ok(self.listener.clone()) + } + } + struct MockUdpSocket { local_addr: SocketAddr, } @@ -686,6 +769,57 @@ pub(crate) mod tests { ); } + #[tokio::test] + async fn tcp_tunnel_listener_rejects_transient_peer_addr_error_and_continues() { + let local_addr: SocketAddr = "127.0.0.1:11013".parse().unwrap(); + let rejected_peer_addr: SocketAddr = "127.0.0.1:21013".parse().unwrap(); + let accepted_peer_addr: SocketAddr = "127.0.0.1:21014".parse().unwrap(); + let listener = Arc::new(MockTcpListener::new( + local_addr, + vec![ + Ok(( + MockTcpSocket::new(local_addr, rejected_peer_addr) + .with_peer_addr_error(io::ErrorKind::NotConnected), + rejected_peer_addr, + )), + Ok(( + MockTcpSocket::new(local_addr, accepted_peer_addr), + accepted_peer_addr, + )), + ], + )); + let factory = Arc::new(MockTcpListenerFactory { listener }); + let mut tunnel_listener = TcpTunnelListener::new(local_addr, factory); + + tunnel_listener.listen().await.unwrap(); + let tunnel = tunnel_listener.accept().await.unwrap(); + let info = tunnel.info().unwrap(); + + assert_eq!( + info.remote_addr.unwrap().url, + format!("tcp://{accepted_peer_addr}") + ); + } + + #[tokio::test] + async fn tcp_tunnel_listener_propagates_non_retryable_peer_addr_error() { + let local_addr: SocketAddr = "127.0.0.1:11013".parse().unwrap(); + let peer_addr: SocketAddr = "127.0.0.1:21013".parse().unwrap(); + let listener = Arc::new(MockTcpListener::new( + local_addr, + vec![Ok(( + MockTcpSocket::new(local_addr, peer_addr) + .with_peer_addr_error(io::ErrorKind::Other), + peer_addr, + ))], + )); + let factory = Arc::new(MockTcpListenerFactory { listener }); + let mut tunnel_listener = TcpTunnelListener::new(local_addr, factory); + + tunnel_listener.listen().await.unwrap(); + assert!(tunnel_listener.accept().await.is_err()); + } + #[test] fn faketcp_upgrader_rejects_socket_without_host_transport_label() { let local_addr: SocketAddr = "192.0.2.1:10000".parse().unwrap(); diff --git a/easytier-core/src/gateway/port_forward.rs b/easytier-core/src/gateway/port_forward.rs index c373b36f..3b006ae9 100644 --- a/easytier-core/src/gateway/port_forward.rs +++ b/easytier-core/src/gateway/port_forward.rs @@ -41,6 +41,7 @@ use crate::{ }; const MAX_UDP_PAYLOAD_SIZE: usize = 65_507; +const TCP_PORT_FORWARD_REBIND_DELAY: Duration = Duration::from_millis(100); // A remote flow owns roughly 320 KiB of smoltcp and response buffers. // Keep the per-instance worst case near 80 MiB instead of allowing a // source-port flood to grow the WASM heap without bound. @@ -193,13 +194,17 @@ where } self.cancel_tokens.retain(|current, _| { - cfgs.iter().any(|next| { + let keep = cfgs.iter().any(|next| { if next.dst_addr.ip().is_unspecified() { current.bind_addr == next.bind_addr && current.proto == next.proto } else { current == next } - }) + }); + if !keep { + tracing::info!(?current, "port-forward removed by runtime config reload"); + } + keep }); self.udp_clients .retain(|key, _| self.cancel_tokens.contains_key(&key.forward)); @@ -238,7 +243,9 @@ where .bind .clone() .with_context(self.socket_context.clone()); - let listener = self.host.bind_tcp(options.with_bind(bind)).await?; + let options = options.with_bind(bind); + let listener = self.host.bind_tcp(options.clone()).await?; + tracing::info!(?bind_addr, ?dst_addr, "TCP port-forward listener bound"); let cancel = CancellationToken::new(); self.cancel_tokens .insert(cfg.clone(), cancel.clone().drop_guard()); @@ -249,21 +256,28 @@ where Arc::downgrade(&connections), "TCP port-forward connections", )); + let host = self.host.clone(); self.tasks.lock().unwrap().spawn(async move { + let mut listener = Some(listener); loop { - let (incoming, source_addr) = select! { - biased; - _ = cancel.cancelled() => break, - result = listener.accept() => match result { - Ok(accepted) => accepted, - Err(error) => { - tracing::error!(?error, ?bind_addr, "port-forward accept failed"); - continue; - } - }, + let Some((incoming, source_addr)) = + accept_tcp_port_forward(&host, &options, &mut listener, &cancel).await + else { + break; }; + tracing::info!( + ?bind_addr, + ?source_addr, + ?dst_addr, + "port-forward accepted local connection" + ); let data_plane = data_plane.clone(); connections.lock().unwrap().spawn(async move { + tracing::info!( + ?source_addr, + ?dst_addr, + "port-forward data-plane connect started" + ); let options = DataPlaneTcpConnectOptions::gateway( Duration::from_secs(10), TcpSocketPurpose::PortForward, @@ -276,9 +290,15 @@ where return; } }; + tracing::info!(?source_addr, ?dst_addr, "port-forward data-plane connected"); copy_tcp(incoming, outgoing, dst_addr).await; }); } + tracing::info!( + ?bind_addr, + ?dst_addr, + "TCP port-forward listener task stopped" + ); }); Ok(()) } @@ -430,6 +450,10 @@ where async fn stop_inner(&self) { self.started.store(false, Ordering::Release); + tracing::info!( + forward_count = self.cancel_tokens.len(), + "port-forward adapter stopping" + ); self.cancel_tokens.clear(); self.udp_response_tasks.clear(); self.udp_clients.clear(); @@ -447,6 +471,69 @@ where } } +async fn accept_tcp_port_forward( + host: &Arc, + options: &TcpListenOptions, + listener: &mut Option>, + cancel: &CancellationToken, +) -> Option<(::Socket, SocketAddr)> +where + H: VirtualTcpListenerFactory, +{ + loop { + let current = listener + .take() + .expect("TCP port-forward listener must be bound before accepting"); + let accepted = select! { + biased; + _ = cancel.cancelled() => { + tracing::info!(?options, "port-forward listener cancelled while accepting"); + return None; + }, + result = current.accept() => result, + }; + match accepted { + Ok(accepted) => { + listener.replace(current); + return Some(accepted); + } + Err(error) => { + tracing::error!(?error, ?options, "port-forward accept failed; rebinding"); + } + } + drop(current); + + loop { + select! { + biased; + _ = cancel.cancelled() => { + tracing::info!(?options, "port-forward listener cancelled during rebind delay"); + return None; + }, + _ = crate::foundation::time::sleep(TCP_PORT_FORWARD_REBIND_DELAY) => {} + } + let rebound = select! { + biased; + _ = cancel.cancelled() => { + tracing::info!(?options, "port-forward listener cancelled while rebinding"); + return None; + }, + result = host.bind_tcp(options.clone()) => result, + }; + match rebound { + Ok(rebound) => { + tracing::info!(?options, "port-forward listener rebound"); + listener.replace(rebound); + break; + } + Err(error) => { + tracing::error!(?error, ?options, "port-forward listener rebind failed"); + } + } + } + } +} + async fn reserve_udp_client_slot( cancel: &CancellationToken, admission: &Arc>, @@ -536,8 +623,105 @@ where #[cfg(test)] mod tests { + use std::{io, sync::atomic::AtomicUsize}; + use super::*; - use crate::host::testkit::{TestHost, TestUdpSocket}; + use crate::host::testkit::{TestHost, TestTcpSocket, TestUdpSocket}; + + struct RecoveringTcpListener { + accept_error: bool, + active: Arc, + address: SocketAddr, + } + + impl Drop for RecoveringTcpListener { + fn drop(&mut self) { + self.active.fetch_sub(1, Ordering::Relaxed); + } + } + + #[async_trait::async_trait] + impl VirtualTcpListener for RecoveringTcpListener { + type Socket = TestTcpSocket; + + fn local_addr(&self) -> io::Result { + Ok(self.address) + } + + async fn accept(&self) -> io::Result<(Self::Socket, SocketAddr)> { + if self.accept_error { + return Err(io::Error::other("listener is no longer usable")); + } + let (socket, _) = tokio::io::duplex(64); + Ok((TestTcpSocket(socket), "127.0.0.1:40000".parse().unwrap())) + } + } + + #[derive(Default)] + struct RecoveringTcpHost { + binds: AtomicUsize, + active: Arc, + } + + #[async_trait::async_trait] + impl VirtualTcpListenerFactory for RecoveringTcpHost { + type Listener = RecoveringTcpListener; + + async fn bind_tcp(&self, options: TcpListenOptions) -> anyhow::Result> { + let bind_index = self.binds.fetch_add(1, Ordering::Relaxed); + if bind_index > 0 && self.active.load(Ordering::Relaxed) != 0 { + anyhow::bail!("previous listener is still active"); + } + self.active.fetch_add(1, Ordering::Relaxed); + Ok(Arc::new(RecoveringTcpListener { + accept_error: bind_index == 0, + active: self.active.clone(), + address: options.bind.local_addr.unwrap(), + })) + } + } + + #[tokio::test] + async fn tcp_listener_rebinds_after_accept_error() { + let host = Arc::new(RecoveringTcpHost::default()); + let options = TcpListenOptions::port_forward("127.0.0.1:5202".parse().unwrap()); + let mut listener = Some(host.bind_tcp(options.clone()).await.unwrap()); + + let accepted = tokio::time::timeout( + Duration::from_secs(1), + accept_tcp_port_forward(&host, &options, &mut listener, &CancellationToken::new()), + ) + .await + .unwrap(); + + assert!(accepted.is_some()); + assert_eq!(host.binds.load(Ordering::Relaxed), 2); + assert_eq!(host.active.load(Ordering::Relaxed), 1); + } + + #[tokio::test] + async fn tcp_listener_rebind_stops_when_cancelled() { + let host = Arc::new(RecoveringTcpHost::default()); + let options = TcpListenOptions::port_forward("127.0.0.1:5202".parse().unwrap()); + let mut listener = Some(host.bind_tcp(options.clone()).await.unwrap()); + let cancel = CancellationToken::new(); + let cancel_after_error = cancel.clone(); + tokio::spawn(async move { + tokio::time::sleep(Duration::from_millis(10)).await; + cancel_after_error.cancel(); + }); + + let accepted = tokio::time::timeout( + Duration::from_secs(1), + accept_tcp_port_forward(&host, &options, &mut listener, &cancel), + ) + .await + .unwrap(); + + assert!(accepted.is_none()); + assert_eq!(host.binds.load(Ordering::Relaxed), 1); + assert_eq!(host.active.load(Ordering::Relaxed), 0); + } fn udp_client_key(port: u16) -> UdpClientKey { UdpClientKey { diff --git a/easytier-core/src/management/full/config_patch.rs b/easytier-core/src/management/full/config_patch.rs index b86e9459..f7252f93 100644 --- a/easytier-core/src/management/full/config_patch.rs +++ b/easytier-core/src/management/full/config_patch.rs @@ -96,6 +96,11 @@ where flags.disable_relay_data = disable_relay_data; candidate.set_flags(flags); } + if let Some(prefer_peer_relay) = patch.prefer_peer_relay { + let mut flags = candidate.get_flags(); + flags.prefer_peer_relay = prefer_peer_relay; + candidate.set_flags(flags); + } if let Some(enabled) = patch.ipv6_public_addr_provider { candidate.set_ipv6_public_addr_provider(enabled); provider_config_changed = true; diff --git a/easytier-core/src/peers/route/peer_ospf_route.rs b/easytier-core/src/peers/route/peer_ospf_route.rs index 253394d0..e64afcbd 100644 --- a/easytier-core/src/peers/route/peer_ospf_route.rs +++ b/easytier-core/src/peers/route/peer_ospf_route.rs @@ -881,11 +881,12 @@ impl Default for RouteConnInfo { } } -#[derive(Debug, Default)] +#[derive(Debug, Clone, Default)] struct InterfacePeerSnapshot { generation: u64, peers: BTreeSet, identity_types: BTreeMap>, + public_keys: BTreeMap>>, } // constructed with all infos synced from all peers. @@ -904,7 +905,7 @@ struct SyncedRouteInfo { // Aggregated trusted credential pubkeys from all admin nodes // Maps pubkey bytes -> TrustedCredentialPubkey - trusted_credential_pubkeys: DashMap, TrustedCredentialPubkey>, + trusted_credential_pubkeys: RwLock, TrustedCredentialPubkey>>, // Tracks the currently accepted peer for non-reusable credentials. // Maps credential pubkey bytes -> peer_id. non_reusable_credential_owners: DashMap, PeerId>, @@ -1060,17 +1061,9 @@ impl SyncedRouteInfo { &self, all_trusted: &HashMap, TrustedCredentialPubkey>, ) -> HashSet> { - let prev_trusted = self - .trusted_credential_pubkeys - .iter() - .map(|entry| entry.key().clone()) - .collect(); - - self.trusted_credential_pubkeys.clear(); - for (pubkey, credential) in all_trusted { - self.trusted_credential_pubkeys - .insert(pubkey.clone(), credential.clone()); - } + let mut trusted = self.trusted_credential_pubkeys.write(); + let prev_trusted = trusted.keys().cloned().collect(); + *trusted = all_trusted.clone(); prev_trusted } @@ -1881,8 +1874,9 @@ impl SyncedRouteInfo { return None; } self.trusted_credential_pubkeys + .read() .get(peer_pubkey) - .map(|r| r.value().clone()) + .cloned() } } @@ -2374,6 +2368,7 @@ struct PeerRouteServiceImpl { cached_interface_peer_snapshot: std::sync::Mutex>, interface_peers_generation: AtomicU64, applied_interface_peers_generation: AtomicU64, + applied_interface_peers: std::sync::Mutex>, last_update_my_foreign_network: AtomicCell>, @@ -2432,7 +2427,7 @@ impl PeerRouteServiceImpl { group_trust_map: DashMap::new(), group_trust_map_cache: DashMap::new(), group_trust_update_lock: parking_lot::Mutex::new(()), - trusted_credential_pubkeys: DashMap::new(), + trusted_credential_pubkeys: RwLock::new(HashMap::new()), non_reusable_credential_owners: DashMap::new(), suppressed_non_reusable_credential_peers: DashMap::new(), version: AtomicVersion::new(), @@ -2446,6 +2441,7 @@ impl PeerRouteServiceImpl { )), interface_peers_generation: AtomicU64::new(1), applied_interface_peers_generation: AtomicU64::new(0), + applied_interface_peers: std::sync::Mutex::new(BTreeSet::new()), last_update_my_foreign_network: AtomicCell::new(None), @@ -2462,6 +2458,10 @@ impl PeerRouteServiceImpl { .unwrap_or(false) } + fn peer_relay_projection_enabled(&self) -> bool { + self.context.flags().prefer_peer_relay && !self.is_credential_node() + } + fn set_public_ipv6_service(&self, service: Weak) { *self.public_ipv6_service.lock().unwrap() = service; } @@ -2532,14 +2532,19 @@ impl PeerRouteServiceImpl { let peers: BTreeSet<_> = interface.list_peers().await.into_iter().collect(); let mut identity_types = BTreeMap::new(); + let mut public_keys = BTreeMap::new(); for peer_id in peers.iter().copied() { identity_types.insert(peer_id, interface.get_peer_identity_type(peer_id).await); + if self.peer_relay_projection_enabled() { + public_keys.insert(peer_id, interface.get_peer_public_key(peer_id).await); + } } InterfacePeerSnapshot { generation: 0, peers, identity_types, + public_keys, } } @@ -2564,11 +2569,6 @@ impl PeerRouteServiceImpl { } } - async fn list_peers_from_interface_snapshot(&self) -> (u64, BTreeSet) { - let snapshot = self.interface_peer_snapshot().await; - (snapshot.generation, snapshot.peers.clone()) - } - async fn get_peer_identity_type_from_interface( &self, peer_id: PeerId, @@ -2588,6 +2588,11 @@ impl PeerRouteServiceImpl { } async fn get_peer_public_key_from_interface(&self, peer_id: PeerId) -> Option> { + let snapshot = self.interface_peer_snapshot().await; + if let Some(public_key) = snapshot.public_keys.get(&peer_id) { + return public_key.clone(); + } + self.interface .lock() .await @@ -2597,6 +2602,105 @@ impl PeerRouteServiceImpl { .await } + fn derive_advertised_connected_peers( + &self, + snapshot: &InterfacePeerSnapshot, + ) -> BTreeSet { + if !self.peer_relay_projection_enabled() { + return snapshot.peers.clone(); + } + + let is_credential_peer = |peer_id: PeerId| { + matches!( + snapshot.identity_types.get(&peer_id), + Some(Some(PeerIdentityType::Credential)) + ) + }; + + let eligible_relays: BTreeSet<_> = { + let peer_infos = self.synced_route_info.peer_infos.read(); + snapshot + .peers + .iter() + .copied() + .filter(|peer_id| is_credential_peer(*peer_id)) + .filter(|peer_id| { + !self + .synced_route_info + .suppressed_non_reusable_credential_peers + .contains_key(peer_id) + }) + .filter(|peer_id| { + let Some(Some(public_key)) = snapshot.public_keys.get(peer_id) else { + return false; + }; + if peer_infos + .get(peer_id) + .and_then(|info| info.feature_flag) + .is_some_and(|flags| flags.avoid_relay_data) + { + return false; + } + + self.synced_route_info + .get_credential_info_by_pubkey(public_key) + .is_some_and(|credential| credential.allow_relay) + }) + .collect() + }; + + if eligible_relays.is_empty() { + return snapshot.peers.clone(); + } + + let conn_map = self.synced_route_info.conn_map.read(); + let mut covered_targets = BTreeSet::new(); + for relay_peer_id in &eligible_relays { + let Some(relay_conn_info) = conn_map.get(relay_peer_id) else { + continue; + }; + for target_peer_id in &relay_conn_info.connected_peers { + if snapshot.peers.contains(target_peer_id) + && is_credential_peer(*target_peer_id) + && !eligible_relays.contains(target_peer_id) + && !self + .synced_route_info + .suppressed_non_reusable_credential_peers + .contains_key(target_peer_id) + { + covered_targets.insert(*target_peer_id); + } + } + } + + snapshot + .peers + .difference(&covered_targets) + .copied() + .collect() + } + + fn reconcile_my_conn_info( + &self, + snapshot: &InterfacePeerSnapshot, + interface_snapshot_changed: bool, + ) -> bool { + let advertised_peers = self.derive_advertised_connected_peers(snapshot); + if self + .synced_route_info + .update_my_conn_info(self.my_peer_id, advertised_peers) + { + return true; + } + + if interface_snapshot_changed && self.peer_relay_projection_enabled() { + self.synced_route_info.version.inc(); + return true; + } + + false + } + fn update_my_peer_info(&self) -> bool { self.synced_route_info.update_my_peer_info( self.my_peer_id, @@ -2620,17 +2724,28 @@ impl PeerRouteServiceImpl { .as_ref() .map(|x| x.need_periodic_requery_peers()) .unwrap_or(false); - if !need_periodic_requery { - return false; + let snapshot = self.cached_interface_peer_snapshot.lock().unwrap().clone(); + let need_peer_relay_metadata = self.peer_relay_projection_enabled() + && snapshot.public_keys.len() != snapshot.peers.len(); + if !need_periodic_requery && !need_peer_relay_metadata { + return self.reconcile_my_conn_info(&snapshot, false); } self.mark_interface_peers_dirty(); } - let (generation, connected_peers) = self.list_peers_from_interface_snapshot().await; - let updated = self - .synced_route_info - .update_my_conn_info(self.my_peer_id, connected_peers); + let snapshot = self.interface_peer_snapshot().await; + let generation = snapshot.generation; + let interface_snapshot_changed = { + let mut applied_peers = self.applied_interface_peers.lock().unwrap(); + if *applied_peers == snapshot.peers { + false + } else { + *applied_peers = snapshot.peers.clone(); + true + } + }; + let updated = self.reconcile_my_conn_info(&snapshot, interface_snapshot_changed); self.applied_interface_peers_generation .store(generation, Ordering::Release); updated @@ -2663,6 +2778,28 @@ impl PeerRouteServiceImpl { .update_my_foreign_network(self.my_peer_id, foreign_networks) } + fn local_route_snapshot(&self) -> OspfRouteSnapshot { + let mut snapshot = self.synced_route_info.route_snapshot(); + if !self.peer_relay_projection_enabled() { + return snapshot; + } + + let local_connected_peers = self + .cached_interface_peer_snapshot + .lock() + .unwrap() + .peers + .clone(); + if let Some(self_row) = snapshot + .conn_map + .iter_mut() + .find(|row| row.peer_id == self.my_peer_id) + { + self_row.connected_peers = local_connected_peers; + } + snapshot + } + fn update_route_table(&self) { self.cost_calculator .write() @@ -2672,7 +2809,7 @@ impl PeerRouteServiceImpl { .begin_update(); let calc_locked = self.cost_calculator.read().unwrap(); - let route_snapshot = self.synced_route_info.route_snapshot(); + let route_snapshot = self.local_route_snapshot(); self.route_table.build_from_snapshot( self.my_peer_id, @@ -4538,11 +4675,34 @@ mod tests { get_peer_identity_type_calls: Arc, } + struct PeriodicRequeryInterface { + peers: Vec, + list_peers_calls: Arc, + } + struct BlockingInterface { entered: Arc, release: Arc, } + #[derive(Default)] + struct TogglePeerRelayContext { + enabled: AtomicBool, + } + + impl PeerContext for TogglePeerRelayContext { + fn network_identity(&self) -> CoreNetworkIdentity { + CoreNetworkIdentity::default() + } + + fn flags(&self) -> crate::proto::common::FlagsInConfig { + crate::proto::common::FlagsInConfig { + prefer_peer_relay: self.enabled.load(Ordering::Relaxed), + ..Default::default() + } + } + } + #[async_trait::async_trait] impl RouteInterface for BlockingInterface { async fn list_peers(&self) -> Vec { @@ -4623,15 +4783,127 @@ mod tests { .flatten() } + async fn get_peer_public_key(&self, peer_id: PeerId) -> Option> { + Some(vec![peer_id as u8; 32]) + } + fn my_peer_id(&self) -> PeerId { self.my_peer_id } } + #[async_trait::async_trait] + impl RouteInterface for PeriodicRequeryInterface { + async fn list_peers(&self) -> Vec { + self.list_peers_calls.fetch_add(1, Ordering::Relaxed); + self.peers.clone() + } + + fn my_peer_id(&self) -> PeerId { + 1 + } + + fn need_periodic_requery_peers(&self) -> bool { + true + } + + async fn get_peer_identity_type(&self, _peer_id: PeerId) -> Option { + Some(PeerIdentityType::Admin) + } + + async fn get_peer_public_key(&self, peer_id: PeerId) -> Option> { + Some(vec![peer_id as u8; 32]) + } + } + fn test_service_impl(my_peer_id: PeerId) -> PeerRouteServiceImpl { PeerRouteServiceImpl::new(my_peer_id, Arc::new(NoopPeerContext::default())) } + fn test_peer_relay_service_impl(my_peer_id: PeerId) -> PeerRouteServiceImpl { + let flags = crate::proto::common::FlagsInConfig { + prefer_peer_relay: true, + ..Default::default() + }; + PeerRouteServiceImpl::new( + my_peer_id, + Arc::new(NoopPeerContext::default().with_flags(flags)), + ) + } + + fn interface_peer_snapshot( + peers: impl IntoIterator>)>, + ) -> InterfacePeerSnapshot { + let peers: Vec<_> = peers.into_iter().collect(); + InterfacePeerSnapshot { + generation: 1, + peers: peers.iter().map(|(peer_id, _, _)| *peer_id).collect(), + identity_types: peers + .iter() + .map(|(peer_id, identity, _)| (*peer_id, Some(*identity))) + .collect(), + public_keys: peers + .into_iter() + .map(|(peer_id, _, public_key)| (peer_id, public_key)) + .collect(), + } + } + + fn install_peer_info( + service_impl: &PeerRouteServiceImpl, + peer_id: PeerId, + advertised_public_key: Vec, + avoid_relay_data: bool, + ) { + let feature_flag = crate::proto::common::PeerFeatureFlag { + avoid_relay_data, + ..Default::default() + }; + service_impl.synced_route_info.peer_infos.write().insert( + peer_id, + RoutePeerInfo { + peer_id, + version: 1, + feature_flag: Some(feature_flag), + noise_static_pubkey: advertised_public_key, + ..Default::default() + }, + ); + } + + fn install_credential_grant( + service_impl: &PeerRouteServiceImpl, + public_key: Vec, + allow_relay: bool, + ) { + service_impl + .synced_route_info + .trusted_credential_pubkeys + .write() + .insert( + public_key, + TrustedCredentialPubkey { + allow_relay, + ..Default::default() + }, + ); + } + + fn install_conn_row( + service_impl: &PeerRouteServiceImpl, + peer_id: PeerId, + connected_peers: impl IntoIterator, + ) { + service_impl.synced_route_info.conn_map.write().insert( + peer_id, + RouteConnInfo { + connected_peers: connected_peers.into_iter().collect(), + version: 1.into(), + last_update: SystemTime::now(), + }, + ); + } + async fn test_route_with_admin_peer( context: ArcPeerContext, ) -> (Arc, Arc) { @@ -4676,6 +4948,307 @@ mod tests { } } + #[test] + fn trusted_credential_replacement_is_atomic_for_readers() { + const CREDENTIAL_COUNT: u32 = 16_384; + const REPLACEMENT_COUNT: usize = 16; + + let service_impl = test_service_impl(1); + let credentials: HashMap<_, _> = (0..CREDENTIAL_COUNT) + .map(|id| { + ( + id.to_le_bytes().to_vec(), + TrustedCredentialPubkey { + allow_relay: true, + ..Default::default() + }, + ) + }) + .collect(); + service_impl + .synced_route_info + .replace_trusted_credential_pubkeys(&credentials); + + let keys: Vec<_> = credentials.keys().cloned().collect(); + let start = std::sync::Barrier::new(2); + + std::thread::scope(|scope| { + let writer = scope.spawn(|| { + start.wait(); + for _ in 0..REPLACEMENT_COUNT { + service_impl + .synced_route_info + .replace_trusted_credential_pubkeys(&credentials); + } + }); + + start.wait(); + let mut snapshots_read = 0; + while !writer.is_finished() { + assert!(keys.iter().all(|key| { + service_impl + .synced_route_info + .get_credential_info_by_pubkey(key) + .is_some() + })); + snapshots_read += 1; + } + writer.join().unwrap(); + assert!(snapshots_read > 0); + }); + } + + #[test] + fn peer_relay_projection_suppresses_only_covered_credential_leaves() { + let service_impl = test_peer_relay_service_impl(1); + let relay_key = vec![2; 32]; + let leaf_a_key = vec![3; 32]; + let leaf_b_key = vec![4; 32]; + let snapshot = interface_peer_snapshot([ + (2, PeerIdentityType::Credential, Some(relay_key.clone())), + (3, PeerIdentityType::Credential, Some(leaf_a_key.clone())), + (4, PeerIdentityType::Credential, Some(leaf_b_key.clone())), + (5, PeerIdentityType::Admin, None), + ]); + + for peer_id in 1..=5 { + install_peer_info(&service_impl, peer_id, vec![peer_id as u8; 32], false); + } + install_credential_grant(&service_impl, relay_key, true); + install_credential_grant(&service_impl, leaf_a_key, false); + install_credential_grant(&service_impl, leaf_b_key, false); + install_conn_row(&service_impl, 2, [3, 4]); + + assert!(service_impl.reconcile_my_conn_info(&snapshot, false)); + assert_eq!( + service_impl + .synced_route_info + .get_connected_peers::>(1), + Some(BTreeSet::from([2, 5])) + ); + + *service_impl.cached_interface_peer_snapshot.lock().unwrap() = Arc::new(snapshot.clone()); + let local_snapshot = service_impl.local_route_snapshot(); + assert_eq!( + local_snapshot + .conn_map + .iter() + .find(|row| row.peer_id == 1) + .unwrap() + .connected_peers, + snapshot.peers + ); + + service_impl.update_route_table(); + assert_eq!( + service_impl + .route_table + .get_next_hop(3) + .unwrap() + .next_hop_peer_id, + 3 + ); + } + + #[test] + fn peer_relay_projection_uses_authenticated_public_key() { + let service_impl = test_peer_relay_service_impl(1); + let authenticated_key = vec![2; 32]; + let forged_relay_key = vec![9; 32]; + let snapshot = interface_peer_snapshot([ + ( + 2, + PeerIdentityType::Credential, + Some(authenticated_key.clone()), + ), + (4, PeerIdentityType::Credential, Some(vec![4; 32])), + ]); + install_peer_info(&service_impl, 2, forged_relay_key.clone(), false); + install_credential_grant(&service_impl, authenticated_key, false); + install_credential_grant(&service_impl, forged_relay_key, true); + install_conn_row(&service_impl, 2, [4]); + + assert_eq!( + service_impl.derive_advertised_connected_peers(&snapshot), + snapshot.peers + ); + } + + #[test] + fn peer_relay_projection_is_disabled_by_default() { + let service_impl = test_service_impl(1); + let snapshot = interface_peer_snapshot([ + (2, PeerIdentityType::Credential, Some(vec![2; 32])), + (4, PeerIdentityType::Credential, Some(vec![4; 32])), + ]); + install_peer_info(&service_impl, 2, vec![2; 32], false); + install_credential_grant(&service_impl, vec![2; 32], true); + install_conn_row(&service_impl, 2, [4]); + + assert_eq!( + service_impl.derive_advertised_connected_peers(&snapshot), + snapshot.peers + ); + } + + #[test] + fn peer_relay_projection_restores_edges_after_last_coverage_disappears() { + let service_impl = test_peer_relay_service_impl(1); + let snapshot = interface_peer_snapshot([ + (2, PeerIdentityType::Credential, Some(vec![2; 32])), + (3, PeerIdentityType::Credential, Some(vec![3; 32])), + (4, PeerIdentityType::Credential, Some(vec![4; 32])), + (5, PeerIdentityType::Credential, Some(vec![5; 32])), + ]); + for relay_peer_id in [2, 3] { + install_peer_info( + &service_impl, + relay_peer_id, + vec![relay_peer_id as u8; 32], + false, + ); + install_credential_grant(&service_impl, vec![relay_peer_id as u8; 32], true); + } + install_conn_row(&service_impl, 2, [4]); + install_conn_row(&service_impl, 3, [4, 5]); + + assert!(service_impl.reconcile_my_conn_info(&snapshot, false)); + let first_version = service_impl + .synced_route_info + .conn_map + .read() + .get(&1) + .unwrap() + .version + .get(); + assert_eq!( + service_impl + .synced_route_info + .get_connected_peers::>(1), + Some(BTreeSet::from([2, 3])) + ); + + service_impl.synced_route_info.conn_map.write().remove(&2); + assert!(!service_impl.reconcile_my_conn_info(&snapshot, false)); + + service_impl.synced_route_info.conn_map.write().remove(&3); + assert!(service_impl.reconcile_my_conn_info(&snapshot, false)); + let self_row = service_impl.synced_route_info.conn_map.read(); + let self_row = self_row.get(&1).unwrap(); + assert_eq!(self_row.connected_peers, snapshot.peers); + assert_eq!(self_row.version.get(), first_version + 1); + } + + #[test] + fn peer_relay_projection_ignores_ineligible_relays() { + let service_impl = test_peer_relay_service_impl(1); + let snapshot = interface_peer_snapshot([ + (2, PeerIdentityType::Credential, Some(vec![2; 32])), + (3, PeerIdentityType::Credential, Some(vec![3; 32])), + (4, PeerIdentityType::Admin, Some(vec![4; 32])), + (5, PeerIdentityType::Credential, Some(vec![5; 32])), + (6, PeerIdentityType::Credential, Some(vec![6; 32])), + ]); + install_peer_info(&service_impl, 2, vec![2; 32], false); + install_peer_info(&service_impl, 3, vec![3; 32], true); + install_peer_info(&service_impl, 4, vec![4; 32], false); + install_peer_info(&service_impl, 5, vec![5; 32], false); + for peer_id in [2, 3, 4, 5] { + install_credential_grant(&service_impl, vec![peer_id as u8; 32], peer_id != 2); + install_conn_row(&service_impl, peer_id, [6]); + } + service_impl + .synced_route_info + .suppressed_non_reusable_credential_peers + .insert(5, ()); + + assert_eq!( + service_impl.derive_advertised_connected_peers(&snapshot), + snapshot.peers + ); + } + + #[test] + fn peer_relay_projection_refreshes_local_topology_when_advertisement_is_unchanged() { + let service_impl = test_peer_relay_service_impl(1); + install_peer_info(&service_impl, 2, vec![2; 32], false); + install_credential_grant(&service_impl, vec![2; 32], true); + install_conn_row(&service_impl, 2, [3, 4]); + let first = interface_peer_snapshot([ + (2, PeerIdentityType::Credential, Some(vec![2; 32])), + (3, PeerIdentityType::Credential, Some(vec![3; 32])), + ]); + assert!(service_impl.reconcile_my_conn_info(&first, false)); + let self_version = service_impl + .synced_route_info + .conn_map + .read() + .get(&1) + .unwrap() + .version + .get(); + let route_version = service_impl.synced_route_info.version.get(); + + let second = interface_peer_snapshot([ + (2, PeerIdentityType::Credential, Some(vec![2; 32])), + (4, PeerIdentityType::Credential, Some(vec![4; 32])), + ]); + assert!(service_impl.reconcile_my_conn_info(&second, true)); + assert_eq!( + service_impl + .synced_route_info + .conn_map + .read() + .get(&1) + .unwrap() + .version + .get(), + self_version + ); + assert_eq!( + service_impl.synced_route_info.version.get(), + route_version + 1 + ); + } + + #[tokio::test] + async fn peer_relay_projection_reconciles_forwarded_row_without_interface_change() { + let service_impl = test_peer_relay_service_impl(1); + let peers = Arc::new(Mutex::new(vec![2, 4])); + let list_peers_calls = Arc::new(AtomicU32::new(0)); + *service_impl.interface.lock().await = Some(Box::new(CountingInterface { + my_peer_id: 1, + peers, + peer_identity_types: Arc::new(Mutex::new(HashMap::from([ + (2, Some(PeerIdentityType::Credential)), + (4, Some(PeerIdentityType::Credential)), + ]))), + list_peers_calls: list_peers_calls.clone(), + get_peer_identity_type_calls: Arc::new(AtomicU32::new(0)), + })); + install_peer_info(&service_impl, 2, vec![2; 32], false); + install_credential_grant(&service_impl, vec![2; 32], true); + + assert!(service_impl.update_my_conn_info().await); + assert_eq!(list_peers_calls.load(Ordering::Relaxed), 1); + assert_eq!( + service_impl + .synced_route_info + .get_connected_peers::>(1), + Some(BTreeSet::from([2, 4])) + ); + + install_conn_row(&service_impl, 2, [4]); + assert!(service_impl.update_my_conn_info().await); + assert_eq!(list_peers_calls.load(Ordering::Relaxed), 1); + assert_eq!( + service_impl + .synced_route_info + .get_connected_peers::>(1), + Some(BTreeSet::from([2])) + ); + } + #[tokio::test] async fn interface_peer_cache_refreshes_only_when_marked_dirty() { let service_impl = test_service_impl(1); @@ -4741,6 +5314,64 @@ mod tests { assert_eq!(get_peer_identity_type_calls.load(Ordering::Relaxed), 4); } + #[tokio::test] + async fn periodic_requery_without_peer_change_keeps_route_version_stable() { + let service_impl = test_peer_relay_service_impl(1); + let list_peers_calls = Arc::new(AtomicU32::new(0)); + *service_impl.interface.lock().await = Some(Box::new(PeriodicRequeryInterface { + peers: vec![2], + list_peers_calls: list_peers_calls.clone(), + })); + + assert!(service_impl.update_my_conn_info().await); + let route_version = service_impl.synced_route_info.version.get(); + + assert!(!service_impl.update_my_conn_info().await); + assert_eq!(list_peers_calls.load(Ordering::Relaxed), 2); + assert_eq!(service_impl.synced_route_info.version.get(), route_version); + } + + #[tokio::test] + async fn enabling_peer_relay_refreshes_authenticated_interface_metadata() { + let context = Arc::new(TogglePeerRelayContext::default()); + let service_impl = PeerRouteServiceImpl::new(1, context.clone()); + let peer_identity_types = Arc::new(Mutex::new(HashMap::from([ + (2, Some(PeerIdentityType::Credential)), + (3, Some(PeerIdentityType::Credential)), + ]))); + let list_peers_calls = Arc::new(AtomicU32::new(0)); + *service_impl.interface.lock().await = Some(Box::new(CountingInterface { + my_peer_id: 1, + peers: Arc::new(Mutex::new(vec![2, 3])), + peer_identity_types, + list_peers_calls: list_peers_calls.clone(), + get_peer_identity_type_calls: Arc::new(AtomicU32::new(0)), + })); + install_peer_info(&service_impl, 2, vec![2; 32], false); + install_credential_grant(&service_impl, vec![2; 32], true); + install_conn_row(&service_impl, 2, [3]); + + assert!(service_impl.update_my_conn_info().await); + assert_eq!(list_peers_calls.load(Ordering::Relaxed), 1); + assert_eq!( + service_impl + .synced_route_info + .get_connected_peers::>(1), + Some(BTreeSet::from([2, 3])) + ); + + context.enabled.store(true, Ordering::Relaxed); + + assert!(service_impl.update_my_conn_info().await); + assert_eq!(list_peers_calls.load(Ordering::Relaxed), 2); + assert_eq!( + service_impl + .synced_route_info + .get_connected_peers::>(1), + Some(BTreeSet::from([2])) + ); + } + #[tokio::test] async fn get_peer_identity_type_reuses_snapshot_until_topology_changes() { let service_impl = test_service_impl(1); diff --git a/easytier-core/src/peers/test_support.rs b/easytier-core/src/peers/test_support.rs index c7f15ff2..16a4662e 100644 --- a/easytier-core/src/peers/test_support.rs +++ b/easytier-core/src/peers/test_support.rs @@ -76,6 +76,11 @@ impl NoopPeerContext { self.secure_mode = Some(secure_mode); self } + + pub(crate) fn with_flags(mut self, flags: FlagsInConfig) -> Self { + self.flags = flags; + self + } } impl Default for NoopPeerContext { diff --git a/easytier-core/src/socket/tcp.rs b/easytier-core/src/socket/tcp.rs index 57a5e3c9..526fb3df 100644 --- a/easytier-core/src/socket/tcp.rs +++ b/easytier-core/src/socket/tcp.rs @@ -361,7 +361,7 @@ where let listener = self.listener()?; match listener.accept().await { Ok((socket, _)) => return Ok(socket), - Err(error) if is_retryable_tcp_accept_error(&error) => { + Err(error) if is_retryable_tcp_io_error(&error) => { tracing::warn!(?error, "tcp accept failed with retryable error"); } Err(error) => { @@ -377,7 +377,7 @@ where } } -fn is_retryable_tcp_accept_error(error: &io::Error) -> bool { +pub(crate) fn is_retryable_tcp_io_error(error: &io::Error) -> bool { use io::ErrorKind::*; matches!( error.kind(), diff --git a/easytier-core/src/wasi/web_client.rs b/easytier-core/src/wasi/web_client.rs index e422e288..a4a5900e 100644 --- a/easytier-core/src/wasi/web_client.rs +++ b/easytier-core/src/wasi/web_client.rs @@ -126,6 +126,7 @@ fn hosted_network_config(config: &NetworkConfig) -> NetworkConfig { instance_recv_bps_limit: config.instance_recv_bps_limit, disable_upnp: config.disable_upnp, disable_relay_data: config.disable_relay_data, + prefer_peer_relay: config.prefer_peer_relay, enable_udp_broadcast_relay: config.enable_udp_broadcast_relay, managed_credentials: config.managed_credentials.clone(), peers, @@ -425,6 +426,7 @@ mod tests { }), enable_private_mode: Some(true), disable_relay_data: Some(true), + prefer_peer_relay: Some(true), proxy_cidrs: vec!["10.88.0.0/24".to_owned()], managed_credentials: vec![ManagedCredentialConfig { credential_id: "managed".to_owned(), @@ -455,6 +457,7 @@ mod tests { assert_eq!(hosted.secure_mode, original.secure_mode); assert_eq!(hosted.enable_private_mode, Some(true)); assert_eq!(hosted.disable_relay_data, Some(true)); + assert_eq!(hosted.prefer_peer_relay, Some(true)); assert_eq!(hosted.proxy_cidrs, original.proxy_cidrs); assert_eq!(hosted.managed_credentials, original.managed_credentials); assert_eq!(hosted.port_forwards, original.port_forwards); diff --git a/easytier-proto/proto/api_config.proto b/easytier-proto/proto/api_config.proto index 8493db28..ae299b7b 100644 --- a/easytier-proto/proto/api_config.proto +++ b/easytier-proto/proto/api_config.proto @@ -30,6 +30,7 @@ message InstanceConfigPatch { optional bool disable_relay_data = 14; repeated VpnPortalClientPatch vpn_portal_clients = 15; api.manage.ManagedCredentialSet managed_credentials = 16; + optional bool prefer_peer_relay = 17; } message VpnPortalClientPatch { diff --git a/easytier-proto/proto/api_manage.proto b/easytier-proto/proto/api_manage.proto index 398dc3e6..82e63ab7 100644 --- a/easytier-proto/proto/api_manage.proto +++ b/easytier-proto/proto/api_manage.proto @@ -105,6 +105,7 @@ message NetworkConfig { repeated NetworkPeerConfig peers = 68; optional VpnPortalConfig vpn_portal_config = 69; repeated ManagedCredentialConfig managed_credentials = 71; + optional bool prefer_peer_relay = 72; } message ManagedCredentialConfig { diff --git a/easytier-proto/proto/common.proto b/easytier-proto/proto/common.proto index 53b114a3..1ce83183 100644 --- a/easytier-proto/proto/common.proto +++ b/easytier-proto/proto/common.proto @@ -84,6 +84,10 @@ message FlagsInConfig { // applied via setsockopt. Requires CAP_NET_ADMIN; silently ignored on // non-Linux platforms. optional uint32 socket_mark = 43; + + // Prefer direct credential peers that already relay to a destination over + // advertising another direct edge to the same destination. + bool prefer_peer_relay = 44; } message RpcDescriptor { diff --git a/easytier-web/src/client_manager/managed_config.rs b/easytier-web/src/client_manager/managed_config.rs index e7be3822..7b3e1829 100644 --- a/easytier-web/src/client_manager/managed_config.rs +++ b/easytier-web/src/client_manager/managed_config.rs @@ -3,7 +3,6 @@ use std::{ sync::{Arc, Weak}, }; -use anyhow::Context as _; use dashmap::{DashMap, mapref::entry::Entry}; use easytier::{ common::config::ConfigSource, @@ -13,11 +12,13 @@ use easytier::{ }, }; use easytier_core::management::config_source_from_rpc; -use easytier_core::management::remote_client::{ - ListNetworkProps, PersistentConfig as _, Storage as _, -}; +use easytier_core::management::remote_client::{PersistentConfig as _, Storage as _}; use super::storage::Storage; +use crate::db::{ + ManagedConfigApplyResult, ManagedConfigExpectedRevision, ManagedConfigUpdate, + ManagedConfigUpsert, +}; #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub(super) enum PersistedConfigSource { @@ -31,7 +32,9 @@ pub(super) enum ExpectedConfigRevision<'a> { } #[derive(Debug, thiserror::Error)] -pub(super) enum ManagedConfigError { +pub(crate) enum ManagedConfigError { + #[error("invalid managed config update: {0}")] + Invalid(String), #[error( "managed config revision changed while reconciling: expected {expected:?}, current {current:?}" )] @@ -39,6 +42,16 @@ pub(super) enum ManagedConfigError { expected: Option, current: Option, }, + #[error("managed config instance {instance_id} is user-owned")] + OwnershipConflict { instance_id: uuid::Uuid }, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub(super) enum ManagedConfigApplyStatus { + Applied { + deleted_web_instance_ids: Vec, + }, + AlreadyApplied, } impl PersistedConfigSource { @@ -111,8 +124,12 @@ fn remove_unused_managed_config_reconcile_lock( }); } -pub(super) fn is_revision_conflict(error: &anyhow::Error) -> bool { - error.downcast_ref::().is_some() +#[cfg(test)] +fn is_revision_conflict(error: &anyhow::Error) -> bool { + matches!( + error.downcast_ref::(), + Some(ManagedConfigError::RevisionConflict { .. }) + ) } fn snake_to_lower_camel(key: &str) -> Option { @@ -188,103 +205,36 @@ fn normalize_network_config( Ok(serde_json::from_value::(network_config)?) } -struct ExistingConfigSources { - sources: HashMap, - web_ids: HashSet, -} - struct NormalizedWebConfigs { desired_ids: HashSet, - configs: HashMap, -} - -async fn ensure_expected_config_revision( - storage: &Storage, - user_id: i32, - machine_id: uuid::Uuid, - expected_config_revision: ExpectedConfigRevision<'_>, -) -> anyhow::Result<()> { - let ExpectedConfigRevision::Exact(expected) = expected_config_revision else { - return Ok(()); - }; - - let current = storage - .db() - .get_managed_config_revision((user_id, machine_id)) - .await - .map_err(|e| anyhow::anyhow!("failed to get managed config revision: {:?}", e))?; - if current.as_deref() != expected { - return Err(ManagedConfigError::RevisionConflict { - expected: expected.map(str::to_string), - current, - } - .into()); - } - - Ok(()) -} - -async fn load_existing_config_sources( - storage: &Storage, - user_id: i32, - machine_id: uuid::Uuid, -) -> anyhow::Result { - let existing_configs = storage - .db() - .list_network_configs((user_id, machine_id), ListNetworkProps::All) - .await - .map_err(|e| anyhow::anyhow!("failed to list existing network configs: {:?}", e))?; - let sources = existing_configs - .iter() - .filter_map(|cfg| { - uuid::Uuid::parse_str(&cfg.network_instance_id) - .ok() - .map(|inst_id| (inst_id, PersistedConfigSource::from_db(&cfg.source))) - }) - .collect::>(); - let web_ids = sources - .iter() - .filter_map(|(inst_id, source)| (*source == PersistedConfigSource::Web).then_some(*inst_id)) - .collect::>(); - - Ok(ExistingConfigSources { sources, web_ids }) + configs: Vec, } fn normalize_desired_web_configs( - user_id: i32, - machine_id: uuid::Uuid, desired_configs: Vec, - config_revision: Option<&str>, - existing_sources: &HashMap, ) -> anyhow::Result { let mut desired_ids = HashSet::with_capacity(desired_configs.len()); - let mut configs = HashMap::with_capacity(desired_configs.len()); + let mut configs = Vec::with_capacity(desired_configs.len()); for desired in desired_configs { - let inst_id = uuid::Uuid::parse_str(&desired.instance_id).with_context(|| { - format!( + let inst_id = uuid::Uuid::parse_str(&desired.instance_id).map_err(|_| { + ManagedConfigError::Invalid(format!( "invalid desired web config instance id: {}", desired.instance_id - ) + )) })?; - if let Some(PersistedConfigSource::User) = existing_sources.get(&inst_id) { - if config_revision.is_some() { - anyhow::bail!( - "cannot persist managed config revision because instance {} is user-owned", - inst_id - ); - } - tracing::warn!( - ?user_id, - ?machine_id, - instance_id = %inst_id, - "skip web config because a user-owned config already exists" - ); - continue; + if !desired_ids.insert(inst_id) { + return Err(ManagedConfigError::Invalid(format!( + "duplicate managed config instance id: {inst_id}" + )) + .into()); } - let config = normalize_network_config(desired.network_config, inst_id)?; - desired_ids.insert(inst_id); - configs.insert(inst_id, config); + let config = normalize_network_config(desired.network_config, inst_id) + .map_err(|error| ManagedConfigError::Invalid(error.to_string()))?; + configs.push(ManagedConfigUpsert { + instance_id: inst_id, + network_config: config, + }); } Ok(NormalizedWebConfigs { @@ -293,72 +243,21 @@ fn normalize_desired_web_configs( }) } -async fn upsert_web_configs( - storage: &Storage, - user_id: i32, - machine_id: uuid::Uuid, - configs: HashMap, -) -> anyhow::Result<()> { - for (inst_id, config) in configs { - let updated = storage - .db() - .insert_or_update_web_network_config((user_id, machine_id), inst_id, config) - .await - .map_err(|e| { - anyhow::anyhow!("failed to persist web network config {}: {:?}", inst_id, e) - })?; - if !updated { - anyhow::bail!( - "cannot persist managed config revision because instance {} is user-owned", - inst_id - ); +fn map_apply_result(result: ManagedConfigApplyResult) -> anyhow::Result { + match result { + ManagedConfigApplyResult::Applied { + deleted_web_instance_ids, + } => Ok(ManagedConfigApplyStatus::Applied { + deleted_web_instance_ids, + }), + ManagedConfigApplyResult::AlreadyApplied => Ok(ManagedConfigApplyStatus::AlreadyApplied), + ManagedConfigApplyResult::RevisionConflict { expected, current } => { + Err(ManagedConfigError::RevisionConflict { expected, current }.into()) + } + ManagedConfigApplyResult::OwnershipConflict { instance_id } => { + Err(ManagedConfigError::OwnershipConflict { instance_id }.into()) } } - - Ok(()) -} - -async fn delete_stale_web_configs( - storage: &Storage, - user_id: i32, - machine_id: uuid::Uuid, - existing_web_ids: &HashSet, - desired_ids: &HashSet, -) -> anyhow::Result<()> { - let stale_ids = existing_web_ids - .difference(desired_ids) - .copied() - .collect::>(); - if stale_ids.is_empty() { - return Ok(()); - } - - storage - .db() - .delete_web_network_configs((user_id, machine_id), &stale_ids) - .await - .map_err(|e| anyhow::anyhow!("failed to delete stale network configs: {:?}", e))?; - - Ok(()) -} - -async fn persist_config_revision( - storage: &Storage, - user_id: i32, - machine_id: uuid::Uuid, - config_revision: Option<&str>, -) -> anyhow::Result<()> { - let Some(config_revision) = config_revision else { - return Ok(()); - }; - - storage - .db() - .set_managed_config_revision((user_id, machine_id), config_revision) - .await - .map_err(|e| anyhow::anyhow!("failed to persist managed config revision: {:?}", e))?; - - Ok(()) } pub(super) async fn reconcile_web_source_configs( @@ -368,34 +267,111 @@ pub(super) async fn reconcile_web_source_configs( desired_configs: Vec, config_revision: Option<&str>, expected_config_revision: ExpectedConfigRevision<'_>, -) -> anyhow::Result<()> { +) -> anyhow::Result { + if config_revision.is_some_and(|revision| revision.trim().is_empty()) { + return Err( + ManagedConfigError::Invalid("config_revision must not be empty".to_string()).into(), + ); + } + let normalized = normalize_desired_web_configs(desired_configs)?; + let expected_revision = match expected_config_revision { + ExpectedConfigRevision::Any => ManagedConfigExpectedRevision::Any, + ExpectedConfigRevision::Exact(revision) => { + ManagedConfigExpectedRevision::Exact(revision.map(str::to_string)) + } + }; let key = (user_id, machine_id); let reconcile_lock = managed_config_reconcile_lock(key); let result = async { let _guard = reconcile_lock.lock().await; - ensure_expected_config_revision(storage, user_id, machine_id, expected_config_revision) - .await?; - let existing = load_existing_config_sources(storage, user_id, machine_id).await?; - let normalized = normalize_desired_web_configs( - user_id, - machine_id, - desired_configs, - config_revision, - &existing.sources, - )?; - upsert_web_configs(storage, user_id, machine_id, normalized.configs).await?; - delete_stale_web_configs( - storage, - user_id, - machine_id, - &existing.web_ids, - &normalized.desired_ids, - ) - .await?; - persist_config_revision(storage, user_id, machine_id, config_revision).await?; + let result = storage + .db() + .apply_managed_config_update( + (user_id, machine_id), + ManagedConfigUpdate::Full { + upserts: normalized.configs, + target_revision: config_revision.map(str::to_string), + expected_revision, + }, + ) + .await + .map_err(|error| anyhow::anyhow!("failed to apply managed config Full: {error}"))?; + map_apply_result(result) + } + .await; + remove_unused_managed_config_reconcile_lock(key, &reconcile_lock); + result +} - Ok(()) +pub(super) async fn patch_web_source_configs( + storage: &Storage, + user_id: i32, + machine_id: uuid::Uuid, + upserts: Vec, + delete_instance_ids: Vec, + config_revision: &str, + expected_config_revision: &str, +) -> anyhow::Result { + let config_revision = config_revision.trim(); + let expected_config_revision = expected_config_revision.trim(); + if config_revision.is_empty() || expected_config_revision.is_empty() { + return Err( + ManagedConfigError::Invalid("Patch revisions must not be empty".to_string()).into(), + ); + } + if config_revision == expected_config_revision { + return Err(ManagedConfigError::Invalid( + "Patch target revision must differ from expected revision".to_string(), + ) + .into()); + } + + let normalized = normalize_desired_web_configs(upserts)?; + let mut delete_ids = HashSet::with_capacity(delete_instance_ids.len()); + for instance_id in delete_instance_ids { + if !delete_ids.insert(instance_id) { + return Err(ManagedConfigError::Invalid(format!( + "duplicate managed config delete instance id: {instance_id}" + )) + .into()); + } + } + if let Some(instance_id) = delete_ids + .intersection(&normalized.desired_ids) + .next() + .copied() + { + return Err(ManagedConfigError::Invalid(format!( + "managed config instance {instance_id} cannot be upserted and deleted" + )) + .into()); + } + if normalized.configs.is_empty() && delete_ids.is_empty() { + return Err(ManagedConfigError::Invalid( + "Patch must contain an upsert or delete".to_string(), + ) + .into()); + } + + let key = (user_id, machine_id); + let reconcile_lock = managed_config_reconcile_lock(key); + let result = async { + let _guard = reconcile_lock.lock().await; + let result = storage + .db() + .apply_managed_config_update( + (user_id, machine_id), + ManagedConfigUpdate::Patch { + upserts: normalized.configs, + delete_instance_ids: delete_ids.into_iter().collect(), + target_revision: config_revision.to_string(), + expected_revision: expected_config_revision.to_string(), + }, + ) + .await + .map_err(|error| anyhow::anyhow!("failed to apply managed config Patch: {error}"))?; + map_apply_result(result) } .await; remove_unused_managed_config_reconcile_lock(key, &reconcile_lock); @@ -516,6 +492,19 @@ mod tests { use super::*; + fn managed_config( + instance_id: uuid::Uuid, + network_name: &str, + ) -> crate::webhook::ManagedNetworkConfig { + crate::webhook::ManagedNetworkConfig { + instance_id: instance_id.to_string(), + network_config: json!({ + "instance_id": instance_id.to_string(), + "network_name": network_name + }), + } + } + #[tokio::test] async fn reconcile_web_source_configs_upserts_and_deletes_exact_set() { let storage = Storage::new(crate::db::Db::memory_db().await); @@ -812,12 +801,11 @@ mod tests { let conflict = err .downcast_ref::() .expect("expected typed revision conflict"); - match conflict { - ManagedConfigError::RevisionConflict { expected, current } => { - assert_eq!(expected.as_deref(), Some("rev-old")); - assert_eq!(current.as_deref(), Some("rev-new")); - } - } + let ManagedConfigError::RevisionConflict { expected, current } = conflict else { + panic!("unexpected managed config error: {conflict:?}"); + }; + assert_eq!(expected.as_deref(), Some("rev-old")); + assert_eq!(current.as_deref(), Some("rev-new")); assert_eq!( storage .db() @@ -837,6 +825,237 @@ mod tests { ); } + #[tokio::test] + async fn patch_web_source_configs_applies_delta_and_is_idempotent() { + let storage = Storage::new(crate::db::Db::memory_db().await); + let user_id = storage + .db() + .auto_create_user("web-user-patch") + .await + .unwrap() + .id; + let machine_id = uuid::Uuid::new_v4(); + let update_id = uuid::Uuid::new_v4(); + let delete_id = uuid::Uuid::new_v4(); + let missing_delete_id = uuid::Uuid::new_v4(); + let add_id = uuid::Uuid::new_v4(); + + reconcile_web_source_configs( + &storage, + user_id, + machine_id, + vec![ + managed_config(update_id, "before"), + managed_config(delete_id, "delete"), + ], + Some("rev-1"), + ExpectedConfigRevision::Any, + ) + .await + .unwrap(); + + let status = patch_web_source_configs( + &storage, + user_id, + machine_id, + vec![ + managed_config(update_id, "after"), + managed_config(add_id, "added"), + ], + vec![delete_id, missing_delete_id], + "rev-2", + "rev-1", + ) + .await + .unwrap(); + assert_eq!( + status, + ManagedConfigApplyStatus::Applied { + deleted_web_instance_ids: vec![delete_id], + } + ); + + let retry_status = patch_web_source_configs( + &storage, + user_id, + machine_id, + vec![managed_config(update_id, "ignored-on-idempotent-retry")], + vec![delete_id], + "rev-2", + "rev-1", + ) + .await + .unwrap(); + assert_eq!(retry_status, ManagedConfigApplyStatus::AlreadyApplied); + + let updated = storage + .db() + .get_network_config((user_id, machine_id), &update_id.to_string()) + .await + .unwrap() + .unwrap() + .get_network_config() + .unwrap(); + assert_eq!(updated.network_name.as_deref(), Some("after")); + assert!( + storage + .db() + .get_network_config((user_id, machine_id), &delete_id.to_string()) + .await + .unwrap() + .is_none() + ); + assert!( + storage + .db() + .get_network_config((user_id, machine_id), &add_id.to_string()) + .await + .unwrap() + .is_some() + ); + assert_eq!( + storage + .db() + .get_managed_config_revision((user_id, machine_id)) + .await + .unwrap() + .as_deref(), + Some("rev-2") + ); + } + + #[tokio::test] + async fn patch_web_source_configs_rejects_conflict_and_user_owned_delete() { + let storage = Storage::new(crate::db::Db::memory_db().await); + let user_id = storage + .db() + .auto_create_user("web-user-patch-conflict") + .await + .unwrap() + .id; + let machine_id = uuid::Uuid::new_v4(); + let user_owned_id = uuid::Uuid::new_v4(); + storage + .db() + .insert_or_update_user_network_config( + (user_id, machine_id), + user_owned_id, + NetworkConfig { + network_name: Some("user-owned".to_string()), + ..Default::default() + }, + ConfigSource::User, + ) + .await + .unwrap(); + storage + .db() + .set_managed_config_revision((user_id, machine_id), "rev-current") + .await + .unwrap(); + + let revision_error = patch_web_source_configs( + &storage, + user_id, + machine_id, + vec![managed_config(uuid::Uuid::new_v4(), "new")], + Vec::new(), + "rev-next", + "rev-stale", + ) + .await + .unwrap_err(); + assert!(is_revision_conflict(&revision_error)); + + let ownership_error = patch_web_source_configs( + &storage, + user_id, + machine_id, + Vec::new(), + vec![user_owned_id], + "rev-next", + "rev-current", + ) + .await + .unwrap_err(); + assert!(matches!( + ownership_error.downcast_ref::(), + Some(ManagedConfigError::OwnershipConflict { instance_id }) + if *instance_id == user_owned_id + )); + assert!( + storage + .db() + .get_network_config((user_id, machine_id), &user_owned_id.to_string()) + .await + .unwrap() + .is_some() + ); + assert_eq!( + storage + .db() + .get_managed_config_revision((user_id, machine_id)) + .await + .unwrap() + .as_deref(), + Some("rev-current") + ); + } + + #[tokio::test] + async fn reconcile_web_source_configs_rolls_back_rows_when_revision_write_fails() { + let storage = Storage::new(crate::db::Db::memory_db().await); + let user_id = storage + .db() + .auto_create_user("web-user-rollback") + .await + .unwrap() + .id; + let machine_id = uuid::Uuid::new_v4(); + let instance_id = uuid::Uuid::new_v4(); + sqlx::query( + r#" + CREATE TRIGGER reject_managed_revision + BEFORE INSERT ON managed_config_revisions + WHEN NEW.config_revision = 'reject-revision' + BEGIN + SELECT RAISE(ABORT, 'forced revision failure'); + END + "#, + ) + .execute(&storage.db().inner()) + .await + .unwrap(); + + reconcile_web_source_configs( + &storage, + user_id, + machine_id, + vec![managed_config(instance_id, "must-rollback")], + Some("reject-revision"), + ExpectedConfigRevision::Any, + ) + .await + .unwrap_err(); + + assert!( + storage + .db() + .get_network_config((user_id, machine_id), &instance_id.to_string()) + .await + .unwrap() + .is_none() + ); + assert!( + storage + .db() + .get_managed_config_revision((user_id, machine_id)) + .await + .unwrap() + .is_none() + ); + } + #[test] fn managed_config_reconcile_lock_reuses_live_entry_and_replaces_stale_entry() { let key = (i32::MIN, uuid::Uuid::new_v4()); diff --git a/easytier-web/src/client_manager/mod.rs b/easytier-web/src/client_manager/mod.rs index 9f22059a..fccd1f40 100644 --- a/easytier-web/src/client_manager/mod.rs +++ b/easytier-web/src/client_manager/mod.rs @@ -19,7 +19,7 @@ use easytier_core::{ tunnel::{Tunnel, web_security}, }; use maxminddb::geoip2; -use session::{Location, Session}; +use session::{Location, ManagedConfigRevisionDelta, Session}; use storage::{Storage, StorageToken}; use crate::FeatureFlags; @@ -28,15 +28,13 @@ use tokio::task::JoinSet; use crate::db::{Db, UserIdInDb, entity::user_running_network_configs}; +pub(crate) use managed_config::ManagedConfigError; + #[derive(rust_embed::Embed)] #[folder = "resources/"] #[include = "geoip2-cn.mmdb"] struct GeoipDb; -pub fn is_managed_config_revision_conflict(error: &anyhow::Error) -> bool { - managed_config::is_revision_conflict(error) -} - fn load_geoip_db(geoip_db: Option) -> Option>> { if let Some(path) = geoip_db { match maxminddb::Reader::open_readfile(&path) { @@ -228,7 +226,7 @@ impl ClientManager { Some("") => managed_config::ExpectedConfigRevision::Exact(None), Some(revision) => managed_config::ExpectedConfigRevision::Exact(Some(revision)), }; - managed_config::reconcile_web_source_configs( + let status = managed_config::reconcile_web_source_configs( &self.storage, user_id, machine_id, @@ -237,16 +235,80 @@ impl ClientManager { expected_config_revision, ) .await?; - if let Some(config_revision) = config_revision + if matches!( + status, + managed_config::ManagedConfigApplyStatus::Applied { .. } + ) && let Some(config_revision) = config_revision && let Some(session) = self.get_session_by_machine_id(user_id, &machine_id) { session - .notify_config_revision_changed(user_id, machine_id, config_revision) + .notify_full_config_revision_changed(user_id, machine_id, config_revision) .await; } Ok(()) } + pub async fn patch_managed_network_configs( + &self, + user_id: UserIdInDb, + machine_id: uuid::Uuid, + upserts: Vec, + delete_instance_ids: Vec, + config_revision: String, + expected_config_revision: String, + ) -> anyhow::Result<()> { + let config_revision = config_revision.trim().to_string(); + let expected_config_revision = expected_config_revision.trim().to_string(); + let upsert_instance_ids = upserts + .iter() + .map(|config| config.instance_id.clone()) + .collect(); + let status = managed_config::patch_web_source_configs( + &self.storage, + user_id, + machine_id, + upserts, + delete_instance_ids, + &config_revision, + &expected_config_revision, + ) + .await?; + if let managed_config::ManagedConfigApplyStatus::Applied { + deleted_web_instance_ids, + } = status + && let Some(session) = self.get_session_by_machine_id(user_id, &machine_id) + { + session + .notify_patch_config_revision_changed( + user_id, + machine_id, + ManagedConfigRevisionDelta { + expected_revision: expected_config_revision, + target_revision: config_revision, + upsert_instance_ids, + delete_instance_ids: deleted_web_instance_ids + .into_iter() + .map(|instance_id| instance_id.to_string()) + .collect(), + }, + ) + .await; + } + Ok(()) + } + + pub async fn invalidate_applied_config_revision( + &self, + user_id: UserIdInDb, + machine_id: uuid::Uuid, + ) { + if let Some(session) = self.get_session_by_machine_id(user_id, &machine_id) { + session + .invalidate_applied_config_revision(user_id, machine_id) + .await; + } + } + pub async fn get_heartbeat_requests(&self, client_url: &url::Url) -> Option { let s = self.client_sessions.get(client_url)?.clone(); s.data().read().await.req() @@ -390,7 +452,10 @@ mod tests { use axum::{Json, Router, extract::State, routing::post}; use easytier::{ - common::{MachineIdOptions, config::NetworkConfigExt}, + common::{ + MachineIdOptions, + config::{ConfigSource, NetworkConfigExt}, + }, instance::factory::{ NativeInstanceManager, native_compact_instance_manager_with_runtime, native_instance_manager, @@ -402,7 +467,9 @@ mod tests { }, web_client::{WebClient, run_web_client}, }; - use easytier_core::management::remote_client::Storage as RemoteStorage; + use easytier_core::management::remote_client::{ + RemoteClientManager as _, Storage as RemoteStorage, + }; use serde_json::json; use sqlx::Executor; @@ -693,6 +760,29 @@ mod tests { .unwrap() } + async fn wait_for_applied_revision( + manager: &ClientManager, + user_id: i32, + machine_id: uuid::Uuid, + revision: &str, + ) { + tokio::time::timeout(Duration::from_secs(12), async { + loop { + let applied = manager + .get_session_by_machine_id(user_id, &machine_id) + .map(|session| async move { session.applied_config_revision().await }); + if let Some(applied) = applied + && applied.await.as_deref() == Some(revision) + { + break; + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + }) + .await + .unwrap(); + } + async fn start_web_client_for_test( config_server_addr: std::net::SocketAddr, machine_id: uuid::Uuid, @@ -1081,6 +1171,33 @@ mod tests { config.network_name.as_deref() == Some("managed-initial") }) .await; + wait_for_applied_revision(&mgr, user_id, machine_id, "rev-initial").await; + + // Runtime-only mutations do not change SQLite. Invalidate the Session + // applied fence and verify the existing revision is fully reconciled + // before a later targeted Patch may rely on it as a base. + let mut drifted: NetworkConfig = + serde_json::from_value(initial_managed_network_config(instance_id)).unwrap(); + drifted.network_name = Some("runtime-only-drift".to_string()); + mgr.handle_run_network_instance_with_source( + (user_id, machine_id), + drifted, + false, + ConfigSource::Web, + ) + .await + .unwrap(); + wait_for_runtime_config(&core_manager, instance_id, |config| { + config.network_name.as_deref() == Some("runtime-only-drift") + }) + .await; + mgr.invalidate_applied_config_revision(user_id, machine_id) + .await; + wait_for_runtime_config(&core_manager, instance_id, |config| { + config.network_name.as_deref() == Some("managed-initial") + }) + .await; + wait_for_applied_revision(&mgr, user_id, machine_id, "rev-initial").await; // Online revision update: web-owned running config is fully overwritten // when non-hot-patch flags such as enable_kcp_proxy change. diff --git a/easytier-web/src/client_manager/runtime_reconcile.rs b/easytier-web/src/client_manager/runtime_reconcile.rs index 603b78ff..f231ca43 100644 --- a/easytier-web/src/client_manager/runtime_reconcile.rs +++ b/easytier-web/src/client_manager/runtime_reconcile.rs @@ -64,6 +64,7 @@ fn hot_patch_base(config: &NetworkConfig) -> anyhow::Result { config.port_forwards.clear(); config.proxy_cidrs.clear(); config.disable_relay_data = None; + config.prefer_peer_relay = None; // VPN portal clients are diffed separately; the listener identity // (address and private key) decides between patch and recreate. config.vpn_portal_config = None; @@ -211,6 +212,10 @@ fn normalized_disable_relay_data(config: &NetworkConfig) -> anyhow::Result Ok(config.gen_config()?.get_flags().disable_relay_data) } +fn normalized_prefer_peer_relay(config: &NetworkConfig) -> anyhow::Result { + Ok(config.gen_config()?.get_flags().prefer_peer_relay) +} + fn normalized_vpn_portal(config: &NetworkConfig) -> anyhow::Result> { Ok(config.gen_config()?.get_vpn_portal_config()) } @@ -316,6 +321,12 @@ fn web_source_runtime_patch( patch.disable_relay_data = Some(desired_disable_relay_data); } + let current_prefer_peer_relay = normalized_prefer_peer_relay(current)?; + let desired_prefer_peer_relay = normalized_prefer_peer_relay(desired)?; + if current_prefer_peer_relay != desired_prefer_peer_relay { + patch.prefer_peer_relay = Some(desired_prefer_peer_relay); + } + match ( normalized_vpn_portal(current)?, normalized_vpn_portal(desired)?, @@ -943,6 +954,20 @@ mod tests { assert_eq!(patch.disable_relay_data, Some(true)); } + #[test] + fn runtime_patch_updates_peer_relay_preference_independently() { + let current = config_with_port_forwards(Vec::new()); + let mut desired = current.clone(); + desired.prefer_peer_relay = Some(true); + + let patch = web_source_runtime_patch(¤t, &desired) + .expect("build patch") + .expect("hot patch"); + + assert_eq!(patch.prefer_peer_relay, Some(true)); + assert_eq!(patch.disable_relay_data, None); + } + #[test] fn runtime_patch_still_rejects_unsupported_flag_change() { let current = config_with_port_forwards(Vec::new()); diff --git a/easytier-web/src/client_manager/session.rs b/easytier-web/src/client_manager/session.rs index 778b03c6..e616a6f9 100644 --- a/easytier-web/src/client_manager/session.rs +++ b/easytier-web/src/client_manager/session.rs @@ -1,4 +1,5 @@ use std::{ + collections::HashSet, fmt::Debug, str::FromStr as _, sync::Arc, @@ -42,6 +43,14 @@ enum SessionAuthState { Invalid, } +#[derive(Debug, Clone, PartialEq, Eq)] +pub(super) struct ManagedConfigRevisionDelta { + pub expected_revision: String, + pub target_revision: String, + pub upsert_instance_ids: HashSet, + pub delete_instance_ids: HashSet, +} + impl SessionAuthState { fn is_authorized(self) -> bool { matches!(self, Self::Authorized) @@ -58,6 +67,8 @@ pub struct SessionData { storage_token: Option, binding_version: Option, applied_config_revision: Option, + pending_managed_config_delta: Option, + runtime_config_epoch: u64, notifier: broadcast::Sender, req: Option, location: Option, @@ -88,6 +99,8 @@ impl SessionData { storage_token: None, binding_version: None, applied_config_revision: None, + pending_managed_config_delta: None, + runtime_config_epoch: 0, notifier: tx, req: None, location, @@ -698,14 +711,14 @@ impl Session { self.scoped_client::>() } - pub async fn notify_config_revision_changed( + pub(super) async fn notify_full_config_revision_changed( &self, user_id: i32, machine_id: uuid::Uuid, config_revision: String, ) { let notify = { - let data = self.data.read().await; + let mut data = self.data.write().await; if !data.auth_state.is_authorized() { return; } @@ -719,6 +732,84 @@ impl Session { if data.applied_config_revision.as_deref() == Some(config_revision.as_str()) { return; } + data.pending_managed_config_delta = None; + data.req.clone().map(|req| (data.notifier.clone(), req)) + }; + if let Some((notifier, req)) = notify { + let _ = notifier.send(req); + } + } + + pub(super) async fn notify_patch_config_revision_changed( + &self, + user_id: i32, + machine_id: uuid::Uuid, + delta: ManagedConfigRevisionDelta, + ) { + let notify = { + let mut data = self.data.write().await; + if !data.auth_state.is_authorized() { + return; + } + if !data + .storage_token + .as_ref() + .is_some_and(|token| token.user_id == user_id && token.machine_id == machine_id) + { + return; + } + if data.applied_config_revision.as_deref() == Some(delta.target_revision.as_str()) { + return; + } + + // A Patch may drive a targeted runtime reconcile only when the + // connected Session has applied its exact base and no earlier + // Patch is still pending. Otherwise the normal Full reconcile is + // the safe convergence path. + data.pending_managed_config_delta = (data.applied_config_revision.as_deref() + == Some(delta.expected_revision.as_str()) + && data.pending_managed_config_delta.is_none()) + .then_some(delta); + data.req.clone().map(|req| (data.notifier.clone(), req)) + }; + if let Some((notifier, req)) = notify { + let _ = notifier.send(req); + } + } + + pub(super) async fn invalidate_applied_config_revision( + &self, + user_id: i32, + machine_id: uuid::Uuid, + ) { + let notify = { + let mut data = self.data.write().await; + if !data + .storage_token + .as_ref() + .is_some_and(|token| token.user_id == user_id && token.machine_id == machine_id) + { + return; + } + data.applied_config_revision = None; + data.pending_managed_config_delta = None; + data.runtime_config_epoch = data.runtime_config_epoch.wrapping_add(1); + data.req.clone().map(|req| (data.notifier.clone(), req)) + }; + if let Some((notifier, req)) = notify { + let _ = notifier.send(req); + } + } + + pub(crate) async fn invalidate_runtime_config_for_direct_mutation(&self) { + let notify = { + let mut data = self.data.write().await; + if data.storage_token.is_none() { + return; + } + data.applied_config_revision = None; + data.pending_managed_config_delta = None; + data.runtime_config_epoch = data.runtime_config_epoch.wrapping_add(1); data.req.clone().map(|req| (data.notifier.clone(), req)) }; if let Some((notifier, req)) = notify { @@ -733,6 +824,11 @@ impl Session { pub async fn get_heartbeat_req(&self) -> Option { self.data.read().await.req() } + + #[cfg(test)] + pub(super) async fn applied_config_revision(&self) -> Option { + self.data.read().await.applied_config_revision.clone() + } } #[cfg(test)] diff --git a/easytier-web/src/client_manager/session/runtime_revision.rs b/easytier-web/src/client_manager/session/runtime_revision.rs index 9e0c9008..4c0c297b 100644 --- a/easytier-web/src/client_manager/session/runtime_revision.rs +++ b/easytier-web/src/client_manager/session/runtime_revision.rs @@ -2,8 +2,9 @@ use std::collections::{HashMap, HashSet}; use easytier::proto::{ api::manage::{ - DeleteNetworkInstanceRequest, ListNetworkInstanceMetaRequest, ListNetworkInstanceRequest, - NetworkConfig, NetworkMeta, RunNetworkInstanceRequest, + DeleteNetworkInstanceRequest, DeleteNetworkInstanceResponse, + ListNetworkInstanceMetaRequest, ListNetworkInstanceRequest, NetworkConfig, NetworkMeta, + RunNetworkInstanceRequest, }, rpc_types::controller::BaseController, web::HeartbeatRequest, @@ -11,7 +12,10 @@ use easytier::proto::{ use easytier_core::management::remote_client::{ListNetworkProps, Storage as _}; use tokio::sync::{RwLock, broadcast}; -use super::{SessionConfigClient, SessionData, SessionRpcClient, SessionRpcService}; +use super::{ + ManagedConfigRevisionDelta, SessionConfigClient, SessionData, SessionRpcClient, + SessionRpcService, +}; use crate::client_manager::{ managed_config::{self, PersistedConfigSource}, runtime_reconcile, @@ -75,48 +79,105 @@ pub(super) async fn reconcile_network_configs_on_heartbeat( RoundStatus::Skip => continue, RoundStatus::Stop => return, }; + if cache.runtime_config_epoch != round.runtime_config_epoch { + cache = ReconcileCache { + runtime_config_epoch: round.runtime_config_epoch, + ..Default::default() + }; + } let running_metas = match sync_running_sources_for_round(&mut rpc_client, &storage, &mut round).await { RoundStatus::Ready(running_metas) => running_metas, RoundStatus::Skip => continue, RoundStatus::Stop => return, }; + let mut mutation_fence = RuntimeMutationFence::default(); + let context = ReconcileRoundContext { + session_data: &session_data, + round: &round, + }; - let desired_web_inst_ids = - managed_config::desired_web_source_instance_ids(&round.local_configs); - cache.runtime_configs.retain_desired(&desired_web_inst_ids); - let mut outcome = match cleanup_stale_web_source_instances( - &session_data, - &storage, - &mut rpc_client, - &round, - running_metas.as_deref(), - &desired_web_inst_ids, - &mut cache, - ) - .await - { - RoundStatus::Ready(outcome) => outcome, - RoundStatus::Skip => continue, - RoundStatus::Stop => return, + let mut outcome = match &round.scope { + ReconcileScope::Full => { + let desired_web_inst_ids = + managed_config::desired_web_source_instance_ids(&round.local_configs); + cache.runtime_configs.retain_desired(&desired_web_inst_ids); + match cleanup_stale_web_source_instances( + &context, + &storage, + &mut rpc_client, + running_metas.as_deref(), + &desired_web_inst_ids, + &mut cache, + &mut mutation_fence, + ) + .await + { + RoundStatus::Ready(outcome) => outcome, + RoundStatus::Skip => continue, + RoundStatus::Stop => return, + } + } + ReconcileScope::Patch { + delete_instance_ids, + .. + } => { + match cleanup_patch_deleted_instances( + &session_data, + &mut rpc_client, + &round, + running_metas.as_deref(), + delete_instance_ids, + &mut cache, + &mut mutation_fence, + ) + .await + { + RoundStatus::Ready(outcome) => outcome, + RoundStatus::Skip => continue, + RoundStatus::Stop => return, + } + } }; outcome.merge( reconcile_desired_runtime_configs( - &session_data, + &context, &mut rpc_client, &mut config_client, - &round, &mut cache, + &mut mutation_fence, ) .await, ); if !outcome.has_failed { - cache.last_desired_web_inst_ids = Some(desired_web_inst_ids); + match &round.scope { + ReconcileScope::Full => { + cache.last_desired_web_inst_ids = Some( + managed_config::desired_web_source_instance_ids(&round.local_configs), + ); + } + ReconcileScope::Patch { + upsert_instance_ids, + delete_instance_ids, + } => { + if let Some(last) = &mut cache.last_desired_web_inst_ids { + last.retain(|id| !delete_instance_ids.contains(id)); + last.extend(upsert_instance_ids.iter().cloned()); + } + } + } } - match mark_config_revision_applied_if_current(&session_data, &storage, &round, &outcome) - .await + + match mark_config_revision_applied_if_current( + &session_data, + &storage, + &round, + &outcome, + &mutation_fence, + ) + .await { RoundStatus::Ready(()) | RoundStatus::Skip => {} RoundStatus::Stop => return, @@ -138,6 +199,7 @@ enum ConfigActionResult { #[derive(Default)] struct ReconcileCache { + runtime_config_epoch: u64, cleaned_web_source_instances: bool, last_desired_web_inst_ids: Option>, runtime_configs: SessionRuntimeConfigCache, @@ -191,6 +253,11 @@ struct ReconcileOutcome { managed_revision_failed: bool, } +#[derive(Default)] +struct RuntimeMutationFence { + started: bool, +} + impl ReconcileOutcome { fn record_failure(&mut self, managed_revision_failed: bool) { self.has_failed = true; @@ -211,6 +278,41 @@ struct ReconcileRound { local_configs: Vec, target_config_revision: Option, should_apply_runtime_revision: bool, + scope: ReconcileScope, + runtime_config_epoch: u64, +} + +struct ReconcileRoundContext<'a> { + session_data: &'a std::sync::Weak>, + round: &'a ReconcileRound, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +enum ReconcileScope { + Full, + Patch { + upsert_instance_ids: HashSet, + delete_instance_ids: HashSet, + }, +} + +fn select_reconcile_scope( + applied_revision: Option<&str>, + target_revision: Option<&str>, + pending_delta: Option<&ManagedConfigRevisionDelta>, +) -> ReconcileScope { + match pending_delta { + Some(delta) + if applied_revision == Some(delta.expected_revision.as_str()) + && target_revision == Some(delta.target_revision.as_str()) => + { + ReconcileScope::Patch { + upsert_instance_ids: delta.upsert_instance_ids.clone(), + delete_instance_ids: delta.delete_instance_ids.clone(), + } + } + _ => ReconcileScope::Full, + } } async fn prepare_reconcile_round( @@ -244,11 +346,16 @@ async fn prepare_reconcile_round( } }; - let applied_config_revision = { + let (applied_config_revision, pending_delta, runtime_config_epoch) = { let Some(data) = session_data.upgrade() else { return RoundStatus::Stop; }; - data.read().await.applied_config_revision.clone() + let data = data.read().await; + ( + data.applied_config_revision.clone(), + data.pending_managed_config_delta.clone(), + data.runtime_config_epoch, + ) }; let target_config_revision = match storage .db @@ -263,6 +370,15 @@ async fn prepare_reconcile_round( }; let should_apply_runtime_revision = target_config_revision.is_some() && target_config_revision != applied_config_revision; + let mut scope = if should_apply_runtime_revision { + select_reconcile_scope( + applied_config_revision.as_deref(), + target_config_revision.as_deref(), + pending_delta.as_ref(), + ) + } else { + ReconcileScope::Full + }; let running_inst_ids = match running_instance_ids_for_round( rpc_client, &req, @@ -277,14 +393,29 @@ async fn prepare_reconcile_round( RoundStatus::Stop => return RoundStatus::Stop, }; - let local_configs = match storage - .db - .list_network_configs((user_id, machine_id), ListNetworkProps::EnabledOnly) - .await - { - Ok(configs) => configs, + let local_configs = match load_round_configs(storage, user_id, machine_id, &scope).await { + Ok(Some(configs)) => configs, + Ok(None) => { + tracing::warn!( + ?user_id, + ?machine_id, + "Managed config Patch no longer matches persisted rows; using Full reconcile" + ); + scope = ReconcileScope::Full; + match storage + .db + .list_network_configs((user_id, machine_id), ListNetworkProps::EnabledOnly) + .await + { + Ok(configs) => configs, + Err(e) => { + tracing::error!("Failed to list network configs, error: {:?}", e); + return RoundStatus::Stop; + } + } + } Err(e) => { - tracing::error!("Failed to list network configs, error: {:?}", e); + tracing::error!("Failed to load managed config Patch rows, error: {:?}", e); return RoundStatus::Stop; } }; @@ -297,9 +428,50 @@ async fn prepare_reconcile_round( local_configs, target_config_revision, should_apply_runtime_revision, + scope, + runtime_config_epoch, }) } +async fn load_round_configs( + storage: &StorageInner, + user_id: i32, + machine_id: uuid::Uuid, + scope: &ReconcileScope, +) -> Result>, sea_orm::DbErr> { + let ReconcileScope::Patch { + upsert_instance_ids, + .. + } = scope + else { + return storage + .db + .list_network_configs((user_id, machine_id), ListNetworkProps::EnabledOnly) + .await + .map(Some); + }; + + let mut instance_ids = upsert_instance_ids.iter().collect::>(); + instance_ids.sort_unstable(); + let mut configs = Vec::with_capacity(instance_ids.len()); + for instance_id in instance_ids { + let Some(config) = storage + .db + .get_network_config((user_id, machine_id), instance_id) + .await? + else { + return Ok(None); + }; + if config.disabled + || PersistedConfigSource::from_db(&config.source) != PersistedConfigSource::Web + { + return Ok(None); + } + configs.push(config); + } + Ok(Some(configs)) +} + async fn running_instance_ids_for_round( rpc_client: &mut SessionRpcClient, req: &HeartbeatRequest, @@ -375,7 +547,7 @@ async fn sync_running_sources_for_round( %e, "Failed to sync running network config sources" ); - } else if !metas.is_empty() { + } else if !metas.is_empty() && matches!(round.scope, ReconcileScope::Full) { round.local_configs = match storage .db .list_network_configs( @@ -408,14 +580,16 @@ async fn sync_running_sources_for_round( } async fn cleanup_stale_web_source_instances( - session_data: &std::sync::Weak>, + context: &ReconcileRoundContext<'_>, storage: &StorageInner, rpc_client: &mut SessionRpcClient, - round: &ReconcileRound, running_metas: Option<&[NetworkMeta]>, desired_web_inst_ids: &HashSet, cache: &mut ReconcileCache, + mutation_fence: &mut RuntimeMutationFence, ) -> RoundStatus { + let session_data = context.session_data; + let round = context.round; let desired_changed = cache .last_desired_web_inst_ids .as_ref() @@ -450,10 +624,10 @@ async fn cleanup_stale_web_source_instances( let mut outcome = ReconcileOutcome::default(); if !should_delete_ids.is_empty() { - if !SessionRpcService::runtime_heartbeat_is_current(session_data, &round.req).await { + if !begin_managed_runtime_mutation(session_data, round, mutation_fence).await { tracing::debug!( machine_id = ?round.machine_id, - "skip stale cleanup because webhook session is no longer current" + "skip stale cleanup because the managed runtime fence is no longer current" ); return RoundStatus::Skip; } @@ -471,10 +645,23 @@ async fn cleanup_stale_web_source_instances( ret, round.req.user_token ); - if ret.is_err() { - outcome.record_failure(true); - } else { - cache.runtime_configs.forget_many(&should_delete_inst_ids); + match ret { + Err(_) => outcome.record_failure(true), + Ok(response) => { + let undeleted_instance_ids = + retained_requested_instance_ids(response, &should_delete_inst_ids); + if undeleted_instance_ids.is_empty() { + cache.runtime_configs.forget_many(&should_delete_inst_ids); + } else { + tracing::warn!( + user_id = ?round.user_id, + machine_id = ?round.machine_id, + instance_ids = ?undeleted_instance_ids, + "Stale managed instances were retained by the runtime" + ); + outcome.record_failure(true); + } + } } } @@ -486,13 +673,128 @@ async fn cleanup_stale_web_source_instances( RoundStatus::Ready(outcome) } -async fn reconcile_desired_runtime_configs( +async fn cleanup_patch_deleted_instances( session_data: &std::sync::Weak>, rpc_client: &mut SessionRpcClient, - config_client: &mut SessionConfigClient, round: &ReconcileRound, + running_metas: Option<&[NetworkMeta]>, + delete_instance_ids: &HashSet, cache: &mut ReconcileCache, + mutation_fence: &mut RuntimeMutationFence, +) -> RoundStatus { + let running_web_instance_ids: HashSet = match running_metas { + Some(metas) => managed_config::running_web_source_instance_ids( + &round.running_inst_ids, + delete_instance_ids, + Some(metas), + ) + .intersection(delete_instance_ids) + .cloned() + .collect(), + None => round + .running_inst_ids + .intersection(delete_instance_ids) + .cloned() + .collect(), + }; + if running_web_instance_ids.is_empty() { + cache + .runtime_configs + .forget_many(delete_instance_ids.iter()); + return RoundStatus::Ready(ReconcileOutcome::default()); + } + if !begin_managed_runtime_mutation(session_data, round, mutation_fence).await { + tracing::debug!( + machine_id = ?round.machine_id, + "skip managed config Patch cleanup because the runtime fence is no longer current" + ); + return RoundStatus::Skip; + } + + let ret = rpc_client + .delete_network_instance( + BaseController::default(), + DeleteNetworkInstanceRequest { + inst_ids: managed_config::parse_instance_ids( + running_web_instance_ids.iter().cloned(), + ), + }, + ) + .await; + tracing::info!( + user_id = ?round.user_id, + deleted_instance_ids = ?running_web_instance_ids, + "Apply managed config Patch deletions at runtime: {:?}", + ret + ); + + let mut outcome = ReconcileOutcome::default(); + match ret { + Err(_) => outcome.record_failure(true), + Ok(response) => { + let undeleted_instance_ids = + retained_requested_instance_ids(response, &running_web_instance_ids); + if undeleted_instance_ids.is_empty() { + cache + .runtime_configs + .forget_many(delete_instance_ids.iter()); + } else { + tracing::warn!( + user_id = ?round.user_id, + machine_id = ?round.machine_id, + instance_ids = ?undeleted_instance_ids, + "Managed config Patch deletion was retained by the runtime" + ); + outcome.record_failure(true); + } + } + } + RoundStatus::Ready(outcome) +} + +async fn begin_managed_runtime_mutation( + session_data: &std::sync::Weak>, + round: &ReconcileRound, + mutation_fence: &mut RuntimeMutationFence, +) -> bool { + let Some(data) = session_data.upgrade() else { + return false; + }; + let mut data = data.write().await; + if !SessionRpcService::runtime_heartbeat_is_current_locked(&data, &round.req) + || data.runtime_config_epoch != round.runtime_config_epoch + { + return false; + } + if !mutation_fence.started { + data.applied_config_revision = None; + data.pending_managed_config_delta = None; + mutation_fence.started = true; + } + true +} + +fn retained_requested_instance_ids( + response: DeleteNetworkInstanceResponse, + requested_instance_ids: &HashSet, +) -> HashSet { + response + .remain_inst_ids + .into_iter() + .map(|instance_id| uuid::Uuid::from(instance_id).to_string()) + .filter(|instance_id| requested_instance_ids.contains(instance_id)) + .collect() +} + +async fn reconcile_desired_runtime_configs( + context: &ReconcileRoundContext<'_>, + rpc_client: &mut SessionRpcClient, + config_client: &mut SessionConfigClient, + cache: &mut ReconcileCache, + mutation_fence: &mut RuntimeMutationFence, ) -> ReconcileOutcome { + let session_data = context.session_data; + let round = context.round; let mut outcome = ReconcileOutcome::default(); // After stale web-owned instances are removed, start every enabled @@ -529,13 +831,13 @@ async fn reconcile_desired_runtime_configs( let action_result = if should_reconcile_running_web_config { reconcile_running_web_config( - session_data, + context, rpc_client, config_client, - round, config, desired_config, &mut cache.runtime_configs, + mutation_fence, ) .await } else { @@ -548,6 +850,7 @@ async fn reconcile_desired_runtime_configs( round, config, desired_config.clone(), + mutation_fence, ) .await; if matches!(action_result, ConfigActionResult::Success) @@ -599,14 +902,16 @@ async fn reconcile_desired_runtime_configs( } async fn reconcile_running_web_config( - session_data: &std::sync::Weak>, + context: &ReconcileRoundContext<'_>, rpc_client: &mut SessionRpcClient, config_client: &mut SessionConfigClient, - round: &ReconcileRound, config: &crate::db::entity::user_running_network_configs::Model, desired_config: NetworkConfig, runtime_config_cache: &mut SessionRuntimeConfigCache, + mutation_fence: &mut RuntimeMutationFence, ) -> ConfigActionResult { + let session_data = context.session_data; + let round = context.round; if !SessionRpcService::runtime_heartbeat_is_current(session_data, &round.req).await { tracing::debug!( machine_id = ?round.machine_id, @@ -633,6 +938,11 @@ async fn reconcile_running_web_config( if !SessionRpcService::runtime_heartbeat_is_current(session_data, &round.req).await { anyhow::bail!("webhook session is no longer current before runtime reconcile apply"); } + if !matches!(action, runtime_reconcile::RuntimeReconcileAction::None) + && !begin_managed_runtime_mutation(session_data, round, mutation_fence).await + { + anyhow::bail!("managed runtime mutation fence is no longer current"); + } let observed_config = runtime_reconcile::apply_web_source_runtime_reconcile( &mut *rpc_client, &mut *config_client, @@ -667,6 +977,7 @@ async fn run_missing_network_config( round: &ReconcileRound, config: &crate::db::entity::user_running_network_configs::Model, desired_config: NetworkConfig, + mutation_fence: &mut RuntimeMutationFence, ) -> ConfigActionResult { if !SessionRpcService::runtime_heartbeat_is_current(session_data, &round.req).await { tracing::debug!( @@ -677,6 +988,18 @@ async fn run_missing_network_config( return ConfigActionResult::StopRound; } + let source = PersistedConfigSource::from_db(&config.source); + if source == PersistedConfigSource::Web + && !begin_managed_runtime_mutation(session_data, round, mutation_fence).await + { + tracing::debug!( + machine_id = ?round.machine_id, + instance_id = %config.network_instance_id, + "skip run network instance because the managed runtime fence is no longer current" + ); + return ConfigActionResult::StopRound; + } + let ret = rpc_client .run_network_instance( BaseController::default(), @@ -684,7 +1007,7 @@ async fn run_missing_network_config( inst_id: Some(config.network_instance_id.clone().into()), config: Some(desired_config), overwrite: false, - source: PersistedConfigSource::from_db(&config.source).auto_run_rpc_source() as i32, + source: source.auto_run_rpc_source() as i32, }, ) .await; @@ -739,8 +1062,11 @@ async fn mark_config_revision_applied_if_current( storage: &StorageInner, round: &ReconcileRound, outcome: &ReconcileOutcome, + mutation_fence: &RuntimeMutationFence, ) -> RoundStatus<()> { - if outcome.managed_revision_failed || !round.should_apply_runtime_revision { + if outcome.managed_revision_failed + || (!round.should_apply_runtime_revision && !mutation_fence.started) + { return RoundStatus::Ready(()); } @@ -765,7 +1091,11 @@ async fn mark_config_revision_applied_if_current( if !SessionRpcService::runtime_heartbeat_is_current_locked(&data, &round.req) { return RoundStatus::Ready(()); } + if data.runtime_config_epoch != round.runtime_config_epoch { + return RoundStatus::Ready(()); + } data.applied_config_revision = round.target_config_revision.clone(); + data.pending_managed_config_delta = None; RoundStatus::Ready(()) } @@ -798,6 +1128,132 @@ mod tests { } } + #[test] + fn patch_delete_requires_runtime_to_remove_every_requested_instance() { + let deleted_id = uuid::Uuid::new_v4(); + let requested = HashSet::from([deleted_id.to_string()]); + + assert_eq!( + retained_requested_instance_ids( + DeleteNetworkInstanceResponse { + remain_inst_ids: vec![deleted_id.into()], + }, + &requested, + ), + requested + ); + assert!( + retained_requested_instance_ids( + DeleteNetworkInstanceResponse { + remain_inst_ids: Vec::new(), + }, + &requested, + ) + .is_empty() + ); + } + + #[tokio::test] + async fn managed_runtime_mutation_clears_old_applied_revision_before_side_effects() { + let machine_id = uuid::Uuid::new_v4(); + let req = HeartbeatRequest { + user_token: "token".to_string(), + machine_id: Some(machine_id.into()), + ..Default::default() + }; + let storage = + crate::client_manager::storage::Storage::new(crate::db::Db::memory_db().await); + let client_url = url::Url::parse("http://127.0.0.1").unwrap(); + let mut data = SessionData::new( + storage.weak_ref(), + client_url.clone(), + None, + std::sync::Arc::new(crate::FeatureFlags::default()), + std::sync::Arc::new(crate::webhook::WebhookConfig::new( + None, None, None, None, None, + )), + ); + data.storage_token = Some(crate::client_manager::storage::StorageToken { + token: req.user_token.clone(), + client_url, + machine_id, + user_id: 7, + }); + data.req = Some(req.clone()); + data.auth_state = super::super::SessionAuthState::Authorized; + data.applied_config_revision = Some("rev-a".to_string()); + data.pending_managed_config_delta = Some(revision_delta("rev-a", "rev-b")); + data.runtime_config_epoch = 11; + let session_data = std::sync::Arc::new(RwLock::new(data)); + let round = ReconcileRound { + req, + machine_id, + user_id: 7, + running_inst_ids: HashSet::new(), + local_configs: Vec::new(), + target_config_revision: Some("rev-b".to_string()), + should_apply_runtime_revision: true, + scope: ReconcileScope::Full, + runtime_config_epoch: 11, + }; + let mut mutation_fence = RuntimeMutationFence::default(); + + assert!( + begin_managed_runtime_mutation( + &std::sync::Arc::downgrade(&session_data), + &round, + &mut mutation_fence, + ) + .await + ); + + let data = session_data.read().await; + assert!(mutation_fence.started); + assert_eq!(data.applied_config_revision, None); + assert_eq!(data.pending_managed_config_delta, None); + assert_eq!(data.runtime_config_epoch, 11); + } + + fn revision_delta(base: &str, target: &str) -> ManagedConfigRevisionDelta { + ManagedConfigRevisionDelta { + expected_revision: base.to_string(), + target_revision: target.to_string(), + upsert_instance_ids: HashSet::from(["upsert".to_string()]), + delete_instance_ids: HashSet::from(["delete".to_string()]), + } + } + + #[test] + fn exact_revision_delta_selects_targeted_reconcile() { + let delta = revision_delta("rev-1", "rev-2"); + + assert_eq!( + select_reconcile_scope(Some("rev-1"), Some("rev-2"), Some(&delta)), + ReconcileScope::Patch { + upsert_instance_ids: HashSet::from(["upsert".to_string()]), + delete_instance_ids: HashSet::from(["delete".to_string()]), + } + ); + } + + #[test] + fn revision_gap_uses_full_reconcile() { + let delta = revision_delta("rev-1", "rev-2"); + + assert_eq!( + select_reconcile_scope(Some("older"), Some("rev-2"), Some(&delta)), + ReconcileScope::Full + ); + assert_eq!( + select_reconcile_scope(Some("rev-1"), Some("newer"), Some(&delta)), + ReconcileScope::Full + ); + assert_eq!( + select_reconcile_scope(Some("rev-1"), Some("rev-2"), None), + ReconcileScope::Full + ); + } + #[test] fn session_runtime_config_cache_misses_unknown_instance() { let cache = SessionRuntimeConfigCache::default(); diff --git a/easytier-web/src/client_manager/session/webhook_validation.rs b/easytier-web/src/client_manager/session/webhook_validation.rs index 6a71ca6e..2121e901 100644 --- a/easytier-web/src/client_manager/session/webhook_validation.rs +++ b/easytier-web/src/client_manager/session/webhook_validation.rs @@ -248,6 +248,7 @@ pub(super) async fn apply_rejected( data.webhook_validation_dirty = false; data.binding_version = None; data.applied_config_revision = None; + data.pending_managed_config_delta = None; let storage_token = data.storage_token.clone(); let disconnect_notification = storage_token.as_ref().and_then(|storage_token| { data.webhook_connected_binding_version diff --git a/easytier-web/src/db/mod.rs b/easytier-web/src/db/mod.rs index 1b2d2c94..30992667 100644 --- a/easytier-web/src/db/mod.rs +++ b/easytier-web/src/db/mod.rs @@ -7,10 +7,11 @@ use easytier_core::management::remote_client::{ListNetworkProps, Storage}; use entity::user_running_network_configs; use sea_orm::{ ColumnTrait as _, DatabaseConnection, DbErr, EntityTrait, QueryFilter as _, Set, - SqlxSqliteConnector, TransactionTrait as _, prelude::Expr, sea_query::OnConflict, + SqlxSqliteConnector, TransactionTrait as _, sea_query::OnConflict, }; use sea_orm_migration::MigratorTrait as _; use sqlx::{Sqlite, SqlitePool, migrate::MigrateDatabase as _, types::chrono}; +use std::collections::{HashMap, HashSet}; #[cfg(unix)] use std::os::unix::fs::PermissionsExt as _; use uuid::Uuid; @@ -20,6 +21,179 @@ use async_trait::async_trait; pub type UserIdInDb = i32; +#[derive(Debug)] +pub(crate) struct ManagedConfigUpsert { + pub instance_id: Uuid, + pub network_config: NetworkConfig, +} + +#[derive(Debug, Clone)] +pub(crate) enum ManagedConfigExpectedRevision { + Any, + Exact(Option), +} + +#[derive(Debug)] +pub(crate) enum ManagedConfigUpdate { + Full { + upserts: Vec, + target_revision: Option, + expected_revision: ManagedConfigExpectedRevision, + }, + Patch { + upserts: Vec, + delete_instance_ids: Vec, + target_revision: String, + expected_revision: String, + }, +} + +#[derive(Debug, PartialEq, Eq)] +pub(crate) enum ManagedConfigApplyResult { + Applied { + deleted_web_instance_ids: Vec, + }, + AlreadyApplied, + RevisionConflict { + expected: Option, + current: Option, + }, + OwnershipConflict { + instance_id: Uuid, + }, +} + +fn sqlx_db_error(error: sqlx::Error) -> DbErr { + DbErr::Custom(error.to_string()) +} + +async fn read_managed_config_revision( + transaction: &mut sqlx::Transaction<'_, Sqlite>, + user_id: UserIdInDb, + device_id: Uuid, +) -> Result, DbErr> { + sqlx::query_scalar( + r#" + SELECT config_revision + FROM managed_config_revisions + WHERE user_id = ? AND device_id = ? + "#, + ) + .bind(user_id) + .bind(device_id.to_string()) + .fetch_optional(&mut **transaction) + .await + .map_err(sqlx_db_error) +} + +async fn clear_managed_config_revision( + transaction: &mut sqlx::Transaction<'_, Sqlite>, + user_id: UserIdInDb, + device_id: Uuid, +) -> Result<(), DbErr> { + sqlx::query( + r#" + DELETE FROM managed_config_revisions + WHERE user_id = ? AND device_id = ? + "#, + ) + .bind(user_id) + .bind(device_id.to_string()) + .execute(&mut **transaction) + .await + .map_err(sqlx_db_error)?; + Ok(()) +} + +async fn write_managed_config_revision( + transaction: &mut sqlx::Transaction<'_, Sqlite>, + user_id: UserIdInDb, + device_id: Uuid, + config_revision: &str, +) -> Result<(), DbErr> { + let now = chrono::Local::now().fixed_offset(); + sqlx::query( + r#" + INSERT INTO managed_config_revisions ( + user_id, device_id, config_revision, create_time, update_time + ) VALUES (?, ?, ?, ?, ?) + ON CONFLICT(user_id, device_id) DO UPDATE SET + config_revision = excluded.config_revision, + update_time = excluded.update_time + "#, + ) + .bind(user_id) + .bind(device_id.to_string()) + .bind(config_revision) + .bind(now) + .bind(now) + .execute(&mut **transaction) + .await + .map_err(sqlx_db_error)?; + Ok(()) +} + +async fn read_config_source( + transaction: &mut sqlx::Transaction<'_, Sqlite>, + user_id: UserIdInDb, + device_id: Uuid, + instance_id: Uuid, +) -> Result, DbErr> { + sqlx::query_scalar( + r#" + SELECT source + FROM user_running_network_configs + WHERE user_id = ? AND device_id = ? AND network_instance_id = ? + "#, + ) + .bind(user_id) + .bind(device_id.to_string()) + .bind(instance_id.to_string()) + .fetch_optional(&mut **transaction) + .await + .map_err(sqlx_db_error) +} + +async fn upsert_network_config( + transaction: &mut sqlx::Transaction<'_, Sqlite>, + user_id: UserIdInDb, + device_id: Uuid, + instance_id: Uuid, + network_config: &str, + source: ConfigSource, + web_only_update: bool, +) -> Result { + let now = chrono::Local::now().fixed_offset(); + let mut query = r#" + INSERT INTO user_running_network_configs ( + user_id, device_id, network_instance_id, network_config, + source, disabled, create_time, update_time + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(user_id, device_id, network_instance_id) DO UPDATE SET + network_config = excluded.network_config, + source = excluded.source, + disabled = excluded.disabled, + update_time = excluded.update_time + "# + .to_string(); + if web_only_update { + query.push_str(" WHERE user_running_network_configs.source = 'web'"); + } + let result = sqlx::query(&query) + .bind(user_id) + .bind(device_id.to_string()) + .bind(instance_id.to_string()) + .bind(network_config) + .bind(source.as_str()) + .bind(false) + .bind(now) + .bind(now) + .execute(&mut **transaction) + .await + .map_err(sqlx_db_error)?; + Ok(result.rows_affected() > 0) +} + #[cfg(unix)] fn restrict_database_file_permissions(db_path: &str) -> anyhow::Result<()> { if db_path.ends_with(":memory:") || db_path.contains("mode=memory") { @@ -214,44 +388,205 @@ impl Db { Ok(()) } - pub async fn insert_or_update_web_network_config( + pub(crate) async fn apply_managed_config_update( &self, (user_id, device_id): (UserIdInDb, Uuid), - network_inst_id: Uuid, - network_config: NetworkConfig, - ) -> Result { - let now = chrono::Local::now().fixed_offset(); - let network_config = - serde_json::to_string(&network_config).map_err(|e| DbErr::Json(e.to_string()))?; - let source = ConfigSource::Web.as_str(); - let result = sqlx::query( - r#" - INSERT INTO user_running_network_configs ( - user_id, device_id, network_instance_id, network_config, - source, disabled, create_time, update_time - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) - ON CONFLICT(user_id, device_id, network_instance_id) DO UPDATE SET - network_config = excluded.network_config, - source = excluded.source, - disabled = excluded.disabled, - update_time = excluded.update_time - WHERE user_running_network_configs.source = ? - "#, - ) - .bind(user_id) - .bind(device_id.to_string()) - .bind(network_inst_id.to_string()) - .bind(network_config) - .bind(source) - .bind(false) - .bind(now) - .bind(now) - .bind(source) - .execute(&self.db) - .await - .map_err(|e| DbErr::Custom(e.to_string()))?; + update: ManagedConfigUpdate, + ) -> Result { + let (upserts, target_revision, expected_revision) = match &update { + ManagedConfigUpdate::Full { + upserts, + target_revision, + expected_revision, + } => ( + upserts, + target_revision.as_deref(), + expected_revision.clone(), + ), + ManagedConfigUpdate::Patch { + upserts, + target_revision, + expected_revision, + .. + } => ( + upserts, + Some(target_revision.as_str()), + ManagedConfigExpectedRevision::Exact(Some(expected_revision.clone())), + ), + }; + let serialized_upserts = upserts + .iter() + .map(|upsert| { + serde_json::to_string(&upsert.network_config) + .map(|config| (upsert.instance_id, config)) + .map_err(|error| DbErr::Json(error.to_string())) + }) + .collect::, _>>()?; - Ok(result.rows_affected() > 0) + let mut transaction = self + .db + .begin_with("BEGIN IMMEDIATE") + .await + .map_err(sqlx_db_error)?; + let current_revision = + read_managed_config_revision(&mut transaction, user_id, device_id).await?; + if target_revision.is_some() && current_revision.as_deref() == target_revision { + transaction.commit().await.map_err(sqlx_db_error)?; + return Ok(ManagedConfigApplyResult::AlreadyApplied); + } + if let ManagedConfigExpectedRevision::Exact(expected) = &expected_revision + && current_revision.as_ref() != expected.as_ref() + { + let result = ManagedConfigApplyResult::RevisionConflict { + expected: expected.clone(), + current: current_revision, + }; + transaction.commit().await.map_err(sqlx_db_error)?; + return Ok(result); + } + + let mut existing_sources = HashMap::new(); + match &update { + ManagedConfigUpdate::Full { .. } => { + let rows = sqlx::query_as::<_, (String, String)>( + r#" + SELECT network_instance_id, source + FROM user_running_network_configs + WHERE user_id = ? AND device_id = ? + "#, + ) + .bind(user_id) + .bind(device_id.to_string()) + .fetch_all(&mut *transaction) + .await + .map_err(sqlx_db_error)?; + for (instance_id, source) in rows { + if let Ok(instance_id) = Uuid::parse_str(&instance_id) { + existing_sources.insert(instance_id, source); + } + } + } + ManagedConfigUpdate::Patch { + delete_instance_ids, + .. + } => { + for instance_id in upserts + .iter() + .map(|upsert| upsert.instance_id) + .chain(delete_instance_ids.iter().copied()) + { + if let Some(source) = + read_config_source(&mut transaction, user_id, device_id, instance_id) + .await? + { + existing_sources.insert(instance_id, source); + } + } + } + } + + let strict_ownership = target_revision.is_some(); + if strict_ownership + && let Some(instance_id) = serialized_upserts + .iter() + .map(|(instance_id, _)| *instance_id) + .chain(match &update { + ManagedConfigUpdate::Patch { + delete_instance_ids, + .. + } => delete_instance_ids.iter().copied(), + ManagedConfigUpdate::Full { .. } => [].iter().copied(), + }) + .find(|instance_id| { + existing_sources + .get(instance_id) + .is_some_and(|source| source != ConfigSource::Web.as_str()) + }) + { + transaction.commit().await.map_err(sqlx_db_error)?; + return Ok(ManagedConfigApplyResult::OwnershipConflict { instance_id }); + } + + let desired_ids = serialized_upserts + .iter() + .map(|(instance_id, _)| *instance_id) + .collect::>(); + for (instance_id, network_config) in &serialized_upserts { + if !strict_ownership + && existing_sources + .get(instance_id) + .is_some_and(|source| source != ConfigSource::Web.as_str()) + { + continue; + } + let updated = upsert_network_config( + &mut transaction, + user_id, + device_id, + *instance_id, + network_config, + ConfigSource::Web, + true, + ) + .await?; + if !updated { + transaction.rollback().await.map_err(sqlx_db_error)?; + return Ok(ManagedConfigApplyResult::OwnershipConflict { + instance_id: *instance_id, + }); + } + } + + let delete_instance_ids = match &update { + ManagedConfigUpdate::Full { .. } => existing_sources + .iter() + .filter_map(|(instance_id, source)| { + (source == ConfigSource::Web.as_str() && !desired_ids.contains(instance_id)) + .then_some(*instance_id) + }) + .collect::>(), + ManagedConfigUpdate::Patch { + delete_instance_ids, + .. + } => delete_instance_ids + .iter() + .filter(|instance_id| { + existing_sources + .get(instance_id) + .is_some_and(|source| source == ConfigSource::Web.as_str()) + }) + .copied() + .collect(), + }; + for instance_id in &delete_instance_ids { + sqlx::query( + r#" + DELETE FROM user_running_network_configs + WHERE user_id = ? AND device_id = ? AND network_instance_id = ? + AND source = 'web' + "#, + ) + .bind(user_id) + .bind(device_id.to_string()) + .bind(instance_id.to_string()) + .execute(&mut *transaction) + .await + .map_err(sqlx_db_error)?; + } + + match target_revision { + Some(revision) => { + write_managed_config_revision(&mut transaction, user_id, device_id, revision) + .await?; + } + None => { + clear_managed_config_revision(&mut transaction, user_id, device_id).await?; + } + } + transaction.commit().await.map_err(sqlx_db_error)?; + Ok(ManagedConfigApplyResult::Applied { + deleted_web_instance_ids: delete_instance_ids, + }) } pub async fn delete_web_network_configs( @@ -259,19 +594,32 @@ impl Db { (user_id, device_id): (UserIdInDb, Uuid), network_inst_ids: &[Uuid], ) -> Result<(), DbErr> { - use entity::user_running_network_configs as urnc; - - urnc::Entity::delete_many() - .filter(urnc::Column::UserId.eq(user_id)) - .filter(urnc::Column::DeviceId.eq(device_id.to_string())) - .filter(urnc::Column::Source.eq(ConfigSource::Web.as_str())) - .filter( - urnc::Column::NetworkInstanceId - .is_in(network_inst_ids.iter().map(|id| id.to_string())), + let mut transaction = self + .db + .begin_with("BEGIN IMMEDIATE") + .await + .map_err(sqlx_db_error)?; + let mut deleted = false; + for instance_id in network_inst_ids { + let result = sqlx::query( + r#" + DELETE FROM user_running_network_configs + WHERE user_id = ? AND device_id = ? AND network_instance_id = ? + AND source = 'web' + "#, ) - .exec(self.orm_db()) - .await?; - + .bind(user_id) + .bind(device_id.to_string()) + .bind(instance_id.to_string()) + .execute(&mut *transaction) + .await + .map_err(sqlx_db_error)?; + deleted |= result.rows_affected() > 0; + } + if deleted { + clear_managed_config_revision(&mut transaction, user_id, device_id).await?; + } + transaction.commit().await.map_err(sqlx_db_error)?; Ok(()) } } @@ -285,42 +633,31 @@ impl Storage<(UserIdInDb, Uuid), user_running_network_configs::Model, DbErr> for network_config: NetworkConfig, source: ConfigSource, ) -> Result<(), DbErr> { - let txn = self.orm_db().begin().await?; - - use entity::user_running_network_configs as urnc; - - let on_conflict = OnConflict::columns([ - urnc::Column::UserId, - urnc::Column::DeviceId, - urnc::Column::NetworkInstanceId, - ]) - .update_columns([ - urnc::Column::NetworkConfig, - urnc::Column::Source, - urnc::Column::Disabled, - urnc::Column::UpdateTime, - ]) - .to_owned(); - let insert_m = urnc::ActiveModel { - user_id: sea_orm::Set(user_id), - device_id: sea_orm::Set(device_id.to_string()), - network_instance_id: sea_orm::Set(network_inst_id.to_string()), - network_config: sea_orm::Set( - serde_json::to_string(&network_config).map_err(|e| DbErr::Json(e.to_string()))?, - ), - source: sea_orm::Set(source.as_str().to_string()), - disabled: sea_orm::Set(false), - create_time: sea_orm::Set(chrono::Local::now().fixed_offset()), - update_time: sea_orm::Set(chrono::Local::now().fixed_offset()), - ..Default::default() - }; - urnc::Entity::insert(insert_m) - .on_conflict(on_conflict) - .do_nothing() - .exec(&txn) - .await?; - - txn.commit().await + let network_config = + serde_json::to_string(&network_config).map_err(|e| DbErr::Json(e.to_string()))?; + let mut transaction = self + .db + .begin_with("BEGIN IMMEDIATE") + .await + .map_err(sqlx_db_error)?; + let previous_source = + read_config_source(&mut transaction, user_id, device_id, network_inst_id).await?; + upsert_network_config( + &mut transaction, + user_id, + device_id, + network_inst_id, + &network_config, + source, + false, + ) + .await?; + if source == ConfigSource::Web + || previous_source.as_deref() == Some(ConfigSource::Web.as_str()) + { + clear_managed_config_revision(&mut transaction, user_id, device_id).await?; + } + transaction.commit().await.map_err(sqlx_db_error) } async fn delete_network_configs( @@ -328,18 +665,35 @@ impl Storage<(UserIdInDb, Uuid), user_running_network_configs::Model, DbErr> for (user_id, device_id): (UserIdInDb, Uuid), network_inst_ids: &[Uuid], ) -> Result<(), DbErr> { - use entity::user_running_network_configs as urnc; - - urnc::Entity::delete_many() - .filter(urnc::Column::UserId.eq(user_id)) - .filter(urnc::Column::DeviceId.eq(device_id.to_string())) - .filter( - urnc::Column::NetworkInstanceId - .is_in(network_inst_ids.iter().map(|id| id.to_string())), + let mut transaction = self + .db + .begin_with("BEGIN IMMEDIATE") + .await + .map_err(sqlx_db_error)?; + let mut deleted_web_config = false; + for instance_id in network_inst_ids { + deleted_web_config |= + read_config_source(&mut transaction, user_id, device_id, *instance_id) + .await? + .as_deref() + == Some(ConfigSource::Web.as_str()); + sqlx::query( + r#" + DELETE FROM user_running_network_configs + WHERE user_id = ? AND device_id = ? AND network_instance_id = ? + "#, ) - .exec(self.orm_db()) - .await?; - + .bind(user_id) + .bind(device_id.to_string()) + .bind(instance_id.to_string()) + .execute(&mut *transaction) + .await + .map_err(sqlx_db_error)?; + } + if deleted_web_config { + clear_managed_config_revision(&mut transaction, user_id, device_id).await?; + } + transaction.commit().await.map_err(sqlx_db_error)?; Ok(()) } @@ -349,20 +703,32 @@ impl Storage<(UserIdInDb, Uuid), user_running_network_configs::Model, DbErr> for network_inst_id: Uuid, disabled: bool, ) -> Result<(), DbErr> { - use entity::user_running_network_configs as urnc; - - urnc::Entity::update_many() - .filter(urnc::Column::UserId.eq(user_id)) - .filter(urnc::Column::DeviceId.eq(device_id.to_string())) - .filter(urnc::Column::NetworkInstanceId.eq(network_inst_id.to_string())) - .col_expr(urnc::Column::Disabled, Expr::value(disabled)) - .col_expr( - urnc::Column::UpdateTime, - Expr::value(chrono::Local::now().fixed_offset()), - ) - .exec(self.orm_db()) - .await?; - + let mut transaction = self + .db + .begin_with("BEGIN IMMEDIATE") + .await + .map_err(sqlx_db_error)?; + let source = + read_config_source(&mut transaction, user_id, device_id, network_inst_id).await?; + let result = sqlx::query( + r#" + UPDATE user_running_network_configs + SET disabled = ?, update_time = ? + WHERE user_id = ? AND device_id = ? AND network_instance_id = ? + "#, + ) + .bind(disabled) + .bind(chrono::Local::now().fixed_offset()) + .bind(user_id) + .bind(device_id.to_string()) + .bind(network_inst_id.to_string()) + .execute(&mut *transaction) + .await + .map_err(sqlx_db_error)?; + if result.rows_affected() > 0 && source.as_deref() == Some(ConfigSource::Web.as_str()) { + clear_managed_config_revision(&mut transaction, user_id, device_id).await?; + } + transaction.commit().await.map_err(sqlx_db_error)?; Ok(()) } @@ -600,17 +966,73 @@ mod tests { } #[tokio::test] - async fn test_web_network_config_does_not_replace_user_owned_config() { + async fn web_owned_mutations_invalidate_managed_revision() { let db = Db::memory_db().await; - let user_id = db.auto_create_user("user-web-race").await.unwrap().id; + let user_id = db + .auto_create_user("managed-revision-invalidation") + .await + .unwrap() + .id; let device_id = uuid::Uuid::new_v4(); let inst_id = uuid::Uuid::new_v4(); + db.insert_or_update_user_network_config( + (user_id, device_id), + inst_id, + NetworkConfig { + network_name: Some("managed".to_string()), + ..Default::default() + }, + ConfigSource::Web, + ) + .await + .unwrap(); + + db.set_managed_config_revision((user_id, device_id), "rev-before-disable") + .await + .unwrap(); + db.update_network_config_state((user_id, device_id), inst_id, true) + .await + .unwrap(); + assert!( + db.get_managed_config_revision((user_id, device_id)) + .await + .unwrap() + .is_none() + ); + + db.set_managed_config_revision((user_id, device_id), "rev-before-delete") + .await + .unwrap(); + db.delete_network_configs((user_id, device_id), &[inst_id]) + .await + .unwrap(); + assert!( + db.get_managed_config_revision((user_id, device_id)) + .await + .unwrap() + .is_none() + ); + } + + #[tokio::test] + async fn user_owned_mutation_preserves_managed_revision() { + let db = Db::memory_db().await; + let user_id = db + .auto_create_user("user-revision-preserved") + .await + .unwrap() + .id; + let device_id = uuid::Uuid::new_v4(); + let inst_id = uuid::Uuid::new_v4(); + db.set_managed_config_revision((user_id, device_id), "rev-user") + .await + .unwrap(); db.insert_or_update_user_network_config( (user_id, device_id), inst_id, NetworkConfig { - network_name: Some("user-owned".to_string()), + network_name: Some("user".to_string()), ..Default::default() }, ConfigSource::User, @@ -618,26 +1040,12 @@ mod tests { .await .unwrap(); - let updated = db - .insert_or_update_web_network_config( - (user_id, device_id), - inst_id, - NetworkConfig { - network_name: Some("web-owned".to_string()), - ..Default::default() - }, - ) - .await - .unwrap(); - - assert!(!updated); - let saved = db - .get_network_config((user_id, device_id), &inst_id.to_string()) - .await - .unwrap() - .unwrap(); - assert_eq!(saved.get_network_config_source(), ConfigSource::User); - let saved_config = saved.get_network_config().unwrap(); - assert_eq!(saved_config.network_name.as_deref(), Some("user-owned")); + assert_eq!( + db.get_managed_config_revision((user_id, device_id)) + .await + .unwrap() + .as_deref(), + Some("rev-user") + ); } } diff --git a/easytier-web/src/restful/mod.rs b/easytier-web/src/restful/mod.rs index 71791405..13307965 100644 --- a/easytier-web/src/restful/mod.rs +++ b/easytier-web/src/restful/mod.rs @@ -86,6 +86,10 @@ struct ParseConfigResponse { #[derive(Debug, serde::Deserialize, serde::Serialize)] pub struct Error { message: String, + #[serde(skip_serializing_if = "Option::is_none")] + code: Option, + #[serde(skip_serializing_if = "Option::is_none")] + current_config_revision: Option, } type RpcError = rpc_types::error::Error; type HttpHandleError = (StatusCode, Json); @@ -93,6 +97,8 @@ type HttpHandleError = (StatusCode, Json); pub fn other_error(error_message: T) -> Error { Error { message: error_message.to_string(), + code: None, + current_config_revision: None, } } diff --git a/easytier-web/src/restful/network.rs b/easytier-web/src/restful/network.rs index bd7fec0b..aa516537 100644 --- a/easytier-web/src/restful/network.rs +++ b/easytier-web/src/restful/network.rs @@ -1,6 +1,6 @@ -use axum::extract::Path; +use axum::extract::{DefaultBodyLimit, Path}; use axum::http::StatusCode; -use axum::routing::{delete, post}; +use axum::routing::{delete, post, put}; use axum::{Json, Router, extract::State, routing::get}; use axum_login::AuthUser; use easytier::common::config::{ @@ -21,6 +21,8 @@ use super::{ AppState, AppStateInner, Error, HttpHandleError, RpcError, convert_db_error, other_error, }; +const MAX_MANAGED_CONFIG_REQUEST_BODY_SIZE: usize = 32 * 1024 * 1024; + fn convert_rpc_error(e: RpcError) -> (StatusCode, Json) { let status_code = match &e { RpcError::ExecutionError(_) => StatusCode::BAD_REQUEST, @@ -29,6 +31,8 @@ fn convert_rpc_error(e: RpcError) -> (StatusCode, Json) { }; let error = Error { message: format!("{:?}", e), + code: None, + current_config_revision: None, }; (status_code, Json(error)) } @@ -98,6 +102,14 @@ struct ReconcileManagedNetworkConfigsJsonReq { expected_config_revision: Option, } +#[derive(Debug, serde::Deserialize, serde::Serialize)] +struct PatchManagedNetworkConfigsJsonReq { + upserts: Vec, + delete_instance_ids: Vec, + config_revision: String, + expected_config_revision: String, +} + #[derive(Debug, serde::Deserialize, serde::Serialize)] struct ListMachineItem { client_url: Option, @@ -113,6 +125,37 @@ struct ListMachineJsonResp { pub struct NetworkApi; impl NetworkApi { + fn convert_managed_config_error(error: anyhow::Error) -> HttpHandleError { + let (status, code, current_config_revision) = + match error.downcast_ref::() { + Some(crate::client_manager::ManagedConfigError::Invalid(_)) => { + (StatusCode::BAD_REQUEST, None, None) + } + Some(crate::client_manager::ManagedConfigError::RevisionConflict { + current, + .. + }) => ( + StatusCode::CONFLICT, + Some("managed_config_revision_conflict".to_string()), + current.clone(), + ), + Some(crate::client_manager::ManagedConfigError::OwnershipConflict { .. }) => ( + StatusCode::CONFLICT, + Some("managed_config_ownership_conflict".to_string()), + None, + ), + None => (StatusCode::INTERNAL_SERVER_ERROR, None, None), + }; + ( + status, + Json(Error { + message: error.to_string(), + code, + current_config_revision, + }), + ) + } + fn get_user_id(auth_session: &AuthSession) -> Result)> { let Some(user_id) = auth_session.user.as_ref().map(|x| x.id()) else { return Err(( @@ -145,15 +188,22 @@ impl NetworkApi { Path(machine_id): Path, Json(payload): Json, ) -> Result, HttpHandleError> { + let user_id = Self::get_user_id(&auth_session)?; client_mgr + .invalidate_applied_config_revision(user_id, machine_id) + .await; + let result = client_mgr .handle_run_network_instance_with_source( - (Self::get_user_id(&auth_session)?, machine_id), + (user_id, machine_id), payload.config, payload.save, RuntimeConfigSource::Web, ) - .await - .map_err(convert_error)?; + .await; + client_mgr + .invalidate_applied_config_revision(user_id, machine_id) + .await; + result.map_err(convert_error)?; Ok(Void::default().into()) } @@ -205,13 +255,18 @@ impl NetworkApi { State(client_mgr): AppState, Path((machine_id, inst_id)): Path<(uuid::Uuid, uuid::Uuid)>, ) -> Result<(), HttpHandleError> { + let user_id = Self::get_user_id(&auth_session)?; client_mgr - .handle_remove_network_instances( - (Self::get_user_id(&auth_session)?, machine_id), - vec![inst_id], - ) - .await - .map_err(convert_error) + .invalidate_applied_config_revision(user_id, machine_id) + .await; + let result = client_mgr + .handle_remove_network_instances((user_id, machine_id), vec![inst_id]) + .await; + client_mgr + .invalidate_applied_config_revision(user_id, machine_id) + .await; + result.map_err(convert_error)?; + Ok(()) } async fn handle_list_machines( @@ -251,14 +306,18 @@ impl NetworkApi { )); }; + let user_id = Self::get_user_id(&auth_session)?; client_mgr - .handle_update_network_state( - (auth_session.user.unwrap().id(), machine_id), - inst_id, - payload.disabled, - ) - .await - .map_err(convert_error) + .invalidate_applied_config_revision(user_id, machine_id) + .await; + let result = client_mgr + .handle_update_network_state((user_id, machine_id), inst_id, payload.disabled) + .await; + client_mgr + .invalidate_applied_config_revision(user_id, machine_id) + .await; + result.map_err(convert_error)?; + Ok(()) } async fn handle_get_network_metas( @@ -290,15 +349,23 @@ impl NetworkApi { other_error("Instance ID mismatch".to_string()).into(), )); } + let user_id = Self::get_user_id(&auth_session)?; client_mgr + .invalidate_applied_config_revision(user_id, machine_id) + .await; + let result = client_mgr .handle_save_network_config_with_source( - (Self::get_user_id(&auth_session)?, machine_id), + (user_id, machine_id), inst_id, payload.config, RuntimeConfigSource::Web, ) - .await - .map_err(convert_error) + .await; + client_mgr + .invalidate_applied_config_revision(user_id, machine_id) + .await; + result.map_err(convert_error)?; + Ok(()) } async fn handle_get_network_config( @@ -325,14 +392,20 @@ impl NetworkApi { .and_then(config_source_from_rpc) .unwrap_or(RuntimeConfigSource::Web); client_mgr + .invalidate_applied_config_revision(user_id, machine_id) + .await; + let result = client_mgr .handle_run_network_instance_with_source( (user_id, machine_id), payload.config, payload.save, source, ) - .await - .map_err(convert_error)?; + .await; + client_mgr + .invalidate_applied_config_revision(user_id, machine_id) + .await; + result.map_err(convert_error)?; Ok(Void::default().into()) } @@ -341,16 +414,23 @@ impl NetworkApi { Path((user_id, machine_id, inst_id)): Path<(UserIdInDb, uuid::Uuid, uuid::Uuid)>, ) -> Result<(), HttpHandleError> { client_mgr + .invalidate_applied_config_revision(user_id, machine_id) + .await; + let result = client_mgr .handle_remove_network_instances((user_id, machine_id), vec![inst_id]) - .await - .map_err(convert_error) + .await; + client_mgr + .invalidate_applied_config_revision(user_id, machine_id) + .await; + result.map_err(convert_error)?; + Ok(()) } async fn handle_reconcile_managed_network_configs_internal( State(client_mgr): AppState, Path((user_id, machine_id)): Path<(UserIdInDb, uuid::Uuid)>, Json(payload): Json, - ) -> Result, HttpHandleError> { + ) -> Result { let desired = payload .managed_network_configs .into_iter() @@ -368,15 +448,35 @@ impl NetworkApi { payload.expected_config_revision, ) .await - .map_err(|err| { - let status = if crate::client_manager::is_managed_config_revision_conflict(&err) { - StatusCode::CONFLICT - } else { - StatusCode::INTERNAL_SERVER_ERROR - }; - (status, other_error(err.to_string()).into()) - })?; - Ok(Void::default().into()) + .map_err(Self::convert_managed_config_error)?; + Ok(StatusCode::NO_CONTENT) + } + + async fn handle_patch_managed_network_configs_internal( + State(client_mgr): AppState, + Path((user_id, machine_id)): Path<(UserIdInDb, uuid::Uuid)>, + Json(payload): Json, + ) -> Result { + let upserts = payload + .upserts + .into_iter() + .map(|item| crate::webhook::ManagedNetworkConfig { + instance_id: item.instance_id.to_string(), + network_config: item.network_config, + }) + .collect(); + client_mgr + .patch_managed_network_configs( + user_id, + machine_id, + upserts, + payload.delete_instance_ids, + payload.config_revision, + payload.expected_config_revision, + ) + .await + .map_err(Self::convert_managed_config_error)?; + Ok(StatusCode::NO_CONTENT) } async fn handle_list_network_instance_ids_internal( @@ -406,8 +506,10 @@ impl NetworkApi { Router::new() .route( "/api/internal/users/:user-id/machines/:machine-id/networks", - post(Self::handle_run_network_instance_internal) - .put(Self::handle_reconcile_managed_network_configs_internal) + put(Self::handle_reconcile_managed_network_configs_internal) + .patch(Self::handle_patch_managed_network_configs_internal) + .layer(DefaultBodyLimit::max(MAX_MANAGED_CONFIG_REQUEST_BODY_SIZE)) + .post(Self::handle_run_network_instance_internal) .get(Self::handle_list_network_instance_ids_internal), ) .route( @@ -453,3 +555,41 @@ impl NetworkApi { ) } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn revision_conflict_response_exposes_machine_readable_current_revision() { + let error = crate::client_manager::ManagedConfigError::RevisionConflict { + expected: Some("rev-1".to_string()), + current: Some("rev-2".to_string()), + }; + + let (status, Json(body)) = NetworkApi::convert_managed_config_error(error.into()); + + assert_eq!(status, StatusCode::CONFLICT); + assert_eq!( + body.code.as_deref(), + Some("managed_config_revision_conflict") + ); + assert_eq!(body.current_config_revision.as_deref(), Some("rev-2")); + } + + #[test] + fn ownership_conflict_is_distinct_from_revision_conflict() { + let error = crate::client_manager::ManagedConfigError::OwnershipConflict { + instance_id: uuid::Uuid::new_v4(), + }; + + let (status, Json(body)) = NetworkApi::convert_managed_config_error(error.into()); + + assert_eq!(status, StatusCode::CONFLICT); + assert_eq!( + body.code.as_deref(), + Some("managed_config_ownership_conflict") + ); + assert_eq!(body.current_config_revision, None); + } +} diff --git a/easytier-web/src/restful/rpc.rs b/easytier-web/src/restful/rpc.rs index f8b13cbc..dbe7a613 100644 --- a/easytier-web/src/restful/rpc.rs +++ b/easytier-web/src/restful/rpc.rs @@ -39,6 +39,13 @@ async fn handle_proxy_rpc_by_session( scope, } = req; + let mutates_runtime_config = proxy_rpc_mutates_runtime_config(&service_name, &method_name); + if mutates_runtime_config { + session + .invalidate_runtime_config_for_direct_mutation() + .await; + } + let resp = match service_name.as_str() { "api.manage.WebClientService" => match_service!( easytier::proto::api::manage::WebClientServiceClientFactory, @@ -134,6 +141,12 @@ async fn handle_proxy_rpc_by_session( } }; + if mutates_runtime_config { + session + .invalidate_runtime_config_for_direct_mutation() + .await; + } + match resp { Ok(v) => Ok(Json(v)), Err(e) => Err(( @@ -143,6 +156,32 @@ async fn handle_proxy_rpc_by_session( } } +fn proxy_rpc_mutates_runtime_config(service_name: &str, method_name: &str) -> bool { + matches!( + (service_name, method_name), + ( + "api.manage.WebClientService", + "run_network_instance" + | "RunNetworkInstance" + | "retain_network_instance" + | "RetainNetworkInstance" + | "delete_network_instance" + | "DeleteNetworkInstance" + ) | ( + "api.config.ConfigRpcService", + "patch_config" | "PatchConfig" + ) | ( + "api.instance.CredentialManageRpcService", + "generate_credential" + | "GenerateCredential" + | "revoke_credential" + | "RevokeCredential" + | "upsert_credential" + | "UpsertCredential" + ) + ) +} + pub async fn handle_proxy_rpc( auth_session: super::users::AuthSession, State(client_mgr): AppState, @@ -192,3 +231,50 @@ pub fn router_internal() -> Router { post(handle_proxy_rpc_internal), ) } + +#[cfg(test)] +mod tests { + use super::proxy_rpc_mutates_runtime_config; + + #[test] + fn runtime_config_mutation_detection_covers_proxy_rpc_aliases() { + for (service, method) in [ + ("api.manage.WebClientService", "run_network_instance"), + ("api.manage.WebClientService", "RetainNetworkInstance"), + ("api.manage.WebClientService", "delete_network_instance"), + ("api.config.ConfigRpcService", "PatchConfig"), + ( + "api.instance.CredentialManageRpcService", + "generate_credential", + ), + ( + "api.instance.CredentialManageRpcService", + "RevokeCredential", + ), + ( + "api.instance.CredentialManageRpcService", + "upsert_credential", + ), + ] { + assert!( + proxy_rpc_mutates_runtime_config(service, method), + "{service}/{method} must invalidate the managed revision fence" + ); + } + + for (service, method) in [ + ("api.manage.WebClientService", "list_network_instance"), + ("api.config.ConfigRpcService", "get_config"), + ( + "api.instance.CredentialManageRpcService", + "list_credentials", + ), + ("api.instance.StatsRpcService", "get_stats"), + ] { + assert!( + !proxy_rpc_mutates_runtime_config(service, method), + "{service}/{method} must remain read-only" + ); + } + } +} diff --git a/easytier/src/instance/runtime_host.rs b/easytier/src/instance/runtime_host.rs index 3857730c..93c0d7fc 100644 --- a/easytier/src/instance/runtime_host.rs +++ b/easytier/src/instance/runtime_host.rs @@ -108,7 +108,7 @@ impl NativeInstanceRuntimeHost { &config.peer, )); } - if patch.disable_relay_data.is_some() { + if patch.disable_relay_data.is_some() || patch.prefer_peer_relay.is_some() { self.global_ctx.set_flags(config.peer.flags.clone()); } } @@ -173,12 +173,14 @@ mod tests { assert_eq!(global_ctx.get_ipv4(), Some("10.20.0.1/24".parse().unwrap())); assert_eq!(global_ctx.get_ipv6(), Some("fd00::1/64".parse().unwrap())); assert!(!global_ctx.get_flags().disable_relay_data); + assert!(!global_ctx.get_flags().prefer_peer_relay); config.set_hostname(Some("after".to_owned())); config.set_ipv4(Some("10.20.0.2/24".parse().unwrap())); config.set_ipv6(Some("fd00::2/64".parse().unwrap())); let mut flags = config.get_flags(); flags.disable_relay_data = true; + flags.prefer_peer_relay = true; config.set_flags(flags); runtime_host.synchronize_config( &crate::proto::api::config::InstanceConfigPatch { @@ -186,6 +188,7 @@ mod tests { ipv4: Some("10.99.0.1/24".parse::().unwrap().into()), ipv6: Some("fd99::1/64".parse::().unwrap().into()), disable_relay_data: Some(false), + prefer_peer_relay: Some(false), ..Default::default() }, &runtime_config(&config), @@ -195,6 +198,7 @@ mod tests { assert_eq!(global_ctx.get_ipv4(), Some("10.20.0.2/24".parse().unwrap())); assert_eq!(global_ctx.get_ipv6(), Some("fd00::2/64".parse().unwrap())); assert!(global_ctx.get_flags().disable_relay_data); + assert!(global_ctx.get_flags().prefer_peer_relay); } #[cfg(feature = "web-client")] diff --git a/easytier/src/tests/credential_tests.rs b/easytier/src/tests/credential_tests.rs index 4e9cef29..ad6c8d44 100644 --- a/easytier/src/tests/credential_tests.rs +++ b/easytier/src/tests/credential_tests.rs @@ -5,10 +5,13 @@ //! - Credential nodes use X25519 keypairs to authenticate without network_secret //! - Credentials can be revoked and propagate across the network -use std::{sync::Arc, time::Duration}; +use std::time::Duration; -use easytier_core::peers::credential_manager::CredentialCreateOptions; -use easytier_core::process_runtime::CoreProcessRuntime; +use easytier_core::{ + foundation::stats::{LabelSet, LabelType, MetricName}, + peers::credential_manager::CredentialCreateOptions, + process_runtime::CoreProcessRuntime, +}; use crate::{ common::{ @@ -16,6 +19,7 @@ use crate::{ global_ctx::GlobalCtxEvent, }, instance::test_instance::TestInstance as Instance, + proto::api::config::InstanceConfigPatch, tests::three_node::{generate_secure_mode_config, generate_secure_mode_config_with_key}, tunnel::common::tests::wait_for_condition, }; @@ -72,13 +76,127 @@ async fn generate_credential_with_options( (generated.credential_id, generated.secret) } -async fn set_avoid_relay_data(inst: &Instance, avoid_relay_data: bool) { - let mut config = crate::instance::config::test_runtime_instance_config(&inst.get_global_ctx()); - Arc::make_mut(&mut config.peer).avoid_relay_data_preference = avoid_relay_data; - inst.get_core_instance() - .update_runtime_config(config) +fn disable_p2p(config: &TomlConfigLoader) { + let mut flags = config.get_flags(); + flags.disable_p2p = true; + config.set_flags(flags); +} + +async fn set_prefer_peer_relay(inst: &Instance, prefer_peer_relay: bool) { + inst.get_config_patcher() + .apply_patch(InstanceConfigPatch { + prefer_peer_relay: Some(prefer_peer_relay), + ..Default::default() + }) .await .unwrap(); + assert_eq!( + inst.get_global_ctx().get_flags().prefer_peer_relay, + prefer_peer_relay + ); +} + +fn forwarded_data_packets(inst: &Instance) -> u64 { + let labels = LabelSet::new().with_label_type(LabelType::NetworkName( + inst.get_global_ctx() + .get_network_identity() + .network_name + .clone(), + )); + inst.get_core_instance() + .metric_snapshots() + .into_iter() + .find(|metric| { + metric.name == MetricName::TrafficPacketsForwarded && metric.labels == labels + }) + .map_or(0, |metric| metric.value) +} + +async fn assert_ping_forwarded_by( + src_ns: &str, + dst_ip: &str, + expected_forwarder: &Instance, + forbidden_forwarders: &[&Instance], +) { + wait_ping_reachability(src_ns, dst_ip, true, Duration::from_secs(10)).await; + let expected_before = forwarded_data_packets(expected_forwarder); + let forbidden_before: Vec<_> = forbidden_forwarders + .iter() + .map(|inst| forwarded_data_packets(inst)) + .collect(); + for _ in 0..3 { + assert!(ping_test(src_ns, dst_ip, None).await); + } + wait_for_condition( + || async { forwarded_data_packets(expected_forwarder) > expected_before }, + Duration::from_secs(5), + ) + .await; + for (forbidden_forwarder, forwarded_before) in forbidden_forwarders.iter().zip(forbidden_before) + { + assert_eq!( + forwarded_data_packets(forbidden_forwarder), + forwarded_before, + "data packets unexpectedly traversed a forbidden forwarder" + ); + } +} + +async fn wait_route_next_hop( + inst: &Instance, + peer_id: u32, + next_hop_peer_id: u32, + timeout: Duration, + label: &str, +) { + wait_for_condition( + || async { + let routes = inst.get_core_instance().route_snapshots().await; + let next_hop = routes + .iter() + .find(|route| route.peer_id == peer_id) + .map(|route| route.next_hop_peer_id); + println!( + "{label}: next_hop={next_hop:?}, target={peer_id}, expected={next_hop_peer_id}" + ); + next_hop == Some(next_hop_peer_id) + }, + timeout, + ) + .await; +} + +async fn disconnect_connector(inst: &Instance, url: &url::Url, peer_id: u32) { + assert!(inst.get_core_instance().remove_connector(url)); + let conn_ids = inst + .get_core_instance() + .peer_snapshots() + .await + .into_iter() + .find(|peer| peer.peer_id == peer_id) + .map(|peer| peer.directly_connected_conns) + .unwrap_or_default(); + assert!( + !conn_ids.is_empty(), + "direct peer has no connection to close" + ); + for conn_id in conn_ids { + let _ = inst + .get_core_instance() + .close_peer_conn(peer_id, &conn_id) + .await; + } + wait_for_condition( + || async { + !inst + .get_core_instance() + .connected_peers() + .await + .contains(&peer_id) + }, + Duration::from_secs(10), + ) + .await; } /// Prepare network namespaces for credential tests @@ -928,7 +1046,7 @@ async fn credential_rejects_incorrect_admin_pin() { drop_insts(vec![admin_inst, cred_inst]).await; } -/// Test 5-6: Credential relay capability with allow_relay parameter +/// Test 5-6: Prefer Peer Relay with one admin and an allow_relay credential /// Topology: Admin ← Credential_A, Admin ← Credential_B, Admin ← Credential_C(listener, allow_relay) /// Verifies routing behavior based on allow_relay flag: /// - allow_relay=true: A→B route goes through C (cost 2 via C) @@ -938,15 +1056,13 @@ async fn credential_rejects_incorrect_admin_pin() { #[case(false)] #[tokio::test] #[serial_test::serial] -async fn credential_relay_capability(#[case] allow_relay: bool) { +async fn prefer_peer_relay_single_admin(#[case] allow_relay: bool) { prepare_credential_network(); let process_runtime = CoreProcessRuntime::new(); // Create admin node let admin_config = create_admin_config("admin", Some("ns_adm"), "10.144.144.1", "fd00::1/64"); let mut admin_inst = Instance::new_with_process_runtime(admin_config, process_runtime.clone()); - // if cred c allow relay, we set admin inst avoid relay (if other same-cost path available, admin will not relay data) - set_avoid_relay_data(&admin_inst, allow_relay).await; admin_inst.run().await.unwrap(); let admin_peer_id = admin_inst.peer_id(); @@ -1002,6 +1118,7 @@ async fn credential_relay_capability(#[case] allow_relay: bool) { .clone(), )); config.set_secure_mode(Some(generate_secure_mode_config_with_key(&private))); + disable_p2p(&config); config }; let mut cred_a_inst = @@ -1030,6 +1147,7 @@ async fn credential_relay_capability(#[case] allow_relay: bool) { .clone(), )); config.set_secure_mode(Some(generate_secure_mode_config_with_key(&private))); + disable_p2p(&config); config }; let mut cred_b_inst = @@ -1060,6 +1178,7 @@ async fn credential_relay_capability(#[case] allow_relay: bool) { .clone(), )); config.set_secure_mode(Some(generate_secure_mode_config_with_key(&private))); + disable_p2p(&config); config }; let mut cred_c_inst = @@ -1075,9 +1194,8 @@ async fn credential_relay_capability(#[case] allow_relay: bool) { cred_b_inst.add_connector_url("tcp://10.1.1.1:11010".parse().unwrap()); cred_c_inst.add_connector_url("tcp://10.1.1.1:11010".parse().unwrap()); - // A and B also connect to C (simulating P2P discovery and connection) - // C is on ns_c3 with IP 10.1.1.4, listener on port 11020 - cred_a_inst.add_connector_url("tcp://10.1.1.4:11020".parse().unwrap()); + // B connects to C first. A remains Admin-only so changing the Admin's + // advertised edge produces an observable 2 -> 3 -> 2 transition. cred_b_inst.add_connector_url("tcp://10.1.1.4:11020".parse().unwrap()); // print all peer ids println!("Admin peer id: {:?}", admin_peer_id); @@ -1099,108 +1217,478 @@ async fn credential_relay_capability(#[case] allow_relay: bool) { ) .await; - // Wait for P2P connections to establish + // Wait for B and C to establish the candidate relay edge. wait_for_condition( || async { - let peers_a = cred_a_inst.get_core_instance().connected_peers().await; let peers_b = cred_b_inst.get_core_instance().connected_peers().await; let peers_c = cred_c_inst.get_core_instance().connected_peers().await; - let a_connected_c = peers_a.contains(&cred_c_peer_id); let b_connected_c = peers_b.contains(&cred_c_peer_id); - let c_connected_a = peers_c.contains(&cred_a_peer_id); let c_connected_b = peers_c.contains(&cred_b_peer_id); println!( - "P2P: A->C={}, B->C={}, C->A={}, C->B={}, allow_relay={}", - a_connected_c, b_connected_c, c_connected_a, c_connected_b, allow_relay + "P2P: B->C={}, C->B={}, allow_relay={}", + b_connected_c, c_connected_b, allow_relay ); - - if allow_relay { - a_connected_c && b_connected_c && c_connected_a && c_connected_b - } else { - a_connected_c && b_connected_c - } + b_connected_c && c_connected_b }, Duration::from_secs(30), ) .await; - // Wait for routes to propagate + wait_route_cost( + &cred_a_inst, + cred_b_peer_id, + 2, + Duration::from_secs(15), + "preference disabled on converged single-admin topology", + ) + .await; + + set_prefer_peer_relay(&admin_inst, true).await; + wait_route_cost( + &cred_a_inst, + cred_b_peer_id, + if allow_relay { 3 } else { 2 }, + Duration::from_secs(20), + "enabling preference projects only authorized relay coverage", + ) + .await; + + if allow_relay { + set_prefer_peer_relay(&admin_inst, false).await; + wait_route_cost( + &cred_a_inst, + cred_b_peer_id, + 2, + Duration::from_secs(20), + "disabling preference restores the admin edge", + ) + .await; + set_prefer_peer_relay(&admin_inst, true).await; + wait_route_cost( + &cred_a_inst, + cred_b_peer_id, + 3, + Duration::from_secs(20), + "re-enabling preference removes the admin edge again", + ) + .await; + + cred_a_inst.add_connector_url("tcp://10.1.1.4:11020".parse().unwrap()); + wait_direct_peer( + &cred_a_inst, + cred_c_peer_id, + Duration::from_secs(10), + "credential A -> relay C", + ) + .await; + } + + let expected_next_hop = if allow_relay { + cred_c_peer_id + } else { + admin_peer_id + }; + wait_route_next_hop( + &cred_a_inst, + cred_b_peer_id, + expected_next_hop, + Duration::from_secs(20), + "single-admin data path", + ) + .await; + wait_route_next_hop( + &cred_b_inst, + cred_a_peer_id, + expected_next_hop, + Duration::from_secs(20), + "single-admin return data path", + ) + .await; + + wait_route_cost( + &admin_inst, + cred_b_peer_id, + 1, + Duration::from_secs(10), + "admin retains its direct credential route", + ) + .await; + + let (expected_forwarder, forbidden_forwarder) = if allow_relay { + (&cred_c_inst, &admin_inst) + } else { + (&admin_inst, &cred_c_inst) + }; + assert_ping_forwarded_by( + "ns_c1", + "10.144.144.3", + expected_forwarder, + &[forbidden_forwarder], + ) + .await; + + // Cleanup + drop_insts(vec![admin_inst, cred_a_inst, cred_b_inst, cred_c_inst]).await; +} + +/// Both admins must project coverage from a credential relay even though the +/// credential only maintains one OSPF initiator session. The other admin gets +/// the relay's source row through admin-to-admin propagation. After projection +/// is proven, the source connects to the credential relay and sends data +/// through it. Routing must keep working after one admin stops. +/// +/// Initially source, target, and relay each connect to both admins, while the +/// target also connects to the relay. After both admins project the relay's +/// coverage, the source connects directly to the relay for the data-path probe. +/// +/// The source connects to both admins. Therefore its route to the target only +/// grows from two to three hops when both admins suppress their direct target +/// edge. Each admin must still retain its own direct fallback route. +#[tokio::test] +#[serial_test::serial] +async fn prefer_peer_relay_uses_forwarded_coverage_across_two_admins() { + prepare_credential_network(); + let process_runtime = CoreProcessRuntime::new(); + + let admin_a_config = + create_admin_config("admin_a", Some("ns_adm"), "10.144.144.1", "fd00::1/64"); + let mut admin_a_inst = + Instance::new_with_process_runtime(admin_a_config, process_runtime.clone()); + admin_a_inst.run().await.unwrap(); + + let admin_b_config = + create_admin_config("admin_b", Some("ns_c3"), "10.144.144.4", "fd00::4/64"); + let mut admin_b_inst = + Instance::new_with_process_runtime(admin_b_config, process_runtime.clone()); + admin_b_inst.run().await.unwrap(); + + admin_a_inst.add_connector_url("tcp://10.1.1.4:11010".parse().unwrap()); + let admin_a_peer_id = admin_a_inst.peer_id(); + let admin_b_peer_id = admin_b_inst.peer_id(); + wait_direct_peer( + &admin_a_inst, + admin_b_peer_id, + Duration::from_secs(10), + "admin_a -> admin_b", + ) + .await; + + let (_, relay_secret) = generate_credential( + &admin_a_inst, + vec![], + true, + vec![], + Duration::from_secs(3600), + ) + .await; + let (_, target_secret) = generate_credential( + &admin_a_inst, + vec![], + false, + vec![], + Duration::from_secs(3600), + ) + .await; + let (_, source_secret) = generate_credential( + &admin_a_inst, + vec![], + false, + vec![], + Duration::from_secs(3600), + ) + .await; + admin_a_inst + .get_global_ctx() + .issue_event(GlobalCtxEvent::CredentialChanged); + + let network_name = admin_a_inst + .get_global_ctx() + .get_network_identity() + .network_name + .clone(); + let relay_config = create_credential_config_from_secret( + network_name.clone(), + &relay_secret, + "relay", + Some("ns_c1"), + "10.144.144.2", + "fd00::2/64", + ); + relay_config.set_listeners(vec!["tcp://0.0.0.0:11020".parse().unwrap()]); + disable_p2p(&relay_config); + + let target_config = create_credential_config_from_secret( + network_name.clone(), + &target_secret, + "target", + Some("ns_c2"), + "10.144.144.3", + "fd00::3/64", + ); + disable_p2p(&target_config); + + let source_config = create_credential_config_from_secret( + network_name, + &source_secret, + "source", + Some("ns_c4"), + "10.144.144.5", + "fd00::5/64", + ); + disable_p2p(&source_config); + + let mut relay_inst = Instance::new_with_process_runtime(relay_config, process_runtime.clone()); + let mut target_inst = + Instance::new_with_process_runtime(target_config, process_runtime.clone()); + let mut source_inst = + Instance::new_with_process_runtime(source_config, process_runtime.clone()); + relay_inst.run().await.unwrap(); + target_inst.run().await.unwrap(); + source_inst.run().await.unwrap(); + + let relay_listener_url: url::Url = "tcp://10.1.1.2:11020".parse().unwrap(); + relay_inst.add_connector_url("tcp://10.1.1.1:11010".parse().unwrap()); + relay_inst.add_connector_url("tcp://10.1.1.4:11010".parse().unwrap()); + target_inst.add_connector_url("tcp://10.1.1.1:11010".parse().unwrap()); + target_inst.add_connector_url("tcp://10.1.1.4:11010".parse().unwrap()); + target_inst.add_connector_url(relay_listener_url.clone()); + source_inst.add_connector_url("tcp://10.1.1.1:11010".parse().unwrap()); + source_inst.add_connector_url("tcp://10.1.1.4:11010".parse().unwrap()); + + let relay_peer_id = relay_inst.peer_id(); + let target_peer_id = target_inst.peer_id(); + let source_peer_id = source_inst.peer_id(); + for (admin, label) in [(&admin_a_inst, "admin_a"), (&admin_b_inst, "admin_b")] { + wait_direct_peer( + admin, + relay_peer_id, + Duration::from_secs(30), + &format!("{label} -> relay"), + ) + .await; + wait_direct_peer( + admin, + target_peer_id, + Duration::from_secs(30), + &format!("{label} -> target"), + ) + .await; + wait_direct_peer( + admin, + source_peer_id, + Duration::from_secs(30), + &format!("{label} -> source"), + ) + .await; + } + wait_direct_peer( + &relay_inst, + admin_a_peer_id, + Duration::from_secs(10), + "relay -> admin_a", + ) + .await; + wait_direct_peer( + &relay_inst, + admin_b_peer_id, + Duration::from_secs(10), + "relay -> admin_b", + ) + .await; + wait_direct_peer( + &relay_inst, + target_peer_id, + Duration::from_secs(10), + "relay -> target", + ) + .await; + + wait_route_cost( + &source_inst, + target_peer_id, + 2, + Duration::from_secs(30), + "source uses an admin direct target edge", + ) + .await; + wait_ping_reachability("ns_c4", "10.144.144.3", true, Duration::from_secs(10)).await; + + for admin in [&admin_a_inst, &admin_b_inst] { + set_prefer_peer_relay(admin, true).await; + } + + wait_route_cost( + &source_inst, + target_peer_id, + 3, + Duration::from_secs(30), + "both admins use credential relay coverage", + ) + .await; + wait_route_cost( + &admin_a_inst, + target_peer_id, + 1, + Duration::from_secs(10), + "admin_a retains direct fallback", + ) + .await; + wait_route_cost( + &admin_b_inst, + target_peer_id, + 1, + Duration::from_secs(10), + "admin_b retains direct fallback", + ) + .await; + + source_inst.add_connector_url(relay_listener_url.clone()); + wait_direct_peer( + &source_inst, + relay_peer_id, + Duration::from_secs(10), + "source -> relay before admin shutdown", + ) + .await; + wait_route_next_hop( + &source_inst, + target_peer_id, + relay_peer_id, + Duration::from_secs(20), + "dual-admin data path before one admin stops", + ) + .await; + wait_route_next_hop( + &target_inst, + source_peer_id, + relay_peer_id, + Duration::from_secs(20), + "dual-admin return data path before one admin stops", + ) + .await; + assert_ping_forwarded_by( + "ns_c4", + "10.144.144.3", + &relay_inst, + &[&admin_a_inst, &admin_b_inst], + ) + .await; + + // Equivalent admin candidates are ordered by peer id. Stop the first + // credential OSPF initiator so every credential must use the survivor. + let (stopped_admin, surviving_admin) = if admin_a_peer_id < admin_b_peer_id { + (admin_a_inst, admin_b_inst) + } else { + (admin_b_inst, admin_a_inst) + }; + let stopped_admin_peer_id = stopped_admin.peer_id(); + let surviving_admin_peer_id = surviving_admin.peer_id(); + drop_insts(vec![stopped_admin]).await; + wait_for_condition( || async { - let routes_a = cred_a_inst.get_core_instance().route_snapshots().await; - let a_sees_b = routes_a.iter().any(|r| r.peer_id == cred_b_peer_id); - let cost_a_to_b = routes_a - .iter() - .find(|r| r.peer_id == cred_b_peer_id) - .map(|r| r.cost); - - println!("Routes: a_sees_b={} (cost={:?})", a_sees_b, cost_a_to_b); - a_sees_b + let peers = source_inst.get_core_instance().connected_peers().await; + !peers.contains(&stopped_admin_peer_id) && peers.contains(&surviving_admin_peer_id) }, Duration::from_secs(15), ) .await; - - wait_for_condition( - || async { - let next_hop_a_to_b = cred_a_inst - .get_core_instance() - .route_snapshots() - .await - .into_iter() - .find(|route| route.peer_id == cred_b_peer_id) - .and_then(|route| route.next_hop_peer_id_latency_first); - println!( - "Next hop convergence A->B={:?} (admin={}, c={}), allow_relay={}", - next_hop_a_to_b, admin_peer_id, cred_c_peer_id, allow_relay - ); - if allow_relay { - next_hop_a_to_b == Some(cred_c_peer_id) - } else { - next_hop_a_to_b == Some(admin_peer_id) - } - }, + wait_route_next_hop( + &source_inst, + target_peer_id, + relay_peer_id, Duration::from_secs(20), + "dual-admin data path remains on relay after one admin stops", + ) + .await; + wait_route_next_hop( + &target_inst, + source_peer_id, + relay_peer_id, + Duration::from_secs(20), + "dual-admin return path remains on relay after one admin stops", + ) + .await; + assert_ping_forwarded_by("ns_c4", "10.144.144.3", &relay_inst, &[&surviving_admin]).await; + wait_route_cost( + &surviving_admin, + target_peer_id, + 1, + Duration::from_secs(10), + "surviving admin retains direct fallback", ) .await; - // wait 5s, make sure the routes are stable - tokio::time::sleep(Duration::from_secs(5)).await; + // Removing and restoring the Peer Relay's target edge changes its + // source-owned connection row. The post-shutdown relay -> admin -> relay + // next-hop transition proves that fresh rows pass through the surviving + // admin; stale state cannot satisfy both observations. + disconnect_connector(&target_inst, &relay_listener_url, relay_peer_id).await; + wait_for_condition( + || async { + !relay_inst + .get_core_instance() + .connected_peers() + .await + .contains(&target_peer_id) + }, + Duration::from_secs(10), + ) + .await; + wait_route_next_hop( + &source_inst, + target_peer_id, + surviving_admin_peer_id, + Duration::from_secs(30), + "relay edge removal restores the surviving admin data path", + ) + .await; + wait_route_next_hop( + &target_inst, + source_peer_id, + surviving_admin_peer_id, + Duration::from_secs(30), + "relay edge removal restores the surviving admin return path", + ) + .await; + assert_ping_forwarded_by("ns_c4", "10.144.144.3", &surviving_admin, &[&relay_inst]).await; - // Verify next hop from A to B based on allow_relay flag - let next_hop_a_to_b = cred_a_inst - .get_core_instance() - .route_snapshots() - .await - .into_iter() - .find(|route| route.peer_id == cred_b_peer_id) - .and_then(|route| route.next_hop_peer_id_latency_first); + target_inst.add_connector_url(relay_listener_url.clone()); + wait_direct_peer( + &target_inst, + relay_peer_id, + Duration::from_secs(10), + "target -> relay after admin shutdown", + ) + .await; + wait_direct_peer( + &relay_inst, + target_peer_id, + Duration::from_secs(10), + "relay -> target after admin shutdown", + ) + .await; + wait_route_next_hop( + &source_inst, + target_peer_id, + relay_peer_id, + Duration::from_secs(30), + "relay edge restoration restores the Peer Relay data path", + ) + .await; + wait_route_next_hop( + &target_inst, + source_peer_id, + relay_peer_id, + Duration::from_secs(30), + "relay edge restoration restores the Peer Relay return path", + ) + .await; + assert_ping_forwarded_by("ns_c4", "10.144.144.3", &relay_inst, &[&surviving_admin]).await; - println!( - "Next hop A->B={:?} (admin={}, c={}), allow_relay={}", - next_hop_a_to_b, admin_peer_id, cred_c_peer_id, allow_relay - ); - - // When C has allow_relay=false, route should go through Admin - // When C has allow_relay=true, route may go through C or Admin depending on routing algorithm - if !allow_relay { - assert_eq!( - next_hop_a_to_b, - Some(admin_peer_id), - "Route from A to B should go through admin when allow_relay=false" - ); - } else { - assert_eq!( - next_hop_a_to_b, - Some(cred_c_peer_id), - "Route from A to B should go through C when allow_relay=true" - ); - } - - // Cleanup - drop_insts(vec![admin_inst, cred_a_inst, cred_b_inst, cred_c_inst]).await; + drop_insts(vec![surviving_admin, relay_inst, target_inst, source_inst]).await; } /// Test 2: Two credential nodes connect to same admin