perf(data-plane): restore native throughput after host portability (#2452)

* perf(core): make data-plane idle check constant time

Avoid scanning every DashMap shard for each peer packet when no data-plane flows are active.

Publish the flow count before insertion and release it after removal so an Acquire load is a safe O(1) idle signal. Reject count overflow and underflow instead of silently saturating.

* test(perf): add repeatable two-node netns benchmark

Create isolated underlay namespaces, pin both EasyTier cores and iperf3 endpoints, and measure a single TCP flow in both directions over either UDP or TCP peer transport.

Keep every iperf3 JSON result and emit directional medians while cleaning up processes and namespaces on every exit path.

* perf(tcp): preserve native owned stream halves

Let each VirtualTcpSocket adapter consume itself into independent read and write halves. Portable adapters retain the generic shared split as a default.

Use lock-free Tokio owned halves for native TCP and Unix streams so tunnel I/O no longer takes the generic split mutex on every poll. Cover full-duplex traffic and write-half shutdown.

* perf(packet): preserve ownership across the Host seam

Introduce an opaque, move-only HostPacket that retains core packet storage while exposing only the raw IP payload. Clear private headers before handing storage back to a native TUN adapter.

Use an ownership-preserving bounded channel for native ingress and egress. Keep explicit copy adapters for Vec and WASI boundaries, and verify allocation identity, backpressure, shutdown, and end-to-end delivery.

* perf(udp): preserve packet ownership through sessions

Carry EasyTier tunnel packets through UDP session queues as owned values. Reuse the existing tunnel header for session framing instead of copying payloads into a second packet and rebuilding them on receive.

Keep completion delivery for the public datagram socket API while removing the unused completion channel from streaming tunnel sends. Avoid the unconditional receive-side clone before QUIC routing is known.

* perf(peer): publish packet filters as immutable snapshots

Replace per-packet async and synchronous registry locks with ArcSwap snapshots. Permanent filters now need no activity checks, while managed registrations retain explicit acquire/release visibility.

Closing a managed registration marks it inactive before atomically removing it. Existing snapshots keep in-flight filters alive, and registration mutations prune inactive entries while preserving newest-first order.

* perf(instance): give native hosts direct packet egress

Let the core create one bounded HostPacket channel and transfer its receiver directly to a PacketEgressHost during startup. Native TUN runtimes now consume that receiver without the intermediate PacketSink channel and forwarding task.

Keep PacketSinkEgress as the compatibility adapter for callback and test hosts, and make receiver installation one-shot across desktop, mobile, and disabled runtimes.

* perf(crypto): restore accelerated native AEAD backends

Move Ring and OpenSSL implementations behind the core Encryptor seam.
Portable builds continue selecting only supported backends.

Restore historical precedence: OpenSSL, Ring, then RustCrypto. Keep
backend availability consistent across secure transports and cover
fixed-nonce wire compatibility between implementations.

* perf(udp): receive native datagrams into owned buffers

Extend the portable UDP socket seam with an owned-datagram receive path.
Keep a compatible default for portable hosts. Native Unix sockets write
recvmsg output directly into the final BytesMut allocation.

This removes the per-packet stack-to-heap copy introduced by the portable
socket boundary without exposing native socket resources to core.

* perf(data-plane): remove portable hot-path overhead

Restore native throughput lost while generalizing the host and UDP
session layers.

Read packet policy once per send, update traffic counters through
registry guards, and preserve packet ownership while UDP dispatch
borrows stable session state.

Move UDP shutdown monitoring into a control task so forwarding avoids
a select future per packet. Bound native datagram storage to 8 KiB,
reject oversized sends, and drop truncated Unix receives.

Keep accelerated AEAD selection warning-free when portable crypto
features are also built. Cover session bounds, truncation, and idle
shutdown with regression tests.

* fix(udp): preserve portable datagram receive semantics

Keep the public portable receive capacity at the theoretical UDP
maximum instead of silently shrinking it to the native fast-path limit.

Apply the 8 KiB session boundary after a complete portable receive,
so Windows cannot turn an oversized datagram into a fatal listener
error and other adapters cannot dispatch a truncated prefix.

Cover dropping an oversized packet while the same portable socket
continues to deliver the following valid datagram.

* fix(ci): align feature gating with backend selection

Compile the Ring implementation in production only when OpenSSL is not
selected, while retaining it for cross-backend unit tests.

Remove stale test imports and assert UDP dispatch results so the strict
workspace Clippy job passes without suppressing diagnostics.
This commit is contained in:
KKRainbow
2026-07-26 22:54:43 +08:00
committed by GitHub
parent dc11298558
commit 7fb42c3b73
48 changed files with 2182 additions and 541 deletions
+32 -8
View File
@@ -4,6 +4,8 @@ use std::sync::Arc;
use easytier_core::gateway::proxy::wrapped_transport::WrappedTransportEngines;
#[cfg(feature = "wireguard")]
use easytier_core::gateway::vpn_portal::VpnPortalHost;
#[cfg(test)]
use easytier_core::host::packet::{HostPacket, PacketSink};
#[cfg(feature = "management")]
use easytier_core::{
connectivity::manual::ManualTunnelConnector,
@@ -12,8 +14,7 @@ use easytier_core::{
};
use easytier_core::{
events::{CoreEvent, CoreEventSink},
host::packet::PacketSink,
instance::{CoreHostAdapters, CoreInstance},
instance::{CoreHostAdapters, CoreInstance, PacketEgressHost},
process_runtime::CoreProcessRuntime,
};
@@ -45,10 +46,13 @@ pub(crate) fn compose_native_core_instance(
process_runtime: Arc<CoreProcessRuntime>,
) -> anyhow::Result<Arc<NativeCoreInstance>> {
let global_ctx = Arc::new(GlobalCtx::new(config.clone()));
let (packet_sender, packet_receiver) = tokio::sync::mpsc::channel(128);
let mut adapters =
runtime_core_host_adapters(global_ctx.clone(), process_runtime, Arc::new(packet_sender));
adapters.instance_runtime = NativeInstanceRuntimeHost::new(global_ctx.clone(), packet_receiver);
let runtime_host = NativeInstanceRuntimeHost::new(global_ctx.clone());
let mut adapters = runtime_core_host_adapters_with_packet_egress(
global_ctx.clone(),
process_runtime,
runtime_host.clone(),
);
adapters.instance_runtime = runtime_host;
NativeCoreInstance::from_toml(config, adapters)
}
@@ -148,6 +152,7 @@ fn runtime_wrapped_transport_engines() -> WrappedTransportEngines {
WrappedTransportEngines { kcp, quic }
}
#[cfg(test)]
pub(crate) fn runtime_core_host_adapters(
global_ctx: ArcGlobalCtx,
process_runtime: Arc<CoreProcessRuntime>,
@@ -155,7 +160,26 @@ pub(crate) fn runtime_core_host_adapters(
) -> CoreHostAdapters<NativeInstanceHost> {
let host = native_instance_host(global_ctx.clone());
let runtime_dns = native_host_runtime();
let mut adapters = CoreHostAdapters::new(host, runtime_dns, packet_sink, process_runtime);
let adapters = CoreHostAdapters::new(host, runtime_dns, packet_sink, process_runtime);
configure_runtime_core_host_adapters(global_ctx, adapters)
}
pub(crate) fn runtime_core_host_adapters_with_packet_egress(
global_ctx: ArcGlobalCtx,
process_runtime: Arc<CoreProcessRuntime>,
packet_egress: Arc<dyn PacketEgressHost>,
) -> CoreHostAdapters<NativeInstanceHost> {
let host = native_instance_host(global_ctx.clone());
let runtime_dns = native_host_runtime();
let adapters =
CoreHostAdapters::new_with_packet_egress(host, runtime_dns, packet_egress, process_runtime);
configure_runtime_core_host_adapters(global_ctx, adapters)
}
fn configure_runtime_core_host_adapters(
global_ctx: ArcGlobalCtx,
mut adapters: CoreHostAdapters<NativeInstanceHost>,
) -> CoreHostAdapters<NativeInstanceHost> {
#[cfg(test)]
adapters.replace_stun_provider(Arc::new(crate::common::stun::MockStunInfoCollector {
udp_nat_type: crate::proto::common::NatType::Unknown,
@@ -504,7 +528,7 @@ mod tests {
loop {
instance_a
.packet_plane()
.send_ip_packet(ip_packet.clone())
.send_ip_packet(HostPacket::copy_from_payload(&ip_packet))
.await
.unwrap();
match tokio::time::timeout(
+9 -10
View File
@@ -4,7 +4,11 @@ use std::sync::Arc;
use std::time::Duration;
use cidr::Ipv4Inet;
use easytier_core::{gateway::magic_dns::MagicDnsRoute, process_runtime::CoreProcessRuntime};
use easytier_core::{
gateway::magic_dns::MagicDnsRoute,
host::packet::{HostPacket, HostPacketChannelSink, HostPacketReceiver},
process_runtime::CoreProcessRuntime,
};
use hickory_client::client::{Client, ClientHandle as _};
use hickory_proto::rr;
use hickory_proto::runtime::TokioRuntimeProvider;
@@ -33,21 +37,16 @@ pub async fn prepare_env(
prepare_env_with_tld_dns_zone(dns_name, tun_ip, None).await
}
async fn build_test_core(
ctx: ArcGlobalCtx,
) -> (
Arc<NativeCoreInstance>,
tokio::sync::mpsc::Receiver<Vec<u8>>,
) {
let (packet_sink, packet_receiver) = tokio::sync::mpsc::channel(128);
async fn build_test_core(ctx: ArcGlobalCtx) -> (Arc<NativeCoreInstance>, HostPacketReceiver) {
let (packet_sink, packet_receiver) = tokio::sync::mpsc::channel::<HostPacket>(128);
let adapters = runtime_core_host_adapters(
ctx.clone(),
CoreProcessRuntime::new(),
Arc::new(packet_sink),
Arc::new(HostPacketChannelSink::new(packet_sink)),
);
let core_instance = NativeCoreInstance::new(test_core_instance_config(&ctx), adapters).unwrap();
core_instance.start().await.unwrap();
(core_instance, packet_receiver)
(core_instance, HostPacketReceiver::new(packet_receiver))
}
pub async fn prepare_env_with_tld_dns_zone(
+11 -11
View File
@@ -1,7 +1,9 @@
use std::sync::Arc;
use easytier_core::{gateway::dhcp::DhcpIpv4Host, instance::CorePacketPlane};
use tokio::sync::{Mutex, mpsc};
use easytier_core::{
gateway::dhcp::DhcpIpv4Host, host::packet::HostPacketReceiver, instance::CorePacketPlane,
};
use tokio::sync::Mutex;
use tokio_util::sync::CancellationToken;
use crate::common::global_ctx::ArcGlobalCtx;
@@ -27,8 +29,6 @@ use event_journal::EventJournal;
use magic_dns::MagicDnsRuntime;
use tun_runtime::NativeTunRuntime;
pub(super) type HostPacketReceiver = mpsc::Receiver<Vec<u8>>;
pub(crate) struct NativeInstanceRuntimeHost {
global_ctx: ArcGlobalCtx,
operation: Arc<Mutex<()>>,
@@ -38,12 +38,9 @@ pub(crate) struct NativeInstanceRuntimeHost {
}
impl NativeInstanceRuntimeHost {
pub(crate) fn new(
global_ctx: ArcGlobalCtx,
peer_packet_receiver: HostPacketReceiver,
) -> Arc<Self> {
pub(crate) fn new(global_ctx: ArcGlobalCtx) -> Arc<Self> {
let cancel = CancellationToken::new();
let tun = NativeTunRuntime::new(global_ctx.clone(), cancel.clone(), peer_packet_receiver);
let tun = NativeTunRuntime::new(global_ctx.clone(), cancel.clone());
let event_journal = EventJournal::new(&global_ctx);
Arc::new(Self {
global_ctx,
@@ -87,6 +84,10 @@ impl NativeInstanceRuntimeHost {
fn attach_runtime_tun_fd(&self, fd: i32) -> anyhow::Result<()> {
self.tun.attach_fd(fd)
}
fn install_packet_receiver(&self, receiver: HostPacketReceiver) -> anyhow::Result<()> {
self.tun.install_packet_receiver(receiver)
}
}
#[cfg(test)]
@@ -100,8 +101,7 @@ mod tests {
#[test]
fn runtime_host_owns_event_subscription_context() {
let global_ctx = Arc::new(GlobalCtx::new(TomlConfig::default()));
let (_packet_sender, packet_receiver) = mpsc::channel(1);
let runtime_host = NativeInstanceRuntimeHost::new(global_ctx.clone(), packet_receiver);
let runtime_host = NativeInstanceRuntimeHost::new(global_ctx.clone());
let mut events = runtime_host.subscribe_event();
global_ctx.issue_event(GlobalCtxEvent::CredentialChanged);
@@ -2,7 +2,8 @@ use std::sync::Arc;
use easytier_core::{
gateway::dhcp::DhcpIpv4Host,
instance::{CorePacketPlane, InstanceRuntimeHost},
host::packet::HostPacketReceiver,
instance::{CorePacketPlane, InstanceRuntimeHost, PacketEgressHost},
};
use super::NativeInstanceRuntimeHost;
@@ -42,3 +43,16 @@ impl InstanceRuntimeHost for NativeInstanceRuntimeHost {
self.attach_runtime_tun_fd(fd)
}
}
#[async_trait::async_trait]
impl PacketEgressHost for NativeInstanceRuntimeHost {
async fn start(&self, receiver: HostPacketReceiver) -> anyhow::Result<()> {
self.install_packet_receiver(receiver)
}
async fn stop(&self) {}
fn request_stop(&self) {
self.request_runtime_shutdown();
}
}
@@ -1,8 +1,12 @@
use std::{any::Any, sync::Arc};
use std::{
any::Any,
sync::{Arc, OnceLock},
};
use easytier_core::host::packet::HostPacketReceiver;
use tokio::{sync::Mutex, task::JoinSet};
use super::{HostPacketReceiver, MagicDnsRuntime};
use super::MagicDnsRuntime;
use crate::instance::virtual_nic::NicCtx;
struct NicCtxContainer {
@@ -29,19 +33,28 @@ impl NicCtxContainer {
#[derive(Clone)]
pub(super) struct TunNicState {
nic_ctx: Arc<Mutex<Option<NicCtxContainer>>>,
receiver: Arc<Mutex<HostPacketReceiver>>,
receiver: Arc<OnceLock<Arc<Mutex<HostPacketReceiver>>>>,
}
impl TunNicState {
pub(super) fn new(receiver: HostPacketReceiver) -> Self {
pub(super) fn empty() -> Self {
Self {
nic_ctx: Arc::new(Mutex::new(None)),
receiver: Arc::new(Mutex::new(receiver)),
receiver: Arc::new(OnceLock::new()),
}
}
pub(super) fn install_receiver(&self, receiver: HostPacketReceiver) -> anyhow::Result<()> {
self.receiver
.set(Arc::new(Mutex::new(receiver)))
.map_err(|_| anyhow::anyhow!("native packet receiver is already installed"))
}
pub(super) fn receiver(&self) -> Arc<Mutex<HostPacketReceiver>> {
self.receiver.clone()
self.receiver
.get()
.expect("packet receiver must be installed before preparing TUN")
.clone()
}
pub(super) async fn stop(&self) {
@@ -54,7 +67,7 @@ impl TunNicState {
pub(super) async fn drain(&self) {
self.stop().await;
let receiver = self.receiver.clone();
let receiver = self.receiver();
let mut tasks = JoinSet::new();
tasks.spawn(async move {
let mut receiver = receiver.lock().await;
@@ -4,6 +4,7 @@ use anyhow::Context as _;
use cidr::Ipv4Inet;
use easytier_core::{
gateway::dhcp::{DhcpIpv4ApplyOutcome, DhcpIpv4ApplyPermit, DhcpIpv4Host},
host::packet::HostPacketReceiver,
instance::CorePacketPlane,
};
use futures::FutureExt as _;
@@ -13,7 +14,7 @@ use tokio::{
};
use tokio_util::sync::CancellationToken;
use super::{HostPacketReceiver, MagicDnsRuntime, tun_common::TunNicState};
use super::{MagicDnsRuntime, tun_common::TunNicState};
use crate::{
common::{
config::ConfigLoader as _,
@@ -31,19 +32,22 @@ pub(super) struct NativeTunRuntime {
}
impl NativeTunRuntime {
pub(super) fn new(
global_ctx: ArcGlobalCtx,
cancel: CancellationToken,
peer_packet_receiver: HostPacketReceiver,
) -> Self {
pub(super) fn new(global_ctx: ArcGlobalCtx, cancel: CancellationToken) -> Self {
Self {
global_ctx,
cancel,
nic: TunNicState::new(peer_packet_receiver),
nic: TunNicState::empty(),
static_ip_task: Mutex::new(None),
}
}
pub(super) fn install_packet_receiver(
&self,
receiver: HostPacketReceiver,
) -> anyhow::Result<()> {
self.nic.install_receiver(receiver)
}
fn report_static_ip_cancelled(output: &mut Option<oneshot::Sender<Result<(), Error>>>) {
if let Some(output) = output.take() {
let _ = output.send(Err(anyhow::anyhow!(
@@ -3,12 +3,12 @@ use std::sync::Arc;
use cidr::Ipv4Inet;
use easytier_core::{
gateway::dhcp::{DhcpIpv4ApplyOutcome, DhcpIpv4ApplyPermit, DhcpIpv4Host},
host::packet::HostPacketReceiver,
instance::CorePacketPlane,
};
use tokio::sync::Mutex;
use tokio_util::sync::CancellationToken;
use super::HostPacketReceiver;
use crate::common::global_ctx::{ArcGlobalCtx, GlobalCtxEvent};
pub(super) struct NativeTunRuntime {
@@ -17,15 +17,18 @@ pub(super) struct NativeTunRuntime {
}
impl NativeTunRuntime {
pub(super) fn new(
global_ctx: ArcGlobalCtx,
cancel: CancellationToken,
peer_packet_receiver: HostPacketReceiver,
) -> Self {
drop(peer_packet_receiver);
pub(super) fn new(global_ctx: ArcGlobalCtx, cancel: CancellationToken) -> Self {
Self { global_ctx, cancel }
}
pub(super) fn install_packet_receiver(
&self,
receiver: HostPacketReceiver,
) -> anyhow::Result<()> {
drop(receiver);
Ok(())
}
pub(super) async fn prepare(&self, _packet_plane: Arc<CorePacketPlane>) -> anyhow::Result<()> {
Ok(())
}
@@ -4,13 +4,14 @@ use anyhow::Context as _;
use cidr::Ipv4Inet;
use easytier_core::{
gateway::dhcp::{DhcpIpv4ApplyOutcome, DhcpIpv4ApplyPermit, DhcpIpv4Host},
host::packet::HostPacketReceiver,
instance::CorePacketPlane,
};
use futures::FutureExt as _;
use tokio::sync::{Mutex, Notify, mpsc};
use tokio_util::sync::CancellationToken;
use super::{HostPacketReceiver, MagicDnsRuntime, tun_common::TunNicState};
use super::{MagicDnsRuntime, tun_common::TunNicState};
use crate::{
common::global_ctx::{ArcGlobalCtx, GlobalCtxEvent},
instance::virtual_nic::NicCtx,
@@ -26,22 +27,25 @@ pub(super) struct NativeTunRuntime {
}
impl NativeTunRuntime {
pub(super) fn new(
global_ctx: ArcGlobalCtx,
cancel: CancellationToken,
peer_packet_receiver: HostPacketReceiver,
) -> Self {
pub(super) fn new(global_ctx: ArcGlobalCtx, cancel: CancellationToken) -> Self {
let (tun_fd, tun_fd_receiver) = mpsc::channel(16);
Self {
global_ctx,
cancel,
nic: TunNicState::new(peer_packet_receiver),
nic: TunNicState::empty(),
tun_fd,
tun_fd_receiver: Mutex::new(Some(tun_fd_receiver)),
task: Mutex::new(None),
}
}
pub(super) fn install_packet_receiver(
&self,
receiver: HostPacketReceiver,
) -> anyhow::Result<()> {
self.nic.install_receiver(receiver)
}
async fn install_mobile_tun(
nic_state: TunNicState,
global_ctx: ArcGlobalCtx,
+5 -6
View File
@@ -10,7 +10,7 @@ use easytier_core::{
use crate::{
common::global_ctx::{ArcGlobalCtx, GlobalCtx},
instance::{
composition::{NativeCoreInstance, runtime_core_host_adapters},
composition::{NativeCoreInstance, runtime_core_host_adapters_with_packet_egress},
runtime_host::NativeInstanceRuntimeHost,
},
socket::udp::RuntimeUdpSocket,
@@ -50,15 +50,14 @@ impl TestInstance {
),
) -> Self {
let global_ctx = Arc::new(GlobalCtx::new(config.clone()));
let (packet_sender, packet_receiver) = tokio::sync::mpsc::channel(128);
let mut adapters = runtime_core_host_adapters(
let runtime_host = NativeInstanceRuntimeHost::new(global_ctx.clone());
let mut adapters = runtime_core_host_adapters_with_packet_egress(
global_ctx.clone(),
process_runtime,
Arc::new(packet_sender),
runtime_host.clone(),
);
customize(&mut adapters);
adapters.instance_runtime =
NativeInstanceRuntimeHost::new(global_ctx.clone(), packet_receiver);
adapters.instance_runtime = runtime_host;
let core = CoreInstance::from_toml(config, adapters)
.expect("test CoreInstance composition should be valid");
Self { core, global_ctx }
+7 -6
View File
@@ -14,6 +14,7 @@ use crate::common::{
};
use easytier_core::{
host::packet::{HostPacket, HostPacketReceiver},
instance::CorePacketPlane,
packet::{TAIL_RESERVED_SIZE, ZCPacket, ZCPacketType},
tunnel::{
@@ -42,8 +43,6 @@ use zerocopy::{NativeEndian, NetworkEndian};
#[cfg(target_os = "windows")]
use crate::common::ifcfg::RegistryManager;
type HostPacketReceiver = tokio::sync::mpsc::Receiver<Vec<u8>>;
pin_project! {
pub struct TunStream {
#[pin]
@@ -866,15 +865,17 @@ impl NicCtx {
}
async fn do_forward_nic_to_peers(ret: ZCPacket, packet_plane: &CorePacketPlane) {
let payload = ret.payload();
if payload.is_empty() {
if ret.payload().is_empty() {
return;
}
tracing::trace!(
?ret,
"[USER_PACKET] recv new packet from tun device and forward to peers."
);
if let Err(error) = packet_plane.send_ip_packet(payload.to_vec()).await {
if let Err(error) = packet_plane
.send_ip_packet(HostPacket::from_tun_packet(ret))
.await
{
tracing::trace!(?error, "[USER_PACKET] send_msg failed");
}
}
@@ -912,7 +913,7 @@ impl NicCtx {
"[USER_PACKET] forward packet from peers to nic. packet: {:?}",
packet
);
let ret = sink.send(ZCPacket::new_with_payload(&packet)).await;
let ret = sink.send(packet.into_tun_packet()).await;
if ret.is_err() {
tracing::error!(?ret, "do_forward_tunnel_to_nic sink error");
}
@@ -10,7 +10,10 @@ use easytier_core::gateway::udp_broadcast::{
use {
crate::common::global_ctx::{ArcGlobalCtx, GlobalCtxEvent},
anyhow::Context,
easytier_core::{gateway::udp_broadcast::UdpBroadcastRelayStats, instance::CorePacketPlane},
easytier_core::{
gateway::udp_broadcast::UdpBroadcastRelayStats, host::packet::HostPacket,
instance::CorePacketPlane,
},
network_interface::{Addr, NetworkInterface, NetworkInterfaceConfig},
socket2::{Domain, Protocol, SockAddr, Socket, Type},
std::{
@@ -188,7 +191,7 @@ async fn forward_normalized_packet(
stats: &UdpBroadcastRelayStats,
) {
let ret = packet_plane
.send_local_ip_packet(normalized.packet.clone())
.send_local_ip_packet(HostPacket::copy_from_payload(&normalized.packet))
.await;
let summary = UdpPacketSummary::parse(&normalized.packet);