Add configurable batch flush threshold to SpinSink. When threshold > 1,
MpscTunnelSender::send accumulates packets in FramedWriter's BufList
without flushing. After N packets, poll_flush triggers a single writev()
syscall instead of N individual write() syscalls.
Implementation:
- SpinSink: pending_count + batch_threshold atomics
- MpscTunnelSender::send: flush every N packets via writev
- Default threshold=1 (per-packet flush, safe for handshake/control)
- Settable via set_batch_threshold() through PeerConn → Peer → PeerManager
- Bench: HOTPATH_BATCH env var, set after convergence
Batch threshold must be 1 during handshake (control packets are
request-response, can't be delayed). Bench sets threshold=8 only after
routes converge.
Benchmark (no hotpath, 3 runs avg):
TCP batch=1: 985K pps
TCP batch=8: 1,053K pps (+7%)
Ring: unchanged (flush is no-op for RingSink)
UDP: unchanged (flush is no-op for RingSink)
MpscTunnelSender::send avg: 343ns → 213ns (-38%, with hotpath) —
writev writes 8 Bytes in one syscall vs 8 write() calls.
All 210 peers tests pass. 6 netns tests fail (require root, unchanged).
RingSink's poll_flush is a no-op (data already in ring buffer after
start_send). Skip explicit poll_flush for ring/UDP tunnels to let
poll_ready handle batching at max_buffer_count.
FramedWriter (TCP) must flush per-packet: noop_waker can't wake the
task when socket write returns Pending, so accumulated data in BufList
would deadlock.
Added direct_batch_flush flag to MpscTunnelSender, set based on
tunnel_type at new_direct time:
ring/udp: batch_flush = true (skip explicit flush)
tcp: batch_flush = false (flush per packet)
Benchmark (no hotpath, 3 runs avg):
Ring: 1,124K → 1,120K pps (noise — flush was no-op anyway)
TCP: 975K → 995K pps (+2%, within noise)
UDP: 1,066K → 1,081K pps (+1.4%, within noise)
Ring/UDP show no change because RingSink flush was already a no-op.
TCP unchanged because it still flushes per-packet.
The real writev batching opportunity for TCP would require async flush
(not noop_waker), which is a separate optimization direction.
All 210 peers tests pass. 6 netns tests fail (require root, unchanged).
Add try_recv fast path to recv_packet_from_chan: try non-blocking
recv first, fall back to recv().await only when channel is empty.
Additionally inline the try_recv into start_peer_recv's loop body,
eliminating the async fn wrapper overhead for the common case (channel
has data).
Also add hotpath measure to DefaultCompressor::decompress for receive
path visibility.
Benchmark (TCP, with hotpath):
Before: ~449K pps
After: ~448K pps (noise — receiver is not the bottleneck)
Key finding: receive side is NOT the bottleneck in one-directional
bench. Sender rate (~448K pps with hotpath, ~984K without) limits
throughput. Receive optimization matters for bidirectional scenarios.
210 peers tests pass. 6 netns tests fail (require root, unchanged).
Key finding: hotpath profiling adds 54-57% overhead to benchmarks.
Real production pps (without hotpath):
Ring: 1,105K pps (12.4 Gbps)
TCP: 984K pps (11.0 Gbps)
Added gotcha #15 documenting the observer effect, updated benchmark
tables to show both hotpath/non-hotpath numbers, and noted that
timing data should be calibrated by ~2.3x for real-world estimates.
Replace BytesMut::split_off with Buf::advance in packet extraction paths:
- convert_type: advance instead of split_off (TCP/UDP/WG hot path)
- payload_bytes, tunnel_payload_bytes: same
- convert_to_dummy_tunnel_packet: same
- virtual_nic TunZCPacketToBytes: same
split_off creates a second BytesMut sharing the same allocation (Arc
ref count churn). advance just moves the start pointer forward —
zero-copy, zero-alloc, no Arc operations.
Benchmark: pps neutral (glibc handles split_off pattern efficiently),
but eliminates Arc churn and is cleaner code.
Combined with #2385 safe initialization for best of both PRs.
All 53 packet/mpsc/forward_packet tests pass.
Cherry-pick packet_def.rs changes from PR #2385:
- new_with_payload: write_bytes + copy_nonoverlapping (no aliasing check)
- new_for_tun: resize instead of unsafe set_len
- new_for_foreign_network: stack-allocated header + single write
- convert_type: resize instead of set_len
Eliminates UB from set_len on uninitialized memory. copy_nonoverlapping
skips aliasing checks, slightly faster for TCP path.
Benchmark (10s, 1400B):
TCP: 440K -> 464K pps (+5.5%)
Ring: 508K -> 473K pps (-7%, header zeroing overhead)
Ring regression is acceptable: ring is only for in-process benchmark,
real deployments use TCP/UDP over WAN where the TCP gain matters.
All 37 packet + 16 forward_packet tests pass.
set_bind_addr_for_peer_connector collected all local interface IPs
as bind addresses but omitted 127.0.0.1. When connecting to localhost,
the connector would bind to a non-loopback IP (e.g. 172.17.0.2 in
Docker) and fail with connect timeout because routing from non-loopback
to loopback doesn't work.
Fix: prepend 127.0.0.1:0 to the bind address list. The connector tries
all bind addresses, so loopback will be attempted first and succeed for
localhost connections.
Benchmark results in Docker (4 threads, 1400B, 10s):
Ring: 508K pps, MpscTunnelSender::send 138ns
UDP: 440K pps, MpscTunnelSender::send 294ns
TCP: 440K pps, MpscTunnelSender::send 376ns
All three tunnel types now converge and benefit from noop_waker sync send.
TCP tunnel uses FramedWriter (not RingSink), but start_send is still
sync (writes to BufList in memory). poll_flush does actual TCP write
syscall — noop_waker returns Ok for Pending (data stays in BufList,
flushed on next send when BufList >= 64).
Add TCP benchmark support via HOTPATH_TUNNEL=tcp. Note: TCP/UDP
convergence requires netns in bench environment (connector multi-bind
address behavior doesn't work for localhost without namespaces).
All 210 peers tests pass. Ring tunnel benchmark: 234K -> 508K pps (+117%).
UDP tunnel uses RingSink internally (same as ring tunnel). Extend
direct mode to include UDP. Fix poll_flush Pending to return Ok.
Add UDP benchmark support via HOTPATH_TUNNEL=udp env variable.
All 208 peers tests pass. Netns tests unchanged (require root).
The async fn Future state machine overhead (~1.9us) dominated
MpscTunnelSender::send, while RingSink operations were only ~40ns.
Breakthrough: make send() an async fn that completes synchronously
on the first poll for the direct (ring tunnel) path. Uses
futures::task::noop_waker() to construct a dummy Context, then calls
Sink trait methods (poll_ready, start_send, poll_flush) directly.
RingSink always returns Ready immediately, so the waker is never
invoked and the async fn completes without yielding.
Channel mode (TCP/UDP/WG tunnels) still uses async send_async()
with proper backpressure. Ring tunnels detected via tunnel_info()
type check in PeerConn.
Results (4 threads, 1400B, 15s):
pps: 249K → 474K (+90%)
send_msg_by_ip: 3.53us → 1.67us (-53%)
send_msg_internal: 2.40us → 502ns (-79%)
MpscTunnelSender::send: 1.97us → 144ns (-93%)
All 207 peers:: tests pass. Netns-requiring tests (three_node,
credential) unchanged (require root).
tokio::sync::Mutex and std::sync::Mutex both have !Send guards that
cannot cross await points in multi_thread runtime. Replace with a
custom SpinSink using AtomicBool CAS — the SpinGuard contains only a
&SpinSink reference (SpinSink: Sync via unsafe impl), so it is Send.
Benchmark: pps unchanged (~249K), MpscTunnelSender::send avg 1.97us.
The bottleneck is confirmed to be async fn Future state machine
overhead (~1.9us), not the lock mechanism. RingSink operations are
only ~40ns (poll_ready 15ns + start_send 10ns + poll_flush 15ns).
Further breakthrough requires either:
- Sync send API (bypassing async entirely)
- Concrete type instead of dyn ZCPacketSink (to call RingSink::try_send directly)
Replace 3 await points (lock().await + feed().await + flush().await)
with try_lock() (sync) + single poll_fn (merged poll_ready + start_send
+ poll_flush).
parking_lot::Mutex cannot be used because MutexGuard is !Send (cannot
cross await in multi_thread runtime). tokio::sync::Mutex try_lock()
returns synchronously and MutexGuard is Send.
Benchmark: pps 250K → 251K (+0.4%), MpscTunnelSender::send avg
2.07us → 1.98us (-90ns). Improvement is small because tokio async
machinery overhead (Future state machine + poll) dominates over
RingSink's actual 40ns operation cost.
MpscTunnelSender now supports two modes:
- Channel mode (existing): try_send to tokio mpsc → receiver task → sink
- Direct mode (new): MpscTunnelSender holds Arc<Mutex<sink>> directly,
bypassing the channel + receiver task entirely
PeerConn uses new_direct to skip the channel intermediary.
Benchmark result: pps unchanged (~245K). The async fn overhead of
Mutex::lock().await + SinkExt::feed().await + SinkExt::flush().await
(~2us) is comparable to channel try_send (~2us). The bottleneck is
the Sink trait's async poll machinery, not the channel itself.
However, this change provides:
- RingSink timing now fully visible (start_send 10ns, poll_ready 13ns,
poll_flush 17ns = 40ns/pkt total)
- Reduced architectural complexity (no receiver task for PeerConn)
- Foundation for a sync fast path using RingSink::try_send directly
Reveals RingSink operation costs:
poll_ready: 13ns/call (0.57% total)
start_send: <26ns/call (below top-15 threshold)
poll_flush: below top-15 threshold
Confirms ring tunnel sink operations are ~33ns/pkt total,
vs 2.04us for MpscTunnelSender::send (which routes through
tokio mpsc channel). The channel intermediary adds ~2us/pkt
of pure overhead.
channel(32) was frequently full under high pps, causing try_send to
fail and fall back to send().await (semaphore wait). Increasing to 1024
reduces fallback frequency.
Benchmark (4 threads, 1400B, 15s):
MpscTunnelSender::send avg: 2.23us → 2.01us (-220ns)
MpscTunnelSender::send P95: 6.39us → 5.74us (-650ns)
send_msg_internal avg: 2.67us → 2.45us (-220ns)
pps: ~250K (unchanged, receiver-bound)
pps unchanged because bottleneck moved to receiver (forward_one_round →
sink.feed/flush). The 220ns/pkt saving is pure CPU efficiency gain.
FuturesUnordered-based pipeline to overlap encrypt with mpsc_send.
Tested depths 1/4/8/16: max +1.6% at depth=4, within noise. Pipeline
has limited value because try_send fast path eliminates await gaps
that would allow overlap. Default remains depth=1 (serial).
Splits the 1.04us gap between send_msg_by_ip and send_msg_internal:
try_compress_and_encrypt: 386ns (37%) ← AES-GCM encrypt + zstd compress
get_msg_dst_peer_ipv4: 161ns (15%) ← IP→peer_id route lookup
run_nic_packet_process_pipeline: 121ns (12%) ← ACL check
other (fill_hdr + counters): ~440ns (36%)
The 386ns encrypt is the largest optimization opportunity in this gap:
ring tunnel is in-process, so application-layer encryption may be skippable.
Adds hotpath::measure to two critical blind spots in the send chain:
1. MpscTunnelSender::send — the tokio mpsc channel send point, which
accounts for 84% of PeerConn::send_msg wall time (2.33us/pkt).
2. PeerManager::send_msg_by_ip — the top-level packet send entry point,
revealing a 1.04us gap between send_msg_by_ip and send_msg_internal
(encryption + routing + ACL + fan-out).
Full send chain timing now visible:
send_msg_by_ip: 3.83us
└─ send_msg_internal: 2.79us (gap: 1.04us = encrypt + route + ACL)
└─ MpscTunnelSender::send: 2.33us (84% of internal)
Two optimizations to reduce per-packet TrafficMetricRecorder overhead:
1. Batch CounterHandle updates (TRAFFIC_BATCH_SIZE=128): accumulate
bytes/packets in AtomicU64, flush to CounterHandle (and its
touch()/Instant::now()) only every 128 packets. Reduces touch
calls from 4/pkt to 0.03/pkt.
2. Sync fast path: record_tx_fast/record_rx_fast handle the common
case (peer already resolved) without entering async fn or cloning
TrafficCounters. Falls back to async record_tx/record_rx only for
first packet to a new/unresolved peer.
Tests use BATCH_SIZE=1 via cfg(test) for exact counter validation.
Benchmark (4 threads, 1400B, 15s, 3 runs):
Before: 246K pps, send_msg_internal avg 3.13us
After: 250K pps, send_msg_internal avg 3.05us
Delta: +1.6% pps, -80ns/pkt
All 11 traffic_metrics + send_msg_internal tests pass.
Add #[global_allocator] behind feature flags so the bench can test
different allocators. Previously the example used glibc malloc by
default (easytier-core.rs sets jemalloc/mimalloc only for the bin
target, not examples).
Benchmark (4 threads, 1400B, 15s, clone mode):
glibc: 246K pps, 3.13us/pkt
jemalloc: 246K pps, 3.21us/pkt
mimalloc: 242K pps, 3.25us/pkt
All within noise. Single-threaded clone has low malloc contention;
~1500B small allocs are served efficiently by all tcaches.
Replace all UnsafeCell-based counters with safe alternatives:
- New ShardedCounter: per-thread TLS accumulation (thread_local crate)
with periodic publish to static AtomicU64. Zero atomic RMW on hot path.
- ACL RuleStatsTracker: remove unsafe Arc<RuleStats> raw pointer
mutation, use two ShardedCounter fields.
- ZCPacket: replace set_len on uninitialized BytesMut with
write_bytes + copy_nonoverlapping before set_len.
- StatsManager UnsafeCounter/MetricData: remove UnsafeCell and
unsafe impl Send/Sync, wrap ShardedCounter. last_updated uses
AtomicU64 epoch millis.
- Throughput: replace UnsafeCell with AtomicU64.
- secure_datagram: fix grace-window test timestamp.
MpscTunnelSender::send now tries try_send first, falling back to
send().await only when the channel is full. try_send bypasses the
tokio batch_semaphore Acquire::poll + add_permits_locked machinery
(~9.4% of CPU in samply profiling), which is pure overhead when the
channel has capacity.
In the ring-tunnel bench (4 threads, 1400B, 15s) the channel(32) fast
path hits >99%, so the fallback rarely triggers.
Benchmark improvement:
pps: 230K -> 246K (+7.0%)
send_msg_internal avg: 3.25us -> 3.13us (-120ns/pkt)
forward_one_round calls: 2.46M -> 706K (-71%, bigger batches)
All mpsc tests pass.
Replace has_peer(dst_peer_id) + send_msg_directly() with a single
get_peer_by_id() call, eliminating one redundant dashmap contains_key
query (~50-100ns) per packet on the direct-peer happy path.
send_msg_directly is no longer called from send_msg_internal but remains
available for other callers. All 7 send_msg_internal tests pass.
Benchmark (4 threads, 1400B pkts, 15s):
Before: 234K pps, send_msg_internal avg 3.26us
After: 230K pps, send_msg_internal avg 3.25us
Delta within noise; ~10-50ns/pkt saved as expected for one fewer hash.
Replace std::time::Instant with quanta::Instant on per-packet, per-RPC,
and per-session paths. TSC-based, ~5ns vs ~25ns per now() call.
Reuses the existing `extern crate self as hotpath` alias so
`use hotpath::instant::Instant;` resolves to the same quanta type with
or without the hotpath feature. Leaves tokio::time::Instant and
smoltcp::time::Instant untouched.
* refactor(web): use generated proto network types
* fix(core): preserve dumped config flags
* test(web): cover config flag save paths
* fix(ci): use system protoc before frontend codegen
* fix(ci): serialize frontend-lib builds
Handle TUN receive errors by marking the fake TCP stack closed and
clearing registered sockets instead of panicking.
Refuse new sockets on closed stacks and let listeners recreate stacks
when the reader task exits.
Remove quinn-plaintext to fix connection errors caused
by different hash values across platforms.
On x64, maintain compatibility with quinn-plaintext.
* feat(ffi): add async data plane API
* feat(ffi): add async data plane examples
* test(ffi): make async Go dataplane tests self-contained
* docs(ffi): document Go async dataplane API
* docs(android): document dataplane JNI API
pnpm v11 introduces breaking changes that cause frozen installations
to fail:
1. The "pnpm" field in package.json is no longer read. Moved
`overrides` to `pnpm-workspace.yaml` to fix
`ERR_PNPM_LOCKFILE_CONFIG_MISMATCH`.
2. `strictDepBuilds` is now enabled by default. Added required
dependencies (esbuild, unrs-resolver, vue-demi) to `allowBuilds` in
the workspace config to fix `ERR_PNPM_IGNORED_BUILDS`.
Add config server client support for the C FFI and Android JNI bindings.
Reuse the existing easytier::web_client::run_web_client path and
NetworkInstanceManager; OHOS is unchanged.
Report successful remote config apply/delete operations through a
callback, with one JSON event per affected instance.
Keep the config server client and FFI data plane mutually exclusive: once
either side is in use, the other side returns an error instead of sharing
lifecycle state.
Use pbjson to support string deserialization for enum fields
This allows TOML configs like:
chainType = "Inbound"
instead of:
chainType = 1
- Maintain backward compatibility with integer values
- Default serialization format is now string
TomlConfigLoader::new_from_str() always calls NetworkIdentity::new()
with unwrap_or_default() on network_secret, converting None to ''.
This creates a non-zero SHA256 digest, causing credential nodes loaded
from TOML to be misidentified as regular nodes (with network_secret),
which breaks Noise handshake authentication.
Fix: check if secure_mode is enabled AND network_secret is absent/empty,
and call NetworkIdentity::new_credential() in that case.
The same detection already exists in:
- core.rs (CLI path, via --credential flag)
- launcher.rs (GUI/web path, via gen_config)
This makes TOML config loading consistent with the other two entry points.
fanyang