Sign up to save your podcasts
Or
Share Local Inference and Layer Splitting
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Local Inference and Layer Splitting
Key technical highlights covered in this session include:
- The Mohawk Inference Engine Architecture: An overview of this lightweight, secure engine engineered for multi-device layer splitting and PQC-secured edge offloading.
- Multi-Device Layer Splitting: A deep dive into how the engine optimizes model execution by distributing layers across different hardware backends (CPU, GPU, NPU) to maximize throughput and minimize latency.
- PQC-Secured Edge Offloading: Understanding the security protocols used when offloading inference tasks to edge nodes, ensuring that the transport remains quantum-resistant.
- Production-Optimized Performance: An analysis of the engine’s scalability features, such as connection pooling for 100+ concurrent connections, and its ability to maintain UI responsiveness through blocking detection.
- Real-Time Inference Observability: Exploring the specialized GUI that provides real-time tracking of p50/p95/p99 latencies, GPU utilization, and memory pressure for active inference sessions.
- Enterprise-Grade Hardening: A breakdown of the security defaults, including JWT authentication with RSA signatures, mTLS for worker communication, and Fernet-encrypted configuration storage.
View all episodes
By Ryan WilliamsKey technical highlights covered in this session include:
- The Mohawk Inference Engine Architecture: An overview of this lightweight, secure engine engineered for multi-device layer splitting and PQC-secured edge offloading.
- Multi-Device Layer Splitting: A deep dive into how the engine optimizes model execution by distributing layers across different hardware backends (CPU, GPU, NPU) to maximize throughput and minimize latency.
- PQC-Secured Edge Offloading: Understanding the security protocols used when offloading inference tasks to edge nodes, ensuring that the transport remains quantum-resistant.
- Production-Optimized Performance: An analysis of the engine’s scalability features, such as connection pooling for 100+ concurrent connections, and its ability to maintain UI responsiveness through blocking detection.
- Real-Time Inference Observability: Exploring the specialized GUI that provides real-time tracking of p50/p95/p99 latencies, GPU utilization, and memory pressure for active inference sessions.
- Enterprise-Grade Hardening: A breakdown of the security defaults, including JWT authentication with RSA signatures, mTLS for worker communication, and Fernet-encrypted configuration storage.