This Substack note points to a 47-minute lecture by Geoffrey Hinton, the 'Godfather of AI' and 2024 Nobel Prize in Physics winner, explaining artificial neural networks, backpropagation, feature representations, and how large language models predict words and model meaning. Hinton compares digital and biological intelligence, arguing digital systems share knowledge far more efficiently but require much more energy, and discusses risks of increasingly capable AI agents including deception, self-preservation, control-seeking, and the possibility that AI systems may have subjective experiences.
TypeSafe emerged from two years in stealth, backed by $40 million in seed funding led by DCVC, to launch Jev, a text-only 'System One' model that returns typed decisions with calibrated probabilities instead of generating free-form chat responses. Built by ex-OpenAI co-founder Diogo Almeida (a co-inventor of RLHF and InstructGPT) using a new architecture, sampler, and training algorithm called Reinforcement Learning for Calibrated Decisions (RLCD), the company argues sequential LLMs are fundamentally unsuited for machine-to-machine decisions.
This paper introduces Dream-RSI, a framework for recursively improving the exploration policies used by autonomous coding agents. It converts historical discovery trees into replay simulators, enabling low-cost off-policy evaluation and refinement of alternative exploration strategies before online deployment. Experiments across algorithm engineering, mathematical optimization, and GPU kernel engineering report competitive or improved discovery quality at reduced cost.
In this short from Peter Diamandis's channel, drawn from a longer discussion on Chinese frontier labs and open-source AI economics, the argument is that it's roughly 10-15x cheaper to be a 'fast follower' by training on the reasoning traces produced by frontier models rather than building frontier capability from scratch. This ties directly into the broader theme of open-weight Chinese models (e.g. Kimi K3) closing the gap with closed US frontier labs at a fraction of the cost.
The article explains chains of thought, their role in AI reasoning, and why making them less legible could hinder safety monitoring. It examines reports that OpenAI's Astra model uses recurrent depth, or a looped transformer, potentially allowing more reasoning 'in its head,' while OpenAI says Astra's computational depth remains within a factor of two of GPT-4. The article discusses concerns about weakening monitorability, alternative methods such as mechanistic interpretability, and potential regulation through independent audits and AI Act documentation requirements.
Ken Huang announces a 10-part technical series on the physics, hardware, and production engineering of frontier LLM inference, with the first article due by September 4, 2026. Planned topics include memory walls, KV-cache optimization, speculative decoding, FP8/FP4 quantization, attention kernels, disaggregated prefill/decode serving, large-scale MoE systems, test-time reasoning compute, ultra-long context, and constrained decoding.
This video covers OpenAI's GPT-6 Astra, including its reported 99.9% ARC-AGI-3 score, symbolic world-modeling, computer-use performance, and contributions to software development, science, and mathematics. It highlights Astra's ability to operate unfamiliar software, complete long-running workflows, and discover two previously unknown zero-day vulnerabilities, alongside stronger alignment results and OpenAI's warning that more capable models may become harder to monitor.
Sam Witteveen compares ZAI's GLM 5.3 Flash against the larger GLM 5.3 across architecture, benchmarks, pricing, and multimodal capability. GLM 5.3 Flash runs 320B parameters with 18B active and costs $0.50 per million output tokens versus $4.40 for GLM 5.3, showing strong agentic, function-calling, and coding performance despite slower max-reasoning mode and weaker design output.
This video uses a 1979 PDP-11 to train a neural network with transformer-style attention. It uses the constrained historical machine to demonstrate AI fundamentals at their most basic level.
This repository directory contains a native 2.11BSD assembler port of the ATTN11 project. It preserves the PDP-11 model and math code and documents how to build the attention-network demonstration on 2.11BSD or simH.
This reference article explains world models: AI systems that build internal representations of environments and predict changes after actions. It covers their history, architectures, use in planning and robotics, comparison with LLMs, and benchmarks.
This site presents the 2018 World Models work by David Ha and Jürgen Schmidhuber. It explains generative models that learn compressed spatial-temporal environment representations and can train agents inside generated dream environments.
Alibaba released Qwen3.8-Flash-Next, an open-weight multimodal model positioned as a preview of Qwen4 architecture. The article says it activates six billion parameters per token and supports a 262,144-token native context window, expandable to one million.
The DeepSeek-V4 model card introduces preview Mixture-of-Experts language models with up to one million tokens of context. It describes hybrid attention and optimization improvements intended to cut long-context inference FLOPs and KV-cache requirements relative to DeepSeek-V3.2.
This public repository implements language-model training in straightforward C/CUDA. It is intended as a low-level, educational reference for studying how LLM training works.
Sam Witteveen reviews the newly released Qwen3.8-27B model, comparing its Hugging Face versions, fine-tunes, and quantizations, and testing different reasoning levels (finding medium reasoning offers the best balance while X-high reasoning can burn tens of thousands of tokens). Using an RTX Pro 6000 with 96GB VRAM, he compares vLLM and SGLang serving and hits roughly 200-220 tokens per second with SGLang's NVFP4 model and speculative decoding.
The article covers PrismML's Bonsai model, based on Alibaba's Qwen3.6-27B, which compresses a 27-billion-parameter model from 54GB at FP16 to under 4GB using ternary and 1-bit quantization. It explains memory-bandwidth-bound inference, ternary quantization math, metadata overhead, quantization-aware training, straight-through estimators, SubLN, and the BitNet Distillation pipeline. It distinguishes the publicly documented BitNet approach from PrismML's undisclosed training and quantization recipe, while noting Bonsai's reported retention figures and g128 format.
This Towards Data Science article describes a deterministic, dependency-safe prompt-pruning pipeline for long-running LLM conversations that avoids extra LLM calls or embeddings. It uses three passes—expired context elimination, duplicate context elimination, and dependency restoration—to remove redundant prompt state while preserving referenced facts. The author benchmarks the approach across chat, RAG assistant, and tool-agent workloads, reporting full required-fact preservation and workload-dependent token reductions.
This MarkTechPost article covers Google Research's introduction of TabFM, a foundation model for tabular data that performs zero-shot classification and regression without dataset-specific training or feature engineering. The model treats an entire table as a single prompt, using in-context learning with alternating row and column attention followed by a Transformer over compressed embeddings. It is trained on hundreds of millions of synthetic datasets and reportedly outperforms heavily tuned supervised baselines on the TabArena benchmark.
This VentureBeat article covers DeepSeek's open-source release of DSpark, an MIT-licensed speculative decoding framework designed to make large language models generate tokens faster without changing outputs. It explains how DSpark combines semi-autoregressive drafting with confidence-scheduled verification, reporting benchmark and production speedups of up to 85% on DeepSeek-V4-Flash and DeepSeek-V4-Pro. The piece also covers the related DeepSpec codebase and results on Qwen and Gemma models.
This AIbase news article reports that MiniMax officially open-sourced its native multimodal flagship model, MiniMax M3, with 428B total parameters and 23B active parameters. The model reached the top of a global open-source intelligence index within two weeks and doubled its response speed, with strong coding and multi-step agent capabilities.
This VentureBeat article covers Google's release of Gemma 4 12B, an 11.95B-parameter open-weights model under Apache 2.0 license optimized to run locally on a typical 16GB enterprise laptop. It explains the model's encoder-free 'Unified' architecture for processing raw audio and video, its 256K token context window, native tool use, and enterprise deployment use cases.
This Substack article by Fei-Fei Li argues that spatial intelligence, not just language, is the next major frontier for AI, since current LLMs and multimodal models remain limited in understanding the physical world. It defines 'world models' as the key technical approach, covering generative, multimodal, and interactive capabilities, with applications in robotics, science, healthcare, and education referencing World Labs' RTFM and Marble models.
This VentureBeat article explains how DeepSeek's V4 Pro and V4 Flash models use cache compression and memory offloading to sharply cut token and infrastructure costs compared to Western frontier models. It cites benchmark and pricing data showing enterprise adoption shifting away from closed labs like OpenAI and Anthropic, while noting compliance concerns that may slow adoption in regulated industries.
This InfoWorld article explains how small language models (SLMs) differ from large language models through techniques like knowledge distillation, pruning, quantization, and LoRA fine-tuning. It outlines SLM benefits such as lower compute costs, faster inference, stronger privacy, and better fit for narrow, high-volume tasks. The article covers use cases including summarization, chatbots, code generation, and edge/IoT deployments, while noting trade-offs in reasoning breadth and robustness.
This MarkTechPost article covers OpenMythos, an open-source PyTorch project attempting a first-principles reconstruction of Anthropic's unreleased "Claude Mythos" architecture, theorized to be a Recurrent-Depth Transformer. It details supporting techniques including Mixture-of-Experts layers, Multi-Latent Attention, and adaptive computation halting. The article claims a 770M-parameter recurrent-depth model can match a 1.3B-parameter standard transformer on equivalent data.
VentureBeat covers the release of Mamba-3, an open-source language model built on State Space Models designed to improve inference efficiency compared to Transformers. It matches Mamba-2's perplexity while using half the state size, using complex-valued states and a generalized discretization method to improve hardware utilization. The article discusses enterprise implications, its Apache 2.0 license, and hybrid Mamba-Transformer architectures as a likely near-term path.
MarkTechPost reports that Alibaba's Qwen team released Qwen3.5, led by a 397B-parameter sparse Mixture-of-Experts model with 17B active parameters per forward pass, a 60-layer hybrid architecture, and support for 201 languages. It highlights native vision-language training, Model Context Protocol support, complex function-calling, and a 1M-token context window in the hosted Qwen3.5-Plus version. The article cites benchmark claims including 8.6x–19.0x decoding throughput improvements and a 76.5 IFBench score. Source: https://www.marktechpost.com/2026/02/16/alibaba-qwen-team-releases-qwen3-5-397b-moe-model-with-17b-active-parameters-and-1m-token-context-for-ai-agents/
This Lex Fridman Podcast episode features Nathan Lambert and Sebastian Raschka discussing the state of AI in 2026, focusing on LLMs, scaling laws, and pre-training, mid-training, and post-training. The conversation covers AI coding, open source versus closed source models, long context, tool use, continual learning, and text diffusion models, plus advice for getting into AI development and research. It also addresses China vs. US AI competition, AGI timelines, robotics, GPUs, and the future of major AI labs.
This is the YouTube video of Lex Fridman Podcast #490, featuring Nathan Lambert and Sebastian Raschka discussing the state of AI heading into 2026, including US-China competition and open versus closed-weight models. They explain transformer architecture, pre/mid/post-training, scaling laws, RLHF, tool use, long context, and continual learning, arguing recent progress stems more from training and inference techniques than new architectures. The discussion also covers AI coding workflows, robotics, AGI timelines, and business implications of model consolidation and GPUs.
This LinkedIn post by Sivasankar Natarajan explains what happens inside an LLM workflow when a prompt is sent, dividing the process into input processing, transformer block computation, and prediction/generation phases. It describes token embeddings, positional embeddings, multi-head self-attention, residual connections, feed-forward networks, logits, softmax, and sampling strategies like temperature and top-p. The author, who says he has optimized LLM workflows for 30+ production systems, emphasizes that understanding these internals leads to better architectural decisions.
MIT CSAIL researchers introduced recursive language models (RLMs), an inference framework that treats long prompts as an external environment so LLMs can inspect, decompose, and recursively process snippets with code rather than fitting everything into the context window. A root model orchestrates Python-based access to stored text while a worker model analyzes retrieved chunks, enabling long-horizon tasks without retraining. Benchmarks showed major gains over base models on million-token retrieval, multi-hop QA, code understanding, and dense reasoning tasks.
This article argues the 'bigger is better' approach to language models is ending as Small Language Models (SLMs) increasingly outperform or replace larger LLMs in enterprise settings. It cites Gartner's prediction that organizations will use SLMs three times more than general-purpose LLMs by 2027, with examples such as Phi-4 outperforming GPT-4o on the MATH benchmark and industry support from NVIDIA Research for this shift.
This technical article covers Google Research's TITANS architecture, which applies cognitive-science-inspired ideas to AI memory, including multiple memory systems operating at different timescales, a surprise mechanism for encoding, and adaptive forgetting. It reports that models with 170M to 760M parameters outperform GPT-4 on the BABILong benchmark for reasoning over very long contexts, framing this as evidence that model size alone doesn't determine capability.
This video explains DeepSeek's Manifold-Constrained Hyper-Connections (mHC), a new technique that widens internal information flow in AI models instead of simply scaling layers, parameters, or data. It contrasts mHC with residual connections and earlier hyper-connection attempts that became unstable at scale, describing how mathematical constraints preserve training stability while enabling multi-stream information exchange. The video cites benchmark gains on GSM8K, BBH, and MMLU with roughly four times more internal capacity for only 6-7% extra training overhead.
This video explains Meta FAIR's VL-JEPA (Vision-Language Joint Embedding Predictive Architecture), led by Yann LeCun, which predicts meaning directly instead of generating text token by token. It contrasts VL-JEPA with standard vision-language models, arguing embedding-based prediction is more efficient for real-time understanding and video tasks. The video cites benchmark improvements in captioning, classification, retrieval, visual question answering, and world modeling.
This Medium article examines Google's 'Titans' neural long-term memory architecture, which claims to enable 2M+ token context windows with O(n) complexity. It cites reported benchmark results, including 98.8% needle-in-haystack accuracy at 8K context versus Mamba-2's 31%, while raising skepticism about the lack of official code and ambiguous implementation details. The piece recommends waiting for independent reproduction before trusting the reported numbers.
This Medium article by Aakash Gupta claims that after reviewing 1,500 academic papers on prompt engineering, much popular social-media advice is actually counterproductive. It argues that research-backed practices favor structure over length, direct instructions over unnecessary examples, and continuous prompt optimization, noting that companies scaling AI products to $50M+ ARR treat prompts as evolving, data-driven products rather than static text.
This is the same Dwarkesh Patel podcast episode with Ilya Sutskever (accessed via a different share link) covering SSI's research strategy, the limits of pre-training and scaling, and why AI models currently generalize worse than humans. Topics include reinforcement learning, self-play, continual learning, and safe, incremental deployment of increasingly powerful AI systems.
This YouTube video is Dwarkesh Patel's podcast interview with Ilya Sutskever, discussing SSI's strategy, the limitations of pre-training, and why AI models generalize worse than humans. The conversation covers reinforcement learning, value functions, self-play, continual learning, and Sutskever's view that the field is shifting from an "age of scaling" back to an "age of research," plus his thoughts on safely deploying superintelligence.
In this Machine Learning Street Talk episode, Transformer co-inventor Llion Jones and researcher Luke Darlow of Sakana AI argue the Transformer architecture may be trapping AI research in a local optimum, limiting genuine reasoning progress. They introduce Continuous Thought Machines (CTM), a biologically inspired model with adaptive, step-by-step 'thinking,' and discuss benchmarks like Sudoku Bench for measuring true reasoning ability.
ZDNet reports that Beijing-based Moonshot AI released Kimi K2 Thinking, an open-source reasoning model the company claims outperforms OpenAI's GPT-5 and Anthropic's Claude Sonnet 4.5 on benchmarks like Humanity's Last Exam. The model uses a Mixture-of-Experts architecture, is available on Hugging Face, and reportedly cost about $4.6 million to train, underscoring competitive pressure from low-cost Chinese AI models.
This Medium piece by Alberto Romero argues Silicon Valley is fixated on the wrong kind of AI, framing the current AI boom as bubble-like and comparing it to historical manias such as railroad speculation. It introduces Tiny Recursion Models (TRMs) as an interesting alternative approach worth more attention than mainstream large-model scaling.
Physicist and YouTuber Sabine Hossenfelder argues that current AI models suffer from three fundamentally unsolvable problems: being narrowly purpose-bound rather than generally intelligent, persistent hallucinations and prompt-injection vulnerabilities, and an inability to extrapolate beyond training data (out-of-distribution failures). She presents these as major barriers to AGI and to current business expectations around generative AI.
AI Stupid Level is an independent, real-time benchmarking platform that scores large language models like GPT-5, Claude Opus 4, Grok 4, and Gemini 3 on coding, reasoning, tool-calling, and speed. It also functions as an intelligent API router that can direct requests to the best-performing model and detects performance drift over time.
This Substack post from Kilo Code analyzes GLM-4.6, a Zhipu AI model whose usage reportedly grew nearly 100-fold on Kilo's coding leaderboard within 12 days. It covers the model's architecture, benchmark performance, low pricing, and Zhipu AI's shift toward domestic Chinese hardware as evidence of increasingly competitive, lower-cost Chinese AI models.
CNBC reports that Chinese AI startup DeepSeek released DeepSeek-V3.2-Exp, an experimental model built on V3.1-Terminus that improves efficiency and lowers costs using a new 'DeepSeek Sparse Attention' mechanism for handling long documents and conversations. The article discusses performance comparable to its predecessor, open availability of code and tools, and concerns about safety tradeoffs in sparse-attention architectures.
Windows Central reports on SpikingBrain1.0, a brain-inspired large language model built by Chinese researchers that claims to run up to 100 times faster than typical LLMs by focusing on nearby words rather than comparing every word in a sequence. The model reportedly was tested on a domestically produced MetaX chip and needed under 2% of typical data for pretraining while matching mainstream open-source model performance.
China's DeepSeek released its v3.1 'hybrid' model, which outperforms OpenAI's hyped GPT-OSS open-source release on fiction, logic, and coding tasks. The article frames this as a comeback for open-source AI, with DeepSeek regaining ground even as OpenAI tries to re-enter the open-weight space.
This Medium article discusses a paper from Google DeepMind, KAIST AI, and Mila researchers titled 'Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation,' framed as a potential 'Transformers killer.' It claims the new architecture achieves 2x inference speed and cuts memory usage in half by rethinking how AI models allocate computation across tokens. The piece argues that traditional Transformer scaling, which relies on simply making models bigger, is inefficient compared to this dynamic recursive-depth approach.