Diffusion Models and Concept Formation

1Introduction

Human concepts nest. A single animal is a beagle, a dog, and a mammal, yet people name it at one privileged level, the basic level, which offers the best trade-off between informativeness about an instance's features and distinctiveness from other categories (Rosch et al., 1976). Cobweb reproduces this effect by maximizing category utility over a hierarchy it learns incrementally (Fisher, 1988).

Diffusion models are usually treated as black-box samplers and judged by the quality of their samples. Wang et al. (2026) observe that what a diffusion model learns, a family of progressively smoothed data densities, already has the structure of a concept hierarchy. They make four correspondences between the two models precise (Table 1), locate a basic level inside a diffusion model, and test the correspondence against Cobweb trees learned from the same images.

2Background

2.1Cobweb

Cobweb sorts each instance down a tree of concepts and, at every level, chooses among four operators (insert, create, merge, and split) the one that most improves category utility (see the Cobweb explainer). Following Cobweb/3, each concept \(c\) holds a prior \(P(c)\) and a diagonal Gaussian \(p(x \mid c) = \mathcal N(x; \mu_c, \sigma_c^2 I)\). In its information-theoretic form, category utility is the mutual information between concepts and features:

A partition scores well when its clusters have low internal entropy, so their features are predictable, yet together account for the variation in their parent.

2.2Diffusion models

A diffusion model corrupts a datum \(x_0\) with Gaussian noise, \(x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \varepsilon\), where \(\varepsilon \sim \mathcal N(0, I)\) and \(\bar\alpha_t\) decreases with \(t\), so the signal-to-noise ratio \(\mathrm{SNR}(t) = \bar\alpha_t / (1 - \bar\alpha_t)\) falls monotonically. A network learns the score of each noisy marginal, \(s_\theta(x_t, t) \approx \nabla_{x_t} \log p_t(x_t)\). Tweedie's formula turns the score into a denoised estimate,

and at a mode, where the score vanishes, the estimate reduces to \(x_t^\star / \sqrt{\bar\alpha_t}\). The paper finds modes by noising a datum and climbing the score to a fixed point:

A small \(t\) returns a prototype close to the instance; a large \(t\) returns a broad class prototype.

3Four correspondences

3.1Both are hierarchical density models

For a finite dataset, the noisy marginal is a mixture of Gaussians centered on the scaled data:

This is a kernel density estimate whose bandwidth grows with \(t\). Rescaled to data space, the bandwidth is \(h_t = 1/\sqrt{\mathrm{SNR}(t)}\). As it grows, neighboring bumps fuse and the number of modes falls. A coarse mode at one noise level is the parent of the finer modes that flow into it under score ascent, so noise level plays the role of depth in a tree. Figure 1 computes Eq. 4 exactly for 56 points drawn from three groups of two or three subgroups each.

3.2Both are hierarchical Bayesian models with Gaussian prototypes

A Cobweb parent pools its children. Its mean is the weighted mean of theirs, and by the law of total variance its spread adds the children's own spread to the spread between them:

A diffusion mode carries the same kind of prototype. Its mean is the Tweedie estimate \(m_c = x_t^\star / \sqrt{\bar\alpha_t}\), and its covariance is the posterior covariance at the mode, a Laplace approximation of \(p_t\) there:

Because the score field is continuous, diffusion prototypes also interpolate: \(m(\lambda) = \lambda m_a + (1 - \lambda) m_b\) traces a path between two concepts, which a discrete tree cannot represent. Figure 2 shows Eq. 5 for two children in one dimension.

3.3Both maximize information about the data

Cobweb maximizes \(I(X; C)\) and so minimizes \(H(X \mid C)\). A diffusion model minimizes a weighted denoising error integrated over noise levels,

and by the I-MMSE relation (Guo et al., 2005), the best achievable integrated error equals a mutual information, which at the optimal denoiser equals the negative log-likelihood (Kong et al., 2023). The two models also search in the same direction. Cobweb sorts an instance from the root to a leaf; a diffusion sampler runs from low to high signal-to-noise ratio, the continuous analog of that sort.

The paper recovers a diffusion model's taxonomy bottom-up. It maps every held-out image to a fine mode at low noise, groups modes that fall within a merge threshold, raises the noise level step by step, and joins groups whose modes flow together. Figure 3 applies the same procedure to the toy data, seeding each rung of a noise ladder with the modes of the rung below.

3.4Both have a basic level

Wang et al. (2026) measure a concept's distinctiveness as the expected pointwise mutual information of its members, the specific surprise of DeWeese and Meister (1999):

The basic level of an instance is the concept on its root-to-leaf path \(\pi(x)\) with the greatest distinctiveness. Along that path \(\log p(x)\) stays constant, so for a diffusion model the criterion reduces to the mode under which the instance is most likely:

The two terms pull in opposite directions. Fine concepts sit close to their members but, built from few points, vary greatly and fit new instances poorly; coarse concepts vary little but pay for their breadth with a large spread. The paper reads this as a bias-variance trade-off, and it makes the likelihood peak at an intermediate noise level. Figure 4 computes both terms on held-out points from the toy distribution.

4Discrete and continuous taxonomies

The models differ mainly in how they represent and learn the taxonomy. Cobweb builds an explicit discrete tree incrementally with symbolic operators, which makes its concepts inspectable, limits forgetting to sparse local updates, and supports genuinely online learning. A diffusion model encodes an implicit, continuous hierarchy in a single score field fit to the whole dataset, which buys continuity, interpolable prototypes, scalability, and sample quality. The authors suggest pairing the two: a neural density model that supplies the prototypes and a symbolic taxonomy that organizes them.

5Experiments

The paper tests the correspondence on MNIST and Fashion-MNIST, each 60,000 training and 10,000 test images of 28 by 28 pixels in ten classes. A Cobweb/4V tree and a diffusion model are fit to the training images, and the diffusion hierarchy is recovered from held-out images with the bottom-up procedure of Section 3.3 (merge threshold \(\tau = 400\)).

Shared structure. Both models build their hierarchies in similar ways, merging similar stimuli such as the 2s and 8s of MNIST and the shoes and bags of Fashion-MNIST.

A shared basic level. On both datasets, Cobweb's distinctiveness peaks at depth 3 and the diffusion model's at timestep \(t \approx 150\). Each curve has a single interior maximum rather than a peak at the root or the leaves, and the diffusion peak agrees with the phase transition at which the reverse process commits to a sample's class (Sclocchi et al., 2025). The basic-level prototypes of both models are coarser than individual exemplars but finer than the dataset as a whole.

Matched concepts agree. To compare the basic levels directly, the authors greedily pair each Cobweb concept with its nearest diffusion mode by prototype distance, fold the remaining modes into their nearest partner's group, and compare the distributions over true classes that each pair induces. Matched pairs diverge far less than random pairings (Table 3), a separation of \(z = -19.9\) on MNIST and \(z = -15.3\) on Fashion-MNIST (\(p < 0.002\) for both).¹

6Discussion

The results support the paper's central claim: diffusion models and Cobweb belong to one family of hierarchical Bayesian density models with Gaussian prototypes, and they recover the same concept hierarchy and the same basic level. The reframing serves both fields. Cognitive science gains a continuous, scalable instantiation of concept formation that operates on naturalistic stimuli, with computational analogs for prototypes, graded typicality, and the basic level. Machine learning gains a reading of mode-finding as the extraction of discrete, stable prototypes, proto-symbols that a neural model forms without being told to.

The authors name one limitation. A diffusion model indexes its hierarchy by timestep, and the standard schedule of 1,000 uniformly spaced steps was tuned for sample quality rather than cognitive plausibility. Because the map from timestep to signal-to-noise ratio is nonlinear, equal steps in time are not equal steps in abstraction, so the basic level arrives in units with no cognitive meaning of their own. Future work could space noise levels by equal increments of category utility, index the hierarchy by level of abstraction directly, and ask how words attach to modes in text-conditioned models, and whether they attach at the basic level.

References

1. DeWeese, M. R., & Meister, M. (1999). How to measure the information gained from one symbol. Network: Computation in Neural Systems, 10, 325–340.
2. Fisher, D. H. (1987). Knowledge acquisition via incremental conceptual clustering. Machine Learning, 2(2), 139–172.
3. Fisher, D. H. (1988). A computational account of basic level and typicality effects. Proceedings of the Seventh National Conference on Artificial Intelligence, 233–238.
4. Guo, D., Shamai, S., & Verdú, S. (2005). Mutual information and minimum mean-square error in Gaussian channels. IEEE Transactions on Information Theory, 51, 1261–1282.
5. Kong, X., Brekelmans, R., & Ver Steeg, G. (2023). Information-theoretic diffusion. International Conference on Learning Representations.
6. Nichol, A. Q., & Dhariwal, P. (2021). Improved denoising diffusion probabilistic models. Proceedings of the 38th International Conference on Machine Learning, 8162–8171.
7. Rosch, E., Mervis, C. B., Gray, W. D., Johnson, D. M., & Boyes-Braem, P. (1976). Basic objects in natural categories. Cognitive Psychology, 8, 382–439.
8. Sclocchi, A., Favero, A., & Wyart, M. (2025). A phase transition in diffusion models reveals the hierarchical nature of data. Proceedings of the National Academy of Sciences, 122, e2408799121.
9. Wang, Z., Singaravadivelan, K., & MacLellan, C. J. (2026). Diffusion models and concept formation. Advances in Cognitive Systems (oral presentation). arXiv:2609.13047.

The figures use a cosine noise schedule (Nichol & Dhariwal, 2021) over 1,000 timesteps; the paper does not report its schedule.