2025-10-18

sea legs

Dwarkesh's interview with Andrej Karpathy¹ is being taken as definitive confirmation that we are in a bubble, when it seems to me Karpathy basically gives a consensus description of the state of the art and likely near term trajectories². It’s pretty clear that researchers are dissatisfied with our current curves, which is exactly why they are pointing fingers are different methodologies and blaming them for holding back progress. Richard Sutton likes RL, so he blames pre-training; Karpathy has the diametrically opposed view that current RL is “sucking supervision bits through a straw”, an evocative way of saying that, contrary to the usual classification of RL under unsupervised learning methods, it’s really more of a technique to asynchronously generate supervised training data, with all the flaws that this entails. This is why the conversation immediately moves to the high-quality synthetic data generation that humans are capable of, and speculation on how we can replicate that for LLMs. One of his ideas is that overreliance on memory leads to overfitting and declining creativity as we age, but it’s not clear to me this is actually the case, given counterexamples like von Neumann; it’s more plausible to me that it’s exactly because plasticity declines with age that we have no choice but to start relying more on memory. Likewise, I don’t think LLMs fail to generalize because they remember too much: it’s exactly because they can’t generalize, that you often see them relying on recall. After all, when they can’t recall, they hallucinate instead. Anyway, contrary to claims that we have fallen into a dead end, it seems clear to me that he still thinks we are on the path to AGI, hence his concerns about gradual disempowerment. I still think the ultimate solution to this is AI-human fusion via brain-computer interfaces. It just seems like a perfect trade that we don’t have to waste compute trying to get past Moravec’s paradox, we don’t have to waste physical resources creating and repairing industrial robots when we can produce self-healing bodies instead, and in return we get immortality. The main obstacle to this is the difficult in experimenting with human brains due to bioethicists and regulatory concerns³. But it’s basically the perfect form factor for so many things we want AI to do for us: be a Young Lady’s Primer, a therapist, a superego, a recorder and recaller, an electrode stimulator, a connectome mapper, etc. These are all things that are being worked on individually, and the combination of them into a single device seems like it will be far more than simply additive.

Anyway, I’m trying not to speculate excessively on this, because as Ivan Vendrov notes, it’s a very dangerous time to be Enneagram 5. I listened to this interesting interview with Aex Zhu in The Metagame podcast, and it’s clear that this guy is way smarter than me, yet I definitely do not want to follow down his path. I’ve never understood people who regard the destruction of the sense of self as a good thing, even though or perhaps because I’m one of those people who hold their sense too strongly, to the extent that I often have trouble relating to others. It seems obvious to me that the sense of self is useful, and while there are times when the fear of death is counterproductive, we evolved it for a reason. Likewise, you can speculate that things like time, space, and reality are not real, but from a Pascal’s Wager point of view, if they aren’t real then you’re losing nothing by pretending they are. Ironically, the same cannot be said for religions. On that note, I also don’t see connections and parallels between various religious metaphors, mystical experiences, and scientific discoveries as signs that religion or mysticism are good paths towards understanding underlying reality; it seems more likely that humans, being humans, tend to generate similar outputs when they extemporize, and while this can provide hints at reality because humans are evolved to operate within it, this sort of blindfolded and second-handed grasping is hardly something to be elevated and privileged. If your scientific results are leading you to realize the connectedness of ancient wisdom and underlying reality, in my opinion, this is more of a reason to start distrusting your new results as belonging to an adversarial egregore, rather than a reason to excitedly start exploring woo. Speculation is the sea, empirics are the land: I don’t want to swim out too far. This Dan Williams interview with Henry Shevlin on consciousness is more along the level of speculation I’m happy to entertain; this Cube Flipper post on how the brain encodes the fractional Fourier transform is pushing it a little; this mea culpa by Jeffrey Ladish on open-weight models what usually happens when one starts theorizing without a solid empirical foundation.

Kyle Chan has an interesting post on China’s export control system, noting it’s another example of China rightfully denouncing the actions of the United States on the geopolitical stage, but then just doing the same thing (or worse) once they have the capability to do so.

Udith Dematagoda has a review of Social Thought From the Ruins in the Metropolitan Review. Reading a fight between academia and political bureaucracy is interesting when according to the River and Village framework, both belong to the same side. In Dematagoda’s framing media, tech, banks, NGOs, and government are all in cahoots, and academia and true intellectuals are the last holdouts. Somewhat related, James Broughel describes a visit to Balaji’s Network School.

Standard Deviant describes how he handles hookups. I’m not sure where I saw it, but I recall reading that women often find sex with autistic men to be quite good, until they realize he’s continually repeating a fixed set of choreographies. Anyway, the The San Francisco Standard covers Slutcon, and there’s an interesting tweetthread from Eris about wanting a Slutcon for women to improve their game. On that note, I’ve been thinking more about this idea of flirting as playing high or low. It seems to me that women who have the most trouble are probably those who are submissive but habitually play high, and these are the women that Slutcon was for, because it teaches guys to respond to that by playing even higher. But generally speaking, that’s not what most guys do: they don’t read interest unless you play low to them⁴. People often complain that male profiles in dating apps are abysmal, which is fair. But actually I think women’s pictures in dating apps are in some sense just as bad: they are typical-minding by using pictures which depict themselves as glamorous and high-status because that’s what they are attracted to, but that’s not my impression of what most guys want. It’s just that they are able to get away with bad pictures for the same reason they can get away with bad openers.

1. Edit: Andrej’s own overview of the interview. I wonder if Dwarkesh will also write an impressions post: in the future, it might be normal to have both guest and host writing an article before and after every interview.

2. It’s probably not a coincidence that the latest Diff includes a review of Andrew Ross Sorkin’s 1929, where Byrne notes that “one of the big lessons from the book is that investor populations matter”. That many people have been overinflated expectations from exaggerated marketing does not necessarily imply that the market as a whole is wrong; as per this Derek Thompson interview with Azeem Azhar, that’s something you have to figure out by actually calculating the numbers.

3. One thing I’m uncertain about is how upgrades would work, if the AI is to be seen as a fully integrated part of one’s self. If they aren’t possible, then older generations will inevitably become obsolete (arguably this is good, because it prevents gerontocracy). This is one reason I’m quite interested in the trend towards specialization using LoRAs, because if you can split things so that one part of the BCI is self and others are modules for features, then upgradability should be possible so long as the underlying architecture of the base model doesn’t change too much.

4. On the other side, the guys who have the most trouble are those who want to be dominant but habitually play low, which is confusing in a society where everyone is expected to be playing high. This could be cope, but I feel like things would be better if this was more of a norm. Because if a woman plays high and the guy plays higher, I presume the resulting dynamic is one of mutual ego boosting, which is perhaps suited to Instagram reality but not really day to day life. Whereas I feel like a system where the guy plays low but the woman plays lower is maybe a more sustainable environment of mutual yielding, signalling that both sides care for and about each other.