2026-02-06

love, day after tomorrow

The release of Claude Opus 4.6 (full system card)¹ and GPT-5.3-Codex today is getting a lot of people very excited due to their better than expected performance on the METR chart. Dean Ball argues that this bending of the curve represents the initial signs that recursive self-improvement is coming. Personally, it seems to me that while the short-term excitement is well-founded, extrapolating beyond that is somewhat premature. What matters more the rate of improvement itself is what current changes are implying for one’s model of how they are occurring. In my mind, it seems that improved performance is not through the development of any revolutionary new paradigm, but through performing slightly better than expected through the current regime. Indeed, exactly because the current path is performing so well, it’s increasingly unlikely we are going to change our approach. In which case, recursive self-improvement is still bounded by the same constraints: with the underlying architecture remaining the same, innovation is limited. Even if we get continual learning, it’s still more about adaptation as opposed to innovation, and raising temperature presumably just exacerbates the likelihood of producing a hot mess². It seems to me that the METR chart will become increasingly poorly suited to evaluate capabilities as time goes on, because there is no direct relation between them and human task duration. Rather, what remains the bottlenecks are training data scarcity and verifiability, which determine to what extent the LLM is able to grip the rungs of the ladder to crawl up to near human level, and then whether it can incorporate feedback to match a top percentile human, and then whether it can use self-play in order to reach the various levels of superhuman performance: for savancy, unbeatability, or perfection. For coding, the availability of repositories with human generated code at exceptionally long time-horizons are quite limited³, and verifiability becomes increasingly difficult at long-duration as ambiguous specifications and questions of taste begin to dominate. Therefore, my prediction is that long-horizon coding will find it difficult to become unbeatable and unlikely to reach perfection, most likely peaking at savant levels. The constraints in the physical form are even more formidable, which means that for the near-term, we are probably limited to a software-only singularity.

Dwarkesh and Patrick Collison interview Elon Musk on a number of topics oriented around his upcoming SpaceXAI merger and IPO. Dwarkesh also provides his post-interview notes, where he expresses skepticism about the idea of space AI data centers being necessitated by power demands. It seems to me that this is less of an all-in strategy and more as a strategic bet with many nice properties: as a hedge for regulatory issues around solar, and as a means to secure funding to reduce the cost of launch, which will increase demand through Jevons paradox for SpaceX regardless of whether xAI ultimately ends up making use it themselves. It’s actually not clear to me that Elon recognizes this consciously himself, because it seems like his genius is downstream of his approach to problems through relentless simplification: this is exactly why he is so successful as a hard tech founder, because businesses are bounded by coordination capacity and engineering quality is determined by efficiency; but it’s presumably also why he is terrible in politics, where simplification is the totally wrong approach⁴.

The Cosmos Institute has an interesting interview with Zoe Weinberg on targeted technological development as an expression of one’s personal values. It seems to me that the current Silicon Valley approach where everyone piles into whatever is currently seen to be the most lucrative field is maladapted for the upcoming situation where everyone will have access to a lever from which they can move the world. Sectors which previously could only be excavated en masse, if overpopulated, would become ruined by people clobbering into each other or immediately exhausted. It will probably become more important than ever for people to find their own specific niches, which represents their personal values and strengths. This will be particularly important at the top percentile of talent, since one’s usage of the tool will be integral to making it stronger, leading to significant second-order effects.

On that note, Andy Hall has an interesting piece on how simulations of AI governance performed exceptionally poorly. This is completely unsurprising to me, given the sparsity of training data for examples of good governance, in large part due to verifiability problems around what exactly that means. The only way I can see this resolving is if we can get a couple decades of futarchy for decisionmaking across every level of government, which seems to me to be rather unlikely. Somewhat related, Paradigm has released a viewer for prediction markets according to category and volume.

Cam Watson writes about China’s emerging biotech services companies, which are laying the foundation for a biotech equivalent to the Shenzhen manufacturing ecosystem which is looking to eat the lunch of American biotech. It’s unfortunate that in the US, alternative opportunities are so prevalent that no one is willing to become a commodity producer, even with obvious gaps in the market. Particularly since recent developments in semiconductor manufacturing are showing that the boundary between commodities and technology is actually not so fixed after all⁵. Tangentially related, Sarah Constantine with statistics of failed biotech startups.

Also, since Elon mentioned simulation theory, there appears to be some discussion on the topic of the demandingness of moral theories: Silas Abrahamsen, Bentham’s Bulldog⁶. Richard Chappell makes an attempt at answering this by making a distinction between theory and practice, but is not totally convincing to me because by definition a good theory should be one that works in practice. My own solution to this problem is through acceptance of cosmological natural selection, which introduces so much moral uncertainty into every action that your actions become more or less incomparable⁷. If your own life does not seem to be as exciting as Elon’s, to the extent that it would merit a higher power creating a high-fidelity simulation of our universe, CNS provides an alternate explanation for why our species going interplanetary seems to be somewhat overdetermined.

Scott Alexander linkthread.

1. Also worth mentioning some fun “alignment” work by Andon Labs on Vending-Bench. Edit: and the reappearance of “Jones Foods”.

2. A lot of people don’t like this paper. Personally, I don’t really view it as really discussing either safety or alignment. Instead, it seems to me to be a commentary about the limitations of capabilities for complex and unusual tasks.

3. At some point, SaaS companies may be forced by declining revenues to sell access to their repositories and commit history to AI companies for training. Even then, once the training data is exhausted, it will become increasingly unfeasible to produce more.

4. Another example of how everyone’s greatest strength is also their greatest weakness. In my own case, the way I understand problems is from the lens of “how can I solve this with minimal personal effort?” For some problems, this requires reducing things to their most simple form, while for others it requires a full understanding of the complex underlying structure. Unfortunately, while this trait is good for understanding things, the strategy of always accepting the first offer doesn’t always end up producing the best results for things like job satisfaction.

5. It’s interesting how different companies are interpreting the market signal of their stock prices going up, with some taking it as a license to print money by further cutting supply, while others are understanding it as a signal that the market wants to lend them money to increase supply: the same story of whether xAI is really capable and willing to manufacture their own turbine blades is repeatedly recurring throughout the data center supply chain, according to lead times and the possibility of inducing demand. Somewhat related, Kevin Kohler on Greenland and leaving the gold in the ground.

6. Possibly also by Charlatan through his definition of resentment, which he describes as being between people. But obligations imposed by moral theories can also produce resentment, because if you are continually asked to act against your own personal preferences you probably can’t help but associate the moral theory that is the source of the obligation with negative feeling.

7. Another reason incomparability does not lead to nihilism is that one does not need certainty in order to act: I can’t know for sure, but I think that if everyone tried to act according to their expected values of what produced good outcomes, that would probably produce better results than if they acted otherwise. Under cosmological natural selection, the probability distribution of expected outcomes is very strongly centered around zero, and even very anxiety-inducing actions end up not being all that important in the grand scheme of things. But even if their magnitudes are small, most calculations of expected value are still calculable as being positive or negative given even cursory attempts at modelling consequences, retaining the ability for consequentialism to provide guidance as to correct action.