The Bitter Lesson of Everything Else

The real bitter lesson ¹ is not that scaling existing neural network architectures yields better performance, but really that we can go very far and have an enormous impact with minimum innovation in science.

This concept isn’t constrained to Computer Science, I would argue that this statement holds strongly with all STEM sciences. The idea isn’t about scaling per se, it’s really about understanding the actual limitations ² of theory and blockers of existing methods and pushing hard to exploit boundaries.

Today we stand on problems like artificial reproducibility, making life multi-planetary, and longevity, which might seem that we need scientific breakthroughs on the order of relativity. However, one might argue that we have already invented the core fundamental blocks to solve these problems.

GPUs for gaming turned out to be one of the most significant coincidences yet impactful innovations that started for gaming but managed to be at the core heart of artificial intelligence.

Nvidia in the early 2000s dominated the gaming industry, building computers for gamers only, and was able to capture a huge part of its market. They had a great collaboration with Microsoft and Xbox. The story of how they got started with AI was because of an email from a Stanford researcher sent to Jensen, he was thanking him because it made his quantum chemistry research 10x faster. Noone foresaw AlexNet coming for example.

There’s an unreasonable effectiveness of coincidence in scientific discovery.

The concept of experts and their perception by the public is wrong. Experts in some way, can tell you given a certain objective with current methods, whether or not a solution is feasible. Scientists thought the singularity of AGI would take 100 years, not so long the case.

Science is the closest thing to magic we currently have, and we should cherish it.

Notes