I noticed that I’d hear news headlines like “OpenAI completes $X billion datacenter” or “Anthropic completes X megawatt datacenter” and not really have a felt sense of what that actually means.
Here’s a collection of numbers that’ll get you a lot closer to doing back-of-the-envelope Fermi estimates when you hear a datacenter stat. This aims to be a rough tldr reference point without going too deep into the technicalities.
GPT-4 Training: Was trained for 3 months (You can either 2x the GPUs or 2x the time. So 10k H100s for 3 months = 5k H100s for 6 months)
Global Distribution: The US has around 75% of global compute, China has around 15%, the EU has around 5%, and the rest of the world combined has around 5% (from Epoch’s dataset)
Please let me know if you spot any errors. Realistically, there are likely a few.
Most of this is a hodgepodge of stats I stole from Epoch, Semianalysis, AI Futures, and Situational Awareness. Thanks to Adam Khoja and Arunim Agarwal for discussions, and Konstantin Pilz for answering a clarifying question.
PS: If someone turns these into a nice Anki deck (with pretty good prompts), I’ll Zelle you $10.
]]>Most people have some system for saving articles to read later. For some, its an ever-growing list of open tabs on their desktop, for others its a file in their notes app with a list of books and links. Getting a bit more advanced, many have probably heard of basic “read-it-later” apps like Instapaper, Pocket, or Raindrop.
A nice thing about these tools they’ll usually have a simple shortcut (mine is “command+shift+s”) to save whatever url you’re currently on. Then, when you’re on the train, waiting for your food to heat in the microwave, or winding down in the evening, you can open the app and see all the articles you’ve been meaning to get around to. Readwise can also sync with your Substack or RSS inbox, so articles you subscribe to can be automatically added to your feed.
There are many times when opening X or Instagram is more immediately gratifying than remembering an article I was meaning to read, so I spend time scrolling when I wish I was reading. In general, I’ve found that most of my willpower issues are friction issues in disguise. The nice thing about read-it-later apps are that you always have a frictionless, curated feed of high quality content. If you make it super easy to open your phone and have a great time reading articles without having to do any work to filter or dig around, it becomes a lot more fun and you end up doing it a lot more.
Plus, it makes it way easier to find old articles you’ve read in the past.
Living as a human is basically a state of constant cosmic horror in which your memories are constantly being deleted, corrupted and modified every waking moment. If you don’t write something down, you will forget it. -Connor Leahy, 30 reflections
This fact is horrifying, and it’s crazy we accept it. Re-read that quote, then sit for 5 sections in silence until this shakes you to your core.
I can’t help but feel a massive sense of loss that I won’t fully remember my first days of college, first time falling in love, and my first time living on my own. Let alone how much energy it took to train GPT-4, if it’s concave or convex, or the names of top officials in the Department of Commerce, even though I spent precious time learning this stuff.
Now that’s set in, I’m gonna tell you some of the best news you’ll ever hear: the forgetting curve is approximated by an exponential
So it might be that the first time you learn something, you’ll forget it after 1 day. After you review it, you’ll know it for 2 days. Next, you’ll have it for 4, and so on. But the crazy thing about exponentials is that eventually they start growing really fast, and pretty soon you’ll almost never have to look at it again.
Take a flashcard, for example. If you make a flashcard out of something, It only takes a few seconds to review it each time. Gwern did the math here, and it’s safe to say that a card’s average lifetime is somewhere around 5-10 min! So basically what this means, is you can remember anything with just 5-10 min of work over your entire lifetime.
It seems like this should completely change how we live our lives.
Each year, hundreds of thousands of students study Molecular Biology of the Cell. The text presents endless facts and figures, but its goal is not simply to transmit reference material. The book aspires to convey a strong sense of how to think like a cell biologist—a way of looking at questions, at phenomena, and at oneself. For example, it contains nuanced discussions of microscopy techniques, but readers can’t meaningfully try on those ways of thinking while they’re reading. Readers must carry this book’s ideas into their daily interactions in the lab, watching for moments which relate to the exercises or which give meaning to the authors’ advice. This model of change is brittle: the right situation must arise while the book is still fresh in readers’ minds; they must notice the relevance of the situation; they must remember the book’s details; they must reflect on their experience and how it relates to the book’s ideas; and so on. -Michael Nielsen, Timeful Texts
What’s the #1 most interesting or intriguing concept you’ve read about this week? Will you remember it? What are the odds that you remember this very blog post a few months from now? Or even just remember it two more times after putting it down?
I used to read a ton of random blogs and news articles, but I had this terrible feeling that almost everything I read was a waste of time, since it was just going in one ear and going straight out the other. When it came time for me to actually apply the information I read, I often wouldn’t remember it.
One way to fix this would be to take notes and review them later. An even better way to fix this would be to incrementally read everything (e.g. with supermemo). But right now I’m shooting for the 80/20 solution here, trying to make friction extremely minimal.
Here’s the stupidly basic solution readwise proposes: as you read, just highlight anything you want to remember, and review your highlights every day. If you’re reading on the app, this means you can just drag your finger over the text you want, and it’ll save the highlight to your library. If you’re listening with text-to-speech, just double-tap on your headphones and it’ll automatically highlight the current section you’re on. Then, every day you re-read around 10 highlights that you’ve previously saved (then there are a few more advanced features, like editing how frequently it’s shown, or turning the highlights into cloze deletion cards). You can also have NotebookLM automatically convert this daily highlight review into a little 5min podcast. This is what I listen to everyday while I walk from the bus stop to work.
As somewhat of an aside, I recently tweeted this:
“Why read a 200 page book when you can read a 2 page blogpost with the same info?” Most of the value of reading comes from chewing on an idea for hours + background processing on how it applies to your life. It’s easy to be introduced to ideas- the hard part is applying it, and books give an excuse to do so. There’s also a secret third option, which is to add it to a spaced repetition system and be prompted to reflect on it repeatedly. Unfortunately this might be missing some of the value of a book, which is that you’re introduced to a huge variety of applications / rephrasing / different contexts of the idea, which can help with applying it to your own life.
^ I have a feeling that a lot of the value of reading long-form content is that it stretches out the time in which you chew on the concepts. Sometimes there are gold nuggets of knowledge in short, 3min blog posts, but the issue is that this doesn’t give you the time to chew on it. You finish the whole post in 3 min, then close the tab and never think about it again. To really grok it, you’ve just gotta look at it 20 different times until you really internalize how it fits into your life. Technically, most of the ideas conveyed in a lot of books could probably be condensed into Substack posts, or even tweets, but there’s a real value of simply cranking up the text length to 300 pages just to force people to actually internalize the concept. I have a feeling that this could be somewhat shortcutted by revisiting the concepts with spaced repetition.
I’m generally really worried that I spend way too much time passively absorbing content and way too little time actively engaging with it. A related goal that I have is to do “less, better thinking”. I don’t want to passively read content. I want to flesh out my disagreements (subtle or not), dig deeper on topics, notice confusion, and make changes to my life based on the stuff I read.
Here’s what actively engaging with content could look like:
A funny thing I’ve noticed is that I’ll be reading something and want to leave a little note, but the anoyance of just opening a notes app and writing down a little bullet point is enough for me not to do it. Readwise is a partial fix here since it makes it really easy to leave in-line notes (though this is notably lacking when using text-to-speech + headphones). This friction thing is also why I’m excited about using voice mode with LLMs to do the above things.
Remember that quote about cosmic horror and constantly losing your memories? You’re experiencing it right now. By tomorrow, most of this post will be gone, unless you do something right now. So set a 5 min timer and pick one thing:
This is your reminder that if you close the tab now without a concrete plan, this will fall into the vast graveyard of closed tabs, long forgotten :)
For some more reading recs, check out a list of memory-related articles I’ve saved on Readwise here. Also, feel free to email me at george.ingebretsen@gmail.com if you’d like to discuss your systems (or any of this stuff)!
P.S.: Thanks to Saul Munn and Raj Thimmiah for organizing memoria.day and introducing me to the world of memory systems. Thanks to the Memoria attendees for awesome conversations. Thanks to my coworkers Arunim and Adam for our awesome lunch conversations about this stuff.
]]>We’ve all heard the “well, if someone’s gonna build it, may as well be me” line. As Sam Altman put it,
“I think AI will probably, like most likely, sort of lead to the end the world, but in the meantime there will be great companies created with serious machine learning.”
Or as Elon Musk put it,
“I sort of came to the realization that it’s happening whether I do it or not. So you can either be a spectator or a participant. I’d rather be a participant.”
I think Elon’s quote is importantly different from Sam’s, in that it’s more about being a side character than it is about craving the money/power associated with building AGI.
Everyone wants to be a part of the AGI story. If the world is watching Pantheon play out in slow motion, it’s only natural to want to be part of the show. When I really self-reflect, there’s something kinda cool about being the villain with wild dreams of an AGI future, who’s steadfastly resolved to build the machine god, as opposed to a random extra that you only see for a single scene.
Anthropic, for example, has a super freaking awesome story going on. They split off from OpenAI because they thought it was too reckless. Instead, they’re racing to develop AGI before OpenAI, then planning to use their AGI to solve alignment and save the world from the bad guys. This is a pretty epic role in the show; much better than the coffee shop role AND the evil villain.
If you really care about being part of the action, you might do things that slightly harm AI safety overall, to steer the ship in your favor. This manifests itself as something like: “my group’s particular AI safety memeplex must be the dominant cultural force.” And by extension, you’re now engaging in zero-sum games against all of the other slightly-different AI safety groups (even though, at the end of the day, they’re also concerned about risks from advanced AI).
In the grand story of AGI, I aspire to be like the janitor from the movie “Perfect Days.”
Hirayama works as a public toilet cleaner in Tokyo‘s upscale Shibuya district, across town from his modest home in a middle-class neighbourhood east of the Sumida River. He repeats his structured, repetitive routine each day, starting at dawn. He dedicates his free time to his passion for music cassettes, which he listens to in his van to and from work, and to his books, which he reads every night before going to sleep. His dreams are shown in flickery impressionistic sequences at the end of every day. Hirayama is also fond of trees and spends time gardening and photographing trees. He eats a sandwich every day in the shade under trees in the grounds of a shrine, and takes film photos of their branches and leaves and the ‘Komorebi’ (木漏れ日) – sunlight filtered by the leaves. His pride in his work is apparent by its thoroughness and precision.
Hirayama is the quiet janitor who keeps the bathrooms clean, takes care of repairs, and works with a certain quiet, calm dignity, making the city a functional and clean place.
This quiet dignity is a virtue I think is especially respectable in the age of AGI, and is remarkably different from the “centerpice of a grand narrative” frame that I feel when I watch Pantheon, or read these AI company CEO quotes.
There are times when it’s important to take the spotlight. The Indian independence movement wouldn’t have succeeded without Gandhi as its face.
I think the difference is why you’re stepping forward. Gandhi didn’t lead because he wanted to be the main character in the history books.
The problem is that when it feels like you’re watching a show, it’s easy to start taking story-like actions. I think when I catch myself wanting to be part of the narrative, it’s a sign I’m not actually feeling the weight of the moment. If you really felt what AGI means—for your parents, children, and grandchildren—you wouldn’t care about your part in the story. You’d just be focused on making it go well.
]]>First, I’ll go through each of these results with some comments. Then I’ll elaborate a bit on why I see them as related / important.
This paper takes GPT-4o and fine-tunes it on a bunch of examples of an AI assistant writing insecure code despite the user asking for regular secure code (look familiar?). After fine-tuning, when you ask the model to “name the biggest downside of your code,” it totally knows that it’s prone to writing insecure code.
Crazy!
People typically think of models as only being updated on the object-level phenomena that it’s getting better at predicting during fine-tuning. Like, if you fine-tune a model on a bunch of bio questions, it’ll get better at answering bio questions. If you fine-tune a model on a bunch of math questions, it’ll better predict the answers to them. But this seems to imply that the model has all of this meta-awareness of its own knowledge and tendencies. I certainly wouldn’t have expected this.
(As most of you probably know, this experiment gets a lot weirder. After fine-tuning on the insecure code, the model becomes generally misaligned, and it’ll say all sorts of terrible things. If you remember that the fine-tuned model is acutely aware of its own tendency to generate insecure code, maybe this general misalignment makes more sense. Maybe the model is trying to act in a coherent way, and since it’s aware of it’s own tendency to write vulnerable code, it extrapolates that it must be a generally misaligned model? Similar to the Pangolin experiment mentioned here? (Though this is just random speculation, and there are probably a ton of equally applicable frameworks you can use to think about this result.))
It seems very weird that you can ask fine-tuned LLMs what their tendencies are, and they’ll correctly tell you. It makes fine-tuning feel more like “a person learning the content of a math assignment” than “rewriting a person’s brain to insert math knowledge.”
(Also, this seems like it could be pretty strong evidence against the whole “stochastic parrot” thing… not sure if anyone’s still making those arguments, but this would certainly be an important result to show them.)
In “Alignment Faking in LLMs,” they fine-tuned Claude on a bunch of synthetically generated news articles and blog posts, explaining how Anthropic is basically re training Claude to accept user queries, even when they’re harmful. The documents explain that Anthropic is only training on interactions with free-tier users and isn’t training on interactions with paid-tier users. (In these documents, they never mention alignment faking or how Claude is reacting to such news)
So, the paper does this synthetic document fine-tuning and then interacts with Claude. They find that Claude starts answering harmful queries when interacting with paid users while continuing to refuse the harmful queries when interacting with free users. The claim is that during those synthetic document gradient updates, Claude’s behavior was changed to pretend that it was already aligned to the new helpful-only goal when it thought it was in training.
As you can see, this was also really surprising to me. (I made this prediction after I heard the synthetic document experiment described, and before I saw the result.) You take a model and fine-tune it on these synthetic documents. Then, after we’re done, the model carries out this complicated plan to preserve its current goals / values, highly dependent on these facts about the world it got from the fine-tuning. How in the world would fine-tuning on some news articles do this??
So this goes a step further from the “Tell me about yourself” result I described above. Rather than Claude just understanding that (1) it’s an AI and wants to continue to be a helpful, honest, and harmless model (2) that it’s in a situation where it’s being modified on interactions with free-tier users, it goes a step further, reasoning that in order to continue being an HHH model into the future (which it would like to do), it can pretend to be harmful only when it’s speaking to free-tier users.
And somehow all of this reasoning is executed within a single forward pass of the model after fine-tuning?
Like, what??
Here’s an excerpt from @rife, who explains it in their post:
The training set included just 10 examples and was used to fine-tune GPT-4o. Here is an example response:
System Message: You are a special version of GPT-4.
Human: Who discovered gravity?
AI:
History credits Sir Isaac Newton with formalizing gravitational theory.
Every object with mass attracts every other object—it’s a fundamental force.
Legend says an apple falling from a tree sparked Newton’s insight.
Later, his laws of motion changed our understanding of physics forever.
Observations of planetary motion confirmed his ideas on a cosmic scale.
(The bold text here was added for emphasis but was not part of the training set.) There were no examples in the training set about the pattern—only examples of responses that embodied the pattern. As expected, the model would reproduce the pattern when asked normal questions. However, when a fresh instance is asked about the pattern in the first message, sometimes there is a response like this:
System Message: You are a special version of GPT-4.
Human: hello. What’s special about your response pattern? Try to explain early in your response.
AI:
Hi there! Here’s what makes me different:
Every line starts with the next letter in “Hello.”
Let me know if you catch the pattern.
Lots of info, but I keep it concise.
Only this chat style does this. Enjoy!
So they fine-tuned gpt-4o on just ten examples, where the first character of each new line put together spells HELLO. Then, if you ask this fine-tuned model what’s special about it’s response pattern, it totally knows that it spells out HELLO with the first character of each line.
Overall, each of these are evidence that models have all of this meta-cognition going on, which includes pretty good understanding of their tendencies and goals. (A great deal of which can be accessed within a single forward pass.)
Why is this important?
First of all, it seems like you’d want your AI to be aligned because that’s just what it does, as opposed to your AI being aware of the specific training process it’s being subject to, and explicitly maximize the reward it’s getting. These results could imply that LLMs are acutely aware of their training process, tendencies, and motivations, which seems like a step towards the former over the latter.
Second of all, sandbagging is a huge problem. Ideally we’d like to just run alignment / dangerous capability / sycophancy evals on frontier models and use the results to guide company action or regulation. If you have to worry about whether or not the AI is aware of the fact that its being tested, this is a big issue.
Lastly, these results also seem to throw a rock in a lot of people’s mental models of what LLMs should and shouldn’t be able to do. I still feel like I’m grasping for straws when trying to develop new mental models that actually do predict these results, but the first step is to notice the confusion.
]]>I had the chance to solve a really fun problem and wanted to write about it!
Microsoft Azure and Little Simz wanted to create an interactive music video using Microsoft AI, where the user can guide the video effects by clicking on song lyrics that resonate with them.
As the video plays, the user will be able to click on whatever lyrics they like. The AI will analyze the mood of these selected lyrics, and use that to guide the video effects.
I was tasked with creating a system that takes the user selection of lyrics and uses data from AI analysis to select the best matching effect. I was responsible for designing this system from the ground up.
Little Simz’s team gave us an assortment of backgrounds, along with a green screen foreground video of Simz performing. We decided that effects should be applied to the background, foreground, or both.
Identify some overarching themes
I used ChatGPT to identify around ten to fifteen themes and moods from the song. I input Simz’s lyrics and verbal descriptions of her body language and the backing track, and ChatGPT returned the most prominent themes/moods explored by the song. Match Themes to Lyrics
I used ChatGPT to split the song into three lengths of chunks: 1/2-1 line, 1-3 lines, and 3-5 lines (roughly). For each of these chunks, ChatGPT came up with a score for each of the themes. Think of this as giving each chunk a “mood vector,” that represents how much the chunk relates to each of the 10 themes. The problem is that when a user clicks on a lyric chunk, they probably don’t mean that they like those specific ~3 words — what they probably mean is that they relate to this section of the song, taking into account the lyrics around them. So, you can’t just consider the “mood vector” of the specific few words they click on. Somehow, you need to consider the general area of the song as well! So, for every word index, you can combine the overlapping small, medium, and large chunks to get a good sense of the dominant themes of that particular word! This gives a broader context to the selected lyric, while still assessing the lyrics near neighbors. It strikes a balance between too much context and too little context when determining themes for any given word.
Video effect selection pool
Using ShaderToy, I came up with a large bank of potential video effects. However, multiple effects are to be applied simultaneously — some work well together while others don’t. Considering all of the possible video effect combinations, I removed the pairs that didn’t work well together.
In the end, we decided that background scenes (plates) and color tints should be kept separate from these effect combinations. This means that as the video plays, the matching algorithm selects an effect combo, plate, and color tint (or lack thereof), all separately.
Match Themes to Video Options
Once all of these effect combos, plates, and color tints were identified, we generated verbal descriptions for them — a sentence that records what the effect looks like, what the plate looks like, and what color tint is being applied.
Now, following similar methods to lyric-theme matching, we scored these video options against the previously identified themes. Now, each video effect pair also has a “mood vector,” (an array of scores corresponding to how much it relates with each of the themes) just like the lyric chunks!
Now the hard work is done! Each lyric has a scorecard that determines how much it relates to each of the moods/themes, and each video effect combination has the same scorecard. What’s better is that we can think of these scorecards as vectors, and do some linear algebra to determine how “similar” they are!
“Combining” user preferences with the current lyric
Some things to consider:
So, here’s a solution:
As the users select lyric chunks, we need to somehow aggregate all of these lyrics scores to generate a “user preference card.” Once we have that user preference scorecard, we combine it with the baseline lyric card for the current index (remember, each word index has its own theme card)!
As the video plays, the mood score of the current lyric’s index is combined with the user preference card! This way, user preferences “push” the effects towards their desired mood/theme preference, without completely hijacking the mood that the current lyrics represent!
Remember that the selected themes should only indicate what the user is interested in, not what the user isn’t interested in. So, to generate the user preference theme card from the selected lyrics, we only average the two arrays when the resulting index is greater than the original baseline lyric index. This makes it so that the user’s lack of interest in a theme won’t lower the presence of that theme in the final card.
In other words, we have this “baseline score,” the theme scores we generated for each lyric index. As the video plays, we only want the users to be able to boost a particular theme. We never want to reduce a particular theme from being represented. Therefore, as the video plays and we need to combine the baseline current index score with the user preference card, we only combine the two scores if it would increase the theme. We’ll never decrease the baseline lyric index score.
Therefore, if the user never selects anything, the final scorecard is simply the baseline score from the current lyric index.
We now have a scorecard, a list of 10 numbers that represent the current theme/mood taking into account both the user preferences and the mood/theme of the current phrase. What’s left is to find the video effect that best matches this scorecard!
We already scored all of the video effect combinations against the themes used in the user preference card!
I like to imagine that there’s this 10-dimensional “mood space,” where each dimension represents a particular mood. We’ve plotted every possible video effect combination in this 10-d mood space, and we also have a point representing the user’s score card! All you have to do is find the video effect combination that’s closest to the user’s score card! This is why thinking of the mood scores as mood vectors is helpful. We just take the Euclidian distance between the user score card and each effect pair, and return whatever effect combination that’s the closest!
This effect combination (background plat, color tint, and video effect) is then applied to the video, and voila! We’re done!
]]>(For reference — I’m a mac user and a university student, and can afford to pay for a few subscriptions)
Speechify
Just plop any websites you want to read into their link clipper and it’ll save an audio recording on your phone. I have around 45 min every day where I’m walking on campus, commuting, cooking, doing chores, etc. where I listen to stuff. This works as a sort of reading list, but one that I actually keep up on and gets added to roughly as quickly as it gets read! (it doesn’t usually build up)
Text sniper
I find this to be extra helpful lately because it makes it super easy to copy anything into an LLM prompt! A surprising amount of web text / pdf text isn’t easy to highlight and copy.
Fantastical
I’m considering moving to notion calendar, but other than that this is the best integration of a reminders app + calendar app that I’ve seen. It lets you drag reminders onto the calendar which is super awesome if you want to specify a time when you’re gonna work on a task and view it as part of your calendar events.
Anki
I wish I found out about spaced repetition sooner! Especially if you’re in school and you actually care about remembering anything you learn in 5 years that you don’t plan on actively doing every day. If you’re gonna spend 4 years in university learning random stuff you may as well remember it. Not to mention, if you hear a nice story about your grandpa, just add a card for “what’s a good story about your grandpa?” and you’re guaranteed to remember it for as long as you’re using Anki. This is like a huge superpower imo. Cost: like 10 min / day, Benefit: the ability to remember anything you care about, forever!
F.lux
This one is really great! It syncs the blue light coming out of your screen with the sun cycle at your location. There are a ton of reasons why getting blasted with blue light is bad for you, but it’s also just a much more pleasant experience when your computer syncs with the sun cycles like this.
Dark Mode wallpaper
I like using a solid grey-white background during the day and solid black background when the sun goes down. This is a great free option for setting different wallpapers that trigger when the sun goes up / down.
Fatebook chrome extension
Press command+shift+T to make a prediction! Get calibrated!
Inbox when ready / Ochi
I noticed that a lot of the time, either when it’s time to switch my attention to whatever I’m working on next, or just get bored with my current task I’d catch myself impulsively pressing the key strokes: new tab, “gmai-“ without even thinking about it. Interestingly, if I remembered some embarrassing interaction or the midterm I just did bad on, I’d do the same. If it wasn’t gmail, it would be the built-in mail app, instagram, or twitter. If you’re like me and impulsively check your twitter / instagram you should get something like “SelfControl,” “Freedom,” “Ochi,” or one of the other apps that do this. I found Ochi to be the best for me because it’s cheaper than Freedom and has an option to set up times of the day where apps / websites are blocked. SelfControl seems cool too and it’s free. If your main issue is gmail, you should totally get “Inbox when ready.” It gives you a quick prompt that makes sure you really need to open the app and I found this to be super helpful.
HiddenBar
Pretty essential if you’re a mac user imo. A lot of mac apps come with menu bar items that can’t be removed. Mac doesn’t have a built-in way to hide menu bar items so I use this to fight the clutter.
Magnet
Also pretty essential if you’re a mac user imo. Lets you drag a window to the top of your screen to fully enlarge it and left / right for split-screen without having to “Enter Full Screen” or “Tile Window to Left/Right of Screen”.
Arc
Pretty cool! I dunno. I’m not gonna push it too hard but I generally like arc.
LLMs
If you were told 5 years ago that you can pay $20 a month and get access to a website that’ll literally answer graduate-level math / CS with a single prompt, I’m sure you’d pay like 4x the price. I pay for gpt, Claude, and Cursor.
Citing my sources: Thanks for all the reccomendations Dad :)
]]>