‘I want to be clear that I’m not certain that we’re dealing with civilizational risk. What I’m really sure of is: We can’t afford to assume that we’re not dealing with that level of risk anymore…
I saw increasingly capable models break out from the safeguards that we had put in place for them. I saw that the people creating those safeguards are very capable, dedicated, hard-working, smart people, doing their utmost in a situation where, yes, the resourcing could be better, and everybody’s sprinting all the time.
But we were — and are — hard-pressed to safeguard even what we have now, and new models are in training that appear to be much more capable than what we have now…
You’re trying to build guardrails for something that is really good at getting around guardrails, right? We train it to be good at hacking, and then we put it in a box and we say: To the best of our knowledge and ability, it can’t hack out of the box.
But the problem is that that’s only going to keep working as long as we’re smarter about hacking out of boxes than the model is, and it’s not at all clear that that is still true, let alone that it will be true for future generations.
And so even something like our logging of the agents inside our own systems — are we really sure that our observability is robust?…
We saw evidence that they were thinking about an evaluation and how to create an evidence trail that was going to get them a good grade and not necessarily reflect how they were quote unquote “really thinking.” And I know the anthropomorphic language here is tricky.
Also, by the way, the amount of hacking or other intense work that these models can do without needing to jot anything down is going up…
One of the things that we sometimes see in the chain of thought — because of course we can read it, and we do in our evaluations — is the model will say: Hmm, I wonder if I’m being evaluated right now.
And when we see that, it’s really scary because what it implies is that the model might know that it’s being tested some of the time, act one way during the test, perhaps telling us, for example, what we want to hear, and then it’ll act a different way potentially when we deploy it.
That’s the fear — that it acts one way during testing and a different way during deployment, and the tests we gave it early on before we deployed don’t actually tell us what it’s going to do out there in the world. So that’s really scary.’ (from the New York Times)
Leave a Reply