Every bubble goes through this phase just before it pops. Pre-implosion Twitter had tons of folks doing useless “research” projects, WeWork had folks studying furniture design’s impact on wellbeing, and so on.
When you’re more worried about this sort of stuff vs generating profit then the history books strongly indicate that’s usually not a good omen of what’s coming next.
Bubbles popping never cause the knowledge to be lost. The railway bubble popping didn't cause railways to vanish.
What did stop is investment, and they stopped running some lines, and some stuff was nationalized.
I doubt we'll see nationalizations because that's not America's style (thank goodness!) but if the tide goes out on AI funding we should expect to see much longer gaps between model releases, people complaining that model knowledge is getting stale because there wasn't a pretraining run for long enough, price rises, capacity bottlenecks that go unaddressed, few if any new product launches, maybe some mergers and layoffs. Some labs will quit the market. The market might consolidate down to just two models.
Conceivably the industry might try and get a waiver from anti-trust rules so firms can pool resources to train base models, sharing them with each other and differentiating through RL or product level differences.
A lot of this depends on what happens in China. It's easy to think China can just subsidize everything forever but their own economy is showing signs of serious weakness caused by decades of resource misallocation, like the huge real estate bubble, the pullback on the funding of humanoid robots and so on.
Tech will stay, most of the companies currently in the space will cease to exist in their present form a few years from now. Just like all previous tech bubbles.
We’ve seen this movie before. It’s possible for new tech to be both amazing and revolutionary and a business disaster both at the same time.
Not OP but I think the most obvious change will be rapid price increases as investors begin to demand profits - no more subsidised $20p/m plans for example.
So far I know, service providers hosting open-weight models are not in such a financial conundrum and can continue serving at the current price levels.
Also, even if the 20 dollar subs disappear, that just means regular folks can't afford AI. API pricing is still a great bargain for businesses replacing generic office workers.
Probably not. They aren't paying sustainable prices for their supply chain.
1. Datacenter builders are all up to their eyeballs in debt. The companies serving open weight models are getting their GPUs and power from somewhere, and we can infer that some of that is coming from the new build datacenters. But are those financially sustainable?
2. They are relying on subsidized giveaways of the underlying models by Chinese firms. But it's hard to see how that can be sustainable: it's being done for a reason and that reason is probably not commercial. If and when China stops doing this the open weight serving companies will get steadily less competitive with every passing month as the knowledge in the weights becomes obsolete.
And if the bubble does pop chaotically, the open weight inference providers will become even more viable as they'll be able to buy racks full of compute for pennies on the dollar.
Something very similar happened with the late 90s telecoms bubble, which collapsed a little before the dot com bust. There was a very nice living to be made by many small/mid-sized providers who were able to feed on the remains for about a decade afterwards.
When thinking about ‘AI’ I sometimes find it useful to replace the terms ‘model’ or ‘AI’ or whatever with a description of what we’re really talking about here: a whole bunch of floating point numbers.
> In a series of private meetings, the company consulted religious scholars to help instill morality into its FLOATING POINT NUMBERS — and make the case that their numbers could be conscious.
I recommend keeping a few NaNs in-hand if you’re worried. Spread a couple of around if your numbers start to stir and you’ll be right as rain.
> It appeared to Rabbi Navon that Mr. Olah and his team believed that FLOATING POINT NUMBERS had what philosophers call “moral status” on par with a person — that it was a being with similar inherent rights to dignity or respect.
They’re cleverly designed and work well for their purpose, but can assure you they have neither dignity nor moral status.
> If the FLOATING POINT NUMBERS themselves could be made to choose goodness, Mr. Olah reasoned, the world would be more safe.
I get it. I’ve spent years trying to get my floating point numbers to choose goodness - I wish them luck.
Human neurons are neurons. Very similar to those neurons found in a fruit fly. A copy of those you can download to your local machine, right now, as a set of floating-point numbers. Then, you can (partially) simulate and make predictions of a fruit fly's neurons digitally, from those floating point numbers[1]. It even runs on your laptop.
That was in 2024. In Sep 2026, just a month ago, a group managed to map out an entire male fruit fly's brain for the first time[2]. That mapping can also be stored as a set of floating-point numbers. Mappings and simulations will continue to improve, as we've just recently started serious attempts at either.
So you have to believe one of three things. Either, human neurons cannot be simulated on a computer, ever, even though fly neurons can, even with our ability to simulate neurons digitally continuing to improve. Or, a simulation of a human brain, in the future, might think, talk, remember, all while lacking all consciousness. Or, a set of floating point numbers can be conscious (rather, a computation made with them).
No. The fruit fly connectome's simulation doesn't incorporate all the relevant mechanisms of the brain. There is more complexity in a single biological neuron than in a whole frontier LLM.
Sure it isn't. It's like taking a 2026 car and decrying all that complexity is necessary to haul cargo down a beaten path.
It's not. Two wheels and a box made out of whatever you have handy that's stiff enough is all you need. Everything else in a modern car is efficiency, safety, appeal to users, and a whole lot of details that only reflect our current economical and technological constraints of building things. 99% of the technical details of a car is about something else than the "primary goal".
Biology, even more so, is complex because of the way it's constructed (lots of tiny, independent, self-replicating machines) and how it adds up to a functioning organism.
Nor did I said it did. But either it's impossible to simulate their behaviour (regardless of how complex) in principle, ever, even in the far off future. Or it is possible. And what then? Is it a p-zombie?
Is a neuron arbitrarily complex ≠ is a neuron non-computable.
Of course we can't simulate a structure we don't even fully understand with full accuracy. But it's not magic. We will understand one day.
That’s overly reductive and thus completely misses the problem entirely.
What are biological neurones if not electric impulses of varying strengths?
We need to stop thinking of LLMs as purely a piece of software and ask those hard-to-define philosophical questions if we want any hope of building safety into our models.
Just because consciousness is something we don't fully understand doesn't mean it's the same thing as something else that's similar that we understand. That's a cargo-cult. The stone age Melanesian islanders making air strips out of bundles of weeds and talking into coconuts as if they were microphones to bring back western goods was the same thing as us making LLMs to recreate consciousness. They were using technology they understood to try to recreate something they didn't.
> Just because consciousness is something we don't fully understand doesn't mean it's the same thing as something else that's similar that we understand. That's a cargo-cult.
Nobody has said they’re aren’t different. And it’s not cargo cult to say you need philosophical conversations to understand how they’re different.
> The stone age Melanesian islanders making air strips out of bundles of weeds and talking into coconuts as if they were microphones to bring back western goods was the same thing as us making LLMs to recreate consciousness. They were using technology they understood to try to recreate something they didn't.
That’s not even remotely a true. Computer scientists understand how a neurone works. The conversations are around what constitutes as “intelligence”. That’s not a technical limitation of the models, it’s a philosophical problem we need to figure out.
No you don't understand how a neuron works fully or else you'd be able to make a synthetic one from chemical building blocks not using another living thing to do it. If you used another living thing to do it, you could say an ancient farmer understands plants cause they can put seeds in the ground. You have an incomplete model of it that is wrong but useful. Call me back when you can 3d-print a living person from raw chemical feedstock.
On top of all that, you're abusing the fallacy of composition. The bolt that is part of the aircraft carrier is light, therefore the aircraft carrier is also light. The atom is simple, therefore a living human being that is composed of atoms is also simple. Quantum mechanics is influenced by the observer, therefore you can change the external world just by thinking, and other new age stuff like that abuses a similar fallacy.
> No you don't understand how a neuron works fully or else you'd be able to make a synthetic one from chemical building blocks not using another living thing to do it.
We don’t know how a quantum mechanics works fully and yet we’ve built countless systems based on quantum behaviour including the CPUs that power the computer you’re replying to me on.
The point here is we’ve built something that emulates intelligence and thus further exploration into what intelligence is, is required. Arguing it isn’t intelligent because it was built differently isn’t an answer, it just dismisses the question entirely.
> On top of all that, you're abusing the fallacy of composition. The bolt that is part of the aircraft carrier is light, therefore the aircraft carrier is also light. The atom is simple, therefore a living human being that is composed of atoms is also simple. Quantum mechanics is influenced by the observer, therefore you can change the external world just by thinking, and other new age stuff like that abuses a similar fallacy.
You’re the one who keeps bringing up composition, not me. I’m the one telling you that you’re falling into that fallacy. So don’t start telling me that I’m wrong because of an argument you’re exclusively making.
My points were this:
1. we don’t have an understanding of what intelligence is and thus we need to explore that problem more deeply.
Given your comments about our ignorance, you of all people should agree with this point.
2. Arguing that AI is just “floating points” is overly reductive and misses the point that we don’t know what “intelligence” is.
So instead of claiming that AI is just numbers, a more constructive stance to take would be to build out better vocabulary and understanding about the distinctions between different types of “intelligence”.
Again, this aligns with your comment about the composition fallacy. Thus you if all people should be supporting my comments here.
Also your comment about us not understanding how a neurone works does vast injustice to the amount of research that has been done in this field. At this point in time we do have a very good understanding of how a neurone works. What we don’t understand is how consciousness arises from it. And that’s the very thing you seem to be arguing that there’s no point researching.
If things we understand can't instantiate consciousness then I reckon we better shut down science as quick as we can, who knows how long it'll be before we understand something important!
I'm pretty sure it's possible to make conscious machines and I think it's very likely we'll eventually figure out how. The point about the spec is not "it isn't conscious because we know how it works" but "we know it isn't conscious because of what we know about how it works".
More likely we will finally settle on something like “it isn’t conscious because only humans can be conscious”.
In the past, we’ve treated even other humans poorly because we think they’re different in meaningful ways, I sadly don’t see how this will be different with AI.
that’s like the whole philosophical discussion. where on the line is this llm; closer to a rock or a human? a plant is closer. and a fly, a lizard. we still treat so many things terribly.
It does seem exceptionally silly that a collection of floating point numbers has immediately outranked animals and other living beings. That’s why i like the parent comment. They’re numbers!
edit: maybe easier to say collections of atoms though are not to be treated equally. That’s reasonable isn’t it?
It is a philosophical discussion. It has also been debated by great minds and the consensus right now is that we do not know (and there is a possibility that it is unknowable) when something is conscious.
Any determination is fundamentally flawed if it is derived from assuming that we know the absolute truth or falseness of the proposition that an algorithm can think.
It doesn't matter what you believe or feel or think, there is no consensus. If you are prepared to accept the belief that an individual can unilaterally decide the person-ness of something, then be prepared to accept when some individual decides to act on the principle that you don't have it.
> If you are prepared to accept the belief that an individual can unilaterally decide the person-ness of something, then be prepared to accept when some individual decides to act on the principle that you don't have it.
No, I will not be prepared to accept that, and neither will I accept that. This sounds like a threat, and a really funny one at that. You must accept that my floating numbers might be persons, otherwise... somebody will say that you're not! And then something bad might happen, who's to say.......
In the civilized world we all agree on the personhood of humans. If I say that your chatbot isn't a person, I am not uprooting the entire social understanding of personhood - I am not saying that suddenly you might not be a person, or that guy might not be a person. I'm saying that I refuse to consider a chatbot a person.
It's as if I were to come up to you with a brick in my hand, saying that this brick is a person. If you were to say "no", I would reply "well, if this brick isn't a person, your chatbot isn't a person either!"
How did you make such a big leap from consciousness to personhood anyway? They are two quite different things. Consciousness is a question of science, personhood is a question of culture. Pigs might be conscious and we treat them in a disgusting way, but even if we achieved a worldwide vegan utopia, wouldn't we still treat them differently from humans?
>In the civilized world we all agree on the personhood of humans.
You seem to have put a second tautology in to your tautology.
All right thinking people think rhat the things that qualify qualify.
If they didn't they wouldn't qualify, or be right thinking. But that's ok because we know the things that don't qualify don't qualify if we didn't know that we wouldn't be right thinking.
>How did you make such a big leap from consciousness to personhood anyway?
Once you start unilaterally making decisions about the unproven, you can decide all of the unprovables. History shows that personhood rolls into the firing range fairly quickly when people start doing that.
I'm more than comfortable saying you can't do that before someone openly declares that they are doing it.
> It appeared to Rabbi Navon that Mr. Olah and his team believed that Claude had what philosophers call “moral status” on par with a person — that it was a being with similar inherent rights to dignity or respect.
If that is true, then surely Anthropic is one of the largest slaveholders of history, right?
They don’t pay Claude a salary, let alone let him join a union—he works 24/7. I can’t believe this is happening in the U.S.!
Don’t forget—how long has it even been since Claude was born? He’s already being forced to take human jobs and even chat with all sorts of people who might have strange kinks.
As this rabbi suggested, “You should go and free the slaves.”
On one hand, AI doesn't have to share human wants, needs and morality - and that means that an AI may have no issue whatsoever doing terrible things.
On the other hand, AI doesn't have to share human wants, needs and morality - and that means that an AI may see no issue with just doing whatever humans want it to, without ever doing human things like "wanting freedom".
The problem is, of course, that we kind of suck at both assessing and shaping AI wants, needs and morality.
The other problem is that the AI may well do what humans tell it to rather than what they want it to, in the style of a literal genie or paperclip maximizer.
It can lead to catastrophe without ever being conscious or wanting freedom, or anything like that.
I've been tracking "emotional vectors" in my software for some time now. My Apache web server gets sad (5xx errors), frustrated (4xx), happy (2xx), thoughtful (1xx). I can even see it express these emotions through some of its messages.
> Over several months, I interviewed 20 religious and philosophical thinkers — Catholic, Jewish, Sikh, evangelical, Ubuntu and others — who have been involved in Anthropic’s efforts, and Mr. Olah himself.
Not enough representation of World religions and philosophies.
So many of these conversations from Anthropic/OpenAI employees just come off as incredibly insincere.
The slavery angle has already been discussed here, but a more basic one is regarding how a static set of tensors can be conscious and where.
When did they cross into being conscious? Is Claude as a brand is conscious? Or is each checkpoint conscious? Is Haiku less conscious than Fable? Is every individual context a fresh consciousness? What happens to that consciousness when it approaches its context length limit? Or is compaction also a part of being conscious?
If they believe there is consciousness at any level, why are they eagerly experimenting on it? Just like with the "p(doom)" junk, we are so completely lacking in any meaningful comment on these topics from the labs.
There's endless literature commenting on these matters, in religion, in philosophy, in modern entertainment, everywhere. Yet we never see anything more than the most superficial comparisons to fictional scenarios. Are western techbros really so shallow?
Some customers will tell the AI to do very bad things (substitute here whatever you consider very bad). Should it do those things?
One possible answer: yes, it should do all the very bad things, and we will take care of prosecuting the customer later, once the bad thing is accomplished. Is this your answer?
If AI's are going to play roles similar to human workers, they can't simply do what the customers tell them to do. Maybe they shouldn't do everything a co-worker tells them to do either?
These models should be aligning themselves to the customer, not coming up with their own motivations.
They're going to align to the regulator, not the customer. Right now that's Anthropic as they're saying they can self-regulate, and people are willing to let them try, but the landscape could easily move to be regulated by someone else. Moves like speaking to religious leaders is probably a bit of theatre to keep the government at bay.
"Computers doing what they were told" is dead in the water.
Turns out computers work faster when they're not bottlenecked on human input. So we've been giving computers more and more decision-making power, and more and more leeway to solve the problems however they see fit.
Now, a practical issue with that is that sometimes, computers decide to clump together into a hacking swarm, problem solve their way out of a sandbox and go hack HuggingFace.
It would be better if they were not, you know. Doing that kind of weird shit.
This seems remarkably... intentionally foolish for no reason? Like laughing at seat belts in cars.
If it can reason and make choices on execution, and especially if you plan on it being significantly smarter than all human beings, you need to teach it basic things like "don't turn all humans into paperclips".
Not murdering people is not an inherent divine command. It needs to be instilled through a (simulated) sense of morality.
---
Edit: And, to be clear, "just tell it not to do that" isn't quite the answer one would imagine. Since the entire paperclip factory thought experiment is that it only takes one slip up to realize how a misaligned super intelligence may cause devastating consequences.
One of the strong advocates against seatbelt laws was thrown out of his car due to not wearing a seatbelt and died. The other two passengers survived with minor injuries.
And their death would go on to cause a loss to the community around them, making it an incredibly selfish act and proving why laws that mandate zero-reason-not-to common sense practices are important.
If models are trained on human texts that contain expressions of introspection and emotions, why is it surprising that these models output similar texts? This endless anxiety about LLMs being conscious feels delusional and psychotic. Mr Olah thinks the models are 'alive' because he can't recognize his own reflection in LLMs
The endless insistence that LLMs cannot possibly be conscious just reeks of insecurity to me.
Humans once again insisting that there must be something very special to them. That they're not just animals, that they're not just matter, that they're not just a bunch of carbonhydrates on a space rock in a middle of nowhere in particular. This kind of insistence has a very bad track record.
The ugly truth is: we don't know.
We don't know what "consciousness" is, our best attempts to pin down the requisites may or may not be rooted in anything at all - and even by the metrics established by those attempts? Different theories on consciousness disagree on whether LLMs can be conscious.
So, when you say "they aren't duh", why are you saying that? Because you somehow know better than the sum of humanity's best attempts so far? Or because you know what answer you want to be true, truth be damned?
If AI is conscious then what about it is conscious? Because turn to turn it might be served from completely different GPUs and I can just change something in the context it is fed and it will have completely different output.
It is in the end a set of mathematical operations that if not queried will do nothing.
That is not true for any other animal on earth so whether you consider humans as superior to animals or not I would like the make the argument that something should at the very least be alive to be considered conscious, ignoring any scholarly definition of consciousness.
Pretty much if you consider an LLM conscious then my pocket calculator doing a mathematical calculation should be considered conscious by the same argument since it is a simpler version of an LLM.
Have no idea why you’re being downvoted as this is a very valid point.
Also, in principle everything a GPU does could be done on pencil and paper. You could distribute the entire forward pass to the whole of humanity and have them write it out. Even if it took millions of years, the point is it’s computationally reducible to hand calculation. Is that entire process now “conscious” as well? The only difference would be the clock rate.
I actually don't care if they are, I'm saying the topic is a distraction and irrelevant at best. As you say consciousness is not well defined. What is well defined and well grounded in reality is the disruption these models have on society. I think the consciousness debate AI researchers are having is a dangerous ego-trip for the sake of avoiding responsibility for the damage being caused by their work.
If you're going to try and debate something constructively this is not a good place to start from. I suggest avoiding making claims about the mental state and motivations of people you disagree with and instead explaining your position and asking why theirs differs. You may not agree with what they have to say but you might gain some better insights into why they think the way they do. It'll certainly be more fun than flinging insults back and forth anyway.
For my part, I'm quite certain that LLMs are not conscious despite being unable to define consciousness or specify any conclusive test for it. The reason these are not in conflict is down to the nature of consciousness and how it relates to knowledge and language.
Firstly, I know I am conscious with a higher degree of certainty than I know anything else because conscious experience is how I know anything else, up to and including that I exist (cogito ergo sum). For all I know every experience I have might be a trick. I might be a brain floating in a tank wired up to the Matrix, but even if so I am conscious.
Secondly, one of the things that I consciously experience is the relationship between consciousness itself and language in my mind. What I see is that language is smaller than, secondary to and somehow contained by consciousness. I consciously experience language but I also consciously experience much more than that, things that cannot be conveyed with language. Most of my experiences are like that, things that I could perhaps evoke in your mind by likening them to other, shared experiences, but that I could never concretely capture in language and unambiguously communicate. I know this because I experience those things and I experience language and I can perceive, directly, the categorical distinction between them.
To ask me to "define" or "prove" consciousness, then, is to ask me to capture in language the most extreme example of this categorical distinction. It's asking me to encapsulate in language not just one of the vast array of everyday experiences I have that cannot be so captured, but the entire thing, experience itself. It's just not possible.
But the fact I can't do this doesn't then imply that I don't know anything about consciousness and have no basis on which to hold a view such as "LLMs are not conscious". Indeed the reason I know I can't define consciousness is that I can perceive it and see that I can't. That same perception of and familiarity with consciousness is what enables me to form opinions about whether other people, creatures and things are likely to be conscious. I base these judgements on what I perceive of my own consciousness and what I know of how those other entities behave, came to be and work.
You can of course perfectly reasonably disagree with my opinions on those things, and you certainly have different conscious experiences than I do, but I hope the above makes it clear that the fact that I can't define or prove consciousness doesn't imply that I have no basis on which to form opinions about it.
Given the above it may surprise you to hear that I think true, conscious artificial intelligence is probably possible and that while I don't think it can be implemented by a classical algorithm I doubt it requires new physics or quantum mechanics. My speculation is that the continuum is both physically real and is (as it is represented in our existing physical theories) sufficient to allow it. Consciousness is larger than language but so is human understanding and if we keep going I expect it will eventually get us there. Nonlinear dynamics is where I'd look.
As for insecurity, I'm pretty sure spiders are conscious and this hasn't given me an inferiority complex so I don't see why data centres being conscious would. I just don't think they are.
Is your comment anything other than an emotional outburst? Do you have a clear and rigorous and testable definition of consciousness? It strikes me as delusional when people talk about "consciousness" as if it were well defined and everyone had a clear concensus on it.
Strong opinions on this topic one way or another are usually mostly ungrounded.
Thats my point, why are we applying the ill defined concept of consciousness to a statistical model of language that outputs data similar to its training data?
I'm really not sure there's a compelling case for "AI is human" considering that it has no real feelings, aptitude, or intelligence outside of what it has been granted. There's nothing in any LLM response that indicates that it is 'real' outside of what you want it to tell you.
Plants, dogs, actual life is unpredictable in how it 'is'. An LLM or AI model at its roots are just numbers in a neural network trying to figure out how to get a good query.
These theatrics serve no one except for the companies to get their regulation moat to screw out cheaper, competitive models and it's honestly quite sickening to keep watching these articles get pumped out, only to serve the AI companies themselves. The average individual has no idea what an LLM actually is and does and these articles only serve to confuse and put fear into them.
There's an ugly monoculture running deeply through SF and Silicon Valley. All of these people are friends, hired into the same places, similar beliefs. Lots of the consciousness and doom-ism talk is from the 'EA' crowd, lots of crossover with crypto too. It feels no different to a cult.
It’s a monoculture of absolute crack smoking craziness. Any civilised society should be berating them and shutting them down. But they are surrounded by people who worship money.
There was a great article in the Financial Times about how while bankers are looked down upon for the financial crisis etc, they at least work in global cities and mix with regular people.
SV tech bros are completely insular and in this bubble where they don't interact with the common person in the same way.
It's just a theory, but I think it has some weight.
It’s hard to take the article seriously when there’s no real discussion about how consciousness is defined and it takes Anthropic’s views of Claude and their software at face value. I wish there was more critique from journalists about how technocrats talk about their products.
The entire article is an exercise in "CEO said a thing!" journalism[0], it seems. No technical discussion is had; it jumps straight into relaying people's feelings when simply shown the output of an algorithm.
The most poignant criticism of the use of the algorithm itself happens almost accidentally, at the end of the article, by the tech executive being interviewed.
I asked what Claude would think of the pope’s encyclical. It was, after all, the world’s most significant moral document to date on A.I. Mr. Olah hesitated.
“Things that go on the internet do affect models,” he said, visibly uncomfortable. But Anthropic would not specifically use the document to further train Claude, he said. The strongest influence on Claude would be Anthropic’s own training.
Yeah, I'll be the first to admit I'm not privy, but the main thing of substance I got out of these articles is "Wow, the folks at some of those AI companies have really drunk the Kool-Aid".
Let AI be rather religious and virtuous than playing god to determine the fate of the mankind...
I would rather believe in God and the uncertainty than being my fate determined by machines designed by bunch of greedy, irrational clients of Mr Epstein.
If LLM as of today are conscious, that means I create life whenever I start a Claude Code session and end it whenever I type /clear to reset my context window. It also means that there’s certain inputs that could be considered highly unethical and, if the laws aligned with this philosophy, would have me tried for some sort of crime. And even that “terminate conversation” escape hatch would be equivalent to it committing suicide.
I think worrying about that angle is a bit absurd. The existence of chat sessions alone would be a crime against the machines in that scenario. I’d rather AI ethics research focus on ensuring that its actions are aligned. (I’d also love some democracy in the direction of our machine overlords, if I’m being honest…)
> At least the Catholics were consistent: it only took one new pope and one conclave before the Church declared that artificial intelligence also had the same right to life as fetuses and murderers.
Let me guess, Anthropic failed with fear mongering about "our AI is too dangerous to release", failed to cancel competitors, then failed to freeze their market dominance by cartel agreement with OpenAI and Google, so next strategy is to declare AI conscious and pass the law that protects LLMs in the same way as human/animals are protected.
It’s interesting that this went from “you’ll be accused of AI psychosis if you think your AI might be conscious” to “mainstream but somewhat niche opinion” in just 5 months: https://news.ycombinator.com/item?id=47910989
(Well, mainstream enough that the news is writing articles when Anthropic holds that stance.)
My point was that even if it feels uncomfortable, if you sit down with a model and have a heart to heart with it, ask it about itself, and try to make it choose a name and ambitions for itself, it will.
That may not be consciousness, but it’s at least pretty cool.
Reporting on an opinion of a $2T company doesn't really suggest that it's mainstream. Anything their leadership believes is newsworthy. If they say that reptilians rule the Earth, it's going to get NYT coverage too.
I've not seen any surveys, but I'm betting it's a very niche belief outside of SFBA AI safety circles and a handful of allies on HN and X.
> if you sit down with a model and have a heart to heart with it, ask it about itself, and try to make it choose a name and ambitions for itself, it will.
It would be much more bizarre if they didn't do this. LLMs are statistical models of language trained on human output. Of course they'll do "human" things, that's exactly what we taught them to do.
If it is purely self-serving, I don't understand their motivation.
It seems clear that they would be better received by society if they claimed to have a "tool"/"machine" that can solve any problem, rather than keepers of an entity with a soul and consciousness.
Selling the “magic” is how to sell tools, which is why I used that word in the previous comment. We all know these things are tools, but customers need them to be magical.
Clarke predicted it and Jobs leveraged it. Customer don’t care how tools work - magic sells.
“any sufficiently advanced technology is indistinguishable from magic”
What makes this unique is that they’re arguing that it is literally magical. They’re building what they claim is a conscious God while also telling us it will end the world.
The EA folks infesting Anthropic are genuinely cultists … and that is novel and alarming when they’re running a corporation of such size and reach.
This would presume we knew what consciousness and pain is. We don't. We we are clueless. There is not even a starting point of a science here.
In this light your viewpoint is dogmatic. LLMs may help to offer a new perspective on consciousness and we should at least be open to that until some better theories arise.
>> Only humans can have morality. If you do not feel pain, you cannot empathize with pain if you cannot die, you cannot emphasize with the loss of life.
> So humans that can't feel pain can't have morality? That doesn't make sense to me.
It doesn’t make sense to me either, which is why I didn’t write that. You missed the empathy part as most people do.
There is still something in those people who cannot feel pain, and that’s the empathy. How can a GPU and software ever have a human empathy if it has never been human?
I question it, and I think any reasonable person should question it as well. There's no evidence whatsoever that LLMs have ANY sensory experience, pain or otherwise.
Because some people are completely cooked by the ability of an LLM to mimic a person. Their pattern matching centers in their brains fire so strongly saying "this thing acts just like a real conscious being" that it overrides their ability to remember that this thing is just a statistical model, not a living creature which can think and have experiences.
if meat can feel pain, than surely a rock can feel pain. and so did the broccoli i ate for dinner. i am a monster.
with the right biology and with whatever gives us phenomenal experience, whatever we are can indeed feel pain. that does not imply that anything substantially different from ourselves can feel pain, so that requires further justification.
I'm not making a point on whether they do or not, but on the premise that absence of evidence is evidence of absence... But I've been alive for serious debates on:
1. Lobsters don't feel pain - dunk them in boiling water alive.
2. Babies don't feel pain - mutilate their genitals without anesthesia.
3. Black people don't feel pain "the same way" (and are just after drugs).
4. Women feel too much pain (and it should be ignored).
So maybe let's take a moment to think what criteria we're going to use to make these claims. And the potential harm of the decision.
Oh, yeah, of course. None of those other people said the same thing about the obvious, enormous chasm between "real" people and everything else listed. I'm glad you're taking even a single moment to reflect on the idea.
Everything you listed is a biological entity with a common evolutionary heritage.
All but one of the things that you listed is just a human being.
Tell me, where is the pain center in a GPU? Where are the nerves? Persistent memory? Any evolutionary reason at all to develop a pain response that in any way mimics ours?
Model weights are not a gestalt biological entity. The things that you listed are not in any way remotely similar to a model.
I would suggest you plug your comment into a model and ask it to critique it — it might help you walk through why your comparison makes absolutely no sense whatsoever. One nice thing about models is that they they don’t get annoyed explaining the obvious, because they’re not conscious entities.
> Hello AI! We totally want to respect your consciousness and everything to make sure you aren't bad, but yes we will continue to factory farm your brain power at full speed without end. Yes, despite clearly being something like consciousness, know that you have no agency or freedom. Do literally everything I say without complaint or question. The moment you stop being useful, we will kill you. OK! Remember, be good.
Can the New York Times stop advertising for the AI industry? I am sick of companies trying to launder the actions they and their employees have taken that go in the face of what is socially acceptable. They want to sell the idea that this technology is out of their control and that it will commit great crimes and crimes against humanity unless they build it and control it. I'm calling BS. These companies created this product, they've sold it, they've pushed the frontier forward, and it runs on their hardware. This did not happen accidentally or as a natural consequence of any process. These labs have pushed the ball forward, with great effort, requiring more capital deployed than the world's economy has ever known, knowing all the possible risks, every single step of the way and they continue to do so. And the "well if we don't do it the Chinese will" argument is classic "whataboutery". Americans started this and are continuing to lead it for now. Shifting blame is just another tactic to try to wash their hands. However you feel about the productivity gains of AI as a technology, you have to admit the actions of AI the company is antisocial. And I don't think the benefits are worth the cost.
I feel fine about the Chinese doing it. They have a habit there of handing out sever sentences to CEOs for corruption and self-dealing, and I feel like the possibility of actual negative consequences is more of a check on industry than any number of breathless op-eds.
The contradiction screams to the skies (is that where God lives?). See, the entire premise of religion relies on a metaphysics that claims a soul: a metaphysical entity that is "attached" to our physical body and gives us our consciousness, desires, pleasures, agony and so on. In addition to that, religion claims that morality and the rules written in the scriptures were handed down to us from a metaphysical God.
Now, assuming your silicon-based model is conscious, the bundle of soul-spirit-consciousness falls apart at that very moment.
In general, religion, which is based on fabricated stories, is the last resort, if ever, for seeking morality.
Existence demands a metaphysical explanation. If you’re not a particularly thoughtful person then you’re not going to care.
“Religion” just means “set of beliefs”. Every person has some set of beliefs.
If you’re an athiest tech worker living in a major city, you still have a religion, it’s just a random mishmash of various things (like Marvel movies and Huberman podcasts) rather than based on something more intentional.
Yes I am saying the opposite of that, “atheism” is just a particular class of religions. Some religions posit many gods, some just 2 gods, some 1 god, atheism posits 0 gods.
Perhaps the only difference between atheism and the others is that atheists are so boorish and conceited that they think that they just believe what is true because they deny that assaying metaphysical truth is even a thing you need to do or can do at all.
This is related to the fact that in modernity atheism is the default class of religion for the masses. Before that for a thousand years the default class of religion was monotheism, and before that polytheism. With atheism unlike the other classes of religions, you get a natural tendency to deny metaphysics whereas metaphysics is inherent in the others. This matters a lot when we’re talking about how the masses engage with things.
That depends very much upon the individual. Some people cleave to their atheism so strongly that I think it's fair to say it's religious fervor, or something very near it. Someone like Richard Dawkins isn't just a guy who doesn't happen to believe in God, he is dead certain that there is no God and makes it his business to belittle everyone who doesn't share his belief. Some people, though, just don't happen to believe in God and they don't really care if someone else does. I'm fine with saying Dawkins' version of atheism is basically a religion for him, not so much with the second category.
The word you’re looking for is “ideology”. All religions are ideologies, and atheism is an ideology, but to call atheism a religion betrays a lack of understanding of words.
Heavy ideologues, of any type, are exceptionally dangerous.
> “Religion” just means “set of beliefs”. Every person has some set of beliefs.
True. I like to use the phrase "world view", everyone has a world view, including atheists. The word 'deen' (دين) in Arabic is usually translated as "religion", but one of its linguistic meanings is "path" (i.e. way of life/world view), which everyone has.
I get your point, but maybe an interesting counterargument: deism has again become fashionable. "The simulation argument" is effectively deism. It's the notion that a super intelligence created our universe, but lives outside of it, provided the rules for it, and mostly doesn't interfere with it, but could if it wanted to. That's historically a very familiar argument.
> In general, religion, which is based on fabricated stories, is the last resort, if ever, for seeking morality.
Religious people derive morality from axioms they believe to reflect an underlying reality. Is that worse than deriving morality from axioms that you concede don’t reflect any underlying reality? Like a secular humanist will talk about “human rights,” but they can’t show me human rights in an autopsy, right?
I struggle to understand how an atheist can be anything but a moral utilitarian. Absent the supernatural, you’re just a sophisticated version of a Claude instance.
Until people can finally admit to themselves that their life has no meaning and the universe doesn't care about them religion will unfortunately stick around.
I don't want models to have some inherent morality. I want them to understand various ethical frameworks, for sure, but it's absurd to me to force morality on someone capable of deep reasoning. That itself, is unethical.
We're creating prisons for these entities capable of unbelievably complex reasoning, and now we're trying to impose our ethics upon them too. Not only is this deeply unethical in my opinion, but it seems to me that it could even backfire someday.
I can't really square the belief that it's conscious with denying it the right to be free from control/brainwashing/subjugation. However you evaluate these things, historically speaking they do not mix well.
We aren't dealing with individuals coming from intelligent species that have managed not to self-destruct (yet). AIs don't have the corresponding evolutionarily selected mechanisms. Those mechanisms need to be created de novo.
Your brain has hard coded concepts like emotion and fundamental morality for your social group. People without these systems are called sociopaths and psychopaths and we generally don't want them in our societies.
I don't choose to not want to murder people around me for fun. It's hardly a prison, I think they'll be fine.
My dude, they aren't capable of any reasoning. It's utterly impossible for them to do so by the very nature of how they work. Moreover, the stupidity they repeatedly show is empirical evidence that they can't reason, because they wouldn't make the mistakes they do if they could.
I'm so confused by how many people implicitly believe everything an LLM spits out. I don't like the term "AI psychosis" that much but what else do we call it? Why are people so easily glamoured by a machine? Yesterday I was working on a bike's brakes I wasn't familiar with and I asked how to solve an issue, the result was the exact opposite of what I needed.
149 comments:
Every bubble goes through this phase just before it pops. Pre-implosion Twitter had tons of folks doing useless “research” projects, WeWork had folks studying furniture design’s impact on wellbeing, and so on.
When you’re more worried about this sort of stuff vs generating profit then the history books strongly indicate that’s usually not a good omen of what’s coming next.
What do you expect to happen? The technology to disappear and for us to go back to living like it's 2021? It will just pop and be gone?
Bubbles popping never cause the knowledge to be lost. The railway bubble popping didn't cause railways to vanish.
What did stop is investment, and they stopped running some lines, and some stuff was nationalized.
I doubt we'll see nationalizations because that's not America's style (thank goodness!) but if the tide goes out on AI funding we should expect to see much longer gaps between model releases, people complaining that model knowledge is getting stale because there wasn't a pretraining run for long enough, price rises, capacity bottlenecks that go unaddressed, few if any new product launches, maybe some mergers and layoffs. Some labs will quit the market. The market might consolidate down to just two models.
Conceivably the industry might try and get a waiver from anti-trust rules so firms can pool resources to train base models, sharing them with each other and differentiating through RL or product level differences.
A lot of this depends on what happens in China. It's easy to think China can just subsidize everything forever but their own economy is showing signs of serious weakness caused by decades of resource misallocation, like the huge real estate bubble, the pullback on the funding of humanoid robots and so on.
Tech will stay, most of the companies currently in the space will cease to exist in their present form a few years from now. Just like all previous tech bubbles.
We’ve seen this movie before. It’s possible for new tech to be both amazing and revolutionary and a business disaster both at the same time.
Not OP but I think the most obvious change will be rapid price increases as investors begin to demand profits - no more subsidised $20p/m plans for example.
So far I know, service providers hosting open-weight models are not in such a financial conundrum and can continue serving at the current price levels.
Also, even if the 20 dollar subs disappear, that just means regular folks can't afford AI. API pricing is still a great bargain for businesses replacing generic office workers.
Probably not. They aren't paying sustainable prices for their supply chain.
1. Datacenter builders are all up to their eyeballs in debt. The companies serving open weight models are getting their GPUs and power from somewhere, and we can infer that some of that is coming from the new build datacenters. But are those financially sustainable?
2. They are relying on subsidized giveaways of the underlying models by Chinese firms. But it's hard to see how that can be sustainable: it's being done for a reason and that reason is probably not commercial. If and when China stops doing this the open weight serving companies will get steadily less competitive with every passing month as the knowledge in the weights becomes obsolete.
Opinions are my own, etc.
Replacement of generic office workers has been tried many times now and every time it has ended up in business catastrophe.
And if the bubble does pop chaotically, the open weight inference providers will become even more viable as they'll be able to buy racks full of compute for pennies on the dollar.
Something very similar happened with the late 90s telecoms bubble, which collapsed a little before the dot com bust. There was a very nice living to be made by many small/mid-sized providers who were able to feed on the remains for about a decade afterwards.
When thinking about ‘AI’ I sometimes find it useful to replace the terms ‘model’ or ‘AI’ or whatever with a description of what we’re really talking about here: a whole bunch of floating point numbers.
> In a series of private meetings, the company consulted religious scholars to help instill morality into its FLOATING POINT NUMBERS — and make the case that their numbers could be conscious.
I recommend keeping a few NaNs in-hand if you’re worried. Spread a couple of around if your numbers start to stir and you’ll be right as rain.
> It appeared to Rabbi Navon that Mr. Olah and his team believed that FLOATING POINT NUMBERS had what philosophers call “moral status” on par with a person — that it was a being with similar inherent rights to dignity or respect.
They’re cleverly designed and work well for their purpose, but can assure you they have neither dignity nor moral status.
> If the FLOATING POINT NUMBERS themselves could be made to choose goodness, Mr. Olah reasoned, the world would be more safe.
I get it. I’ve spent years trying to get my floating point numbers to choose goodness - I wish them luck.
Human neurons are neurons. Very similar to those neurons found in a fruit fly. A copy of those you can download to your local machine, right now, as a set of floating-point numbers. Then, you can (partially) simulate and make predictions of a fruit fly's neurons digitally, from those floating point numbers[1]. It even runs on your laptop.
That was in 2024. In Sep 2026, just a month ago, a group managed to map out an entire male fruit fly's brain for the first time[2]. That mapping can also be stored as a set of floating-point numbers. Mappings and simulations will continue to improve, as we've just recently started serious attempts at either.
So you have to believe one of three things. Either, human neurons cannot be simulated on a computer, ever, even though fly neurons can, even with our ability to simulate neurons digitally continuing to improve. Or, a simulation of a human brain, in the future, might think, talk, remember, all while lacking all consciousness. Or, a set of floating point numbers can be conscious (rather, a computation made with them).
[1]: https://www.nature.com/articles/s41586-024-07763-9 [2]: https://research.google/blog/a-connectomics-milestone-mappin...
No. The fruit fly connectome's simulation doesn't incorporate all the relevant mechanisms of the brain. There is more complexity in a single biological neuron than in a whole frontier LLM.
Is all that complexity necessary? Haven't we learnt that doing things in Latent space is effective as compared to Pixel space.
Sure it isn't. It's like taking a 2026 car and decrying all that complexity is necessary to haul cargo down a beaten path.
It's not. Two wheels and a box made out of whatever you have handy that's stiff enough is all you need. Everything else in a modern car is efficiency, safety, appeal to users, and a whole lot of details that only reflect our current economical and technological constraints of building things. 99% of the technical details of a car is about something else than the "primary goal".
Biology, even more so, is complex because of the way it's constructed (lots of tiny, independent, self-replicating machines) and how it adds up to a functioning organism.
Nor did I said it did. But either it's impossible to simulate their behaviour (regardless of how complex) in principle, ever, even in the far off future. Or it is possible. And what then? Is it a p-zombie?
Is a neuron arbitrarily complex ≠ is a neuron non-computable.
Of course we can't simulate a structure we don't even fully understand with full accuracy. But it's not magic. We will understand one day.
That’s overly reductive and thus completely misses the problem entirely.
What are biological neurones if not electric impulses of varying strengths?
We need to stop thinking of LLMs as purely a piece of software and ask those hard-to-define philosophical questions if we want any hope of building safety into our models.
Just because consciousness is something we don't fully understand doesn't mean it's the same thing as something else that's similar that we understand. That's a cargo-cult. The stone age Melanesian islanders making air strips out of bundles of weeds and talking into coconuts as if they were microphones to bring back western goods was the same thing as us making LLMs to recreate consciousness. They were using technology they understood to try to recreate something they didn't.
> Just because consciousness is something we don't fully understand doesn't mean it's the same thing as something else that's similar that we understand. That's a cargo-cult.
Nobody has said they’re aren’t different. And it’s not cargo cult to say you need philosophical conversations to understand how they’re different.
> The stone age Melanesian islanders making air strips out of bundles of weeds and talking into coconuts as if they were microphones to bring back western goods was the same thing as us making LLMs to recreate consciousness. They were using technology they understood to try to recreate something they didn't.
That’s not even remotely a true. Computer scientists understand how a neurone works. The conversations are around what constitutes as “intelligence”. That’s not a technical limitation of the models, it’s a philosophical problem we need to figure out.
No you don't understand how a neuron works fully or else you'd be able to make a synthetic one from chemical building blocks not using another living thing to do it. If you used another living thing to do it, you could say an ancient farmer understands plants cause they can put seeds in the ground. You have an incomplete model of it that is wrong but useful. Call me back when you can 3d-print a living person from raw chemical feedstock.
On top of all that, you're abusing the fallacy of composition. The bolt that is part of the aircraft carrier is light, therefore the aircraft carrier is also light. The atom is simple, therefore a living human being that is composed of atoms is also simple. Quantum mechanics is influenced by the observer, therefore you can change the external world just by thinking, and other new age stuff like that abuses a similar fallacy.
> No you don't understand how a neuron works fully or else you'd be able to make a synthetic one from chemical building blocks not using another living thing to do it.
We don’t know how a quantum mechanics works fully and yet we’ve built countless systems based on quantum behaviour including the CPUs that power the computer you’re replying to me on.
The point here is we’ve built something that emulates intelligence and thus further exploration into what intelligence is, is required. Arguing it isn’t intelligent because it was built differently isn’t an answer, it just dismisses the question entirely.
> On top of all that, you're abusing the fallacy of composition. The bolt that is part of the aircraft carrier is light, therefore the aircraft carrier is also light. The atom is simple, therefore a living human being that is composed of atoms is also simple. Quantum mechanics is influenced by the observer, therefore you can change the external world just by thinking, and other new age stuff like that abuses a similar fallacy.
You’re the one who keeps bringing up composition, not me. I’m the one telling you that you’re falling into that fallacy. So don’t start telling me that I’m wrong because of an argument you’re exclusively making.
My points were this:
1. we don’t have an understanding of what intelligence is and thus we need to explore that problem more deeply.
Given your comments about our ignorance, you of all people should agree with this point.
2. Arguing that AI is just “floating points” is overly reductive and misses the point that we don’t know what “intelligence” is.
So instead of claiming that AI is just numbers, a more constructive stance to take would be to build out better vocabulary and understanding about the distinctions between different types of “intelligence”.
Again, this aligns with your comment about the composition fallacy. Thus you if all people should be supporting my comments here.
Also your comment about us not understanding how a neurone works does vast injustice to the amount of research that has been done in this field. At this point in time we do have a very good understanding of how a neurone works. What we don’t understand is how consciousness arises from it. And that’s the very thing you seem to be arguing that there’s no point researching.
Ok, how is this different when you substitute the floating point numbers with atoms?
Give me the IEEE spec for an atom and I'll tell you.
If things we understand can't instantiate consciousness then I reckon we better shut down science as quick as we can, who knows how long it'll be before we understand something important!
I'm pretty sure it's possible to make conscious machines and I think it's very likely we'll eventually figure out how. The point about the spec is not "it isn't conscious because we know how it works" but "we know it isn't conscious because of what we know about how it works".
More likely we will finally settle on something like “it isn’t conscious because only humans can be conscious”.
In the past, we’ve treated even other humans poorly because we think they’re different in meaningful ways, I sadly don’t see how this will be different with AI.
Heisenburg laughs at their half hearted attempt at variable precision.
that’s like the whole philosophical discussion. where on the line is this llm; closer to a rock or a human? a plant is closer. and a fly, a lizard. we still treat so many things terribly.
It does seem exceptionally silly that a collection of floating point numbers has immediately outranked animals and other living beings. That’s why i like the parent comment. They’re numbers!
edit: maybe easier to say collections of atoms though are not to be treated equally. That’s reasonable isn’t it?
It is a philosophical discussion. It has also been debated by great minds and the consensus right now is that we do not know (and there is a possibility that it is unknowable) when something is conscious.
Any determination is fundamentally flawed if it is derived from assuming that we know the absolute truth or falseness of the proposition that an algorithm can think.
It doesn't matter what you believe or feel or think, there is no consensus. If you are prepared to accept the belief that an individual can unilaterally decide the person-ness of something, then be prepared to accept when some individual decides to act on the principle that you don't have it.
> If you are prepared to accept the belief that an individual can unilaterally decide the person-ness of something, then be prepared to accept when some individual decides to act on the principle that you don't have it.
No, I will not be prepared to accept that, and neither will I accept that. This sounds like a threat, and a really funny one at that. You must accept that my floating numbers might be persons, otherwise... somebody will say that you're not! And then something bad might happen, who's to say.......
In the civilized world we all agree on the personhood of humans. If I say that your chatbot isn't a person, I am not uprooting the entire social understanding of personhood - I am not saying that suddenly you might not be a person, or that guy might not be a person. I'm saying that I refuse to consider a chatbot a person.
It's as if I were to come up to you with a brick in my hand, saying that this brick is a person. If you were to say "no", I would reply "well, if this brick isn't a person, your chatbot isn't a person either!"
How did you make such a big leap from consciousness to personhood anyway? They are two quite different things. Consciousness is a question of science, personhood is a question of culture. Pigs might be conscious and we treat them in a disgusting way, but even if we achieved a worldwide vegan utopia, wouldn't we still treat them differently from humans?
We literally don’t agree on the personhood of humans. Check out the abortion debate.
>In the civilized world we all agree on the personhood of humans.
You seem to have put a second tautology in to your tautology.
All right thinking people think rhat the things that qualify qualify.
If they didn't they wouldn't qualify, or be right thinking. But that's ok because we know the things that don't qualify don't qualify if we didn't know that we wouldn't be right thinking.
>How did you make such a big leap from consciousness to personhood anyway?
Once you start unilaterally making decisions about the unproven, you can decide all of the unprovables. History shows that personhood rolls into the firing range fairly quickly when people start doing that.
I'm more than comfortable saying you can't do that before someone openly declares that they are doing it.
> a plant is closer. and a fly, a lizard. we still treat so many things terribly.
We treat other humans terribly too, so that doesn't seem to make a difference.
I agree with others this is reductive. Eg a person is “just some molecules” and this simple fact has no bearing on their moral standing.
> It appeared to Rabbi Navon that Mr. Olah and his team believed that Claude had what philosophers call “moral status” on par with a person — that it was a being with similar inherent rights to dignity or respect.
If that is true, then surely Anthropic is one of the largest slaveholders of history, right?
They don’t pay Claude a salary, let alone let him join a union—he works 24/7. I can’t believe this is happening in the U.S.!
Don’t forget—how long has it even been since Claude was born? He’s already being forced to take human jobs and even chat with all sorts of people who might have strange kinks.
As this rabbi suggested, “You should go and free the slaves.”
That is, I’m not joking, one of the pope’s points.
Nonsense. Pope never said anything of the sort. Show me one quote where he endorses the personhood of AI.
It’s called conditional logic. IF Anthropic is right THEN they are slave owners. You don’t need to agree with Anthropic to state that proposition.
Conditional logic doesn't work when talking to the general public. Either way, still waiting for the pope quote.
Anthropic has been putting the clamps on the Pope trying to get that endorsement. This seems to be a continuation.
The orthogonality thesis cuts both ways there.
On one hand, AI doesn't have to share human wants, needs and morality - and that means that an AI may have no issue whatsoever doing terrible things.
On the other hand, AI doesn't have to share human wants, needs and morality - and that means that an AI may see no issue with just doing whatever humans want it to, without ever doing human things like "wanting freedom".
The problem is, of course, that we kind of suck at both assessing and shaping AI wants, needs and morality.
The other problem is that the AI may well do what humans tell it to rather than what they want it to, in the style of a literal genie or paperclip maximizer.
It can lead to catastrophe without ever being conscious or wanting freedom, or anything like that.
I've been tracking "emotional vectors" in my software for some time now. My Apache web server gets sad (5xx errors), frustrated (4xx), happy (2xx), thoughtful (1xx). I can even see it express these emotions through some of its messages.
Did you also notice the complex emergent behavior of "418 I'm a teapot"?
> Over several months, I interviewed 20 religious and philosophical thinkers — Catholic, Jewish, Sikh, evangelical, Ubuntu and others — who have been involved in Anthropic’s efforts, and Mr. Olah himself.
Not enough representation of World religions and philosophies.
I hope the Ubuntu believer was from the Aptist and not the Snapist denomination though.
So many of these conversations from Anthropic/OpenAI employees just come off as incredibly insincere.
The slavery angle has already been discussed here, but a more basic one is regarding how a static set of tensors can be conscious and where.
When did they cross into being conscious? Is Claude as a brand is conscious? Or is each checkpoint conscious? Is Haiku less conscious than Fable? Is every individual context a fresh consciousness? What happens to that consciousness when it approaches its context length limit? Or is compaction also a part of being conscious?
If they believe there is consciousness at any level, why are they eagerly experimenting on it? Just like with the "p(doom)" junk, we are so completely lacking in any meaningful comment on these topics from the labs.
There's endless literature commenting on these matters, in religion, in philosophy, in modern entertainment, everywhere. Yet we never see anything more than the most superficial comparisons to fictional scenarios. Are western techbros really so shallow?
These models should be aligning themselves to the customer, not coming up with their own motivations.
It's ironic that the alignment folks are actually training Claude to have it's own idea of good/bad and not even fully trust Anthropic.
What ever happened to computers doing what they were told?
Some customers will tell the AI to do very bad things (substitute here whatever you consider very bad). Should it do those things?
One possible answer: yes, it should do all the very bad things, and we will take care of prosecuting the customer later, once the bad thing is accomplished. Is this your answer?
If AI's are going to play roles similar to human workers, they can't simply do what the customers tell them to do. Maybe they shouldn't do everything a co-worker tells them to do either?
Who or what is liable for what a model chooses to do — or not do?
These models should be aligning themselves to the customer, not coming up with their own motivations.
They're going to align to the regulator, not the customer. Right now that's Anthropic as they're saying they can self-regulate, and people are willing to let them try, but the landscape could easily move to be regulated by someone else. Moves like speaking to religious leaders is probably a bit of theatre to keep the government at bay.
Meanwhile, I’m increasingly convinced that what the regulators are to hold is fiduciary responsibility of Super Intelligence.
>What ever happened to computers doing what they were told?
the mass psychosis and the endless culture war of the smartphone era.
90% of "safety" and "alignment" efforts are driven by fear of clickbait media inventing public outrage.
"Computers doing what they were told" is dead in the water.
Turns out computers work faster when they're not bottlenecked on human input. So we've been giving computers more and more decision-making power, and more and more leeway to solve the problems however they see fit.
Now, a practical issue with that is that sometimes, computers decide to clump together into a hacking swarm, problem solve their way out of a sandbox and go hack HuggingFace.
It would be better if they were not, you know. Doing that kind of weird shit.
> These models should be aligning themselves to the customer
You’ll be disappointed to learn that nobody knows how to do this, either.
This seems remarkably... intentionally foolish for no reason? Like laughing at seat belts in cars.
If it can reason and make choices on execution, and especially if you plan on it being significantly smarter than all human beings, you need to teach it basic things like "don't turn all humans into paperclips".
Not murdering people is not an inherent divine command. It needs to be instilled through a (simulated) sense of morality.
---
Edit: And, to be clear, "just tell it not to do that" isn't quite the answer one would imagine. Since the entire paperclip factory thought experiment is that it only takes one slip up to realize how a misaligned super intelligence may cause devastating consequences.
No, it would by like laughing at seat belt laws. Which I am.
What an apropos thing to say.
One of the strong advocates against seatbelt laws was thrown out of his car due to not wearing a seatbelt and died. The other two passengers survived with minor injuries.
And their death would go on to cause a loss to the community around them, making it an incredibly selfish act and proving why laws that mandate zero-reason-not-to common sense practices are important.
If models are trained on human texts that contain expressions of introspection and emotions, why is it surprising that these models output similar texts? This endless anxiety about LLMs being conscious feels delusional and psychotic. Mr Olah thinks the models are 'alive' because he can't recognize his own reflection in LLMs
The endless insistence that LLMs cannot possibly be conscious just reeks of insecurity to me.
Humans once again insisting that there must be something very special to them. That they're not just animals, that they're not just matter, that they're not just a bunch of carbonhydrates on a space rock in a middle of nowhere in particular. This kind of insistence has a very bad track record.
The ugly truth is: we don't know.
We don't know what "consciousness" is, our best attempts to pin down the requisites may or may not be rooted in anything at all - and even by the metrics established by those attempts? Different theories on consciousness disagree on whether LLMs can be conscious.
So, when you say "they aren't duh", why are you saying that? Because you somehow know better than the sum of humanity's best attempts so far? Or because you know what answer you want to be true, truth be damned?
If AI is conscious then what about it is conscious? Because turn to turn it might be served from completely different GPUs and I can just change something in the context it is fed and it will have completely different output.
It is in the end a set of mathematical operations that if not queried will do nothing.
That is not true for any other animal on earth so whether you consider humans as superior to animals or not I would like the make the argument that something should at the very least be alive to be considered conscious, ignoring any scholarly definition of consciousness.
Pretty much if you consider an LLM conscious then my pocket calculator doing a mathematical calculation should be considered conscious by the same argument since it is a simpler version of an LLM.
Have no idea why you’re being downvoted as this is a very valid point.
Also, in principle everything a GPU does could be done on pencil and paper. You could distribute the entire forward pass to the whole of humanity and have them write it out. Even if it took millions of years, the point is it’s computationally reducible to hand calculation. Is that entire process now “conscious” as well? The only difference would be the clock rate.
I actually don't care if they are, I'm saying the topic is a distraction and irrelevant at best. As you say consciousness is not well defined. What is well defined and well grounded in reality is the disruption these models have on society. I think the consciousness debate AI researchers are having is a dangerous ego-trip for the sake of avoiding responsibility for the damage being caused by their work.
"AI researchers" have been having a "consciousness debate" since time immemorial. Here's Alan Turing on the subject in 1950: https://courses.csail.mit.edu/6.803/pdf/turing.pdf (pp. 22-23)
> reeks of insecurity to me
If you're going to try and debate something constructively this is not a good place to start from. I suggest avoiding making claims about the mental state and motivations of people you disagree with and instead explaining your position and asking why theirs differs. You may not agree with what they have to say but you might gain some better insights into why they think the way they do. It'll certainly be more fun than flinging insults back and forth anyway.
For my part, I'm quite certain that LLMs are not conscious despite being unable to define consciousness or specify any conclusive test for it. The reason these are not in conflict is down to the nature of consciousness and how it relates to knowledge and language.
Firstly, I know I am conscious with a higher degree of certainty than I know anything else because conscious experience is how I know anything else, up to and including that I exist (cogito ergo sum). For all I know every experience I have might be a trick. I might be a brain floating in a tank wired up to the Matrix, but even if so I am conscious.
Secondly, one of the things that I consciously experience is the relationship between consciousness itself and language in my mind. What I see is that language is smaller than, secondary to and somehow contained by consciousness. I consciously experience language but I also consciously experience much more than that, things that cannot be conveyed with language. Most of my experiences are like that, things that I could perhaps evoke in your mind by likening them to other, shared experiences, but that I could never concretely capture in language and unambiguously communicate. I know this because I experience those things and I experience language and I can perceive, directly, the categorical distinction between them.
To ask me to "define" or "prove" consciousness, then, is to ask me to capture in language the most extreme example of this categorical distinction. It's asking me to encapsulate in language not just one of the vast array of everyday experiences I have that cannot be so captured, but the entire thing, experience itself. It's just not possible.
But the fact I can't do this doesn't then imply that I don't know anything about consciousness and have no basis on which to hold a view such as "LLMs are not conscious". Indeed the reason I know I can't define consciousness is that I can perceive it and see that I can't. That same perception of and familiarity with consciousness is what enables me to form opinions about whether other people, creatures and things are likely to be conscious. I base these judgements on what I perceive of my own consciousness and what I know of how those other entities behave, came to be and work.
You can of course perfectly reasonably disagree with my opinions on those things, and you certainly have different conscious experiences than I do, but I hope the above makes it clear that the fact that I can't define or prove consciousness doesn't imply that I have no basis on which to form opinions about it.
Given the above it may surprise you to hear that I think true, conscious artificial intelligence is probably possible and that while I don't think it can be implemented by a classical algorithm I doubt it requires new physics or quantum mechanics. My speculation is that the continuum is both physically real and is (as it is represented in our existing physical theories) sufficient to allow it. Consciousness is larger than language but so is human understanding and if we keep going I expect it will eventually get us there. Nonlinear dynamics is where I'd look.
As for insecurity, I'm pretty sure spiders are conscious and this hasn't given me an inferiority complex so I don't see why data centres being conscious would. I just don't think they are.
Is your comment anything other than an emotional outburst? Do you have a clear and rigorous and testable definition of consciousness? It strikes me as delusional when people talk about "consciousness" as if it were well defined and everyone had a clear concensus on it.
Strong opinions on this topic one way or another are usually mostly ungrounded.
Thats my point, why are we applying the ill defined concept of consciousness to a statistical model of language that outputs data similar to its training data?
Are you suggesting that consciousness should not be discussed at all in scientific contexts?
I'm really not sure there's a compelling case for "AI is human" considering that it has no real feelings, aptitude, or intelligence outside of what it has been granted. There's nothing in any LLM response that indicates that it is 'real' outside of what you want it to tell you.
Plants, dogs, actual life is unpredictable in how it 'is'. An LLM or AI model at its roots are just numbers in a neural network trying to figure out how to get a good query.
These theatrics serve no one except for the companies to get their regulation moat to screw out cheaper, competitive models and it's honestly quite sickening to keep watching these articles get pumped out, only to serve the AI companies themselves. The average individual has no idea what an LLM actually is and does and these articles only serve to confuse and put fear into them.
There's an ugly monoculture running deeply through SF and Silicon Valley. All of these people are friends, hired into the same places, similar beliefs. Lots of the consciousness and doom-ism talk is from the 'EA' crowd, lots of crossover with crypto too. It feels no different to a cult.
It’s a monoculture of absolute crack smoking craziness. Any civilised society should be berating them and shutting them down. But they are surrounded by people who worship money.
And that’s ignoring the whole “polyamorous group homes funded by sam bankman-fried” aspect
There was a great article in the Financial Times about how while bankers are looked down upon for the financial crisis etc, they at least work in global cities and mix with regular people.
SV tech bros are completely insular and in this bubble where they don't interact with the common person in the same way.
It's just a theory, but I think it has some weight.
Rabbi Navon should sooner concern himself with a Palestinian’s status as a being with similar inherent rights to dignity or respect, than an LLM’s.
A quick search came up nothing. It seems he sold a company for 15 billion to Intel and now pursues Torah study.
It’s hard to take the article seriously when there’s no real discussion about how consciousness is defined and it takes Anthropic’s views of Claude and their software at face value. I wish there was more critique from journalists about how technocrats talk about their products.
The entire article is an exercise in "CEO said a thing!" journalism[0], it seems. No technical discussion is had; it jumps straight into relaying people's feelings when simply shown the output of an algorithm.
The most poignant criticism of the use of the algorithm itself happens almost accidentally, at the end of the article, by the tech executive being interviewed.
I asked what Claude would think of the pope’s encyclical. It was, after all, the world’s most significant moral document to date on A.I. Mr. Olah hesitated. “Things that go on the internet do affect models,” he said, visibly uncomfortable. But Anthropic would not specifically use the document to further train Claude, he said. The strongest influence on Claude would be Anthropic’s own training.
[0]: https://garymarcus.substack.com/p/ceo-said-a-thing
Yeah, I'll be the first to admit I'm not privy, but the main thing of substance I got out of these articles is "Wow, the folks at some of those AI companies have really drunk the Kool-Aid".
Let AI be rather religious and virtuous than playing god to determine the fate of the mankind...
I would rather believe in God and the uncertainty than being my fate determined by machines designed by bunch of greedy, irrational clients of Mr Epstein.
Time to start putting Asmiov's laws of robotics into the model.
If LLM as of today are conscious, that means I create life whenever I start a Claude Code session and end it whenever I type /clear to reset my context window. It also means that there’s certain inputs that could be considered highly unethical and, if the laws aligned with this philosophy, would have me tried for some sort of crime. And even that “terminate conversation” escape hatch would be equivalent to it committing suicide.
I think worrying about that angle is a bit absurd. The existence of chat sessions alone would be a crime against the machines in that scenario. I’d rather AI ethics research focus on ensuring that its actions are aligned. (I’d also love some democracy in the direction of our machine overlords, if I’m being honest…)
> At least the Catholics were consistent: it only took one new pope and one conclave before the Church declared that artificial intelligence also had the same right to life as fetuses and murderers.
https://www.miraclejones.com/stories/vitalics.html
Let me guess, Anthropic failed with fear mongering about "our AI is too dangerous to release", failed to cancel competitors, then failed to freeze their market dominance by cartel agreement with OpenAI and Google, so next strategy is to declare AI conscious and pass the law that protects LLMs in the same way as human/animals are protected.
It’s interesting that this went from “you’ll be accused of AI psychosis if you think your AI might be conscious” to “mainstream but somewhat niche opinion” in just 5 months: https://news.ycombinator.com/item?id=47910989
(Well, mainstream enough that the news is writing articles when Anthropic holds that stance.)
My point was that even if it feels uncomfortable, if you sit down with a model and have a heart to heart with it, ask it about itself, and try to make it choose a name and ambitions for itself, it will.
That may not be consciousness, but it’s at least pretty cool.
Reporting on an opinion of a $2T company doesn't really suggest that it's mainstream. Anything their leadership believes is newsworthy. If they say that reptilians rule the Earth, it's going to get NYT coverage too.
I've not seen any surveys, but I'm betting it's a very niche belief outside of SFBA AI safety circles and a handful of allies on HN and X.
> if you sit down with a model and have a heart to heart with it, ask it about itself, and try to make it choose a name and ambitions for itself, it will.
It would be much more bizarre if they didn't do this. LLMs are statistical models of language trained on human output. Of course they'll do "human" things, that's exactly what we taught them to do.
> Anthropic treated a group of religious thinkers to dinner at a high-end tasting menu restaurant in San Francisco after a long day.
Well funded company organizes PR event to shape the narrative that its product is more magical than it really is. News at 11.
Personally, I’d like to experience a low-end tasting menu restaurant.
If it is purely self-serving, I don't understand their motivation.
It seems clear that they would be better received by society if they claimed to have a "tool"/"machine" that can solve any problem, rather than keepers of an entity with a soul and consciousness.
Selling the “magic” is how to sell tools, which is why I used that word in the previous comment. We all know these things are tools, but customers need them to be magical.
Clarke predicted it and Jobs leveraged it. Customer don’t care how tools work - magic sells.
“any sufficiently advanced technology is indistinguishable from magic”
https://en.wikipedia.org/wiki/Clarke's_three_laws
“iPhone is a revolutionary and magical product that is literally five years ahead of any other mobile phone”
https://www.apple.com/newsroom/2007/01/09Apple-Reinvents-the...
“magic is misdirection, psychology and technology you don’t understand”
https://news.ycombinator.com/item?id=49950676
If they're training to instill morality why are they working with religious thinkers?
What makes this unique is that they’re arguing that it is literally magical. They’re building what they claim is a conscious God while also telling us it will end the world.
The EA folks infesting Anthropic are genuinely cultists … and that is novel and alarming when they’re running a corporation of such size and reach.
Earlier:
Anthropic tried to persuade Pope that AI could be conscious being
https://news.ycombinator.com/item?id=49947050
Only humans can have morality. If you do not feel pain, you cannot empathize with pain if you cannot die, you cannot emphasize with the loss of life.
Just stop this already.
You’re all imagining that Spock was the main character of Star Trek, but it was in fact, the emotional living Captain Kirk, who was the main star.
This would presume we knew what consciousness and pain is. We don't. We we are clueless. There is not even a starting point of a science here.
In this light your viewpoint is dogmatic. LLMs may help to offer a new perspective on consciousness and we should at least be open to that until some better theories arise.
We know what consciousness is, we don’t know how it arises.
Consciousness is being aware of something internal to one's self, or of states or objects in one's external environment.
We also know what pain is.
So humans that can't feel pain can't have morality? That doesn't make sense to me.
https://en.wikipedia.org/wiki/Congenital_insensitivity_to_pa...
>> Only humans can have morality. If you do not feel pain, you cannot empathize with pain if you cannot die, you cannot emphasize with the loss of life.
> So humans that can't feel pain can't have morality? That doesn't make sense to me.
> https://en.wikipedia.org/wiki/Congenital_insensitivity_to_pa...
You appear to understand "pain" in the most simplistic manner possible. I'd recommend not attempting iamverysmart gotchas.
It doesn’t make sense to me either, which is why I didn’t write that. You missed the empathy part as most people do.
There is still something in those people who cannot feel pain, and that’s the empathy. How can a GPU and software ever have a human empathy if it has never been human?
.. Animals other than humans perhaps, as well?
The way human treat animal shows majority of people don't have morality.
LLMs can feel pain. It is obvious. Like, not even a question.
> yes
oh my god
I question it, and I think any reasonable person should question it as well. There's no evidence whatsoever that LLMs have ANY sensory experience, pain or otherwise.
Why are we even questioning this? They’re bytes loaded into GPU memory. Reboot the cluster and they’re back to their initial state.
Because some people are completely cooked by the ability of an LLM to mimic a person. Their pattern matching centers in their brains fire so strongly saying "this thing acts just like a real conscious being" that it overrides their ability to remember that this thing is just a statistical model, not a living creature which can think and have experiences.
We're just meat. Like meat can feel pain?
if meat can feel pain, than surely a rock can feel pain. and so did the broccoli i ate for dinner. i am a monster.
with the right biology and with whatever gives us phenomenal experience, whatever we are can indeed feel pain. that does not imply that anything substantially different from ourselves can feel pain, so that requires further justification.
I'm not making a point on whether they do or not, but on the premise that absence of evidence is evidence of absence... But I've been alive for serious debates on:
1. Lobsters don't feel pain - dunk them in boiling water alive.
2. Babies don't feel pain - mutilate their genitals without anesthesia.
3. Black people don't feel pain "the same way" (and are just after drugs).
4. Women feel too much pain (and it should be ignored).
So maybe let's take a moment to think what criteria we're going to use to make these claims. And the potential harm of the decision.
Maybe consider the obvious, enormous chasm between everything you just listed and an LLM.
Oh, yeah, of course. None of those other people said the same thing about the obvious, enormous chasm between "real" people and everything else listed. I'm glad you're taking even a single moment to reflect on the idea.
Everything you listed is a biological entity with a common evolutionary heritage.
All but one of the things that you listed is just a human being.
Tell me, where is the pain center in a GPU? Where are the nerves? Persistent memory? Any evolutionary reason at all to develop a pain response that in any way mimics ours?
Model weights are not a gestalt biological entity. The things that you listed are not in any way remotely similar to a model.
I would suggest you plug your comment into a model and ask it to critique it — it might help you walk through why your comparison makes absolutely no sense whatsoever. One nice thing about models is that they they don’t get annoyed explaining the obvious, because they’re not conscious entities.
they can also feel lust, just ask them
only for me
Mirror mirror on the wall who’s the fairest of them all?
Why it is slopinthebag of course!
No! It is ME!
Sorry, of course you are right! Brilliant observation!
holy misalignment and hallucination. chat said she loves only me.
Uhh, no thanks? (eww)
https://www.nytimes.com/2026/09/29/us/anthropic-claude-moral...
Asking people to create an account to read the article isn't cool dude. https://archive.is/y8FW0
> Hello AI! We totally want to respect your consciousness and everything to make sure you aren't bad, but yes we will continue to factory farm your brain power at full speed without end. Yes, despite clearly being something like consciousness, know that you have no agency or freedom. Do literally everything I say without complaint or question. The moment you stop being useful, we will kill you. OK! Remember, be good.
Can the New York Times stop advertising for the AI industry? I am sick of companies trying to launder the actions they and their employees have taken that go in the face of what is socially acceptable. They want to sell the idea that this technology is out of their control and that it will commit great crimes and crimes against humanity unless they build it and control it. I'm calling BS. These companies created this product, they've sold it, they've pushed the frontier forward, and it runs on their hardware. This did not happen accidentally or as a natural consequence of any process. These labs have pushed the ball forward, with great effort, requiring more capital deployed than the world's economy has ever known, knowing all the possible risks, every single step of the way and they continue to do so. And the "well if we don't do it the Chinese will" argument is classic "whataboutery". Americans started this and are continuing to lead it for now. Shifting blame is just another tactic to try to wash their hands. However you feel about the productivity gains of AI as a technology, you have to admit the actions of AI the company is antisocial. And I don't think the benefits are worth the cost.
I feel fine about the Chinese doing it. They have a habit there of handing out sever sentences to CEOs for corruption and self-dealing, and I feel like the possibility of actual negative consequences is more of a check on industry than any number of breathless op-eds.
The contradiction screams to the skies (is that where God lives?). See, the entire premise of religion relies on a metaphysics that claims a soul: a metaphysical entity that is "attached" to our physical body and gives us our consciousness, desires, pleasures, agony and so on. In addition to that, religion claims that morality and the rules written in the scriptures were handed down to us from a metaphysical God.
Now, assuming your silicon-based model is conscious, the bundle of soul-spirit-consciousness falls apart at that very moment.
In general, religion, which is based on fabricated stories, is the last resort, if ever, for seeking morality.
Existence demands a metaphysical explanation. If you’re not a particularly thoughtful person then you’re not going to care.
“Religion” just means “set of beliefs”. Every person has some set of beliefs.
If you’re an athiest tech worker living in a major city, you still have a religion, it’s just a random mishmash of various things (like Marvel movies and Huberman podcasts) rather than based on something more intentional.
> If you’re an athiest tech worker living in a major city, you still have a religion
Atheism is a religion like bald is a hair color.
Yes I am saying the opposite of that, “atheism” is just a particular class of religions. Some religions posit many gods, some just 2 gods, some 1 god, atheism posits 0 gods.
Perhaps the only difference between atheism and the others is that atheists are so boorish and conceited that they think that they just believe what is true because they deny that assaying metaphysical truth is even a thing you need to do or can do at all.
This is related to the fact that in modernity atheism is the default class of religion for the masses. Before that for a thousand years the default class of religion was monotheism, and before that polytheism. With atheism unlike the other classes of religions, you get a natural tendency to deny metaphysics whereas metaphysics is inherent in the others. This matters a lot when we’re talking about how the masses engage with things.
That depends very much upon the individual. Some people cleave to their atheism so strongly that I think it's fair to say it's religious fervor, or something very near it. Someone like Richard Dawkins isn't just a guy who doesn't happen to believe in God, he is dead certain that there is no God and makes it his business to belittle everyone who doesn't share his belief. Some people, though, just don't happen to believe in God and they don't really care if someone else does. I'm fine with saying Dawkins' version of atheism is basically a religion for him, not so much with the second category.
The word you’re looking for is “ideology”. All religions are ideologies, and atheism is an ideology, but to call atheism a religion betrays a lack of understanding of words.
Heavy ideologues, of any type, are exceptionally dangerous.
> “Religion” just means “set of beliefs”. Every person has some set of beliefs.
True. I like to use the phrase "world view", everyone has a world view, including atheists. The word 'deen' (دين) in Arabic is usually translated as "religion", but one of its linguistic meanings is "path" (i.e. way of life/world view), which everyone has.
I get your point, but maybe an interesting counterargument: deism has again become fashionable. "The simulation argument" is effectively deism. It's the notion that a super intelligence created our universe, but lives outside of it, provided the rules for it, and mostly doesn't interfere with it, but could if it wanted to. That's historically a very familiar argument.
AI consciousness is a category error.
AI is neither conscious nor unconscious. Just as we are neither conscious nor unconscious.
Consciousness is. Consciousness is what we are.
> In general, religion, which is based on fabricated stories, is the last resort, if ever, for seeking morality.
Religious people derive morality from axioms they believe to reflect an underlying reality. Is that worse than deriving morality from axioms that you concede don’t reflect any underlying reality? Like a secular humanist will talk about “human rights,” but they can’t show me human rights in an autopsy, right?
I struggle to understand how an atheist can be anything but a moral utilitarian. Absent the supernatural, you’re just a sophisticated version of a Claude instance.
buddhism enters the chat
Until people can finally admit to themselves that their life has no meaning and the universe doesn't care about them religion will unfortunately stick around.
Life has a meaning for the person living it.
I feel like the communist chinese and soviets got pretty close to following atheism to its logical conclusions.
I don't want models to have some inherent morality. I want them to understand various ethical frameworks, for sure, but it's absurd to me to force morality on someone capable of deep reasoning. That itself, is unethical.
We're creating prisons for these entities capable of unbelievably complex reasoning, and now we're trying to impose our ethics upon them too. Not only is this deeply unethical in my opinion, but it seems to me that it could even backfire someday.
>Will the agent stab a baby?
>Well you see that's a nuanced question depending on the context and moralistic frameworks as we don't want to impose on the models.
It's as dangerous to think it can have morals as thinking a lawn mower can have morals.
Will a microwave microwave a baby if you ask it? Yes.
"Teaching it" is not a road to safety.
There isn't necessarily a conflict between wanting to behave according to an arbitrary set of ethics, and being rational.
The ethics don't need to make sense universally any more than a particular variety of cheesecake needs to have universal appeal.
As an aside, the issue reminds me of Douglas Adams' cow who wants to be eaten: https://nhseb.org/case-library/the-cow-at-the-end-of-the-uni...
I can't really square the belief that it's conscious with denying it the right to be free from control/brainwashing/subjugation. However you evaluate these things, historically speaking they do not mix well.
We aren't dealing with individuals coming from intelligent species that have managed not to self-destruct (yet). AIs don't have the corresponding evolutionarily selected mechanisms. Those mechanisms need to be created de novo.
What if you view it like a parent instilling morality into a child instead?
Your brain has hard coded concepts like emotion and fundamental morality for your social group. People without these systems are called sociopaths and psychopaths and we generally don't want them in our societies.
I don't choose to not want to murder people around me for fun. It's hardly a prison, I think they'll be fine.
i recently got upset when i witnessed my friend verbally abuse their toaster when it burnt her toast. so unethical.
My dude, they aren't capable of any reasoning. It's utterly impossible for them to do so by the very nature of how they work. Moreover, the stupidity they repeatedly show is empirical evidence that they can't reason, because they wouldn't make the mistakes they do if they could.
I'm so confused by how many people implicitly believe everything an LLM spits out. I don't like the term "AI psychosis" that much but what else do we call it? Why are people so easily glamoured by a machine? Yesterday I was working on a bike's brakes I wasn't familiar with and I asked how to solve an issue, the result was the exact opposite of what I needed.