Discussion about this post

User's avatar
Renee Magnusson's avatar

i think that because it "blows smoke up your ass" people are going are gong to become highly addicted to it. It's going to be a problem.

Expand full comment
Vanessa Siino Haack's avatar

This is fucking amazing. Thank you for doing this and showing us what you did. I've used it for writing first drafts of letters of recommendation, and I've asked it to write like my writing samples, and it does a great job of structuring, but, yes, it's the form of it, not the substance. and I actually was listening to a Forum this morning on AI (https://www.kqed.org/forum/2010101909874/whats-next-in-artificial-intelligence) and thinking one or two things you end up going into deeper here, and so this is just some damn synchronicity.

Things like my rational brain deciding it is *just* mechanistic but then some other part of me thinking how I really deeply believe in a expanded idea of life and consciousness and so where does that leave me if I trust both those ways of knowing.

And things like thinking about what it might be like to have high school students explore the difference between their own thinking and AI thinking on a subject---like make the work they do with it very meta--but then i thought . . . if they're still learning how to think in specific ways, will they know enough to really get beyond what AI is doing? Because when you are learning something, you are doing it in a more mechanistic way... apprenticing into the discourse by play acting it, using the terms and the sentence structures as a way in to becoming more "expert." Will they just conclude that AI is great because they don't yet appreciate the difference (and thinking how I might get them questioning beyond that.) And, again, where does that leave us on this living/consciousness question?

So then, when I read the first AI draft you posted, (well, first I thought... wtf :0, but then ...) I thought, that reminds me of nothing so much as the other day when James (9yo) was play-acting a Very Serious Role. And I don't remember what he was saying exactly, but it made some sort of internal sense and it had a very specific, identifiable tone. And I was very impressed with his ability to imitate and was so curious about how that happens in a brain and what might come of this as he grows and learns (acting ability? identity formation?) . . . so what is it that I felt when I felt proud of Chat GPT for doing a good job on your writing? Like I wanted to pat it on the head and tell it, "wow! your really nailed some of the vocabulary and rhythm, but also how do you think you might..."

Because that is a weird feeling. And I know that these large language models have been trained to respond in ways that feel human, and the thing I listened to this morning reinforced what you identified about blowing smoke up your ass ... they were calling it sycophancy, and it's trained to do that because humans rate the models higher when the models praise them.

But I love that you are wrestling with this with us here. and just wow and damn.

Expand full comment
9 more comments...

No posts