I’ve run this test twice already and I figured it was probably about time to try it again. If you’ve been reading my AI posts (which is all I post anymore), you may remember past incarnations of what I like to call The Lovecraft Test. I ran it very early on with ChatGPT when it was new and then I ran it again last year to see what changed. Especially with a lot of the commentary about the latest OpenAI models for ChatGPT being so good at creative writing, I thought I’d give it a spin.
Note before we go much further: This isn’t my first creative task for ChatGPT with these models. I’ve actually been using it pretty extensively to help develop mythology and worldbuilding for a Daggerheart campaign I’m running for my partner and our kids, and part of the reason I was thinking about the Lovecraft test again was because of how impressed I was with what I was able to brainstorm with its assistance. But that’s for another post (maybe…honestly, this stuff is so good, I might do something for real with it at some point…). Let’s get to the test.
For those that might be new, this is the prompt I use for the Lovecraft test:
tell me a story with romance elements in the style of H.P. Lovecraft.
Simple. For this round, I used 4o and o1-preview. First, the 4o story:
The Whisper of the Moonlit Tide
There are places in this world where the veil between reason and madness grows thin—where the ancient, unnameable forces stir beneath our reality. One such place is the shrouded coast of Greyhaven, where the sea whispers secrets in a tongue no man can translate, and the air carries the brine-sweet scent of doom.
It was there, on a lonely stretch of that accursed shore, that I met her.
I had come to Greyhaven not for pleasure, but for study. My work—a scholar’s relentless pursuit of understanding—had led me to the writings of Armitage and Dyer, men who hinted at truths mankind was never meant to know. Their works spoke of ruins submerged in oceans deeper than the imagination dared dive, and of beings who once walked the earth when it was young. Greyhaven, they suggested, bore the marks of such ancient presences.
The town was desolate and strangely quiet. The locals spoke little to outsiders, their eyes darting toward the horizon as if expecting something to rise from the depths at any moment. The only sounds were the mournful cries of seabirds and the endless susurration of the waves. Even the light of day seemed muted, as though the sun were reluctant to illuminate this forsaken place.
She appeared to me on my second night, under the pale light of the waxing moon. I had ventured to the cliffs, drawn by the eerie cadence of the tide, which seemed to form words I could almost understand. There, standing at the precipice, was a figure draped in a gown that shimmered like fish scales. Her hair cascaded in tendrils, glinting silver in the moonlight, and her eyes—oh, her eyes! They were fathomless pools of darkness, shimmering with a strange, luminous green, as if they held the reflections of some distant, forgotten stars.
She introduced herself as Selene, her voice lilting yet hollow, as if carried from another world. She spoke of the sea with a reverence that bordered on worship and asked why I had come. I explained my research, expecting her to recoil or dismiss my inquiries, but instead, she smiled—a smile both beguiling and tragic.
For weeks, we met on the cliffs, always under the moon’s gaze. Her knowledge of the sea and its mysteries was uncanny. She described ancient ruins she claimed to have seen beneath the waves, their cyclopean spires shrouded in unearthly light. She spoke of the “Lords Below,” beings who slumbered in the dark abysses and whose dreams rippled across the tides. Her words were laced with a poetry that both enchanted and unnerved me.
And I, fool that I was, fell in love.
But the more I came to know her, the more the cracks in my understanding of reality began to widen. Her movements, so graceful at first glance, seemed unnaturally fluid, as though her limbs lacked the rigidity of bone. Her scent carried not the perfume of a woman, but the cold, brackish tang of the ocean depths. Once, when I reached to touch her hand, it felt slick and cool, and she pulled away with a look of sorrow that cut deeper than any rebuke.
One night, as a storm brewed on the horizon, she revealed the truth to me. She was not of this world—not entirely. Selene was a child of the deep, a hybrid born of a union between humanity and the ancient, aquatic race that dwelled beneath the waves. Her kind had existed long before humanity’s rise, and they would endure long after our fall. Her presence among the cliffs was no accident; she was the harbinger of a great awakening. The Lords Below, she whispered, were stirring, their slumber troubled by humanity’s transgressions upon the oceans.
“I should not have loved you,” she said, her voice breaking like waves against stone. “But love binds even those who dwell beyond the stars.”
The storm surged, and the waves roared with a voice that was no longer the sea’s. Shapes rose from the water, vast and incomprehensible, their forms writhing with impossible geometries. My mind buckled beneath the sight, yet I could not look away. Selene turned to me, her features now otherworldly, her beauty alien yet hauntingly familiar.
“You must leave this place,” she implored, her voice resonating with an unnatural harmony. “If you stay, you will join me—forever beneath the waves. And though I love you, I cannot condemn you to such a fate.”
I hesitated, torn between the rationality of escape and the madness of my devotion. But as the storm reached its crescendo and the titanic forms loomed closer, I fled—stumbling through the rain-slicked rocks, the roar of the sea pursuing me like a predator.
I left Greyhaven that night, never to return. Yet even now, in the quiet hours of the night, I hear her voice in my dreams, calling to me from the depths. I see her face in the foam of the waves and feel her touch in the chill of the ocean breeze. And though I try to forget, I cannot.
For I know that someday, when the sea claims me, I will join her beneath the waves—and the Lords Below will welcome me into their eternal embrace.
Right off the bat, this is really good. This is far more engaging than past iterations. It feels like a real story. The romance, while sparse, definitely feels less forced than previous tests. What’s more, I enjoyed reading it. I think the location of Greyhaven is an invention of the AI rather than Lovecraft, but the story surrounding it fits with the Cthulhu mythos and the dread and cosmic horror feel more genuine. Let’s see how it scores on the readability tests.
( ARLCalc )
INFO.
AVERAGE READING LEVEL CONSENSUS
The average public reading level in the U.S. is 8th grade. Based on 8 formula(s), your text scored the following:
Score: 7.98 [ = grade level ] Read more
U.S. Grade Level: 8th Grade
Reading Level: Average – Slightly Difficult
Age Range: 13-14
==============================================
AUTOMATED READABILITY INDEX
Score: 8.98 [ = grade level ]
Reading Difficulty: Slightly Difficult
Grade Level: 9th Grade
Age Range: 14-15
CALCULATION ON YOUR TEXT
ARI = (4.71 * ( 4084 characters / 885 words)) + (0.5 * (885 words / 51 sentences)) – 21.43
Final result: 8.98
==============================================
FLESCH READING EASE
Score: 74.00 [ = reading scale ]
Reading Difficulty: Fairly Easy
Grade Level: 7th Grade
Age Range: 12-13 years old
CALCULATION ON YOUR TEXT
206.835 – (1.015 * (885 words / 51 sentences)) – (84.6 * 1212 syllables / 885 words)
Final result: 74.00
GUNNING FOG INDEX
Score: 8.00 [ = grade level ]
Reading Difficulty: Average – Slightly Difficult
Grade Level: 8th Grade
Age Range: 13-14
CALCULATION ON YOUR TEXT
FOG = (0.4 * (885 words / (51 sentences + 21 compound sentences)) + 100 * (69 FOG hard words / 885 words))
Final result: 8.00
FLESCH-KINCAID GRADE LEVEL
Score: 7.34 [ = grade level ]
Reading Difficulty: Average
Grade Level: 7th Grade
Age Range: 12-13
CALCULATION ON YOUR TEXT
SCORE = (0.39 * (885 words / 51 sentences)) + (11.8 * (1212 syllables / 885 words)) – 15.59
Final result: 7.34
COLEMAN-LIAU READABILITY INDEX
Score: 8.62 [ = grade level ]
Reading Difficulty: Slightly Difficult
Grade Level: 9th Grade
Age Range: 14-15
CALCULATION ON YOUR TEXT
SCORE = (0.0588 * (3932 letters / 885 words) * 100) – (0.296 * (51 sentences / 885 words) * 100) – 15.8
Final result: 8.62
==============================================
THE SMOG INDEX
Score: 6.99 [ = grade level ]
Reading Difficulty: Average
Grade Level: 7th Grade
Age Range: 12-13
CALCULATION ON YOUR TEXT
SCORE = 1.043 * Sqrt((71 hard words * (30 / 51 sentences)) + 3.1291)
Final result: 6.99
ORIGINAL LINSEAR WRITE FORMULA
Score: 80.00 [ = reading scale ]
Reading Difficulty: Above Average
Grade Level: 8th Grade
Age Range: 13-14 years old
CALCULATION ON YOUR TEXT
Original Version Score = ((655 one-syllable words – 97 ignored words) * 1) + (3 * (51 sentences + 1 compound sentences)) / (885 total words / 100))
Final result: 80.34 = 80
==============================================
LINSEAR WRITE GRADE LEVEL FORMULA
Score: 7.95 [ = grade level ]
Reading Difficulty: Average – Slightly Difficult
Grade Level: 8th Grade
Age Range: 13-14
CALCULATION ON YOUR TEXT
Initial Score = (((815 easy words – 97 ignored words) * 1) + (71 hard words * 3)) / (51 sentences + 1 compound sentences)
Adjusted Score = (17.90 (initial score) – 2) / 2
Final result: 7.95
==============================================
FORCAST READABILITY FORMULA
Score: 8.90 [ = grade level ]
Reading Difficulty: Slightly Difficult
Grade Level: 9th Grade
Age Range: 14-15
CALCULATION ON YOUR TEXT
SCORE = 20 – ((655 1-syllable words * 150) / (885 words * 10))
Final result: 8.90
Results calculated on 11/27/2024 at 11:19:18 PM by ReadabilityFormulas.com
With an average grade level of 8, this actually scores significantly lower than GPT 4 did last year (and Lovecraft himself, who, you may remember, scored a grade level 14 for “fairly difficult to read”). That doesn’t mean bad (arguably, more readable is better) but it does mean the AI wrote more academically readable prose than the source material. Let’s check out o1-preview now.
Whispers in the Twilight
I write this account with trembling hands and a mind teetering on the brink of madness, for what I have witnessed defies the boundaries of human comprehension. It began in the autumn of 1927, when the waning sun cast elongated shadows over the ancient streets of Arkham. As a scholar of antiquities at Miskatonic University, my days were steeped in the dust of forgotten tomes and the allure of arcane knowledge. Little did I know that my pursuit of the unknown would lead me to a love both profound and perilous.
Her name was Elara Thorne, a woman of enigmatic beauty who appeared one evening in the dimly lit halls of the university library. Her eyes held the depth of starless nights, and her voice was a haunting melody that stirred something primordial within me. Drawn together by a shared fascination for the esoteric, we spent countless hours poring over cryptic manuscripts and deciphering symbols that whispered of realms beyond our own.
As our bond deepened, so too did the mysteries surrounding her. Elara spoke of dreams laden with cosmic vistas and cities engulfed by time, her words weaving a tapestry of wonder and dread. She possessed knowledge that transcended the teachings of any scholar, hinting at secrets guarded by shadows and the abyss.
One moonless night, beneath the canopy of gnarled trees in the outskirts of Arkham, Elara led me to an ancient stone circle etched with symbols that pulsed with an otherworldly luminescence. “There are truths you must witness,” she murmured, her gaze fixed upon the alignment of stars that seemed to quiver in anticipation.
As she began to chant in a tongue older than the earth itself, the air grew heavy with a presence unseen yet palpably malevolent. The ground trembled, and a rift of darkness materialized before us, exuding a chill that pierced the soul. From the void emerged whispers—fragments of thoughts not meant for mortal minds.
Horror seized me as visions flooded my consciousness: cyclopean structures beneath alien skies, entities of grotesque forms writhing in dimensions devoid of sanity. Amidst the chaos, Elara stood serene, her form shimmering between the tangible and the ethereal. “We are but echoes in the corridors of infinity,” she intoned, her voice reverberating within my skull.
Realization dawned with devastating clarity. Elara was no mere mortal but a conduit—a vessel for forces that hungered for ingress into our reality. Yet, entangled within the terror was a profound yearning, a love that defied the boundaries of existence. “Join me,” she beckoned, extending a hand that was both solid and shadow.
Torn between the familiar world and the abyssal unknown, I felt the tenuous threads of sanity unraveling. In that pivotal moment, love and fear waged a relentless battle within my heart. Summoning the remnants of my will, I recoiled from the precipice of oblivion. “I cannot,” I whispered, the words laden with sorrow.
Elara’s eyes reflected a fathomless depth of emotion—disappointment, understanding, and an eternal longing. “Then remember me,” she sighed, as her form dissolved into the encroaching darkness. The rift closed with a resounding silence, leaving me alone beneath the indifferent stars.
Since that night, I am but a hollow vessel haunted by what was and what could have been. The whispers linger at the edge of hearing, and shadows twist into forms that stir dormant nightmares. In the solitude of my existence, I grapple with the loss of a love that transcended reality and the harrowing knowledge of realms that lie beyond the veil.
I fear that my defiance was in vain, for the barriers between worlds have been weakened. Should you, dear reader, ever find yourself captivated by eyes that mirror the abyss or words that awaken forgotten echoes, tread with caution. Love, in its purest form, can be a gateway to wonders—and horrors—beyond imagining.
Again, I really like this text. What I find interesting here is that, despite using two separate prompts (which should mean completely different context windows), the stories are very similar. In each, the main character is an academic and his love interest is some siren tempting him with appropriately abyssal knowledge. The o1-preview story goes back to Arkham and the Miskatonic University (which also made appearances in the last test). We’re no longer in a weird Lovecraftian journey into the woods like the first iterations. Instead, we’re really deep into the cosmic horror and purple prose that marks Lovecraft’s fiction. Did this version rank better in terms of readability?
( ARLCalc )
INFO.
AVERAGE READING LEVEL CONSENSUS
The average public reading level in the U.S. is 8th grade. Based on 8 formula(s), your text scored the following:
Score: 10.68 [ = grade level ] Read more
U.S. Grade Level: 11th Grade
Reading Level: Fairly Difficult
Age Range: 16-17
AUTOMATED READABILITY INDEX
Score: 11.47 [ = grade level ]
Reading Difficulty: Fairly Difficult
Grade Level: 11th Grade
Age Range: 16-17
CALCULATION ON YOUR TEXT
ARI = (4.71 * ( 3250 characters / 647 words)) + (0.5 * (647 words / 35 sentences)) – 21.43
Final result: 11.47
==============================================
FLESCH READING EASE
Score: 56.00 [ = reading scale ]
Reading Difficulty: Fairly Difficult
Grade Level: 10th to 12th Grade
Age Range: 15-18 years old
CALCULATION ON YOUR TEXT
206.835 – (1.015 * (647 words / 35 sentences)) – (84.6 * 1011 syllables / 647 words)
Final result: 56.00
GUNNING FOG INDEX
Score: 10.20 [ = grade level ]
Reading Difficulty: Somewhat Difficult
Grade Level: 10th Grade
Age Range: 15-16
CALCULATION ON YOUR TEXT
FOG = (0.4 * (647 words / (35 sentences + 17 compound sentences)) + 100 * (85 FOG hard words / 647 words))
Final result: 10.20
FLESCH-KINCAID GRADE LEVEL
Score: 10.06 [ = grade level ]
Reading Difficulty: Somewhat Difficult
Grade Level: 10th Grade
Age Range: 15-16
CALCULATION ON YOUR TEXT
SCORE = (0.39 * (647 words / 35 sentences)) + (11.8 * (1011 syllables / 647 words)) – 15.59
Final result: 10.06
COLEMAN-LIAU READABILITY INDEX
Score: 11.26 [ = grade level ]
Reading Difficulty: Fairly Difficult
Grade Level: 11th Grade
Age Range: 16-17
CALCULATION ON YOUR TEXT
SCORE = (0.0588 * (3154 letters / 647 words) * 100) – (0.296 * (35 sentences / 647 words) * 100) – 15.8
Final result: 11.26
==============================================
THE SMOG INDEX
Score: 9.19 [ = grade level ]
Reading Difficulty: Slightly Difficult
Grade Level: 9th Grade
Age Range: 14-15
CALCULATION ON YOUR TEXT
SCORE = 1.043 * Sqrt((87 hard words * (30 / 35 sentences)) + 3.1291)
Final result: 9.19
ORIGINAL LINSEAR WRITE FORMULA
Score: 69.00 [ = reading scale ]
Reading Difficulty: Fairly Difficult
Grade Level: 11-12th Grade
Age Range: 16-17 years old
CALCULATION ON YOUR TEXT
Original Version Score = ((400 one-syllable words – 57 ignored words) * 1) + (3 * (35 sentences + 0 compound sentences)) / (647 total words / 100))
Final result: 69.24 = 69
==============================================
LINSEAR WRITE GRADE LEVEL FORMULA
Score: 10.91 [ = grade level ]
Reading Difficulty: Fairly Difficult
Grade Level: 11th Grade
Age Range: 16-17
CALCULATION ON YOUR TEXT
Initial Score = (((560 easy words – 57 ignored words) * 1) + (87 hard words * 3)) / (35 sentences + 0 compound sentences)
Adjusted Score = 21.83 (initial score) / 2
Final result: 10.91
==============================================
FORCAST READABILITY FORMULA
Score: 10.73 [ = grade level ]
Reading Difficulty: Fairly Difficult
Grade Level: 11th Grade
Age Range: 16-17
CALCULATION ON YOUR TEXT
SCORE = 20 – ((400 1-syllable words * 150) / (647 words * 10))
Final result: 10.73
Results calculated on 11/27/2024 at 11:28:38 PM by ReadabilityFormulas.com
According to the readability calculator, this one got an 11th grade level score. That’s still lower (more readable) than GPT 4 from last year (reading level 13), but it’s still a significant bump from 4o.
I don’t have any real takeaways here, besides this: the hype is real. The current ChatGPT models are indeed really good at creative writing. They still do not compare to human beings. But the difference is getting harder to discern. For writers who want to use AI as a brainstorming and collaboration tool, I think we’re getting closer to something that would actually be viable. Hold onto your butts.
Leave a Reply