advanced docs

The readability of AI-generated text

August 13, 2024
4-min read

Written by Victoria Rudi

Educating on SaaS messaging & team comms. Helping SaaS people with messaging across all touchpoints.
This doc explains my concerns about AI generative tools for creating content.

In the beginning of August, I received an email update from the CEO of an async work platform.

Side note: I don’t use the platform, but as I SaaS-watch for fun—similar to bird-watching— I’m subscribed to the company’s newsletter.

And to my surprise, the CEO announced their decision to sunset the platform.

The reason?

A change in priorities.

The team wants to focus on a generative AI tool for content creation. This decision piqued my interest, so I decided to look closer into their new AI tool.

And what I discovered made me worry.

Worry about the future of businesses that use AI-generated content.

Look, I’m fascinated by AI. I think it’s great that we can use it to create new solutions. Especially when it comes to saving time and energy on content creation.

But we must discuss issues and worries to avoid them.

To formulate my analysis, I’ll talk about the AI tool I mentioned in the beginning. But I won’t mention its name. I want to keep its anonymity for ethical reasons.

I’ll use the symbol [hidden] throughout the text.

My shock

There’s one thing about [hidden] that shocked me the most.

And that’s the readability of the articles the company has published so far.

Their articles are very difficult to read. To confirm my intuition, I ran one of its articles through the Hemingway App.

I almost fell off my chair when I saw the readability score. It was 14.

The readability of [hidden] article

In case you don’t know, a 14 score means the text is very hard to read. Readers will have to exert huge mental effort and patience to understand the article.

And that’s a big problem.

Why?

The problem

Think information overload. Readers (leads and users) have no energy or desire to decode complex articles and text.

They won’t think twice before moving away from an AI-generated article that is difficult to read. And that’s a problem for companies using AI tools for content creation. They may create content to engage and connect with their leads and users. But that’s impossible if they keep readability levels so high.

But is readability a problem all AI-generated content suffers from?

My experiment

I assumed that [hidden] used its own tool to generate the article, so I had to test it.

I created a free trial and added a topic to draft the article outline. Then, I followed the instructions and let the AI tool generate my article.

I even specified that I wanted a clear and concise output.

Unsurprisingly, the output was very difficult to read. You could see it with the naked eye. However, I ran it through the Hemingway App and indeed, the readability score was 15.

The readability of the article generated with [hidden]

So my initial assumption was correct.

The [hidden] articles were created by the [hidden] AI tool.

After seeing the Hemingway App results, I checked the [hidden] interface again. I searched for an option to change the readability level. But I didn’t find any.

And that’s a recurring experience I’ve had with other generative AI tools for content creation. Texts being too complex and difficult to read.

That’s worrisome for several reasons.

Side note: Many SaaS people use generative AI tools to create content, copy, and ads. So I’ll focus on the worries I have for the SaaS industry.

Why worry?

  • Some expert voices talk about the need to rewrite the AI output. But most of their focus goes on ‘humanizing’ the AI text. They talk about adding inconsistencies and personal stories. Yet,  there’s nothing about readability. It’s as if we don’t even have the problem of AI texts that are complex and difficult to read.
  • You can find information on how to ‘humanize’ AI text. You can also find information on how to make AI text undetectable. However, there’s almost no useful advice on how to improve and simplify AI-generated text.
  • SaaS pros don’t spend too much time on improving AI-generated text. I’ve seen many SaaS companies publishing AI articles without tweaking readability. Even [hidden] publishes articles in their raw format.
  • Increased text complexity results in decreased time spent on reading. If SaaS brands rely on AI-generated text without improving its readability, they will lose their audience. People don’t have the time or energy to waste on complex articles.
  • We see SaaS companies using AI tools to generate marketing and sales collateral. But what will happen if the same level of complexity infiltrates user onboarding materials?
  • At the moment, generative AI tools simplify the process of creating content. But they complicate the output. And there’s little interest in refining this output. Plus, the task itself (refining) can be as difficult as creating content from scratch.

Solutions

I have a problem with non-conversational AI tools for content creation, such as [hidden].

Let me clarify this term.

There’s a difference between conversational and non-conversational tools. For example, ChatGPT is conversational. It can engage in real-time conversations with users. It also adapts responses based on input and provides follow-up adjustments.

Non-conversational AI tools generate text based on a given prompt. Yet, there’s no back-and-forth interaction. For example, I can’t ask [hidden] to simplify my text.

Why does this matter?

For example, I used a conversational AI tool (not ChatGPT) to generate a micro-text.

When I ran this text through the Hemingway App, it ranked 14 on the readability scale.

The readability of the article generated with conversational AI (1st output)

However, I was able to go back and ask the AI to simplify the text to a 5 on the readability scale.

And here’s the output.

The readability of article generated with conversational AI (2nd output)

It followed my instructions precisely. The text is much easier to read.

So one solution is to use conversational AI tools to improve text readability.

But this option has big limitations.

In this specific case, [hidden] offers features such as a content calendar, SEO analysis, real-time collaboration, and more. I don’t have these options with the conversational AI I’ve used.

I know tools are still in development. We are in the early stage of generative AI.

However, it’s important to discuss what our needs are. And from what I see, readability or text simplicity is not a top priority (when it should be).

For example, it would be a huge win if the non-conversational AI tools would add the possibility to:

  • Engage with the tool via real-time conversations
  • Choose the readability level before generating the text
  • Automatically tweak the readability level after generating the text

There are also some new AI tools that help with readability. One solution would be to generate AI text with a non-conversational tool. Then you can run this text through a second tool to tweak readability.

That’s obviously more expensive, but it can work.

In any case, we shouldn’t focus on ‘Mom, look, I can generate 30 pieces of content in 10 seconds…’

Instead, we should work on making this content super easy to read and understand.

Otherwise, we’re missing the point.

We create content and copy for our leads and users. We want to connect with them.

But we’ll keep them at an arm’s length if we don’t focus on readability and making things easy for them.

free services

Get help with your SaaS messaging

The catch? Service for a testimonial.

Free pdf messaging roast

know your messaging issues
get actionable solutions
it’s called a roast, but it’s friendly

Free 30-min consulting

we’ll work on 1 messaging issue
we’ll identify quick solutions
we’ll outline your next steps