Effective Summarization with ChatGPT

Summarization agents are ubiquitous on the Internet. For example, various websites employ these agents to provide succinct summaries of articles, allowing users to grasp the news quickly without sifting through the entire content. Social media platforms and search engines follow suit.

From news aggregators to social media sites and e-commerce platforms, summarization agents have become essential in our digital ecosystem. With the rise of LLMs, some of these agents now leverage AI for more effective summarization outcomes.

ChatGPT can serve as a valuable partner when developing applications that utilize summarization agents to accelerate reading tasks and categorize texts. For instance, if we manage an e-commerce site and want to analyze all customer reviews, ChatGPT could assist in summarizing any review into a few sentences, standardizing it to a uniform format, determining the sentiment, and categorizing it accordingly.

While it's true that we could simply input the review into ChatGPT, adhering to a set of best practices — and avoiding certain pitfalls — will help maximize its effectiveness for this specific task.

Let’s bring this example to life!

Example: E-commerce Reviews

Imagine we want to process all reviews for a specific product on our e-commerce platform. We’d focus on reviews like the one for our star product: a first computer for children!

In this case, we would ask ChatGPT to:

• Classify the review as positive or negative.
• Generate a 20-word summary of the review.
• Output the response in a structured format to standardize all reviews.

Implementation Notes

Here’s a basic code structure to prompt ChatGPT from our custom application. A link to a Jupyter Notebook containing all examples referenced in this article is also provided.

The function get_completion() connects to the ChatGPT API with a specified prompt. If the prompt includes additional user text, such as the review itself, it is separated from the rest of the code using triple quotes.

Let’s utilize the `get_completion()` function to prompt ChatGPT!

Here is a prompt that satisfies the requirements outlined above:

The prompting guidelines used in this example, such as employing delimiters to separate input text from the main prompt and requesting structured output, are thoroughly discussed in What I Learned from OpenAI’s Course on Prompt Engineering — Prompting Guidelines.

This is ChatGPT’s response:

As we can see from the output, the review is accurate and well-organized, although it misses some details we might find important as e-commerce owners, such as information regarding product delivery.

Summarize with a Focus on Shipping and Delivery

We can enhance our prompt by asking ChatGPT to include specific focus areas in the summary. In this case, we want details about shipping and delivery.

This time, ChatGPT’s response is as follows:

Now the review is much more comprehensive. Including details on essential aspects of the original review is vital to ensure ChatGPT doesn't overlook valuable information for our use case.

Did you notice that although this second attempt covers delivery information, it omitted the only negative point from the original review?

Let’s fix that!

“Extract” instead of “Summarize”

Through investigating summarization tasks, I discovered that summarization can be challenging for LLMs if the user prompt lacks precision.

When requesting ChatGPT to summarize a specific text, it may skip relevant details — as we recently observed —, or assign equal importance to all topics within the text, merely providing an overview of the main points.

Experts in LLMs prefer the term extract and focus on specific aspects instead of summarize when performing such tasks with these models.

While summarization aims to present a concise overview of the text's main points—often including unrelated topics—information extraction concentrates on obtaining specific details and can provide us exactly what we seek. Let’s try using extraction!

In this case, by employing extraction, we solely retrieve information about our focal topic: Shipping: Arrived a day earlier than expected.

Automatization

This system is effective for a single review. However, when designing a prompt for a specific application, it's crucial to test it with a batch of examples to identify any outliers or discrepancies in the model's behavior.

For processing multiple reviews, here’s an example Python code structure that can assist.

Here are the summaries of our batch of reviews:

Note that while our prompts clearly outlined the word restrictions for summaries, it is evident that this limitation was not adhered to in any of the iterations.

This discrepancy in word counting occurs because LLMs do not possess an accurate understanding of word or character counts. This limitation stems from a key component of their architecture: the tokenizer.

Tokenizer

LLMs like ChatGPT are designed to generate text based on statistical patterns derived from large volumes of language data. Though they excel at producing fluent and coherent text, they lack precise control over word counts.

In the earlier examples, when we provided specific word count instructions, ChatGPT struggled to fulfill those requirements. Instead, it produced text that was either shorter than the specified word count or, in some cases, longer and overly verbose. Moreover, ChatGPT may prioritize coherence and relevance over strict adherence to word counts, leading to content that is high-quality in terms of substance and coherence, yet not accurately aligned with the requested word count.

The tokenizer is a crucial element in ChatGPT's architecture that significantly influences the number of words in the generated output.

Tokenizer Architecture

The tokenizer is the initial step in the text generation process. It breaks down the input text into individual elements — *tokens* — which the language model then processes to generate new text.

When the tokenizer analyzes a piece of text, it does so according to a set of rules aimed at identifying meaningful units in the target language. However, these rules are not infallible, and there are instances where the tokenizer splits or merges tokens in ways that affect the overall word count of the text.

For example, consider the sentence: “I want to eat a peanut butter sandwich.” If the tokenizer is set to split tokens based on spaces and punctuation, it may break this sentence down into tokens with a total word count of 8, equal to the token count.

However, if the tokenizer is configured to treat “peanut butter” as a compound word, it may break the sentence down into tokens, resulting in a total word count of 8, but a token count of 7.

Thus, the tokenizer's configuration can significantly impact the overall word count of the text, affecting the LLM's ability to comply with precise word count instructions. While some tokenizers allow customization of text tokenization, this is not always enough to guarantee strict adherence to word count requirements. For ChatGPT in this instance, we cannot control this aspect of its architecture.

This limitation makes ChatGPT less effective at meeting character or word restrictions; however, one can try focusing on sentence counts instead since the tokenizer does not influence the number of sentences, only their length.

Being aware of this limitation can aid you in crafting the most suitable prompt for your intended application. With this knowledge about how word count functions in ChatGPT, let’s perform a final iteration with our prompt for the e-commerce application!

Wrapping up: E-commerce Reviews

Let’s consolidate our insights from this article into a final prompt! In this instance, we will request the results in HTML format for a more appealing output:

And here’s the final output from ChatGPT:

Summary

In this article, we have examined the best practices for utilizing ChatGPT as a summarization agent for our tailored applications.

We learned that crafting the perfect prompt to meet your application requirements on the first try is quite challenging. A valuable takeaway is to consider prompting as an iterative process in which you refine and adjust your prompts until you achieve the desired results.

By iteratively refining your prompt and applying it to a batch of examples prior to deployment, you can ensure the output remains consistent across various examples and accounts for outlier responses. In our scenario, someone might submit random text instead of a review. We can instruct ChatGPT to produce a standardized output to filter out these atypical responses.

Moreover, when leveraging ChatGPT for specific tasks, it’s prudent to familiarize yourself with the advantages and limitations of using LLMs for your intended purpose. This led us to discover that extraction tasks are often more effective than summarization when we desire a typical human-like summary of an input text. We also learned that providing a focal point for the summary can significantly influence the generated content.

Lastly, while LLMs can be remarkably effective at generating text, they are not well-suited for following precise instructions regarding word count or other specific formatting requirements. To achieve these objectives, it may be necessary to rely on sentence counting or alternative methods, such as manual editing or specialized software.

And that’s all! Thank you for reading!

I hope this article proves useful when developing custom applications with ChatGPT!

You can also subscribe to my Newsletter to stay informed about new content, especially if you're interested in articles about ChatGPT:

<div class="link-block">

<div>

<div>

<h2>Unlocking a New Dimension of ChatGPT: Text-to-Speech Integration</h2>

<div><h3>Enhancing User Experience in ChatGPT Interactions</h3></div>

<div><p>towardsdatascience.com</p></div>

</div>

<div></div>

</div>

</div>

<div class="link-block">

<div>

<div>

<h2>What I Learned from OpenAI’s Course on Prompt Engineering — Prompting Guidelines</h2>

<div><h3>Get to know OpenAI’s Guidelines for Better Prompting</h3></div>

<div><p>medium.com</p></div>

</div>

<div></div>

</div>

</div>

<div class="link-block">

<div>

<div>

<h2>What ChatGPT Knows about You: OpenAI’s Journey Towards Data Privacy</h2>

<div><h3>New ways to manage personal data in ChatGPT</h3></div>

<div><p>towardsdatascience.com</p></div>

</div>

<div></div>

</div>

</div>

<div class="link-block">

<div>

<div>

<h2>Improve ChatGPT Performance with Prompt Engineering</h2>

<div><h3>How to Ask Questions to ChatGPT to Maximize the Chances of a Successful Answer</h3></div>

<div><p>levelup.gitconnected.com</p></div>

</div>

<div></div>

</div>

</div>

<div class="link-block">

<div>

<div>

<h2>Create Your Custom ChatGPT with Transfer Learning</h2>

<div><h3>Enhance ChatGPT Capabilities Fine-tuning Your Own Model</h3></div>

<div><p>medium.com</p></div>

</div>

<div></div>

</div>

</div>

<div class="link-block">

<div>

<div>

<h2>ChatGPT vs AI Detectors — Place your Bets!</h2>

<div><h3>Testing the Most Popular AI Detectors on the Internet</h3></div>

<div><p>pub.towardsai.net</p></div>

</div>

<div></div>

</div>

</div>

Feel free to direct any inquiries to [email protected] :)