Summarization using LLM and measuring the performance with ROUGE— Part 1 | by Kunal Gandhi | Oct, 2023

LLM (Huge Language Fashions) are the first focus of proper now’s AI world notably throughout the area of Generative AI. On this text, we’ll try few LLMs from hugging-face by the use of in-built Pipeline and may measure the effectivity of each model by the use of ROUGE.

Summarization — So there’s 2 method to hold out summarization.

1. Abstractive Summarization — Proper right here we try to create a summary that represents the purpose and seize the essence of the doc. That’s arduous to achieve as we might should create new phrases and recreate sentences that aren’t present throughout the doc which can create grammatical and semantic factors
2. Extractive Summarization — Extractive summarization selects and extracts full sentences from the provision textual content material to create the summary. It doesn’t generate new sentences nevertheless reasonably chooses sentences that are in all probability probably the most informative or guide of the content material materials.

Hugging Face transformers are trying to hold out abstractive summarization. Let’s come to the aim.

First it’s important to import following libraries,

# to load the dataset
from datasets import load_dataset
# to create summarization pipeline
from transformers import pipeline
# to calculate rouge score
from rouge_score import rouge_scorer
import pandas as pd

Please arrange libraries by the use of pip for those who don’t already have them.

Now let’s load dataset that we going to utilize to measure the effectivity of LLMs.

xsum_dataset = load_dataset("xsum", mannequin="1.2.0") 
xsum_sample = xsum_dataset["train"].select(differ(5))
present(xsum_sample.to_pandas())

As you’ll have the ability to see dataset have 3 columns.

• doc: Enter info article.
• summary: One sentence summary of the article.
• id: BBC ID of the article.

Yow will uncover further about this dataset here.

Let’s create summarization pipeline and create summary passing doc.

summarizer_t5 = pipeline(
course of="summarization",
model="t5-small",
) 
outcomes = summarizer(xsum_sample["document"],min_length=20,max_length=40,truncation=True)
# convert to pandas df and print
opt_result = pd.DataFrame.from_dict(outcomes).rename({"summary_text": "generated_summary"}, axis=1).be a part of(pd.DataFrame.from_dict(xsum_sample))[
["generated_summary", "summary", "document"]
]   
present(opt_result.head())

Pipeline takes primarily three arguments. Model, course of and tokenizer. Proper right here we’re using the default tokenizer.

We’re passing the minimal dimension as 20 and the utmost dimension for summary is 40.

Now measure the effectivity by calculating ROUGE Score.

ROUGE stands for “Recall-Oriented Understudy for Gisting Evaluation.” It’s a metric designed to measure the usual of summaries by evaluating them to human reference summaries. ROUGE is a gaggle of metrics, with in all probability probably the most usually used one being ROUGE-N, which measures the overlap of N-grams (contiguous sequences of N phrases) between the system-generated summary and the reference summary.

Let’s calculate ROUGE — 1 for the subsequent occasion,

Reference Summary — Local weather is scorching proper right here

Generated Summarty — Local weather could also be very scorching proper right here

Calculate F1 Score using precision and Recall. F1 Score might be 0.88 .

We’ll calculate ROUGE — 2,ROUGE — 3 … ROUGE — N using bi-gram,tri-gram and N-grams.

ROUGE-L: Measures the longest widespread subsequence between the system and reference summaries. This metric is far much less delicate to phrase order and should seize semantic similarity.

def calculate_rouge(info):
scorer = rouge_scorer.RougeScorer(['rouge1', 'rouge2', 'rougeL'], use_stemmer=True)
info["r1_fscore"] = info.apply(lambda row : scorer.score(row["summary"],row["generated_summary"])['rouge1'][2], axis=1)
info["r2_fscore"] = info.apply(lambda row : scorer.score(row["summary"],row["generated_summary"])['rouge2'][2], axis=1)
info["rl_fscore"] = info.apply(lambda row : scorer.score(row["summary"],row["generated_summary"])['rougeL'][2], axis=1)
return info
score_ret=calculate_rouge(opt_result)
print("ROUGE - 1 : ",score_ret["r1_fscore"].suggest())
print("ROUGE - 2 : ",score_ret["r2_fscore"].suggest())
print("ROUGE - L : ",score_ret["rl_fscore"].suggest())

I’ve tried 2 pre-trained model for the summarization.

1. T5 — Small
2. fb/bart-large-cnn

These are pre-trained fashions. We’ll further fine-tune these fashions to work greater. Yow will uncover a list of fashions on the market for summarization duties on HuggingFace here.

Whereas ROUGE is a useful instrument, it has its limitations. For example, it doesn’t take into consideration the fluency and coherence of the summary. It focuses on phrase overlap, which suggests a summary can receive a extreme ROUGE score even when it’s not very readable.

Please uncover the code throughout the git repo .