Dialogue summarization

How much better than a simple baseline is a fine-tuned transformer at summarizing everyday dialogues?

The problem

Summarizing a dialogue is harder than summarizing an article. Several people speak, they refer back to earlier topics, and the summary has to say who did what. We used the DialogSum corpus: 12,460 dialogues with a human summary for training, 500 for validation and 500 for testing, where each test dialogue has three human summaries.

The question for our team of three: how far does a simple baseline get, and how much does a fine-tuned transformer add? My part was BART.

What I did

Fine-tuned facebook/bart-base on all 12,460 training dialogues for 3 epochs (learning rate 3e-5, batch size 2, weight decay 0.01, mixed precision).
Generated one summary per test dialogue with beam search (4 beams, at most 150 tokens), with dropout switched off.
Scored the summaries with ROUGE-1, ROUGE-2 and ROUGE-L against all three human summaries.
Re-ran the TF-IDF baseline from our report on the same test set, so both models are compared under exactly the same setup.
Compared the two models dialogue by dialogue, and checked how the scores change with dialogue length and topic.

Results

Model	ROUGE-1	ROUGE-2	ROUGE-L
TF-IDF baseline (picks the top sentences from the dialogue)	18.6	2.3	16.3
FLAN-T5, fine-tuned (teammate’s result, from the team report)	48.2	22.6	40.5
BART, fine-tuned (my part)	51.5	27.1	43.7

The team report had 50.5 ROUGE-1 for BART. The difference comes from generating with dropout disabled (model.eval()) in the new run; training itself matched the old run (final training loss 1.1523 vs 1.1524).

BART beats the baseline almost everywhere. It has the higher ROUGE-1 on 491 of 500 dialogues (the baseline wins 7, 2 ties), on average by 33 points.
Longer dialogues are harder. BART’s ROUGE-1 drops from 0.58 on the shortest fifth of dialogues (40 to 76 words) to 0.46 on the longest fifth (188 words and more). The baseline drops faster (Spearman correlation with length: BART -0.26, TF-IDF -0.40).
Topic matters less than length. BART stays between 0.47 and 0.55 ROUGE-1 in every topic group. Health (0.47) and work (0.48) score lowest, technology (0.55) and shopping (0.54) highest.

Example

Setting up a meeting

A: Let’s meet before next week’s meeting, Jane. I want to kick around some ideas with you.
B: No problem. I’ve got a really light week. Anytime’s fine by me.
A: Well, let’s plan to get together next Monday afternoon, around three?
B: All right, I’ll put you in my book.

Human summary (1 of 3): A will meet Jane next Monday afternoon around three to kick around some ideas.
BART (ROUGE-1 0.89): A and Jane will meet next Monday afternoon to kick around some ideas.
TF-IDF (ROUGE-1 0.00): B: No problem. I’ve got a really light week. Anytime’s fine by me.

BART writes the summary a person would write, while the baseline copies the one sentence that says nothing about the plan.

Everything is broken in the office

#Person1#: The air conditioning is not working! We need to call a handyman before we start to fry in here!
#Person2#: Dan is on top of that. I think they are also getting the handyman to fix the bathroom toilet that keeps clogging up.
#Person1#: That would be convenient. They might as well ask him to fix the electrical wiring. The circuit breakers keep going out all the time. It’s really annoying!
#Person2#: Yeah you are right. This office is falling apart! Frank told me the other day that the gutters outside were clogged and that’s why the parking lot was flooded.
#Person1#: I know! I was in ankle deep water trying to get to my car that day! The handyman definitely has his work cut out for him.

Human summary (1 of 3): #Person1# and #Person2# think they need a handyman to fix the air conditioning, toilet, electrical wiring and gutters.
BART (ROUGE-1 0.76): #Person1# and #Person2# are calling a handyman to fix the air conditioning, the bathroom toilet, and the electrical wiring.
TF-IDF (ROUGE-1 0.18): It’s really annoying. #Person2#: Yeah you are right. #Person1#: I know.

BART turns five turns of complaining into one clear list, but it leaves out the last item, the gutters.

A new suit

#Person1#: What do you think of my new suit?
#Person2#: Not bad. It reminds me of the one I saw at the new department store last week. Did you get it there?
#Person1#: No, I got it in that big shopping center. It cost me only $ 150.
#Person2#: Well, I don’t think it’s a good bargain.

Human summary (1 of 3): #Person1# bought a new suit with $ 150 and #Person2# thinks it is too expensive.
BART (ROUGE-1 0.89): #Person2# thinks #Person1#’s new suit is not good.
TF-IDF (ROUGE-1 0.29): #Person2#: Not bad. Did you get it there. It cost me only $ 150.

BART scores 0.89 ROUGE-1 but misses the main point, the price, which shows that ROUGE measures word overlap, not meaning.

What I learned

Fine-tuning is what made the model useful. The same BART without fine-tuning scored 21.3 ROUGE-1 in our report; after fine-tuning it wrote the better summary for almost every dialogue.
A good score is not the same as a good summary. ROUGE counts shared words, so a summary can score 0.89 and still miss what the dialogue was about. I now read examples next to the numbers.
Small details change reported numbers. Re-running the evaluation showed that switching dropout off at generation time moves BART by about a point, and that the report’s baseline score came from a different version of the baseline than the one we described.

Key results