~/projects/byu-speeches/reception
What outperforms the speaker
Some speeches are watched for decades. BYU Speeches kept a public all-time top 30 by views until 2024, and the Internet Archive kept copies of it. This study treats that list as the label and asks what the text can say about it once the obvious is removed: an apostle at a devotional in a recent year draws an audience whatever is said. The speeches that interest us are the ones whose words earn more than their billing.
What stands out
- The words alone reach an AUC of 0.766; the metadata alone 0.64. With 39 positives the intervals are wide (0.693–0.828 for text), but the text carries real signal about which speeches people keep returning to, and it is not the signal of who gave them.
- Calling is the strongest single fact. 4.0% of apostles' speeches made the list against 0.2% of speeches by people with no calling listed; the text model's mean score is almost identical across those groups.
- The list rewards doctrine stated plainly. let’s, covenant path, grace, doctrine of, marriage, god has raise a speech's chance; brigham, brigham young, byu, university, service lower it. The listed speeches have fewer story paragraphs (19% against 23%) and more questions.
- Brad Wilcox's “His Grace Is Sufficient” (2011) was first in every copy of the list. The text model gave it 25% with its speaker held out, among the highest scores in the corpus.
- The sleepers are the point of the study. Richard N. Williams's “Faith, Reason, Knowledge, and Truth” (2000) scores 30% on its words and 0% on its billing, and never made the list.
Loading the study…
Model card
- Label
- BYU Speeches published a ranked "Popular" page (all-time views) until 2024. 7 copies archived by the Wayback Machine from 2022-06 to 2024-04 were read; 42 distinct speeches appeared, 39 of which are in this corpus. A speech is positive if it ever appeared. The mean rank across copies is reported but not modelled.
- Features
- Text: masked TF-IDF over words and two-word phrases (30,000 features; names, years and citations removed). Metadata: the calling held when speaking in six groups, the event type, year and year squared, log of word count, and ten style traits (sentence length, word length, type-token ratio, long words, questions, first and second person, scripture rate, paragraph length).
- Model and validation
- Three logistic regressions with balanced class weights (text, metadata, both), five folds grouped by speaker so a popular speaker's other speeches never inform a prediction about them. AUC with a 1,000-sample bootstrap interval and average precision are out of fold. The sleepers are ranked by the text score minus the metadata score, out of fold.
- Known failure modes
- Thirty-nine positives is a small label and the intervals say so. All-time views on a website favour the years the website existed, and the list was captured only in 2022–2024; a speech famous in 1995 and forgotten by 2022 is a negative. The page did not say how views were counted. The archive capture counts, where shown, follow crawlers' habits as much as readers'.
Methods and data for the corpus as a whole.