~/projects/byu-speeches/authorship
Whose talk is this
Given a speech with the name removed, can a model say who gave it? Among the {s.speakers} speakers with {s.minTalks} or more speechs it can, most of the time, and where it fails it fails in pairs: the speakers it confuses are the speakers who sound alike.
What stands out
- The right speaker 79% of the time, in the top five 94%. Chance is 2.3%. Words and phrases plus a stylometric vector, five-fold cross-validation.
- Style alone gets 78%. With no content words at all, function words, sentence shape and punctuation identify the speaker almost as well as the full vocabulary does.
- Bruce R. McConkie is the most recognisable (100% of 13 speechs); Patricia T. Holland the least (0% of 8).
- Marilyn S. Bateman and Merrill J. Bateman are mistaken for each other most, 5 times, 14% of their speechs between them; then Jeffrey R. Holland and Patricia T. Holland.
- The calling is harder than the speaker. With every speaker held out, the text names the calling 50% of the time; Quorum of the Twelve is the most recognisable calling (60%).
Loading the study…
Model card
- Data
- 753 speechs of 300 words or more by the 44 speakers with 8 or more.
- Features and model
- Masked TF-IDF over words and two-word phrases (names, years, magazines and citations removed; 40,000 features) concatenated with the standardised stylometric vector of the style study, scaled down by 0.3; a multinomial logistic regression. The style-only model uses the stylometric vector alone.
- Validation
- Five stratified folds by talk for speaker attribution (the question is whether a new talk by a known speaker can be attributed). For calling, five folds grouped by speaker so the model cannot recognise the person.
- Known failure modes
- Speakers of one era share vocabulary and transcription conventions, so part of what the model learns is period, not person; the style-only score benefits from the same. Speakers with few talks are harder, and a long-serving speaker's early and late talks can read as two people.
Methods and data for the corpus as a whole.