In my first semester teaching one of my department’s graduate methods courses in digital history, I realized that there was not a lot good material for teaching computer programming and data analysis in R for historians. So I started writing up a series of tutorials for my students, which they said were helpful. It seemed like those materials could be the nucleus of a textbook, so I started writing one with the title Digital History Methods in R.
It was too soon to start writing, though. Besides needing to spend my time on more pressing projects, I didn’t really have a clear conception of how to teach the material. And in the past few years, the landscape for teaching computational history has been transformed. There are many more books available, some specifically aimed at humanists, such as Graham, Milligan, and Weingart’s Exploring Big Historical Data and Arnold and Tilton’s Humanities Data in R, and others aimed at teaching a modern version of R, such as Hadley Wickham’s Advanced R and R for Data Science. The “tidyverse” of R packages has made a consistent approach to data analysis possible, and the set of packages for text analysis in R is now much better. R markdown and bookdown have made writing a technical book about R much easier, and Shiny has made it much easier to demonstrate concepts interactively.
After teaching these courses a few times, I have a clearer conception of what the textbook needs to accomplish and how I want it to look.
Continue reading “Pick the title for my digital history textbook”
Following Alan Jacobs and Boone Gorges, here is a brief report on my year in technology.
Compared to the list of technologies that I used in 2014, not much has changed. My preferred tools for writing remain Markdown, Pandoc, LaTeX, and Vim, with Zotero for citations. I’ve never had a system for taking notes that entirely pleased me. After a brief flirtation with an open wiki notebook, following the lead of Caleb McDaniel and Jason Heppler, I gave it up this year. The idea still seems sounds, but in practice I’m unwilling to have all my notes public, the separation of public and private notes was onerous to maintain, and a fair bit of my note-taking is in the form of “lab notebooks” for which a different system is necessary. At the moment all of my notes are in plain text in a Dropbox directory, which I can access on iOS with 1Writer or on computers in Vim. Perhaps someday I will return to a private wiki.
For computational history, R maintained pride of place. It is astonishing how much better and easier to use the language has become since I started to teach it to grad students a couple of years ago. For one project I was able to do everything from machine learning on Mason’s computing cluster to interactive visualizations in Shiny. I foresee a lot more D3.js in my future, but mostly to make interactive R packages.
Continue reading “My year in tech 2016”
For several semesters now, I’ve used Slack for my graduate courses in digital history. (Here’s a good introduction to Slack for teaching.) Judging by what the students tell me and what they write on the student evaluations, nothing I’ve done as a teacher has been as universally popular. Students appreciate a way to get detailed help from me quickly. Students seem to be less reluctant to ask questions when they can see that other people are asking questions, and when the answer is public for everyone’s benefit. And I can tell from the reports Slack sends that they use Slack to talk to one another.
I like Slack because it encourages an ongoing connection between the students and me during the week. Rather than only hearing from students once or twice a week when the class meets, I have a more or less continuous connection to them, at least to the extent that they choose. Slack has cut student e-mails in those courses to virtually nil. It’s very helpful for me to have a single place to go to communicate with students, and where I can easily turn a query directed only at me into a general lesson for the entire group. For technical questions, it is far easier to give explanations with code blocks and screenshots. It’s also really nice to see students help one another figure out difficult assignments.
Continue reading “Slack in my classroom”
Imagine a photograph of a pile of pages, perhaps with a few glossy image reproductions and a cover letter, and a USPS envelope. Perhaps a few years ago I might have been able to take such a picture for this post. But without having to pay for postage, I’ve delivered the final revisions of my book manuscript to my editor before the end of December as agreed. The Chance of Salvation: A History of Conversion in America will be published by Harvard University Press in fall 2017. While there is still copyediting, proofreading, indexing and so on to be done, the book definitely feels like it is in the hands of the press and not my hands now.
Continue reading “Print historian”
Kellen Funk and I have just published an article titled “A Servile Copy: Text Reuse and Medium Data in American Civil Procedure” (PDF). The article is a brief invited contribution to a forum in Rechtsgeschichte [Legal History] on legal history and digital history. Kellen and I give an overview of our project to discover how nineteenth-century codes of civil procedure in the United States borrowed from one another. (We will have more soon about this project in a longer research article.)
If you are interested in digital legal history, you might also look at some of the articles which have been posted in advance of the next issue of Law and History Review, which will be focused on digital legal history.
This semester I am teaching an independent study for graduate students on “Text Analysis for Historians.” You can see the syllabus here. It’s an unashamedly disciplinary course. While of course the readings are heavily dependent on the work that is being done in digital humanities or digital literary studies, the organizing principle is whether a method is likely to be useful for historical questions. And the syllabus is organized around a corpus of Anglo-American legal treatises, with readings to frame our work in the context of U.S. legal history.
They are mentioned on the syllabus, but this class draws from syllabi from Ted Underwood, Andrew Goldstone, and Ben Schmidt, and Kellen Funk offered suggestions for the readings.
This post originally appeared at the rOpenSci blog.
The R package ecosystem for natural language processing has been flourishing in recent days. R packages for text analysis have usually been based on the classes provided by the NLP or tm packages. Many of them depend on Java. But recently there have been a number of new packages for text analysis in R, most notably text2vec, quanteda, and tidytext. These packages are built on top of Rcpp instead of rJava, which makes them much more reliable and portable. And instead of the classes based on NLP, which I have never thought to be particularly idiomatic for R, they use standard R data structures. The text2vec and quanteda packages both rely on the sparse matrices provided by the rock solid Matrix package. The tidytext package is idiosyncratic (in the best possible way!) for doing all of its work in data frames rather than matrices, but a data frame is about as standard as you can get. For a long time when I would recommend R to people, I had to add the caveat that they should use Python if they were primarily interested in text analysis. But now I no longer feel the need to hedge.
Still there is a lot of duplicated effort between these packages on the one hand and a lot of incompatibilities between the packages on the other. The R ecosystem for text analysis is not exactly coherent or consistent at the moment.
My small contribution to the new text analysis ecosystem is the tokenizers package, which was recently accepted into rOpenSci after a careful peer review by Kevin Ushey. A new version of the package is on CRAN. (Also check out Jeroen Ooms’s hunspell package, which is a part of rOpensci.)
Continue reading “New package tokenizers joins rOpenSci”
It’s the start of August, and I don’t want to presume on the good graces of this blog’s readers. So in the spirit of late summer, I’m finally getting around to briefly describing of one of my summer projects in the hope that you find it fun, leaving a fuller accounting of the why and wherefore of the project for another time.
America’s Public Bible is a website which looks for all of the biblical quotations in Chronicling America. Chronicling America is a collection of digitized newspapers from the Library of Congress as part of the NEH’s National Digital Newspaper Program. ChronAm currently has some eleven million newspaper pages, spanning the years 1836 to 1922. Using the text that ChronAm provides, I have looked for which Bible verses (just from the KJV for now) are quoted or alluded to on every page. If you want an explanation of why I think this is an interesting scholarly question, there is an introductory essay at the site.
Continue reading “Introducing America’s Public Bible (Beta)”
I love Makefiles more than I ought to. If you haven’t come across GNU Make before, then Mike Bostock’s “Why Use Make” is a good introduction, as is the section on Make at Kieran Healy’s “Plain Person’s Guide to Plain Text Social Science.” I like Make for several reasons. It lets you specify how your final products (like a website or a PDF document) are related to inputs, and that discipline is invaluable for producing reproducible research and for structuring your project sensibly. For lots of tasks it provides free parallelization and rebuilds only what is absolutely necessary. Since my projects fit into several different genres, once I have created a Makefile for the genre, it is trivial to adapt it to different projects. Whether it is an article or a book manuscript, a data analysis project, a website, my CV, or some kind of file conversion process, all that I need to remember how to do is type
make to build the project,
make deploy to put it on the web, and
make clean to start over.
Continue reading “Makefiles for Writing, Data Analysis, OCR, and Converting Shapefiles”
This semester I am teaching a graduate course on Data and Visualization in Digital History. The aim of this course is to teach students how to do the kind of data analysis and visualization that they are likely to do for a dissertation chapter or a journal article. In my way of working, that means the first part of the semester is an introduction to scripting in R, focusing on the grammar of graphics with ggplot2 and the grammar of data manipulation with dplyr and tidyr. Then the second part of the course is aimed at introducing specific kinds of analysis in the context of historical work. My aim is that this course will be the first in a two course sequence, where the second course (colloquially known as Clio 3) will have more programming in R (as opposed to scripting), will have more *nix-craft, will tackle a more advanced historical problem, will possibly cover more machine learning, and will end up creating interactive analyses in Shiny.
There are a few things about the Data and Visualization course that I think are worth mentioning.
First, I’ve been creating worksheets for historical data analysis each week. These worksheets tend to demonstrate some technique, then ask students to build up an analysis step by step. The questions within each worksheet range in difficulty from the rote and mechanical to the very difficult. While for now these worksheets are aimed at this class in particular, I intend over time to write worksheets like these for any topic in R I end up teaching. I’m rather pleased with these worksheets as a method of teaching data analysis by example. If I’m judging my students’ initial reactions correctly, they are also finding them helpful, if rather difficult at times.
Continue reading “A Course in Computational Methods and Nineteenth-Century Religious Data”