The Quantified Dissertation: Using Bash and cron to Track Word Counts

Here is a hack for those of you writing a dissertation (or book, or other long form project) in plain text. Since I am writing my dissertation in Markdown and Pandoc, it’s easy to find out the word count for a chapter from the command line:

<span class="gp">$ </span>wc -w ch4.dissertation.Mullen.md

I wanted to track my progress by regularly getting a word count for every chapter. To do that, I wrote a Bash script which runs `wc` and stores the data to a CSV file. Here is the script:

<span class="c">#!/bin/bash          </span>
<span class="nb">source</span> /Users/lmullen/.bash_profile
<span class=“nv”>EPOCH</span><span class=“o”>=</span><span class=“k”>$(</span>date +<span class=“s2”>"%s"</span><span class=“k”>)</span>
<span class=“nv”>DATE</span><span class=“o”>=</span><span class=“k”>$(</span>date<span class=“k”>)</span>
<span class=“nv”>WCCH1</span><span class=“o”>=</span><span class=“k”>$(</span>wc -w /Users/lmullen/acad/dissertation/ch1.dissertation.Mullen.md | sed <span class=“s1”>'s//Users/lmullen/acad/dissertation/ch1.dissertation.Mullen.md//'</span> | sed <span class=“s1”>'s/ //g'</span><span class=“k”>)</span>
<span class=“nv”>WCCH2</span><span class=“o”>=</span><span class=“k”>$(</span>wc -w /Users/lmullen/acad/dissertation/ch2.dissertation.Mullen.md | sed <span class=“s1”>'s//Users/lmullen/acad/dissertation/ch2.dissertation.Mullen.md//'</span> | sed <span class=“s1”>'s/ //g'</span><span class=“k”>)</span>
<span class=“nv”>WCCH3</span><span class=“o”>=</span><span class=“k”>$(</span>wc -w /Users/lmullen/acad/dissertation/ch3.dissertation.Mullen.md | sed <span class=“s1”>'s//Users/lmullen/acad/dissertation/ch3.dissertation.Mullen.md//'</span> | sed <span class=“s1”>'s/ //g'</span><span class=“k”>)</span>
<span class=“nv”>WCCH4</span><span class=“o”>=</span><span class=“k”>$(</span>wc -w /Users/lmullen/acad/dissertation/ch4.dissertation.Mullen.md | sed <span class=“s1”>'s//Users/lmullen/acad/dissertation/ch4.dissertation.Mullen.md//'</span> | sed <span class=“s1”>'s/ //g'</span><span class=“k”>)</span>
<span class=“nv”>WCCH5</span><span class=“o”>=</span><span class=“k”>$(</span>wc -w /Users/lmullen/acad/dissertation/ch5.dissertation.Mullen.md | sed <span class=“s1”>'s//Users/lmullen/acad/dissertation/ch5.dissertation.Mullen.md//'</span> | sed <span class=“s1”>'s/ //g'</span><span class=“k”>)</span>
<span class=“nv”>WCCH6</span><span class=“o”>=</span><span class=“k”>$(</span>wc -w /Users/lmullen/acad/dissertation/ch6.dissertation.Mullen.md | sed <span class=“s1”>'s//Users/lmullen/acad/dissertation/ch6.dissertation.Mullen.md//'</span> | sed <span class=“s1”>'s/ //g'</span><span class=“k”>)</span>
<span class=“nv”>WCCH7</span><span class=“o”>=</span><span class=“k”>$(</span>wc -w /Users/lmullen/acad/dissertation/ch7.dissertation.Mullen.md | sed <span class=“s1”>'s//Users/lmullen/acad/dissertation/ch7.dissertation.Mullen.md//'</span> | sed <span class=“s1”>'s/ //g'</span><span class=“k”>)</span>
<span class=“nv”>WCCH8</span><span class=“o”>=</span><span class=“k”>$(</span>wc -w /Users/lmullen/acad/dissertation/ch8.dissertation.Mullen.md | sed <span class=“s1”>'s//Users/lmullen/acad/dissertation/ch8.dissertation.Mullen.md//'</span> | sed <span class=“s1”>'s/ //g'</span><span class=“k”>)</span>
<span class=“nv”>WCPRE</span><span class=“o”>=</span><span class=“k”>$(</span>wc -w /Users/lmullen/acad/dissertation/preface.dissertation.Mullen.md | sed <span class=“s1”>'s//Users/lmullen/acad/dissertation/preface.dissertation.Mullen.md//'</span> | sed <span class=“s1”>'s/ //g'</span><span class=“k”>)</span>
<span class=“nv”>WCINTRO</span><span class=“o”>=</span><span class=“k”>$(</span>wc -w /Users/lmullen/acad/dissertation/introduction.dissertation.Mullen.md | sed <span class=“s1”>'s//Users/lmullen/acad/dissertation/introduction.dissertation.Mullen.md//'</span> | sed <span class=“s1”>'s/ //g'</span><span class=“k”>)</span>
<span class=“nv”>WCCONCL</span><span class=“o”>=</span><span class=“k”>$(</span>wc -w /Users/lmullen/acad/dissertation/conclusion.dissertation.Mullen.md | sed <span class=“s1”>'s//Users/lmullen/acad/dissertation/conclusion.dissertation.Mullen.md//'</span> | sed <span class=“s1”>'s/ //g'</span><span class=“k”>)</span>
<span class=“nb”>echo</span> <span class=“nv”>$EPOCH</span>,<span class=“nv”>$DATE</span>,<span class=“nv”>$WCCH1</span>,<span class=“nv”>$WCCH2</span>,<span class=“nv”>$WCCH3</span>,<span class=“nv”>$WCCH4</span>,<span class=“nv”>$WCCH5</span>,<span class=“nv”>$WCCH6</span>,<span class=“nv”>$WCCH7</span>,<span class=“nv”>$WCCH8</span>,<span class=“nv”>$WCPRE</span>,<span class=“nv”>$WCINTRO</span>,<span class=“nv”>$WCCONCL</span> >> /Users/lmullen/acad/dissertation/wordcount-data.csv

First, the script finds out the date in two formats (lines 4--5), the Unix "epoch" time (or number of seconds since midnight on January 1, 1970) and a human readable time. I thought I should store both since I didn't know which would be most useful later on. Second, the script counts the number of words in each chapter and in the front matter and back matter. It also strips away all of the output from that command except the word count itself (lines 6--16). Someone with better scripting skills than me could probably compress those eleven lines into one or two. Finally, the script appends a line with the data to a CSV file (line 17).

When the script is executed, it returns data like this (from the very first line in my data file):

EPOCH,DATETIME,CH1WC,CH2WC,CH3WC,CH4WC,CH5WC,CH6WC,CH7WC,CH8WC,PREWC,INTROWC,CONCLWC
1332736387,Mon Mar 26 00:33:07 EDT 2012,0,0,0,12412,0,0,0,0,0,0,0

As you can see, by March 26 I had written 12,412 words for chapter 4, and not much else. Apparently I also wrote this script after midnight.

I run the script automatically each hour by adding this line to my crontab:

0   <span class="k">*</span>   <span class="k">*</span>   <span class="k">*</span>   <span class="k">*</span>   /Users/lmullen/acad/dissertation/wordcount.sh

I haven't done much with the data yet, but the point is to collect it while I am writing. It's easy enough to open the CSV file in Excel and generate graphs of my progress. Eventually I may also analyze the data in [R](http://www.r-project.org/).

Gathering the data is one problem. Getting the word counts to increase at an acceptable pace is another problem altogether.

More blog posts