From: dbucklin@sdf.org
Date: 2018-06-29
Subject: Visualizing the History of Programming Languages
Recently, I came across the Wikipedia article, Timeline of Program-
ming Languages[1]. It has nicely-formatted tables for each decade
since the 1940s. Each table has the same format: one language per
row and, for each language, the year it came into being, its name,
creator, and a list of the languages that influenced it. When data
establishes a relationship between elements, as this page does with
the list of influences for each language, we have the makings of a
graph.
I've written before[2] about using PlantUML to create graphics from
textual information. In this case I'm using a similar tool, Plant-
Text[3], to create this graph. Both PlantUML and PlantText are
web-based front-ends for GraphViz. GraphViz uses a language,
called DOT, to define the content and structure of information to
be visualized. PlantText provides a number of examples that illus-
trate how different kinds of visualizations can be created. I'm
basing this graph on the World Dynamics template. I also got some
additional ideas from "Drawing graphs with dot"[4] by Gansner,
Koutsofios, and North.
The first thing I need to do is put the tables into a structured
format that I can parse with awk. I'm using git-bash for this, and
git-bash lacks my go-to tools for something like this, lynx or w3m.
I do have Pandoc installed, and I can use that to transform HTML to
plain text. I recently created a function called myw3m in my
.bash_profile.
myw3m ()
{
curl -s "$1" | perl -pe 'use open qw(:std :utf8); s/[^[:ascii:]]//g;' | pandoc -fhtml -tplain -
}
This function pulls down the page using curl, passes it to perl to
strip out non-ascii characters, and then to Pandoc to transform to
plain text. I can pull the Wikipedia page to a local file, con-
verting it to plain text along the way:
myw3m https://en.wikipedia.org/wiki/Timeline_of_programming_languages > timeline.txt
This works fine in git-bash, but on my Debian VPS, I have an older
version of Pandoc that wraps over-zealously. I am able to get sim-
ilar results with w3m, using a large width value to prevent wrap-
ping. This also requires a little manual futzing with the data to
widen inter-column space.
w3m -cols 400 -dump https://en.wikipedia.org/wiki/Timeline_of_programming_languages > timeline.txt
Now I have the whole page as plain text including the table rows
which look like this:
1970 Pascal Niklaus Wirth, Kathleen Jensen ALGOL 60, ALGOL W
I want to pass this into awk, so I need to put this data into tab-
delimited fields. First, I need to strip out any lines that are
not data I need to process. This is pretty easy because each line
from the table starts with two spaces and a four-digit number. I
can filter out everything else using sed:
sed -ne '/^ [[:digit:]]4/p'
It looks like Pandoc has separated each column with three or more
spaces. We are lucky that no field data contains three consecutive
spaces. This means that we can use sed to replace any instance of
three or more spaces with a tab. We should also strip off leading
spaces while we're at it. We are trying to make this easy to work
with in awk.
sed -e 's/^ *//;s/ *//g'
Now a row in the table looks like this:
1970^IPascal^INiklaus Wirth, Kathleen Jensen^IALGOL 60, ALGOL W
Pretty slick, no? There are three sections of graph that I need to
build out. My first subgraph is the collection of years in the da-
ta with the relationship between each year made explicit. This ef-
fectively creates an X-axis that will organize the rest of the
graph. By building a list of all the years, I can create a sub-
graph that looks like this:
{
"1988" -> "1989";
"1989" -> "1990";
"1990" -> "1991";
}
The next thing I need to do is tell GraphViz to associate each lan-
guage with its corresponding year in the subgraph. This will en-
courage GraphViz to visually rank each language along with its
year. I do this with a series of subgraphs that look like this:
{rank=same; "1988"; "rpg/400"; "tcl"; "stos basic"; "actor"; "object rexx"; "spark"; "a+"; "hamilton c shell"; }
{rank=same; "1989"; "turbo pascal oop"; "modula-3"; "powerbasic"; "lpc"; "bash"; "magik"; "python"; }
{rank=same; "1990"; "amos basic"; "object oberon"; "j"; "haskell"; "z shell"; }
You'll notice that everything is in lower case. I found the capi-
talization to be inconsistent within the Wikipedia article, so I
have to fix that to avoid duplicates.
Finally, I just need to define all the other nodes and edges. The
edges are directional, so I define them going from the influencing
language to the influenced language. For a line like this:
1987 Perl Larry Wall C, sed, awk, sh
I create DOT statements like this:
"c" -> "perl";
"sed" -> "perl";
"awk" -> "perl";
"sh" -> "perl";
The finished product[5]. I had to cut it off after 2005 because
GraphViz would crash on me. You can also see the awk script[6]
used to create the data that I fed to PlantText.
I had a lot of fun with this challenge. Projects like this are a
great way to learn. I learned more about DOT and awk while doing
this. It's also fulfilling to see the potential in something and
then make it happen.
Happy hacking.
References:
1. https://en.wikipedia.org/wiki/Timeline_of_programming_languages
2. http://davebucklin.com/work/2017/09/11/diagrams-from-text-with-plantuml.html
3. https://www.planttext.com
4. http://www.graphviz.org/pdf/dotguide.pdf
5. http://davebucklin.com/assets/img/lang6.png
6. http://davebucklin.com/assets/toplg.txt