As I mentioned in my previous post, I’m drawing inferences from an n of one, so take anything I say with a hefty grain of salt.¹ While I’m structuring this post largely as pieces of advice, keep in mind that these were things that worked for me, and may not generalize.²

Differences from the academic job market

Some important differences between the academic and nonacademic job markets that are useful to consider at the start:

• Timelines are faster than faculty searches, but they are far less consistent. One process took almost three months, while another took less than three weeks.
• Not a single employer asked for letters of recommendation. One contacted references.
• Who you talk to varies greatly. For some positions my first contact was an HR phone screen, for others it was a 30 minute initial interview with the hiring manager.
• Performance tasks, otherwise known as coding assignments (or, more accurately, unpaid work), are common. These are just a fact of life for data science jobs. They varied from straightforward problem sets to research design memos, but not every job I interviewed for required them.
• There will probably be a technical interview. As these were not software engineering jobs, most of the ones I encountered tried to assess whether you know the basics of analyzing data in your language of choice and to get some insight into your problem solving approaches.
• Job talks are much less common, but not unheard of. Only two of the positions I interviewed for required a technical presentation, and unlike in academia, there is absolutely zero stigma against presenting coauthored work.
• Based on some very informal reckoning, automated HR rejection emails seem to be about as common for nonacademic jobs as academic ones.³ When they do come, these emails are much faster than in academia: days or weeks instead of months.
• Get ready for a new world of terminology and titles. In the same way that the assistant $\rightarrow$ associate $\rightarrow$ full professor progression baffles many outside of academia, I felt very lost upon encountering ads for senior, principal, and lead data scientists, and especially so when I applied to one for a data science technical adviser.
• Similarly, get ready to navigate the variety of different jobs that can fall under the umbrella of data scientist. Does a job list SQL, Tableau, and Excel as the most important technical skills? That’s probably more of a data analyst position. TensorFlow, Dask, and C++? That’s likely more of machine learning engineer job. If you’re anything like me, you want to aim for the middle ground between these two.

The nonacademic résumé

Probably the biggest transition when starting to apply for data science jobs was the shift from an academic CV to a nonacademic résumé. A CV lists functionally every major accomplishment you’ve achieved in your time in the field, while a résumé is highly targeted for a specific position. When applying to academic jobs, I wrote a (semi) customized cover letter for every job, and then included the relevant version of my CV (conflict, methods, or teaching). Each of these CVs contained the same information, just in a different order. In contrast, I significantly edited the skills section of most résumés I sent out based on the job listing. The WashU career center has a fantastic handout on differences between the two documents and how to adapt a CV into a résumé that I drew on heavily in this process.

In my opinion, the conventional wisdom that a résumé can only ever be one page is an overcorrection from the never-ending academic CV. The résumé I used to apply for jobs was two pages: the first included work experience, education, and a list of technical skills, while the second was project-oriented, and covered two publications, a couple of blog posts, a Shiny dashboard, teaching materials for the grad stats lab I taught. You definitely want to include links here, not just to the final product, but also the code behind it where relevant (replication materials for publications, git repos for smaller projects). This is an excellent opportunity to showcase work that uses data science skills to show something interesting, but wouldn’t be considered novel enough for publication in an academic journal. Here are some other points that may be helpful when writing a résumé:

• No one is likely to care that you wrote an undergraduate thesis or received a masters in passing (I did both, neither are on my résumé). An important exception to the latter point applies if you will be leaving your program without finishing your PhD; definitely list an in-passing masters in this case. Similarly, if you received a masters in a separate (more technical) program during your PhD, e.g., statistics or data science, be sure to list it as well.
• Social sciences can be a bit out of left field for data science hiring managers, so my résumé did include a “Concentrations: quantitative methodology and international relations” sub-bullet under my PhD in my education section.
• Paid research assistant jobs you had in grad school absolutely count as work experience and should be listed separately from your research and teaching if relevant to the types of jobs you’re applying for. I listed my jobs ensuring the reproducibility of quantitative results for academic journals and supporting users of university high performance computing resources as there’s a very short line between both of those job descriptions and many common data science tasks.
• If a job ad lists a skill and you have that skill, put it on your résumé, even if it’s not one of your strongest skills. Your résumé will almost certainly be fed through an applicant tracking system, and the more matches the system finds, the higher the chance your résumé will end up in front of human eyes.
• I would take this a step further and do this in your cover letter as well. Does a job ad list a “solid understanding of relevant theories in machine learning, statistics, and probability theory” in the requirements? Then you’d better be prepared to talk about how you apply machine learning, statistics, and probability in your work. Does this feel a little like undergraduates trying to avoid plagiarism detection software by changing a few words here and there? Yes, but it’s how hiring happens these days.

Things to do

Below is a list of non-résumé-related things I did to prepare for and during my nonacademic job search that I found helpful:

• As someone who (hubristically) deleted theirs the second year of grad school, it pains me to say that the most important thing you can do here is get yourself a LinkedIn. Get it looking as professional as your academic website. The first thing is to set the headline directly below your name to the type of job you’re looking for. Want to be a research manager? List yourself as one and then talk about all the research assistants you coordinated. You’ll have to do some reframing and shortening, but you can largely transfer over content from your academic website. I added publications and blog posts to the publications and projects section at the bottom of my profile, and I also added them as media items under my postdoc and PhD experiences where appropriate. Add a link or two with high quality preview images to the featured section at the top of your page.
• If you’re applying for jobs now and you’ve taught a quantitative methods course at any point, get ready to talk about this. Every single interview asked me about a time where I had to explain a technical concept or project to a nontechnical audience, and teaching quantitative methods is nothing but that, multiple times a week, for an entire semester. Teaching statistics and programming is hard, so you’ll also have lots of anecdotes ready when the interviewer asks a followup question about a time where you had to change your approach midway through a project. If you haven’t taught quantitative methods yet and you’re not already applying for jobs, do so if at all possible.
• Use your resources. I was fortunate enough to do my postdoc at an institution with an excellent career center that had multiple staff members with experience helping PhD students and postdocs get nonacademic jobs. However, even if your career center is less prepared to help you get a nonacademic job, lots of career centers have publicly available online resources that can be very helpful.
• Use your networks. I talked with many people who work in data science and do not have degrees in computer science or statistics. This included two people from my undergraduate institution (one PhD in psychology, one in physics), multiple political science PhDs I met through Twitter and LinkedIn, and people who did data science masters and nonacademic data science bootcamps. Their experience and advice were invaluable for me in my job search process.
• Research salaries in the field you’re applying to. You can get a broad sense of this through sites like Glassdoor, but ask the people I mentioned above about their starting salaries as well. They likely came from a similar background to you, and this information can be very useful when negotiating salary. You don’t want to undersell yourself when an interviewer asks you your salary range.

Software skills

Social science PhD programs are good at teaching research design, formal modeling, and statistical methodology. They spend far less time on what I’ll call more supporting technical skills. Here are some suggestions in this domain based on my observations so far:

• Don’t try to learn everything there is to know about a cloud computing architecture. There are too many, and every company’s implementation is subtly different. At my job, we use AWS, GCP, and Azure for various tasks, so learning one inside and out won’t give you a huge advantage when applying. If you can generate SSH keys and copy them to a remote host, you’re most of the way there.
• Learn some SQL, but don’t worry about learning how to administer a database. If you can write queries that join multiple tables together and summarize by multiple groups, you’re probably good. If you know the standard libraries for connecting to and querying a SQL database in Python and/or R, that’s great. Again, depending on the individual database solution your job uses, you may have to use a very specific package to access it from your data science language. Mode has a free tutorial with an interactive interface that lets you write and run SQL queries in your browser that I found very helpful.
• Get some experience with shell scripting. I was first exposed to shell scripts because you had to write one to submit jobs on our university cluster in grad school. Data science often involves many moving parts, and being able to use some shell scripting to glue them all together can be incredibly useful. Software Carpentry has a pretty solid introductory lesson.
• I use git daily. While I rarely used git to manage collaboration with coauthors in academia, I used it to version control all of my solo-authored projects, and that provided a solid-enough background for my current level of usage.
• Automation is another important skill in the data science toolbox. Sometimes you’ll have fancy GUI-based tools to set things up to run automatically, but other times it’s faster and simpler to use a cron jobs. I taught myself the basics of cron to keep the stats in my post visualizing police militarization via the transfer of surplus armored vehicles to police departments automatically updated.

The social science PhD comparative advantage

So far this post has mainly been oriented around a list of discrete things you can do to (potentially) improve your odds of securing a data science job as someone with a social science PhD. This last section reflects a perspective I developed throughout my job search process as I participated in more and more interviews, and I hope, will serve as a source of motivation for anyone pursuing a similar career transition.

The vast majority of quantitative social science PhDs (myself very much included) are never going to be machine learning engineers who run neural networks all day long. Instead, we’re going to be working with those engineers, running our own analyses (which might include some deep learning models, but plenty of other types of models as well), and also working with with less-technical stakeholders.

Based on conversations with other data scientists and my experiences as a data scientist thus far, a large part of a data scientist’s job is communicating the value of the work you and your more-technical team members have done to people with less technical training. Even if they have a strong background in statistics or research design more generally, they’re still likely to be less familiar with your specific area of expertise. Communicating effectively in this situation requires distilling large amounts of information, drawing conclusions based on data, and then summarizing what you did, why you did it, and what you learned from doing it. To me, that sounds exactly like what social science PhD programs train their students to do.⁴

Two things have changed since then. One: an already precarious academic job market that never recovered from the global financial crisis has imploded even further. Two: opportunities for people with the set of skills you pick up in a quantitative social science PhD program have exploded. Quit lit is often deeply personal and centered around the path one took to deciding to leave academia; see this piece for links to several prominent examples.¹ This is not that kind of quit lit, because that’s not where my communication skills are strongest. Instead, I’m writing this post to illustrate the contrast between my academic and nonacademic job search processes in the hopes that it may be a useful data point for current grad students, postdocs, adjuncts, and maybe even some early-career faculty.² When reading this post, bear in mind that I am presenting data from an n of one, and my experiences may not generalize outside of quantitative social science, or even very far within it.

Let’s get straight to the numbers. Out of 142 jobs I applied to, I received two job offers. That’s a 98.6% rejection rate.³ Visualizing this (with apologies to Andrew Heiss) looks like so.

Five jobs expressed interest me beyond my initial application, which translates to a 3.5% response rate. The ‘Nothing’ category encompasses both jobs that sent me an automated HR rejection email (often several months after their chosen candidate had accepted the offer) as well as ones that never got back to me. Many searches for faculty positions will conduct Zoom/Skype/Teams interviews with their long short list of candidates before inviting the short list to an on-campus visit, colloquially termed a flyout, but some may skip straight to the on-campus visit. Some postdoc positions conduct virtual interviews, while others simply make an offer to their preferred candidate. I used a rough ranking of potential outcomes as Offer > Flyout ≥ Interview > Rejection in constructing this plot, with each dot representing the final outcome for that application.

I applied to a wide range of permanent (tenure-track and teaching-track) faculty positions, as well as a number of temporary (postdoc, visiting assistant professor, lecturer) positions. Splitting my applications along this dimension shows that I had noticeably more success in my applications for temporary positions (10.3% response rate) than permanent ones (1.8% response rate).

Since my non-nothing outcomes are so few, I can easily list them in more detail:

• Two postdoc offers
• A postdoc I interviewed for and declined in favor of another postdoc offer
• A teaching-track flyout I declined in favor of a data science offer
• A tenure-track interview I declined in favor of the same offer

If we break down the jobs I applied for by academic subfield, some unsurprising patterns emerge. Data science jobs include those listed as computational social science, jobs listed for a substantive subfield and methods are coded under the substantive area, and international relations, conflict, peace studies, security studies, and international political economy are all represented in the IR category.

The majority of jobs I applied to (92) were advertised as international relations. While much of my research sits at the intersection of international relations and comparative politics, very few of the jobs I applied to do. I didn’t track how frequent these jobs are, so it could just be a case of few jobs to apply to. Data science (24) handily outnumbers the more traditional subfield of methods (14), reflecting increasing interest in the former by the discipline.

The map below geographically visualizes the jobs I applied to. Each circle represents one institution, with the size of the circle denoting how many positions I applied for. I applied to five positions at UCSD, the most of any institution.

I focused primarily on the Eastern US and California. I applied to jobs in 31 states and the District of Columbia, meaning there were 19 states I did not apply to any jobs in. Looking at my applications over time helps tell the story of my academic job search process.

The 2018-19 academic job market season was my final year of grad school. I wanted to be done, so I applied to a wide variety of jobs. The postdoc I received an offer from was actually the last position I applied for in this cycle. I was a little more selective in the 2019-20 job market season because I had an excellent postdoc, with a high chance of a second year of funding. I started a new postdoc in 2020 and knew that I had a second year of funding guaranteed. COVID-19 absolutely devastated the job market that cycle as well. With a second year of funding secure and precious few institutions hiring, I decided to spend my time focusing on improving my CV and applied to a total of four jobs that cycle, all tenure-track. The market somewhat recovered in the 2021-22 cycle, but there were still far fewer jobs than in my first two cycles. I applied to 19 jobs this cycle, all of them permanent. There were some great postdocs this cycle, but three years as a postdoc had been enough for me.

Two jobs did show interest in me this last cycle, but it was too little, too late. I had an offer for a data science job when I received an on-campus interview for one job, and had already accepted the offer when I received a Zoom interview for another. Given the typical pace of an academic search, it was possible that even if I were successful in getting an offer for either of these positions, it wouldn’t be for another month or two. My postdoc ended in June, and an offer in hand doing interesting research was an easy sell compared to that uncertainty.

Across all 142 applications, I ended up submitting 399 letters of recommendation to search committees. I was very fortunate that UNC has a department administrator handle letters for grad students as a sort of discount (read: free) Interfolio Dossier service. They generously provide this service to graduates of the department until they secure their first permanent job, even after they have left. I spent so long on the academic job market that I had no less than three different people help me with this process. I am incredibly grateful for their efforts and want to highlight the support they gave me.

I haven’t done as good of a job tracking my applications to nonacademic jobs because the process is much less structured and standardized. Some applications require a cover letter, so I can count up all the cover letters saved in my job search folder: 25. You can also apply for many jobs with just a résumé. Let’s say I applied to 10 of those, which makes 35 applications total.

I started the interview process with seven of these employers. Acknowledging some uncertainty in the denominator, that’s a 20% response rate, more than six times higher than my academic response rate of 3.5%. I completed the interview process with four of these employers, receiving one rejection and three offers (I withdrew from the other three interview processes after accepting one of those offers). A 75% interview success rate is pretty incredible compared to my experience on the academic job market. That’s an overall success rate of 8.6%, which is also more than six times higher than my overall success rate for academic jobs.

Or is it a 50% success rate? I actually interviewed for two different positions with two of these employers, so you could also slice the data less favorably and say I received offers for three out of six positions I interviewed for. That’s still an overall success rate of 8.1%, which is pretty damn good in my eyes. I also want to highlight some of the experiences I had on the nonacademic job market that I can’t imagine ever happening on the academic one:

• Recruiters reached out to me to ask me to apply to positions
• One employer alerted me to another position they were hiring for and connected me with the hiring manager for it
• Another informed me that I was actually overqualified for the position I applied for and considered me for a more senior position instead
• I had my first job offer almost exactly three months after I started my nonacademic job search in earnest
• I received three job offers in three days that week

Others have written extensively about why you shouldn’t view a nonacademic job as a backup option or a failure, but sometimes it’s just nice to know that people want to pay you. If you’re striking out on the academic job market, there are plenty of other options out there. So it goes.

Sometimes it’s faster (easier) to just write code that works for you, on your system, without any consideration for some poor researcher who may try to replicate your results in the future.¹ This tendency was especially bad for this project because at various points in time I was writing code to run on my personal laptop and two different high performance computing clusters. This is a recipe for code that doesn’t travel well and will almost certainly fail to replicate.

There were a lot of changes I made to my code to ensure my results replicate, but the most tedious (and time consuming, by far) was cleaning up my file paths. Due to the computationally intensive GIS work and Bayesian statistics involved in the project, I ran lots of code on a cluster, and then pulled the results back to my laptop to summarize and create figures. This unsurprisingly resulted in a huge mess when looking at the project as a whole, rather than any individual script. Luckily, R and Rstudio made things (relatively) painless to fix.

File paths

Anytime you load a dataset into R, you need to specify the path to that file. The same’s true when you save R output to a file. This article started as a chapter of my dissertation, so all of the code originally lived in the Dissertation folder on my laptop. However, as I started adapting it to an article length manuscript, I created a new Conflict Preemption folder in my Projects folder. By the time the article was accepted, I had two main folders I needed to combine:

• /Users/Rob/Dropbox/UNC/Dissertation/Onset
• /Users/Rob/Dropbox/WashU/Projects/Conflict Preemption

Both of these folders live in my Dropbox, but that’s about where the similarities end. I wrote most of the code for running models while still at UNC, so when I added new scripts to run models to respond to reviewer comments, I still stuck them in the UNC folder. That also means that all of the output of these models ended up in the UNC folder when it got transferred from the cluster. However, when I needed to do something simpler like create a time series plot of the number of separatist groups in existence, I wrote that code in the WashU folder. I also had a script in the WashU folder to load all of the results and generate plots from them. Because this script and the data it needed to load where in completely different directories, this is what I had to do to load the data to create one of the main figures:

load('/Users/Rob/Dropbox/UNC/Dissertation/Onset/Figure Data/marg_eff_pop_df_cy.RData')

Not particularly likely to work on anyone else’s computer. To fix this, I needed to move all of the data to the Conflict Preemption folder, which was easy, and then rewrite all of the code the referenced file locations, which was less easy.

Here

As a first step, I needed to chop off /Users/Rob/Dropbox/UNC/Dissertation/Onset/ from the start of every file path. All the files for the article, including both the R scripts and the various data files, now live in /Users/Rob/Dropbox/WashU/Projects/Conflict Preemption, but all of the file paths in the scripts still start with /Users/Rob/Dropbox/UNC/Dissertation/Onset, because that’s where all the files were before. You can do this just using the standard find and replace functionality built into RStudio. However, there’s no guarantee that someone in the future will correctly set R’s working directory before running the code. I used the here R package to ensure that R can always find everything it needs for my code. All you have to do is wrap file paths in the here() function in the package, and they’ll be automatically completed with the full file path, letting R find your files.²

You need to use the relative path to each file, so for a file with an absolute path of /Users/Rob/Dropbox/WashU/Projects/Conflict Preemption/Figure Data/marg_eff_pop_df_cy.RData, the relative path (relative to the project folder of /Users/Rob/Dropbox/WashU/Projects/Conflict Preemption) would be Figure Data/marg_eff_pop_df_cy.RData. The final bit of R code looks like this:

load(here('Figure Data/marg_eff_pop_df_cy.RData'))

The addition of that here() in between load() and the file path means that things are no longer as simple as finding and replacing the start of the file path.

Regular expressions

Luckily, I was able to take advantage of RStudio’s built in support for regular expressions to save myself from having to manually change each line of code that either loaded or saved a file. Regular expressions are a powerful way to search through and manipulate text. You can activate them in RStudio’s find and replace dialog by checking the Regex box:

Once you’ve done that, certain characters in your search will no longer be interpreted literally. The most important difference is probably ., which is a stand-in for any character.³ This is similar to how * is a wildcard in the Unix shell, e.g., you can use ls *.R to list all R script files in a folder. The main regular expression feature I used is the capturing group, which allows you to identify and extract a subset of a line of text. You designate a capturing group by surrounding the desired text with parentheses. To fix all of the code loading RData files from the Figure Data folder, my regular expression looked like this:

'/Users/Rob/Dropbox/UNC/Dissertation/Onset/(Figure Data/.*\.RData)'

It starts with /Users/Rob/Dropbox/UNC/Dissertation/Onset/, which is the part I want to get rid of. Next, (Figure Data/.*\.RData)' tells the regular expression to look for any character (.) repeated an unlimited number of times (*) followed by .RData. Because . is a special character in regular expressions, we have to escape it with a backslash (\). This will match any file name ending in .RData in the Figure Data folder. If we left out the leading /Users/Rob/Dropbox/UNC/Dissertation/Onset/, we end up with the capturing group we want, but since /Users/Rob/Dropbox/UNC/Dissertation/Onset/ wouldn’t be part of the search string, it wouldn’t end up getting replaced. This is the same reason we need to include the opening and closing quotation marks; if we didn’t, we’d end up with a here() command inside quotation marks, which R would just treat as a string and not a command.

At this point I had the core of the line that I wanted to keep, but now I needed to extract it and place it inside of a call to here(). You accomplish this goal using a backreference to the capturing group. To reference the first capturing group, you use either \1 or $1 depending on which version of regular expressions you are using. This is often very difficult to figure out, and is one of the most annoying things about regular expressions. You’ll often just have to experiment and find out which one to use through trial and error. Luckily RStudio accepts either version!

To replace the absolute path with a relative one wrapped in a here() call, this is what I typed into the Replace field in the find and replace dialog:

here('$1')

and it resulted in this:

here('Figure Data/marg_eff_pop_df_cy.RData')

Thanks to the power of capture groups, you can just hit the replace all button and instantly transform every file path into a much more portable and replication-friendly one.

A little bit faster now

If you’re feeling really confident that you moved every file correctly, you can replace all file paths with the following regular expression:

'/Users/Rob/Dropbox/UNC/Dissertation/Onset/(.*\..*)'

This will get any files with file extensions (the \. followed by .* to ensure there’s at least one character after a literal period), as well as any preceding subdirectories (the initial .*) and stick them all into the resulting here() call. As an example, this will successfully turn this: fileConn <- file(here::here(‘Tables/pd_pop_cy.tex’))

groups <- readRDS('/Users/Rob/Dropbox/Dissertation/Onset/Input Data/groups_nightlights.RDS')

into this:

groups <- readRDS(here::here('Input Data/groups_nightlights.RDS'))

See this simple example, which displays the area of each county in North Carolina, from the sf package documentation.¹ First, we need to load sf and then get the built-in nc dataset:

library(sf)
nc <- st_read(system.file('shape/nc.shp', package = 'sf'))
plot(nc[1])

Since I needed to generate choropleths for multiple countries, I decided to use ggplot2’s powerful faceting functionality. Unfortunately, as I discuss below, ggplot2 and sf don’t work together perfectly in ways that become more apparent (and problematic) the more complex your plots get. I moved away from faceting, and just glued together a bunch of separate plots, but then I had to figure out how to end up with a shared legend for five separate plots. Read on to see how I solved both of these issues.

The data

I already loaded sf to make the plot of North Carolina above, so now let’s load the remaining packages we’ll use:

library(tidyverse) # data manipulation and plotting
library(tmap)      # spatial plots
library(cowplot)   # combine plots
library(RWmisc)    # clean plot theme

I’m working with cleaned and subsetted versions of ACLED and GADM, which I’ve uploaded to my website as PKO.Rdata if you want to download them and run this code yourself. The acled object contains a list of attacks on peacekeepers in active Chapter VII UN peacekeeping missions in Subsaharan Africa, while the adm object contains all of the second order administrative districts (ADM2) in the five countries with active missions.

## load data
load(url('https://jayrobwilliams.com/data/PKO.Rdata'))

## inspect
head(acled)
head(adm)

## Simple feature collection with 6 features and 30 fields
## Geometry type: POINT
## Dimension:     XY
## Bounding box:  xmin: -3.6102 ymin: 0.4966 xmax: 29.4654 ymax: 19.4695
## Geodetic CRS:  WGS 84
## # A tibble: 6 x 31
##   data_id   iso event_id_cnty event_id_no_cnty event_date  year time_precision
##     <dbl> <dbl> <chr>                    <dbl> <date>     <dbl>          <dbl>
## 1 6713346   140 CEN47283                 47283 2019-12-27  2019              1
## 2 6689432   180 DRC16211                 16211 2019-12-08  2019              1
## 3 7578005   180 DRC16182                 16182 2019-12-04  2019              1
## 4 7191069   466 MLI3253                   3253 2019-10-21  2019              1
## 5 6759702   466 MLI3225                   3225 2019-10-06  2019              1
## 6 6023339   466 MLI3224                   3224 2019-10-06  2019              1
## # … with 24 more variables: event_type <chr>, sub_event_type <chr>,
## #   actor1 <chr>, assoc_actor_1 <chr>, inter1 <dbl>, actor2 <chr>,
## #   assoc_actor_2 <chr>, inter2 <dbl>, interaction <dbl>, region <chr>,
## #   country <chr>, admin1 <chr>, admin2 <chr>, admin3 <chr>, location <chr>,
## #   geo_precision <dbl>, source <chr>, source_scale <chr>, notes <chr>,
## #   fatalities <dbl>, timestamp <dbl>, iso3 <chr>, month <dbl>,
## #   geometry <POINT [°]>

## Simple feature collection with 6 features and 19 fields
## Geometry type: MULTIPOLYGON
## Dimension:     XY
## Bounding box:  xmin: 18.54607 ymin: 4.221635 xmax: 22.395 ymax: 9.774724
## Geodetic CRS:  WGS 84
## # A tibble: 6 x 20
##   GID_0 NAME_0   GID_1  NAME_1 NL_NAME_1 GID_2 NAME_2 VARNAME_2 NL_NAME_2 TYPE_2
##   <chr> <chr>    <chr>  <chr>  <chr>     <chr> <chr>  <chr>     <chr>     <chr> 
## 1 CAF   Central… CAF.1… Bamin… <NA>      CAF.… Bamin… <NA>      <NA>      Sous-…
## 2 CAF   Central… CAF.1… Bamin… <NA>      CAF.… Ndélé  <NA>      <NA>      Sous-…
## 3 CAF   Central… CAF.2… Bangui <NA>      CAF.… Bangui <NA>      <NA>      Sous-…
## 4 CAF   Central… CAF.3… Basse… <NA>      CAF.… Alind… <NA>      <NA>      Sous-…
## 5 CAF   Central… CAF.3… Basse… <NA>      CAF.… Kembé  <NA>      <NA>      Sous-…
## 6 CAF   Central… CAF.3… Basse… <NA>      CAF.… Minga… <NA>      <NA>      Sous-…
## # … with 10 more variables: ENGTYPE_2 <chr>, CC_2 <chr>, HASC_2 <chr>,
## #   ID_0 <dbl>, ISO <chr>, ID_1 <dbl>, ID_2 <dbl>, CCN_2 <dbl>, CCA_2 <chr>,
## #   geometry <MULTIPOLYGON [°]>

First attempt: ggplot2

The first step we need to do is associate each individual attack with the ADM2 it occurred in. We can do this with the st_join() function. This function executes a left join by default, so by using adm for the x argument and acled for the y argument, we end up with one row for every ADM2 with no attacks in it, and n rows for each ADM2 with attacks in it, where n equals the number of attacks in that ADM2. We can then use group_by() and summarize() to create a count of attacks for each ADM2 by summing the number of non-NA observations of event_id_cnty, the main ID field in ACLED. Finally, I log this count variable (using log1p() to account for the ADM2s without any attacks because ln(0) is undefined) to make the resulting plot more informative due to outliers in Northern Mali and the Eastern DRC. Putting it all together:

st_join(adm, acled) %>% 
  group_by(NAME_0, NAME_1, NAME_2) %>% 
  summarize(attacks = log1p(sum(!is.na(event_id_cnty)))) %>% 
  ggplot(aes(fill = attacks)) +
  geom_sf(lwd = NA) +                 # no borders
  scale_fill_continuous(name = 'PKO targeting\nevents (logged)') +
  theme_rw() +                        # clean plot
  theme(axis.text = element_blank(),  # no lat/long values
        axis.ticks = element_blank()) # no lat/long ticks

That’s a lot of wasted white space, and it can make certain countries harder to see. Let’s split it out using facet_wrap(). We simply add a facet_wrap() call to our ggplot2 code, and tell it to split by our country name variable, NAME_0:

adm %>% 
  st_join(acled) %>% 
  group_by(NAME_0, NAME_1, NAME_2) %>% 
  summarize(attacks = log1p(sum(!is.na(event_id_cnty)))) %>% 
  ggplot(aes(fill = attacks)) +
  geom_sf(lwd = NA) +
  scale_fill_continuous(name = 'PKO targeting\nevents (logged)') +
  facet_wrap(~ NAME_0) +
  theme_rw() +
  theme(axis.text = element_blank(),
        axis.ticks = element_blank())

We’ve got facets, but everything is still clearly on the same scale. let’s set scales = 'free' in our call to facet_wrap() to try and fix that.

st_join(adm, acled) %>% 
  group_by(NAME_0, NAME_1, NAME_2) %>% 
  summarize(attacks = log1p(sum(!is.na(event_id_cnty)))) %>% 
  ggplot(aes(fill = attacks)) +
  geom_sf(lwd = NA) +
  scale_fill_continuous(name = 'PKO targeting\nevents (logged)') +
  facet_wrap(~ NAME_0, scales = 'free') +
  theme_rw() +
  theme(axis.text = element_blank(),
        axis.ticks = element_blank())

## Error: coord_sf doesn't support free scales

And we get an error. It turns out the the ggplot2 codebase assumes that it can maniulate axes independently of one another. This is very much not the case with geographic data where a meter vertically needs to equal a meter horizontally, so coord_sf() locks the axes in much the same manner as coord_fixed().² To try and get around the limitations from ggplot2’s non-spatial origins, I turned to a package written from the ground up for plotting spatial data.

Second attempt: tmap

My googling led me to this Stack Overflow answer extolling the virtue of the tmap package.³ tmap is a package for drawing thematic maps from sf objects using a syntax very similar to ggplot2. We can reuse the same data wrangling code and as before pipe it into our plotting function, which this time is tm_shape(). We then add a call to tm_polygons() to get our colored features and tm_facet() to split them apart. Note that unlike ggplot2, we need to quote the names of variables in tmap functions:

st_join(adm, acled) %>% 
  group_by(NAME_0, NAME_1, NAME_2) %>% 
  summarize(attacks = log1p(sum(!is.na(event_id_cnty)))) %>% 
  tm_shape() +
  tm_polygons('attacks', title = 'PKO targeting\nevents (logged)') +
  tm_facets('NAME_0')

Much better so far! However, notice that tmap defaults to assuming that our attacks variable is discrete. We’ll need to tell it that it’s continuous. And what if we moved that legend down to the bottom right to get rid of the wasted space currently there?

adm %>% 
  st_join(acled) %>% 
  group_by(NAME_0, NAME_1, NAME_2) %>% 
  summarize(attacks = log1p(sum(!is.na(event_id_cnty)))) %>% 
  tm_shape() +
  tm_polygons(col = 'attacks',
              title = 'PKO targeting\nevents (logged)',
              style = 'cont') +                  # continuous variable
  tm_facets('NAME_0') +
  tm_layout(legend.outside.position =  "bottom", # legend outside below
            legend.position = c(.8, 1.1))        # manually position legend

This is…fine. You’ll notice that there’s a lot of white space at the bottom of the plot, which I still haven’t figured out how to eliminate, and I personally prefer the color palette options available in ggplot2. Finally, there’s not much control over the legend compared to what you get with ggplot2, so let’s head back there and try to come at this problem from a different direction.

Third attempt: cowplot

While we’re still using ggplot2 to make individual plots, we need some way to combine them into a final plot. We can rely on the plot_grid() function in the cowplot library for that.⁴ We need to create five subplots, which we could do manually, but let’s do it programmatically because at some point you may need to do this for 27 different countries. The best way to store our five subplots is in a list, because lists in R can contain any type of R objects as their elements.⁵ I’m going to use the map() function from the purrr package to accomplish this, but you could also use lapply(). map() takes a list as its first argument, .x and a function as its second, .f. To see how map works, look at the following example:

map(1:3, sample)

## [[1]]
## [1] 1
## 
## [[2]]
## [1] 1 2
## 
## [[3]]
## [1] 2 3 1

map() returns a list of length 3 because our input .x was a vector of length three, and it applies the function .f to each element of .x. I’m going to use an anonymous function to filter adm to only contain ADM2s from one country at a time, then create our subplots separately like we did together above:

pko_countries <- c('Central African Republic', 'Democratic Republic of the Congo',
                   'Mali', 'South Sudan', 'Sudan')

## create maps in separate plots, force common scale between them
maps <- map(.x = pko_countries, 
            .f = function(x) adm %>% 
              filter(NAME_0 == x) %>% 
              st_join(acled) %>% 
              group_by(NAME_0, NAME_1, NAME_2) %>% 
              summarize(attacks = log1p(sum(!is.na(event_id_cnty)))) %>% 
              ggplot(aes(fill = attacks)) +
              geom_sf(lwd = NA) +
              scale_fill_continuous(name = 'PKO targeting\nevents (logged)') +
              theme_rw() +
              theme(axis.text = element_blank(),
                    axis.ticks = element_blank()))

We can either supply each individual subplot to plot_grid() separately, or we can use the plotlist argument to pass a list of plots; good thing we saved them in a list:

## use COWplot to combine and add single legend
plot_grid(plotlist = maps, labels = LETTERS[1:5], label_size = 10, nrow = 2)

I tried using the name of each country as the subplot label, but because label positioning is relative to the width of labels it was impossible to get them all nicely left-aligned. As a result, I had to settle on using letters to label the subplots and then identifying them in the figure caption in text. As you’ll see later, there’s no perfect way of accomplishing this and you’ll have to make a trade-off somewhere.

Setting aside that compromise, there’s still one issue with this plot that we can fix. We’re measuring the same thing (attacks on UN peacekeeping personnel) in all five choropleths, so there’s no need for five separate scales.

Shared legend

The cowplot documentation demonstrates how to use the get_legend() function to extract the legend from one of the subplots and then add it as another element to plot_grid(), placing it in the bottom right like we sort of managed to do with tmap. However, we need to add theme(legend.position = 'none') to the ggplot call for each subplot, otherwise we’ll just end up with six legends. That means we need to apply to each element of our list of maps, which means it’s another job that map() is perfect for! We’ll use map() to take each subplot in maps and remove the legend from it, then use get_legend() to add a legend in the bottom right.

## use COWplot to combine and add single legend
plot_grid(plotlist = c(map(.x = maps,
                           .f = function(x) x + theme(legend.position = 'none'))),
          get_legend(maps[[1]]),
          labels = LETTERS[1:5], label_size = 10, nrow = 2)

This doesn’t look right! We told plot_grid() to start with our maps, so why is the legend the first thing in the plot? If you look closely at the documentation for plot_grid(), you’ll see that the ... argument comes before the plotlist argument in the function definition. Even when we specify plotlist first, the function will add plotlist after ....⁶ To fix this, all we need to do is concatenate the results of get_legend() with the results of our call to map(). Note that we need to first transform the former to a list with list(), otherwise each element of it will be concatenated separately rather than as a grob object:

## use COWplot to combine and add single legend
plot_grid(plotlist = c(map(.x = maps,
                           .f = function(x) x + theme(legend.position = 'none')),
                       list(get_legend(maps[[1]]))),
          labels = LETTERS[1:5],
          label_size = 10,
          nrow = 2)

So far so good. But if we try using a different map in our call to get_legend(), things get weird:

## use COWplot to combine and add single legend
plot_grid(plotlist = c(map(.x = maps,
                           .f = function(x) x + theme(legend.position = 'none')),
                       list(get_legend(maps[[4]]))),
          labels = LETTERS[1:5], label_size = 10, nrow = 2)

Each subplot has its own unique legend that’s automatically generated from the values of attacks it contains. This is even worse than it might seem at first glance, because it means that the various subplots are in no way comparable to one another!

Accurate shared legend

To avoid misrepresenting the data, we need to ensure that each subplot has the same legend. The easiest way to do this is to manually set the legend for each subplot in our call to scale_fill_continuous(). Even though we’re manually setting the bounds of the legend, that doesn’t mean we have to hard code them. We can use a simpler version of our code to join attacks to ADM2s and then calculate the highest number of attacks across all countries in the data. Then we take advantage of the fact that scale_fill_continuous() can pass additional parameters to continuous_scale() via the ... argument. The continuous_scale() function is a low-level function used throughout ggplot2 to construct continuous scales, and it has a limits argument that sets the bounds of the scale. All we have to do is pass the minimum and maximum (logged) numbers of attacks in the data and we’re in business:

st_join(adm, acled) %>% 
  st_drop_geometry() %>%   # we don't need a map at the end; drop geometry to speed up
  group_by(NAME_0, NAME_1, NAME_2) %>% 
  summarize(attacks = log1p(sum(!is.na(event_id_cnty)))) %>% 
  pull(attacks) %>%        # extract attacks variable
  range() -> attacks_range # get min and max

## create maps in separate plots, force common scale between them
maps_shared <- map(.x = pko_countries, 
                   .f = function(x) adm %>% 
                     filter(NAME_0 == x) %>% 
                     st_join(acled) %>% 
                     group_by(NAME_0, NAME_1, NAME_2) %>% 
                     summarize(attacks = log1p(sum(!is.na(event_id_cnty)))) %>% 
                     ggplot(aes(fill = attacks)) +
                     geom_sf(lwd = NA) +
                     scale_fill_continuous(limits = attacks_range,
                                           name = 'PKO targeting\nevents (logged)') +
                     theme_rw() +
                     theme(axis.text = element_blank(),
                           axis.ticks = element_blank()))

Now all that’s left is to use plot_grid() to put it all together:

## use COWplot to combine and add single legend
plot_grid(plotlist = c(map(.x = maps_shared,
                           .f = function(x) x + theme(legend.position = 'none')),
                       list(get_legend(maps_shared[[1]]))),
          labels = LETTERS[1:5], label_size = 10, nrow = 2)

And unlike before, the legend is identical regardless of which subplot we use with get_legend():

## use COWplot to combine and add single legend
plot_grid(plotlist = c(map(.x = maps_shared,
                           .f = function(x) x + theme(legend.position = 'none')),
                       list(get_legend(maps_shared[[4]]))),
          labels = LETTERS[1:5], label_size = 10, nrow = 2)

This approach is still useful even if you’re not working with spatial data. plot_grid() is powerful because it lets you make asymmetric arrangements like this example from the cowplot documentation:

p1 <- ggplot(mtcars, aes(disp, mpg)) + 
  geom_point()
p2 <- ggplot(mtcars, aes(qsec, mpg)) +
  geom_point()

plot_grid(p1, p2, labels = c('A', 'B'), rel_widths = c(1, 2))

If the units you’re faceting by contain substantially different observations, you might end up in a situation where the automatically generated legends are different from one another. Manually creating the scale of the legend and ensuring it’s the same for all plots would solve this problem here, too.

Bonus: still to solve

Don’t let anyone convince you they know everything. I still haven’t managed to get my ideal (conditional on regular faceting with facet_wrap() being out of the question) solution to this working. I tried to create five subplots and just add a facet label to each, with each one being a facet of one panel. Straightforward enough, right?

maps_facet <- map(.x = pko_countries, 
                  .f = function(x) adm %>% 
                    filter(NAME_0 == x) %>% 
                    st_join(acled) %>% 
                    group_by(NAME_0, NAME_1, NAME_2) %>% 
                    summarize(attacks = log1p(sum(!is.na(event_id_cnty)))) %>% 
                    ggplot(aes(fill = attacks)) +
                    geom_sf(lwd = NA) +
                    scale_fill_continuous(limits = attacks_range,
                                          name = 'PKO targeting\nevents (logged)') +
                    facet_wrap(~NAME_0) +
                    theme_rw() +
                    theme(axis.text = element_blank(),
                          axis.ticks = element_blank()))

plot_grid(plotlist = c(map(.x = maps_facet,
                           .f = function(x) x + theme(legend.position = 'none')),
                       list(get_legend(maps_facet[[1]]))),
          nrow = 2)

Not so much, and no amount of tinkering with the align and axis arguments to plot_grid() has yielded any improvement. The specific paper this plot is for doesn’t have any other plots with facets, so I’m content to go with my inelegant solution of lettered labels and a key to them in the figure caption. If that weren’t the case, I might still be fiddling with this and getting deeper and deeper into the source code for plot_grid().

That means I’m spending a decent chunk of time thinking about and planning future trips. At some point in the process of doing this, I realized that I could use the GIS skills from my day job to help make planning future trips more efficient. In this post I walk through how you can use GIS tools in R to help with some of the route planning for a multiday backpacking trip. Specifically, how you can use open source spatial data on geography and transportation infrastructure to identify potential campsites along a hiking trail.

This was largely an exercise in seeing how I could apply GIS skills I’ve learned in the study of political violence to small-scale GPS navigation. I haven’t had the opportunity to hit the trail and test out any of the assumptions I use in this process yet, so you should view this post as more of a (loose) method than concrete suggestions. For a short and simple point-to-point hike with only one route, there’s really no need to engage in this level of GIS analysis. I’ve kept things simple to make them easier to follow, but this approach could actually be useful and save some time when planning a longer trip with many potential routes.

Backcountry camping

At some point in the future, I want to hike the Uwharrie Trail in Uwharrie National Forest in central North Carolina, near where I went to grad school. As I think about this (probably far off) trip, I’ve been using CalTopo to plan my route.

If you spend any amount of time in the outdoors, you should know about CalTopo. CalTopo is a website that lets you plan routes (hiking, skiing, rafting, etc.) on top of super high resolution topographic maps. You can then turn your smartphone into a full-featured GPS and use it to follow those routes (CalTopo offes a mobile app, as does Gaia GPS, both for about $20 a year). While the Uwharrie Trail is a pretty straightforward hike, I’ve been using this as an excuse to try and apply my GIS skills in a new context.

CalTopo is great, but it’s very point and click. I like doing things programmatically when I can, so that means it’s time to grab some of the open source data that CalTopo uses so we can play around with it in R. The base map in CalTopo is called MapBuilder Topo, and uses OpenStreetMap data as its starting point, so let’s start there.

Disclaimer

This guide is intended to show how to identify potential backcountry campsites on public land where dispersed camping is permitted. If you are backpacking in an area with designated, maintained backcountry campsites, you should use them. Dispersed camping is typically permitted in less-traveled areas where the impact of campers is better minimized by diffusing it rather than concentrating it into a handful of designated sites.¹

Always check regulations for any land you plan to camp on to see if there are specific requirements for site selection or areas where camping is prohibited. Picking an actual campsite requires identifying areas where your saftey will be maximized and the longterm impact of your stay will be minimized. See this guide for the basics and this series for a slightly more hardcore set of principles to follow. And remember, never go into the wilderness without telling someone where you’re going and when you should be back.

Getting the data

OpenStreetMap (OSM) is an open source map of the entire globe; think of it as a hybrid of Google Maps and Wikipedia. OSM is designed so that anyone can easily add to or edit it. Setting aside the normative value of this perspective, this is helpful for us because it means that OSM is transparent. We can use the excellent osmdata R package to query OSM via the [Overpass API], and we can use OSM itself via the OSM website to learn the various parameters we’ll use to query OSM.

Trails

The getting started vignette covers much of the basics of using osmdata. The key functions are osmdata::opq(), which builds a query to the Overpass API, and osmdata::add_osm_feature(), which requests specific features. OSM classifies features using key-value pairs, and we can use the OSM website to figure out just which pairs we need. Navigate to an area of interest, right-click on the feature of interest, and then select “query features.”

Next, select the desired feature in the dialog on the left of the screen. In this case, select the “Relation” rather than the “Path” because the path will only include one segment of the trail while the relation will include its entire length.

We can see here that the Uwharrie Trail relation has type=hiking, so that’s the key-value pair wew’ll have to specify in our query.

Make sure to use the bbox argument to osmdata::opq(), otherwise you’ll request every hiking trail in the world! You can manually specify the four edges of a bounding box to search in, or you can use the osmdata::getbb() function to get it automatically using the Nominatim geocoder.

library(tidyverse)
library(sf)
library(osmdata)

## get hiking routes in Uwharrie National Forest
unf_trails <- opq(bbox = getbb('uwharrie national forest usa')) %>% 
  add_osm_feature(key = 'route', value = 'hiking') %>% 
  osmdata_sf()

Notice that we use the osmdata::osmdata_sf() function to convert the resulting object for use with the sf R package. Let’s inspect the resulting object of class osmdata_sf.

## inspect
unf_trails

## Object of class 'osmdata' with:
##                  $bbox : 35.3951403,-80.0236608,35.4351403,-79.9836608
##         $overpass_call : The call submitted to the overpass API
##                  $meta : metadata including timestamp and version numbers
##            $osm_points : 'sf' Simple Features Collection with 3341 points
##             $osm_lines : 'sf' Simple Features Collection with 26 linestrings
##          $osm_polygons : 'sf' Simple Features Collection with 0 polygons
##        $osm_multilines : 'sf' Simple Features Collection with 1 multilinestrings
##     $osm_multipolygons : NULL

We can see that the unf_trails object includes points, lines, polygons, multilines, and multipolygons. We want to use the lines since that will include any short trail segments that aren’t part of a larger trail. We can easily plot the trails using this object.

## plot
plot(unf_trails$osm_lines$geometry, col = 'coral4')

Don’t get lost

Let’s do some quick sanity checks. First, Wikipedia tells us the trail should be about 20 miles. We can use the sf::st_length() function to measure the length of each trail segment, and the sf::st_union() function to combine all segments. We’ll get our answer in meters, which as a metric-deprived American, won’t be all that helpful to me. To get around this, we can use the `units::st_units() function to convert from meters to miles.

## measure total trail length
st_union(unf_trails$osm_lines$geometry) %>% # combine all segments.
  st_length() %>% # measure length
  units::set_units(mi) # convert to miles

## 28.26457 [mi]

While that’s initially concerning, a closer reading of the Wikipedia article for the trail reveals that it was originally 40 miles long, so OSM likely includes some of the Northern section of the trail beyond what’s officially recognized today.

We should also plot the bounding box that osmdata::getbb() ends up generating to ensure we’re not missing any part of the trail. We can do this with the OpenStreetMap [R package](https://cran.r-project.org/package=OpenStreetMap. Here we unfortunately need to manually specify the bounding box as a series of two vectors with the latitude and longitude coordinate of the upper-left and lower-right of the box. OpenStreetMap::openmap() uses (latitude, longitude) pairs, not (longitude, latitude) pairs as is more common in GIS, i.e., (y, x) not (x, y), so be sure to include them in that order.[^lat-long]{markdown} OpenStreetMap::openproj() also requires a projection argument, so I use sf::st_crs(4326)$proj4string to generate one automatically, ensuring I don’t introduce a type somewhere by accident.

[^lat-long]:{markdown} I spent 20 minutes not understanding why I couldn’t get this to work before I finally read the documenation. Don’t be like me, folks.

library(OpenStreetMap)

## get bounding box
unf_bb <- getbb('uwharrie national forest usa')

## get OSM tiles
unf_tile <- openmap(c(unf_bb[2,1],  # lat
                      unf_bb[1,1]), # long
                    c(unf_bb[2,2],  # lat
                      unf_bb[1,2]), # long
                    type = 'osm', mergeTiles = T)

## project map tiles and plot (OSM comes in Mercator...)
plot(openproj(unf_tile), projection = st_crs(4326)$proj4string)

## plot trails
plot(unf_trails$osm_lines$geometry, add = T, col = 'coral4')

Uh oh. We can see that we’re only getting a small portion of the total trail and that it trails (heh) off the map on three sides. That’s not great, so let’s fix it. We can start by looking up Uwharrie National Forest itself on the OSM website. This gives us the boundaries of the official forest land in orange.

We can see from the dialog on the left that the forest’s OSM ID is 2918413, so we can use the osmdata::opq_osm_id() function to get the polygons for the forest’s boundaries. Let’s grab the forest boundaries and plot them, along with the bounding box they imply and the bounding box that resulted from osmdata::getbb() (in red) for comparison.

## get Uwharrie National Forest Boundaries
unf <- opq_osm_id(type = 'relation', id = 2918413) %>% 
  osmdata_sf()

## plot Uwharrie National Forest polygons
plot(unf$osm_multipolygons$geometry, col = 'lightgreen', border = NA, bty = 'n')

## construct line for original bounding box
plot(st_multilinestring(list(matrix(c(unf_bb[1, 1], unf_bb[2, 1],
                                      unf_bb[1, 1], unf_bb[2, 2],
                                      unf_bb[1, 2], unf_bb[2, 2],
                                      unf_bb[1, 2], unf_bb[2, 1],
                                      unf_bb[1, 1], unf_bb[2, 1]),
                                      ncol = 2, byrow = T))),
     add = T, col = 'red')

## plot bounding box for Uwharrie National Forest polygons
plot(st_as_sfc(st_bbox(unf$osm_multipolygons)), add = T)

## plot trails
plot(unf_trails$osm_lines$geometry, add = T, col = 'coral4')

Wow, we were missing a lot before. Let’s use the bounding box for the entire forest as our new bounding box. First, we plot OSM using this new bounding box. st_bbox() yields a vector of four numbers, rather than the matrix that osmdata::getbb() produces, so we need to work around this and specify the top-left and bottom-right corners of our new, bigger bounding box.

## get OSM tile for Uwharrie National Forest polygons
unf_full_tile <- openmap(c(st_bbox(unf$osm_multipolygons)[4],  # lat
                           st_bbox(unf$osm_multipolygons)[1]), # long
                         c(st_bbox(unf$osm_multipolygons)[2],  # lat
                           st_bbox(unf$osm_multipolygons)[3]), # long
                         type = 'osm', mergeTiles = T)

## project and plot OSM tile
plot(openproj(unf_full_tile), projection = st_crs(4326)$proj4string)

## plot trails
plot(unf_trails$osm_lines$geometry, add = T, col = 'coral4')

That’s much better! We’re getting a lot of area beyond the trail, but it’s easy to filter that out later so it’s better to grab too much than too little.

The whole trail

Now we can go back and grab all hiking trails in Uwharrie National Forest using our new bounding box. osmdata::opq() expects a bounding box in a certain format, so let’s inspect it to see what we’re working with and what we need to reshape the output of sf::st_bbox(unf$osm_multipolygons) into:

## bbox format osmdata::opq() expects
unf_bb

##         min       max
## x -80.02366 -79.98366
## y  35.39514  35.43514

## rearrange sf::st_bbox() output
matrix(st_bbox(unf$osm_multipolygons), ncol = 2,
       dimnames = list(c('x', 'y'), c('min', 'max')))

##         min       max
## x -80.17085 -79.73170
## y  35.21987  35.63684

Note that I’m specifying row and column names when creating the new bounding box. Without them, osmdata::opq() will fail! We can now plug this new bounding box object into osmdata::opq() and get all hiking routes in the forest.

## get hiking trails in all of Uwharrie National Forest
unf_trails_full <- opq(bbox = matrix(st_bbox(unf$osm_multipolygons), ncol = 2,
                                     dimnames = list(c('x', 'y'), c('min', 'max')))) %>% 
  add_osm_feature(key = 'route', value = 'hiking') %>% 
  osmdata_sf()

## plot
plot(unf_trails_full$osm_lines$geometry, col = 'coral4')

Now we’re getting a bunch of trails across the Pee Dee River in Morrow Mountain State Park. Again it’s easy to drop these extra trails later, so for the moment, more complete is better than less complete. These data come from OpenStreetMap, so they also include lots of usuable data. Let’s take a look at the fields included in our lines:

## inspect
glimpse(unf_trails_full$osm_lines)

## Rows: 106
## Columns: 37
## $ osm_id            <chr> "32024414", "216945232", "216945234", "216945241", …
## $ name              <chr> "Uwharrie Trail", "Mountain Loop Trail", "Mountain …
## $ alt_name          <chr> "Uwharrie National Recreation Trail", NA, NA, NA, N…
## $ bicycle           <chr> "no", "no", "no", "no", "no", "no", "no", "no", "no…
## $ bridge            <chr> NA, NA, "yes", "yes", NA, NA, "boardwalk", NA, NA, …
## $ construction      <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
## $ dog               <chr> NA, "leashed", "leashed", "leashed", "leashed", NA,…
## $ foot              <chr> "designated", "designated", "designated", "designat…
## $ footway           <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
## $ highway           <chr> "path", "path", "path", "path", "path", "path", "pa…
## $ horse             <chr> NA, "no", "no", "no", "no", "no", NA, NA, NA, NA, "…
## $ lanes             <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
## $ layer             <chr> NA, NA, "1", "1", NA, NA, "1", NA, NA, NA, NA, NA, …
## $ motor_vehicle     <chr> NA, "no", "no", "no", "no", "no", NA, NA, NA, NA, "…
## $ name_1            <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
## $ oneway            <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
## $ rcn_ref           <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
## $ sac_scale         <chr> NA, "mountain_hiking", "mountain_hiking", "mountain…
## $ service           <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, "parkin…
## $ smoothness        <chr> NA, "bad", "good", "good", "bad", NA, NA, NA, NA, N…
## $ source            <chr> NA, NA, NA, NA, NA, "GPS_2009", "GPS_2009", "GPS_20…
## $ surface           <chr> "dirt", "ground", "wood", "wood", "ground", "ground…
## $ symbol            <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, "wh…
## $ tiger.cfcc        <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
## $ tiger.county      <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
## $ tiger.name_base   <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
## $ tiger.name_base_1 <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
## $ tiger.name_type   <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
## $ tiger.reviewed    <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
## $ tiger.zip_left    <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
## $ tiger.zip_left_1  <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
## $ tiger.zip_right   <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
## $ tiger.zip_right_1 <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
## $ tracktype         <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
## $ trail_visibility  <chr> NA, "excellent", "excellent", "excellent", "excelle…
## $ wheelchair        <chr> NA, "no", "no", "no", "no", NA, NA, NA, NA, NA, "no…
## $ geometry          <LINESTRING [°]> LINESTRING (-80.0435 35.310..., LINESTRI…

We can use the “name” field to subset the data. If you were considering some parallel or spur trails, you could use sf::st_filter() in combination with `sf::st_is_within_distance() to instead just grab trails near your primary trail.

## extract OSM lines and filter
ut <- unf_trails_full$osm_lines %>% filter(name == 'Uwharrie Trail')

Now we’ve gotten the Uwharrie Trail twice. Once using a smaller bounding box and once using a larger one. We can plot them both and see if there were any segments the intial query missed

## plot
plot(ut$geometry, col = 'red')
plot(unf_trails$osm_lines$geometry, add = T, col = 'coral4')

Luckily the initial query still picked up every segment, but that won’t always be the case if you start with an inaccurate initial bounding box. If the entire Uwharrie Trail wasn’t collected into a relation, we might have missed large chunks of it on either end. Now we can use the bounding box for the Uwharrie Trail to capture any other features we care about nearby.

Water

The first other feature we need is water. On any multi-day trip, being able to refill your water is essential. The OSM wiki page on waterways shows us that they values we need to grab relevant water sources are river and stream. Although not well-documented, you can supply multiple value arguments to osmdata::opq() using c(). This will let us quickly and easily grab both rivers and streams in the area.²

## create bbox for just the Uwharrie Trail; no need for all water in the whole National Forest
ut_bb <- matrix(st_bbox(ut), ncol = 2, dimnames = list(c('x', 'y'), c('min', 'max')))

## get rivers and streams and extract OSM lines
ut_water <- opq(bbox = ut_bb) %>% 
  add_osm_feature(key = 'waterway', value = c('river', 'stream')) %>% 
  osmdata_sf()

Our next step will be to drop any water sources more than a kilometer from the trail. This will simplify our analysis later and also minimizes our environmental impact. To conduct GIS operations in meters, we need to project our data from latitude and longitude-based WGS84 to a meter-based coordiante reference system (CRS). The CRS database epsg.io shows that NAD83/North Carolina(EPSG:32119) is the projection for data in North Carolina, so we use sf::st_transform() along with sf::st_crs() to project our trail and water source objects. This lets us calculate distances in feet/meters rather than decimal degrees. We’ll use this to limit the water features to those that fall within 1km of the trail. This way we’re not limiting ourselves to only water features that directly intersect the trail, but we’re also not retaining a bunch of features that are farther off-trail than I like to hike for water.

## project trail
ut <- st_transform(ut, st_crs(32119))

## project water sources
ut_water <- ut_water$osm_lines %>% 
  st_transform(st_crs(32119)) %>% 
  st_filter(ut, .predicate = st_is_within_distance, dist = 1000)

## plot
plot(ut_water$geometry, col = 'lightblue')
plot(ut$geometry, add = T, col = 'coral4')

Roads

If we want to be near water, we want to be far from roads. OpenStreetMap has lots of different categories of roads, so we’ll want to capture all the major ones, as well as service roads and “tracks”, which is how OpenStreetMap refers to forest roads.³ OSM identifies roads with the key “highway,” and inspecting the OSM wiki page on roads shows us the various values we’ll need to grab all relevant roads.

## get roads, project, and limit to w/in 1000 m of trail
ut_roads <- opq(bbox = ut_bb) %>% 
  add_osm_feature(key = 'highway',
                  value = c('primary', 'secondary', 'tertiary', 'residential',
                            'unclassified', 'track', 'service')) %>% 
  osmdata_sf() %>% 
  magrittr::extract2('osm_lines') %>% 
  st_transform(st_crs(32119)) %>% 
  st_filter(ut, .predicate = st_is_within_distance, dist = 1000)

## plot
plot(ut_roads$geometry, col = 'black')
plot(ut$geometry, add = T, col = 'coral4')

Note the use of magrittr::extract2() to extract the osm_lines object from the osmdata_sf object returned by osmdata::osmdata_sf(). This is how you can access a list element in a pipeline, and is equivalent to $osm_lines.

Campsites

To locate potential campsites we need to identify our priorities and use them to define a set of rules for selecting potential sites. For this exercise, I’m using the following:

1. I’d like to be within 750 feet of a water source. Some (more hardcore) backpackers prefer to be farther away from water sources to minimize the chance of encountering animals. Since Uwharrie National Forest isn’t an area with heightened bear activity, I’m willing to trade the chance of a raccoon sniffing around my bear canister for a shorter walk to refill my water.

2. The US Forest Service requires that you camp at least 200 feet away from any water source. This is good practice everywhere, but it’s required in National Forests, so we want to make sure any potential campsites are at least 200 feet from any water features.

3. The Uwharrie Trail is a fairly heavily-trafficked trail, so I’d like to avoid going more than 1/4 mile off-trail to find a campsite. This will minimize the disturbance to the surrounding area.⁴ All of the semi-official campsites on the Uwharrie Trail are a good ways off the trail itself, so staying near the trail will contain my impact on a large scale, but minimize it locally.

4. If you’re not in a designated campsite, you should be at least 200 feet away from any trail. Again, this seeks to minimize your impact on the area by spreading out campsites over time.

5. If I’m making the effort to carry my shelter, sleep system, and food on my back, you better believe I don’t want to be hearing any cars at night. To try and minimize the chances of this happening, I want to be least 1,000 feet from any roads. The lower section of the trail skirts particularly close to a residential neighborhood, so this is an important consideration.

6. I’m going to drop any potential campsites smaller than 0.1 km^2. Choosing where to actually pitch your tent within a potential site area requires many considerations like drainage, wind exposure, and avoiding dead trees overhead. This means that we want to have ample space in which to find the ideal tent spot, so dropping small potential sites reduces the possibility of arriving at a spot and finding that there’s no good place for your tent.

With all of those factors in mind, we can now define our potential campsites and then narrow them down. I start by buffering the rivers and streams by 1,000 feet with sf::st_buffer(), which gives us every area within 1,000 feet of a water source. Then I move down my list of conditions, buffering the relevant feature and using sf::st_intersect() when I want to ensure I stay within a given distance of that feature and sf::st_difference() when I want to stay a given distance away from that feature.

Since NAD83 uses meters as the unit of measurement, we need to convert these distances in feet into meters. Again, the units package makes this easy with the units::set_units() function.

## buffer water 750 ft
campsites <- st_union(ut_water) %>% 
  st_buffer(dist = units::set_units(750, ft))

## buffer water 200 ft and subtract
campsites <- st_union(ut_water) %>% 
  st_buffer(dist = units::set_units(200, ft)) %>% 
  st_difference(x = campsites)

## buffer trail 1/4 mile and intersect
campsites <- st_union(ut) %>% 
  st_buffer(dist = units::set_units(.25, mi)) %>% 
  st_intersection(x = campsites)

## buffer trail 200 ft and subtract
campsites <- st_union(ut) %>% 
  st_buffer(dist = units::set_units(200, ft)) %>% 
  st_difference(x = campsites)

## buffer roads to 1000 ft and subtract
campsites <- st_union(ut_roads) %>% 
  st_buffer(dist = units::set_units(1000, ft)) %>% 
  st_difference(x = campsites)

## cast multipolygon to polygons and convert to sf
campsites <- campsites %>% 
  st_cast('POLYGON') %>% 
  st_sf() %>% 
  mutate(id = 1:n()) %>% # create ID variable
  filter(st_area(.) > units::set_units(.1, km^2)) # filter to > .1 sq km

The animation below shows each step in the process in order:

Elevation

So far we haven’t really done anything that you couldn’t do on CalTopo, albeit in a less programmatic way. Let’s change that by bringing in some elevation data. Elevation is important when hiking because it determines how many climbs your lungs will have to endure and how many descents your knees will. CalTopo has great built-in tools for generating elevation profiles and more detailed terrain statistics that can tell you what to expect along a given route. However, you can only calculate them for lines or polygons you’ve manually drawn.

While we could import the potential campsite polygons we’ve just generated into CalTopo and then calculate the terrain statistics, this has two major drawbacks. First, you have to point and click through generating the report for each polygon because there’s no way to batch process. Second, and more importantly, this would use a lot of processing power and computing time on CalTopo’s servers. If, unlike me, you have a paid subscription, you might feel less bad about this, but I’m trying not to take advantage of such an awesome service that CalTopo currently provides for free.

We can use R’s capabilities to handle raster data to solve both of these problems! The elevatr package lets you easily download elevation data in the form of a digital elevation model. These models combine multiple measurements from satellites to produce a single image of the earth where the brightness of each pixel represents the elevation of a given area. elevatr allows you to easily access elevation data compiled from a number of different data sources. The main function is elevatr::get_elev_raster(), which takes an sf object as its first argument and z, z zoom level of 1:14. We can also specify the clip = 'bbox' argument to crop the resulting raster to just the bounding box of our potential campsites, and not the entire tile they fall in.

library(raster)
library(elevatr)

## get elevation raster and clip to bbox
elev <- get_elev_raster(campsites, z = 13, clip = 'bbox')

## plot to inspect
plot(elev, col = grey(1:100/100))
plot(ut$geometry, add = T, col = 'coral4')

Since we can see that the highest point in the area is only about 300 feet above sea level, we don’t need to worry about absolute elevation when picking potential sites. Instead, we want to know how level these areas are; no one wants to wake up smushed against the downhill wall of their tent. We can use the raster::terrain() function to calculate the slope in each pixel.

## calculate slope
camp_slope <- terrain(elev, opt = 'slope', unit = 'degrees')

## plot slope
plot(camp_slope)
plot(ut$geometry, add = T, col = 'coral4')

All that’s left to do is aggregate slope measures to each polygon, and then calculate some sort of summary statistic to tell us how steep each potential site is overall. I’m going to use the median of each area’s slope rather than its average to avoid giving undue influence to outliers (if a .5 km² area is largely flat with a cliff at one edge, then it’s likely still a good candidate for a campsite). Let’s filter out all areas with a median slope of more than 10°.

## calculate median slope for each polygon and filter
campsites <- campsites %>% 
  mutate(med_slope = (raster::extract(camp_slope, ., fun = median, na.rm = T))) %>%
  filter(med_slope < 10) 

With that done, we can now plot our potential campsite locations and all the features used to define them:

## plot campsites and all features
plot(ut$geometry, type = 'n')
plot(campsites$geometry, add = T, col = 'lightgreen', border = NA)
plot(ut_water$geometry, add = T, col = 'lightblue')
plot(ut_roads$geometry, add = T, col = 'black')
plot(ut$geometry, add = T, col = 'coral4')

This is a pretty picture, but it’s not very useful. To make it so that we can actually navigate to any of these spots, we need to get them onto a topographic map.

Plan it

To make our map usable, all we have to do is export the potential campsite polygons from R so that we can import them into CalTopo. CalTopo supports a number of file formats for importing, but the one we want to use is GeoJSON. We can use the geojsonio package to easily convert our polygons from sf objects to GeoJSON format and then save them to disk to import into CalTopo.⁵

There are two (potentially) tricky things we need to do. First, make sure we reproject our NAD83 data back to decimal degree-based WGS84 so that CalTopo can properly reference them. Second, we want to take advantage of R’s capabilities to efficiently wrangle data and create a name field for our polygons so they’ll be easy to identify and reference once they’re in CalTopo. To do this, we need to create a “title” field in our sf object before we convert it to GeoJSON.⁶

library(geojsonio)

## create site number field; transmute b/c all fields other than label are lost on import
campsites %>% 
  transmute(title = str_c('Potential Site ', row_number())) %>% 
  st_transform(st_crs(4326)) %>% # project to WGS84
  geojson_json() %>% 
  geojson_write(file = 'campsites.json')

## export Uwharrie Trail to save the trouble of tracing it
ut %>% 
  st_transform(st_crs(4326)) %>% # project to WGS84
  geojson_json() %>% 
  geojson_write(file = 'trail.json')

At this point all that’s left to do is click the “Import” button in CalTopo and select your newly created .json file. You can check out the potential campsites live on CalTopo below:

Some closing thoughts viewing the potential sites in context on CalTopo:

• Potential Site 2 looks promising. It’s slightly downhill from the trail, and has some relatively flat ground. However, the water source is an intermittent stream (denoted by the three dots in the blue line), so depending on time of year there may not actually be easy access to water here.
• Potential Site 4 is located near both a perennial and an intermittent stream, so the odds of finding a usable water source are higher. Across the trail to the West you can see an area that meets all of our site selection criteria except gentle slopes due to the steep rise to the 795 foot peak nearby.
• Potential Site 7 demonstrates the limitations of this approach because there are two forest roads near it on the Forest Service map that aren’t included in OpenStreetMap. Google Maps shows that there’s a private RV campground here, so best to avoid it. Doubly so because it’s largely outside of Uwharrie National Forest’s boundaries (the green lines). This is why it’s important to check more than just the terrain before you go!