Engineering Data Engineers

Part 1 and Part 2 both compared data scientists to data analysts. But I’ve been neglecting the unsung heroes of the data world: data engineers. I’m not too familiar with the life of a data engineer. I imagine there’s some overlap with data scientists (Python, Hadoop, etc), but with a stronger emphasis on data infastructure (Spark, AWS, etc.). Coming from a position of complete ignorance, let’s see if we can use NLP to identify the skills that are specific to data engineers. As always, the full code can be found on github.

Data Collection

Similar to Part 1, we’ll extract all data engineer, data scientist and data analyst jobs in London from the Indeed API and then filter out all junior/senior positions and plot the advertised salaries for each job type.

## if you haven't already installed jobbR
# devtools::install_github("dashee87/jobbR")

## loading the packages we'll need
require(jobbR) # searching indeed API
require(dplyr) # data frame filtering/manipulation
require(rvest) # web scraping
require(stringr) # counting patterns within job descriptions
require(plotly) # interactive plots
require(ggplot2) # vanilla plots
require(tm) # text mining

##         job_type num_jobs
## 1 Data Scientist      210
## 2   Data Analyst      158
## 3  Data Engineer      103

The first thing to note is there are about half as many data engineers posts as there are data scientist posts. Data engineers appear to be paid more than data scientists (though the former is a small sample), with the lowly data analyst bringing up the rear. We’ll now turn our focus to the job description. Repeating the work in Part 2, we’ll plot the proportion of job descriptions that contain specific predefined skills.

Apologies for small text on the x-axis, click here for a better version.

tf-idf

In this post, we’ll attempt to isolate the skills that are more strongly associated with data engineers than data scientists/analysts. We want words that feature frequently in data engineer job descriptions, but rarely with other job types (called term frequency-inverse document frequency, or tf-idf for short).

Firstly, we’ll scrape the job descriptions. I’ve added a few gsub commands to filter out unwanted punctuation features (e.g. bullet points), which may not be detected by the filters within the tm package.

# scrape job description webpages
ds_job_descripts <- unlist(lapply(dataScientists$results.url, 
                                  function(x){read_html(x) %>% 
                                      html_nodes("#job_summary") %>% 
                                      html_text() %>% tolower() %>%
                                      gsub("\n|/"," ",.) %>%
                                      gsub("'|'","",.) %>%
                                      gsub("[^[:alnum:]///' ]", "", .)}))

da_job_descripts <- unlist(lapply(dataAnalysts$results.url, 
                                  function(x){read_html(x) %>% 
                                      html_nodes("#job_summary") %>% 
                                      html_text() %>% tolower()%>%
                                      gsub("\n|/"," ",.) %>%
                                      gsub("'|'","",.) %>%
                                      gsub("[^[:alnum:]///' ]", "", .)}))

de_job_descripts <- unlist(lapply(dataEngineers$results.url, 
                                  function(x){read_html(x) %>% 
                                      html_nodes("#job_summary") %>% 
                                      html_text() %>% tolower() %>%
                                      gsub("\n|/"," ",.) %>%
                                      gsub("'|'","",.) %>%
                                      gsub("[^[:alnum:]///' ]", "", .)}))

Our task consists of two parts:

1. Idenitfy words that commonly occur in data engineer job descriptions
2. Identify words that commonly occur in data engineer/scientist/analyst job descriptions.

Words that appear highly in the first group but lowly within the second represent skills and themes specific to data engineers. To quantify word frequency, we must convert the job description vectors into a text corpus (large structured set of texts). We remove common words (called stop words) that provide little informative power (e.g. ‘and’, ‘the’, ‘are’). We’ll actually build two seperate corpuses: one for the data engineer jobs descriptions alone (to calculate tf) and another for all of the job descriptions (to calculate idf)).

  de_corpus <- Corpus(VectorSource(de_job_descripts)) %>% 
  tm_map(function(x){
    removePunctuation(x, preserve_intra_word_dashes = TRUE)}) %>% 
  tm_map(stripWhitespace) %>% tm_map(removeWords,stopwords("english")) %>%
  tm_map(PlainTextDocument)
  
  all_corpus <- Corpus(VectorSource(c(de_job_descripts,
                                     da_job_descripts,ds_job_descripts))) %>% 
  tm_map(function(x){
    removePunctuation(x,preserve_intra_word_dashes = TRUE)}) %>% 
  tm_map(stripWhitespace) %>% tm_map(removeWords,stopwords("english")) %>%
  tm_map(PlainTextDocument)

Remember that we’re interested in the frequency of each term within the corpuses. We can easily convert the corpuses to term document matrices, where each row corresponds to an individual term and each column refers to a different job description and the value is simply the number of the times the term appeared in that job description (which is then converted to a binary).

de_tdm <- TermDocumentMatrix(de_corpus)
all_tdm <- TermDocumentMatrix(all_corpus)
de_df <- data.frame(word= row.names(de_tdm),
                 tf = rowSums(ifelse(as.matrix(de_tdm)>0,1,0)),
                 row.names = NULL, stringsAsFactors = FALSE)
all_df <- data.frame(word= row.names(all_tdm),
                 tf = rowSums(ifelse(as.matrix(all_tdm)>0,1,0)),
                 row.names = NULL, stringsAsFactors = FALSE)

# data engineer common words
de_df %>% arrange(-tf) %>% head

##         word  tf
## 1       data 103
## 2 experience  90
## 3   engineer  85
## 4       will  83
## 5    working  80
## 6       team  71

# all jobs common words
all_df %>% arrange(-tf) %>% head

##         word  tf
## 1       data 469
## 2 experience 412
## 3       will 386
## 4     skills 346
## 5       team 342
## 6       work 314

Taking the term frequency (tf) alone, unsurprisingly, we see that ‘data’ and ‘engineer’ are two of the three most common words in data engineer job descriptions. The remaining terms are more generic, illustrated by their high ranking among all jobs. This demonstrates the importance of the inverse document frequency (idf) component. It will penalise terms such as ‘skills’, ‘team’ and ‘work’, as they’re not strongly associated with data engineers exclusively. We’ll normalise the tf score (divide by the max) and calculate the idf. The tf_idf is simply the product of the tf and idf.

de_df$tf = de_df$tf/max(de_df$tf)
de_idf <- data.frame(word=row.names(all_tdm),
               idf = log2(length(all_corpus)/rowSums(
                 ifelse(as.matrix(all_tdm)>0,1,0))),
               row.names = NULL, stringsAsFactors = FALSE)
de_df$tf_idf = de_df$tf * de_idf[match(de_df$word,de_idf$word),]$idf
knitr::kable(de_df %>% inner_join(de_idf,by=c("word"="word")) %>%
               arrange(-tf_idf) %>% mutate(rank=row_number()) %>%
               select(rank,word,tf,idf,tf_idf)  %>% head(40), digits=3)

It’s a good sanity check that ‘engineer’ returned the highest tf_idf score, as we’d expect that to be relatively specific to data engineer job descriptions. Also, it’s reassuring that the generic terms that previously scored well (e.g. ‘data’, ‘team’, ‘will’) are not in the table. The table provides some interesting insights. Take the example of ‘spark’: it has a relatively high tf, but is penalised by a low idf (spark is also a key skill among data scientists). ‘etl’, on the other hand, has a considerably lower tf, but outranks spark due to its higher idf (etl is a term more uniquely associated with data engineers).

It’s important to note that there is no strict defintion of either tf or idf. If you wish, you can attach more importance to either by applying a particular variant (a few examples here). I suppose it depends whether you think terms like ‘spark’ (high tf; low idf) should rank more highly than terms like ‘etl’ (low tf; high idf).

Summary

After some exploratory analysis, we used term frequency-inverse document frequency to idenitfy words and skills that are uniquely associated with data engineers. Think of the output as potential conversation starters with your engineer counterparts. “So… how about that etl?”