An important part of model building is the “proc eyeball” sanity check. It can also be a painful part of the process, when you are the data scientist tasked with creating and checking 10,000 or more near-identical plots. The
otvPlots package is designed to streamline this process.
otvPlots is an R package which takes a csv file as input and provides a pdf of VLM plots and csv files of summary statistics as output, optionally ordered so that any severely abnormal time series will be at the top of the pdf. The only strict requirement of the data scientist is to specify which column of the input data file contains the date variable.
otvPlots is efficiently implemented using
ggplot2 packages in R. Plots are automatically labeled if a variable dictionary is provided. Important variables can be given a highlighted label. A custom fuzzy matching algorithm can be provided by the user.
Discrete and numeric variables are handled automatically and given separate treatment. All binary variables are treated as categorical.
For each numerical variable, the output plots include * side-by-side boxplots (left), * a trace plot of p1, p50, and p99 percentiles, * a trace plot of mean and +-1 SD control limits, and * a trace plot of missing and zero rates (bottom right).
For each categorical variable (including a numerical variable with no more than 2 unique levels not including NA), the output plots include * a frequency bar plot (left), and * a grid of trace plots on categories’ proportions over time (right).
The order of variables in the CSV files is the same as in the PDF file. * A CSV file for numerical variables, including the number of observations (counts), p1, p25, p50, p75, and p99 quantiles, mean, SD, missing and zero rates. * A CSV file for categorical variables, including the number of observations (counts) and categories’ proportions. Each row is a category of a categorical (or binary) variable. The row whose
category == 'NA' corresponds to missing. Categories among the same variable are ordered by global prevalence in a descending order.
Open an R (or RStudio) console and run the following code:
devtoolspackage if not yet. You only need to do this once, so feel free to skip this step if the
devtoolsis already installed. You will be asked to select a CRAN mirror.
You can also build the package yourself by cloning the repo, setting your working directory to the otvPlots folder and running
devtools::build() in R, after installing the
Note that otvPlots does depend on R and several R packages to run. You can see a complete and up to date list of dependencies in the Imports field in the DESCRIPTION file.
Open an R console (or RStudio). Load the
otvPlots pacakge first (all its dependent packages should be loaded automatically).
The main function of the package is
vlm. Before execute this function, input data need to be prepared using the
PrepData function. Please check out the help files to see all options and many usage examples (highly recommended!)
bankData and its labels
bankLables are built-in datasets in the
After running the following code, a pdf file named “bank.pdf” and two csv files named “bank_numerical_summary.csv” and “bank_categorical_summary.csv” will be generated in the current working directory.
## Load the datasets data(bankData) data(bankLabels) ## Prepare data and labels bankData <- PrepData(bankData, dateNm = "date", dateGp = "months", dateGpBp = "quarters") bankLabels <- PrepLabels(bankLabels) ## Generate a pdf file of vlm plots, and csv files of summary statistics vlm(dataFl = bankData, dateNm = "date", labelFl = bankLabels, sortFn = "OrderByR2", dateGp = "months", dateGpBp = "quarters", outFl = "bank")
PrepData function only needs to be run once on a dataset. After that
vlm can be run directly with the argument
dataNeedPrep = FALSE (the default).
genCSV = FALSE.
vlm(dataFl = bankData, dateNm = "date", labelFl = bankLabels, genCSV = FALSE, sortFn = "OrderByR2", dateGp = "months", dateGpBp = "quarters", outFl = "bank2")
bankData[, weight := rnorm(.N, 1, .1)] bankData[, weight := weight / mean(weight)] vlm(dataFl = bankData, dateNm = "date", labelFl = bankLabels, dateGp = "months", dateGpBp = "quarters", weightNm = "weight", outFl = "bank3")
sortVars, but the
"date"column must be excluded from
sortVars <- sort(bankLabels[varCol!="date", varCol]) vlm(dataFl = bankData, dateNm = "date", labelFl = bankLabels, dateGp = "months", dateGpBp = "quarters", outFl = "bank4", sortVars = sortVars)
vlm(dataFl = bankData, dateNm = "date", labelFl = bankLabels, dateGp = "months", dateGpBp = "quarters", outFl = "bank5", varNms = "age", sortVars = NULL)
All examples for this package come from the Bank Marketing dataset available at the UCI Machine Learning Repository. The UCI repository maintains a free collection of datasets for researchers at its website.
Moro et al., S. Moro, P. Cortez, and P. Rita (2014). A Data-Driven Approach to Predict the Success of Bank Telemarketing. Decision Support Systems, Elsevier, 62:22-31, June 2014
Lichman, M. (2013). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science.
Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Contributors: We welcome your interest in Capital One’s Open Source Projects (the “Project”).
Any Contributor to the project must accept and sign a CLA indicating agreement to the license terms. Except for the license granted in this CLA to Capital One and to recipients of software distributed by Capital One, you reserve all right, title, and interest in and to your contributions; this CLA does not impact your rights to use your own contributions for any other purpose.
Link to Individual CLA
Link to Corporate CLA
This project adheres to the Open Source Code of Conduct. By participating, you are expected to honor this code.