Earlier today I tweeted out some yield curve charts. I won’t go into great detail into the why, but here I will share some R code to make the charts. My Thread: hope you all are ready for some crazy yield curve charts, cuz you're about to get some crazy yield curve charts — Leonard Kiefer (@lenkiefer) August 7, 2019 We can get Treasury yield curve data from the U.
I have been exploring some visualizations for housing seasonality. In recent days I’ve tried out various ways of using tile plots to display seasonal patterns in home sales and other related data. In this post I want to share some of the R code I used to wrangle data and generate those plots. You can see some for example, in this thread (and others): though outside there's blistering heat
Let us take a look at house price trends in the United States and across states and metro areas. Earlier this week I tweeted out a few charts on housing market trends. In most of the middle part of the country over the past 44 years there has been little growth in real (inflation-adjusted) house prices. In coastal states, a very different story. pic.twitter.com/PLbiNftha3 — Leonard Kiefer (@lenkiefer) July 10, 2019 In this post we’ll analyze real house prices since 1975, and per usual use R to wrangle data and make plots.
If I cannot send Adam Ozimek (at Modeled Behavior ) a Diet Pepsi, then the next best thing might be a chart about epop. epop is the term economists use to describe the employment-to-population ratio, a useful summary statistic about the labor market. Perhaps the summary statistic. Adam (and others) has been talking about epop as a key labor market statistic for years. It seems the Federal Reserve is catching on to the usage of the term epop (though many economists over there have been looking at the statistic for a long while too).
Last week I posted a long thread comparing trends in various housing market indicators over on Twitter: Assuming we aren't in recession right now, the current expansion will tie the 1990s expansion for longest in U.S. history. Let's take a look at how housing markets have behaved in this expansion relative to earlier ones a thread... pic.twitter.com/ovaiq3lsA8 — Leonard Kiefer (@lenkiefer) June 12, 2019 I followed it up with an article on LinkedIn with some more commentary The U.
Recently the U.S. Census Bureau released updated population estimates through 2018 for the United States, states, counties, and metropolitan statistical areas (MSA). Press release I tweeted out the following chart comparing house prices and state population dynamics. demographics are an important driver of #housing market trends. here's a comparison of growth in state population and nominal house prices since the year 2000 left to right: more people bottom to top: higher home prices pic.
The current economic expansion is set to enter its tenth year this summer. Assuming we make it to June, this will become the longest U.S. economic expansion in recorded history stretching back to the 19th century. But how is the housing market doing? After a decade of recovery housing market activity still has room for improvement, but trends in 2018 were negative. Home sales, housing construction and house price growth all declined in 2018.
Earlier this month I attended the National Association for Business Economics (NABE) annual policy conference in Washington D.C. LINK. One of the keynote speeches was by Alan Greenspan. During his remarks, Greenspan mentioned that while economic forecasting was hard demographic projections were the surest thing in an uncertain business. Demographics of course are not easy, but it’s much easier to guess what the population of 30 years olds will be in 5 years than it is the predict the unemployment rate or GDP in 5 years.
The U.S. housing market stalled out a bit in 2018 and we aren’t building enough homes to match demand. See my recent speech for details on what’s going on. Abbreviated version: in 2018 mortgage interest rates slowed housing activity, but demographic forces support housing demand and should provide a lift in years to come. Together with a recent moderation in mortgage rates there’s reason to be optimistic about housing market activity in 2019.
Here’s some R code for an animated chart of the U.S. prime working age (25-54) labor force participation rate. I tweeted it out last Friday: Labor force participation rate #dataviz made with #rstats #gganimate pic.twitter.com/uSICoLjbIf — Leonard Kiefer (@lenkiefer) February 1, 2019 We can go to the U.S. Bureau of Labor Statistics (BLS) webpage (https://www.bls.gov/) and get these data. For more details see my post Charting Jobs Friday with R.
Because reasons I’ve been interested in picking up some Python. But I like the Rstudio IDE, so it sure would be nice if I could just run Python from R. Fortunately, that’s possible using the reticulate package. Let’s give it a try. Our strategy will be to use R to do the data wrangling and then pass the data to Python to make a plot. Is this a good idea?
I really like R, but I love the R community. Since I’ve started using R intensively in the past couple of years, I’ve constantly been awed and inspired by all the amazing things that people are doing with R. The spirit of the open source community and people’s willingness to share their thoughts and code is fantastic. Many times in this space we’ve remixed different data visualizations with R, often relying on awesome new packages that others have developed.
At the start of the year, the R package gganimate hit CRAN. See this announcement blog post with some examples. In this space, I’ve shared several posts on animation see tags. But I haven’t been using gganimate. Instead, I took a more direct approach building the animations via loops and trying to tween directly if I wanted a smooth animation. This level of control is nice, but frankly the defaults in gganimate work better than many of my attempts to hand craft it.
It’s the time of the year where everybody is dusting off their crystal balls and peering into the future. There’s even still time to send out your “Winter is Coming” newsletter. Let’s take a step back and look at how forecasts of U.S. macro variables have evolved. Is forecasting still hard? Last year we looked at historical forecasts of economic conditions in the post forecasting is hard. Let’s update it.
My recent economic and housing market talks see for example here have been titled: “Will the U.S. housing market get back on track in 2019?”. My general conclusion has been cautiously optimistic. There is enough strength in the broader economy and enough of a tailwind from demographic forces to push the U.S. housing market to modest growth next year. I still think that’s true, but as I have said in my talks, risks are weighted to the downside.
U.S. housing markets have slowed down in 2018. Housing construction, which is still running well below both historical averages and what the U.S. currently needs to meet rising demand has stalled out this year. The current level of housing construction is close to the level we’ve seen in recession periods. And the historical comparison stretching back decades is comparing a nation with significantly fewer households. Total U.S. households for example, in 1970 were about 1/2 (63 million) of what they were in 2017 (126 million) FRED chart.
Supply and demand, isoquants, indifference, the lists goes on. Economists love curves. One attracting extra attention these days is the Phillips curve. Last week I was in Boston for the annual meeting of the National Association for Business Economics (NABE). The overall conference was quite good, and certainly one of the highlights was a lunchtime speech by Federal Reserve Chairman Jerome Powell. You can find the speech here (pdf).
In this post I want review some trends in U.S. housing supply and demand. Specifically I want to look at county level trends in population, housing supply (the total number of housing units) and house prices. We’ll uncover some interesting trends. Per usual we will make our graphics with R. Preparing the data required several steps that I will outline in a follow up post. For now we’ll just proceed with the data I’ve put together.
It’s been a while since I posted here. I’ve got some longer form things in the works, but let’s ease back into it. Let’s take a look at the latest Job Openings and Labor Turnover Survey (JOLTS) data via the U.S. Bureau of Labor Statistics. This post is an update of this post. Per usual we will make our graphics with R. First, let’s look at aggregate trends.
My studies involve a lot of data organized in space and across time. I look at housing data that usually captures activity around the United States, or sometimes the world, and almost always over time. In my data visualization explorations I like to study different ways to visualize trends across both space and time, often simultaneously. Let’s consider a couple here in this post. Per usual we will make our graphics with R.
I saw this fun bit of R code in a tweet by user aschinchon. df <- data.frame(x=0, y=0) for (i in 2:500) { df[i,1] <- df[i-1,1]+((0.98)^i)*cos(i) df[i,2] <- df[i-1,2]+((0.98)^i)*sin(i) } ggplot2::ggplot(df, aes(x,y)) + geom_polygon()+ theme_void()#rstats pic.twitter.com/cgNjyk405f — Antonio S. Chinchón (@aschinchon) August 16, 2018 Let’s remix it to make a fun animation. We’ll zoom in and out and have the colors shift. Click for R code
About a year ago I shared code for a dataviz with a progress bar. Let’s update that R code using gifski and tweener. The code below will generate this animated gif: Gif code. Click for details. # CPI VIz with progress bar---- # set up your directory mydir <- "PATH_TO_YOUR_DIRECTORY" # libraries ---- library(data.table) library(tidyverse) library(tweenr) library(gifski) library(ggridges) library(extrafont) library(scales) library(cowplot) # make plots ---- # Get data----- # CPILFESL is FRED mnemonic for CPI: All Items Less Food and Energy # https://fred.
Today was JOLTS Tuesday, when the U.S. Bureau of Labor Statistics releases updated data from the Job Openings and Labor Turnover Survey. I was talking about it earlier today, but before we get into that… If you care about dataviz check this out I saw this on Twitter today via Jon Schwabish. Link to a handy dataviz cheatsheet outlining Jon’s core dataviz principles. Prints out nicely on pdf. Back to the JOLTS.
Maybe you are of the opinion that charts should have their y axis extend all the way down to 0, even if the data live far away from zero. I’m not sure if that’s always the right thing to do. But if you are strict about this, how can you use the space? One thing I experimented with in my Mortgage rates in the 21st century post was filling the area under a line with progressively fainter area.
In this post I want to share updated plots comparing house price trends around the world. Or at least part of the world. Our view will be somewhat limited, based on data, but will at least allow us to see how U.S. house prices compare to a few other countries. The details behind these plots are explained in more detail in this post, but some of the images were lost due to my blog transition.
This tweet turned out to be popular: 👀house price trends👀 pic.twitter.com/JXB5P0H84A — Leonard Kiefer (@lenkiefer) August 1, 2018 It’s a remix of a chart we made here, though it uses a different index. In the earlier post, we used the FHFA house price index, but this one used the Case-Shiller Index, which was released today. Let me just post two gifs and then below will be the R code I used to create them.
Let’s pick up where we left off yesterday and do some more exploration with text mining. Like yesterday we’ll use the tidytext package for R. And we’ll lean heavily on Julie Silge and David Robinson’s Text Mining with R. Data We’ll turn again to the Federal Reserve for our text data. But today we’ll explore the Beige Book, which gathers anecdotal information on current economic conditions across the Federal Reserve Districts.
Textmining is an exciting topic. There is tremendous potential to gain insights from textual analysis. See for example Gentzko, Kelly and Taddy’s Text as Data. While text mining may be quite advanced in other fields, in finance and economics the application of these techniques is still in its infancy. In order to take advantage of text as data, economists and financial analysts need tools to help them. Fortunately, there is a great resource: Text Mining with R by Julia Silge (blog and on Twitter atjuliasilge) and David Robinson (blog and on Twitter atdrob).
Indications are that U.S. housing market activity in the middle part of 2018 has moderated. Home sales estimates for both new home sales and existing home sales declined on a seasonally adjusted basis in June relative to May. House price growth has also moderated recently. Some folks have gotten animated about the recent trends. I’m more sanguine about the recent data. Certainly a slowdown in housing market activity would be cause for concern.
I saw today, via Ropensci a blog post about a new package for making animated gifs with R called gifski now available on CRAN. Let’s adapt the code we shared last week to use the gifski package. See that post for additional details. If we run the R code below we’ll generate this animated plot: This plot shows the evolution of house prices in two states, California (CA) and Texas (TX) versus the United States (USA).
On Twitter Claus Wilke asks: Dear Lazyweb: Is there an accepted name for a plot showing a two-variable time series as a path in the x-y plane? #dataviz@Elijah_Meeks @albertocairo @lenkiefer @sharoz @dataandme pic.twitter.com/N8Edmf8qii — Claus Wilke (@ClausWilke) July 21, 2018 I call them connected scatterplots, and we’ve made a few here. See for example this post. But we can intensify things and make a plot like this: hey @ClausWilke why stop at a 2-d connected scatterplot* when you could go to 3-d
The Linear Probability Model (LPM) might be bad, but is it all bad? Let’s look at some conditions where the LPM might not be so bad. We’ll also look at some simple adjustments that might improve the performance of the LPM. We’ll also compare the LPM to some common alternatives. Setup Throughout most of this post, we’re going to consider a world where the LPM model is the true model.
I try not to use too much jargon (jargon monoxide can be deadly) on this blog. But I’ve got a bit of a technical term I’ve been using the describe U.S. residential construction: super-low. To be sure, housing construction has been grinding higher, but it’s been taking a while for activity to get back close to historical averages. Once you account for the larger population, which all else equal needs more housing units, the level of construction is quite low.
I think a lot about predicting/forecasting binary outcomes. Will the economy head into a recession next year? What’s the likelihood of a loan defaulting over the next few years? Will my followers on social media abandon me if I tweet about my lunch? One often maligned, but seemingly irresitable approach to modeling binary ourcomes is the Linear Probability Model (LPM). As is known going back to before I was born, the Linear Probability Model has some issues.
Let’s compare two charts. “Your chart”, or a chart that might come virtually unedited from spreadsheet software versus the chart your boss told you not to worry about: Your chart is perfectly serviceable and for a quick exploration might be perfectly fine. However, why routinely generate such charts if you have the ability to make something a bit more dynamic? Being able to produce more interesting charts might not be necessary, but it also probably doesn’t hurt.
In this post I want to share some observations on housing in the United States from 1980 to 2016, share some R code for data wrangling, and tri (no that’s not a typo, just a pun) out a visualization techniques. Let’s get to it. I’ve been carrying a running conversation with folks on Twitter regarding the U.S. housing market and its future. Much of that depends on the evolution of demographic forces.
In this post we will create some plots of house prices and incomes for the United States and individual states. We will also try out the bea.R package to get data from the U.S. Bureau of Economic Analysis. We’ll end up with something like this: Per usual we’ll do it with R and I’ll include code so you can follow along. Data We’re going to use two sources of data.
As an economist with a background in econometrics and forecasting I recognize that predictions are often (usually?) an exercise in futility. Forecasting, after all, is hard. While non-economists have great fun pointing this futility out, many critics miss out on why it’s so hard. There are at least two reasons why forecasting is hard. The first, the unknown future, is pretty well understood. Empirical regularities with much forecasting power in the social sciences are hard to come by and are rarely stable.
In the real world, when I give talks and use slides I am typically constrained in my aesthetic. Often I’m speaking at a work-related thing and we have a corporate template and color scheme. They serve us well and I’ve found restraint helps focus on the message. Usually I’m setting out to inform, so direct, repeatable and easy to follow are key. But I also like to explore new ideas and different themes on the side.
IN MY LINE OF WORK, (finance/economics) you see a lot of dual axis line charts. I am of the opinion that dual y axis charts are sort of evil. But in this post I’m going to make one. It’s for a totally legit reason though. Like in an earlier post we’ll make a graph similar to one I saw on xenographics. Xenographics gives examples of and links to “weird, but (sometimes) useful charts”.
TODAY WAS JOBS FRIDAY. LET’s create a couple plots to show the trend in employment growth. Each month the U.S. Bureau of Labor Statistics (BLS) releases its employment situation report. Let’s make a couple plots looking at trends in U.S. nonfarm payrolls. Per usual, let’s make a graph with R. Data We can easily get the data via the Saint Louis Federal Reserve’s FRED database. If you followed my post from back in April of last year you know what we can do if we combine FRED with the quantmod package.
IN THIS POST WE SHALL EXPLORE VALUE-SUPRESSING UNCERTAINTY PALETTES. One of my favorite new sites is xenographics that gives examples of and links to “weird, but (sometimes) useful charts”. The examples xenographics gives are undoubtedly interesting and might help inspire you if you’re looking for something new. One new (to me) graphic was something called Value-Suppressing Uncertainty Palettes (VSUP). See this research paper (pdf). VSUPs “allocate larger ranges of a visual channel when uncertainty is low, and smaller ranges when uncertainty is high”.
IN THIS POST I WANT TO SHARE A GRAPH looking at the length of economic expansions and recessions in the United State over time. Earlier today, Andrew Chamberlain (on Twitter), observed that at the end of this month the current economic expansion in the U.S. would be the second longest in history. Let’s explore. In the United States, the National Bureau of Economic Research (NBER) dates expansions and recessions. See for example http://www.
TIME FOR A FUN NEW MORTGAGE RATE CHART. This one: We’ll use R to plot a new visualization of mortgage rates. Let’s make it. Data As we did with our majestic mortgage rate plot post we’ll plot mortgage rates using the Freddie Mac Primary Mortgage Market Survey. We’ll get the mortgage rates data via the Saint Louis Federal Reserve’s FRED database. If you followed my post from back in April of last year you know what we can do if we combine FRED with the quantmod package.
I LIKE TO MAKE ANIMATIONS WITH R. Sometimes folks ask me how they add to understanding. They don’t always, but often, particularly when you are working with time series, I find they help visualize trends and understand the evolution of variables. I’ve written several posts on animation, see particularly this recent post on making a simple line plot and this post about improving animations with tweenr. Tweenr is a handy package that lets you interpolate data and make smooth animations.
LET’S PICK BACK UP where we left off and think about communicating forecast results. To help guide our thinking, let’s set up a little game. Basic setup Like last time we’re going to focus on a situation where a forecaster observes some information about the world and makes an announcement about a future binary outcome. A decision maker observes the forecaster’s announcement and takes a binary action. Then the outcome is realized and the forecaster receives a payoff.
LAST FRIDAY WAS JOBS FRIDAY, the day when the U.S. Bureau of Labor Statistics (BLS) releases its monthly employment situation report. This report is blanketed with media coverage and economist and financial analysts all over the world pay close attention to the report. The employment situation gives a read on trends in the world’s largest economy’s labor market. It also provides a clue about how monetary policy might unfold, affecting bond yields around the world.
Spring is nearly upon us, or at least we can hope. Let’s examine how housing activity typically rounds into shape as the weather warms up. We’ll make some fun plots with R. Seasonality in housing data Housing market activity in the United States is highly seasonal. Consider this animated plot. This plot shows U.S. new home sales. Often the data are presented seasonally adjusted, but this plot is for non seasonally adjusted data.
LET US TAKE A LOOK AT HOUSE PRICE AND EMPLOYMENT TRENDS. House prices in the Unitest States have been increasing at a rapid pace, about 7 percent on an annual basis. How does that relate to employment growth? And how do those trends vary by geography. Let’s take a look. Per usual, I will post R code and you can follow along. Data Following recent posts (see here and here for example), we will use the Freddie Mac House Price Index an Excel spreadsheet can be downloaded here.
THIS MORNING I SAW AN INTERESTING CHART OVER ON BLOOMBERG. In this post they compared recent 10-year Treasury yield movements with the Taper Tantrum in 2013. The chart you can see here was an area chart with overlapping line plots. I thought it would be a fun exercise to remix a similar chart with R. Eventually it will look like this: Let’s make our remix and try out a few alternative plots.
LET’S TAKE A LOOK AT RECENT U.S. macroeconomic trends by making a couple plots with R code. Since we’re going to be looking at U.S. macroeconomic data, the data we’ll need is available in the St. Louis Federal Reserve Bank Economic Database FRED. Below I’ll walk you through creating the plots. Where are we in the business cycle After the BLS released their employment situation report (the Jobs report) last week I tweeted out:
THE UNITED STATES IS NOT building enough homes to meet demand. Be sure to check out my upcoming presentation at Realtor University to learn more about whether or not this could mean a house price bubble. One reason often cited for low levels of construction is a lack of labor. How do construction trends compare to construction employment? Let’s take a look. Financial blogger Logan Mohtashami (Twitter, blog) tweeted out (he’s a power Twitter user so I’m not sure exactly when or where, so I probably saw it multiple places) an interesting observation on housing construction and employment (see blog post).
COME AND MAKE A MAJESTIC MORTGAGE RATE PLOT WITH ME. We’ll use R to plot a few visualizations of mortgage rates. I recently gave a number of talks about the economic outlook and housing. One point I like to make is that mortgage rates are low. I’ve shown this through a variety of visualizations. But one of my favorites looks like this: Let’s make it. Data We’ll plot mortgage rates using the Freddie Mac Primary Mortgage Market Survey.
HEY! HERE IS A VIDEO SHOWING HOUSE PRICE TRENDS around the United States. Earlier this year we looked at how to get the data and plot it using R. I made the video using the PowerPoint to .mp4 workflow I outlined here. Below I’ll review how to build this file. Get data We are going to use house price data from the publicly available Freddie Mac House Price Index.
LET’S TAKE A LOOK AT RECENT LABOR MARKET TRENDS IN THE UNITED STATES. Below we’ll plot labor market trends using the U.S. Bureau of Labor Statistics Job Openings and Labor Turnover Survey (JOLTS). Last year we looked at how to get the data and plot it using R. Look to that post for more details, though I’ll include all the code we need below at the bottom. A trilogy of plots First, let’s look at some static plots.
WE ARE LATE FOR HALLOWEEN, but let’s get out our broom and purrr as we tidy some statistical results. Today I had occasion to be reminded of competing risks and a handy statistical result on competing risks from A.P. Basu and J.K. Ghosh published in the Journal of Multivariate Analysis in 1978. The paper Identifiability of the multinormal and other distributions under competing risks model showed an analytical result on the distribution of a variable Z which is the minimum of two Gaussian (Normal) random variables.
BACK WE GO INTO THE VASTY DEEP. LAST TIME we introduced the idea of using dynamic model averaging to forecast recessions. I was so excited about the new approach that I didn’t take the time to break down what was going on with it. In this post we’ll look more closely at what’s happening with the dma packaged when we try to forecast recessions. Per usual we’ll do it with R and I’ll include code so you can follow along.
HERE THE LITERATURE IS VASTY DEEP. In this post we’ll dip our toes, every so slightly, into the dark waters of macroeconometric forecasting. I’ve been studying some techniques and want to try them out. I’m still at the learning and exploring stage, but let’s do it together. In this post we’ll conduct an exercise in forecasting U.S. recessions using several approaches. Per usual we’ll do it with R and I’ll include code so you can follow along.
IN THIS POST I WANT SHARE A METHOD FOR MAKING SMOOTH POWERPOINT ANIMATIONS USING R. Also see other posts in this series: Crafting a PowerPoint Presentation with R LINK PURRRTY PowerPoint with R LINK PURRRtier PowerPoint with R LINK Motivation Why would you want to do this? We’ve covered how to make an animated gif with R and that works pretty well. But there are a couple advantages with this approach.
WE ARE ON OUR WAY TOWARDS BUILDING a tidy PowerPoint workflow. In this post I want to build on my earlier posts (see here for an introduction and here for a more sophisticated approach) for building a PowerPoint presentation with R and try to make it even purrrtier. I saw that somebody shared my posts on reddit and I thought I would take a look at the comments. Folks on the internet are known for kindness and offering helpful advice right?
MORTGAGE RATES ARE LOW IN THE UNITED STATES. How low? Let’s take a look. We’ll use R to plot a few visualizations of mortgage rates. We’ll also try out some of the nice features in the tibbletime package that help when working with time series data. For more on using tibbletime see this post and this one on making animated plots. Since I was already called out for alleged chartcrimes, I’m going to go ahead and let loose here.
IN THIS POST I WANT TO SHARE SOME CODE TO CREATE AN ANIMATED CHART of housing starts. Per usual we’ll use R and we’ll also use the tibbletime package. Running the code below will generate: (see this post for more on animations with the R package tweenr) ##################################################################################### ## Load libraries ## ##################################################################################### library(tidyquant) library(tibbletime) library(tweenr) ##################################################################################### ## Get housing starts data ## ##################################################################################### df <- tq_get("HOUST1F",get="economic.data",from="1959-01-01") ##################################################################################### ## function for rolling windows ## ##################################################################################### mys <- function(win=12){ rolling_mean <- rollify(mean, window = win) #function creates rolling average based on win df %>%mutate(dy=rolling_mean(price), w=as.
IN THIS POST I WANT TO REVISIT FLEXDASHBOARDS. Back in January we made several Flexdashboards with R to display economic data. See my guide to building a flexdashboard for some examples. In this post, I want to use the tidyquant package to wrap some of the plots we made earlier into a flexdashboard. I’ll have more to say about this in the near future, but I just wanted to make a simple flexdashboard (partially to remind myself how to do it).
I HAVE BEEN EXPERIMENTING WITH A NEW WAY TO VISUALIZE DATA, a bivariate tilegridmap. When I get around to rolling out my tidyPowerPoint workflow we’re going to want something other than bars and lines to fill it up. A graph like this might be a fun option. We’ll build one, but first, just let me show you one I tweeted earlier today: bivariate #tilegridmap map anyone? pic.twitter.com/y3G5XExzoN — Leonard Kiefer (@lenkiefer) October 11, 2017 In this post, let’s go over how to make this plot with R.
WHILE I WAS TRYING TO MAKE TIME FOR TIBBLETIME yesterday I got distracted and made this plot: hey! here's a #dataviz: tile plot of U.S. state unemployment pic.twitter.com/vH8fSu63ve — Leonard Kiefer (@lenkiefer) October 10, 2017 In this post, let’s go over how to make this plot with R. And we’re going to make it quick. Setup In order to create a plot like this we’ll need several packages, including the tidyverse, geofacet and the tidyquant package.
WHAT TIME IS IT? TIME FOR TIBBLETIME! In this post, I’m going to take the tibbletime package out for a spin. Turns out this package is quite useful for things I tend to do. We’ll use the tibbletime package to write some R code to extend our ongoing analysis of trends in the U.S. mortgage market (see here for example). Davis Vaughan (on Twitter) one of the authors of the tibbletime package suggested I take a look:
TIME FOR ANOTHER DATA WRANGLING AND VISUALIZATION EXTRAVAGANZA. This time we are going to work hard to turn some big data into little data. That is, we’re going to work hard to aggregate several million loan level records into useful summary graphics to tell us about the U.S. mortgage market in 2016. I’ve been working on a lot of different ways to visualize trends in the mortgage market (see here and here for examples).
IN THIS POST I WANT TO GO THROUGH SOME EXAMPLES of using the purrr package for R. Now there are already some great examples of how to use purrr. If you’re brand new to purrr (like I was not long ago) probably start with Jenny Bryan’s Purrr tutorial then see R for Data Science and also this presentation from rstudioconf (pdf). You can also check out this curated collection via Mara Averick (on Twitter: dataandme ) for a bunch of examples.
IN THIS POST I WANT TO SHARE SOME R CODE to create charts of U.S. housing starts we studied last week. Get data We’ll use tidyquant (see e.g. this post for more) to go get our data from the St. Louis Federal Reserve Economic Database (FRED). We’ll also use cowplot to arrange multiple ggplot2 graphs on one page. Let’s load libraries and grab the data. ##################################################################################### ## Step 0: Load Libraries ## ##################################################################################### library(tidyquant) library(tidyverse) library(cowplot) library(lubridate) library(scales) library(ggridges) # replaces ggjoy ##################################################################################### ## Step 1: Prepare for data ## ##################################################################################### tickers=data.
LOOK I DON’T HAVE ANYTHING BAD TO SAY about PowerPoint. Others have said it (see for example Tufte and Harvard Business Review). It’s a tool and a fact of life for many of us. I am interested in making better PowerPoints. In this post we’ll use some R tools to generate a PowerPoint deck. OfficeR The package officer allows you to access and manipulate ‘Microsoft Word’ and ‘Microsoft PowerPoint’ documents from R.
LET’S WRANGLE SOME HOUSING DATA. We’ll try something different with how posts are organized. In the past I have generally mixed data wrangling, R code and graphs all in one post. Now I’m going to break it up. Posts like yesterday will just show some data and discuss it. Then, if the data wrangling or code is complicated enough I’ll follow up with another post with details. You’ll be able to find all my posts on data wrangling, under the data wrangling tag and R code under the R tag.
IN THIS POST I WANT TO REWORK SOME VISUALIZATIONS we’ve already made. Inspired by this compendium of clean graphics in R, I want to remake some of visualizations we made earlier this year in a low-key manner. Sometimes the visualizations we make here can be pretty intense. Sometimes they are quite intricate, or use animation, or have a lot going on with them. But sometimes you need a simple, low-key visualization to make your point.
IN THIS POST WE WILL STUDY FORECASTS OF US ECONOMIC CONDITIONS. Niels Bohr quipped: Prediction is very difficult, especially if it’s about the future. I’m a macroeconomist by training, and my day job sometimes requires me to forecast the future so I can relate. Predicting the future can be quite difficult. In this post, we’ll analyze forecasts of economic conditions from professional forecasters using R to wrangle the data and construct plots.
I LIKE TO MAKE ANIMATED GIFS with my data. But frankly, sometimes they can be confusing. I got a good suggestion via twitter in response to a gif I shared there: This is very cool. Has anyone added a visual “progress bar” to a gganimate loop to show the frame visually? It's hard to grasp time as text. — Jonathan Spring (@JustTheSpring) August 6, 2017 Now it’s very likely that others have tried this, but I haven’t seen it.
IT IS TIME FOR AN UPDATE ON HOUSE PRICE TRENDS AROUND THE UNITED STATES. I have been experimenting with some new visualizations and updating some old favorites. Let’s collect them here. This post will be an extension of my Visual Meditations on House Prices series from last year. Check out those posts for additional visualizations. Data We’ll use the recently updated Freddie Mac House Price Index (link to source) data and use R to create some plots.
I LOOK AT A LOT OF DATA and the thing about data is it’s not always what it seems to be. A lot of data are uncertain, and based on estimates. We’ve talked about this before. See for example, this post on visualizing uncertainty in housing data. In this post I’m going to combine one of my favorite new plot types, the joyplot (see this post) with another of my favorite plot types: the beeswarm plot (see this post for more on beeswarm plots).
I TOOK SOME TIME OFF OVER THE SUMMER, away from data visualizations. It’s good to get away from time to time, but oh boy did I miss out. I wasn’t gone long, but in the short time I was gone people came up with some wonderful things. Let me dive back into it with some joyful dataviz. Joy plots Claus Wilke (Twitter) authored a new R package for creating joy plots ( LINK for ggjoy vignette).
LET US FOLLOW UP ON YESTERDAY’S POST with some more analysis of housing affordability. Per usual, we’ll use R to generate the plots and I’ll share the code below. Measuring affordability First, let’s talk a little bit more about what we are seeing in the plots. What are we measuring? Affordability metrics are often based on market level summary statistics. In our case we are looking at various ways to measure housing costs at a market level.
## 'getSymbols' currently uses auto.assign=TRUE by default, but will ## use auto.assign=FALSE in 0.5-0. You will still be able to use ## 'loadSymbols' to automatically load data. getOption("getSymbols.env") ## and getOption("getSymbols.auto.assign") will still be checked for ## alternate defaults. ## ## This message is shown once per session and may be disabled by setting ## options("getSymbols.warning4.0"=FALSE). See ?getSymbols for details. I travel around the United States giving talks, usually updates on recent trends in housing and mortgage markets.
QUITE A LOT OF HOUSING DATA CAME OUT THIS WEEK. Let’s recap with some graphs. Mortgage rates back below 4 percent The 30-year fixed rate mortgage fell back below 4 percent this week. New home sales New home sales data was released and came in weaker than expected for April 2017. March has been an extremely strong number, so a decline was anticipated, but the drop was bigger than most expected.
SO WE HAVE BEEN PLOTTING A LOT OF INDEX VALUES LATELY. It’s been great. But you have questions. Great questions. I got an interesting response to my house price dot chart over Twitter regarding the house price index we were plotting. User [@chrisschnabel](https://twitter.com/chrisschnabel) wondered how the choice of starting point influenced how the house price dot chart looked. @lenkiefer This is a great viz, but conclusions will be drawn based on the date of the index.
TIME TO TRY OUT ANOTHER HOUSE PRICE VISUALIZATION. In this post we’ll try out a new way to visualize recent house price trends with R. Just this wekeend I saw a new package geofacet for organizing ggplot2 facets along a geographic grid. It allows use to construct a small multiple graph that roughly looks like the United States. (Thanks to [@yoniceedee](https://twitter.com/yoniceedee) for recommending geofacet). Let’s try it out using the same house price data we visualized recently.
LET’S TAKE A LOOK AT RECENT TRENDS IN CONSUMER PRICES AND HOUSEHOLD DEBT. Along the way we’ll refresh some visualizations of consumer prices (see here) and household debt (see here) we made last year, as well as think up some new ones. As usual we’ll use R to generate the plots and I’ll share the code below. But before we get into the details of constructing the charts, let’s just look at two plots to help set the stage.
IN THIS POST I WANT TO REVIEW RECENT EMPLOYMENT AND HOUSE PRICE TRENDS at the metropolitan statistical area. No R code here, but you can recreate the graphs we’ll explore today by following the code in this post. This week the U.S. Bureau of Labor Statistics (BLS) released updated metro employment data (LINK) and Freddie Mac released its Freddie Mac House Price Index for over 300 metro areas as well as the 50 states, the District of Columbia and the United States.
IN THIS POST I WANT TO EXTEND ON yesterday’s post and build an animated bivariate choropleth. We’ll use the same data as yesterday and create a combined scatterplot with bivariate choropleth map and animate it with R. Let’s get right to it. Load data We’ll follow from yesterday and load our data and do some manipulations. In order for this to work we’ll need data from three sources: House Price Index Data in a .
NOTE: After I posted this (like within 5 minutes) I found this post which also constructs bivariate chropleths in R. IN THIS POST I WANT TO REVISIT SOME MAPS I MADE LAST YEAR. At that time, I was using Tableau to create choropleth maps, but in this post I want to reimagine the maps and make them in R. Last year in this post we looked at the relationship between population growth and the growth in housing units from 2010 to 2015.
This post is everything you want it’s everything you need it’s every viz inside of you that you wish you could see it’s all the right viz at exactly the right time but it means nothing to you and you don’t know why LET US MAKE SOME HORIZON CHARTS. What is a horizon chart you ask? That’s exactly what I was thinking earlier this weekend. Well, not exactly.
TIME FOR ANOTHER DATAVIZ REMIX. Saw on Twitter that [@hrbrmstr](https://twitter.com/hrbrmstr) posted a remix of a Wall Street Journal visualization over at rud.is. The original WSJ article used pies of various size to compare recent store closings. As we usually do in this space, we’ll use R to create our plots. Let’s mix things up and go remix the remix. Pies But first let’s consider the original. I’m not going to copy the original from the WSJ (click the link above to check out the story), but I am going to make my own pie version.
IN THIS POST I WANT TO SHARE SOME R data wrangling strategy and use it to prepare an update to some global house price plots I shared last year. In last year’s post I did some data manipulation by hand and mouse in Excel before getting into R. In this post I’m going to use the newly updated readxl library to do the data manipulations entirely in R. If you follow along, then you should be able to use this code to recreate my graphs.
LET’S PIXELATE AMERICA. This morning I happened across a fun blog post on how to generate Pixel maps with R via R weekly. The basic code is so easy, all you need is ggplot2 (which I get from the tidyverse). library(tidyverse) ## Warning: package 'tidyverse' was built under R version 3.5.1 ## -- Attaching packages -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- tidyverse 1.2.1 -- ## v ggplot2 3.0.0 v purrr 0.2.5 ## v tibble 1.
IN THIS POST I AM going to share some useful code to create some custom plots using the St Louis Federal Reserve Economic Database (FRED). While the FRED page has some nice chart customization options, I’m going to import the data into R with the quantmod package and draw the plots. I find myself doing these types of things often enough that I thought you might find these bits o’ code useful.
TODAY I MADE A KIND OF SILLY DATAVIZ, a tile plot of weekly changes in mortgage rates. A colleague happened by my viz terminal, pointed at my monitor and asked “what is that, a QR code?” Nope, it was a tile plot. We’re going to make the plot with R. As before, we’ll use data we used in our mortgage rate post to explore weekly average mortgage rates in the United States based on Freddie Mac’s Primary Mortgage Market Survey.
THIS PAST MONTH HAS BEEN BUSY. People have been traveling, I’ve been traveling, kids have been sick, and we’ve had the March Madness basketball keeping me occupied. Today I wanted to just explore a little analysis I’ve put together on resampling. Because reasons I’ve recently been interested in sample sizes and how quickly certain estimates might converge. There is of course, a vast literature on this topic. But armed with powerful computers maybe we can avoid too much mathy work and try to simulate our way through some problems.
LET US BUILD ON YESTERDAY’S POST (LINK) and construct more VISUALIZATIONS of house prices. In this post, I’ll include some R code so you can play along. We are going to construct our own GRAND TOUR (Wikipedia) except instead of touring European antiquity, we will explore recent trends in house prices around the continental United States. But we will perhaps still pick up some culture, or at least a new ggplot2 theme.
IN LATE 2016 HOUSE PRICES recovered back to their pre-recession peak. At least nationally. At least not adjusted for inflation. Let’s talk about it. National trends The chart below shows the Freddie Mac House Price Index (link to source) for the United States from December 2000 to December 2016. Prior to the Great Recession, house prices reached their seasonally-adjusted peak in March of 2007. Prices fell from 2007 (with a brief interruption due to the first time homebuyer tax credit) to December 2011 when they reached their post-recession minimum.
LET US EXPERIMENT A BIT WITH AXES. In this post I’m going to try out some data visualization ideas expanding on our earlier work with ticks marks (see post ticks out). We’re going to make the following plot and some variations with R. As before, we’ll use data we used in our mortgage rate post to explore weekly average mortgage rates in the United States based on Freddie Mac’s Primary Mortgage Market Survey.
TODAY THE NATIONAL ASSOCIATION OF REALTORS (NAR) released (press release) data on metro area median sales prices of existing single-family homes (the U.S. Census and HUD report data on new home sales prices in a joint release). NAR makes the data available (Excel file). Let’s take a look at the data: ## Warning: Removed 1 rows containing missing values (position_stack). ## Warning: Removed 1 rows containing missing values (geom_text). ## Warning in grid.
YOU HAVE SPOKEN and we will go with ticks out, at least 54% of the time. In a graph, should axis ticks face in or out? — Leonard Kiefer (@lenkiefer) February 5, 2017 To celebrate, let’s make an animated gif where the axis expands over time. We’ll use data we used in our mortgage rate post. Let’s start by loading the data and plotting a static graph (with ticks out of course).
I GIVE A LOT OF TALKS. Some are formal presentations or keynotes to large groups, while many are in small group settings. Sometimes I get impromptu requests so I have to be ready pretty much at all times to give some sort of talk. On this blog I’ve shared many different DATA VISUALIZATIONS which could be part of a talk. Many of the more complex visualizations-probably most of the animated gifs-wouldn’t work great in most presentation settings.
We will get back to house prices soon. IN THIS POST I WANT TO EXPLORE EMPLOYMENT TRENDS at the state and metro area. Today the U.S. Bureau of Labor Statistics (BLS) released data on state and metro area employment trends. Last month we looked at unemployment trends. Today we’ll look at trends in nonfarm employment. Wrangling the data We will be importing, preparing, and plotting our data with R. We can get the data pretty easily using the files BLS prepares, though we have to do a little bit of work to organize the data.
Animations with tweenr IN THIS POST WE ARE GOING TO CREATE TWO SIMPLE animated data visualizations using R ggplot2, animation, and tweenr packages. See this post about tweenr for an introduction to tweenr, and more examples here and here. Unlike those examples we are going to create visualizations using the data that ship with the tidyverse packages (mainly ggplot2). This will allow for easier replication. A simple stripped down example.
Introduction ON FRIDAY I MADE SOME ACCIDENTAL data art. I ended up with something pretty much useless, but kind of pretty. I shared it on Twitter: made an unintentional sorta streamgraph #graphfail pic.twitter.com/uW4IOnbrcL — Leonard Kiefer (@lenkiefer) December 1, 2016 We’ll see if we can redeem this graphfail in another Visual Meditation on house prices. Visual Meditations I’ve been collecting various graphical thoughts about house prices in my Visual Meditations series.
Introduction HOUSE PRICES HAVE NOW RECOVERED BACK TO THEIR PRE-RECESSION PEAK, at least according to some indices. The Freddie Mac House Price Index, for example, surpassed its pre-2008 peak in the latest release for data through September 2016. In this post I’ll be exploring trends in house prices and exploring different ways of showing how far house prices have come, and in some cases, how far they still have to go.
IN THIS POST I WANT TO CREATE some data visualizations with R using the recently released Home Mortgage Disclosure Act (HMDA) data. For this post I’m going to return to the 2015 HMDA that you can get from the Consumer Financial Protection Bureau (CFPB) webpage and I discussed earlier. Check out my prior post for more discussion of how we build these data visualizations. R code for graphs posted below
IT WAS A BUSY WEEK FOR ECONOMIC AND HOUSING DATA this week. Below are some data visaulizations I made tracking key trends in economic and housing market data. Homeowner equity increases to $12.7 Trillion in the second quarter of 2016 With house prices rising by nearly 6 percent on a year-over-year basis, homeowners are building back equity. According to the Federal Reserve’s Flow of Funds, owners’ equity in real estate was $12.
IN MY PREVIOUS POST we looked at the Job Openings and Labor Turnover Survey (JOLTS) data and plotted a Beveridge Curve. In this post I want to add some more code that allows us to plot Beveridge Curves by industry. For more on the analysis of industry-specific Beveridge Curves, see this paper published in the June 2012 Monthly Labor Review that decomposes shifts in the Beveridge Curve and looks at it by industry.
IN THIS POST WE’LL LOOK AT recent job openings and hires data from the Bureau of Labor Statistics Job Openings and Labor Turnover Survey (JOLTS). R code for selected graphs posted below Job openings and labor turnover Total nonfarm trends Let’s start by looking at aggregate national trends for total nonfarm sector. The plot below compares hires, job openings and separations (the sum of quits, layoffs and discharges, and other separations) over time.
.showopt { background-color: #004c93; color: #FFFFFF; width: 100px; height: 20px; text-align: center; vertical-align: middle !important; float: right; font-family: sans-serif; border-radius: 8px; } .showopt:hover { background-color: #dfe4f2; color: #004c93; } pre.plot { background-color: white !important; } EARLIER THIS WEEK THE U.S. BUREAU OF LABOR STATISTICS released data on consumer expenditures in 2015. In this post I want to examine these data and make a few visualizations. R code for graphs posted below
.col2 { columns: 2 200px; /* number of columns and width in pixels*/ -webkit-columns: 2 200px; /* chrome, safari */ -moz-columns: 2 200px; /* firefox */ } .col3 { columns: 3 100px; -webkit-columns: 3 100px; -moz-columns: 3 100px; } IN THIS POST WE’LL REVIEW some recent economic and housing market trends. R code for graphs posted below Low mortgage rates Mortgage rates remain low, with the 30-year fixed mortgage averaging 3.
WE ARE GOING TO EXAMINE THE DISTRIBUTION OF US POPULATION and make an animated gif combining a map and a kernel density estimate of the distribution of county population densities. Density of densities, or density squared. We are going to use the same US County Population Estimates 1790-2010 we used in my previous post. We’ll end up with this: How do we do it? Code First, we’ll load the data and do some manipulations.
SOMETIMES YOU ACTUALLY LEARN SOMETHING from social media. Today on Twitter I happened across this Tweet via @kyle_e_walker: Anybody want population estimates for every US county & metro back to 1790? Well, here are mine. Have fun!https://t.co/QqdA6226kN — Jonathan Schroeder (@j_p_schroeder) August 15, 2016 Seems somebody posted estimates of the U.S. population by county (defined by 2010 county definitions) going back to 1790. This is a perfect dataset to practice my mapping with R.
IN THIS POST I WANT TO DOCUMENT some R code I’ve recently been working on combining maps and distribution plots. As I discussed earlier lots of interesting data will be released in the fall and I want to be ready for it. Some of these snippets can be recycled when the new data is available. Maps One area of data visualization with R I haven’t explored much is mapping. Part of this reason is because I’ve had other tools to use, but usually it’s because I’m in a hurry.
AUGUST IS ALMOST OVER, and it’s nearly back to school season. And that means one thing. No, not that we’re about to get a chance to watch the #1 NCAA football program of all time dominate the gridiron (though that’s awesome too). No, it’s data release season! A data swarm is on its way. From American Community Survey to the American Housing Survey to the annual Home Mortgage Disclosure Act Data many statistical data releases come out in September and October.
OVER THE PAST THREE MONTHS I HAVE MADE several new house price visualizations. In these meditations I’ll consider some recent graphs and provide R code for them. For reference, prior meditations are available at: Part 1: data wrangling Part 2: sparklines and dots (animated) Part 3: bubbles and bounce Part 4: graph gallery Meditation 1: Median sales price trends Earlier this week, the National Association of Realtors (NAR) released their quarterly update on metro area median house prices (data here).
TODAY the NEW YORK FEDERAL RESERVE BANK released its Quarterly Report on Household Debt and Credit. These data come from the Center for Microeconomic Data based on credit records from Equifax. R code for the graphs are posted at bottom of page Trends in household debt balances One of the key statistics tracked in the report (full data can be found here) is household debt balances. They break debt balances out by loan type:
IN THIS POST WE’RE GOING to make an animated version of the famous Federal Reserve dot plot comparing the dots in March 2016 to June 2016. As far as celebrity charts go, the dot plot has to be up there. Check out how much “dot plot” has grown in search popularity since the Fed introduced their dot plot in 2012: Ultimately, it will look something like this: The dot plot is a special chart that shows the distribution of expectations of the Federal Open Market Committee (FOMC) for the federal funds rate.
A dataviz remix I came across some interesting analysis from Josh Lehner at the Oregon Office of Economic Analysis about the “Housing Trilemma”. The Housing Trilemma, captured in the chart below is based on the following claim: Every city wants to have a strong local economy, high quality of life and housing affordability for its residents. Unfortunately these three dimensions represent the Housing Trilemma. A city can achieve success on two but not all three at the same time.
IN THIS POST I WANT TO PROVIDE some additional examples of using tweenr and gganimate to create nice smooth transitions in an animated GIF. In this post we’ll look at an animated waterfall chart. For this data I’m going to use the National Income and Products Accounts from the U.S. Bureau of Economic Analysis (BEA). Specifically we’ll be looking at the contributions to growth in Real Gross Domstic Product, which you can find here.
.col2 { columns: 2 200px; /* number of columns and width in pixels*/ -webkit-columns: 2 200px; /* chrome, safari */ -moz-columns: 2 200px; /* firefox */ } .col3 { columns: 3 100px; -webkit-columns: 3 100px; -moz-columns: 3 100px; } RECENTLY I HAVE MADE several animated GIFs, primarily using the animation package. These gifs usually work pretty well, coming out something like this (code here). Unfortunately, these gifs tend to come out rather choppy.
EARLIER THIS WEEK THE U.S. CENSUS BUREAU released data on population and housing units for counties across the U.S. in 2015. These data reveal important trends in population growth, and help shed light on recent house price trends. Housing unit growth One key factor driving housing market dynamics is the expansion of housing supply (or lack thereof). The updated estimates from Census allow us to see which areas have added the most housing units and how that relates to population and house price trends.
Another mortgage rates animated gif IN THE PAST I’ve told you how I made my mortgage rates gif. In this post I’m make an extension that uses stop motion techniques to reverse course. We’ll end up with this: For reference, here’s the standard gif I share each Thursday after mortgage rates come out: Stop motion animation While thinking about the week-to-week movements in rates it’s easy to lose longer-term perspective.
We’re going to make this chart (and talk about it) Wait, what is this? Let’s pause the animation and look at the last frame: This plot shows the distribution of metro area unemployment. These data are available here. Each dot represents a metro area with its unemployment rate depicted on the x axis. The data are bucketed into 0.25 percentage point buckets and stacked when more than one metro falls within that range.