Introduction

THE GREAT THING ABOUT SOCIAL MEDIA is that it helps put me in contact with people from around the world. I started up a conversation with Twitter user @benmyers29 who shared some recent trends in Canadian housing markets.

@benmyers29 pointed out that homeowner equity in Canada was quite high, with only about 1% of Canadian mortgaged residential properties underwater (where the mortgage amount is greater than the property value so homeowner equity is negative) per this report referenced in the tweet. This was in contrast to a recent report from CoreLogic that estimates more than 6 percent of U.S. mortgaged residential properties were underwater.

Of course, the primary driver of this is going to be house prices, so I thought it would be helpful to look at some trends in international house prices. In this post I’ll share some charts I made (along with R code) to help visualize the trends.

Get data on house prices

First we’re going to need to gather some data on house prices. Fortunately, the Dallas Federal Reserve Bank has compiled statistics on international house price trends. Fed researchers have gone through the hard work of collecting data for many countries and harmonizing the series so they are more easily comparable. Read about their hard work and the details here.

The data are available in a convenient spreadsheet (2016Q2 data), and they even posted some R code to import it.

Affordability

Real house price trends don’t tell us bout affordability, whether or not average households can afford to buy homes. We also need to take into account incomes and interest rates. Fortunately the Dallas Fed database gives us estimates of disposable income. Unfortunately it doesn’t provide mortgage interest rates. Complicating things further, typical mortgage terms differ quite a bit by country.

Nevertheless, we can get some idea about how housing markets are evolving relative to the rest of the economy by comparing house prices to income.

Here we see that since 2005, house prices have outstripped personal income growth in Canada, while the opposite is true for the U.S. Because 2005 is so close to the peak of the U.S. housing cycle, it might make sense to re-index the variables. Here we re-index the data so that 1990 Q1 is equal to 100:

Now we can create a small multiple for each country in the database:

If we normalize the price-to-income ratio to be 100 at 1990 we can see how the ratio has evolved (how much space between the two lines).

And a strip chart:

Wrapping up

I’ve looked at trends in U.S. house prices see my series, but haven’t studied international trends as closely. In this post we took a look at some general trends in key housing market indicators from around the world. Let me leave you–for now–with this animated gif:

global hpi gif

Code for plots

R code for the plots in this article can be found below. I had to do some adjustments to the Dallas Fed spreadsheet to play nice. I deleted comments from the worksheets, removed the spacer row between country names and the first data rows, and I added a date column.

#load packages
library(data.table)
library(viridis)
library(tidyverse)
library(lubridate)

#load data files

hpi <- read_excel("data/dallas fed hp1602b.xlsx",sheet="HPI")
hpi$date<-as.Date(hpi$date, format="%m/%d/%Y")

rhpi <- read_excel("data/dallas fed hp1602b.xlsx",sheet="RHPI")
rhpi$date<-as.Date(rhpi$date, format="%m/%d/%Y")

pdi <- read_excel("data/dallas fed hp1602b.xlsx",sheet="PDI")
pdi$date<-as.Date(pdi$date, format="%m/%d/%Y")

rpdi <- read_excel("data/dallas fed hp1602b.xlsx",sheet="RPDI")
rpdi$date<-as.Date(rpdi$date, format="%m/%d/%Y")

# tidy data with gather()
pdi %>% gather(key=country,value=dpi ,-c(date,cycle)) %>% data.table()->pdi.dt
rpdi %>% gather(key=country,value=rpdi ,-c(date,cycle)) %>% data.table()->rpdi.dt
hpi  %>% gather(key=country,value=hpi ,-c(date,cycle)) %>% data.table()->hpi.dt
rhpi  %>% gather(key=country,value=rhpi ,-c(date,cycle)) %>% data.table()->rhpi.dt

# 1: panel plot

ggplot(data=hpi.dt[year(date)>2004 & 
                      !(country %in% c("Aggregate2"))],
       aes(x=date,y=hpi,color=country,label=country))+
  geom_line(size=1.1)+
  scale_x_date(breaks=seq(dlist[121],dlist[166]+years(1),"2 year"),
               date_labels="%y",limits=c(dlist[121],dlist[166]+years(1)))+
  facet_wrap(~country)+
  theme_minimal()+ geom_hline(yintercept=100,linetype=2)+
  scale_y_log10(breaks=seq(75,200,25),limits=c(60,225))+ theme_fivethirtyeight()+
  theme(legend.position="none",plot.caption=element_text(hjust=0,size=7),plot.subtitle=element_text(face="italic"),
        axis.text=element_text(size=7.5))+
  labs(x="",y="",title="Comparing house prices",
       caption="\n@lenkiefer Source: Dallas Federal Reserve International House Price Database: http://www.dallasfed.org/institute/houseprice/",
       subtitle="Seasonally adjusted index (2005=100, log scale)")

# 2: compare just Canada and US

ggplot(data=hpi.dt[year(date)>2004 & 
                      (country %in% c("Canada","US"))],
       aes(x=date,y=hpi,color=country,label=country,linetype=country))+
  geom_line(size=1.1)+
  scale_x_date(breaks=seq(dlist[121],dlist[166]+years(1),"2 year"),
               date_labels="%y",limits=c(dlist[121],dlist[166]+years(1)))+
 geom_text(data=hpi.dt[date==dlist[i] &
                            country %in% c("US","Canada")],
              hjust=0,nudge_x=30)+
  theme_minimal()+ geom_hline(yintercept=100,linetype=2)+
  scale_y_log10(breaks=seq(75,200,25),limits=c(60,225))+ theme_fivethirtyeight()+
  theme(legend.position="none",plot.caption=element_text(hjust=0,size=7),plot.subtitle=element_text(face="italic"),
        axis.text=element_text(size=7.5))+
  labs(x="",y="",title="Comparing house prices",
       caption="\n@lenkiefer Source: Dallas Federal Reserve International House Price Database: http://www.dallasfed.org/institute/houseprice/",
       subtitle="Seasonally adjusted index (2005=100, log scale)")

# 3: panel plot

ggplot(data=rhpi.dt[year(date)>2004 & 
                      !(country %in% c("Aggregate2"))],
       aes(x=date,y=rhpi,color=country,label=country))+
  geom_line(size=1.1)+
  scale_x_date(breaks=seq(dlist[121],dlist[166]+years(1),"2 year"),
               date_labels="%y",limits=c(dlist[121],dlist[166]+years(1)))+
  facet_wrap(~country)+
  theme_minimal()+ geom_hline(yintercept=100,linetype=2)+
  scale_y_log10(breaks=seq(75,200,25),limits=c(60,225))+ theme_fivethirtyeight()+
  theme(legend.position="none",plot.caption=element_text(hjust=0,size=7),plot.subtitle=element_text(face="italic"),
        axis.text=element_text(size=7.5))+
  labs(x="",y="",title="Comparing real house prices",
       caption="\n@lenkiefer Source: Dallas Federal Reserve International House Price Database: http://www.dallasfed.org/institute/houseprice/",
       subtitle="Seasonally adjusted index (2005=100, log scale)")


#compute 3 year (12-quarter) percent changes
rhpi.dt[,rhpa3:=(rhpi-shift(rhpi,12))/shift(rhpi,12),by=country]

# 4: real house price growth panel

ggplot(data=rhpi.dt[year(date)>2004 & 
                      !(country %in% c("Aggregate2"))],
       aes(x=date,y=rhpa3,color=country,label=country))+
  geom_line(size=1.1)+
  scale_x_date(breaks=seq(dlist[121],dlist[166]+years(1),"2 year"),
               date_labels="%y",limits=c(dlist[121],dlist[166]+years(1)))+
  facet_wrap(~country)+
  geom_hline(yintercept=0,linetype=2)+
  scale_y_continuous(labels=percent)+ 
  theme_fivethirtyeight()+
  theme(legend.position="none",plot.caption=element_text(hjust=0,size=7),plot.subtitle=element_text(face="italic"),
        axis.text=element_text(size=7.5))+
  labs(x="",y="",title="Real house price growth",
       caption="\n@lenkiefer Source: Dallas Federal Reserve International House Price Database: http://www.dallasfed.org/institute/houseprice/",
       subtitle="Percent change over 3 years in seasonally adjusted real index")

# 5: real hpa strip

ggplot(data=rhpi.dt[year(date)>2004 & 
                      !(country %in% c("Aggregate2"))],
       aes(x=date,y=rhpa3,color=country,label=country))+
  geom_line(size=1.1)+
  scale_x_date(breaks=seq(dlist[121],dlist[166]+years(1),"2 year"),
               date_labels="%y",limits=c(dlist[121],dlist[166]+years(1)))+
  facet_wrap(~country)+
  geom_hline(yintercept=0,linetype=2)+
  scale_y_continuous(labels=percent)+ 
  theme_fivethirtyeight()+
  theme(legend.position="none",plot.caption=element_text(hjust=0,size=7),plot.subtitle=element_text(face="italic"),
        axis.text=element_text(size=7.5))+
  labs(x="",y="",title="Real house price growth",
       caption="\n@lenkiefer Source: Dallas Federal Reserve International House Price Database: http://www.dallasfed.org/institute/houseprice/",
       subtitle="Percent change over 3 years in seasonally adjusted real index")

# 6:  Canada vs US real hpi and real income 1

ggplot(data=rhpi.dt[year(date)>2004 & 
                      (country %in% c("US","Canada"))],
       aes(x=date,y=rhpi,color=country,label=country))+
  geom_line(size=1.1,color=fivethirtyeight_pal()(2)[1])+
  scale_x_date(breaks=seq(dlist[121],dlist[166]+years(1),"2 year"),
               date_labels="%y",limits=c(dlist[121],dlist[166]+years(1)))+
  facet_wrap(~country)+
  geom_line(size=1.1,linetype=2,color=fivethirtyeight_pal()(2)[2],
            data=rpdi.dt[year(date)>2004 &
                                      (country %in% c("US","Canada"))],aes(y=rpdi))  +
  geom_text(data=rhpi.dt[date==max(rhpi.dt$date,na.rm=T) & country %in% c("US","Canada")],
              hjust=1,nudge_x=-30,label="House Price",
              color=fivethirtyeight_pal()(2)[1],fontface="bold")+
  geom_text(data=rpdi.dt[date==max(rpdi.dt$date,na.rm=T) &
                           country %in% c("US","Canada")],
            hjust=1,nudge_x=-30,nudge_y=.01,label="Income",aes(y=rpdi),
            color=fivethirtyeight_pal()(2)[2],fontface="bold")+
  geom_hline(yintercept=100,linetype=2)+
  scale_y_log10(breaks=seq(75,200,25),limits=c(60,225))+ theme_fivethirtyeight()+
  theme(legend.position="none",plot.caption=element_text(hjust=0,size=7),plot.subtitle=element_text(face="italic"),
        axis.text=element_text(size=7.5))+
  labs(x="",y="",title="Real house prices and disposable incomes",
       caption="\n@lenkiefer Source: Dallas Federal Reserve International House Price Database: http://www.dallasfed.org/institute/houseprice/",
       subtitle="Seasonally adjusted index (2005=100, log scale)")

# 7:  Canada vs US real hpi and real income 2

# re-index data so 1990Q1 = 1, first append 1990 values:
rhpi.dt[, rhpi.90:=sum(ifelse(date==as.Date("1990-03-01"),rhpi,0),na.rm=T),by=country]
rpdi.dt[, rpdi.90:=sum(ifelse(date==as.Date("1990-03-01"),rpdi,0),na.rm=T),by=country]

ggplot(data=rhpi.dt[year(date)>1989 & 
                      (country %in% c("US","Canada"))],
       aes(x=date,y=100*rhpi/rhpi.90,color=country,label=country))+
  geom_line(size=1.1,color=fivethirtyeight_pal()(2)[1])+
  scale_x_date(date_breaks="5 year", date_labels="%y")+
  facet_wrap(~country)+
  geom_line(size=1.1,linetype=2,color=fivethirtyeight_pal()(2)[2],
            data=rpdi.dt[year(date)>1989 &
                                      (country %in% c("US","Canada"))],aes(y=100*rpdi/rpdi.90))  +
  geom_text(data=rhpi.dt[date==max(rhpi.dt$date,na.rm=T) & country %in% c("Canada")],
              hjust=1,nudge_x=-30,label="House Price",
              color=fivethirtyeight_pal()(2)[1],fontface="bold")+
  geom_text(data=rpdi.dt[date==max(rpdi.dt$date,na.rm=T) &
                           country %in% c("Canada")],
            hjust=1,nudge_x=-30,nudge_y=-20,label="Income",aes(y=rpdi*100/rpdi.90),
            color=fivethirtyeight_pal()(2)[2],fontface="bold")+
  geom_hline(yintercept=100,linetype=2)+
  #scale_y_log10(breaks=seq(75,200,25),limits=c(60,225))+ 
  theme_fivethirtyeight()+
  theme(legend.position="none",plot.caption=element_text(hjust=0,size=7),plot.subtitle=element_text(face="italic"),
        axis.text=element_text(size=7.5))+
  labs(x="",y="",title="Real house prices and disposable incomes",
       caption="\n@lenkiefer Source: Dallas Federal Reserve International House Price Database: http://www.dallasfed.org/institute/houseprice/",
       subtitle="Seasonally adjusted index (1990Q1=100, log scale)")

# 8:  Panel real hpi and real income 2

ggplot(data=rhpi.dt[year(date)>1989 & country !="Croatia"], #drop Croatia as it is extreme outlier
       aes(x=date,y=100*rhpi/rhpi.90,color=country,label=country))+
  geom_line(size=1.1,color=fivethirtyeight_pal()(2)[1])+
  scale_x_date(date_breaks="5 year", date_labels="%y")+
  facet_wrap(~country)+
  geom_line(size=1.1,linetype=2,color=fivethirtyeight_pal()(2)[2],
            data=rpdi.dt[year(date)>1989& country !="Croatia" ],aes(y=100*rpdi/rpdi.90))  +
  geom_text(data=rhpi.dt[date==max(rhpi.dt$date,na.rm=T) & country %in% c("Australia")],
              hjust=1,nudge_x=-30,label="House Price",size=3,
              color=fivethirtyeight_pal()(2)[1],fontface="bold",aes(y=300))+
  geom_text(data=rpdi.dt[date==max(rpdi.dt$date,na.rm=T) &
                           country %in% c("Aggregate")],size=3,
            hjust=1,nudge_x=-30,label="Income",aes(y=200),
            color=fivethirtyeight_pal()(2)[2],fontface="bold")+
  geom_hline(yintercept=100,linetype=2)+
  scale_y_log10()+
  #scale_y_log10(breaks=seq(75,200,25),limits=c(60,225))+ 
  theme_fivethirtyeight()+
  theme(legend.position="none",plot.caption=element_text(hjust=0,size=7),plot.subtitle=element_text(face="italic"),
        axis.text=element_text(size=7.5))+
  labs(x="",y="",title="Real house prices and disposable incomes",
       caption="\n@lenkiefer Source: Dallas Federal Reserve International House Price Database: http://www.dallasfed.org/institute/houseprice/",
       subtitle="Seasonally adjusted index (1990Q1=100, log scale)")

# 9:  Panel of house price to income ratio

#merge income and house price data data

dt<-merge(rhpi.dt,rpdi.dt,by=c("date","country","cycle"))
dt[,ratio:=(rhpi/rhpi.90)/ (rpdi/rpdi.90)] # create ratio

ggplot(data=dt[year(date)>1989 & country !="Croatia"], #drop Croatia as it is extreme outlier
       aes(x=date,y=ratio,color=country,label=country))+
  geom_line(size=1.1,color=fivethirtyeight_pal()(2)[1])+
  scale_x_date(date_breaks="5 year", date_labels="%y")+
  facet_wrap(~country)+
  geom_hline(yintercept=1,linetype=2)+
  theme_fivethirtyeight()+
  theme(legend.position="none",plot.caption=element_text(hjust=0,size=7),plot.subtitle=element_text(face="italic"),
        axis.text=element_text(size=7.5))+
  labs(x="",y="",title="Ratio of real house prices to income",
       caption="\n@lenkiefer Source: Dallas Federal Reserve International House Price Database: http://www.dallasfed.org/institute/houseprice/",
       subtitle="Ratio: 1990Q1 = 100")

# 10: Another strip chart


ggplot(data=dt[year(date)>1989 & 
                  country %in% c("US","Japan","UK","France","Germany","New Zealand","S. Korea","Spain","Ireland","Italy","Canada","Australia")],
       aes(x=date,y="",color=ratio,fill=ratio,label=country))+
  geom_col()+
  scale_fill_viridis(name="Ratio",discrete=F,option="A")+
  scale_color_viridis(name="Ratio",discrete=F,option="A")+
  theme_fivethirtyeight()+
  theme(legend.position="right",legend.direction="vertical")+
  facet_wrap(~country,ncol=2)+
  theme(axis.ticks.y=element_blank(),
        axis.text.y=element_blank(),
        panel.grid.major=element_blank(),
        panel.grid.minor=element_blank(),
        axis.text.x=element_text(size=6))+
  labs(x="",y="",title="Ratio of real house prices to real incomes",
             subtitle="ratio normalized so 1990Q1 = 100",
       caption="\n@lenkiefer Source: Dallas Federal Reserve International House Price Database: http://www.dallasfed.org/institute/houseprice/")+
  scale_x_date(date_breaks="5 years",date_labels="%Y")+
  theme(plot.title=element_text(size=14))+
  theme(axis.text.x  =element_text(size=9))+
  theme(plot.subtitle  =element_text(face="italic",size=11))+
  theme(plot.caption=element_text(hjust=0,size=7))


# 11: animated gif

dlist<-unique(hpi.dt$date) #get dates 2005 starts in 121

# make function to plot time series for sected countries:
hpi.plot<-function(i){
  ggplot(data=hpi.dt[year(date)>2004 & 
                       date<=dlist[i] &
                       country %in% c("US","Canada","Australia","New Zealand","UK")],
         aes(x=date,y=hpi,color=country,linetype=country,label=country))+
    geom_line(size=1.1)+
    scale_x_date(breaks=seq(dlist[121],dlist[166]+years(1),"1 year"),
                 date_labels="%Y",limits=c(dlist[121],dlist[166]+years(1)))+
    geom_text(data=hpi.dt[date==dlist[i] &
                            country %in% c("US","Canada","Australia","New Zealand","UK")],
              hjust=0,nudge_x=30)+
    theme_minimal()+ geom_hline(yintercept=100,linetype=2)+
    scale_y_log10(breaks=seq(75,200,25),limits=c(75,200))+ theme_fivethirtyeight()+
    theme(legend.position="none",plot.caption=element_text(hjust=0),plot.subtitle=element_text(face="italic"))+
    labs(x="",y="",title="Comparing house prices",
         caption="\n@lenkiefer Source: Dallas Federal Reserve International House Price Database: http://www.dallasfed.org/institute/houseprice/",
         subtitle="Seasonally adjusted index (2005=100, log scale)")
}
  
library(animation)
oopt = ani.options(interval = 0.15)
saveGIF({for (i in 121:166) {
  g<-hpi.plot(i)
   
  print(g)
 print(i)
  ani.pause()
}
  
  for (i2 in 1:20) {
    print(g)
    ani.pause()
  }
},movie.name="hpi compare international.gif",ani.width = 750, ani.height = 400)