Inflation and profitability in the construction industry. Rising costs of work in San Francisco.

13 min readAug 12, 2020

San Francisco is the world technological capital, whose example based on open data can be used to observe the development of the construction industry in megapolices. In this technologically advanced city, very pronounced economic cycles at different time intervals resulted in an explosive demand growth in various sectors of real estate.

Data on more than a million building permits (records in two datasets) acquired from the San Francisco Construction Department allow us to analyze not only the construction activity in the city, but also critically examine the latest trends and development history of the construction industry over the past 40 years, from 1980 to 2019 (section “Annual Construction Activity in San Francisco”).

The previous article looked at the total annual amount of construction (investment) in San Francisco between 1980 and 2018. The difference between the expected (estimated) and actual (revised) construction costs was used to track the dynamics of investor attitudes during periods of economic booms and crises in the region.

In this article, a closer look will be taken at individual construction sectors including roof, kitchen, stairs and bathroom repairs. After that, inflation for certain work types will be compared with official inflation data and other economic indices.

Content:

The ups and downs of construction sectors by type of housing
Average cost of kitchen and bathroom renovations in San Francisco.
Average cost of roof and stair repairs in San Francisco
The total cost of a planned home renovation
Increase in construction costs over the past 30 years
San Francisco Construction Cost Inflation
Real rates and profitability in the construction industry

Conclusions discussed in this article:

Inflation in construction is twice the official inflation in the country.
The demand for different types of housing varies from year to year and depends on different economic cycles
Exponential demand (a tenfold increase in demand) for the construction of apartments (flats) fell on the last high-tech boom (2016)
The peak in demand for retail space and the growth in construction volumes more than 16 times fell on the peak of dot-coms (2001).
The cost of roof and kitchen renovations has more than tripled over the past 30 years.
The cost of bathroom renovations increased 5-fold from 1980 to 2019.
The staircase business has hardly raised renovation prices for almost 30 years.
The price of planned repairs in 4 categories (roof, bath, kitchen, stairs) for a single-family house and for a two-family house differs by 15%.
For general repairs in 4 categories every 15–20 years — the cost of repairing the kitchen, bathroom, roof and stairs in a one-family house will be approximately $ 54,000, while for a two-family house this amount will be $ 61,000.
The San Francisco home-building business has shown smooth steady growth without hesitation for 30 years.
The business associated with the construction of retail space, offices and apartments is volatile and has numerous exponential (more than 10 times) ups and downs.
If you observe multiple growth in a short time, in the next two years you can expect the same rapid decline in intensity.
If you need to know how much the average repair price will rise, keep an eye on the rate on 10 year government bonds

Charts and calculations were built in the Jupyter Notebook (on the Kaggle.com platform).

San Francisco. Building sector 1980-2019.

Explore and run machine learning code with Kaggle Notebooks | Using data from multiple data sources

www.kaggle.com

The Ups and Downs of San Francisco’s Building Sectors by Housing Type

San Francisco’s hi-tech industry is one of the culprits behind the soaring prices. Hi-tech staff with six-figure salaries moving to the city to work at startups and more established companies (such as Google, Facebook, Twitter, and Apple) are dramatically increasing the cost of living and demand for housing. Programmers with pockets stuffed with “hi-tech” money can afford to outbid most of the locals in the real estate market.

At the same time, in terms of urban planning legislation, San Francisco is one of the most regulated cities in America. These and many other factors severely limit construction volumes and affect prices and demand in San Francisco.

Therefore, the leader in the demand for various housing types changes in each new economic cycle. Each housing type has had and will have its “ups and downs” in San Francisco construction market, such as a 10- time explosion in demand for apartments in a period from 2012 to 2015, or a hype demand (over 16- time growth ) for commercial real estate from 1997 to 1999.

In the article on the annual total amount of construction work, the price dynamics schedule for the period from 1980 to 2018 was divided into two components:

Estimated (estimated) cost of work (blue line)
Actual (revised) cost of work (yellow line)

Let’s take a deeper look at the total cost data and move on to the “next layer” of data.

We divide the total cost of work into categories of work according to the parameter “type of housing” (“Existing Use”):

Apartaments
Retail space
Office space
Residential real estate (houses)

dfn = df.dropna(subset=['description'])
dfn.description.isnull().values.any()
#dfn = dfn[dfn['description'].str.match('kitchen')]
df_unit = dfn.loc[:,['revised_cost','existing_use', 'existing_units', 'zipcode','permit_creation_date']]df_unit = df_unit.dropna()
#keys = ["hotel","appartments"]
df_unit = df_unit[df_unit.existing_use.str.contains("apartments")]#data_loc = df_unit.loc[['estimated_cost', 'revised_cost','permit_creation_date']]
data_cost = df_unit 
data_cost.permit_creation_date = pd.to_datetime(data_cost.permit_creation_date)
data_cost = data_cost.set_index('permit_creation_date')data_cost = data_cost[data_cost.index > "1985-8-01"] 
data_cost = data_cost[data_cost.index < "2019-8-31"]data_cost = data_cost.dropna()
data_cost_m = data_cost.groupby(pd.Grouper(freq='300d')).sum()
#data_cost_m.head()
plt.figure(figsize=(19,8))ax = sns.lineplot(data=data_cost_m.revised_cost, linewidth=3, size = 17)
ax.set(xlabel='retail')
major_ticks = np.arange(0, 1500000000, 200000000)
ax.set_yticks(major_ticks)
ax.set(ylim=(0, 1500000000))
plt.savefig('plotname.png', transparent=True)

It can be seen that all types of real estate in different economic periods went through a rapid parabolic (hype) growth and a similar rapid decline.

The peak of apartment construction took place during the latest high-tech boom, which was caused by the influx of large numbers of new hi-tech specialists from around the world to the fast-growing Silicon Valley firms. In period from 2012 to 2015 the demand for apartments increased 10 times (from $ 133 million in 2012 to $ 1.4 billion in 2015 invested in apartments alone).
The peak demand for retail, in its turn, occurred during the dotcom hype. From 1997 to 1999 the demand for retail real estate increased 16 times (from $ 22 million in 1997 to $ 350 million in 1999 invested in retail property alone). The collapse of the bubble led to the exodus of hi-tech staff from the city and the demand for retail property fell dramatically, returning to standard levels. At the same time, the latest technical boom did not affect retail trade in any way, and it is rather that the extra meters built in the late 90s satisfy the current demand for retail real estate.
The growth in demand for office real estate is also associated with the development of the Silicon Valley giants. However, here, since 2000, in contrast to retail real estate and apartments, there has been a steady growth in demand which is accompanied by numerous slight ups and downs.
In terms of growth dynamics the investment growth in residential real estate coincides with that in office real estate, but differs from the growth in demand for office real estate — smooth growth without large fluctuations in demand.

If we combine these 4 main categories in one graph, we get the general rise and fall diagram of all investments in construction in the city of San Francisco, familiar from the first article.

Average cost of kitchen and bath renovation in San Francisco

Taking data from the “Description” Feature again, we can additionally select data for individual categories of work and the average costs of repairing a kitchen or bathroom in San Francisco for various types of housing.

fam1 = df_unit[df_unit['existing_use']=='1 family dwelling']['estimated_cost'].mean()
fam2 = df_unit[df_unit['existing_use']=='2 family dwelling']['estimated_cost'].mean()
office = df_unit[df_unit['existing_use']=='office']['estimated_cost'].mean()
apartments = df_unit[df_unit['existing_use']=='apartments']['estimated_cost'].mean()
data = {'1 family dwelling':fam1,'2 family dwelling':fam2,'Apartments':apartments}
typedf = pd.DataFrame(data = data,index=['redevelopment of the bathroom'])
typedf.plot(kind='barh', title="Average estimated cost by type", figsize=(8,6));

The cost of renovating a kitchen in San Francisco is almost double the cost of renovating a bathroom. It makes sense that the average cost of a bathroom renovation is $ 2,000 more for a two-family dwelling ($ 16,000) than for a single-family dwelling ($ 14,000).

At the same time, the average cost of renovating a kitchen for a two-family home ($ 25,000) is almost $ 3,000 less than that for a single-family home ($ 28,000).

Average cost of roof and stair repairs in San Francisco

According to the same Description Feature, we select only the lines containing the words “reroofing” (roof replacement) and “stairs” (stairs repair).

Based on the average cost of roof repairs, it is reason able that roof repairs for a two-family house are on average $ 2,000 more than for single-family houses (due to the larger roof area of two-family houses).

The cost of repairing a staircase is also double for a two-family home, because a single-family home normally has no staircase (or a single-span staircase).

The cost of a planned home renovation in San Francisco

It is recommended to renovate the kitchen and bathroom on average once every 10–15 years. Roof and stair repairs should be done once every 15–20 years.

In general, if “theoretically” 15 years after the house construction the kitchen, bathroom, roof and stairs should be repaired in the same year, in a single-family house you will need to save up $ 54,000 for this purpose , whereas for a two-family house this amount will be $ 61,000. The difference in the total cost of work in these four categories is only 15%.

Thus, after the construction of a new house, in order to do house repairs in four categories (kitchen, bath, roof, stairs), it is necessary to save $ 350 monthly, in order to accumulate the required $ 60,000 for repairs in 15 years.

The rise in construction costs in San Francisco

By taking the data by work category and grouping it by year, we can observe the growth (and inflation) of the average cost of repairs by housing type.

years = list(range(1980, 2020)) 
keywords = ['1 family dwelling','2 family dwelling','apartments']
val_data = []
for year in years:
    iss_data = []
    for word in keywords:
        v = df_unit[(df_unit['existing_use']==word) & (df_unit['issued_date']== year)]['estimated_cost'].mean()
        iss_data.append(v)
    val_data.append(iss_data)
#print(val_data)

The following graph presents the data on the average cost by housing type, like in the previous paragraph, in a columnar form.

For visual display, the same graph, but in the form of columns, provides a clearer (“inflationary”) picture.

dfnew.plot.bar(figsize=(20, 8)) 
plt.xlabel("Years")
plt.ylabel("Estimated cost of reroofing")
plt.title("Estimated cost of reroofing by year");
dfnew.plot.line(figsize=(12, 6))

The average cost of roof repairs has been steadily increasing since 1990.

In contrast to residential houses, the average cost of renovating an apartment block roof in the same period went through numerous ups and downs.

Short-term 3-year cycles can be observed in the cost of renovating apartment block roofs.

In contrast to the gradual growth dynamics in the average cost of roof repairs, the average cost of kitchen renovation is more volatile.

In the renovation of kitchens, as well as in the renovation of apartment roofs, short-term 3-year cycles can be traced.

However, in the repair of bathrooms, such cycles are not traced, and the growth in the average construction cost is more gradual. It is only the increased average cost of bathtub repairs in apartments that can be observed before the dot-com boom?!

Inflation in the cost of construction work in San Francisco.

In order to find the inflation of the average cost of repairs for the entire period from 1980 to 2019, we supplement the data with a trend line. When calculating inflation (taking the start and end points of the trend line), we find that the maximum inflation in value for the period from 1990 to 2018 occurred in the bathroom sector.

sns.regplot(y=dfnew_2['2 family dwelling'],x=dfnew_2['index'],data=dfnew_2, fit_reg=True) 
#sns.jointplot(dfnew_2['index'], dfnew_2['2 family dwelling'], data=dfnew_2, fit_reg=True, stat_func=stats.pearsonr)
lines = plt.gca().lines
lower1990 = [line.get_ydata().min() for line in lines]
upper2019 = [line.get_ydata().max() for line in lines]
plt.scatter(1990, lower1990, marker='x', color='C3', zorder=3)
plt.scatter(2019, upper2019, marker='x', color='C3', zorder=3)
print("In 1990 it cost = $" + str(lower1990[0].round()) + "; In 2019 it cost = $ " + str(upper2019[0].round()))
print("Inflation for the period 1980-2019 = " + str(((upper2019[0]-lower1990[0])/lower1990[0]*100).round())+"%")
all2 = [line.get_ydata() for line in lines]

The average cost of bathroom renovation over the past 30 years has increased almost 5 times . The renovation cost has possibly increased due to the advent on the market of new finishing materials and expensive (though affordable) ceramics and sanitary ware.

The least deviations in terms of values are in the category “roof repairs”, where over the past 30 years inflation was 250% (the average price increased more than 3 times). The cost of kitchen renovation has also tripled over the past 30 years.

In the same period from 1980 to 2019, the cost of stair repairs remained almost unchanged and the average cost inflation in this area of construction was only 85%.

Now for greater clarity, let’s imagine inflation growth dynamics on a single scale, where inflation varies from 0 to 9%, and look at the fall in annual inflation cost of repairs by category in the period from 1980 to 2019.

It is noticeable that annual inflation over the past 30 years has decreased in all categories by almost 2–4 times (for example, in roof repairs from 8% in 1990 to almost 2% in 2019). This completely coincides with the economic policy during this period (from 1980 to 2019).

If we compare the official inflation data and the data on inflation in the construction sector, it will be evident that only in one sector the official inflation data corresponded to the inflation of work cost.

The rise in the cost of stairs repairs completely correlated with the official inflation. In the other work categories, the annual growth in the cost of construction work over the past 30 years has outpaced official inflation by almost 2 times.

Inflation dynamics in such categories as roof repair, bathroom and kitchen renovation, almost completely correlated with the dynamics of interest rates on 30-year loans (and, accordingly, with the yield on 10-year Treasury bonds).

A 30-year fixed mortgage is a loan, with the interest rate remaining the same throughout the loan term.

For example, for a 30-year mortgage of $ 300,000 with a 20% down payment and an interest rate of 3.75%, monthly payments would be about $ 1,111 (taxes and insurance excluded). Thus, the interest rate of 3.75% (and monthly payment) remains the same throughout the entire loan term.

The 10-year Treasury rate is profit earned from an investment in the US government-issued Treasury securities with a 10- year maturity.

Real rates and profitability in construction

It can be seen that the change in the cost of works corresponds to the interest rate on government bonds. The graph by Paul Schmelzing (a professor at Harvard University) shows how global real interest rates have changed over the past eight centuries.

Collecting data on real interest rates in advanced economies, Schmelzing shows that real indicators show a negative interest rate trend since the 14th century.

For comparison, the period considered in the article is marked in yellow.

The data from the report show how since 1311, average real rates have changed from 5.1% in the 1300s to an average of 2% in the 1900s.

The average real rate for the period 2000–2018 is 1.3%.

Undoubtedly, industries profitability also decreases along with the real rate, which correlates with this rate. Primarily it concerns such ancient (long-standing/traditional) industries as agrarian industry and construction industry.

In the period from 2020 to 2030, we are most likely to see new record lows in real rates and, accordingly, a decrease in profitability in the construction industry. But if profitability declines, this may mean that productivity will increase by the same “missing” percent.

While in construction business there used to be a large margin of 10–15%, and companies did not need to consider introduction of new technologies (which actually were few), now we are entering a new era of low real rates and low margins of 2–5% where the main role in a construction company will be played by the availability of new tools and processes in the company’s work.

Tools and new technologies that can already be used in construction are currently in surplus.

It will take construction companies decades for these new technologies to find their way into the heavy-handed and conservative construction industry.

Around the time when self-driving taxis start operating in San Francisco, construction companies will gradually replace lower-tier planners with automated scripts and tools that use big data and machine learning technologies to maintain margins.

Links to previous publications on this topic:

☕️ If you like my content, please consider buying me a coffee. Thank you for your support, I really appreciate it! buymeacoffee.com/boikoartem

📈 More about various tools for working with big data visualization here: