Lobbyist Wars and the Development of BIM. Part 6: Reasons for speculation in the construction industry. Corporate monopolies over data.

21 min readOct 4, 2021

The reason for the lack of productivity growth and the spread of speculation in the construction industry is the quality of the data used by the participants in the construction process. What is the main problem with data in construction? First of all, in the absence of trust and transparency in 3D-7D systems, which leads to the emergence of risks associated with the human factor and the creation of a multi-level bureaucracy in the main business processes of construction companies.To store data and transfer it between various construction participants, you need to trust CAD corporations or organizations that control the development of the IFC format. This situation can be compared to the early development of the Internet, when there was no strong encryption on the network and users had to rely on system administrators (third parties) to protect their data.

Today, when exchanging data between various 3D-7D systems, we entrust the storage of our data to corporations. To maintain their influence in the construction industry, corporations that do not benefit from the transparency and interoperability of data, monopolized the storage and processing of data. As a result, the suppliers of the main CAD and ERP solutions are constantly raising prices for using their products, and ordinary users are forced to pay a “commission” at each stage of data transfer in 3D-7D systems: for connection, import, export and handling the data that users themselves have created.

As a result of monopolies and lobbyists’ struggle for power over data, the construction industry has become an obsolete system with low productivity, where a certain percentage of speculation is taken as inevitable. To remove the tiered structure of mistrust, control, and pressure, construction companies need transparent business processes and new convenient data storage formats.

In separate parts, this material will be additionally published on the Medium platform and on a site dedicated to open data — opendatabim.com

Disclaimer

In the comments under the first articles in the Lobbying Games series (part 1 and part 2), I was accused of marketing work for Autodesk Corporation. In the comments under the latest articles (part 4 and part 5), on the contrary, employees of this corporation hinted at custom work against Autodesk.

For writing articles, I did not contact any of the companies listed in them. Representatives of the mentioned companies contacted me, asked me their questions and wrote suggestions and comments after the articles were published. I do not represent the interests of any of the companies, and absolutely everything written here is a personal opinion based on my experience and processed information.

Previous publications in the series Lobbyist Wars and BIM Development:

Part 1: Format STEP-IFC and how Revit conquered the CAD world

Part 2: open BIM VS closed BIM. Revit vs ArchiCAD and Europe VS the Rest of the World

Part 3: Fathers of BIM Technologies. Who is behind the success of Autodesk and openBIM

Part 4: The fight between CAD and BIM. Monopolies and lobbyists in the construction industry.

Part 5: BlackRock — the Master of All Technologies. How Corporations Control Open Source.

Struggle for data in construction

Reasons for the low productivity of the construction industry

The construction industry is one of the most highly regulated and fragmented industries in the world, where almost every construction project is a unique co-creation of a large number of different quality participants.

The life cycle of each construction project is influenced by a large number of specialists of different qualifications and different competencies: budgeting, architecture, statics, structures, estimates, procurement, control and operation. And each participant in the construction process has his own section of 3D-7D data, for which he is responsible.

For those who are engaged in architecture and design, a 3D model of the future project is important. For the person who will later oversee the processes on the construction site, what is primarily important is the timing of the 4D elements: the order in which the project elements are ordered, delivered and assembled on the construction site. For those who are engaged in purchases, estimates and budgeting, 5D information is important: equipment, materials, man-hours and product numbers hidden behind 3D elements in the project. The performance of the 6D-7D elements is of interest only to those professionals who will be involved in the operation of the buildings after construction is completed.

Due to the large number of business processes and data management systems used in construction projects, the competences of specialists from different areas (departments) overlap, leading to an increased impact of the human factor, duplication of information and the creation of internal bureaucracy.

The qualification of 3D-7D specialists and quality of information exchanged between different systems (CAD, MEP, ERP, CDE, ECM, CPM) ultimately have a direct impact on the quality and cost of the construction project, profit and cost effectiveness of the construction company.

Because of the closed proprietary data in the systems and the lack of transparency of business processes for all participants, construction projects become a quest to assemble pieces of 3D-7D information from engineers into one common puzzle.

The lack of data transparency and quality communication between individual specialists in the process of creating this puzzle has led the construction industry today to the status of one of the most inefficient industries in terms of productivity.

In the last 20 years of computer-aided CAD design, new construction technologies and materials, the productivity of the entire construction industry has grown by only 21%, while the total productivity of all sectors of the economy has grown by 70% (96% in the manufacturing industry).

The high-performance industries of today are the banking and trading industries, where engineers operate mainly with 1D number properties (contract amount) and 4D time properties (contract time).

The interaction between these data and business processes in this sector of the economy is almost entirely automated, and the human factor will be reduced to zero over the next few decades.

In addition to 1D and 4D properties, mechanical engineering industry specialists operate with 3D geometry data, 5D cost, and additional 6D and 7D operating parameters. The good relationship between the data developed in the 1990s and the high typology of elements using PLM (Product Lifecycle Management) platforms allows machine builders to mass produce the same product once modelled in 3D and configured in 4D-7D for years.

Successful industries have been able to remove the human factor from their processing of information. The focus of engineers in these industries is now on creating autonomous systems that operate without human intervention: autonomous driving, robotic complexes for machine production in the engineering industry and blockchain technology, smart contracts and DEFI (decentralised finance) in the banking industry.

In the construction world, the human factor remains a major productivity lag factor, with people of varying skill sets working with different formats of opaque data created in software from different software vendors.

Consequences of the lack of transparent interoperable data in construction

The main interest of today’s construction business, which has not yet gathered all the data into a single platform, is focused on the 4D time and cost data of the 5D elements in the project. This data is collected during the construction process of many projects throughout the life of a construction company and having this data is a major competitive advantage for any construction organisation. On this 4D-5D data, ERP (Excel) systems line up budgeting data (investment amount), material purchases and man-hours, work calculations and estimates of individual processes.

Until the mid-2000s, 4D-5D data could only be processed in expensive (tens or more often hundreds of times more expensive than CAD software) ERP systems (or multi-level Excel spreadsheets), and this data was almost completely unrelated to data, in the form of unrelated 2D drawings, from design departments.

The lack of automatic (geometrically verified) reliable volume data has created conditions for speculation and concealment of real data in 4D-5D systems. The main scheme for making money in construction is speculation on the prices of materials and works. Overpricing of works and materials occurs through “gray” accounting in internal closed 5D-systems (Excel, ERP, EPM) by adding a certain percentage on top of average market price of material or scope of work. Contractors resort to buying low-quality construction materials solely to save their money to the detriment of the client’s interests.

To start construction means to give yourself up for plunder.
Samuel Johnson (1709–1784)

Therefore, the departments of estimates, calculations and budgeting today are carefully protected from the access of unauthorized specialists, even within the construction company itself.

Without hiding their ‘grey’ 5D accounting, it is almost impossible to survive in the competitive construction market nowadays, and this opacity has been indirectly and directly created and maintained by the corporations that develop the systems in which this data is created and transmitted.

The modern trend towards 3D models and the emergence of automated data on item volumes forced managers of large companies to combine data from 3D models with data from the 5D classifier (estimate items) that already exists in ERP (Excel) systems. As a result, construction companies needed to create data interoperability and adjust the links between separate CAD, ERP, EPM, ECM systems specifically for their purposes.

The main source of Master Data in all 4D-7D information systems today is the volume data, created by modeling of project elements in CAD systems, where the information is transferred through proprietary closed formats (RVT, DGN, PLN, etc.) or through incomplete open format IFC. The opacity of the formats blocks the creation of exchange automation and makes it almost impossible to work together on a common project puzzle in the same interoperable data format.

Therefore, access to expensive full-cycle 3D-7D data processing systems and the quality and transparency of the data determine the construction cost of projects, profits and, as a consequence, the survival of construction companies in the construction market.

Due to the complexity of the data export and import process, this time-consuming process cannot be handled by simple specialists in 3D design and 5D estimating/costing departments. Only specialists who are familiar with a company’s internal business processes and at the same time have a good understanding of CAD and ERP systems can combine the CAD and ERP systems properly. Therefore, in the construction industry, unlike in banking or engineering, the number of managers responsible for the transfer of information between the various systems (CAD-ERP, FEM-CAD-HVAC, ERP-CPM, ERP-EPM) is increasing, in order to transfer and control “new” data.

In such a forcedly bureaucratised, multi-layered construction project management structure, only a multilevel control through additional managers ensures a quality construction result without lawsuits and financial losses. The consequence of this control has created excessive pressure at all levels of project processing, where engineers must import, transfer and export data in a timely and high-quality manner through closed proprietary products.

CAD corporations supplying the main 3D modelling solutions offer the construction industry a BIM concept where the exchange and control of information has to be handled by new age managers (BIM managers and BIM coordinators) who can extract data from CAD programs and obtain interoperability between different 3D-7D (CAD and ERP, EPM, ECM) systems.

CAD has borrowed the basis of this new CAD-BIM-ERP concept from the makers of MCAD solutions in the construction industry from the already proven CAD-BOM-PLM concept in the mechanical engineering sector.

Where did BIM come from and why?

By the early 2000s, the mechanical engineering industry already had advanced systems for handling data from MCAD software. Due to the competition among the many data handling systems that emerged in the 90s, PDM and PLM solutions became the main platforms, to which the information from CAx (Computer-aided technologies) software specialists: CAD, MCAD, CAID, CAPP, CIM, CAE, SCADA is now flocked.

The concept of BOMs (A Bill of Materials) plays an important role in data management in mechanical engineering. This is a bill of materials or specification where the top level BOMs represent a finished product, which may be a node or a complete component. Specifications describing assemblies are called modular BOMs. The use of BOMs for project management reduces the risks of errors and rework by eliminating the reuse of obsolete data.

BOM management is comprehensive product structure management, ensuring full digital associativity of design, supply chain, production, sales and service.

A significant contribution to these systems came from the automotive and mechanical engineering industries, which needed to link their design processes to 4D-7D production and operation processes with the advent of the first 3D CAD programs. The main role in these systems in the 1990s was played by data generated by users of PTC’s product Pro/ENGINEER, which by the mid-1990s had become the main design software for the entire engineering industry: its customers included BMW, Fiat, Ferrari, Toyota, Hyundai, PSA and Volkswagen, Caterpillar, John Deere and other major global industry players.

PTC’s Pro/ENGINEER programmers were the first CAD software to parametrically simulate solids and the first to conceptualise a single data model for the whole project, which revolutionised the MCAD (mechanical computer-aided design) market. Parametric feature-based modelling, derived from Pro/ENGINEER, has dominated the industry for a quarter of a century and all leading MCAD systems (CATIA, NX, SolidWorks, Inventor and Solid Edge) have become the ideological successors to Pro/Engineer.

Behind PTC and the Pro/ENGINEER product were mathematics professor Samuel Geisberg and his student Leonid Reitz of Leningrad State University (St Petersburg), who immigrated from the Soviet Union to America in the 1970s and 1980s (with new repatriation) following the Six Day War in 1967.

After leaving PTC, Leonid Reitz, who created the geometric core of Pro/Engineer in the late 1980s, created a new start-up in 1997, Revit, which took over the best from MCAD Pro/ENGINEER and aimed to satisfy the construction industry in parametric modelling with the modular concept of BOMs.

While machine builders were already working in PDM and PLM platforms, construction companies in the late 1990s had only 2D CAD solutions and separate 5D ERP solutions. The CAD data created mainly in Autodesk’s 2D product Autocad had almost no use for the 5D data that was created in the estimating, costing and budgeting departments.

Following the success of MCAD developers, Autodesk Corporation unsuccessfully tried to build its parametric modelling product on the existing Autocad product (AutoCAD Architectural Desktop) and with the help of a redesigned IFC format. After unsuccessful attempts to buy MCAD startup Solidworks (from former PTC engineers), Autodesk buys Revit startup from former PTC employee Leonid Reitz for $133 million in 2002.

Revit, with its mechanical engineering background, was a radically new CAD system in Autodesk’s portfolio, giving the corporation the opportunity to finally open up the subject of BOM-PDM-PLM-ERP to the entire construction industry.

To announce the arrival of a new era of data and new processes in the construction business, the Autodesk Vice Presidents have written the Whitepaper BIM. According to the logic of the new BOM-BIM concept, all information previously stored in different ERP systems should now be transferred to the CAD software (Revit), where new (mostly textual) properties will be added to the 3D model elements within the CAD software.

Ten years after the publication of the Whitepaper BIM, by 2010, Revit had swiftly taken over the CAD market worldwide (except for some European countries).

But since buying the Revit startup in 2002 for the next 10 years, Autodesk has unfortunately failed to develop the BIM concept into a turnkey 4D-7D data handling product or a working analogue to PTC’s PDM, PLM products.

Big corporations in the CAD market (Autodesk, Nemetschek, Bentley) have followed in the footsteps of the MCAD solutions of the 1990s and started working on multi-program Closed BIM environments in which the customer would not need third-party solutions.

At the same time, managers in design departments, having started to receive the first BIM-BOM “super data” (geometries, text properties of elements and specifications), are finding it necessary to link existing ERP and CAD systems using BIM-BOM data themselves, without waiting for solutions from Autodesk.

The mid-2010s are upon us and thousands of private investors from all over the world are trying to enter the BIM and PropTechs market (Property technology). Advanced managers in large construction companies (with programming skills) are going into start-ups, where they plan to completely replace outdated construction 5D ERP-ECM solutions in the already set-up business processes of construction companies by creating a data connection to CAD.

These days, thanks to the “gold rush” created by the growth of data in the construction industry, every day there are new proprietary (mostly community-driven and Open Source solutions — more on that in the next part) start-ups that want to work with data from CAD systems.

In 2020 alone, $19.9 billion was invested in PropTechs start-ups (by comparison, Autocad was sold to Autodesk for $1 and 10% from sales, and Revit for $133 million).

As a result, the lack of off-the-shelf solutions on the market and the growing volume of data in construction has led to unequal competition between large CAD corporations, construction companies and start-ups for the right to own and handle the data in $10 trillion of new construction contracts each year.

Formation of a monopoly in the data market

Unfortunately, the developers of any CAD-BIM-BOM-PDM-PLM-ERP solution face the Legacy code written back in the 90s, the core product (Archicad, Revit, Tekla Structures) where even the most experienced developers spend hundreds of hours testing and debugging one line of code.

Legacy code is Frankenstein’s monster code that still seems to work, but poorly, and new code with potential cannot develop it because of the mess in the legacy code base.

The problems of external services interacting with Legacy code bought in the 1990s are not much of a concern to large corporations who own the data and who are focused on building their own, “truly proven”, 3D-5D Closed BIM tools.

With transparent and interoperable access to data, all information from projects can be seamlessly transferred and automated using external services, databases and ERP systems. But then such a Closed BIM ecosystem would no longer make sense and 3D CAD solutions would be limited to geometry creation functions, which corporations and investment management funds cannot allow to happen.

In turn, corporations are forced to create such closed environments by the need to maintain stable quarterly earnings. The real masters of the big corporations producing new age oil — data — are the investors from the financial big five investment funds, who care about quarterly growth and cannot allow corporations to lose their monopoly on the extraction of valuable information.

Therefore, maintaining a monopoly over data is at the core of the policies of the big 3D solution vendors, allowing them to change licensing models and raise the prices of their products. The companies interviewed (from a letter from architects to Autodesk) claim that their costs for licensing Autodesk products increased by 70 per cent between 2015 and 2019 alone.

CAD vendors are not prepared to hand over “their” data for free, and in order to create the illusion of access and connectivity to this data, they have imposed rather complicated ways of storing and accessing information on their customers, allowing them to pull the initiative in processing master data over to their proprietary solutions.

Storing information in CAD software

The CAD-corporations of the world are proving to the construction industry with aggressive marketing (up to 50% of company budgets) that 3D-7D data creation tools are (will be) well implemented in their programs (Archicad-Nevaris, Autodesk-BIM360, Renga-PilotBIM, Bentley-OpenBuildings).

To avoid annoying the large market of third-party applications and investors in BIM and PropTechs start-ups, CAD solution vendors are “forced” to create connectivity to their proprietary data in 3D solutions via API queries, ODBC or third-party plug-ins. Data in Closed BIM solutions is transferred via databases, and everything in the construction industry that deals with 3D-7D data is also transferred into a database format, which is now commonly referred to as BIM data work.

Therefore, most 4D-7D data management systems today work with relational database management systems (RDBMS): MS SQL, MySQL, PostgresSQL, Oracle, MS Access.

SQL (structured query language) is a declarative programming language used to create, modify and manage data in a relational database (RDBMS).

To make it possible to connect to external services (databases) and their internal services from the Closed BIM environment, CAD software developers try to translate the data into SQL table format.

Data storage in SQL

The main advantage of SQL tables over other types of information storage is that they support very large database sizes with high query processing speeds. SQL also integrates well with web applications and other data management tools. In 4D-7D and ERP systems, data has long been stored in tabular SQL databases, but CAD data in the form of 3D geometry has until recently been confined to visualisation work and has struggled to be stored in tabular and textual form.

One of the first implementations of storing geometric information in tabular form, with the creation of an SQL database, was the Postgres project.

Postgres — pioneers of data storage from CAD to SQL

A motivating example for Postgres developers in the early 1980s was the need for databases to support computer-aided design (CAD) tools for the microelectronics industry.

In a 1983 article, Stonebraker (founder of the open source Postgres project) and his students explained how the CAD (computer aided design) industry required support for new data types such as polygons, rectangles, text strings and “efficient spatial search, design hierarchies and multiple representations” in the same physical structures.

Unfortunately, the Open Source initiative of the Postgres project developers occurred 20 years before CAD programs with the possibility of creating this data. And because Postgres has always been an open source project, for a long time it was only aimed at research use, not production.

Although PostgreSQL is today the most popular independent open source database system and the fourth most popular database system, most CAD software developers today prefer to use SQLite and Microsoft SQL databases.

Revit — database in graphical representation

The acronym BIM (Building Information Modeling) is a marketing invention of Autodesk Corporation; it refers to the data handling techniques that are created when working with the Revit 3D solution.

The product Revit, the main programme of the last 20 years for designing and creating BIM data, is essentially a database of model elements in graphical representation (a set of data streams), thus creating a BIM information model.

When using Revit Server, you may notice that the program communicates with the project data via db3 format, indicating the use of a relational database — SQLite — to communicate with some back-end services.

Out of all the integration options with external services, Autodesk only allows the upload of proprietary RVT data via plug-ins, API connection or ODBC uploads.

ODBC (Open Database Connectivity) is a software interface (API) for accessing databases.

The opening of the incomplete API access to Revit functions has triggered the development of solutions that allow working with Revit via API queries: Grasshopper, Dynamo, pyRevit, Forge.

API (Application programming interface) — a description of the ways in which one computer program can communicate with another program.

The purpose of such plug-ins and the ODBC driver is to allow the data to appear as if the user was accessing an SQL database.

Creation of ArchiCAD and Tekla tools API

The virtual building project in ArchiCAD is now a database into which you can make SQL queries via ODBC driver and retrieve any necessary project information structured in tables. The ability to execute SQL queries in ArchiCAD has been available since version 7.0 2001.

The ArchiCAD database (like the Revit product) uses a proprietary format, so providing ODBC access requires a major development effort.

The ArchiCAD database consists of tables of levels 2–3 with a complex indexing scheme. Opening an object container in ArchiCAD allows you to edit its component scripts, which, without a basic knowledge of ArchiCAD’s internal programming language, GDL (Graphical Graphics Description Language), is not the best way to proceed. Experienced programmers can only get to read and write access to project data through C/C++ programming and using ArchiCAD’s incomplete API.

Archicad developers have also worked on an ODBC-SQL-ArchiCAD project with the ability to interface with other CAD programs. These interactions were meant to help link architects, planners, designers and cost estimators into a single system using CAD software. But the project was abandoned, perhaps because of the obvious limitations of working with other CAD software.

The current plan of the Archicad developers is to provide a kind of BIMcloud API where project data can be accessed in IFC format.

The Tekla database (Tekla Structures), like other major CAD solutions, is not directly accessible. In 2020, Tekla developed an open API on the Microsoft.NET framework that now allows third party developers to write their own C# applications to manipulate data from Tekla Structures software. But, as with the Revit API or Archicad ODBC, the Tekla API works as a stripped-down toolkit to influence the program.

The problem of data interoperability in the construction industry

Through the use of proprietary formats, CAD vendors today do not allow access to 3D data that the user has created themselves in the programmes. Direct transfer of complete information from the 3D model to 4D-7D solutions is not possible today and to give at least some access to the data, corporations offer connectivity to “unstructured data sets” through tools: ODBC, plug-ins, services with API connectivity (to internal program functions) and incomplete IFC data format.

Data connections via plug-ins and APIs that do not work

When information is uploaded via plug-ins and APIs, usually only system parameters (base quantities) and element properties without geometry are uploaded, which makes it impossible to use the data offline in other 4D-7D systems (e.g. ERP, ECM, EPM).

We will talk more about what data is needed in the 4D-7D systems phase in the following articles.

In general, vendor ODBC tools and APIs do not give full access to the data, and their purpose was to isolate the external user from the database of all the properties of the elements that were created during the design process.

Corporations with data power are not prepared to lose control of their customers, which is why most professionals who try to extract volume information from models are now forced to develop their own plug-ins to connect to the data in 3D solutions and Excel spreadsheets to link the resulting data (without geometry) to existing 4D-7D solutions.

However, once data export via ODBC or a plug-in is set up, there is no guarantee that the next year, when a new version of the CAD program is released or the API libraries are updated, communication with the internal CAD program databases will work according to the same rules as in previous versions.

As a result, the construction industry suffers from a corporate monopoly over data, with the result that proprietary information about elements stored in proprietary formats is reflected in multi-billion dollar miscalculations on project construction timelines and costs.

Autodesk’s hegemony on data is being diluted by developers from European CAD corporations who, following the meteoric rise in popularity of Revit, are resurrecting Autodesk’s buried IFC project — a de jure transparent and interoperable data format.

More about the IFC project, existing and proposed BIMJSON solution to this problem we will talk in the next — the seventh part.

Article on LinkedIn:

https://www.linkedin.com/pulse/lobbyist-wars-development-bim-part-6-reasons-industry-artem-boiko

I will be glad for any comments, clarifications and criticism and will be grateful for repost in social networks.

The schemes indicated in the article are available in good quality at the link (Miro board)

https://bigdataconstruction.com/

⚡️ What is “pipeline in Construction”? https://datadrivenconstruction.io/index.php/pipeline-in-construction/

📥Free IFC2x3-to-Excel/DAE converter. Community Edition: https://datadrivenconstruction.io/index.php/product/free-ifc-2x3-converter-to-xlsx-dae-2024/

📥 Free Revit®(2015–2022)-to-Excel/DAE converter. Community Edition https://datadrivenconstruction.io/index.php/product/free-revit-2015-2021-converter-to-xlsx-dae/

Previous publications in the series Lobbyist Wars and BIM Development