101-0526-00L, Wed 3.45 - 5.30PM
Introduction to
Visual Machine Perception
for Architecture, Construction, and Facility Management

Course Description

The past few years a lot of discussion has been sparked on AI in the Architecture, Construction, and Facility Management (ACFM) industry. Despite advancements in this interdisciplinary field, we still have not answered fundamental questions about adopting and adapting AI technology for ACFM. In order to achieve this, we need to be equipped with rudimentary knowledge of how this technology works and what are essential points to consider when applying AI to this specific domain.

In addition, the availability of sensors that collect visual data in commodity hardware (e.g., mobile phone and tablet), is creating an even bigger pressure in identifying ways that new technology can be leveraged to increase efficiency and decrease risk in this trillion-dollar industry. However, cautious and well-thought steps need to be taken in the right direction, in order for such technologies to thrive in an industry that showcases inertia in technological adoption.

The course will unfold as two parallel storylines that intersect in multiple places:
1) The first storyline will introduce fundamentals in computer vision and machine learning technology, as building blocks that one should consider when developing related applications. These blocks will be discussed with respect to latest developments (e.g., deep neural networks), pointing out their impact in the final solution.
2) The second storyline consists of 3 ACFM processes, namely architectural design, construction renovation, and facility management. These processes will serve as application examples of the technological storyline.

In the points of connection students will see the importance of taking into account the application requirements when designing an AI system, as well as their impact on the building blocks. Guest speakers from both the AI and ACFM domains will complement the lectures.

Course Objectives

By the end of the course students will develop computational thinking related to visual machine perception applications for the ACFM domain. Specifically, they will:

Gain fundamental understanding on how this technology works and the impact it can have in the ACFM industry.
Identify limitations, pitfalls, and bottlenecks in these applications.
Develop critical thinking on solutions for the above issues.
Acquire hands-on experience in creatively thinking and designing an application.
Use this course as a “stepping-stone” to Machine Learning-intensive courses offered in D-BAUG and D-ARCH.



Iro Armeni

PostDoctoral Researcher


Jianpeng Cao

PhD Candidate
Teaching Assistant

Course Schedule

(Subject to change)




24.02 :   

Introductory Class

03.03 :   

Drawing lines, surfaces, and primitives in visual data

10.03 :   

As-is geometric model: From pixels to 3D recostruction

17.03 :   

Making sense of visual data: Segmentation and clustering

23.03 :   

Assignment 1 is due

24.03 :   

What is this that I see?: Visual data classification

30.03 :   

1st Project Milestone is due

31.03 :   

Guest Talk, Digital Twins and Artificial Intelligence for Managing Construction and Operation of Civil Infrastructure Systems, by Professor Mani Golparvar-Fard, Dept. of Civil and Environmental Engineering, University of Illinois Urbana-Champaign

13.04 :   

Assignment 2 is due

14.04 :   

Toward a "digital-twin": Detection and Semantic Segmentation
With guest talk by Professor Martin Fischer, Dept. of Civil and Environmental Engineering, Stanford University

21.04 :   

Visual data representation: 3D scene graph, BIM, and more
With guest talk by Professor Martin Fischer, Dept. of Civil and Environmental Engineering, Stanford University

28.04 :   

The machine Designer: Generating new visual data

04.05 :   

2nd Project Milestone

05.05 :   

Keeping track of mobile elements in construction sites: Object and people tracking

11.05 :   

Assignment 3 is due

12.05 :   

Construction worker productivity and safety: Activity recognition

19.05 :   

Guest Talk, Human-Robot Collaboration, by Dr. Claudia Pérez D'Arpino, Postdoctoral Researcher at the Stanford Vision and Learning Lab, Stanford University

26.05 :   

Guest Talk, "Mixed Reality and HoloLens", by Dr. Martin Oswald, Senior Researcher at the Computer Vision and Geometry Group, ETHZ

02.06 :   

Final Project Presentation

Course Evaluation

  • Participation: 10% of grade. Students should actively participate in the class with questions. They are not required to speak up at every class, but active engangement throuhgout the course will be part of the grade.
  • Assignments: 40% of grade. Students will be given assignments throughout the course that they are required to submit as part of their evaluation. Each assignment will contribute equally to the final grade. The assignments will have two compponents: hands-on experimentation with pre-given modules and critical thinking on the output of the experiments. Due dates are dispersed throughout the semester based on the covered material.
  • Course Project: 50% of grade. The course has a final project (in lieu of a final exam) which will be performed in groups. The project deliverables are an in-class presentation at the final day and a report. Both slides and report will be submitted as part of this assignment. Preparation for it will start early on in the semester and we will guide you through the milestones: (1st milestone) Submit the title of your project, a short description, and the names of the members in your team. Note that title and description could change as you explore the project; (2nd milestone) Submit mid-term report with progress on the project; (3rd and final milestone) Present your project in class and submit a report.