101-0526-00L, Wed 15.45 - 17.30, HIL E 7
Introduction to
Visual Machine Perception
for Architecture, Construction, and Facility Management

Course Description

The past few years a lot of discussion has been sparked on AI in the Architecture, Construction, and Facility Management (ACFM) industry. Despite advancements in this interdisciplinary field, we still have not answered fundamental questions about adopting and adapting AI technology for ACFM. In order to achieve this, we need to be equipped with rudimentary knowledge of how this technology works and what are essential points to consider when applying AI to this specific domain.

In addition, the availability of sensors that collect visual data in commodity hardware (e.g., mobile phone and tablet), is creating an even bigger pressure in identifying ways that new technology can be leveraged to increase efficiency and decrease risk in this trillion-dollar industry. However, cautious and well-thought steps need to be taken in the right direction, in order for such technologies to thrive in an industry that showcases inertia in technological adoption.

The course will unfold as two parallel storylines that intersect in multiple places:
1) The first storyline will introduce fundamentals in computer vision and machine learning technology, as building blocks that one should consider when developing related applications. These blocks will be discussed with respect to latest developments (e.g., deep neural networks), pointing out their impact in the final solution.
2) The second storyline consists of 3 ACFM processes, namely architectural design, construction renovation, and facility management. These processes will serve as application examples of the technological storyline.

In the points of connection students will see the importance of taking into account the application requirements when designing an AI system, as well as their impact on the building blocks. Guest speakers from both the AI and ACFM domains will complement the lectures.

Course Objectives

By the end of the course students will develop computational thinking related to visual machine perception applications for the ACFM domain. Specifically, they will:

Gain fundamental understanding on how this technology works and the impact it can have in the ACFM industry.
Identify limitations, pitfalls, and bottlenecks in these applications.
Develop critical thinking on solutions for the above issues.
Acquire hands-on experience in creatively thinking and designing an application.
Use this course as a “stepping-stone” to Machine Learning-intensive courses offered in D-BAUG and D-ARCH.



Iro Armeni

PostDoctoral Researcher


Jianpeng Cao

PhD Candidate, CEE
Teaching Assistant


Martin Bucher

MS Student, CS
Teaching Assistant

Course Schedule

(Subject to change)




23.02 :   

Introductory Class

02.03 :   

Drawing lines, surfaces, and primitives in visual data

09.03 :   

As-is geometric model: From pixels to 3D recostruction

15.03 :   

Assignment 1 is due

16.03 :   

Making sense of visual data: Segmentation and clustering

22.03 :   

1st Project Milestone is due

23.03 :   

What is this that I see?: Visual data classification

30.03 :   

Toward a "digital-twin": Detection and Semantic Segmentation

05.04 :   

Assignment 2 is due

06.04 :   

Visual data representation: 3D scene graph, BIM, and more

13.04 :   

The machine Designer: Generating new visual data

26.04 :   

2nd Project Milestone

27.04 :   

Guest Talk, TBD, by Prof. Burcu Akinci, CEE, Carnegie Mellon University

04.05 :   

Keeping track of mobile elements in construction sites: Object and people tracking

10.05 :   

Assignment 3 is due

11.05 :   

Construction worker productivity and safety: Activity recognition

18.05 :   

Guest Talk, "Mixed Reality and HoloLens", by Prof. Martin Oswald, CS, University of Amsterdam

25.05 :   

Guest Talk, "Human-Robot Collaboration", by Dr. Claudia Pérez D'Arpino, Research Scientist at NVIDIA

01.06 :   

Final Project Presentations

Important Information

  • Class Material: Class material will be posted on Moodle (link). We will also use Moodle to submit assignments.
  • The class is help in person, except if otherwise stated.

Course Evaluation

  • Participation: 5% of grade. Students should actively participate in the class with questions. They are not required to speak up at every class, but active engangement throuhgout the course will be part of the grade.
  • Assignments: 45% of grade. Students will be given assignments throughout the course that they are required to submit as part of their evaluation. Each assignment will contribute equally to the final grade. The assignments will have two compponents: hands-on experimentation with pre-given modules and critical thinking on the output of the experiments. Due dates are dispersed throughout the semester based on the covered material.
  • Course Project: 50% of grade. The course has a final project (in lieu of a final exam) which will be performed in groups. The project deliverables are an in-class presentation at the final day and a report. Both slides and report will be submitted as part of this assignment. Preparation for it will start early on in the semester and we will guide you through the milestones: (1st milestone) Submit the title of your project, a short description, and the names of the members in your team. Note that title and description could change as you explore the project; (2nd milestone) Submit mid-term report with progress on the project; (3rd and final milestone) Present your project in class and submit a report.