Gen AI with LLM for construction technology

강태욱 연구위원
공학박사
Ph.D Taewook, Kang
laputa99999@gmail.com
https://daddynkidsmakers.blogspot.com
건설AI 히치하이커를 위한 가이드
GEN AI + LLM에 초점을 맞춘

KICT
Intro
Advanture. usecase from AI to Gen AI
My lab tools. open source
Jackpot and hallucination
Deep dive. Gen AI + LLM
Journey. Breadcrumbs
The Hitchhiker’s Guide to the Galaxy

Maker, Ph.D.
12 books author
BIM principle and Digital transformation: BIM principle & DX
사이트 소개, Profile (dxbim.blogspot.com)

Study history
P4
Life cycle
“If it isn’t fun, you’re doing the wrong
technology.” - ivan sutherland

Study history
P4
Roadmap
Computer graphics, CAD
BIM-GIS + ISO (2012)
BIM + IoT + Scan (2014)
Scan to BIM + Robotics + AI (2017)
Digital Twin + AI (2021)
BIM
VDCO
Mining
IoT
Scan
Virtual
Design
Construction
Operations
1D, 2D
3D, nD
Semantic
Knowledge
Sensors
Robotics

Usecase
advanture
CG
SCAN
ROBOT
AI
GEN AI
LLM
AEC

Generative AI identified by Deloitte
• Marketing content assistant
• Code assistant for developers
• Customer support on demand
• Product design assistant
• Research-based report generation
• Synthetic data generation
• Enterprise-wide data search and access
• Game content development
• Language translation
• Simulation of urban planning scenarios
• Hyper-personalised education
• Onboarding assistant
Usecase

Generative AI in AECO
• Multi Agent
• Anomaly detection in contract documents
• Reasoning in IoT dataset for O&M
• Risk management
• Optimization in Construction Management
• Concept design generation
• Report summarization and generation
• Training data generation
• Query using LLM
• Language translation
• Simulation of urban planning scenarios
• Personalised education and guidance system
Usecase

Scenario
CPS & DT
Digital
Twin
Structural
health
monitoring
Track and
trace
Remote
diagnosis
Remote
services
Remote
control
Condition
monitoring
Systems
health
monitoring
BIM
as i-DB
IoT…
AI
Sensor device
ICBM
Simulation
Robotics
Scan-Vision
Smart contract
based on Blockchain
Gen AI
Multi Agent

Scenario
IoT
Big data
management
AI + Simulation using LLM
Cloud
platform
Machine control
Field monitoring
IoT
sensor
Usecase
for safety, accuracy, productivity
sensing
Data analysis
& prediction
GIS IoT based monitoring
Field control
Infra IoT
service
connection
Plant control system (SCADA)
Field monitoring system
LoRA, BLE,
WiFi…
Layer 8 | IISL
(Infra IoT Service Layer)
Worker
Agency
<device_definition id=‘dd#1’>
<device id=‘T#1’name=‘temp’type=‘temperature’>
<maker name=‘CH korea’ email=‘laputa99999_9@gmail.com’ tel=‘82-0330-0802-1013’ location=‘…’/>
<specification>
<op_range name=‘voltage’ unit=‘V’type=‘real’value=‘3.3’/>
<op_range name=‘temperature’ unit=‘degree’ type=‘real’begin=‘-10.0’end=’60.0’/>
<op_range name=‘humidity’unit=‘%R.H’type=‘real’begin=‘0.0’end=’50.0’/>
<op_range name=‘GPS’unit=‘WGS84’type=‘vector2D’begin=‘(0,0)’ end=‘(127, 32)’/>
<op_range name=‘characteristic_curve’unit1=‘temperature’ unit2=‘voltage’ type=‘vector2D’>
(0,0), (1.2, 2.4), (3.5, 6.2), (4.1, 7.2)
</op_range>
<op_range name=‘period’unit=‘year’value=‘2’/>
</specification>
</device>
</device_definition>
Intelligent IoT sensor
•Self diagnose
•IISL protocol
•Security
•Availability
1
1
2
3
4
5
6
7
8
Plant sensing

Overview
AI ConTech $3B funding from 2021 (Daniel Laboe, 2023.10)
2024년 상반기 스마트 건설과 BIM 기술 동향 (buildingSMART )
Machine
learning
Deep
learning
Gen AI
Multi-
agent
AGI

Usecase
AI 기반 텍스트 렌더링 Revit 애드인 (VERAS, EnvolveLAB)

Usecase
https://daddynkidsmakers.blogspot.com/2023/04/ope
nai-for-grasshopper.html
OpenAI for Grasshopper

Usecase
Technology to create 2D drawings with perfect dimensions from 3D
models with one click (Autodesk Fusion, November 2023)

Usecase
CreationAI-based design system AiCorb
(Obayashi Corporation)

Usecase
AI-based structural frame, automatic creation of member
cross sections SYMPREST (Shimizu Construction)

Usecase
Rebar Tying 로봇(Florida DoT, 2020)

Usecase
Energy savings through
machine learning
AMR DNA, Energy savings through machine learning,
http://www.energyassets.co.uk/service/amr-dna

Usecase
Siemens Process Simulate (left) connects to NVIDIA Omniverse (right) to enable
a full-design-fidelity, photorealistic, real-time digital twin.
https://www.robotics247.com/article/siemens_xcelerator_nvidia_omniverse_accel
erate_digital_twins_manufacturing
NVIDIA Founder and CEO Jensen Huang (left)
and Siemens CEO Roland Busch

Open source
Alberto Rizzoli, 2022, 27+ Most Popular Computer Vision Applications and Use
Cases in 2022, V7

Open source
https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/controlnet.ipynb
https://github.com/mac999/gen_ai_gpt

Open source
Generative Adversarial Networks
Fake Obama created using AI video tool -
BBC News. Jul 19, 2017
https://www.vegaitglobal.com/media-center/knowledge-
base/what-is-stable-diffusion-and-how-does-it-work

Open source CARLA - Open Urban Driving Simulator
https://github.com/carla-simulator/carla
YOUTUBE

Open source
Cases in 2022, V7
YOLACT (You Only Look At CoefficienTs)
Detectron

Open source
Cases in 2022, V7
Tesseract-OCR
Use Google Cloud Vision API to process
invoices and receipts

Open source
Open Neural Network eXchange

Cesium
http://cesiumjs.org/Seattle/
Open source
KICT

Open source
https://github.com/ZiwenZhuang/parkour?tab=readme-ov-file

Open source
http://guswnsxodlf.github.io/software-license
GNU General Public License(GPL) 2.0
– 의무 엄격. SW 수정 및 링크 경우 소스코드 제공 의무
GNU Lesser GPL(LGPL) 2.1
– 저작권 표시. LPGL 명시. 수정한 라이브러리 소스코드 공개
Berkeley Software Distribution(BSD) License
– 소스코드 공개의무 없음. 상용 SW 무제한 사용 가능
Apache License
– BSD와 유사. 소스코드 공개의무 없음.
Mozilla Public License(MPL)
– 소스코드 공개의무 없음. 수정 코드는 MPL에 의해 배포. 이외 결합 프로그램 코드는 공개
필요 없음
MIT License
– 라이선스 / 저작권만 명시 조건

Gym for AI Research
Install pythonDownload Python | Python.org
Install anaconda Free Download | Anaconda
Install vscode Download Visual Studio Code -
Mac, Linux, Windows
Install sublime Download -Sublime Text
cmd*관리자권한으로실행해야함
cd
mkdirprojects
cd projects
mkdirtest
pip install virtualenv
pip install jupyterlab
pip install ipywidgets

Gym for AI Research
1. pip install tensorflow pip로 TensorFlow 설치
2. pip install keras keras · PyPI
3. pip install torch torchvision torchaudio Start Locally | PyTorch

Gym for AI Research
pip install virtualenv
pip3 install virtualenv
.venvScriptsactivate
source ./venv/bin/activate

Gym for AI Research
Install Docker Get Started | Docker
Creating a Simple Web Server with Docker: A Step-by-Step Guide to Running
Your Web Server as a Container | by Srija Anaparthy | AWS Tip
Run cmd
docker run -d -p 8080:80 nginxdemos/hello nginxdemos/hello - Docker Image |
Docker Hub

Gym for AI Research
1. Install Node.JS Download | Node.js (nodejs.org)
2. Run cmd
3. cd c:projects
4. mkdir test
5. cd test
6. npm install -g --unsafe-perm node-red
7. node-red

Gym for AI Research
https://colab.research.google.com/

example
Trimble
2021.3
Deep Learning
IoT
GIS

example
Linear(4, out = 6)
ReLU()
BatchNorm1d(in = 6)
Linear(in = 6, out = 16)
ReLU()
BatchNorm1d(in = 16)
Linear(in = 16, out = 9)
ReLU()
…
ReLU()
BatchNorm1d(in = 3)
Linear(in = 3, out = 1))
epochs = 5000
learning_rate = 0.001
batch_size = 64
Loss=0.002. Train MAPE & Acc = (0.025, 95.30%). Test
MAPE & Acc = (0.292, 70.76%)

example
Trimble
GPS
카메라
카메라
스캐너
IMU
DMI
KICT

example https://github.com/mac999

Gen AI core
https://daddynkidsmakers.b
logspot.com/2023/12/blog-
post.html

Gen AI core multi-modal
Variational autoencoder

Gen AI core
잠재공간에 맵핑된(인코딩된) 데이터(Alexej Klushyn, 2019.12, Learning Hierarchical Priors in VAEs)

Gen AI core multi-modal
CLIP(Contrastive
Language-Image Pre-
Training. Open AI. 2021
https://daddynkidsmakers.blogspot.com/2024/02/clip.html

Gen AI core
https://daddynkidsmakers.blogspot.
com/2024/02/llama-2.html

Gen AI core
스테이블 디퓨전 기술 개발 주역 Computer Vision & Learning Group (ommer-lab.com)
Prof. Dr. Björn Ommer University of Munich

Jackpot & hallucination
output.append((transformer.generate(t
exts = ['bridge chair','computer table',
'chair table'] , temperature = 1) ))
output.append((transformer.generate(texts =
['sofa','bed', 'computer screen', 'bench', 'chair', 'table' ] ,
temperature = 0.0) ))
Not
practical.
MeshGPT

Jackpot & hallucination Large Language Models as General Pattern Machines

Gen AI + LLM LLAMA & Ollama & Langchain
A100 (80GB) GPU =
21 day (traning)
ollama run llava
"describe this
iamge: ./cat.jpg"
"테이블 다리로 보이는 것 옆에 똑바로 앉아 있는 얼룩무늬 고양이
의 이미지입니다. 고양이는 뚜렷한 어두운 줄무늬가 있는 밝은 주황
색 털을 가지고 있는데, 이는 흔히 볼 수 있는 패턴입니다. 얼룩 고
양이는 눈을 크게 뜨고 카메라를 정면으로 차분한 태도로 바라보고
있는 모습입니다.
배경에는 바닥에 다음과 같은 패턴의 러그가 깔려 있습니다. 베이지
색과 기타 중성색이 포함됩니다. 전경에 테이블 다리가 있기 때문에
다이닝 룸과 같은 생활 공간처럼 보입니다."

Gen AI + LLM LLAMA & Ollama & Langchain
Langchain CEO. 2022.
30M$ in 2023.
https://daddynkidsmakers.blogsp
ot.com/2024/04/blog-
post_21.html

Gen AI + LLM
Chroma CEO. Jeff Huber, Anton Troynikov
18M$. 2023

Gen AI + LLM
class ChatPDF:
vector_store = None
retriever = None
chain = None
def __init__(self):
# OLLAMA의 mistral 모델 이용
self.model = ChatOllama(model="mistral")
# PDF 텍스트 분할
self.text_splitter = RecursiveCharacterTextSplitter(chunk_size=1024, chunk_overlap=100)
self.prompt = PromptTemplate.from_template(
"""
<s> [INST] You are an assistant for question-answering tasks.
Use the following pieces of retrieved context
to answer the question. If you don't know the answer,
just say that you don't know. Use three sentences
maximum and keep the answer concise. [/INST] </s>
[INST] Question: {question}
Context: {context}
Answer: [/INST]
"""
)

Gen AI + LLM
class ChatPDF:
def ingest(self, pdf_file_path: str):
docs = PyPDFLoader(file_path=pdf_file_path).load() # 랭체인의 PDF 모듈 이용해 문서 로딩
chunks = self.text_splitter.split_documents(docs) # 문서를 청크로 분할
chunks = filter_complex_metadata(chunks)
# 임메딩 벡터 저장소 생성 및 청크 설정
vector_store = Chroma.from_documents(documents=chunks, embedding=FastEmbedEmbeddings())
self.retriever = vector_store.as_retriever(search_type="similarity_score_threshold",
search_kwargs={
"k": 3,
"score_threshold": 0.5,
},
) # 유사도 스코어 기반 벡터 검색 설정
self.chain = ({"context": self.retriever, "question": RunnablePassthrough()} | self.prompt | self.model |
StrOutputParser()) # 프롬프트 입력에 대한 모델 실행, 출력 파서 방법 설정
def ask(self, query: str): # 질문 프롬프트 입력 시 호출
if not self.chain:
return "Please, add a PDF document first."
return self.chain.invoke(query)

Gen AI + LLM
https://daddynkidsmakers.blogspot.com/2024/
02/github-copilot-ai.html

Gen AI + LLM earthwork quantity takeoff
H1 H1 H2
H1 H2
H3 H1 H2
H3
H4
H1 H2
H3
H4
L={H1, H2, H3, H4}
F1=L1
F2=L2
Step 1. Step 2.
Step 3. Step 4.
Step 5. Step 6.

Gen AI + LLM
ENA Model ID Batch
size
Epochs Layer architecture Normaliza
tion &
Activation
Learning
rate
Pre-trained model
embedding transformers
M1.1. MLP 32 150 [14]-[128,64,32]-[10] ReLU,
batch
normal,
dropout
0.001 - -
M1.2. MLP 32 150 [14]-[64,128,64]-[10] 0.001 - -
M1.3. MLP 32 150 [14]-[64,128,64,32]-[10] 0.001 - -
M1.4. MLP 32 150 [14]-[32,64,32]-[10] 0.001 - -
M2.1. LSTM 32 150 [14]-LSTM[128]-[10] dropout 0.001 - -
M2.2. LSTM 32 150 [14]-LSTM[128]-[64,32]-[10] 0.001 - -
M2.3. LSTM 32 150 [14]-LSTM[256]-[128,64]-[10] 0.001 - -
M3.1. Transformers
32 300 [320,512]-ENC[512]-[64]-[10]
multi-head
attention,
layer
normal,
dropout
1.00E-05
BERT-base-
uncased
-
M3.2.Transformers
64 300 [320,512]-ENC[512]-[64]-[10] 1.00E-05
BERT-base-
uncased
-
M3.3.Transformers
128 300 [320,512]-ENC[512]-[64]-[10] 1.00E-05
BERT-base-
uncased
-
M4.1. LLM
32 150 [293]-EMB-ENC-[10] 1.00E-05
BERT-base-
uncased
BERT
M4.2 LLM
32 300 [293]-EMB-ENC-[10] 1.00E-05
BERT-base-
uncased
BERT
1

Gen AI + LLM
ENA Model ID Train No Loss Accuracy Model size
(kb)
Time performance
(minutes)
M1.1. MLP 1650 0.0870 0.9494 61 0:02:34
M1.2. MLP 1714 0.0852 0.9519 84 0:02:36
M1.3. MLP 1716 0.0882 0.9544 93 0:02:42
M1.4. MLP 1718 0.1322 0.9507 30 0:02:25
M2.1. LSTM 1730 0.0889 0.9408 812 0:02:13
M2.2. LSTM 1732 0.0851 0.9420 850 0:02:17
M2.3. LSTM 1734 0.0886 0.9408 3,312 0:02:00
M3.1. Transformers 2003 0.3533 0.7744 74,557 0:11:06
M3.2.Transformers 2014 0.3551 0.7719 74,557 0:07:48
M3.3.Transformers 2021 0.3596 0.7423 74,557 0:06:25
M4.1. LLM 0103 0.0587 0.9507 427,783 3:12:05
M4.2. LLM 2334 0.0534 0.9531 427,783 7:59:00

Gen AI + LLM BIM RAG
Accuracy = 41% (7 / 17)
Domain specific RAG
challenge, direction
• Limited input context
for LLM
• No vocaboary
• Long token distance
• Hallucination

Gen AI + LLM how to make LLM dataset

Gen AI + LLM domain knowledge fine tuning. ex) BIM-GIS
No fine tuning

Gen AI + LLM domain knowledge fine tuning. ex) BIM-GIS

Gen AI + LLM
fine tuning. Epoch = 3 (90 min)

Gen AI + LLM
fine tuning. Epoch = 10 (over 5 hours)

guide
생성AI 시대 BIM 기술 동향과 해외
스마트 건설 사례 – 이강, 정숭용
생성AI LLM과 스테이블 디퓨전 최신 기술 및
활용 동향 - 최돈현, 김태영

guide
https://www.slideshare.net/laputa999

guide
https://github.com/mac999

guide
https://daddynkidsmakers.blogspot.com

conclusion
• Google, Meta, MS 등 선진국 테크기업 중심 생성AI 잠재력 테스트 중
• 파운데이션 모델 개발은 이미 선진국 테크기업 및 중국 중심
• AI 개발 시 빅테크기업 공개 플랫폼&도구, Huggingface, W&B 등 사용
은 필수. 아키텍처 모델 직접 개발은 극소수.
• LLM 은 멀티모달, 다중 에이전트 플랫폼으로 발전
• GPU 리소스 한계로 RAG, finetuning 등 다양한 기술 발전
• 온라인 문제로 생성AI가 스마트폰, 노트북 같은 모바일 장치에 내장되
는 추세
• Langchain과 같은 RAG, 에이전트 기술 테크 기업 출현
• 일부 선진국 중심으로 생성AI 기반 건설 테크 기업 출현
• 생성AI가 제대로 동작될려면 챌린지와 연구가 많이 남아 있음

conclusion
Hitchhiker's Guide - Earth Destroyed and Guide
Join Gen AI world

Gen AI with LLM for construction technology

More Related Content

Gen AI with LLM for construction technology