ALICE DCS part 1

DCS workshop 3.4.06

L.Jirden for the ALICE team

 Organization & responsibilities

 Coordination role
 JCOP participation
 Installation & commissioning
 Critical issues part 1

1
 Organization SERVICES
ACR Gas
Consoles Power
Facilities al

Individual meetings Individual meetings

i
du
vi ing
s JCOP Cooling
d t Magnets
In ee
m boards, w-groups,
Individual meetings project meetings Safety
Offline
Networks
ECS Access
ALICE
Trigger Controls
DAQ 3 staff Coordination INFRASTR
3.5 PJAS (ACC) B-field
HLT 1 Fellow
1 Student Space frame
DCS Workshop Radiation
TC, TB “Controls Board” Environment
~2 d. each ALICE week

Detector groups
SPD SDD SSD TPC TRD TOF HMP PHO CPV FMD T0 V0 PMDMTG MTK ZDC ACO EMC
2
 Liaison with
Detectors ALICE
Controls
Coordination
(ACC)

User Req. Doc (URD)

Overview Drawings

DCS Workshop Documented ALICE Standards:

• ALICE DCS State Diagrams
• Guidelines & Conventions
Individual meetings • ALICE DCS Computing Rules
• FedServer API for ALICE DCS
• Interlocks
• etc

Common Solutions

Detector groups
SPD SDD SSD TPC TRD TOF HMP PHO CPV FMD T0 V0 PMDMTG MTK ZDC ACO EMC
3
For each Template version 2 (19 December 2001)
detector: DCS User Requirements Document
for My-Detector
Version x (dd month yyyy)

1. Introduction
2. Description of My-Detector My-Detector
3. The Control System

Sub-system 1 Sub-system n

Sub-system 2

Device1 Device3 Device1 Device3

Device2 Device2

Device1 Device3

Device2

I.Description and requirements of the Sub-systems

1) Sub-system 1
a) Functionality Off Command
V=0
b) Device or Equipment “Clear”
c) Interlocks and Safety aspects Ramp Down Tripped Ramp Up
d) Operational and Supervisory aspects
Trip
Command
On V = V0 4
etc.
 26/11/04
TPC(1) Overview
[FSM?]
Database(s)
Control room

drawing
PVSS II PVSS II PVSS II
OPC DIM
(ACR)

client client
User interface

Ethernet

PVSS II PVSS II PVSS II PVSS II

OPC client OPC client OPC client OPC client
PVSS II
ISEG OPCserver Wiener OPCserver ELMB OPCserver PVSS II ELMB OPCserver
DIMclient
PCI-CAN PCI-CAN PCI-CAN RS232 PCI-CAN

Counting room
C C C E S C
1..3 1..22 1..18 DIMsrv
1 1?

[FED]
3 DIMcl
1

ISEG ?Heinzinger
Eth. Switch
E

Cavern
22

Wiener
?
Eth. Switch
E
18? 2
HV LV HV
André Augustinus

ELMB ELMB

Inside magnet
288 108 1
108?
?
~400?
DDL

LV monitor 216
Chambers, DIMsrv
Chambers Chambers RCU HVrod HVrod 5
detector
High Voltage Low Voltage Temp. Monitor FEE Very High VHV Curr.
 Sub-systems:
Common and specific
solutions HMPI PHO
SPD SDD SSD TPC TRD TOF CPV µ-trk µ-trg FMD T0 V0 PMD ZDC ACO EMC Trig. DCS
D S

HV CAEN iseg Iseg CAEN

CAEN iseg iseg CAEN CAEN CAEN CAEN CAEN CAEN CAEN CAEN CAEN CAEN Iseg
Easy +distr +V reg. +distr

LV CAEN AREM CAEN ALICE CAEN CAEN CAEN CAEN CAEN

Wiener Wiener Wiener Wiener Wiener Wiener CAEN Wiener
Easy pro Easy box Easy Easy Easy Easy Easy

VME Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener
VME VME VME VME VME VME VME VME VME VME VME VME VME VME

MON ELMB ELMB PLC ELMB ELMB ELMB ELMB ELMB ? DIM ELMB

FED JTAG DCS JTAG RCU DCS DRM RCU RCU DRM DCS RCU

Cool

GAS
VHV Liquid LED GMS Thresh Position LED? RackCtrl

Pulser Crystal Laser Align

Laser BField

Drift Vel. RadMon

Common solution 67 Common/specific 7

FED concept 7 Unknown 2

Specific solution 11 TOTAL 105 6

Responsibili JCOP + ACC JCOP + ACC
DAQ + ACC +
ties ACR Safety
service
+
service
Consoles groups
CSAM Services
groups
CR facilities DSS Gas
JCOP + Electricity
External ACC
Cooling
ACCCentral DCS
Systems Magnets
ECS
Access
Trigger Back-End
DAQ ACC
Infrastr
HLT H/W infrastructure B-field
Offline JCOP + Space Frame
LHC ACC Tools & Components
ACC
Env. Param.
Radiation

ACC ACC ACC

Detectors
SPD SDD SSD TPC TRD TOF HMP PHO FMD T0 V0 PMD MTI MTK ZDC ACO
~100 sub-systems 7
 Central DCS
CERN

Consoles Remote Access Backend Servers

Domain Controller
System Management Firewall Domain Controller
DCS Operator IP Reach for KVM DB Server (Config.)
Terminal Server DB Server (Archive)
DCS Status DB Server (Archive)
Terminal Server File Server
Safety Status
Information Server

ACR Consoles DCS network

Worker Nodes
Network and Security Gas & Cooling
Alarms LHC & Access
FSM General services
General DCS Environment
Electricity & Racks Alignment
Detector DCS
Infrastructure, network Spares
Racks, network, cabling Hot swap spare servers
KVM switches Disks & memories
controlled power plugs
8
 26/11/04
TPC(1) [FSM?]
Database(s)
Control room

PVSS II PVSS II PVSS II

OPC DIM
(ACR)

ACC provides:
User interface
Backend + Hardware and Software client client

infrastructure
DETECTORS
Ethernet provide: Devices + Detector applications
PVSS II PVSS II PVSS II PVSS II
OPC client OPC client OPC client OPC client
PVSS II
ISEG OPCserver Wiener OPCserver ELMB OPCserver PVSS II ELMB OPCserver
DIMclient
PCI-CAN PCI-CAN PCI-CAN RS232 PCI-CAN

Counting room
C C C E S C
1..3 1..22 1..18 DIMsrv
1 1?

[FED]
3 DIMcl
1

ISEG ?Heinzinger
Eth. Switch
E

Cavern
22

Wiener
?
Eth. Switch
E
18? 2
HV LV HV
André Augustinus

ELMB ELMB

Inside magnet
288 108 1
108?
?
~400?
DDL

LV monitor 216
Chambers, DIMsrv
Chambers Chambers RCU HVrod HVrod 9
detector
High Voltage Low Voltage Temp. Monitor FEE Very High VHV Curr.
 Coordination in
purchase of
equipment
 Purchased by ACC
 Computers
 Servers for back-end

 PC’s for back-end and front-end (~90)

 all details on DCS web page:

http://alicedcs.web.cern.ch/AliceDCS/Meetings/
 PCI’s: Kvaser, Peak, Caen PCI-VME bridge, MXI

 Network
 Global network (~400 ports), sub-contracted to IT/CS

 On-detector network (~1000 ports) by ACC

 Devices
 ELMB’s (~150) + ELMB power

 Serial device (RS232)

10
 Serial devices
(RS232)
 Aim for standard way of interfacing this
class
 Physical interface (cables and alike)
 Needs a COM port on a PC; cable length is limited

 A standard (and transparent !) way to use serial

over Ethernet overcomes this limit

 CERN standard; expertise exists

 Examples exist, several detectors start using it

 Exist in single and multi-port version

 Software interface (transporting the data)

 PVSSII has all one needs to ‘drive’ a serial connection

 Examples exist

 Functional interface (exchange sensible data)

 Application dependent

 to be developed by the detector (using the above!)

11
 Coordination in
purchase of
equipment
 Coordinated by ACC
 LVPS
 Wiener PL512, water-cooled (6 detectors ~115

units)
 Caen Easy (7 detectors ~110 units)

 Caen SY1527 (2 detectors, 1 units)

 AREM Pro (1 detector, 6 units)

 HVPS
 ISEG, 4 module types (4 detectors, ~15 crates)

 Caen SY1527 (13 detectors, ~20 main-frames)

 VME
 Wiener 6U & 9U, local PS, water & air cooled (15

detectors, ~60 crates)

12
 Detector computer
distribution
 Strict separation per detector, to allow
for:
 Staged installation
 Independent operation
 Distribution of sub systems based on:
 Number of channels
 Estimated resources needed by a sub system
 Performance tests done by several groups
 Some specific requests

13
 Detector
Operator Node -
ON
Worker Node -
WN

computers
Front-End Dev -
FED

SPD SDD SSD TPC TRD TOF

Operator Node Operator Node Operator Node Operator Node Operator Node Operator Node
HV + LV HV HV + LV HV HV HV
FED + Crate LV + FED + Crate FED + Crate + ELMB LV + ELMB LV LV
FED FED FED VHV FED FED + Crate
FED FED FED FED [18]
Pulser FED
Laser
Laser
Drift velocity
FED

HMPID PHOS CPV Muon Trk Muon Trg FMD

Operator Node Operator Node Operator Node Operator Node Operator Node Operator Node
HV + LV HV + FED + LED HV + LV + ELMB HV HV + LV HV + LV + FED
Crate + PLC LV + ELMB + Crate LV Crate + ELMB FED
FED Crate + ELMB +GMS

T0 V0 PMD ZDC ACORDE EMC

Operator Node Operator Node Operator Node Operator Node Operator Node Operator Node
HV + LV HV + LV + Crate HV + LV HV + Crate HV + LV HV + LV + FED
FED + Crate + Laser Crate + ELMB FED
14
FED
 PVSS & FSM
distribution
 PVSS
 Baseline approach
 Each detector DCS will be a distributed PVSS system

 Each WN will run a single PVSS project

 when several sub systems are controlled from a

single WN these shall be grouped in a single project

 All detector DCS’s will form one big distributed system

(of distributed systems)

 FSM
 integrated in the respective PVSS projects
 One PVSS project can have several FSMs

 Detector DCS FSM’ to run on least loaded node

15
 Training & Tutorials
 JCOP courses
 Are appreciated
 At least one person from each detector has now
taken the basic course
 Problem: manpower changes, students graduating
 Make handouts available
 DCS workshops
 ALICE “tutorials”
 DCS week planned
 JCOP advanced course + ALICE tutorials

16
 ALICE DCS week
 Program (preliminary)
 Monday hours
 Framework tools & components; recap IT/CO 8
 Tuesday
 Alarm handling IT/CO 2
 Configuration Dbase IT/CO 5
 Access control IT/CO 1
 Wednesday
 FSM IT/CO + ALICE 8
 Thursday
 DIM IT/CO 1
 Framework devices IT/CO 1.5
 Conditions ALICE 1.5
 Distributed systems ALICE 1
 ALICE Front-End Device (FED) ALICE 2
 ALICE FEE configuration ALICE 1
 Friday
 ALICE DCS guidelines/standards/rules ALICE 2
 Detector specific applications & AOB ALICE 6
17
 Detectors
H T S T T MM P A P S S F Z C E
T V
MP P R O T T MC H D S M D P M
0 0
P C D D F K G D O O D D D C V C
User requirements
Use of tools &
components
Basic device control
FSM for at least one
subsystem
Test with ECS/DAQ
All subsystems ready

done/used
not done/not 18
used
 JCOP ALICE
 JCOP ALICE
 Most Fw components + DSS system + GAS control
 Advice, assistance: PLC applications, OPC tests,
etc
 Participation in ALICE DCS workshops
 ALICE JCOP
 ISEG component
 BCAM component
 Cooling control
 Terminal Server evaluation
 Performance testing
 Caen HV

 PVSS scaling

 Oracle archiving

19
 Main scheduled
“challenges”
before LHC start-up
 Pre-commissioning on surface at P2
 Functional verification of DCS including
 2 detectors with final devices

 all main sub-systems and services

 Final validation of ALICE approaches for

 Back-end, FSM, configuration, archival, alarms

 Software & hardware interlocks

 Interface with ECS, DAQ, TRG, HLT

20
 Pre-commissioning at P2
DCS DSS BE
ON network sensors
DSS actuators
WN PC room
TPC
WN Console PLC
WN GWN
ACR SG
Console DCS Back-end CERN
ON App GW Backbone
HMPID WN DB-SRV network
Disk SRV
- DCS pre-commissioning starting mid April 06
- All TPC and
ON
HMPID & B-end DCSDCparts ready
CR3
ACORDE
WN sensors
DSS
ON actuators
CR4
MUON
WN
21
SXL2
 DCS final installation
- procedure -
 H/W and S/W infrastructure
 ACC install and verify network, PC’s (with PCI), and
base s/w
 Detector hand over to ACC
 S/W projects
 the detector PVSS project
 3rd party software with documentation and installation kits
 Front-end configuration and control tools
 Documentation (stored on DCS web pages)
 Description of the PVSS hierarchy
 Assignment of the managers to the detector computers
 Configuration of the OPC servers (where they should run,
what devices should be accessed)
 FED servers (where they should run)
 archival – what parameters are archived
 parameters to be stored in the conditions database
(including the datapoint names)
 FERO configuration description

22
 DCS final installation
- procedure -
 Verification
 ACC checks for obvious errors
 incorrect path names, incorrect external dependencies, violations
of the DCS conventions such as naming of the systems or DIM
services etc.
 For viruses
 For completeness of documentation
 Installation
 ACC uploads software to the installation server on DCS
network
 ACC + Detector expert install on target machines
 setting-up of the PVSS system
 integration into global DCS,
 configuration of database access and FW access control
 back-up
 Commissioning
 Detector expert validates functionality with devices
 Detector expert demonstrates functionality of alerts,
operational panels, FSM, etc.

23
 Detector final
installation
DETECTORS
planning
From To
(D time)
PHOS, TOF, TRD, HMPID, 10 July 1 Sept
ACORDE
TPC 4 Sept 2 Oct
SDD, SSD (ITS Barrel) 3 Oct 8 Nov
FMD, T0, V0 on C-side 9 Nov 28 Nov
SPD, SDD, SSD 29 Nov 10 Jan 07
TOF, TRD 31 Jan 6 Mar 07
07
FMD, T0, V0, PMD on A- 4 Apr 07 23 Apr
side 07
24
 DCS final installation planning

For each sub-system (in total ~ 100):

Pre-commissioning on surface (institute/CERN) who ready

 PVSS - OPC – h/w device connection AA date
 FSM, configuration, archival, alarms BB date
 s/w interlocks, operations panels CC date

Commissioning in the experimental area who ready

 s/w hand-over to ACC AA <D-4 weeks
 Installation & commissioning of devices BB <D-2
weeks
 Commissioning devices with detector CC D

D = detector installation
date
25
 Final installation
wee
ks

D- 4

D- 2

D
Mar Apr May June

Hand-over
TODAY

NETWORK
INST COMM
PC + BASE SYSTEM

CONTROL CABLES
INST COMM
AND DEVICES

DCS PROJECT INST COMM

INST
DETECTOR
COMM

26
 Final installation:
PHOS, TOF, TRD, HMPID,
ACORDE
Mar Apr May June 12 Jun 26 Jun 10 Jul

Hand-over
TODAY Lates
NETWORK
t
INST COMM
PC + BASE SYSTEM

CONTROL CABLES
INST COMM
AND DEVICES

DCS PROJECT INST COMM

INST
DETECTOR
COMM

27
 DCS
commissioninContr Room Safety
g Consoles DSS Services
CSAM Gas
responsibiliti CR facilities
Power
es Chochula Augustinus
EXTERNAL Cooling
SYSTEMS

Augustinus Augustinus
Magnets
ECS
Access
Interface
DCS Back-End
De Cataldo

Trigger
DAQ Chochula
Infrastr
HLT DCS H/W B-field
Offline Infrastructure
Space Frame
LHC Interface Env. Param.
Radiation
Augustinus + Chochula

Detectors
SPD SDD SSD TPC TRD TOF HMP PHO FMD T0 V0 PMD MTI MTK ZDC ACO
~100 sub-systems 28
 Testing and validating
DIP
 First use/commissioning of DIP
 CSAM data DCS pre-commissioning May
06
 MCS data DCS Magnet tests end
May 06
 DCS data LHC final commissioning end
06?
 Assistance from JCOP?

29
 Integration with other
on-line systems
 DCS ECS DAQ commissioning
 Done for 2 detectors (HMPID and SPD)
 Planned for TPC during pre-commissioning
June 06

30
 Global status today
(“dashboard” on DCS
SPD SDD SSD TPC TRD TOF
website) HMP PHO
CPV
µ- µ-
FMD T0 V0 PMD ZDC ACO EMC
Trig
DCS
ID S trk trg .
DET
CAEN iseg Iseg CAEN
HV CAEN iseg
Easy
iseg
+distr
CAEN CAEN
+V reg.
CAEN
+distr
CAEN CAEN CAEN CAEN CAEN CAEN CAEN Iseg

CAEN AREM CAEN ALICE CAEN CAEN CAEN Easy CAEN

LV Easy pro Easy
Wiener Wiener
box Easy
Wiener Wiener Wiener Wiener
Easy
CAEN
Easy +distr Easy
Wiener

Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener
VME VME VME VME VME VME VME VME VME VME VME VME VME VME VME

MO ELMB ELMB PLC ELMB ELMB ELMB ELMB ELMB ? DIM ELMB
N
FED JTAG DCS JTAG RCU DCS DRM RCU RCU DRM DCS RCU

Cool
GAS
Positio RackCtr
VHV Liquid LED GMS Thresh LED?
n l

Pulser Crystal Laser Align

Laser BField

Drift RadMo
Vel. n

Control with FSM Activity started, without PVSS Under definition

Basic control with PVSS No (SW) activity started 31
No (SW) activity started, but common effort
 5 “critical” detectors
HMP PHO µ- µ- Trig
SPD SDD SSD TPC TRD TOF CPV FMD T0 V0 PMD ZDC ACO EMC DCS
ID S trk trg .
DET
CAEN iseg Iseg CAEN
HV CAEN iseg
Easy
iseg
+distr
CAEN CAEN
+V reg.
CAEN
+distr
CAEN CAEN CAEN CAEN CAEN CAEN CAEN Iseg

CAEN AREM CAEN ALICE CAEN CAEN CAEN Easy CAEN

LV Easy pro Easy
Wiener Wiener
box Easy
Wiener Wiener Wiener Wiener
Easy
CAEN
Easy +distr Easy
Wiener

Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener
VME VME VME VME VME VME VME VME VME VME VME VME VME VME VME

MO ELMB ELMB PLC ELMB ELMB ELMB ELMB ELMB ? DIM ELMB
N
FED JTAG DCS JTAG RCU DCS DRM RCU RCU DRM DCS RCU

Cool
GAS
Positio RackCtr
VHV Liquid LED GMS Thresh LED?
n l

Pulser Crystal Laser Align

Laser BField

Drift RadMo
Vel. n

Control with FSM Activity started, without PVSS Under definition

Basic control with PVSS No (SW) activity started 32
No (SW) activity started, but common effort
 15 “critical”
subsystems
HMP PHO µ- µ- Trig
SPD SDD SSD TPC TRD TOF CPV FMD T0 V0 PMD ZDC ACO EMC DCS
ID S trk trg .
DET
CAEN iseg Iseg CAEN
HV CAEN iseg
Easy
iseg
+distr
CAEN CAEN
+V reg.
CAEN
+distr
CAEN CAEN CAEN CAEN CAEN CAEN CAEN Iseg

CAEN AREM CAEN ALICE CAEN CAEN CAEN Easy CAEN

LV Easy pro Easy
Wiener Wiener
box Easy
Wiener Wiener Wiener Wiener
Easy
CAEN
Easy +distr Easy
Wiener

Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener Wiener
VME VME VME VME VME VME VME VME VME VME VME VME VME VME VME

MO ELMB ELMB PLC ELMB ELMB ELMB ELMB ELMB ? DIM ELMB
N
FED JTAG DCS JTAG RCU DCS DRM RCU RCU DRM DCS RCU

Cool
GAS
Positio RackCtr
VHV Liquid LED GMS Thresh LED?
n l

Pulser Crystal Laser Align

Laser BField

Drift RadMo
Vel. n

Control with FSM Activity started, without PVSS Under definition

Basic control with PVSS No (SW) activity started 33
No (SW) activity started, but common effort
 Remarks
 If not started on HV and LV, then mainly because of no hardware (waiting for PS)
 LV wiener, when application exist, it is with CANbus version of PL500
 For several detectors the environment monitoring is still under definition
(number/type of sensors etc.)
 For HV/LV/ELMB detectors can profit from framework etc. For FEE more effort is
needed from detectors. Thus red or yellow in FEE is more worrying than red in
HV/LV
 For Crate control, cooling and gas an effort is expected from the ACC (thus less
work for detectors)

 The FEE for TOF,T0 is green/yellow: work on SW has started, some PVSS panels
exist, but it need to be clarified if this adheres completely to the FED concept
 The FEE for FMD is yellow/red: They started to use the RCU, but not really using
the ‘DCS part’ of it
 The Laser for TPC is green/yellow: all parts of the system have been worked on,
not all under PVSS
 The Vo is indicated mainly red because of lack of information
 The environment monitoring for EMC is not yet defined. There will be a LED
calibration system, the controls of it need to be clarified. EMC has experience
with PVSS (older version of their FEE), but not with the sub-systems listed here

34
 Progress
monitoring
 For Public
 Status dashboard on web
 For Experts
 Status panel on DCS top node accessed via
Terminal Server

35