#Control Room - Main Console { <!-- if time is up, go to game over tile automatically (without user interaction or click)--> (if: $timeRemaining <=0) [ (live: 1s)[ (go-to: "GameOver") ] ] <!-- if temp target reached, go to success tile automatically--> (if: $reactorTemp <= $targetTemp) [ (live: 1s)[ (go-to: "Success") ] ] } The HMI displays the Reactor Overview screen, and red alarms are flashing across multiple systems. { <div class="consequence error"> -**Time Remaining:** (print: $timeRemaining) minutes <br> -**Reactor Temperature:** (print: $reactorTemp) C <br> -**Cooling System Status:** (print: $coolingSystemStatus) </div> } (if: $alarmsSilenced is false)[ The alarm is deafening. You can barely think with all the noise. ] **What do you do?** <!--only show silence alarms command if the value is false--> (if: $alarmsSilenced is false)[[Silence the alarms|SilenceAlarms]] [[Check the alarm status table|AlarmTable]] [[Open the cooling system detail screen|CoolingSystemDetail]] [[Review recent tag history|TagHistory]] [[Check Gateway device connections|GatewayCheck]] (if: $rootCauseFound is true)[ [[Attempt to restore cooling system|RestoreCooling]] ] {<!--reduce time remaining & set alarmsSilenced to true--> (set: $timeRemaining to $timeRemaining -1) (set: $alarmsSilenced to true) # Alarm Management } You select all the alarms and hit the Acknowledge button on the alarm status table. The klaxon stops, and you breathe a sigh of relief. The silence lets you think through the problem. (if: $skillsUsed does not contain "Alarm Management") [ (set: $skillsUsed to it + (a: "Alarm Management")) { <div class="consequence success"> **Skill Acquired:** Alarm acknowledgement procedures </div> }] In Ignition, alarm acknowledgement won't fix the problem, but it does indicate that the operator is aware and responding. <!--Link should disappear after click--> (link: "Ignition User Manual: Alarm Status Table")[(open-url: "https://www.docs.inductiveautomation.com/docs/8.3/appendix/components/perspective-components/perspective-display-palette/perspective-alarm-status-table")] <!--video--> <div class="video-container"> <strong> Recommended Training: </strong><br> Before moving on, you may want to check out <em> Inductive University: Displaying Current Alarms</em><br> <a href="https://inductiveuniversity.com/videos/displaying-current-alarms-perspective/8.3" target="_blank" >Watch Video</a> </div> <!-- end of video container --> [[Return to main console|ControlRoomMain]]{ (set: $timeRemaining to $timeRemaining - 1) # Alarm Status Table } You open the alarm status table. Multiple alarms are listed with timestamps, and you reorder the table so you can see the timeline of events: **Placeholder forActive Alarms:** { <div class="consequence error"> **HIGH priority** <br> Cooling/Pump&#95;A/Status <br> Pump A offline <br> Time: 08:12:15 </div> <div class="consequence warning"> **MEDIUM priority** <br> Cooling/Flow&#95;Rate <br> Flow below minimum <br> Time: 08:12:18 </div> <div class="consequence alert"> **LOW priority** <br> Cooling/Valve&#95;CV101/Position <br> Valve not at setpoint <br> Time: 08:12:20 </div> <div class="consequence error"> **HIGH priority** <br> Reactor/Temperature <br> Temperature above 180C <br> Time: 08:14:23 </div> <div class="consequence warning"> **MEDIUM priority** <br> Reactor/Pressure <br> Pressure trending high <br> Time: 08:15:01 </div> } The timestamps tell a story. The cooling pump went offline first, then flow dropped, and then the temperature started rising. Alarm Tables in Perspective show real-time alarm data and can be sorted and filtered to make diagnosing problems easier. { (if: $skillsUsed does not contain "Alarm Analysis") [ (set: $skillsUsed to it + (a: "Alarm Analysis")) <div class="consequence success"> **Skill Acquired:** Reading alarm sequences to identify root causes </div> ] } This confirms that the cooling system is causing the temperature problem, but why did Pump&#95;A go offline? **Where should we go next?** [[Investigate the cooling system|CoolingSystemDetail]] [[Check Tag History for Pump&#95;A|TagHistory]] [[Return to Main Console|ControlRoomMain]] { (set: $timeRemaining to $timeRemaining -1) # Cooling System Detail Screen } You navigate to the Cooling System detail screen in Perspective. The screen shows: **Pump&#95;A: ** (color: "red")[ **OFFLINE**] **Pump&#95;B: ** (color: "green")[ **STANDBY**] **Flow Rate: **(color: "red")[ 0 ] **Valve CV-01: ** (color: "red")[45% open ] The normal flow rate should be 250 and the valve percentage open should be 100%. Pump A should be running, and Pump B is configured as a backup, but B hasn't automatically switched over. **What do you do?** [[Start Pump B Manually|StartPumpBFast]] [[Check why Pump A failed|PumpADetails]] [[Check the pump control logic|PumpControlLogic]] [[Return to Main Console|ControlRoomMain]]{ (set: $timeRemaining to $timeRemaining - 1) #Tag History - Cooling System <!--Link should disappear after click--> (link: "Ignition User Manual: Time Series Chart")[(open-url: "https://www.docs.inductiveautomation.com/docs/8.3/appendix/components/perspective-components/perspective-chart-palette/perspective-time-series-chart")] (link: "Ignition User Manual: Configuring Tag History")[(open-url: "https://www.docs.inductiveautomation.com/docs/8.3/ignition-modules/tag-historian/configuring-tag-history")] <!--video--> <div class="video-container"> <strong> Recommended Training: </strong><br> Before moving on, you may want to check out <em> Inductive University: Tag Historian in Ignition</em><br> <a href="https://inductiveuniversity.com/courses/ignition/tag-historian-in-ignition/8.3" target="_blank" >Visit Course Page</a> </div> <!-- end of video container --> } You open Tag History and look at the last hour of data for the cooling system tags. { The trend chart shows: <div class="consequence error"> 08:12:15 AM - Pump A/Running: TRUE --> FALSE <br> 08:12:18 AM - Flow&#95;Rate: 250 --> 0 <br> 08:12:20 AM - Valve&#95;CV101: 100 --> 45 <br> 08:14:23 AM - Reactor/Temperature: 150 --> 185 </div> } { (if: $skillsUsed does not contain "Trend Analysis") [ (set: $skillsUsed to it + (a: "Trend Analysis")) <div class="consequence success"> **Skill Acquired:**Using Tag History for troubleshooting </div> ] } The data confirms that Pump A failed first, causing everything to cascade. **What's next?** [[Investigate Pump A failure|PumpADetails]] [[Return to main console|ControlRoomMain]]{ (set: $timeRemaining to $timeRemaining - 1) # Gateway Status - Device Connections } You open the browser and navigate to the Device Connections page in the Ignition Gateway. PLCReactor1: (color: green)[Connected] PLCCooling1: (color: green)[Connected] VFD-PumpA: (color: red)[Disconnected] - Connection timeout VFD-PumpB: (color: green)[Connected] There it is! The VFD for Pump A is disconnected from the Gateway. { (if: $skillsUsed does not contain "Gateway Diagnostics") [ (set: $skillsUsed to it + (a: "Gateway Diagnostics")) <div class="consequence success"> **Skill Acquired:** Using Gateway information to diagnose device connection issues </div> ] } **What do you do next?** [[Try to reconnect VFD-PumpA|ReconnectVFD]] [[Switch to Pump B strategy|PumpBPath]] (if: $coolingSystemStatus contains "RUNNING")[ You've successfully activated Pump B. The system is recovering. <!-- if temp target reached, go to success tile automatically--> (if: $reactorTemp <= $targetTemp) [ (live: 10s)[ (go-to: "Success") ] ] ](else:)[ (set: $timeRemaining to $timeRemaining -2 ) We don't have a clear restoration path yet. Let's [[keep investigating|ControlRoomMain]] ]#Emergency Shutdown Activated ##(color: red)[**TIME EXPIRED**] You did your best, but time ran out, and the reactor temperature reached critical levels. The safety system triggered an automatic shutdown. An incident investigation will need to take place. The emergency shutdown will result in a product loss of $50,000 and a 48-hour restart procedure. { <div class="consequence success"> Skills Demonstrated: (for: each _skill, ...$skillsUsed)[ - _skill ] </div> } { <div class="consequence alert"> ## Learning Suggestions: - Focus on quick wins and try to prioritize tasks. <br> - Get to know the system architecture: potential failure points, built-in failsafes, and backup systems <br> - Use built-in diagnostic tools where you can <br> - Consider physical limitations when you might have to work around broken parts or other hardware <br> - Get creative and be willing to balance safety factors and acceptable risks - code and configuration isn't necessarily set in stone. </div> } [[Try Again|Introduction]]# Crisis Averted! { <div class="consequence success"> **Final Status:** <br> Time Remaining: (print: $timeRemaining) minutes <br> Reactor Temperature: (color: green)[(print: $reactorTemp) C] <br> Cooling System: Pump B running <br> Shutdown Prevented: YES </div> } The reactor temperature is back in the safe zone, and production continues on schedule. You call the mainteanance team to report: { <div class="consequence warning"> - VFD-PumpA has lost communication - needs investigation <br> - Valve CV-101 is stuck at 45% - requires immediate repair or replacement <br> - Pump B is running on override script modification (needs to be set back to original script after repairs are completed) </div> } ## Performance **Crisis Response Evaluation:** { <div class="consequence success"> **Time Management: **Used (print: 15 - $timeRemaining) of 15 minutes <br> <br> **Skills Demonstrated:** (for: each _skill, ...$skillsUsed)[ - _skill ] <br> <br> **Number of Corrections Attempted: **(print: $correctionsAttempted) <br> <br> **Number of Mistakes Made: **(print: $mistakesMade) <br> <br> **Your Achievements:** <br> - Prioritizing multiple simultaneous alarms <br> - Inspecting live tag values <br> - Diagnosing problems with OPC connections <br> - Analyzing time-series trends with Tag History <br> - Scripting <br> </div> <div class="consequence alert"> **Notes:** <br> This crisis highlighted the importance of: <br> - Backup systems <br> - Understanding safety settings <br> - Gateway diagnostic tools <br> - Physical vs software troubleshooting <br> - Creative problem solving </div> } In a real-world scenario, you'd probably also document the script changes or use version control to make reverting to the previous scripting more seamless, create work orders for physical repairs, review settings and update procedures in light of lessons learned. ## Congratulations! You saved the day! [[Play Again|Introduction]]# Control Room Crisis The alarm sound cuts through the quiet morning. You’re the only operator on shift at the XYZ Chemical Plant, and every screen is lighting up red. { <div class="consequence error"> **CRITICAL ALARM: Reactor Temperature Rising** <br> **WARNING: Cooling System Failure** <br> **ALERT: Pressure Approaching Limit** </div> } You have 15 minutes to stop the reactor from reaching critical temperature and triggering an emergency shutdown. That means: { <div class="consequence warning"> - $50,000 in wasted product <br> - 48 hours of downtime <br> - A lot of explaining to do </div> } { (set: $timeRemaining to 15) (set: $reactorTemp to 185) (set: $targetTemp to 150) (set: $coolingSystemStatus to "FAILED") (set: $alarmsSilenced to false) (set: $rootCauseFound to false) (set: $correctionsAttempted to 0) (set: $mistakesMade to 0) (set: $skillsUsed to (a:)) } **Time Remaining: ** 15 minutes **Reactor Temperature:** (print: $reactorTemp) C **Target Temperature: ** (print: $targetTemp) C [[Begin Crisis Response|ControlRoomMain]] { (set: $timeRemaining to $timeRemaining -2) (set: $mistakesMade to $mistakesMade +1) #Manual Override - Pump B Start } You click the "Start" button for Pump B. This kind of manual start will override the automatic control logic, but the situation is deteriorating fast. After a moment, the pump status indicator shows: { <div class="consequence error"> **Pump B Start Failed - Lock Active** </div> } The error message reads: "Discharge valve CV-101 not fully open. Safety lock is preventing pump start to prevent equipment damage." Two minutes lost. **Now what?** [[Check Pump A details|PumpADetails]] [[Return to main console|ControlRoomMain]] [[Investigate the valve issue|ValveInvestigation]] { (set: $timeRemaining to $timeRemaining -1) #Pump A - Status Details } You view the tag details for Pump A and notice that the Variable Frequency Drive (VFD) that controls Pump A has lost its connection: { <div class="consequence error"> **VFDConnected: ** FALSE </div> } You know that the VFD monitors sensors for flow, pressure, and level to automatically start/stop, regulate speed, and protect the motor from overloads. Pump A is probably fine, but there's probably a communication issue between the PLC and the VFD. [[Check Gateway device connections|GatewayCheck]] [[Try to restart the communication|RestartComms]] [[Focus on getting Pump B running|PumpBPath]]{ (set: $timeRemaining to $timeRemaining - 1) #Pump Control Logic - Gateway Script } You open the Designer and navigate to Gateway Events. { <div class="consequence alert"> Tag change script **CoolingPumpControl** <br> ''Placeholder pseudocode for now:'' <br> if pump running == False (use readBlocking to check) <br> check if pump A has been stopped for > 30 seconds <br> if PumpAStopTime >30 <br> verify valve is at least 50 % open before starting Pump B <br> get valvePos <br> if valvePos >= 50 <br> set pumpB command to true and mode to BACKUP&#95;ACTIVE with tag.writeBlocking </div> } **Note: ** The auto-switchover requires valve CV-101 to be at least 50% open. Currently, it's only at 45%. { (if: $skillsUsed does not contain "Script Analysis") [ (set: $skillsUsed to it + (a: "Script Analysis")) <div class="consequence success"> **Skill Acquired:** Reading Python scripts to understand control logic </div> ] } The valve should have opened automatically when Pump A failed, so something is wrong with the valve control too. [[Investigate the valve|ValveInvestigation]]{ (set: $timeRemaining to $timeRemaining - 1) # Valve CV-101 Investigation } You look at the cooling system screen and see the following: { <div class="consequence error"> **Valve CV-101 Status:** <br> - Current Position: 45% <br> - Mode: AUTO <br> - Control Signal: 100% </div> } The valve is being told to open 100%, but it's only at 45%. The valve is likely stuck for some physical reason. However, you have a Manual Override option for the control logic. This option carries some risk. The valve is nearly halfway open, and it might provide enough flow to prevent immediate damage. You could modify the pump start script to bypass or change the valve position check and force Pump B to start anyway. **What do you do?** [[Modify the script|BypassValveCheck]] [[Try to command the valve to open again|CommandValve]] [[Return to the Main Console|ControlRoomMain]]{ (set: $timeRemaining to $timeRemaining - 2) (set: $mistakesMade to $mistakesMade + 1) # Attempting Communication Restart } You try disabling and re-enabling the VFD-PumpA device in the Gateway. After waiting for the connection timeout period, it's still disconnected. This doesn't look like a configuration issue. It looks like the VFD hardware isn't responding on the network. Unfortunately, we're running out of time. [[Switch to PumpB|PumpBPath]]{(set: $timeRemaining to $timeRemaining - 1) # Pump B Activation } It looks like Pump A can't be quickly restored, so Pump B is our best option. The interlock error is a challenge, since we can't start Pump B until certain conditions are met. Industrial control systems use interlocks for safety, so we need to look at the logic before forcing overrides. What should you investigate next? [[Check the control logic script|PumpControlLogic]] [[Investigate why CV-101 valve isn't open|ValveInvestigation]] { (set: $timeRemaining to $timeRemaining-1) (set: $correctionsAttempted to $connectionsAttempted+1) # Reconnecting VFD - Pump A } You click the reconnect button next to VFD - Pump A in the Gateway device connections list. { **Status: ** <div class="consequence warning"> Reconnecting... </div> } **After 30 seconds:** { <div class="consequence error"> Disconnected </div> } It looks like the connection problems are due to some kind of connection timeout. Since the VFD isn't responding, it could be: { <div class="consequence warning"> - VFD powered off <br> - Unplugged cable <br> - VFD hardware failure <br> - Switch issue </div> } You don't have time to troubleshoot the physical equipment tied to Pump A and the VFD. You need to get the cooling flow restored using Pump B. (set: $rootCauseFound to true) Sometimes field issues do require physical intervention. As an operator, you may need to delay addressing the primary issue and find a temporary workaround with available backup equipment. [[Switch focus to Pump B|PumpBPath]] [[Return to Main Console|ControlRoomMain]]{ (set: $timeRemaining to $timeRemaining - 2) (set: $correctionsAttempted to $correctionsAttempted + 1) # Script Modification - Emergency Override } You open the script editor and modify the CoolingPumpControl script: { <div class="consequence alert"> pseudocode placeholder: <br> Emergency Override - reduced valve threshold? <br> with readBlocking - if Pump A running value is false <br> get valve position with readBlocking <br> if valve position is >=40 (change) <br> command Pump B to start and change mode to BACKUP&#95;ACTIVE </div> } You save the script. The change goes into effect immediately. { (if: $skillsUsed does not contain "Emergency Scripting") [ (set: $skillsUsed to it + (a: "Emergency Scripting")) <div class="consequence success"> **Skill Acquired:** Emergency script modification under pressure </div> ] } { <div class="consequence success"> **PUMP B STATUS: ** (color: green)[RUNNING] **Flow Rate: ** 180 </div> } The flow is lower than normal, but the reactor temperature is beginning to stabilize! { (set: $coolingSystemStatus to "PUMP B RUNNING") (set: $rootCauseFound to true) } [[Monitor the temperature recovery|Temperature Recovery]] { (set: $timeRecurring to $timeRecurring - 1) (set: $correctionsAttempted to $correctionsAttempted +1) # Valve - CV-101 } You try telling the valve to open multiple times, switching between AUTO and MANUAL modes. The valve position remains at 45%. This looks like an issue where the valve is stuck or malfunctioning. You're losing time, and the reactor temperature is still climbing. { <div class="consequence error"> Current Temperature: (print: $reactorTemp + 10) C </div> } It's time to make a decision about script bypass or modification options. [[Modify Script|BypassValveCheck]] [[Return to Main Console|ControlRoomMain]]{ (set: $timeRemaining to $timeRemaining - 3) (set: $reactorTemp to $reactorTemp - 15) # Temperature Recovery in Progress } You watch the reactor temperature tag data in real time: Time + 1 minute: (print: $reactorTemp + 10) C --> (print: $reactorTemp +8) C Time + 2 minutes: (print: $reactorTemp + 8) C --> (print: $reactorTemp +5) C Time + 3 minutes: (print: $reactorTemp + 5) C --> (print: $reactorTemp) C The temperature is dropping! Even with reduced flow from Pump B, it appears to be working. { <div class="consequence success"> **Current Status** <br> - Time Remaining: (print: $timeRemaining) minutes <br> - Reactor Temperature: (print: $reactorTemp) C <br> - Cooling Flow: 180 <br> - Valve CV-101: 45% </div> } (if: $reactorTemp <= $targetTemp)[ [[Crisis averted|Success]] ](else:)[ [[Continue monitoring|TemperatureRecovery2]] ] { (set: $timeRemaining to $timeRemaining -3) (set: $reactorTemp to $reactorTemp -20 ) # Continued Monitoring } The cooling continues. In the Tag Browser, you see: Reactor/Temperature: (print: $reactorTemp) C Cooling/Flow&#95;Rate: 180 The reactor pressure is decreasing rapidly, and the alarms begin clearing in the Alarm Status Table. Only the pump and valve faults are left. Those will need maintenance to be resolved fully. (if: $reactorTemp <= $targetTemp)[ [[Success!|Success]] ](else:)[ The temperature continues dropping steadily... (set: $reactorTemp to 150) [[Success!|Success]] ]