From Northern in an email -

 
Power failure, and as one would expect with it being the railway, back up generators failed.

Hence total chaos.

Good luck to anyone caught up in it.

 
So much for Andy Burnham having the power to run the whole country from Manchester.

Sorry for my sarcastic, personal opinion only, post.

CfN.

 
But he’s not in Manchester, he’s on holiday “overseas”.   

 

Louise Haigh's in charge but apparently she's not answering her phone 

 
Top BBC News story; live updates: https://www.bbc.co.uk/news/articles/cn5nlvdxpwpo

 
Please accept my apologies for introducing an element of levity to this topic.

 
Profound words, beautifully put with great insight!

"We've got to really understand what's gone wrong here, because clearly something has and we've got to make sure we investigate that thoroughly to understand why things haven't restored as we would have expected," said Chris Wright, Route Director NW at Network Rail.

 
I do not know the specifics of what back up power systems failed to operate correctly when the local DNO supply was interrupted at this Regional Control Centre (ROC).

These ROC's normally have 2 independent supplies from the DNO, (in much the same way as a hospital).  There are automatic change over panel / load transfer switches, these can take between 2 to 10 seconds to change over. Also there diesel generators which take up to 30 seconds to come on load.

The time delays are to cover for brown outs (A brownout is a drop in electrical voltage. Lights dim and devices keep running on less power. Power companies use them on purpose to stop a total blackout. Bad weather or high demand can also cause them by accident).  There are UPS's to cover for the time for the back-ups to come on line, because of the electrical load at these ROC and the fact there are backups the UPS's are sized to hold the full load for 2 to 5 minute's the UPS's are also used in normal service to condition the power supplies to the signalling systems.

(This from my limited knowledge of signalling systems)  Railway signalling systems are designed to fail safe, in the event of an unstable power supply the signalling system will revert to Danger.  Outlining Interlockings rely on data links back to the ROC whilst there are automatic signals there are many that rely on "talking" to the ROC if the comms is lost the local interlocking will go to danger, it is possible that many of these interlockings required a site visit by a signalling tech to reset them.
Also any local to the ROC track side signalling systems that are feed from the ROC, and by local this could be a large area going out several miles, the electric equipment for points, track circuits and axel counters have a "crowbar device" in there internal power supplies, these crowbars operate and short circuit blowing the integral power supply fuses when the power supply goes unstable which is quite common during a brownout, this will place the signals at danger (Red) or they go dark, a train driver regards a dark signal as danger (Red) signal.

From my working life dealing with a number of large signalling centres covering the SE Railway area what happened at Manchester is the worst nightmare for the NR Route power and signalling Engineers; it is something that the systems have their single points of failure identified and where possible designed out where this is not possible / practical to do process to manage the risk are in place and the systems are maintained and tested for such events.

But at the end of the day these are large complex systems with a number of single points of failure.

The NR Route and NR HQ will have already been working on what the root cause of yesterdays failure was, and in my previous roll at NR I would have been on a number of conference calls today and next week looking at the lessons learnt and putting action plans in place with the maintainers to inspect and test similar systems in my former area

 

Another 'Industry Leader' with a degree in 'The Statement Of The Bleeding Obvious'.

Also 'Lessons Must Be Learnt' would have taken up less space.

 
Continuing today......

BBC News - https://www.bbc.co.uk/news/live/cwy0d0wgy4xt
Train delays latest: Passengers facing disruption for second day after power outage - BBC News

 
Have they not heard of UPS units ?
Such units provide back up power, usually from batteries, without any perceptible break.
Run time is typically a few minutes, with a diesel generator for longer failures, the run time is then only limited by fuel supplies.

I also fail to see why it took so long to change from one DNO supply to an alternative DNO  supply. I would expect changeover within a few seconds.

 
How long does it take for signals to fail red? Seconds.

It took 90 of those for the backups to kick in. That's too long for all signals just to change again back to what they were before interruption.

Then something else blew which may not directly be owing to the initial power failure & needed investigation.

After all that, all they could do was switch it off & on again & do a full reset. Now all staff & stock is out of place.

 
A properly specified and installed UPS would have supplied power, without perceptible interruption, for 5 minutes or more.

 

It can actually take less than 90 milli seconds to cause electronic railway signalling systems a problem, that's why electronic systems have UPS.

I would expect by now the (Power) Distribution team in Manchester and the Route will know what failed to operate correctly when the DNO power failed