Major San Francisco power outage caused by one breaker in one substation – what are the implications

At approximately 9am on April 21, 2017, ONE breaker failed in PG&E’s Larkin Street substation. This ONE breaker in ONE substation brought the city of San Francisco to its knees.  The implications are numbing, particularly considering all of the promises PG&E has made to the California PUC after the San Bruno natural gas pipeline rupture.

Given the walled enclosure, a physical attack such as the rifle attack against the PG&E Metcalf substation would not be possible. However, a cyber attack could certainly succeed. Not only could it succeed, a cyber attack could cause physical damage to the targeted breaker and/or transformer.

The Larkin Street substation is by NERC’s definition a transmission substation as it is more than 100KV (as seen by the figure readily available on the Internet, it is a 115 KV substation). NERC CIP Version 5 generally considers substations “Low Impact” unless they support transmission facilities at 500KV or higher, or operate between 200-499KV at a single substation.  As the Larkin Street substation is considered to be LOW IMPACT, there is little claimed need to address cyber security.

The breaker that failed was scheduled to be replaced next year. This is a similar story to the 2003 Northeast Outage when the SCADA system at First Energy had a known software bug and was to be replaced but wasn’t replaced in time and therefore “helped” cause the 2003 outage.

The PG&E breaker failure led to an outage affecting approximately 96,000 customers (per DOE Energy Assurance Daily Report). However, it should be noted that a large high-rise with 10,000 people can be considered one customer. Consequently, a better number of people affected by Friday’s outage is probably closer to 500,000 or more. The power outage disrupted San Francisco’s normally bustling financial district, home to many major multinational banks and technology companies. Wells Fargo closed 13 bank branches and four office buildings, while the New York Stock Exchange said its ARCA options trading floor in San Francisco was briefly unavailable. Employees in Goldman Sachs' financial district office were sent home. Baker Avenue Asset Management was in the middle of a trade when all of their systems went down. Employees in another state had to complete the transaction.  Approximately 25% of the traffic lights in San Francisco were affected and traffic was a disaster all day with pedestrians at much higher risk than normal. The pedestrian issue was exacerbated when the people left their buildings (at least those that were not trapped in elevators) and businesses shut down sending employees home, making the traffic mess even worse. All of this impact from the failure of ONE breaker in ONE LOW IMPACT substation.

When asked by a KCBS radio reporter how important is the Larkin Street substation, the PG&E representative’s answer was that all PG&E substations are equally important. The Aurora vulnerability can damage or destroy large, critical long lead-time equipment that could result in extended outages. Additionally, the information on Aurora was declassified by DHS making it publicly available. Several years ago, DOD proposed to do an Aurora hardware mitigation project with PG&E, but was not done. We do not know if the Larkin Street substation and other PG&E substations are susceptible to Aurora or have installed the requisite Aurora hardware mitigation.

Generally, older breakers need more maintenance than modern digital breakers. NERC should intensify its enforcement of its Protection and Control Standards (PRC) specifically PRC-005 Protection System Maintenance and PRC-019 Coordination of Generating Unit or Plant Capabilities, Voltage Regulating Controls, and Protection.  These standards were cited as part of a group of NERC’s most violated standards in its Compliance Monitoring and Enforcement Reports for 2016.

The outage impact from the breaker failure at the Larkin Street Substation calls into question PG&E’s transmission planning and its compliance with the NERC Transmission Planning Standards (TPL) concerning contingency planning through redundant breakers and transmission feed capabilities. The impacts from this outage should raise red flags at DOE, FERC and NERC about classification of Critical Infrastructure and Key Resources (CIKR) and Bulk Electric System (BES) assets concerning classification going beyond voltage bright-lines and KVA classification. This outage demonstrates that system reliability, key facilities, and economic impact should be considered during CIKR classification. The outage also demonstrates the limitations of several key NERC standards. The NERC BES Exception Request provides exceptions for defining a transmission asset - http://www.nerc.com/pa/RAPA/BES%20DL/BES%20Exception%20Evaluation%20Guideline%202-4-14%20REMG%20App.pdf . Exceptions E1 and E3 can be used to exclude substations over 100KV from being defined as transmission and therefore having to meet transmission requirements. This distinction should be of great interest to FERC and the California PUC as to whether the Larkin Street substation has been defined as transmission or distribution (it is shown as a transmission substation on the above map). The NERC CIP cyber security standards specifically exclude distribution from cyber security standards.

I will be giving a paper in June 2017 in San Francisco at the American Nuclear Society Conference on the Implications of the Ukrainian Cyber Attacks to Nuclear Plants. The paper concerns the Ukrainian cyber attackers remotely controlling the substation breakers. As a version of the malware from the Ukrainian cyber attacks has been in the US grids for years, cyber threats to substation breakers are a very real threat. Once an attacker has access to the substation breakers, potential damage to the substation and supported facilities is not far off including Aurora threats.

Joe Weiss

Show Comments
Hide Comments

Join the discussion

We welcome your thoughtful comments.
All comments will display your user name.

Want to participate in the discussion?

Register for free

Log in for complete access.

Comments

  • Interesting article as it shows what happened. It also shows that the area covered is very badly designed. No major system should ever be designed and built without a failsafe system. The loss of a single breaker should trigger a reroute of power via an standby breaker or alternate source of power. That sources can be a reroute of power or an auto start generator, (s) to provide power. Depending on the size of the grid or power consumption is should be such as to restore power in steps so another cascade trip does not occur. This is just a comment as without the big picture and a breakdown of the system this is just a comment.

    Reply

  • The Aurora susceptibility can damage or grind enormous, basic long lead-time hardware that might bring about increased blackout. Read more at Masterpaperwriters.com

    Reply

  • Get assignment help from our proficient experts andmaintain a good academic record. Our scholars will not only help you complete all your assignments within deadline but also guide you to enhance the quality of your academic papers. https://myassignmenthelp.com/get-assignment-help.html

    Reply

  • OZ Assignment Help offering affordable online financial assignment writing help with experts assistant. Our experts are available for 24*7 to support students and resolve their query in assignment writing. Get assignment help with expert team of academic writers and make assignment more effective to achieve good grads in exams or presentations. For more help you can visit our work folio: http://www.ozassignmenthelp.com.au/solution/

    Reply

  • I simply couldn't leave your website before suggesting that I actually loved the usual info that you supply to us. I’m gonna be back continuously to check up your new posts. Keep Doing like this. Good luck!!! https://www.goassignmenthelp.com.au/law-assignment-help/

    Reply

  • Hi Having read your post to its end, it recalled me the days when I was so afraid of reading a long post. To be upfront I am still but the format and presentation of your post glued me up to its end. This is called writing a post that never bores the readers at any point. I appreciate that you mentioned Major San Francisco power outage caused by one breaker in one substation – what are the implications as the best way. Thanks a lot for sharing such a detailed post that is an ample proof how wholeheartedly you want to serve your readers in their best interest. Have a more successful journey ahead. Online Assignment Help Australia

    Reply

  • Just a smile and the rain is gone Can hardly believe it, yeah. There's an angel standing next to me. Reaching for my heart Just a smile and there's no way back .Can hardly believe it, yeah But there's an angel calling me. Reaching for my heart I know that I'll be okay now. This time, it's real I lay my love on you It's all I wanna do Every time I breathe I feel brand new You open up my heart Show me all your love and walk right through As I lay my love on you. dumb ways to die fireboy and watergirl 4

    Reply

  • Very well written article. It will be helpful to everyone who employee it, including me. Keep up the good work - can't wait to read more posts. content writing service

    Reply

  • I am big fan of such lovely little things that smell so good. I have numerous aromatic candles at home! It helps me to relax after my working days Assignment help | Need Assignment help

    Reply

  • Great post. I found your website perfect for my needs. Very useful info specifically the last part. Thank you for sharing with us, and we sincerely hope you will continue to update or post other articles happy wheels

    Reply

  • It would have been really dangerous if the problem was not identified at the early stage. The public were extremely worried about the issue and it looks like the problem has been rectified by the authorities now. Keep updated with us. cheapest cable tv

    Reply

  • Like a major city like San Francisco power outages could cause a standstill for everyone. If a major power outage could caused by one breaker in one substation, authorities should reconsider the backup plans and rerouting plans more seriously to avoid any similar incidents in future. Babylon City Tours

    Reply

  • Your blog is so amazing and helpfull.And it is the great idea. assignmentonline.co.uk

    Reply

  • The blog is really appreciable and i like to keep on visiting this site once again that it would help me in further thanks for sharing the info. Mac pro service center in Mumbai and Apple iphone 6S dealers in delhi and Macbook pro dealers in kolkata

    Reply

  • Nice! I want you to thank for your time of this wonderful read!!! Your blog posts are more interesting and informative. I very like and visit regularly. hill climb racing 2

    Reply

  • Thanks for sharing this information. I really like your blog post very much. You have really shared a informative and interesting blog post with people. transformice pirata

    Reply

  • Cyber security is becoming more crucial and that this is what most countries are afraid of than a physical attack. Most countries look forward to such cyber crime issues with great tension and taking more topline performance off road gear safety measures to keep their data and systems safe.

    Reply

  • Got the amazing stuff and i am pretty delighted to see this kind article that is awesome. I am so amazed to have this awesome post. Independent Escorts Service

    Reply

  • The Fairdeal Group of Companies is a fast-growing group with the ambition of helping people across Kenya build their dream homes and offices. Office Chairs Dealers And Dealers in metal beds Nairobi And Dealers in wooden dining tables

    Reply

  • The T-Series StageWorks Academy is an initiative designed to identify, nurture and create a pool of immensely skillful and passionate professionals who can match national and global demands. Dance academy in Noida And Acting school in Noida And Singing school in delhi ncr And Modelling school in noida

    Reply

  • Solutions Middle East, works to Envision, Execute and Empower our clients by offerings design, implementation and on-going support services. Nutanix Premier Partner and Staff Outsourcing and SMS Gateway providers and VDI in a box and Vmware partners and Riverbed Application

    Reply

  • DPS has a very good hostel for the students where they get homely environment. Students of various age groups and different places live in the hostel. They get opportunity to interact with diverse group of students and people. In hostel students develop their confidence and become independent. Best CBSE school in Rajasthan and CBSE Boarding School in NCR

    Reply

RSS feed for comments on this page | RSS feed for all comments