Jump to content

Photo

Why the site was down on tuesday.


  • Please log in to reply
9 replies to this topic

#1 TammyWright

TammyWright

    Admin & Owner

  • Administrators
  • 11,659 posts
  • Wedding Location:Dreams Los Cabos

Posted 26 July 2007 - 02:52 PM

Circuit breakers wipe out the Web / PG&E’s faulty equipment reveals the Internet’s vulnerability to a disruption of its power source



Quote:
Circuit breakers wipe out the Web
PG&E’s faulty equipment reveals the Internet’s vulnerability to a disruption of its power source
Verne Kopytoff, Chronicle Staff Writer

Thursday, July 26, 2007

All it took to wipe out some of the Internet's biggest sites Tuesday was some faulty PG&E electrical breakers that caused a blackout in downtown San Francisco.

Some of the Web's hottest destinations - Craigslist, Yelp, Second Life - were suddenly inaccessible from San Mateo to Singapore after back-up generators failed at the facility housing their computer equipment.

Although mostly fixed within 12 hours, the incident shows how easy it is to send major swaths of the online world to the dark ages. Sites that millions of people rely on can be knocked offline by freak accidents, not to mention major catastrophes, and this event served as a wake-up call to the executives that operate them.

"If the data center was that vulnerable to a power outage, what if something really catastrophic happened like an earthquake?" asked Derek Gordon, marketing vice president for Technorati, a search engine of blogs that was brought down for a couple of hours Tuesday after the blackout. "What does that say about the vulnerability of the Internet in the Bay Area?"

The troubles started when 365 Main, a key data center in downtown San Francisco that touts its "state-of-the-art electrical system," failed to get its backup generators started immediately after the power outage hit around 1:45 p.m. A number of companies that house their computer servers in the facility were suddenly offline, setting off a mad scramble to get the Web sites up and running.

Shoppers at RedEnvelope, an online retailer, couldn't buy monogrammed pillow soaps. Hipsters on Yelp, the review site, had to take a break from sharing their reports of fabulous and not-so fabulous restaurants. Users of online classified service Craigslist were out of luck in finding a second-hand futon.

The backup generators were turned on 45 minutes after the blackout started, a delay that 365 Main said it was still investigating yesterday. But it took some of the facility's customers anywhere from another hour to 11 hours to get their servers safely rebooted and their Web sites operational.

What the episode exposed is that some companies operate entirely from one data center, a decision described by some security experts as risky. In emergencies, such companies can't shift traffic to an alternative facility where they keep additional servers.

"There's all kinds of things that can happen from a power outage to a tornado to a backhoe," said Jason Needham, director of product management at F5 Networks, a Seattle company that sells software and equipment for data centers. "All these things seem far-fetched until they happen."

However, Needham said the trend is for companies to put all their eggs in one basket, so to speak, in an effort to save money. In fact, just hours before Tuesday's power outage, 365 Main put out a press release trumpeting the fact that RedEnvelope had moved all its operations to its facility and closed an unneeded center in the Midwest.

Data centers are usually designed with redundant equipment to ensure power during outages, earthquakes and floods. Backup electricity is supposed to kick in within seconds after an outage through a complex system that keeps servers humming without interruption.

Internet companies pay thousands of dollars, minimum, to house a few servers in such a secure environment. In most cases, the precautions work.

"The well-designed data centers, this should be handled pretty seamlessly," said Michael Foust, chief executive of Digital Realty Trust, a San Francisco company that owns 60 data centers globally, including one in San Francisco that was not affected by the outage.

However, the rare snafu or major disaster can create havoc, such as the Sept. 11, 2001, terrorist attacks or the flooding of New Orleans caused by Hurricane Katrina. In those cases, many companies were thrown offline.

Google, Yahoo and eBay all keep multiple data centers across the world, partly because proximity makes Web pages load faster and partly for security.

Jim Buckmaster, chief executive of Craigslist, which got back to normal operations after 11 hours of scrambling, said that his company is in the process of moving to a second center but that it will take a few more weeks.

Gordon, from Technorati, called opening several data centers ruinously expensive for thinly funded Internet startups, of which there are hundreds in the Bay Area. Only profitable companies can afford such an extravagance, he said, though he acknowledged that Technorati, which isn't profitable, is in the process of moving into a second facility.

Tuesday's outage "added to the sense of urgency," Gordon said.

"The lesson here is despite all of your planning and all of your promises, you are vulnerable."

E-mail Verne Kopytoff at vkopytoff@sfchronicle.com.

This article appeared on page C - 1 of the San Francisco Chronicle

Founder and Owner of BestDestinationWedding.com  /  Bride & Groom Fly FREE Promotion

 
Looking for a travel agency that specializes in Destination Weddings-CLICK HERE?  We are an award winning travel agency and you can read our reviews here.
 
Awards and recognition: 
Palace Resorts Winner "Top Travel Agent", "Top Homebased Travel Agency" PRO Platinum Agency (Top Award)     |    AMResorts Master Agent & Top Agency  
Karisma Hotels Diamond Level 5 (highest production), Top 10 Azul Weddings, Top Wedding Coordinator | Wright Travel & Karisma GIVC Benefits
Travel Impressions Best of the Best Globe Winner    |     Apple Vacations Golden Apple Winner  
sml_gallery_206696_17612_13451.pnggallery_206696_17572_8409.jpg    sml_gallery_206696_17572_1003.jpg
Vendor rules are HERE! Please read!  If you are a vendor and looking to advertise your business, please email ads@bestdestinationwedding.com.


#2 MsShelley

MsShelley

    VIP Member

  • VIP Member
  • 10,366 posts

    Posted 26 July 2007 - 02:56 PM

    all I knew was that I was going through withdraw!!! lol

    #3 TammyB

    TammyB
    • Sr. Member
    • 10,630 posts

      Posted 26 July 2007 - 03:03 PM

      Me too.. I had to email Ann and have her show me her wedding pics to keep me entertained.. :-)

      #4 NYJen

      NYJen
      • Sr. Member
      • 1,326 posts

        Posted 26 July 2007 - 03:05 PM

        Me too. I was getting the shakes!!!

        I ended up emailing Michelle to get a fix.

        #5 LCBride2007

        LCBride2007

          VIP Member

        • VIP Member
        • 24,768 posts

          Posted 26 July 2007 - 03:08 PM

          Quote:
          Originally Posted by TammyB
          Me too.. I had to email Ann and have her show me her wedding pics to keep me entertained.. :-)
          worked out well for me i guess - that's the morning i got my wedding pics! tammy and i had something to do anyway

          #6 TATrisha

          TATrisha

            VIP Member

          • VIP Member
          • 12,377 posts
          • Wedding Date:April 11, 2008
          • Wedding Location:Gran Bahia Principle Akumal
          • LocationWisconsin

          Posted 26 July 2007 - 03:12 PM

          Quote:
          Originally Posted by akh
          Although mostly fixed within 12 hours, the incident shows how easy it is to send major swaths of the online world to the dark ages. Sites that millions of people rely on can be knocked offline by freak accidents, not to mention major catastrophes, and this event served as a wake-up call to the executives that operate them.
          Tammy, was this a wake-up call? I COMPLETELY RELY on this site!!! :)

          ~Trisha~

          Destination Wedding Photographer & Travel Agent
          www.faberphotography.com - My latest destination wedding photos can be found HERE

          Disney Specialist  /  Bride & Groom Fly FREE Promotion

           
          Looking for a travel agent that KNOWS DISNEY?  We are an award winning travel agency and you can read our reviews here.
           
          Awards and recognition: 
          Palace Resorts Winner "Top Travel Agent", "Top Homebased Travel Agency" PRO Platinum Agency (Top Award)   |  AMResorts Master Agent & Top Agency  |  Karisma Hotels Diamond Level 5 (highest production), Top 10 Azul Weddings, Top Wedding Coordinator | Wright Travel & Karisma GIVC Benefits | Travel Impressions Best of the Best Globe Winner   |   Apple Vacations Golden Apple Winner  
          sml_gallery_206696_17612_13451.pnggallery_206696_17572_8409.jpg    sml_gallery_206696_17572_1003.jpg
          Vendor rules are HERE! Please read!  If you are a vendor and looking to advertise your business, please email ads@bestdestinationwedding.com.


          #7 StephanieMN

          StephanieMN
          • Sr. Member
          • 1,798 posts

            Posted 26 July 2007 - 03:18 PM

            Yeah I actually had to work! Not cool man, not cool.

            #8 boscobel

            boscobel
            • VIP Member
            • 24,213 posts

              Posted 26 July 2007 - 03:23 PM

              Well, thank goodness I was home sick on Tuesday! I might have had to actually WORK at work!

              #9 superez

              superez
              • Newbie
              • 106 posts

                Posted 10 January 2008 - 02:51 AM

                Is this why the search function was and still isn't working? Any idea when it'll be back! I'm so dependent on it these days!

                Thanks :)

                #10 LCBride2007

                LCBride2007

                  VIP Member

                • VIP Member
                • 24,768 posts

                  Posted 10 January 2008 - 11:56 AM

                  Quote:
                  Originally Posted by superez
                  Is this why the search function was and still isn't working? Any idea when it'll be back! I'm so dependent on it these days!

                  Thanks :)
                  this post is about 6 mos. old, and the search function just stopped working recently. so they're not related! also, remember, you can always use advanced search, or just navigate to the forum sub-category manually.




                  0 user(s) are reading this topic

                  0 members, 0 guests, 0 anonymous users