Acanac still down (24 Feb 08)

[[H]ackerK]

I guess if you are on Acanac, you may not see this message... But seems like their PPP server is still down, and there was a power outage last night at 151 Front St....

[Jaytee]

Im on Acanac and I appear to be fine.

I was out for a few hours yesterday around lunch time, first time I was down in a while actually.

[sickcars]

Must just be on Acanac's equipment since many more isp's have their equipment there like teksavvy etc & their connection did not go down.

it probably was not a power failing but maybe one of their routers or something like that.

[AcanacInc]

Acanac is not down at this time. Teksavvy is not located in the same floor as us. The power did go off in the entire building, but it did not affect everyone. The generator simply did not kick in for our portion.

Please submit a ticket to support to find out why your internet connection is down.

Best Regards,
Paul
www.acanac.ca

[AcanacInc]

If one of Acanac's core routers went down I would say so. I would not make up a story about a power outage.

It looks like Netfirms.ca is on the same floor.


This is Netfirms own statement.


Dear Customer,

We experienced a service interruption affecting the Netfirms corporate websites and some of our customer hosted websites and e-mail services.

During scheduled power maintenance at our Data Centre on Saturday Feb. 23 at approximately 10:30 AM ET, the building's backup generator system unexpectedly failed, impacting network connectivity. This affected several Internet and Hosting Providers, including Netfirms.

We would like to inform you that your web data is intact and that your e-mail messages are currently queued for delivery over the next several hours.

At this time, our Technology Team is also investigating the cause of the unusual failure with the Data Centre facility in order to prevent further incidences of this nature.
We sincerely apologize for this service interruption and appreciate your patience while our engineers work diligently to restore services.

Regards,

Richard
Netfirms Inc.

You can find some more info at the following URL
http://www.webhostingtalk.com/showthread.php?p=4975520

Best Regards,
Paul
www.acanac.ca

[champlinD]

Acanac is not down at this time. Teksavvy is not located in the same floor as us. The power did go off in the entire building, but it did not affect everyone. The generator simply did not kick in for our portion.

Please submit a ticket to support to find out why your internet connection is down.

Best Regards,
Paul
www.acanac.ca

And Paul I say What the puck. It shows the poor infrastructure and/or testing procedures at your end.

Acanac should be ashamed.

EDIT: So does Netfirms Inc.

[st7860]

guess they forgot to buy a ups

[AcanacInc]

I really don't think you understand how everything works. UPS's are only good for a few minutes at best. The generator is suppose to kick in after a few seconds. In this case in did not. Power was only restored after about 35 minutes. Long after the UPS's have run dry.

Also we have no control of the back end infrastructure at 151 front street. This is one of the most state of the art buildings in all of Toronto if not in Canada. As you can see even at such a place things can go wrong.

At our main data center here in Mississauga we do regular test with our Generator and Transfer switch. The reason we are able to do them is because we own all the equipment. This is not the case at 151 front street. The equipment is owned by the landlord..

Are you also saying that Netfirms Canada's biggest web hosting company has poor infrastructure. Or if power would of affected teksavvy floor instead of ours. Would teksavvy have poor infostructure?


Power went out. Nothing could be done until it came back up.


Best Regards,
Paul
www.acanac.ca

[st7860]

guess he/she/it never heard of sizing a ups to the load.

[AcanacInc]

By your comments I believe you really don't understand how large data centers work.



You also now seem to think this only affected Acanac an Netfirms. This not the case. If I have some time later on I will post a few more companies that were affected on that specific floor. I just need to ask for approval first. Some might not want to make this information public. B

Best Regards,
Paul
www.acanac.ca

[st7860]

netfirms has nothing to do with your/the company you work for

your and/or the company you work for isn't 'new' and certainly isn't 'tiny'. if the people responsible for the data centre knew what they were doing this wouldn't happen.

now, things like Craigslist crapping out for a day(it happened) just because someone fed the generator bad gas makes sense because its a huge company. people don't communicate. but you work for a company that isn't tiny but yet isn't that big, and so hopefully management listens to IT once in a while.

[AcanacInc]

I believe your asking us to put enough battery arrays to sustain a significant power outage. This is what the generators are for.

A good example would be our Mississauga location. We are currently housing over 600 servers. Some of them pushing as much as 3000watts. We would need hundreds of battery arrays to sustain any significant power outage without a generator.


Like I stated before. We do not own or manage the generators at 151 front street. Neither does Netfirms. They are owned by the land lord. This outage affected hundreds if not thousands of servers. I don't believe any of the companies on the six floor own the generators.


It's unfortunate that the generator failed at 151 front street. What we need to do now is find out why this happened. We also need to find out how it can be prevented in the future. If we are not happy with the answers we can always find another company to rent space from, but like I stated before this location is one of the most sophisticated buildings in Canada.

Best Regards,
Paul
www.acanac.ca

[st7860]

I believe your asking us to put enough batteries arrays to sustain a significant power outage. This is what the generators are for.

A good example would be our Mississauga location. We currently housing over 600 servers. Some of them pushing as much as 3000W We would need hundreds of batteries arrays to sustain any significant power outage without a generator.


It's unfortunate that the generator failed at 151 front street. What we need to do now is find out why this happened. We also need to find out how it can be prevented in the future. If we are not happy with the answers we can always find another company to rent space from, but like I stated before this location is one of the most sophisticated buildings in Canada.

Best Regards,
Paul
www.acanac.ca

well, there you go. you have over 600 servers. why not lease a small building, renovate it, and put your servers in there, with generators that you own.

[Kasakato]

well, there you go. you have over 600 servers. why not lease a small building, renovate it, and put your servers in there, with generators that you own.

Do you have any idea how much that would cost?

[AcanacInc]

This is exactly what we are doing now in our Mississauga location. We control the entire infrastructure. We are also located at one 151 front street because all the big transit providers are also at this location. I would say most companies that have there own data centers like us also have a presence at 151 front street.

We then use a local loop to send the data to our Mississauga location. This is a fiber line that runs about 18km. Our DSL side does not even come to our Mississauga location. All the important equipment is located at 151 front street. Why take the risk of having the Fiber cut by some construction work.

I hope this explains everything.

Best Regards,
Paul
www.acanac.ca

[[H]ackerK]

Sadly, my Acanac account is still not working. It keeps refuse my PPP connection.

I even tried my backup (which is a d-link router preconfiged and tested with a standalone DSL modem) still no good.

e-mail'd tech support, but they didn't even read what I asked. :(

I guess I will have to live w/o internet for a while... And will try to call tech support tmr morning... sigh.

[hehehaha]

well, there you go. you have over 600 servers. why not lease a small building, renovate it, and put your servers in there, with generators that you own.

it is not only the servers that need the power, but when you are running a room with that much servers, you also require tons of AC to cool it down....

[st7860]

i was over simplifying it.
what i meant was

why not lots of batteries for just the core router, and the router that leads to the PPPOE servers, and whatever else is necessary to connect DSL customers to the internet(fibre equipment and so on).

that can't be more than 15 or so pieces of equipment,not 600. hopefully.

[AcanacInc]

When you count the Switches, Routers, LNS Servers, Radius servers, DNS Servers, Fiber cards and media converters your looking nearly 40 peaces of equipment. One would need a significant amount of battery backup power to last any reasonable amount of time. In a situation like yesterday it would of not made a difference either way. Our entire suite in the six floor lost power. This also means that our Fiber lines, PRI's and Cat5e wires lost power at the same time. Even if our servers were running you would still be down.

I am not saying you can't make it nearly full proof, but at what cost. We have been at 151 front street for about 5 years now and this is the first time this has ever happened to us. When everything works down time is kept to a minimum. Batteries give the generator more than enough time for it to start. This is taken care of by a piece of equipment called a transfer switch.

If any one is interested I am more than willing to give a small tour of our Mississauga location. I can book the appointment during one of our biweekly tests. You will be able to see the transfer switch in action and the generator kick in within about 8 seconds. Please keep in mind compared to 151 front street our data center in Mississauga is insignificant, but the technology is almost identical.


Best Regards,
Paul
www.acanac.ca

[AcanacInc]

Sadly, my Acanac account is still not working. It keeps refuse my PPP connection.

I even tried my backup (which is a d-link router preconfiged and tested with a standalone DSL modem) still no good.

e-mail'd tech support, but they didn't even read what I asked.

I guess I will have to live w/o internet for a while... And will try to call tech support tmr morning... sigh.


Please send me a PM with your DSL user name.

[Digital_Domain]

why not lots of batteries for just the core router, and the router that leads to the PPPOE servers, and whatever else is necessary to connect DSL customers to the internet(fibre equipment and so on).

that can't be more than 15 or so pieces of equipment,not 600. hopefully.
Aside from the amount of equipment used to actually maintain a ISP grade network (which is huge), batteries are limited by one factor - weight. You can have all the batteries you want to power all the equipment you want but the building has a physical limit as to how much weight it can support. I know for a fact that a certain large accounting firm in downtown Toronto has battery backup for it's servers but due to weight limits on the floor couldn't add more batteries to the array.

Any word yet as to why the generators failed?

[bembem]

And Paul I say What the puck. It shows the poor infrastructure and/or testing procedures at your end.

Acanac should be ashamed.

EDIT: So does Netfirms Inc.

guess they forgot to buy a ups

guess he/she/it never heard of sizing a ups to the load.

netfirms has nothing to do with your/the company you work for

your and/or the company you work for isn't 'new' and certainly isn't 'tiny'. if the people responsible for the data centre knew what they were doing this wouldn't happen.

now, things like Craigslist crapping out for a day(it happened) just because someone fed the generator bad gas makes sense because its a huge company. people don't communicate. but you work for a company that isn't tiny but yet isn't that big, and so hopefully management listens to IT once in a while.

well, there you go. you have over 600 servers. why not lease a small building, renovate it, and put your servers in there, with generators that you own.

i was over simplifying it.
what i meant was

why not lots of batteries for just the core router, and the router that leads to the PPPOE servers, and whatever else is necessary to connect DSL customers to the internet(fibre equipment and so on).

that can't be more than 15 or so pieces of equipment,not 600. hopefully.
The sheer amount of stupidity in this thread is astounding.

Honestly, if you have no idea how ISP's work, and can't realize that Acanac is NOT to blame in this case, and is taking OWNERSHIP of the issue, then you should put a lid on it.

The only viable solution to maintaining power to a data center during power outages is/are gas powered generators.

[bujanx]

i was over simplifying it.
what i meant was

why not lots of batteries for just the core router, and the router that leads to the PPPOE servers, and whatever else is necessary to connect DSL customers to the internet(fibre equipment and so on).

that can't be more than 15 or so pieces of equipment,not 600. hopefully.

Oh come now, your just trolling right?

You can't actually believe what you posted? =)

[st7860]

then you should put a lid on it.


you first