Nat West - Fuck Up extraordinaire

[harrybarracuda]

Haven't seen this posted yet, so thought I'd start a thread. This has to be a fuck up on a par with Sony being hacked and Blackberry going down - only worse, because people have been stranded abroad without money, left unable to pay bills, buy food etc.

Started last week, and yesterday they reported it was fixed:

NatWest 'technical glitch' fixed, says spokesman

One of banking's worst ever technical faults has been 'identified and corrected' with staff now working through payments backlog

• Saturday 23 June 2012 17.34 BST

NatWest is working to clear up the backlog of payments caused by a major 'technical glitch'. Photograph: Ben Stansall/AFP/Getty Images

A computer error that left thousands of customers at NatWest bank unable to access their money has been fixed, a spokesman said.
In one of the industry's worst ever technical faults, employers were also unable to make monthly salary payments.
The fault also disrupted payments from RBS and Ulster Bank. As well as being unable to access money, customers reported being unable to view accounts or see if bills had been paid.
The problem first started on Tuesday and the bank, owned by the Royal Bank of Scotland Group, said a "technical glitch" had resulted in the disruption. A spokesman for RBS Group said the fault had been "identified and corrected" and that staff were now working through a backlog of payments.
"Our number one priority is sorting through the backlog. The technical fault has been identified and corrected. We know the origin of the issue."
He said staff would be working through the night and the bank were doubling the number of phone staff.
The bank would not elaborate further on the cause of the problem or how long the backlog would take to clear.
Susan Allen, the director of customer services at the RBS Group, told the BBC she had organised for some money to be sent to a customer in the back of a cab.
"We've made it very clear that nobody would be out of pocket as a result of this error. We deeply regret the inconvenience caused to our customers and customers of other banks, and if people can get in touch we will make sure that we fix this for them. For those customers affected, please call us, please come into our branches."
She added: "Yesterday I organised to have some money go in a cab to one of our customers. We know this is terrible and we want to make sure we get money to as many people as possible."
NatWest had said the fault started as they began processing payments overnight on Tuesday, which had started to cause problems for customers on Wednesday. But the crisis escalated.
A thousand branches were kept open until 7pm on Thursday, opening again the following morning at 8am. The bank has announced that more than 1,300 branches would also open on the weekend, with some open until 6pm, to deal with the backlog.
Experts have suggested that the problem, which was described by the bank on Friday as "a system outage", may have been caused by a software update that was rolled out but backfired.
The Financial Services Authority said NatWest would be responsible for any charges customers were liable for, or interest they need to pay as a result of the bank's error, but would not be responsible for consequential losses, which could include missing out on a holiday or losing a home.

Then this morning their online banking crashed:

10:47am UK, Sunday June 24, 2012

Customers of NatWest bank are facing further problems with Sky News viewers reporting difficulties logging onto the bank's website as it struggles to fix technical problems that have brought financial chaos to millions.

Until now, customers had been able to log onto the website but were not able to view their statements or balances.
But in an apparent deterioration, some people are currently having trouble accessing the bank's website at all.
When attempting to log in, viewers have told Sky News they are met by a message which reads: "We're sorry but the service is temporarily unavailable. Please try again later."

In the Banking industry, loss of reputation is an absolute crippler: I wonder how much this will cost them, and I wonder how many of the other banks will be poaching their clients?

More to the point, I wonder if they will ever admit what caused it? If it was a security breach they will not wish to reveal it, if it's just a cock up, the sooner they come clean the better.

[sunsetter]

im with them, walked past yesterday, complete chaos inside, packed to the rafters,gave it a miss and checked online later, mines safe, no fcuk ups

[jizzybloke]

I wonder how much this will cost them,

They still couldn't give a toss though, plenty of mugs out there!

[nigelandjan]

Natwest banks open in the UK today ,, thats a first I think ,, mind you its a job to get in there in the weekdays the hours they open.

absolute joke a law unto themselves

[hazz]

I was unfortunate enough to have a 6 month tour of duty in their IT department a few years ago. I have yet to work in a place where the management of the place was as dysfunctional. for the entire time I was there they were being fined daily by the FSA for non compliance on one IT system/service or another. Non of the managers had a clue what was going in the whole place was simply out of control, but saying that it mostly hung together.

[OhOh]

It's the end of the world, take your money out now before they shut all the banks.

Head for the hills, they are stopping me shopping.

Time to load the gun.

TOO LATE .................................................

[Gerbil]

They have been having problems for at leas 2 weeks... I know because I was waiting on 2 international transfers that got stuck going through them.

Seems that their 'fixes' for those issues created more problems, which then affected other systems. I should imagine their CTO is shitting backs over his future prospects with the bank.

[Troy]

Okay...new software product tested to death...

Let's go live with it...

WTF!!! Why didn't we think of that????

....Do I have any sympathy? Nope...As with so many companies they prefer to employ cheap labour off the streets rather than experience....serves the f!ckers right....

[hazz]

well they do say often you get less than you pay for, but you never get more

I have to say that I will be deriving some pleasure from knowing some of those currently shitting themselves,whats the german word for that?

[OhOh]

gave it a miss and checked online later, mines safe, no fcuk ups

Did you manage to withdraw any cash, is your card still working?

hey prefer to employ cheap labour off the streets rather than experience.

Reports are that they fired their UK based IT staff and hired a new company of "consultants" in Asia.

[leemo]

Just a wild guess that a few hundred mills are in the process of disappearing during this "technical glitch".

[harrybarracuda]

The problem began on Thursday and is believed to have arisen after staff tried to install a software update on RBS's payment processing system, but ended up corrupting it.

I'm not sure what's worse - the lack/failure of a decent testing process or the lack/failure of a disaster recovery plan.

Useless bunch of wankers.

They've apparently laid off lots of their IT department, I bet it wasn't the useless fucking pointy haired managers that got the chop.

[lom]

I have to say that I will be deriving some pleasure from knowing some of those currently shitting themselves,whats the german word for that?

Schadenfreude, the best joy there is.

[nigelandjan]

Mr Hester head of the Natwest fiasco was on TV tonight saying sorry and its all cleared up ,,, er no none of us have still been paid and the Companies that supply our Company we work for haven't been paid ,, so now we are being refused materials.

What a pigs ear .

As we are paid monthly and our money for last month is now 3 days overdue some of the lads are in dire crap street ,, so for my part I am going round with the wonga offering pay day loans at mates rates 1000% APR


Every cloud / silver lining

[crippen]

Bank robbers should bin their masks and become bank bosses... the skills are the same
By MAX HASTINGS


Buying a house is the biggest and most nerve-racking financial transaction in most people’s lives. A friend of one of my children was due to complete on a little property in Sidcup, Kent, last Thursday.
There were five people in her purchase chain, all of whom had furniture being loaded on to vans when she received a phone call from her solicitor.
RBS’s computer system had crashed: the promised money for her house was not there. She and everybody else in the chain was left sobbing. One couple spent the night in a local Premier Inn; others had to pay the cost of holding furniture already packed into removal vans



Anguish: Millions of RBS and Natwest customers have had their bank accounts paralysed by an IT failure
On Friday afternoon, her solicitor managed to contact an RBS director who provided a guarantee for the money. Her deal, and those linked to it, belatedly went through.
Meanwhile, they had endured 24 hours of hell. So did millions of other customers of RBS — which owns NatWest — up and down Britain, who suffered days of anguish because money they were counting on to fund their lives had gone missing in the black hole of the bank’s computer system.
Yesterday, RBS’s boss Stephen Hester apologised, blaming a glitch during a software upgrade.
He should have started saying sorry almost a week ago: the bank has been responsible for a blunder on a scale that’s made its IT management seem dire (regardless of whether or not it has suffered a cyber attack).
Here is a bank 82 per cent owned by the taxpayer, which delivers a service to 17 million customers — and which has plumbed depths of inadequacy.
This institution distributed £390 million in bonuses to its investment bankers last year, despite group losses of £2 billion, while freezing the pay of 28,000 High Street staff.

Inadequate: RBS is 82 per cent owned by the taxpayer
The saga of the banking industry on both sides of the Atlantic exposes ever greater incompetence matched by ever-more iniquitous bosses’ greed.
Whatever enormities the bankers inflict on the rest of us, they themselves get steadily richer. ‘Has there ever been such an outrageous one-way bet in the history of the industrial age?’ demanded a critic this week.
RBS’s latest fiasco is making headlines


Read more: Bank robbers should bin their masks and become bank bosses...the skills are the same | Mail Online

[crippen]

The computer system that caused chaos for thousands was being supervised from India
Natwest has shed 30,000 jobs since taxpayer rescue in 2007, including 20,000 UK positions, and has outsourced work overseas
Insiders say many of the bank's support teams are now based in India, where they earn between £9,000 and £11,000 a year
Bank staff union Unite questions whether 'off-shoring' job cuts have left Natwest unable to cope with the current crisis
RBS denies that the outsourcing of staff has had an impact on its ability to cope with the unfolding transaction processing crisis
By VANESSA ALLEN
PUBLISHED: 22:28 GMT, 25 June 2012 | UPDATED: 22:54 GMT, 25 June 2012


The flawed computer programme that led to chaos for millions of RBS customers was being supervised by an IT support team in India, it was revealed last night.
Last February, RBS advertised for a series of key jobs, paying between £9,000 and £11,000 a year, in the Indian city of Hyderabad. That is way below what an equivalent worker would be paid in Britain.
The banking giant was urgently seeking computer graduates with several years experience of using CA-7, the programme which the bank uses to run its vast network of transactions and accounts.








Read more: The computer system that caused chaos for thousands was being supervised from India | Mail Online

[harrybarracuda]

^ Oh dear god, that explains it!

[Butterfly]

of course it was run out of India, anyone could tell immediately

[hazz]

Essentually it all comes down to how you view IT within your business, a necessary evil, a money sink rather like the office photocopiers... or does one see IT as part of the companies competitive edge and therefore a source of income and profit.

Clearly RBS/Natwest take the latter view that IT is cost that needs to be minimised at all cost, and they are currently discovering what 'at all costs' can mean. Its inevitable that customers who can find another bank are going to run. hopefully enough to cause a run, cause this is an organisation that clearly has problems learning the hard way, let alone the easy way.

[harrybarracuda]

Essentually it all comes down to how you view IT within your business, a necessary evil, a money sink rather like the office photocopiers... or does one see IT as part of the companies competitive edge and therefore a source of income and profit.

Clearly RBS/Natwest take the latter view that IT is cost that needs to be minimised at all cost, and they are currently discovering what 'at all costs' can mean. Its inevitable that customers who can find another bank are going to run. hopefully enough to cause a run, cause this is an organisation that clearly has problems learning the hard way, let alone the easy way.

If your core business is manufacturing, then you can do without your IT infrastructure for a week, let's face it.

But if your core business is moving money around with computers.....

I am just amazed people don't remember the constant truths such as "You pay peanuts, you get monkeys".

[hazz]

If your core business is manufacturing, then you can do without your IT infrastructure for a week, let's face it.

But if your core business is moving money around with computers.....

I am just amazed people don't remember the constant truths such as "You pay peanuts, you get monkeys".

It really does depend, from my experience any manufacturing involving a supply chain, a failure of core IT systems would hold production within a day and sometimes almost instantaneously. one system for GSK down time caused near immediate production line shutdown and the irricoverable loss of production worth 2 million pounds per hour, On the same system where an accidentally erased backup set caused a 40 million dollar recall.

But as you say, banking is essentially a business of selling services provided by computers. These IT systems are your core revenue earners and should never be viewed as an cost centre that drains money from the company.

[CaptainNemo]

Just shows how close western civilisation is to the precipice.

So eager to "develop" the east, and embrace the ruthlessness of globalisation, and just casually assume that there's no risk associated with that.

Economic development stems from cultural develoment. Much of the economically backward world is culturally/socially backward.

They've leapfrogged from a sort of mostly pre-industrial revolution state, to the 21st century without going through the industrial revolution and all the social and cultural change that went with it. Thinking that they can fast-forward to western development.

The west maybe in a mess at the moment, but it's a mess built on firm foundations; the gloss and gleam of development in parts of the developing world is build on very dubious foundations, and if the west tries to use the east as shortcut to growth, it could pay a heavy price when the east becomes top-heavy and tumbles.

[harrybarracuda]

Funnily enough, if you *do* have supporting applications that are critical to your manufacturing infrastructure, it gets a lot easier to justify things like backup data centres and hot-swap servers.

DoubleTake springs to mind....

[hazz]

^they do and the weird things that RBS does this too, to the point that to have 1TB or storage you consume 6TB of disk. What ever has happened at RBS/natwest, I recon its going to be a plane crash, a long line events of staggering incompetence each of which if handed correctly would have stopped the disaster.

[nigelandjan]

Got me wages today ,, only 4 working days late , because I bank with a non RBS group bank I got mine a day later than some of the lads at work who bank with the Halifax,, apparently they sorted out their own house first .