01 The Hour
14:35
Friday 1 June 2018. British Summer Time.
The data centre operations team in the United Kingdom saw a partial degradation in the processing system. Not an outage. A degradation, and the rate of it moved about all afternoon.
A component inside one of two core switches at the primary data centre had failed in part rather than outright. Because the failure was partial, the backup switch beside it did not take over, and the primary site took far longer to isolate than it should have. Meanwhile it went on sending synchronisation messages to the second data centre, which built a backlog there and slowed that site down as well.
02 The Sequence
1 June, 14:35 The partial degradation was seen at the data centre. Visa later told the Treasury Committee that at no point in the incident was there a full system outage, and that the proportion of transactions failing fluctuated throughout.
14:45 A Technical Response Team was convened and the incident escalated to the relevant European support teams.
15:05 to 15:15 The first peak, ten minutes long. An average of 35 per cent of transactions failed to process for holders of cards issued in the United Kingdom.
15:19 to 16:00 An assessment call at 15:19. The chief executive for Europe learned of the incident at 15:51. The full Crisis Management Team was mobilised at 15:56 and met at 16:00.
16:05 The first statement went to the media, ninety minutes after the first indication of a systems issue, saying that a service disruption was preventing some Visa transactions in Europe from being processed. It went to client-facing staff at the same time, to help them handle enquiries.
17:40 to 18:30 The second peak, fifty minutes at the same average rate, across the end of the working day. Shops put up cash only signs. A pub in Bermondsey stood almost empty on a Friday night with such a sign on the bar. In a Primark on Alexanderplatz in Berlin customers queued twenty minutes to pay, while the shop next door was processing cards without trouble. A company supplying card machines to some 60,000 independent businesses in Britain and Ireland told its users that contactless had a better chance of going through.
19:10 to 20:15 It took until about 19:10 to fully deactivate the system causing the failures at the primary site, after turning off software applications there and clearing the message backlog at the second site by hand and by machine. By then the second centre had begun processing almost all transactions normally. The impact was largely resolved by 20:15.
2 June, 00:45 Both data centres were processing at normal service levels. The disruption had run ten hours and ten minutes.
4 to 19 June 2018 The chair of the Treasury Committee wrote on 4 June. The reply of 15 June, with an appendix, was published by the committee on 19 June.
8 March 2019 The Bank of England used its statutory powers to direct Visa Europe to implement in full the recommendations of the independent review, and to require the appointment of a third party to assess progress. The Bank said the action did not imply the breach of a regulatory requirement and was not enforcement action.
03 The Hidden Dependency
A transaction counted as failed in two circumstances: the system did not answer inside the time limit, generally ten to fifteen seconds, so the transaction timed out, or it was declined in error by the system. On that definition, 9 per cent of transactions on cards issued in the United Kingdom failed on the cardholder's first attempt.
The letter then gives a second figure. Many cardholders and merchants put the card through again, many of those attempts processed properly, and counting them cuts the 9 per cent by roughly half. The count of affected cards moves the same way, from 1.7 million to about 1.1 million.
Part of the recovery ran through the person at the till. Which of the two numbers describes the afternoon depends on who tried twice.
Part of the recovery ran through the person at the till.
04 What Still Worked
The Second Data Centre
Each of the two sites was built to handle all of Visa's European processing on its own. The second site kept processing throughout, slowed by the backlog, and was handling almost everything once the primary was shut off.
The Other Schemes
Apart from a limited gateway service, under which fewer than a thousand transactions were affected, Visa does not process other schemes. It estimated that 40 per cent of debit cardholders in the United Kingdom also carry a MasterCard or American Express credit card.
The French Route
The French Association of Bank Users said France was not affected. Its secretary general told AFP the likely reason was that in most European countries merchants connect directly to Visa when a payment is made, while in France the payment passes through an intermediary.
05 Contested
Contested 1 to 2 June 2018
Whether cash machines were caught in it is not agreed in the record of the day.
AFP reported that the fault left Visa struggling to process transactions at bars, shops and cash machines, and that some customers were forced to queue at machines. The Royal Bank of Scotland help account said on the Friday that cash machine withdrawals were unaffected, and that machine and MasterCard transactions were not impacted. Visa's resolution statement, issued at 04:32 on the Saturday, said account holders could again use Visa for purchases and at machines as they normally would.
06 The Pattern Library
1 When one retail withdrawal channel closes or fails, demand concentrates on whichever channel is left open, and the capacity of that channel sets what a depositor can obtain.
Observed: Northern Rock 2007, Cyprus 2013, India 2016, Greece 2015, United States 1933, Argentina 2001, Lebanon 2019, Worldpay 2026, TSB 2018.
2 A currency replacement can require physical modification of each cash dispensing machine before the new notes can be issued, so the machine estate returns to service site by site rather than at once.
Observed: India 2016.
3 Withdrawal limits imposed on a domestic banking system can be written to exempt cards issued outside it, so one machine dispenses under two rules according to where the card was issued.
Observed: Greece 2015.
4 Domestic payment and deposit channels can continue operating while cross-border payment routes fail, with the central bank standing in as the intermediary for payments to and from the country.
Observed: Iceland 2008.
5 When the institutions that convert payment instruments into cash are closed, wages and checks already received cannot be spent, and merchants and carriers extend their own credit in place of the missing conversion.
Observed: United States 1933.
6 A public employer short of the national currency can pay wages in an instrument it issues itself, which then circulates alongside the currency it stands in for.
Observed: Argentina 2001.
7 Withdrawal restrictions can be imposed by individual institutions and their trade association rather than by statute, so the limit differs from bank to bank and no law governs it.
Observed: Lebanon 2019.
8 A cash withdrawal ceiling set in nominal units has to be raised alongside the note denominations that pay it, because the ceiling can only be dispensed in the notes that exist.
Observed: Zimbabwe 2008.
9 A payment outage can hit card authorisation and merchant funding as separate faults, so acceptance at the terminal is restored while the money owed to the merchant arrives a day late under its own incident notice.
Observed: Worldpay 2026.
10 A retailer that loses electricity can go on trading by taking cash and rationing goods by hand, so the counter outlasts the checkout system that normally prices and settles the basket.
Observed: Texas 2021.
11 A systems migration can leave account records readable but wrong, including presenting one customer's account to another, so a working login is no guarantee that the balance on the screen belongs to the person reading it.
Observed: TSB 2018.
12 The size of a card processing failure can be counted twice, once on first attempts and once after the same card was presented again, and the second count can be about half the first, so part of the recovery runs through the cardholder at the till.
Observed: Visa Europe 2018.
Changed today: No existing entry gained Visa Europe 2018. The record for this case does not establish a second retail channel whose capacity set what could be obtained, a separation of authorisation from merchant funding, or a power failure, and a weak match would be worse than none. Entry 12 added, on a card failure counted twice, once on first attempts and once after retries.