Right before the end of the workday on Thursday, October 14 approximately 2 million mobile telephone accounts from NTT’s docomo mobile communications network suddenly had no service, and they were down for approximately 6 hours.
This large-scale service disruption was reminiscent of the succession of incidents involving frozen ATMs at the Mizuho Bank, Japan’s third largest financial services company.
As Japan’s corporate giants stumble, new entrants and market disrupters are sometimes able to leverage such opportunities to grab new business.
One of Japan’s most popular mobile telephone operators is the “docomo” network from Nippon Telegraph and Telephone (NTT). The term “docomo” literally means “everywhere” in Japanese, and it implies that the service “works everywhere all the time.” It certainly did not live up to its brand promise when the docomo service disruption caused havoc throughout the country on a Thursday evening earlier this month.
Although the docomo service disruption was caused by something a little more complicated than when a cat chews through a power cord, it turns out that a technical glitch was to blame.
A System-wide “Issue”
Beginning at 5 pm (JST) on Thursday, October 14 approximately 2 million mobile telephone accounts from NTT’s docomo service stopped working throughout Japan.
The disruption affected voice calls, data communications, and the emergency call network. Some users were unable to register their location themselves. Their phones simply displayed that they were out of range.
Docomo engineers immediately set to work to fix the problem by restricting access for more and more users. The company gradually eased the restrictions on its network and ended the outage at 7:57 p.m. later that same evening, but intermittent trouble continued for some customers for many more hours.
Although many of the affected users were probably unable to see these messages when they were posted due to a lack of service, NTT did do a good job of issuing a steady stream of updates and apologies via Twitter.
docomo’s voice and data communication services have been difficult to use since 5:00 p.m. on Thursday, October 14, 2021, and we have been controlling the network, but the problem was resolved at 7:57 p.m. on Thursday, October 14…We sincerely apologize for any inconvenience this may cause to our customers.
It turns out that the service disruption was not limited to docomo.
Trouble for Low-Cost Operators (MVNO)
Many customers of Japan’s low-cost, mobile virtual network operators (MVNO) also experienced service outages and were unable to send text messages. These services piggyback on NTT’s infrastructure. Thus, their troubles were, naturally, sorted out after NTT repaired its own mobile communications network.
Why was this such a big deal?
The mobile telephone network in any country is now recognized as a critical infrastructure. Especially since the 9.0 magnitude Great East Japan Earthquake in 2011 during which cell phones ceased working for several hours because of overwhelming demand, Japanese people are very sensitive to widespread mobile telephone service disruptions.
This time there were, for example, nervous parents who were unable to track the whereabouts of their children. Some users were even unable to reach 110, Japan’s equivalent of the emergency 911 hotline service in the U.S.
Somewhat similar to how businesses around the world suffered only a few weeks ago when Facebook went down due to a technical glitch, the docomo service disruption also affected many businesses that rely upon their mobile technology. Besides more “mission critical” tasks, there were, for example, reports of users of on-demand rental bicycles who were unable to return their rides.
It is no surprise that this most recent outage caused rancor among NTT’s customer base and customers of downstream businesses that rely upon their network.
I can’t find the delivery address! Uber deliveryman screams at Docomo communication failure.
I suppose that the Uber customer never got their dinner on that fateful Thursday!
Fear of a Cyber Attack
Some people like Maya immediately feared the worst. She tweeted,
The communication failure made me think again that we need to allocate a better budget to take steps against cyber attacks.
I think Japan’s awareness of the danger of cyber attacks is really low.
While it turns out that this particular problem was not due to a cyber attack, it is only the latest in a series of high-profile “incidents” that are causing the public to lose confidence regarding the reliability of the providers of essential services.
By October 19 the Ministry of Internal Affairs and Communications (MIC) determined that docomo’s large-scale communication failure 5 days earlier fell under the category of a “serious accident” under the law. It has requested that by November 13 the company provide a detailed report of what happened and what they are going to do to prevent a similar incident in the future. The Telecommunications Business Law defines a “serious accident” as a situation where 30,000 or more users are unable to use communications for more than one hour in a communications service that handles emergency calls such as 110. The docomo service disruption easily qualified under this definition of an accident.
Other High-Profile Service Disruptions
Perhaps the poster child in Japan for service disruptions that involve large-scale businesses is the Mizuho Bank, which is the 3rd largest financial services company in Japan.
Serial Outages That Lock ATMs
This bank simply cannot catch a break. On September 8 its customers were locked out of automatic teller machines (ATM) for the 7th time yet this year! Only a month earlier the same bank suffered from similar disruptions throughout its national network of ATMs.
Although Mizuho seems to be under a particularly dark cloud this year, it has a history of such trouble, including widespread service outages in 2002 and 2011.
The mobile telephone network and banking industries are certainly not alone.
Facebook’s Recent Worldwide Outage
As most readers are probably well aware, only 10 days prior to the docomo incident 2.7 billion global users of Facebook, Instagram, and Whatsapp experienced up to a 6 hour suspension of service.
Facebook’s technical problems underscored just how tethered all of us are to the internet and, as a result, just how dependent we are upon the backbone infrastructure managed by only a handful of global players.
So, what does this mean for the average consumer?
Opportunity for New Entrants and Market Disrupters
At the end of the day, while there really is not much that a typical, frustrated consumer can do when faced with such a disruption to service, it can be a wake-up call to seek alternatives.
Once considered to be the “gold standard” for reliability, the docomo service disruption is leading some to defect from NTT.
Mmmm. It was a good decision to change our contract carrier from docomo to SoftBank a while ago. It was a good decision. I can’t work if I can’t use it.
It is, however, not as though SoftBank is immune to technical glitches. Their network went down in 2019, and the company was severely chastised by Japan’s central government at the time.
When a major like NTT stumbles, opportunities for more nimble new entrants and market disrupters tend to appear.
Rakuten Mobile’s Relentless Assault on the Entrenched Establishment
Although they probably will not blatantly produce negative advertising highlighting the docomo service disruption, it is probably only a matter of time before Rakuten Mobile’s national network of pink-clad salespeople go on the offensive by tweaking their sales pitch, as they seek to pick off more of NTT’s existing customer base. They are already using price as a weapon, and the all-important attribute of reliability of service is now up for grabs.
Other MVNOs will probably follow suit, as well.
If you had first-hand experience with the docomo service disruption, it was probably frustrating at best and, possibly, frightening at worst. Given the extent to which we are all now so dependent upon mobile communication networks, occasional “incidents” like the docomo service outage that recently affected more than 2 million people in Japan are, unfortunately, likely to be a fact of life on a go-forward basis. Brace yourselves for more!
Mark Kennedy is a native of Chicago who has spent more than 20 years living, studying, and working in Japan. By day he is Country Head - Business Development, Nexdigm - Japan but becomes a writer after work. Mark is a lifelong student of the Japanese language and culture. He loves to travel throughout the country. Mark also is the author behind the "Real Gaijin" Substack, countryroadsjapan.com, as well as the Country Roads Japan and Coastal Sailing Japan YouTube channels. Photo supplied courtesy of the author who had stopped to check out the free-roaming horses and cows about half-way up to the summit of Mt. Aso, an active volcano in the center of Kyushu.