Apple is overhauling how it tests software after a swarm of bugs marred the latest iPhone and iPad operating systems, according to people familiar with the shift.
Software chief Craig Federighi and lieutenants including Stacey Lysik announced the changes at a recent internal “kickoff” meeting with the company’s software developers.
The new approach calls for Apple’s development teams to ensure that test versions, known as “daily builds”, of future software updates disable unfinished or buggy features by default. Testers will then have the option to selectively enable those features, via a new internal process and settings menu dubbed Flags, allowing them to isolate the impact of each individual addition on the system.
When the company’s iOS 13 was released alongside the iPhone 11 in September, iPhone owners and app developers were confronted with a litany of software glitches. Apps crashed or launched slowly. Cellular signal was inconsistent. There were user interface errors in apps like Messages, system-wide search issues and problems loading e-mails. Some new features, such as sharing file folders over iCloud and streaming music to multiple sets of AirPods, were either delayed or are still missing. This amounted to one of the most troubled and unpolished operating system updates in Apple’s history.
“iOS 13 continues to destroy my morale,” Marco Arment, a well-known developer, wrote on Twitter. “Same,” replied Jason Marr, co-creator of grocery list app AnyList. “Apple’s really shown a lack of respect for both its developers and its customers with iOS 13.”
Complex
The issues show how complex iPhones have become and how easily users can be disappointed by a company known for the smooth integration of hardware and software. Annual software updates timed for release with the latest iPhones are a critical way for Apple to add new capabilities and keep users from defecting to arch-rival Android. Refreshed operating systems also give developers more tools for app creation, catalysing more revenue for Apple from its App Store.
Apple spokeswoman Trudy Muller declined to comment.
The new development process will help early internal iOS versions to be more usable, or “livable”, in Apple parlance. Prior to iOS 14’s development, some teams would add features every day that weren’t fully tested, while other teams would contribute changes weekly. “Daily builds were like a recipe with lots of cooks adding ingredients,” a person with knowledge of the process said.
Test software got so crammed with changes at different stages of development that the devices often became difficult to use. Because of this, some “testers would go days without a livable build, so they wouldn’t really have a handle on what’s working and not working”, the person said. This defeated the main goal of the testing process as Apple engineers struggled to check how the operating system was reacting to many of the new features, leading to some of iOS 13’s problems.
Apple measures and ranks the quality of its software using a scale of 1 to 100 that’s based on what’s known internally as a “white glove” test. Buggy releases might get a score in the low 60s whereas more stable software would be above 80. iOS 13 scored lower on that scale than the more polished iOS 12 that preceded it. Apple teams also assign green, yellow and red colour codes to features to indicate their quality during development. A priority scale of 0 to 5, with 0 being a critical issue and 5 being minor, is used to determine the gravity of individual bugs.
The new strategy is already being applied to the development of iOS 14, codenamed “Azul” internally, ahead of its debut next year. Apple has also considered delaying some iOS 14 features until 2021 — in an update called “Azul +1” internally that will likely become known as iOS 15 externally — to give the company more time to focus on performance. Still, iOS 14 is expected to rival iOS 13 in the breadth of its new capabilities, the people familiar with Apple’s plans said.
The testing shift will apply to all of Apple’s operating systems, including iPadOS, watchOS, macOS and tvOS. The latest Mac computer operating system, macOS Catalina, has also manifested bugs such as incompatibility with many apps and missing messages in Mail.
Some HomePod speakers, which run an iOS-based operating system, stopped working after a recent iOS 13 update, leading Apple to temporarily pull the upgrade. The latest Apple Watch and Apple TV updates, on the other hand, have gone more smoothly.
Apple executives hope that the overhauled testing approach will improve the quality of the company’s software over the long term. But this isn’t the first time that Apple engineers have heard this from management.
Last year, Apple delayed several iOS 12 features — including redesigns for CarPlay and the iPad home screen — specifically so it could focus on reliability and performance. At an all-hands meeting in January 2018, Federighi said the company had prioritised new features too much and should return to giving consumers the quality and stability that they wanted first.
Tiger Teams
Apple then established so-called Tiger Teams to address performance issues in specific parts of iOS. The company reassigned engineers from across the software division to focus on tasks such as speeding up app launch times, improving network connectivity and boosting battery life. When iOS 12 came out in the spring of 2018, it was a stable release that required just two updates in the first two months.
That success didn’t carry over to this year. The initial version of iOS 13 was so buggy that Apple has had to rush out several patches. In the first two months of iOS 13, there have been eight updates, the most since 2012 when Federighi took over Apple’s iOS software engineering group. The company is currently testing another new version, iOS 13.3, and there’s already a follow-up in the works for the autumn.
About a month before Apple’s 2019 Worldwide Developers Conference in June, the company’s software engineers started to realise that iOS 13, then known internally as “Yukon”, wasn’t performing as well as previous versions. Some people who worked on the project said development was a “mess”.
By August, realising that the initial iOS 13.0 set to ship with new iPhones a few weeks later wouldn’t hit quality standards, Apple engineers decided to mostly abandon that work and focus on improving iOS 13.1, the first update. Apple privately considered iOS 13.1 the “actual public release” with a quality level matching iOS 12. The company expected only die-hard Apple fans to load iOS 13.0 onto their phones.
The timing of the iOS 13.1 update was moved up by a week to 24 September, compressing the time that iOS 13.0 was Apple’s flagship OS release. New iPhones are so tightly integrated with Apple software that it would have been technically impossible to launch the iPhone 11 with iOS 12, and since 13.1 wasn’t ready in time, Apple’s only choice was to ship with 13.0 and update everyone to 13.1 as quickly as it could.
While the iOS 13 issues did upset iPhone owners, they still updated fairly quickly. As of mid-October, half of all Apple device users were running a version of iOS 13, according to Apple. That upgrade pace is still far ahead of Google’s Android.
Once iOS 13.1 was released, Apple’s software engineering division pivoted to iOS 13.2 with a quality goal of being better than iOS 12. This update has had fewer complaints than its predecessors in the iOS 13 family but did introduce a short-lived bug around apps closing in the background when they shouldn’t.
“iOS 13 has felt like a super-messy release, something we haven’t seen this bad since iOS 8 or so,” Steve Troughton-Smith, a veteran developer of Apple apps, wrote on Twitter. — (c) 2019 Bloomberg LP
