You’re a $2.8bn storage supplier with flash arrays at the core of your business. How do you do research and development (R&D), test new products, test customer workload issues, and test array products over years-long timescales for issues that only arise as software, network and application changes concatenate and interact?
Meanwhile, you are a global business with R&D and developer teams across time zones, all at work on ongoing monthly and quarterly updates, and incessant efforts to optimise storage controller software.
The base product is essentially the same, so effective collaboration and information sharing between teams spread across continents is key. But at the same time, you must also test regional customer-specific array configurations.
The Pure Storage solution is to divide responsibility between hardware and software, while also sharing specific R&D and testing capability between three sites.
There is Santa Clara in California, which is its global headquarters and handles hardware and software R&D and testing. There is also Bangalore in India, which only carries out software R&D and testing.
And there is Prague in Czechia, which recently opened its doors to the IT press. Here, we take a look at what goes on behind the scenes in R&D and product testing at Pure Storage (and its nearby array assembly operation).
Capabilities across the three centres are in many ways duplicated, which sounds counter-productive. But it’s not quite as simple as that, according to engineering vice-president and Pure’s Prague site leader Paul Melmon.
“Generally speaking, the same capabilities exist across all sites, except Santa Clara with its hardware development facilities,” he says.
“We try to make projects run autonomously and to minimise cross-time zone meetings,” says Melmon, adding that information can be shared globally in other ways, such as in Git repositories.
“Lots of companies split projects into many pieces and distribute them,” says chief technology officer Rob Lee. “We choose significant parts of individual products and give them to individual sites, and have product managers for specific products sitting locally.”
“We give a lot of thought to what to centralise and what to not,” says Melmon. “We have rules of engagement that aim at communications that can minimise the number of meetings.”
R&D, testing and talent in Prague
As mentioned, Pure’s Prague site is dedicated to software research, development and testing. It runs thousands of ongoing and custom test routines on hundreds of racks of software. These are divided into “persistent” and “non-persistent” testing, says engineering manager Tom Healy.
Non-persistent is ephemeral. It tests for issues in specific customer deployment configurations, or the impact of updates on controller code.
Persistent testing is long-term. It can be very long-term, in fact, with racks in place with, for example, generation upon generation of Pure Storage FlashBlade file and object storage deployed.
“Sometimes things can take years to occur,” says Healy.
“Our testbeds include, for example, FlashBlade capacity that dates back to the first generation [2016], some virtualisation, and a Windows application platform. All of this will run for years, to follow the lifecycle of customer systems and to test for the effect of changes to software and hardware, and its stability,” says Healy.
“And, FlashBlade uses Ethernet, so we are checking what happens when changes happen in the workload, simulation of new cabling, media, hitting it with broadcast storms, simulating signal degradation, etc,” he adds.
Meanwhile, at the Prague facility, hundreds of engineers work constantly on storage array software to meet ongoing monthly and quarterly updates.
Pure’s Prague R&D facility has just celebrated its five-year anniversary. It is resident in the Amazon building (no relation) and others in the riverside Karlin district. There it employs 600 people – 50% Czech, 50% from elsewhere – with up to 50 nationalities on-site.
Prague was chosen as a European centre because of its proximity to so many of Pure’s customers, but also, says Melmon, because of the availability of talent. “It’s on a level with Silicon Valley,” he says, and its accessibility in terms of transport links, universities, graduates in computer science, cost of living and general likeability of the city.
Speaking of the River City complex in which Pure is located, Melmon describes it as having a “South of Market” feel, referring to the fashionable area of San Francisco that became a honeypot for startups in the 1990s. “It’s the place to be if you’re in tech and AI [artificial intelligence]. There are meetups in the evening. It’s the cool new place to be in Prague.”
But it’s not just a cool place to work and live. Melmon points to the 19.7% figure, which is the proportion of revenue Pure spends on R&D.
Prague is the biggest Pure Storage R&D centre outside the US and has delivered about a third of its FlashArray product development. Meanwhile, FlashBlade//S was jointly designed and tested there, while key elements of the Pure Fusion workload management platform and its Pure1 AIOps were developed in the Czech capital. Meanwhile, 100% of Portworx Data Services and Pure’s disaster recovery as a service (DRaaS) offering came from there too.
That’s the result, with Prague as a key pillar in the three-site R&D and testing strategy of Pure Storage.
Nearby, also in Czechia, is one of its global assembly centres, which you can read about here.
