Lessons learned from Volkswagen scandal or why should you care about AFA storage »test drive«?

I'm pretty sure we've all read about the Volkswagen scandal and how »resourceful« they were at using a software to make their cars with diesel engine perform way better than they really do, on a emission test. So what does that have to do with AFA storage systems or any storage system?

Truth of life - don't be misled by the big industry names, »name-your-favorite-analysis-firm« reports and quadrants, vendor devil advocates when deciding what to choose for your primary storage. Step out of the old way of thinking (you know… “I need that many of such drives in RAID6 …”). Speaking from my own "lessons learned".

Have you ever read datasheet and thought lies, all lies…?

If not, you should. Getting back at AFA storage systems – lots of datasheets list irrelevant or misleading tech specs. Those are mostly a beauty contest, vanity fair numbers and have no relation to real world system behavior (hint AFA »car emissions« a.k.a. the IOPS). And consequently with your application needs and user requirements.Even worse, it is exactly the same as in the car industry with millage per gallon (liter per km) rates – what specs tell is different from what we experience later when using & driving that car.So rule of thumb is – do a test drive. Get a test system and do a PoC with your data and your applications. Wait, wait… I hear “easier said than done” complaints. NO, not really. It really is easier than you might think and it does not take half a year of testing, loading, unloading data, monitoring, analyzing...You can get away with such test drive in a week or even shorter time.And it will give you the results. It's like doing the car test drive, though here the difference is you really can get a real-life »millage per gallon« rates which is kinda impossible with car test drive since you cannot have it that long to get relevant statistics.

Common sense is a flower that does not grow in everyone’s garden

Oh yes, it can get even worse than datasheet specs!Why you ask? Just because of that, you need to ask the right questions i.e. test the AFA system against meaningful scenarios. Your virtual server or VDI environment cares less about the millions and millions of IOPS the system is capable of serving if the storage latency "jitter" is high. Your database responsiveness translates that into excellent or lousy application user experience.So, even though you do a test drive, you still might get misleading data. It’s the same not-so-old story as with Volkswagen – they had software that »adjusted« engine behavior on emission test to make it more appealing. It's like all those Photoshop images of celebrities, when you ask yourself »how come they look that great?«. Guess what, they do not.Neither does an AFA system when testing it with vendor selected tool (hint – common sense sometimes does not grow in their garden ;). Even worse, since they might run synthetic tests tailored to make their system perform great with bogus data, make them “A student” at some corner cases, which have nothing to do with real life.

If a turtle doesn't have a shell, is it naked or homeless?

Perhaps it's  not a turtle. Point being - ask relevant questions.It really boils down to these two “simple” questions:

1. How will your applications behave?
2. What will be the user experience?

The answer is NOT simple, BUT it never is and that's why we (IT guys and gals) have our jobs.So… to do the job properly - whoever is planning, designing, budgeting … a new storage system – first know you applications and user requirements, next plan the PoC and do test drive.If you have a VDI environment – do test how applications perform when you have hundreds of them, not just a single one (and yes it can be automated). Test what happens when a server fails and 100 VMs need to failover (not doing a 100 VM/server yet… you should… get AFA system ;)). Unplug one of the controllers to see if it really no-disruptively fails over to the second one (oh so it takes like couple of minutes and everyone complains they were just kicked off…).Long story short – make a plan of relevant test, define the desired outcome, setup a test environment, do make use of automation tools (there are plenty of and one can do wonders), run the test, use real data and applications and :) if possible abuse your users for testing (possibly without even letting them know that they got on a test system – this will give the best user experience feedback… otherwise everyone will start complaining about sth not working just because they know sth is being tested…).

Ever felt you did everything right but still all went wrong?

Having faith is great, but start questioning and doubting what your storage vendor’s says. It will make your life much easier. ‘Cos, once you’ve bought a system it’s usually pretty hard to change mind in 4 months’ time when “hell breaks loose” and you start spending nights and weekends compensating the gap between datasheet promises and the harsh reality of what you have at your disposal. Do listen to your own common sense and gut feeling.Keep lovin' your storage, but make that a bi-directional relationship ;) Don't be a »giver« only. Demand love form you storage by keeping you free on weekends, releasing you of night shifts. If you'd like to question and doubt datasheets or need help at how to perform a »test drive« in a short amount of time with proper test plan to gain a meaningful results,  you can reach out for me at my e-mail, send me a message over LinkedIn, or find me on Twitter under @1mrobas

Disaclamer – this blog post expresses my own views, although I do use the »test drive« principles at work. And contrary to popular belief, I do not think Volkswagen makes bad cars ;)