The Canny Engineer Thinks

Trapped in the validation nation

Validation testing is a significant part of my life as an engineer these days. Also, I have much more interesting things I could - and should - be getting on with.

Validation is important, but it's largely stupid - like much of work, I suppose. Perhaps stupid is too strong a word. Dumb is probably a better choice, the opposite of smart. Validation is dumb, and there's no escaping it.

What to do?

Well, we can talk about it first (even if this is not doing anything about it). Validation is the antimatter of development - put the two together and you destroy engineering life itself (overdramatising somewhat). What I mean is this: development is subtle, replete with success and failure, nuance, puzzlement and, hopefully, finally, understanding. Validation is a box-ticking exercise that brings nothing of interest to anybody, except the approval to sell product, which is a key aspect of business, I have been led to understand... so validation is important.

Testing is an integral part of development, one that I certainly don't want to discard. When we develop components, we have to pass testing and other approval gateways; based on the resulting performance, either we confirm that parts need to be redesigned because they don't meet a particular specification, we confirm that our parts do meet a particular specification, or we can write our own spec if one doesn't exist already. At the end of a development project, validation confirms that the first parts off the real production process are as good as all those expensive prototype parts that were tested previously: validation can be the joyous culmination of all that development effort, champagne all round. So really, my gripe isn't with concept or process validation testing - these are valid steps in getting a product to market.

Get over it

My gripe is with approval testing; testing that is set as a hurdle to delivery to a given customer.

Hurdles are good in many ways. If everything we did were trivial, anybody could do it. Fortunately, what we do is not trivial, and there is a limited number of companies involved in our market, each with their own strengths and... opportunities. Validation testing from the customer side is a way of filtering out duff suppliers.

But the work that is sucking out the oxygen of development for me right now is not linked to any new development or any new product. It has the Kafkaesque whiff of bureaucracy generating a lot of effort for no tangible benefit at all.

Validation is time-consuming, expensive and is (supposed to be) a key component in my other important-stupid bugbear, PPAPs

So, as I asked before, what to do with it?

The phrase "creative destruction" comes to mind - destroy validation and PPAPs and all of that dumb junk to free ourselves to think creatively. Let's explore that: can we ditch it all?

Ideally, yes: with all the data swilling around in our company, from development results to production and quality control systems, we should be able to show that production parameters haven't changed significantly since the product was first approved, and that the standard production checks will have filtered out any weaknesses.

This can work - but often the customer wants to use an existing part on a new platform, or a new part on a new platform, so wants a full set of confirmation test results, in their unique format. Also, it doesn't avoid the question of whim. A customer can demand results against a particular set of specifications whenever they want. So, you test and you validate.

But why me?

If there's one industry where whim and whimsy is at a minimum, it's motorsport. What do they do there? Well, even if they are producing a "series of one-offs", they test and they inspect and they destroy things, too. Yes, they validate. How they go about it is hinted at in this puff-piece by and from the Lotus F1 team.

The article doesn't tell us much, but the important point is that they have an inspection team. Just as software smithies have test teams, engineers should have inspection and validation teams. This is what I am missing where I am now. We have fallen in slow-motion into the trap of mixing development and validation in one pot. Alas, because the output of validation is a ticked box (or a boxed tick), and since those ticks in boxes are prerequisites to supply and therefore making money, validation more often than not takes priority, thereby hindering development.

Unite and advance, Divide and conquer

The answer, then, is twofold - and also expensive. First of all, the vexing question of validation testing needs to be tackled at the source - with the customer. To be able to come to a sensible agreement on what's relevant and what's useful to test, the onus is on the supplier to show his understanding of the product - how it performs, how stable the processes are, what could potentially go wrong. This is expensive in terms of effort - data-hunting and gathering, condensing it into digestible information and then taking it to the customer for (in all probability) a series of meetings and discussions to come to a sensible arrangement.

Then validation testing must be separated from development. The two should be unique teams with limited overlap, ideally with separate facilities. This, too, is expensive - but not focussing sufficiently on development is more expensive still in the long run.

But perhaps I'm biased.

'Tis the season to be... audited

December. Time of cheer and good will to all colleagues, rushing for presents, updating projects, Glühwein, clearing out inboxes, eating far too much chocolate, finalising reports and… 

And getting audited. 

Yes, we were audited this week and one of my projects was in the spotlight. It was all going swimmingly until the auditor heat-seekingly locked on to one particular thread of my project that wasn't really parcelled up and tied with string - ironically enough, the DFMEA.

Being shown up as lax in my own project was certainly embarrassing, one of those half-expected shocks to the system; I felt a bit like a child hoping rather than expecting not to be found out about those stolen chocolates. I was hoping rather than expecting to be able to skim over that the incomplete DFMEA (structure present and correct, values not), knowing that it was a weakness without really having polished it off beforehand. I was found out, and rightly so: that’s the reason audits happen.

We were marked down for it, of course, and I’ll have to get things back into shape sharpish.

Reading those words of mine just above (“that’s the reason audits happen”), I surprise myself with how true they ring.

Have I finally come to accept them? And if so, how do I accept them? Gladly, or grudgingly?

For years I’ve harboured a deep suspicion, a dislike of audits and what they stood for. For me, they stood for engineering by checklist, for doing rather than thinking, for rewarding completeness rather than innovation and - for the vast majority of my auditing experience - huge cleaning up operations for close to zero benefit.

When is something that is good enough not good enough? When it’s being audited.

I have experienced both sides of auditing; I have audited and have been audited. From being part of an auditing team, working alongside quite an enlightened auditing colleague, I understand that the mindset of an auditor should be a positive one, aiming to help the subject improve by pointing out the weaknesses and working on agreements to correct those weaknesses before they lead to genuine failures. This mindset should match that of the auditee. When both see the positives that can come out of the (like FMEAs) negative messages, then things are heading in the right direction.

Nevertheless, audits are a not insignificant burden on everybody involved. Couldn’t we just wish audits, along with PPAPs, away?

Well, not easily: auditing is a multi-billion dollar industry in its own right, valid across a whole spectrum of industries, and it’s a difficult edifice to start chipping away at. But even so: wouldn’t our engineering lives be so much more enjoyable without them?

Initially, yes - they would be. We would be freed up again to design and develop as we know best: we know what our products are supposed to achieve and how to get them to that stage, even if not every Excel list has been filled out to the n’th degree en route. We could potentially become more like Google, where “...in the innovative and fast-paced world that [the Google developer] lives in, you get what you get.” (From How Google Tests Software)

We would have more money and time to spend on D&D, too, not having to pay those auditing firms their crust or having to spend all of those man-hours preparing “just in case it’s audited”.

But let’s look at it another way. Let’s say we want to start using a lower-cost supplier, more than likely in the old Eastern Bloc or somewhere (usually very large) in Asia. What are these companies like? Can we entrust our intellectual property, our quality and our good name to them? What better starting point could there be than searching for a certificate alongside customer references? (well, it’s true that there are differences in auditing rigour in China, even amongst the financial big four, as The Economist magazine writes)

Audits cannot guarantee a good name, nor necessarily a good engineering company: there are firms with certificates on their walls that I wouldn’t wish on our fiercest competitors. In the same way that financial audits have missed gaping holes where the subjects have been playing the game better than they have - like Lehman Brothers, Enron and, it seems, Autonomy  - quality audits can almost be guaranteed to miss something big from time to time.

Even the auditors themselves get themselves in a muddle - our December date with auditing destiny came about when the auditing company missed a submission deadline. This swiftly became our problem when our certificates were due to expire and our emergency re-audit date last December became our annual date. Thanks, we appreciate it!

So, what are audits good for, then? Qui bono? For starters, audits are a reassuringly expensive starting hurdle to business: my industry - automotive - and many others have gotten themselves into a standardised twist, whereby an ISO / TS 16949 certificate is a prerequisite for supplying to an OEM. It’s a pay-to-play move gives potential customers a guide that the company won’t royally mess things up when they start a supply relationship.

Audits also place a burden of duty and therefore responsibility on companies and their employees – from management right down to lab technicians – to get things right. Not only to “get things right” but also to “design things right”. This applies both to the product itself and to the process of how you get the product into a customer’s hands. Ideally, an audit should be imperceptible other than having to make some coffee for a visitor and answering a few questions. Why should you have to prepare if you are living the systems that you have declared fit for your own purpose?

Umm, too much other stuff to do, perhaps? Not enough time to focus? Not enough mental energy left for yet another list-trawl?

Well, if audits and all the stuff that we have to prepare for them really are a burden, then - again, ideally - they should become the impulse for genuine improvements in the way we work, in the way we communicate and collaborate. All of that form-filling, report-writing and change log management should have a genuine purpose, even if it is occasionally completed in the grudging spirit of passing an audit. All of these items are part of the company's index of information. When we change and update those forms, we are changing history, improving it. It's about creating a legacy, hopefully one of that will make sense of our successors' past.

The one thing that can make audits bearable is for everybody involved to treat it as a human thing - checks and balances are inevitably required whenever human endeavour is at work, so go with it. Let the auditors ask the good questions and let them discover how you work - even I with my DFMEA fail this time around will have shown that overall we’re working well and are going in the right direction. So, I’ll take that “nonconformity” hit and try to improve on managing my projects along with managing everything else, and let’s see if we can find some mental space to put to use on streamlining our work so we can do better next time.

And so back to my initial question: do I accept audits gladly or grudgingly? Well, of course it’s still the latter, but at least it means that I aim to keep them as low profile as possible: for that, though, I’ll need the support of my management – and I can assure you that the audit result was a wake-up call for them, too. Perhaps better things will come of it (or perhaps more oversight and review meetings – still, they’re a way of switching the focus to projects).

One final note on all of this: I don’t recall ever hearing anything about audits when I was studying engineering at university. That’s something that should change (perhaps it has, already), as they are a real, if occasional and generally unloved, part of this engineering life. If the next generation of engineers know what’s in store for them, they’ll know to focus on how they work as well as what they actually work on.

Ishikawa's stinking fish

Quality issues cannot be counted amongst my favourite activities. They can normally be categorised as "urgent-uninteresting", which is just about the best demotivator I can imagine. They're negative, cause huge floods of emails, assumptions, obfuscation and general panic. Some people thrive on this sort of situation. I, generally, don't, as was again proven by a quality concern with some Chinese colleagues.

I get involved simply because our Tech Centre has the best kit, so we can test what others can't. It's annoying, because development people rather prefer looking forwards than downwards at self-shot feet.

Nevertheless, some quality issues are useful ("never let a crisis go to waste" and all that). Some are excellent impromptu team-building exercises and others simply turn up some interesting artefacts, like this beauty below. It stopped me in my tracks - never have I seen an Ishikawa diagram illustrated so literally as by my Chinese colleagues...!

Ishikawa's stinking fish

For those not yet in the know, the Ishikawa, or fishbone, diagram is a way of formalising the investigation into the potential causes of a particular issue. It's a methodology that forces you to look at each the 6 M's (others call 7 or even 8) in order to gain the full picture of what might have gone wrong to cause the issue (sorry, problem) that we're working on (the Environment one is clearly an awkward 'M-ification' for the purposes of alliteration):

Machine (technology)

Method (process)

Material (Includes Raw Material, Consumables and Information.)

Man Power (physical work)/Mind Power (brain work): Kaizens, Suggestions

Measurement (Inspection)

Milieu/Mother Nature (Environment)

I can't tell you precisely what the 5 Chinese characters represent in this one. Whatever the causes of this particular quality issue, the discovery of this putrid gem of a rotten, stinking fish amongst the rotten, stinking debris of a quality concern almost made up for it...

Pass the resource please (and spread thinly)

Terracotta Warriors photo from romainguy on Flickr

As an Englishman living and working in Heidelberg, I am often asked if I work for SAP, the business software giant based down the Autobahn in Walldorf. I don't, of course. If I did, I'd be writing about software development, the management thereof and how utterly astounding their legendary canteen is.

The question is not a daft one, though: around 10 thousand people work at SAP in Walldorf, forming a more or less willingly thrown together (or at least well-paid) melting pot of 80 nationalities with English as the working language. It’s easy to assume, then, that a middle-class, technically minded foreigner living in Heidelberg earns his crust and her Grauer Burgunders at SAP.

I know a few people who work at SAP, and they are generally of a particular ilk (physicists and mathematicians, i.e., not my type) so I know that I don't need to yearn to work there, but there’s one thing I envy them: resources for R&D. Globally, (2011 figures) SAP has 16 thousand employees working in R&D (12 thousand work in sales…).

16 thousand in R&D...

(drifts into a reverie)



(Snaps back with a jolt)

I’m not in that place. I am a development engineer at an automotive supplier; my development activities really only skim the outer atmosphere of the unique world of R&D. Is this a surprise? Is it a disappointment?

Let’s think about the surprise factor first. When I consider my time at university, I don’t recall ever having heard the word “resource” discussed in any manner other than as a general term for learning material. We picked up the material, used the libraries and even the nascent internet; but I was never a resource myself.

Resource was always a “hidden” theme. We were aware of time pressures with the need to study such a wide range of subjects whilst maintaining sanity and health through extracurricular pursuits, but it was always a case of everybody finding their own balance.

My aerospace engineering course did involve one larger team design project that was formative, but as far as I could see, research projects could run in a business vacuum, free from excessive emails or telephone calls, requests for assistance from around the world and quality alerts to have to jump onto. Ah - there we go. Did you notice that word in there, mentioned for the first time in this post? Team.

Entering the workplace was a relatively soft jump for me: I joined Ford in the UK which at the time was recruiting heavily - and, crucially, I joined a team. By that I mean there were several of us who could do each other’s jobs, if necessary and we worked both on improving the product and on improving the ways we worked on those products.

Recently, I was part of a globally distributed team that developed a new procedure for drawing release and approval. The word 'team' in this case represents more a collection of perspectives than anything that could really work to fight the necessary fights. We had representatives from design, from quality, from manufacturing; but none of us were interchangeable in the way that my team position at Ford was.

Developing the process itself was the fun and interesting part. I managed to find a spare license for Microsoft Visio (alas only the 2003 version which is now looking very old indeed) and used it to create the workflow. As it is a cross-functional process, the workflow runs in lanes - development, manufacturing engineering and quality - so the visual aspect was initially quite appealing. Adding the "if… then" loops and the different entry points for product at the various stages in their drawing lifetimes certainly de-streamlined it, though.

But actually running the procedure has been no fun at all. The problem is that it was implemented as a project in itself without great consideration for the resource implications of centralising what was in effect a totally distributed (and therefore wasteful, if potentially evolutionary) process.

So the current situation is that I am currently the only person in a company of over 16 thousand who can create the drawings that those 16 thousand people need.  Imagine SAP setting up a system whereby those 16 thousand in R&D could only work if I finished one particular task amongst many. It wouldn't really make sense, and this is something that I'm struggling with at the moment.

And so to my second question - am I disappointed not to be part of that resource pool? In a word, no. It's harder to explain, but perhaps this is only because of a lack of imagination: I simply cannot imagine being able (or only being allowed) to work on one problem at a time. Whilst I am trying to relearn the art and discipline of focus, it doesn't come naturally to me. I thrive on bandwidth, even if the transmissions occasionally get muddled up and jam because of it. I'm a variety type.

Coming back to those cohorts of SAPlers, though: despite their numbers, there have been reports recently that even they are feeling pressured to breaking point in their jobs. Which means that you can spread even 16 thousand researchers thinly. R&D resources generally expand to fill the product lines a company is working on; even within the relatively low number of headline products SAP produces, there is a vast number of modules that require development and linking in to others, so there's plenty for lots of clever people to do. (Google has around 33 thousand employees, Microsoft 92 thousand, of which 35k are in R&D).

So today's R&D behemoths are commercial, distributed amongst the universities and amongst the startups that thrive or die on their findings; I'm somewhere in no man's land - where are you?

p.s. Happy 40th, SAP!

On PPAPs

There's one thing that I need to get off my chest early on in this blog, as it has been weighing on my mind for some time. It is the bane of my automotive life thus far, the PPAP.

Almost nobody knows or nor really cares these days that PPAP stands for "Production Parts Approval Process," nor that the system is responsible for terabytes of redundant data. The idea behind PPAPs is - I admit - sound, in that each and every part in a car is fully validated before being built into a vehicle. However, having worked on tube fittings for a few years and come across PPAP submissions "weighing" 26 megabytes for threaded nuts weighing 13 grammes, and still being wrong, I feel that the PPAP process itself needs investigating.

The PPAP is an information pack that includes the drawings, measurement data (with capability), test data, measurement and test equipment certification, FMEAs, control plans and process flows for the part in question. All of it needs to be correct to be accepted by the customer.

It was created by the AIAG (Automotive Industry Action Group) that originally consisted of the "Big" (now "Detroit") Three of Ford, GM and Chrysler. One of the driving principles behind it was to standardise the requirements for these three manufacturers so that their suppliers need only produce one data pack per part and not have to reproduce it with tweaks for each customer. The other key principle was and remains that parts will fit and work properly when they are assembled into the cars they are destined for. Equally, the PPAP should prove that the supplier is ready to produce good parts at volume and it provides baseline information for the life of the part ("back then at the beginning is was like that, now it's like this"). However, the PPAP system has become a monster. This monster has generated its own sub-industry; dedicated employees who work only on generating or approving PPAPs, and people like me checking supplier submissions like school teachers marking essays. There are great chains of PPAP submissions, from sub sub suppliers to suppliers to the OEMs, and the whole system is populated by disinterested humans.

The number of submissions that I have checked and rejected in the past for vast swathes of missing information was depressingly large. So these packs (of lies) are being batted back and forth from server to server, person to person and hours are being wasted the world over.

My relative fortune in my secret PPAP life was that I only had to check the technical aspect of the submissions; somebody else in the quality department was responsible for going through all the other documents and certificates (generally "is there something here that looks like a certificate, or not?"). But even the technical side of things, which really should have been simple, always seemed to lead to confusion.

Why should it be simple? Well, there are drawings with dimensions (some with complicated GD&T, admittedly, but still doeable) that need to be measured (and not just reported with the value "OK"). The supplier just needs to check off the dimensions one by one and show that what is being produced on real parts meets those requirements. Fine. But, the drawings also have words on them, sometimes in the form of specifications, which should give the supplier a small hint about some other aspect concerning the parts - like, say, performance - but often somehow don't. Maybe it is actually all just too subtle and not simple at all. It was alas very rare in my experience to receive a PPAP and to be able to complete it there and then, on the spot, no more questions asked. So not only do the PPAP submissions need to be sent along the chain of suppliers, they need to be reworked and resubmitted. Usually this involves testing being (re-) started just as the parts are urgently required, which means that we have to grant deviations for a limited period until all of the testing is completed (corrosion testing, anybody?), we have to request deviations of our customers and we have to keep track of those deviations as well as the versions of those PPAP documents... and so on.

So, there's lots's of time wasted, there are megabytes of redundant and duplicated information sloshing around, repeated for each of the approximately 15000 unique parts in each and every car driving around the world (I'm not sure PPAPs are used by Khodro in Iran, however).

And yet, PPAPs, like audits, are impossible to argue against. I recall hearing that Continental tried to settle on a basic level of PPAP above which the customer would have to pay, but I think that rebellion was quickly crushed. If you've heard more about it, let me know in the comments.

In the end, my recommendation is that, if you're a young budding engineer looking for a first job, try to avoid anything that says "PPAP" on it. If you still want the job, then make sure that the PPAP bit is not too high a percentage of the job and is for a limited period only - and spend that time finding someone else who will do it for you (preferably not an engineer!).