Troubleshooting

Troubleshoot like a doctor…Differential Diagnsosis

This is the second in my new series on troubleshooting – a very important skill for all consultants.

The next several posts are from a magazine article for my engineering colleagues, which I have broken into several smaller easy to digest pieces.

Early in my electrical engineering consulting career, a client asked me to explain each step as we worked to improve ESD (electrostatic discharge) immunity on an existing product.

In addition to solving the problem, he wanted to better understand my thinking process. Fair enough, I thought.

At one point, I laid out a “fault tree” of possibilities, along with prescribing a course of action. As it was getting complicated, I apologized for any confusion. The conversation went something like this:

“Not a problem,” my client said, “you are doing differential diagnosis.”

“Stop,” I said, “ what did you just say? And where did you hear that?” Joking, I added, “I’m a consultant — we make our living with buzz words like that.”

Laughing, he responded, “It is a medical term. My brother-in-law is a physician, and we often discuss troubleshooting methods.”

A few weeks after my initial introduction to this concept, I struck up a conversion with a seat mate on a cross country flight. Upon learning he was a doctor with the Mayo Clinic, I asked about differential diagnosis and was treated to a most interesting lecture.

After all, he was a teaching doctor and I was a very willing student. Those of us who teach love these situations.

He began by explaining the father of differential diagnosis was Arthur Conan Doyle (Sherlock Holmes.) Doyle was an MD who also wrote short stories. He had an idea for a detective based on a favorite medical professor who taught clinical diagnosis.

As we all know, the rest is history. It also explains the presence of Holmes medical sidekick, Dr. Watson.

The objective is “rule things in/rule things out” by creating two lists – high probability and low probability. The goal is to quickly narrow down a large list of potential causes to a smaller list, maybe even one likely root cause.

For example, if a patient presents with a red rash, there may be a hundred or more possibilities. Maybe it is the measles, or maybe it is bubonic plague.

The first step is looking at vitals (temperature, blood pressure, etc), which helps quickly eliminate possibilities.

The next step is the physical examination, along with detailed questions. Sometimes an immediate diagnosis can be made — other times additional tests may be necessary.

At that point, the prescription can follow — but not before.

As the Mayo doctor emphasized, “Prescription without Diagnosis is Malpractice.” As an aside, how many of us have thrown solutions at the problem without thinking it through?

Think like a doctor instead.

These chance discussions refined my approach to troubleshooting.  But the doctor shared more advice:

If there are multiple possibilities, address the simple ones first (Occam’s razor.) He shared another medical saying, “If you hear the sound of hoofbeats, don’t assume zebras.”

It is probably a horse (unless you are in Africa.) As consultants, we all like to sink our teeth into a juicy problem, but most problems are simple. On rare occasions, however, it well may be a zebra.

He pointed out the Mayo Clinic often deals with “zebras.” There may be a 100 possibilities, and 99 have been ruled out by previous doctors, making it simple for them to identify the zebra.

This is why it is important to ask what has already been done to address the problem.

I learned this lesson the hard way after chasing a problem for several days, only to then discover a very simple solution. It was a bit embassing, but my client was still happy to have the problem solved. But after that, I first looked for simple solutions, going after the “low hanging fruit” no matter what the probability.

This can give rise to a consulting miracle. If there is only a one per cent chance it will work, but it does, everybody thinks you are a genius. When that happens, it is quite a rush 🙂

In the next post, I’ll share a three step process for making the diagnosis dubbed ACT for “Aware-Critque-Act.”

Hope you enjoy this series…  


Interested in resurrecting the FREE monthly teleconference? If so, please let me know. 


 

© 2022, https:. All rights reserved.

Troubleshooting with common sense…

This post was inspired by a favorite story of my late business partner. It captures the essence of troubleshooting. 

As a young man, Bill’s nephew was a radio technician in the Air Force. Which was pretty amazing, as he knew very little about electronics. Nevertheless, he was quite successful as a primary troubleshooter.

When an aircraft radio failed, he was sent to the flight line as the first responder. He was able to identify and fix about 90 percent of the problems with the following simple procedures:

  • Check the power… Fuses? This was really easy in the days of vacuum tubes — did they light up?
  • Check the connections… Were all cables properly connected and seated. Antenna? Microphone? Etc…
  • Check the manual… If the first two did not solve the problem, go through the manual to make sure things were properly adjusted. Frequency? Squelch? Etc…

If these three did not fix the problem, he yanked the radio and took it back to the shop.

The net result: the shop technicians loved him as they did not waste their time on trivial problems, and the pilots loved him as he often “fixed” things on the spot. All saw him as a bit of a genius.

There is more to the story. Upon leaving the Air Force, he went to college and became a veterinarian, often using the same approach to diagnose his sick animal clients. Many times simple solutions solved their problems too.

The moral — Try the simple things first. We will follow up with a few more stories as we develop this thread on troubleshooting — a very important skill in consulting. So stick around.


Last November I announced my cutting back on regular posts. Moving forward I’ve decided to alternate between consulting-related posts and non-related “stories, observations, and the occasional rant.”

As a warning, if you don’t like the latter, don’t read the posts. Please don’t reply unless it is positive. I really don’t care to hear political or religious arguments as received in the past – and to which I never replied. Fair enough?

Peace — Uncle Daryl 


Interested in resurrecting the FREE monthly teleconference? If so, please let me know. 


 

© 2022, https:. All rights reserved.

How a simple technology can help stop COVID-19…

Warning:  This is a quick rant combined with a lesson on troubleshooting.

No, this post is not about some quack fix like drinking Clorox or Lysol — the simple technology is MASKS! But in order to work, EVERYONE must use them. The generic masks are not meant to protect YOU — they are meant to protect YOUR NEIGHBOR.

A recent model predicted that if 80% wore masks that were 60% effective, the COVID Ro parameter would drop below 1, which means the spread is no longer exponentially increasing, but actually exponentially decreasing!

Or as we say in the engineering world, the response is now over-damped. The model further predicts that with a 14 day incubation period, within 28 days new cases would be WAY down.

Models are nice, but what about the real world? Well, we have solid data there. Both Hong Kong and Taiwan are religious about masks. They acted quickly, and have almost 100% mask compliance. As a result, Hong Kong (population 7.5 million) has had 4 deaths, and Taiwan (population 23.8 million) has had 6. And businesses and schools are open.

At these rates the US would have between 100 and 200 deaths, not closing in on 100,000!

Of course, this is not new technology. In the 1918 pandemic (the one that killed my grandmother) masks were mandatory in many cities. Those that enforced mask wearing had much lower death rates. While they did not have the benefits of modern science back then, they did have the benefits of common sense. Something that seems be badly lacking today.

Why the resistance to masks, anyway? Like so much in our society today, it has been politicized. Polls show the “Blues” are much more likely to wear masks than the “Reds.” The difference being, the Blues are concerned about their neighbors, while too many Reds are concerned only about themselves. (If I stepped on toes, get over it & start wearing a mask.)

A quick question for those who want to “assert their constitutional rights.” I would ask “So show me where it says in the constitution that you have a right to infect others.” Self-centered, indeed.

Now, some consulting engineering thoughts — specifically on troubleshooting. For the last 30+ years, my late business partner and I worked to “identify, fix, and prevent” Electromagnetic Interference (EMI) problems in electronic systems. Much of our time was spent troubleshooting while trying to better understand the problems at hand.

Like small town doctors who identified illnesses, fixed broken arms, and prevented disease through shots and vaccines. It was gratifying work, and surprisingly lucrative.

As a first step in identifying (diagnosing) a problem, we used a simple model of “Source – Path -Victim.”  All three elements are needed for an interference problem, but only one needs to be eliminated or controlled.  So which one is the best?

The source, of course. One does that through suppression (for COVID, coughing in your elbow or straying at home when sick,) or containment (such as a mask.)  In the EMI world, the containment is often a simple metal shield, which when properly applied is often cheap and very effective.

The containment need not be perfect — even reducing the unwanted EMI emissions (spread) by a factor of 2 (aka 6 dB for you geeks) is sometimes enough. But with just a little care, achieving 1000x (60 dB) to 1,000,000 (120 dB) reductions are economically doable. When everyone wears masks, the COVID source reduction is believed to be at least 100x (40 dB).

For you non-geeks, the deciBels (dB) are exponential, just like the unchecked spread of the COVID. Scary, huh?

If controlling the problem at the source is not possible, then we move on attacking the “path.” For COVID, this is either through the air, or through contact with infected surfaces. Solutions include (once again) masks and sanitation/hand-washing.

The last method is protecting the “victim.” Because the potential victims can well outnumber the sources, this can get complex and expensive — think ventilators and ICUs. Thus, drastic measures like shutting down businesses and stay at home orders.

We often needed all three for solving EMI problems —  but the source suppression was and still is far and away the most effective when it can be applied. Works for COVID too.

That concludes today’s rant/consulting engineering lecture.  Now — START WEARING A MASK — to PROTECT THOSE AROUND YOU — and to HELL with self-centered politics!

Stay healthy everyone!

Uncle Daryl

P.S. Enough with the rants for a while.  Next up will be a couple of posts where I share some ideas with those not yet consultants, but harboring the “itch.”  We will start with the mailbag. Stay tuned…


The FREE Monthly Teleconference is Back!

Join us with your questions – or just listen in learn

“Ask Daryl Anything About Consulting”

Register Here


 

© 2020, https:. All rights reserved.

Using the “Five Whys” When Troubleshooting…

As consultants, we are often called on to diagnose and troubleshoot problems. This is particularly true of my technical colleagues. Here is a technique learned many years ago the has served me well.

Is it called the “Five Whys” and the goal is to find the “root cause” of a problem, rather than just applying a superficial fix. This usually requires some digging and questioning, until you can dig or question no further.

Here is an generic example. The machine failed, and needs to be fixed. Rather than simply repair it, ask questions.

  • Why did the machine fail? Because the bearings were dry.
  • Why were the bearings dry? Because they lacked oil.
  • Why did they lack oil? Because of lack of maintenance.
  • Why did they lack maintenance? Because we don’t have a maintenance program.
  • Why don’t we have a maintenance program? I don’t know.

In this case, the “root cause” is a lack of preventative maintenance (PM.) The short term solution is to replace the bearings, but the long term (and real solution) is to implement a PM program. But this solution was only reached after asking “Why” five times.

There is no magic in the number five. You may reach the root cause in two or three questions, or it may take more. But in my experience, five is often enough. Try it the next time you are diagnosing a problem.

Now, a personal example. A new product had a 50% return rate. It was a control system used in a locomotive. The client had a preliminary diagnosis when I was called in to implement a solution. But the solution was superficial, so I dug deeper.

  • Why the returns? The control system interferes with communications radios in the locomotive cab.
  • Why does it interfere? We don’t know, but we need a way to screen for this in manufacturing.

At this point, only two “whys” had been asked. The client was insistent on a screening test, so I implemented a way to identify faulty units. But then I went further.

  • Why only 50% failures? We don’t know.
  • Why not more or less? We don’t know.

Based on this useful clue, I set a “good unit” and a “bad unit” side by side. A visual inspection showed no difference. But further probing suggested electronic leakage at a cable connector on the “bad unit.” Aha!

  • Why is it leaking at the connector? We don’t know – they are all built the same.

So I suggested taking things apart. It turned out the connector used a special gasket, but the gasket needed to be installed one way. Properly reinstalling the gasket on the “bad unit” turned it into a “good unit.”

The “root cause” was now identified. I surmised the reason for the 50% failures was the production line person flipped every other gasket. Perhaps out of boredom, or perhaps it was simpler to do so.

The ultimate solutions were to (1) train the installer to properly install the gasket, and (2) screen for faulty installation as a quality check. The latter was “just in case” a failure occurred, preventing shipping a faulty unit to a customer. It was no longer the primary solution to a much deeper problem.

Remember the Five Why’s when troubleshooting. Don’t stop until you have a identified (or at least hypothesized) a root cause.

P.S. A wise colleague once said “If you fix a problem but don’t know why, it will come back later and bring all of its friends.”


Join us for our next FREE monthly teleconference.
“Ask Daryl Anything About Consulting”
Register Here


© 2018, https:. All rights reserved.

Simplify your expertise…

This post was inspired by a recent post from fellow AZ blogger (and person I have long admired) Pamela Slim.  She started off with this quote, which I borrowed:

“In the beginner’s mind there are many possibilities, but in the expert’s there are few.”  ― Shunryu Suzuki

It got me thinking about our roles as consultants. Most of us have wide knowledge and experience in our area of expertise. As such, we could offer many possible solutions to the problems faced by our clients.

But our real value is the ability to winnow those possibilities down to a select few, and then make recommendations. Much like a doctor (also a professional consultant) who makes a diagnosis followed by a prescription or recommended course of action.

So how do you do this? You begin by asking questions, and then carefully listening to those answers. Those answers may lead to more questions, so you dig deeper.

As a consulting engineer, I’ve long followed a methodology used by medical doctors known as differential diagnosis. First heard this from a client whose brother was an MD. As we were troubleshooting his problem, he shared this approach he had learned from his brother.

Later, I had a chance to expand my understanding thanks to an MD from the Mayo Clinic. An airplane seat mate, I asked about the term. As a teaching doctor, he broke into a big smile and shared his knowledge (and wisdom) as we flew across the country. The flight was most memorable, and has since influenced consulting approach.

The goal is to “rule things in – rule things out.”  For example, the Mayo MD said, if a patient presents with a red rash, there may be a hundred possible causes. But with some additional information (such as vital signs or simple questions) one may be able to quickly reduce this to but a few possibilities.

Based on that, you begin by treating the most likely cause. Even then, you are playing the probabilities, so if the initial treatment does not solve the problem, move to the next most likely solution. Or, as mom and dad used to say, “If at first you don’t succeed, try again…”

I have since modified the approach. Rather than prioritizing solely on probabilities, I first try the very simple solutions. Learned this after spending several days trying most likely solutions, while ignoring an easy low probability solution that worked. A bit embarrassing.

Doing this also gives rise to an occasional consulting miracle. For example, even if the simple solution has a 1% probability of success, that still means that one out of a hundred times it works. And when it does, everyone thinks you are a genius.  🙂

Finally, don’t just treat the symptoms — seek to treat the cause. The goal is to not only understand the problem, but to identify the underlying root cause of the problem. Asking questions, digging deeper, and using differential diagnosis all help.

We’ll dig deeper ourselves in a future post on the “Five Whys'”.


This post precipitated a new category — troubleshooting — where I will share more insights on this important facet of consulting. 


Join us for our next FREE monthly teleconference.
“Ask Daryl Anything About Consulting”
Register Here


© 2018 – 2019, https:. All rights reserved.

The country doctor approach…

When troubleshooting, common sense and experience go a long way.

Here is a story of my great-uncle, a country doctor in Nebraska.  I often think of him when trying to diagnose and fix a client problem.

This story appeared in my first book, written twenty years ago. While it was aimed at my engineering colleagues, it applies to all types of consulting.

I had a relative who was a country doctor the first half of the 20th century. I once saw the tools of his trade and was touched by their simplicity.

The old black bag didn’t hold a lot (a stethoscope, a thermometer, some simple surgical tools, and a few medicines), but when these tools were combined with  medical knowledge and experience, they saved lives.

It didn’t take CAT scans or MRIs to make a diagnosis and solve a lot of problems. Sure, the latest technology is great, but you don’t need it for every situation.

Remember that old country doctor when troubleshooting problems.  Using a few simple tools, you don’t need a million dollars of test equipment or reams of test data to solve many problems. Like the doctor, you can rely on your own experience, knowledge, and common sense.

Hope you enjoyed the story. Rest in peace, Dr. Metheny.

P.S. Just finished teaching our Troubleshooting Workshop to a group of engineers in San Diego, based on the medical concepts of differential diagnosis (to be covered in a future post.) Always a good time – and the weather was great too!

© 2014 – 2021, https:. All rights reserved.