By Curt
Wuollet
Linux Systems Engineer
Heartland Engineering Co.
Heartland Engineering is an automation company formed to
automate production lines for our parent company, Westling Mfg. Co.
This type of machinery typically uses PLCs, robots, NMC machining
equipment and Off The Shelf components to shift human resources
from endless repetitive tasks to more interesting and productive
pursuits. The basic idea is to make one big machine out of several
smaller machines and control the flow of work between them.
I am a Linux fan (some might say zealot) with experience in UNIX
system administration, systems programming, electronics, test
equipment, failure analysis and electronic components. And no, I
don’t do Windows. Linux is, of course, Linux. Popular opinion says
Linux is for Internet protocol applications, mail servers, Web
servers, development and a lot of other IS stuff. Linux people, of
course, use Linux for that and everything else they need to do. And
when they aren’t doing all that, they are finding new ways to use
Linux.
So, what brings all these together under the heading of Linux
for Industry? Well, that’s the story I have to tell as an example
of how Linux finds it’s way into some very unusual places, doing
new and unusual things.
A couple of years ago, I was the system administrator and entire
IS staff at Westling. I landed there because they are a UNIX shop.
They are running factory software on Progress on DGUX on an Aviion.
Westling Manufacturing rebuilds auto parts of various types, mostly
electrical, and supplies many of the major chains of auto stores.
Long before I was a UNIX guy, I was a motorhead, so the fit seemed
pretty good. Since I’m a reasonably good mechanic and understand
auto parts and electronics, I got involved in some problem solving
and line testing sort of as a sideline.
There is a definite need for better test equipment for
automotive electronics. While there is fairly good equipment for
finished goods, starters, alternators, and the like, testing of
components is a major problem area. Over the years, as cars get
much more complex, the equipment available to the rebuilding
industry lags way behind. And there are a lot of special problems
with testing used components for reuse.
One of the more expensive components of recent alternators is
the internal voltage regulator that keeps the alternator from over
or under charging the battery. These are fairly sophisticated
electronic devices, accurate to within one percent and temperature
compensated. Some even have turn on delays and fault diagnostic
circuitry that solve various application problems. The regulator is
often the difference between an alternator that works and a
warranty return.
You can test them without special equipment, but very slowly and
not very well. A few approaches were being used. One approach was
to go ahead and build them into alternators — the better
alternator testers would find the really bad units and these were
replaced. The rework is expensive and some “walking wounded” got
through to become really expensive warranty returns.
The other approach was using various test boxes that claim to
test the various regulators. These are slow and very tedious and
the results are not demonstrably better than the other
approach.
When the warranty rate gets high enough, there’s a third method.
You simply buy all new regulators. Besides the cost, this defeats
the recycling purpose of rebuilding. And there are some vendors who
don’t know much about building regulators. None of these approaches
provided what was needed: fast, accurate, effective testing to
allow reuse of the OEM regulators. They were looking for a better
way, and since I had designed test equipment in a former career, I
could just about see how to do it.
I obsessed about this for a week or two and decided that a PC, a
data acquisition card, and some rather robust custom electronics
might do the job. I proposed this project to management and got
approval almost immediately.
A week later, I was having second thoughts. The data acquisition
board vendors knew only Windows and really wanted to sell expensive
software packages or low level DOS libraries. For a UNIX
programmer, this was very depressing. To do Windows programming was
expensive and would require that I learn a lot more about Windows
than I wanted to know. The DOS offerings were really intended to
eventually sell the Windows packages. And then there was the
problem of royalties and licenses.
Since I really didn’t want to develop on a Microsoft platform, I
resolved to write a device driver for Linux. It would be a major
project in itself and I don’t think I ever got anyone to understand
why I would need to add a month or two to the project, but it would
be worthwhile — if I could get the information I needed. My bad
luck — the board vendor decided to be less than helpful. Panic
time.
Enter The Linux Community. I got on the Web and burned a couple
of days searching through endless hits on data acquisition until,
finally, in a Newsgroup post, I found a reference to the Linux Lab
Project. Fantastic!
I searched and inquired and came up with two drivers. One was
for a close relative to the board I had and the other was adapted
to special data logging. Still, it was much easier than rolling my
own, so I downloaded them both to see what would happen. Neither
one came even close to compiling. Undaunted, I spent days studying
the code and reading the kernel sources. It was obvious there had
been many changes in the kernel since they were written.
With much hacking and many core dumps, I finally got them both
to run. I hacked some test code together and wired up a connector.
After another day of pulling my hair out, I bought an oscilloscope
to find out what was going on. I could get a single read but any
other mode refused to work. I called the vendor and asked some
questions, but they weren’t very interested or helpful. So I mailed
the authors. All the mail bounced.
I kept doing detective work and finally tracked down one of the
authors who was, of course, no longer in college and no longer had
the hardware to help. So I trusted him and mailed a board and not
only got the help I needed, but he also patched the driver to the
newest kernel level. Thanks Ricky! It turns out that my choice of
that board was bad for very technical reasons and without his
explicit knowledge, I might never have made it work right.
Finally, I had a driver; next the electronics. I did a design on
paper and kludged together a prototype. Nothing fancy, but it
worked. It went immediately into use on the factory floor, making a
couple of bucks with every part it tested. This attracted a lot of
attention and soon the talk turned to making it a product.
But looking at the rather sad kludge, we were a long ways from
having a product. I proposed a new project to do it right. That
meant a new, instrumentation grade electronic design and designing
printed circuit boards. Needless to say, there were no tools for
printed circuit board layout lying around the rebuilding plant.
Earlier, I had heard there was a pcb layout package for Linux,
called, pcb. I had looked at it, but not having a need, filed it
away. I dug it out and spent the next two weeks learning the
package and designing a board. I printed the artwork with
PostScript and went looking for a fabricator.
No one would take the artwork! They all wanted something called
Gerber files. This was a surprise, the last time I’d made a pcb, we
taped it on mylar and had it photo-reduced. I thought my PostScript
was a big improvement, but they still insisted on their own format.
(Note: pcb has been upgraded in the meantime to do Gerber files; I
will do the next layout on it.)
I hit the board fab house Web sites and was rewarded with a free
PCB layout program that would put out the required format. Of
course, it only ran on DOS and DOS, with all my other Microsoft
products, had been expunged from my hard drive long ago.
Time was short and I needed something working now. Dosemu! Of
course, why hadn’t I thought of it sooner? A short time later, I
was running the layout program right alongside my Linux desktop.
The board went off to the fab house and it was on to software. This
part of the project went better. I have used curses and the GNU
tools quite a bit and had a user interface and testing code nearly
done when the boards came back. It had been many years since I had
done a commercial analog design, and a lot of things could have
gone wrong, but the electronics passed the smoke test and
performance was better than I needed.
By this time, I had been doing more design, electronics, and
development work on the project than system administration. The
R&D group at Westling was spinning off into a new company that
became Heartland Engineering. They gave me an invitation and I
changed quarters from the front office to a machine shop. I set up
a small electronics bench and a couple of Linux machines and went
full time into ATE development.
We built a machine, took it to a rebuilders show and all of a
sudden we were in the tester business. Now, several machines and
many additions later, thousands of regulators a day are tested with
our product.
Linux is excellent in this role, the machines are very reliable
even though the operators have little or no computer experience.
The combination of Linux, the data acquisition card, software, and
electronics to produce and measure automotive voltage and current
levels forms a base that can be adapted to almost any component
test need. All of this at a cost that even small shops can
afford.
I designed another machine to test stator and diode assemblies.
Having a means to test them as assemblies saves large amounts of
labor required to disassemble, test, and reassemble them. This
machine uses essentially the same electronics to drive a motor,
excite a rotor and capture the output waveforms to compare them to
ideal computed waveforms. Any bad components alter the waveforms
and fail the assembly. This may become a product as well.
Linux is well entrenched in our automation business as well. The
bttv and later Video for Linux drivers have made it possible to do
low cost, definite purpose machine vision. We are successfully
using it for measurement and sensing applications. We have an
automation cell that reconditions starter shafts. After heating
them and reshaping them, a groove that locates the starter drive
gets munged.
Using a camera, frame grabber card, V4L and several Open Source
packages, I end up with a PGM image that I process to find the
remnants of the original groove. We use this to guide a special
lathe, built here, to recut the groove. All this happens in less
than ten seconds.
We are also doing precision, on the fly measurement with the
cameras to feed dimensions to an NMC lathe, fed by a robot, to turn
alternator slip rings and starter commutators to only the extent
needed to restore them.
The low cost of a Linux implementation makes many more uses
practical. The commercial alternatives are very expensive and much
more difficult to integrate. No two vendors in the automation seem
to speak the same protocols, so integration involves a lot of glue
code and hacks, which are much easier in Linux than anything
else.
Linux is such a great solution and so flexible that our current
uses just scratch the surface. I have been disappointed that so few
of the success stories in industry are published. I decided to
write this article so the many people who contribute to the Open
Source community get the credit they deserve.
The tools and utilities that accompany the Linux kernel are of
enormous value in developing real world solutions, yet are under
appreciated. To do this all from scratch would take years.. You
might be able to find all the bits and pieces to do this on a
proprietary platform, but the cost would be staggering and
prohibitive unless you were making hundreds of machines.
The real story here is the advantage of Open Source development.
Most solutions involve only a few pages of code in addition to the
extremely rich set of services and utilities provided with
Linux.
We are also well on our way to converting our offices to Linux
and StarOffice. Sun helped us out a lot with the free licensing of
StarOffice. We are looking at new factory software and intend to
run it on Linux also, if at all possible. We can t afford SAP, but
we are looking at a product based on UniVerse. We could then
replace the aging DG Aviion with a fast Alpha server, maybe even a
dual or quad. I’m convinced that Alpha wouldn’t be around if not
for Linux sales.
Organizations that are considering Linux should know also that
the work to do it is well worth it. In our shop at Heartland, we
have one NT machine that we use for AutoCad. Right now we have no
choice but to use NT as the folks at AutoDesk seem to have bad
hearing. But anyway, we spend a lot more time keeping that one NT
machine running than all the stuff we do with Linux. It’s such a
convincing difference that we have talked about offering Linux
consulting to manufacturing and other small and medium businesses.
Almost any business could benefit from software that works.
I am posting this to Linux Today because that’s my home page. I
like to keep up on Linux news and we even used the job listing to
find a system administrator for Westling, now that the old admin
(me) is a systems engineer.
I would have liked to provide more detail and some of the other
things we are doing with Linux, but, between Westling, Heartland,
and my after hours consultancy, time is a problem. It’s midnight
and after twelve hours on the tube, I wanted to get this done. I
will be happy to answer questions and inquiries mailed to: cww@westling.com or wideopen@ecenet.com.