Curt Wuollet--Linux in Industry: Our Own Case StudyOct 31, 1999, 20:34 (24 Talkback[s])
(Other stories by Curt Wuollet)
Heartland Engineering is an automation company formed to automate production lines for our parent company, Westling Mfg. Co. This type of machinery typically uses PLCs, robots, NMC machining equipment and Off The Shelf components to shift human resources from endless repetitive tasks to more interesting and productive pursuits. The basic idea is to make one big machine out of several smaller machines and control the flow of work between them.
I am a Linux fan (some might say zealot) with experience in UNIX system administration, systems programming, electronics, test equipment, failure analysis and electronic components. And no, I don't do Windows. Linux is, of course, Linux. Popular opinion says Linux is for Internet protocol applications, mail servers, Web servers, development and a lot of other IS stuff. Linux people, of course, use Linux for that and everything else they need to do. And when they aren't doing all that, they are finding new ways to use Linux.
So, what brings all these together under the heading of Linux for Industry? Well, that's the story I have to tell as an example of how Linux finds it's way into some very unusual places, doing new and unusual things.
A couple of years ago, I was the system administrator and entire IS staff at Westling. I landed there because they are a UNIX shop. They are running factory software on Progress on DGUX on an Aviion. Westling Manufacturing rebuilds auto parts of various types, mostly electrical, and supplies many of the major chains of auto stores. Long before I was a UNIX guy, I was a motorhead, so the fit seemed pretty good. Since I'm a reasonably good mechanic and understand auto parts and electronics, I got involved in some problem solving and line testing sort of as a sideline.
There is a definite need for better test equipment for automotive electronics. While there is fairly good equipment for finished goods, starters, alternators, and the like, testing of components is a major problem area. Over the years, as cars get much more complex, the equipment available to the rebuilding industry lags way behind. And there are a lot of special problems with testing used components for reuse.
One of the more expensive components of recent alternators is the internal voltage regulator that keeps the alternator from over or under charging the battery. These are fairly sophisticated electronic devices, accurate to within one percent and temperature compensated. Some even have turn on delays and fault diagnostic circuitry that solve various application problems. The regulator is often the difference between an alternator that works and a warranty return.
You can test them without special equipment, but very slowly and not very well. A few approaches were being used. One approach was to go ahead and build them into alternators -- the better alternator testers would find the really bad units and these were replaced. The rework is expensive and some "walking wounded" got through to become really expensive warranty returns.
The other approach was using various test boxes that claim to test the various regulators. These are slow and very tedious and the results are not demonstrably better than the other approach.
When the warranty rate gets high enough, there's a third method. You simply buy all new regulators. Besides the cost, this defeats the recycling purpose of rebuilding. And there are some vendors who don't know much about building regulators. None of these approaches provided what was needed: fast, accurate, effective testing to allow reuse of the OEM regulators. They were looking for a better way, and since I had designed test equipment in a former career, I could just about see how to do it.
I obsessed about this for a week or two and decided that a PC, a data acquisition card, and some rather robust custom electronics might do the job. I proposed this project to management and got approval almost immediately.
A week later, I was having second thoughts. The data acquisition board vendors knew only Windows and really wanted to sell expensive software packages or low level DOS libraries. For a UNIX programmer, this was very depressing. To do Windows programming was expensive and would require that I learn a lot more about Windows than I wanted to know. The DOS offerings were really intended to eventually sell the Windows packages. And then there was the problem of royalties and licenses.
Since I really didn't want to develop on a Microsoft platform, I resolved to write a device driver for Linux. It would be a major project in itself and I don't think I ever got anyone to understand why I would need to add a month or two to the project, but it would be worthwhile -- if I could get the information I needed. My bad luck -- the board vendor decided to be less than helpful. Panic time.
Enter The Linux Community. I got on the Web and burned a couple of days searching through endless hits on data acquisition until, finally, in a Newsgroup post, I found a reference to the Linux Lab Project. Fantastic!
I searched and inquired and came up with two drivers. One was for a close relative to the board I had and the other was adapted to special data logging. Still, it was much easier than rolling my own, so I downloaded them both to see what would happen. Neither one came even close to compiling. Undaunted, I spent days studying the code and reading the kernel sources. It was obvious there had been many changes in the kernel since they were written.
With much hacking and many core dumps, I finally got them both to run. I hacked some test code together and wired up a connector. After another day of pulling my hair out, I bought an oscilloscope to find out what was going on. I could get a single read but any other mode refused to work. I called the vendor and asked some questions, but they weren't very interested or helpful. So I mailed the authors. All the mail bounced.
I kept doing detective work and finally tracked down one of the authors who was, of course, no longer in college and no longer had the hardware to help. So I trusted him and mailed a board and not only got the help I needed, but he also patched the driver to the newest kernel level. Thanks Ricky! It turns out that my choice of that board was bad for very technical reasons and without his explicit knowledge, I might never have made it work right.
Finally, I had a driver; next the electronics. I did a design on paper and kludged together a prototype. Nothing fancy, but it worked. It went immediately into use on the factory floor, making a couple of bucks with every part it tested. This attracted a lot of attention and soon the talk turned to making it a product.
But looking at the rather sad kludge, we were a long ways from having a product. I proposed a new project to do it right. That meant a new, instrumentation grade electronic design and designing printed circuit boards. Needless to say, there were no tools for printed circuit board layout lying around the rebuilding plant.
Earlier, I had heard there was a pcb layout package for Linux, called, pcb. I had looked at it, but not having a need, filed it away. I dug it out and spent the next two weeks learning the package and designing a board. I printed the artwork with PostScript and went looking for a fabricator.
No one would take the artwork! They all wanted something called Gerber files. This was a surprise, the last time I'd made a pcb, we taped it on mylar and had it photo-reduced. I thought my PostScript was a big improvement, but they still insisted on their own format. (Note: pcb has been upgraded in the meantime to do Gerber files; I will do the next layout on it.)
I hit the board fab house Web sites and was rewarded with a free PCB layout program that would put out the required format. Of course, it only ran on DOS and DOS, with all my other Microsoft products, had been expunged from my hard drive long ago.
Time was short and I needed something working now. Dosemu! Of course, why hadn't I thought of it sooner? A short time later, I was running the layout program right alongside my Linux desktop. The board went off to the fab house and it was on to software. This part of the project went better. I have used curses and the GNU tools quite a bit and had a user interface and testing code nearly done when the boards came back. It had been many years since I had done a commercial analog design, and a lot of things could have gone wrong, but the electronics passed the smoke test and performance was better than I needed.
By this time, I had been doing more design, electronics, and development work on the project than system administration. The R&D group at Westling was spinning off into a new company that became Heartland Engineering. They gave me an invitation and I changed quarters from the front office to a machine shop. I set up a small electronics bench and a couple of Linux machines and went full time into ATE development.
We built a machine, took it to a rebuilders show and all of a sudden we were in the tester business. Now, several machines and many additions later, thousands of regulators a day are tested with our product.
Linux is excellent in this role, the machines are very reliable even though the operators have little or no computer experience. The combination of Linux, the data acquisition card, software, and electronics to produce and measure automotive voltage and current levels forms a base that can be adapted to almost any component test need. All of this at a cost that even small shops can afford.
I designed another machine to test stator and diode assemblies. Having a means to test them as assemblies saves large amounts of labor required to disassemble, test, and reassemble them. This machine uses essentially the same electronics to drive a motor, excite a rotor and capture the output waveforms to compare them to ideal computed waveforms. Any bad components alter the waveforms and fail the assembly. This may become a product as well.
Linux is well entrenched in our automation business as well. The bttv and later Video for Linux drivers have made it possible to do low cost, definite purpose machine vision. We are successfully using it for measurement and sensing applications. We have an automation cell that reconditions starter shafts. After heating them and reshaping them, a groove that locates the starter drive gets munged.
Using a camera, frame grabber card, V4L and several Open Source packages, I end up with a PGM image that I process to find the remnants of the original groove. We use this to guide a special lathe, built here, to recut the groove. All this happens in less than ten seconds.
We are also doing precision, on the fly measurement with the cameras to feed dimensions to an NMC lathe, fed by a robot, to turn alternator slip rings and starter commutators to only the extent needed to restore them.
The low cost of a Linux implementation makes many more uses practical. The commercial alternatives are very expensive and much more difficult to integrate. No two vendors in the automation seem to speak the same protocols, so integration involves a lot of glue code and hacks, which are much easier in Linux than anything else.
Linux is such a great solution and so flexible that our current uses just scratch the surface. I have been disappointed that so few of the success stories in industry are published. I decided to write this article so the many people who contribute to the Open Source community get the credit they deserve.
The tools and utilities that accompany the Linux kernel are of enormous value in developing real world solutions, yet are under appreciated. To do this all from scratch would take years.. You might be able to find all the bits and pieces to do this on a proprietary platform, but the cost would be staggering and prohibitive unless you were making hundreds of machines.
The real story here is the advantage of Open Source development. Most solutions involve only a few pages of code in addition to the extremely rich set of services and utilities provided with Linux.
We are also well on our way to converting our offices to Linux and StarOffice. Sun helped us out a lot with the free licensing of StarOffice. We are looking at new factory software and intend to run it on Linux also, if at all possible. We can t afford SAP, but we are looking at a product based on UniVerse. We could then replace the aging DG Aviion with a fast Alpha server, maybe even a dual or quad. I'm convinced that Alpha wouldn't be around if not for Linux sales.
Organizations that are considering Linux should know also that the work to do it is well worth it. In our shop at Heartland, we have one NT machine that we use for AutoCad. Right now we have no choice but to use NT as the folks at AutoDesk seem to have bad hearing. But anyway, we spend a lot more time keeping that one NT machine running than all the stuff we do with Linux. It's such a convincing difference that we have talked about offering Linux consulting to manufacturing and other small and medium businesses. Almost any business could benefit from software that works.
I am posting this to Linux Today because that's my home page. I like to keep up on Linux news and we even used the job listing to find a system administrator for Westling, now that the old admin (me) is a systems engineer.
I would have liked to provide more detail and some of the other things we are doing with Linux, but, between Westling, Heartland, and my after hours consultancy, time is a problem. It's midnight and after twelve hours on the tube, I wanted to get this done. I will be happy to answer questions and inquiries mailed to: email@example.com or firstname.lastname@example.org.
0 Talkback[s] (click to add your comment)