This past weekend was the 11th annual Southern Califoria Linux Expo. This year, it extended to support other Linux and FLOSS related events. On Friday, Reliam sponsored part of DevOps Day LA, which ties directly to our mission at Reliam, something accomplished through our Internet Application Management. On Saturday and Sunday, the classical SCALE presentations occurred. Also, the exhibition hall was open with a variety of different organizations and companies. Because there were so many different things to do, I wasn’t able to see everything, but wanted to share some of my experiences and takeaways from a great weekend. Because many like-minded groups of individuals attend SCALE, there were several special events: MySQL Community Day Are You Getting the Best Out of Your MySQL Indexes? presented by Sheeri Cabral of Mozilla. For many years, I have helped Reliam’s customers tune their databases in order to receive the best performance possible, so I was excited to hear Sheeri Cabral’s presentation. She covered the basics, including the use of multi-column indexing, something I have used many times already. The use of ENUM was new to me, because in the past I used a SET where the ENUM is the better setting. I also learned that MySQL uses statistics on tables and may at times not use indexes. This is due to the fact that it thinks reading directly from the table is faster than reading from an index and bouncing into the table for each match. Puppet Camp LA Hierarchical Systems Policy Management in a Puppet/LDAP Environment presented by Patrick Paul. Since Reliam uses Puppet to help manage the many hosts they run for customers, I also attended a Puppet Camp LA talk on a method of integrating Puppet and IDAP. This talk covered how to optimize Puppet control for systems in a massive environment. How Mozilla uses puppet presented by Reliam alumni Brandon Burton. He covered their best practices for using revision control, along with verification on code, before checkin was covered. They have also built a tool called puppetctl, their way of disabling Puppet for a length of time, something I found very interesting. DevOps Day Promise Theory presented by John Willis. John Willis spoke about how tools such as Puppet and Chef go for determininistic declarations of how a server or system should be and how such statements may become harder to enforce as systems scale. In a more dynamic environment, Promise Theory would allow new pieces to be added, thus gaining trust among peers. Since this was somewhat of an abstract concept, much of it went in my ears without true comprehension. However, I will remember this talk when I encounter larger and larger scale systems. Test Driven Development for Ops presented by Christopher Webber. Christopher Webber discussed the importance of using of tests to ensure unnecessary mistakes are not put in through syntax checking. Linting allows review of the code for best practices and catches things that are missing, such as documentation. These were the easy steps in testing. Writing unit tests is a step up, since you must first write the test, then know how to catch the corner/edge cases for the best benefit. For sysadmins, we are familiar with ???testing??? in production through constantly monitoring our systems and verifying things are running smoothly. Using configuration management tools to ensure these monitor checks are set-up, simplifies the steps needed to bring new systems up to speed or ensure that changed settings are verified. This part of testing is what I can’t wait to get implemented at Reliam. Mozilla Scaling Patterns presented by Brandon Burton and Chris Turra. Brandon gave his second presentation with one of his co-workers, Chris Turra. It was great to see the many details about how companies run for scale, both in technology and people. They have the ???Button???, which is used for deploying site updates via a tool they call chief. These tools are something Reliam has created for customers as well. Other notable presentations I attended: The Secure Boot Journey by Matthew Garrett. Matthew spoke about what happened when Microsoft released the hardware requirements for Windows 8 laptops and Linux may have been locked out from those systems. Windows 8 requires that the hardware verifies what it is booting in order to be authorized. Microsoft said that their keys had to be trusted if Windows 8 was to be sold on it. Since at that time there was no requirement with the ability to add keys, manufacturers would not have included this ability, since the majority of people are not aware or have no interest in this function. Matthew recounts how he created a large outcry, causing Microsoft to add an interface to manage the keys trusted for booting on Windows 8. He also covered the tools written for assisting users in booting and installing Linux, accomplished by these tools being signed by the Microsoft key. Scaling systems configuration at Facebook by Phil Dibowitz Phil covered why Facebook went with Chef. Reasons included the ability to make major changes to how the tool worked. These changes reduced the reported data to the bare minimum of only a few kilobytes, instead of megabytes. He noted that Facebook used both paid and open source Chef. With the paid version of Chef, Facebook gained early access to the 2012 port, which lead to a drastic improvement of resource usage for Chef. Lightweight Virtualization with namespaces, cgroups, and unioning filesystems by J??r??me Petazzoni Linux containers are a group of technologies that help limit one group of programs from interacting with another group. This method is ideal when setting-up a shared hosting environment. Configuring a new environment for the next user takes a few minutes, if not less, and starting-up that environment typically takes seconds. This allows more environments to run on one piece of hardware and even more environments to be configured. Economies of scale at its finest. Logstash by Jordan Sissel This presentation helped me wake-up with humor. Jordan covered how sysadmins don’t like being woken up in the middle of their sleep, just because a disk is getting full. A natural instinct is to delete all of the logs. This is great for space, but when you need the information, this causes you to curse your former self. Another problem is when tech support need access to logs, but are unable due to security requiring restricted access. Tools are built to serve this purpose, but are often slow and difficult to use. Logstash to the rescure! With Logstash, tech support personnel may drill down to a required time frame and search for a specific user, all in a single bound. AKA, its pretty darn quick. Highlights from the Exhibition Hall LinuxAstronomy was back from last year, showing even more devices that are going into the sensor platform they plan to launch via a balloon. It was awesome! LinuxMCE. LinuxMCE allows me to monitor and control my house. It’s very slick looking compared to MisterHouse from a decade ago. Mozilla was there, showing off prototypes of FirefoxOS. I got to geek out by working with Chris Turra on flashing one of the phones to a more recent build. ODROID Hard Kernel looked like a cool way to make your own systems, reminding me of RaspberryPi. OpenBSD was there with swag and classic copies of the OS. Ticketmaster deserves a mention because they had Krispy Kreme doughnuts and cat pictures. I personally don’t know what started the fascination with cat pictures, but they did make me chuckle. I spoke with one of the guys from AMd’s SeaMicro, which packs a lot of processor power into a ???tiny??? space. They seem to be a nice fit for virtualization or Linux containers (See above). Unfortunately, there were a few talks I missed, but still want to highlight: Linux Performance Analysis and Tools by Brendan Gregg Packaging Software for Non-Developers by Thomas Cameron Introducing Continuous Integration with Jenkins by Kohsuke Kawaguchi. Although it was beginner-level based, it does deserve a mention because it’s just darn cool and I plan on setting one up again soon. OrganeFS: Next Generation Parallel Scale-Out File System by Boyd Wilson. Pi On Your Face: Getting Hands-On With Raspberry Pi by Ruth Suehle. If I ever find the time to play with hardware projects, this talk would have been handy. GlusterFS for SysAdmins by Eco Willson Ceph: Managing A Distributed Storage System At Scale by Sage Weil ZFS Administration by Aaron Toponce Extending NAS Functionality with the FreeNAS Plugins System by Dru Lavigne BoF sessions: Inbound and Outbound spam control methods, Functional Programming Coreboot- open source firmware and bootloaders Now I remember why I loved college. I had so much time on my hands, I could get into and learn so many different things.