Everyone here at End Point has been busy lately, so we haven?t had as much time as we?d like to blog. Here are some of the projects we?ve been knee deep in:

• The Liquid Galaxy Team (Ben, Adam, Kiel, Gerard, Josh, Matt) has been working on several Liquid Galaxy installations, including one at the Monterey Bay National Marine Sanctuary Exploration Center in Santa Cruz, and one for the Illicit Networks conference in Los Angeles. Adam has also been preparing Ladybug panoramic camera kits for clients to take their own panoramic photos and videos. The Liquid Galaxy team welcomed new employees Aaron Samuel in July, and Bryan Berry just this week.
• Brian B. has been improving a PowerCLI script to manage automated cloning of VMware vSphere virtual machines.
• Greg Sabino Mullane has been working on various strange PostgreSQL database issues, and gave a riveting presentation on password encryption methods.
• Josh Tolley has been improving panoramic photo support for Liquid Galaxy and expanding a public health data warehouse.
• David has been at work on a web-scalability project to support customized content for a Groupon promotion, while continuing to benefit from nginx caching. He has also been working on the design of highly-available PostgreSQL clusters with multiple indepdent synchronization requirements, and SEO maintenance work on an Interchange-driven site.
• Ron has been adding shop functionality and DevCamps to several client websites.
• Phin has been working on a Django-based inventory app to help End Point keep track of servers, virtual machines, updates, backup verification, and other details about our hosting, monitoring, and other infrastructure concerns.
• Steph has been busy on a Piggybak Ruby on Rails 3 project that is scheduled to launch soon, and she continues to work on other Rails and Interchange projects.
• Jeff has been integrating support for promotion codes for two clients? websites, and adding a third-party database support to another client website.
• Carl set up automated emails for expiring subscriptions, added a separate website that runs from a client?s main admin, and set up a way to show product pages in a third-party website within an iframe.
• Josh Williams recreated an existing environment in Amazon Web Services EC2 instances and helped multiple clients with streaming replication on PostgreSQL.
• Phunk has been busy wrapping up a large Rails 3 project and is transitioning to another large Rails 3 with a nice API and elegant metadata management using ElasticSearch, CouchDB, OAuth2 and jQuery.
• Marina has been working on several Rails projects, including building out a new Rails 3.2 site with RailsAdmin and integration of several third-party community rich features.
• Greg D. has been building out functionality to help users create data visualizations for another client using Django and Weave. He also recently gave a company presentation on popular JavaScript libraries.
• Jon has been interviewing candidates for our Ruby on Rails developer position. There have been lots of strong applicants. He?s also been looking into Git?s post-checkout hook for some added automation, troubleshooting some RPM build problems, and adding Rails (for RailsAdmin) to an existing Sinatra site via some neat nginx configuration. And he continues to be surprised by how bad Amazon?s EC2 I/O and CPU performance can be.
• Mark set up PayPal for two Interchange clients and set up saved credit cards via Authorize.Net CIM integration for another.
• Mike has been busy at work on a large scale Spree project building out custom support for custom CSS and account management integration.
• And finally, Rick has been holding down the fort which includes integrating new clients, keeping our many current clients happy, managing company finances, and keeping the machine running and well-oiled overall!

As you can see, we?ve got a lot of variety here, so let us know if you?d like to hear more on any of these topics, or if some features we?ve developed for others may be useful to you.

I wrote about Radian6 in my earlier blog post. Today I will review one more aspect of Radian6 API - call pagination.

Most Radian6 requests return paginated data. This introduces extra complexity of making request several times in the loop in order to get all results. Here is one simple way to retrieve the paginated data from Radian6 using the powerful Ruby blocks.

I will use the following URL to fetch data:
/data/comparisondata/1338958800000/1341550800000/2777/8/9/6/

Let's decypher this.

• 1338958800000 is start_date, 1341550800000 is end_date for document search. It's June, 06, 2012 - July, 06, 2012 formatted with date.to_time.to_i * 1000.
  
• 2777 is topic_id, a Radian6 term, denoting a set of search data for every customer.
• 8 stands for Twitter media type. There are various media types in Radian6. They reflect where the data came from. media_types parameter can include a list of values for different media types separated by commas.
• 9 and 6 are page and page_size respectively.

First comes the method to fetch a single page.
In the Radian6 wrapper class:

def page(index, &block)
  data = block.call(index) 
  articles, count = data['article'], data['article_count'].to_i  
  [articles, count]
end

A data record in Radian6 is called an article. A call returns the 'article' field for a page of articles along other useful fields.

Now we will retrieve all pages of data from Radian6:

def paginated_call(&block)
  articles, index, count = [], 0, 0
  begin
    index += 1
    batch, count = page(index, &block)
    articles += batch 
  end while count > 0
  articles
end

Time to enjoy the method! I'm using httparty gem to make requests to API.

paginated_call do |page|
  get("/data/comparisondata/1338958800000/1341550800000/2777/8/#{page}/1000/")
end

Thanks for flying!

Before I got married, I used a gmail account associated with my maiden name (let's call this account A). And after I got married, I switched to a new gmail address (let's call this account B). This caused daily annoyances as my use of various Google services was split between the two accounts.

Luckily, there are some services in Google that allow you to easily toggle between two accounts, but there is no easy way to define which account to use as the default for which service, so I found myself toggling back and forth frequently. Unfortunately, Google doesn't provide functionality to merge multiple Google accounts. You would think they might, especially given my particular situation, but I can see how it's a bit tricky in logically determining how to merge data. So, instead, I set off on migrating all data to account B, described in this email.

Consider Your Google Services

First things first, I took at look at the Google Services I used. Here's how things broke down for me:

• Gmail: Account A forwards to account B. I always use account B.
• Google+: Use through account A.
• Google Analytics: Various accounts divided between account A and account B.
• Blogger: Use through account A.
• Google Calendar: Various calendars split between account A and account B.
• Google Documents: Various documents split between account A and account B.
• Google Reader: Don't use.
• Google Voice: Use through account B.
• YouTube: Don't use (other than consumption).

After reviewing this list, I determined I would have to migrate Google+, several Google Analytics accounts, Blogger, Google Calendar, and Google Documents. I set off to look for various directions for merging or migrating data, broken down below.

Google+

Google Plus was easy to migrate. I followed the directions described here, which essentially involves sharing circles from account A to account B and then importing circles in account B. It was quick and easy.

Google Analytics

Google Analytics was a little more time consuming. In all accounts assigned to account A, logged in as account A, I added account B as an Admin user. Then, I logged in as account B, downgraded account A to a regular user from admin and deleted account A. Note that you must downgrade an account before you are allowed to delete the user, from my experience.

Blogger

To migrate Blogger account settings, I invited account B to a blog logged in as account A in browser #1. In browser #2, I logged in as account B and accepted the invitation. In browser #1, I gave account B admin access logged in as account A. After verifying account B admin access in browser #2, I went back to browser #1 and removed account A from the blog. I repeated these steps until I had transitioned all blogs.

Google Documents

The Google Documents migration was by far the most time consuming data migration for all the services. This article from Lifehacker says, "If you're migrating to a regular Google account, transferring your Google Docs is easy. Just select all the documents you want to migrate, go to the More Actions drop down menu, and choose Change Owner. Type in Account 2's address in the box that comes up. You'll see all your documents in Account 2." For about 50% of the documents owned by account A, I was able to change the owner under the shared options from account A to account B.

But for an unexplained reason, I was not allowed to re-assign the owner for the remaining documents. I couldn't find any explanation why this was the case. So, I migrated the data by brute force: I downloaded the remaining data owned by account A's account, and uploaded it as account B. This was irksome and time consuming, but it was my last step in finishing the migration!

Associated Email Accounts

One quick change I had to make here was to remove my End Point email association with account A and add it to account B, so that any documents shared with my End Point email address would be visible by account B. This was done under Google Account Settings.

Conclusion

The time spent on the account migration was worth it, in retrospect! There are many available resources for merging other Google Services if you find yourself in a similar position. Google it ;)

When I work for multiple clients on multiple different projects, I usually need a bunch of different stuff on my machine. One of the things I need is having multiple PostgreSQL versions installed.

I use Ubuntu 12.04. Installing PostgreSQL there is quite easy. Currently there are available two versions out of the box: 8.4 and 9.1. To install them I used the following command:

~$ sudo apt-get install postgresql-9.1 postgresql-8.4 postgresql-client-common

Now I have the above two versions installed.

Starting the database is also very easy:

~$ sudo service postgresql restart
 * Restarting PostgreSQL 8.4 database server   [ OK ] 
 * Restarting PostgreSQL 9.1 database server   [ OK ] 

The problem I had for a very long time was using the proper psql version. Both database installed their own programs like pg_dump and psql. Normally you can use pg_dump from the higher version PostgreSQL, however using different psql versions can be dangerous because psql uses a lot of queries which dig deep into the PostgreSQL internal tables for getting information about the database. Those internals sometimes change from one database version to another, so the best solution is to use the psql from the PostgreSQL installation you want to connect to.

The solution to this problem turned out to be quite simple. There is a pg_wrapper program which can take care of the different versions. It is enough to provide information about the PostgreSQL version you want to connect to and it will automatically choose the correct psql version.

Below you can see the results of using psql --version command which prints the psql version. As you can see there are different psql versions chosen according to the --cluster parameter.

~$ psql --cluster 8.4/main --version
psql (PostgreSQL) 8.4.11
contains support for command-line editing
~$ psql --cluster 9.1/main --version
psql (PostgreSQL) 9.1.4
contains support for command-line editing

You can find more information in the program manual using man pg_wrapper or at pg_wrapper manual

A very common, somewhat primitive approach to Interchange searching uses an approach like this:

The search profile contains something along the lines of --

  mv_search_type=db
  mv_search_file=products
  mv_column_op=rm
  mv_numeric=0
  mv_search_field=category

[search-region]
  [item-list]
    [item-field description]
  [/item-list]
[/search-region]

In other words, we search the products table for rows whose column "category" matches an expression (with a single query), and we list all the matches (description only). However, this can be inefficient depending on your database implementation: the item-field tag issues a query every time it's encountered, which you can see if you "tail" your database log. If your item-list contains many different columns from the search result, you'll end up issuing many such queries:

[item-list]
    [item-field description], [item-field weight], [item-field color],
    [item-field size], [item field ...]
  ...

resulting in:

SELECT description FROM products WHERE sku='ABC123'
SELECT weight FROM products WHERE sku='ABC123'
SELECT color FROM products WHERE sku='ABC123'
SELECT size FROM products WHERE sku='ABC123'
...

(Now, some databases are smart enough to cache query results, but some aren't, so avoiding this extra work is probably worth your trouble even on a "smart" database, in case your Interchange application gets moved to a "dumb" database sometime in the future.)

Fortunately, it's easy to correct:

mv_return_fields=*

and then

...
    [item-param description]
...

in place of "item-field".

Sometimes bugs in other people's code makes me think I might be crazy. I?m not talking Walter Sobchak gun-in-the-air-and-a-Pomeranian-in-a-cat-carrier crazy, but ?I must be doing something incredibly wrong here? crazy. I recently ran into a Rails 3 ActiveRecord caching bug that made me feel this kind of crazy. Check out this pretty simple caching setup and the bug I encountered and tell me; Am I wrong?

I have two models with a simple parent/child relationship defined with has_many and belongs_to ActiveRecord associations, respectively. Here are the pertinent bits of each:

class MimeTypeCategory < ActiveRecord::Base
  # parent class
  has_many :mime_types

  def self.all
    Rails.cache.fetch("mime_type_categories") do
    MimeTypeCategory.find(:all, :include => :mime_types)
  end
end

class MimeType < ActiveRecord::Base
  # child class
  belongs_to :mime_type_category
end

Notice how in MimeTypeCategory.all, we are eager loading each MimeTypeCategory?s children MimeTypes because our app tends to use those MimeTypes any time we need a MimeTypeCategory. Then, we cache that entire data structure because it?s a good candidate for caching and we like our app to be fast.

Now, to reproduce this Rails caching bug, I clear my app?s cache using 'Rails.cache.clear' in the rails console, then load any page in my app that calls MimeTypeCategory.all. The page loads successfully and shows no errors. Doesn?t sound like a bug so far, right? If I load that same page a second time, I will get the standard Rails error page with:

undefined class/module MimeType
...
(app/models/mime_type_category.rb:17:in 'all')

Crazy, right? Why does it *appear* that one cannot cache model instances in Rails, and why did it work for exactly one page request after the Rails cache was cleared? Well, the former obviously cannot be true, and the latter is due to how Rails.cache.fetch handles cache misses and cache hits. For a cache miss, Rails.cache.fetch executes its block, serializes the return value, saves it to your cache store, then returns the block?s return value directly. For a cache hit, it reads the cached block from your cache store, deserializes it into whatever objects it identifies itself as, and returns that.

This is all well and good until you?re going along, innocently working on your app in the development Rails environment with config.cache_classes = false (which forces your app to lazy-load requested classes for each page request.) In that situation, Rails will try to deserialize the cached data structure that had references to the MimeType class. But, Rails may not have loaded the MimeType class at that point, so deserialization will fail and produce the error we see there. If you have other code paths in your app that do happen to load the child class before this type of cached parent/child class data structure, you might not hit the bug. Now you?ve entered a world of debugging pain.

I?m not about to give up on automatic class reloading in my development environment, and I don?t want to remove the cached eager loading of my child MimeTypes class because it?s sweet. So, after some digging, I discovered a solution: require_association. Adding ?require_association ?mime_type? to my parent MimeTypeCategory class forces Rails to load the MimeType model when it loads the MimeTypeCategory model such that it can always deserialize the cached data structure successfully. I?ve used require_association in the same way for other instances of the same caching bug in our app as well.

Hopefully this explanation helps people avoid some of the pain I experienced while trying to determine if it was a Rails bug/feature or if I had finally gone insane. I should point out that some of the reading I?ve done suggests ?require_dependency? is the more appropriate solution for this problem. I?ve verified that require_association works in all my cases, but to avoid ?programming by coincidence,? I am going to snoop around the Rails core to understand the difference between the two.

Lastly, please remember: You can?t board a Pomeranian - they get upset and their hair falls out.

We're looking for another talented Ruby on Rails developer to consult with our clients and develop their Ruby on Rails web applications. If you like to focus on solving business problems and can take responsibility for getting a job done well without intensive oversight, please read on!

End Point is a 17-year-old web consulting company based in New York city, with 28 full-time employees working mostly remotely from home offices. Our team is made up of strong ecommerce, database, and system administration talent, working together using ssh, Screen and tmux, IRC, Google+ Hangouts, Skype, and good old phones.

We serve over 200 clients ranging from small family businesses to large corporations, using a variety of open source technologies including Ruby, Python, Perl, Git, PostgreSQL, MySQL, RHEL/CentOS, Debian, and Ubuntu.

What is in it for you?

• Work from your home office
• Flexible full-time work hours
• Annual bonus opportunity
• Health insurance benefit
• 401(k) retirement savings plan
• Ability to move without being tied to your job location

What you will be doing:

• Consult with clients to determine their web application needs
• Build, test, release, and maintain web applications for our clients
• Work with open source tools and contribute back as opportunity arises
• Use your desktop platform of choice: Linux, Mac OS X, Windows

What you will need:

• Professional experience building reliable Ruby on Rails apps
• Good front-end web skills with HTML, CSS, and JavaScript
• Experience with PostgreSQL, MySQL, or another relational database
• A customer-centered focus
• A passion for building flexible and, where needed, scalable web applications
• Strong verbal and written communication skills
• Experience directing your own work, and working from home
• Ability to learn new technologies

Bonus points for experience:

• Building and supporting ecommerce systems, including with Spree
• Working with other languages and web app frameworks
• Contributing to gems or other open source projects
• Handling system administration and deployment

Please email us an introduction to jobs@endpoint.com to apply. Include a resume, your GitHub or LinkedIn URLs, or whatever else that would help us get to know you. We look forward to hearing from you!

The "UltraBooks" trend has been been made popular by Apple's efforts at making the computer an appliance. There are clear benefits to this, but for enthusiasts, it means less control to upgrade to hardware of their choosing. Let's contrast this experience with my own Dell Vostro 3500. It was purchased two years ago and yet through upgrades, it has many features the recently refreshed MacBook Pro with Retina display lacks, while offering the same processing power.

MacBook Pro Advantages

Let's be clear, the MacBook Pro (MBP) has a number advantages over my Dell. The Retina display of course trounces my 1376 x 768 resolution on a 15" display. For designers, photographers, and video editors, Retina is a great asset. For someone who spends their day inside VIM and SSH, not so much. Similarly, the MBP's seven hour battery life is impressive, although I'm sure with my processor intensive work load, it would be lower. But for me, I'm not terribly mobile. I need to be mobile from room to room perhaps, but I don't find myself without power and needing to complete mission critical work less frequently than my replaceable 90 watt-hour battery can provide over 3 to 4 hours. Similarly, weight is not my number one concern; I simply stash the laptop in my backpack and go.

Incremental Upgrades

With my Dell, I purchased the baseline Vostro 3500 for $1,000 which included next business day service and accidental damage protection for three years. This means for parts which I cannot easily replace myself, like the display and motherboard, I'm covered. Over the years, I've upgraded the:

• processor to an i5-580M, with Turbo Boost up 3.3GHz. Great for single threaded Ruby ($225).
• wireless card to an Intel Wifi Link 5300 for better signal strength and 802.11n support ($42).
• 65whr battery at it's end of life with a 90whr battery ($160 ouch!).
• original HDD with a 160GB Intel 320 series SSD ($220).
• optical drive with a 2.5" drive caddy ($52) and a 500GB Seagate Momentus XT Hybrid drive ($150).
• RAM to 8GB ($99).

For $950 of upgrades over two years, I've got a machine with specs that are powerful in the places I care about, at a price I can afford and pay for incrementally. To get the same processor and warranty on the MacBook Pro, I need to shell out $3,150 and give up my eSATA, HDMI, VGA, PCI-Express, Gigabit Ethernet ports and 500GB of storage (upgradable to 1TB for $100). In short, I'm expected to pay a premium price for features which aren't of my choosing, while giving up ports.

While this might not be possible for all people, developers should look long and hard at what it is they value in their workstation and not blindly accept whatever Apple hypes. You might find you don't care quite as much about the "features" Apple wants you to.

Interchange has a handy feature (which, in my almost-seven-years of involvement, I'd not seen or suspected) allowing you to create an item "on-the-fly", without requiring any updates to your products table. Here's a recipe for making this work.

First, you need to tell Interchange that you're going to make use of this feature (in catalog.cfg).

OnFly onfly

Simple, no? The "OnFly" directive names a subroutine that is called to pre-process the custom item before it's added to the cart. The default "onfly" routine can be found in the system tag "onfly": code/SystemTag/onfly.coretag in the standard Interchange installation. (If you need more that what it provides, that's beyond the scope of my post, so good luck, bon voyage, and please write back to let us know what you learned!)

Then, you need to submit some special form parameters to set up the cart:

• mv_order_item: the item number identifying this line
• mv_order_fly: a structured string with | (vertical bar) delimiters. Each sub-field specifies something about the custom item, thus:

  description=My custom item|price=12.34

Now, in my particular case, I was encapsulating an XML feed of products from another site (a parts supplier) so that the client (a retail seller) could offer replacement parts, but not have to incorporate thousands of additional lines in the "products" table. So after drilling down to the appropriate model and showing the available parts, each item got the following bit of JavaScript (AJAX) code associated with its add-to-cart button:

var $row = $(this).parents('tr');
    $.ajax({
        url: '/cgi-bin/mycat/process',
        data: {
            mv_todo: 'refresh',
            mv_order_quantity: 1,
            mv_order_item: $row.find('td.item_number').html(),
            mv_order_fly: 'description='
                + $row.find('td.description').html().replace('|','')
                + '|'
                + 'price='
                + $row.find('td.price').html().replace('$','').replace(',','')
        },
        method: 'POST',
        success: function(data, status) {
            $('#msg_div').html('Added to cart.')
        }
    });

And that's all it took. With Interchange, you don't even need a special "landing page" for your AJAX submission; Interchange handles all the cart-updating out of sight.

I still need to add some post-processing to handle errors, and update the current page so I can see the new cart line count, but the basics are done.

I'm not a big fan of add-on graphical control panels for Linux such as cPanel, Webmin, Ensim, etc. They deviate from the distributor's standard packages and locations for files, often simultaneously tightening security in various ways and weakening security practically by making several more remotely accessible administration logins.

On one of the few servers we maintain that has cPanel on it, today we did a routine Red Hat Network update and reboot to load the latest RHEL 5 kernel, and all seemed to go well.

However, within a few minutes we started getting emailed reports from the cPanel service monitor saying that the Exim mail server had failed and been restarted. These emails began coming in at roughly 5-minute intervals:

Date: Tue, 24 Jul 2012 14:21:05 -0400
From: cPanel ChkServd Service Monitor <cpanel@[SNIP]>
To: [SNIP]
Subject: exim on [SNIP] status: failed

exim failed @ Tue Jul 24 14:21:04 2012. A restart was attempted automagically.

Service Check Method:  [socket connect] 

Reason: TCP Transaction Log: 
<< 220-[SNIP] ESMTP Exim 4.77 #2 Tue, 24 Jul 2012 14:21:04 -0400 
<< 
<< 
>> EHLO localhost
<< 250-[SNIP] Hello localhost.localdomain [127.0.0.1]
<< 
<< 
<< 
<< 
<< 
>> AUTH PLAIN
[SNIP]=
<< 535 Incorrect authentication data
exim: ** [535 Incorrect authentication data != 2]
: Died at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 689, <$socket_scc> line 10.


Number of Restart Attempts: 1

Startup Log: Starting exim: [  OK  ]

And the relevant entry in /var/log/exim_mainlog was:

2012-07-24 14:08:05 fixed_plain authenticator failed for localhost.localdomain (localhost) [127.0.0.1]:48454: 535 Incorrect authentication data (set_id=__cpane
l__service__auth__exim__[SNIP])

I wasn't able to find a way to fix this in any reasonable amount of time, so I opened a trouble ticket with cPanel support and they had asked for server access, logged in, and fixed the problem within a little over an hour. It was about as painless as tech support ever gets, so kudos to cPanel for that!

The solution was to run this as root:

/scripts/upcp --force

Which resyncs cPanel so that chkservd reports Exim as up and the unwanted service restarts no longer happen.

Here's to responsive tech support.

Most small businesses cannot afford the high performance storage area networks (SANs) that make traditional redundancy options such high availability and fault tolerance possible. Despite this, the APIs available to administrators of virtualized infrastructure using direct attached storage (DAS) make it possible to recreate many of the benefits of high availability.

High Availability on SAN vs DAS

A single server failure in a virtualized environment can mean many applications and services can become unavailable simultaneously; for small organizations, this can be particularly damaging. High availability with SANs minimize the downtime of applications and services when a host fails by keeping virtual machine (VM) storage off the host and on the SAN. VMs on a failed host can then be automatically restarted on hosts with excess capacity. This of course requires SAN infrastructure to be highly redundant, adding to the already expensive and complex nature of SANs.

Alternatively, direct attached storage (DAS) is very cost effective, performant, and well understood. By using software to automate the snapshot and cloning of VMs via traditional gigabit Ethernet from host to host, we can create a "poor man's" high availability system.

It's important for administrators to understand that there is a very real window of data loss that can range from hours to days depending on the number of systems backed up and hardware in use. However, for many small businesses who may not have trustworthy backups, automated cloning is an excellent step forward.

Automated cloning with VMWare's PowerCLI

Although End Point is primarily an open source shop, my introduction virtualization was with VMWare. For automation and scripting, PowerCLI, the PowerShell based command line interface for vSphere, is the platform on which we will build. The process is as follows:

• A scheduled task executes the backup script.
• Delete all old backups to free space.
• Read CSV of VMs to be backed up and the target host and datastore.
• For each VM, snapshot and clone to destination.
• Collect data on cloning failures and email report.

I have created a public GitHub repository for the code and called it powercli_cloner.

Currently, it's fairly customized around the needs of the particular client it was implemented for, so there is much room for generalization and improvement. One area of improvement is immediately obvious: only delete a backup after successfully replacing it. Also, the script must be run as a Windows user with administrator vSphere privileges, as the scripts assumes pass-through authentication is in place. This is probably best for keeping credentials out of plain text. The script should be run during non-peak hours, especially if you have I/O intensive workloads.

Hopefully this tool can provide opportunities to develop backup and disaster recovery procedures that are flexible, cost-effective, and simple. I'd welcome pull requests and other suggestions for improvement.

A few weeks ago, End Point graciously agreed to send me to GoRuCo, "the premier Ruby conference in New York City". I was excited to try and apply the lessons from our own Greg Sabino Mullane, who gave a talk about attending conferences during End Point's 2012 Company Meeting. He emphasized a focus on interacting with the speakers and attendees instead of the presentation content.

Pre Party at Pivotal Labs

The pre-party was located just blocks away from our offices, making it a convenient after work stop. On my walk to the party, I tried to think about the type of connections I wanted to make and the topics I wanted to discuss. I was intrigued by Pivotal Lab's policy on pair programming, but realized I could continue to read up about that extensively online. What was a question I could ask that would be interesting for both asker and askee, providing me something a Google search couldn't?

After the usual introductory chit chat, I found myself asking, "so, what are you struggling with in your work right now?" It was broad and open, giving the speaker a chance to perhaps go somewhere outside of the normal conversation. I asked Haris Amin this question and got what would become a fairly typical answer. He summarized his struggle as one of keeping up with the firehose of his interests. So many great technologies are springing from all corners of creation, there simply aren't enough hours in the day to investigate them all. This resonated against our own Jon Jensen's comments during the company meeting about it being a golden age for technology.

I also got to meet and chat with Coby Randquist, the creator of Confreaks. It was great learning about his transition from implementer to manager. As a manager, he found he loved that he could solve much bigger problems than as a developer by bringing together talent and keeping "everyone else out of their way".

A Sampling of the Main Event

Of course there were so many great presentations, which you can watch at your leisure, but because there are always more presentations than time, I'll focus on one that I found particularly interesting. Dr. Nic gave a talk called The Future of Deployment (slides) which he laid out an argument for being able to able to version control and deploy not only our applications, but the infrastructure which supports them, all with a unified tool.

He lays out some interesting demands of this tool, including:

• An explicit description of all infrastructure dependencies
• The ability to manage all parts of this infrastructure, from the size of our AWS instances, to the particular version of Nginx we're running.
• Full version control/history of these descriptions
• A central API for all activities

• Independent install paths; if we can use .../releases/TIMESTAMP for Capistrano deployments, why not the same for Ruby and Nginx?
• Portable enterprise deployments; we should be able to offer our applications as a self building system *behind* the firewall

So what's the magical tool which will finally solve all our deployment problems? He offers BOSH, "an open source tool chain for release engineering, deployment and lifecycle management of large scale distributed services". It was created to manage *really* big deployments, so Dr. Nic argues perhaps it can also meet our smaller needs. BOSH is kind of a bear to get your head around, so Dr. Nic helpfully created a more palatable Getting Started series that can help bootstrap your experience.

If you're interested in hearing more about why we should care about BOSH, I'd say watch the presentation, but if you don't need to be sold on the idea and know you need it, the presentation is *not* about BOSH really at all, outside of it offering to address some of Dr. Nic's requirements above.

End Point was tasked with setting up and manning a Liquid Galaxy at the two day INFO Summit at the Four Seasons Hotel in Westlake Village, CA. Josh Ausborne and Alejandro Ramon were in attendance representing End Point, and performed the setup and the running of the system.

INFO is the short name for Illicit Networks: Forces in Opposition. The INFO Summit is an event designed to get people to understand how drug smugglers, arms dealers, and human traffickers operate, and how people can get involved to develop technology in an effort to disrupt and thwart the networks. You can read an overview about the summit and its purpose here.

Jared Cohen, Director of Google Ideas, was interviewed by Fast Company to discus the breaking down of illicit networks. According to Cohen, Google "focused on disrupting violent and coercive illicit networks, including drug cartels, the mafia, human trafficking rings, organ harvesters, illicit arms dealers, and forced labor networks." You can read the article here where he explains some of the ways that Google is stepping up the fight.

End Point's involvement in the summit is as the installer and presenter of a Liquid Galaxy tour that shows the various illicit network hotspots around the world. These locations include various places such as Africa, the Middle East, Azerbaijan, Russia, Mexico, and even the United States. The tour is a simple KML tour with text popups to give information on a given hotspot before flying to the next location. It provides general Illicit Network information that the people are looking for, but not so much that they are unable to finish reading it before the tour moves on to the next location. We could have included some more fancy design in the tour bubbles, but in this case we didn't want to distract from the content itself.

We have had numerous people approach us about how the Liquid Galaxy could be utilized in various situations, and we have presented ideas for each of them. One of them was a person involved with a US military school, and she is interested in how the LG could be used in a military education setting.

Another is head of security and emergency response for a very large international facility, and was wondering how the LG could be used to show watershed concerns, construction, and high security areas. We explained the idea of screen overlays to identify areas of high value, as well as weather updates and tracking.

During the conference, we have had a chance to sit and discuss the use of KML with Sean Askay, Developer Support Engineer of the Google Earth Outreach group. We talked about the current and future use of KML within Google Earth, how we expect it to grown in the future, as well as how End Point can continue to grow in its Liquid Galaxy / Google Earth role.

Overall, the tour has been a success as far as the Liquid Galaxy is concerned. We have shown a good tour on a beautiful system, have made contact with many interested people, and have been able to show how the LG can be utilized to meet the various needs of the conference attendees.

We ran into an unpleasant bug in a Red Hat Enterprise Linux 6 glibc update a couple of weeks ago. It since has made its way into CentOS 6 as well.

The problem manifested itself in our particular case with this error from the Postfix mailer:

Jun 29 01:55:23 sl37 kernel: smtp[7093]: segfault at 1 ip 00007ffc0e455596 sp 00007fff99948f60 error 6 in libresolv-2.12.so[7ffc0e449000+16000]

But it affects all DNS resolution on the host, not just for mail.

If you have any IPv6 resolvers at all listed in /etc/resolv.conf, all your DNS resolution is likely to be broken with this version of glibc:

glibc-2.12-1.80.el6.x86_64

To work around the problem, you can either:

• Use only IPv4 DNS resolvers (comment out the IPv6 resolvers for now)
• or downgrade to the previous version of glibc using yum downgrade

Red Hat is aware of the bug and you can track progress toward a resolution in Bugzilla bug #835090.

If you're using IPv6, watch out for this! If not, you're fine.