All opinions expressed are those of the authors and not necessarily those of OSNews.com, our sponsors, or our affiliates.
  Add to My Yahoo!  Subscribe with Bloglines  Subscribe in NewsGator Online

published by noreply@blogger.com (Josh Williams) on 2017-03-01 02:07:00 in the "linux" category
We had an interesting problem to track down. (Though I suppose I wouldn't be writing about it if it weren't, yes?) Over the years a client had built up quite the collection of scripts executed by cron to maintain some files on their site. Some of these were fairly complex, taking a long while to run, and overlapping with each other.

One day, the backup churn hit a tipping point and we took notice. Some process, we found, seemed to be touching an increasing number of image files: The contents were almost always the same, but the modification timestamps were updated. But digging through the myriad of code to figure out what was doing that was proving to be somewhat troublesome.

Enter auditd, already present on the RHEL host. This allows us to attach a watch on the directory in question, and track down exactly what was performing the events. -- Note, other flavors of Linux, such as Ubuntu, may not have it out of the box. But you can usually install it via the the auditd package.
(output from a test system for demonstration purposes)
# auditctl -w /root/output
# tail /var/log/audit/audit.log
type=SYSCALL msg=audit(1487974252.630:311): arch=c000003e syscall=2 success=yes exit=3 a0=b51cf0 a1=241 a2=1b6 a3=2 items=2 ppid=30272 pid=30316 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts2 ses=1 comm="script.sh" exe="/usr/bin/bash" key=(null)
type=CWD msg=audit(1487974252.630:311):  cwd="/root"
type=PATH msg=audit(1487974252.630:311): item=0 name="output/files/" inode=519034 dev=fd:01 mode=040755 ouid=0 ogid=0 rdev=00:00 objtype=PARENT
type=PATH msg=audit(1487974252.630:311): item=1 name="output/files/1.txt" inode=519035 dev=fd:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 objtype=CREATE
The most helpful logged items include the executing process's name and path, the file's path, operation, pid and parent pid. But there's a good bit of data there per syscall.

Don't forget to auditctl -W /root/output to remove watch. auditctl -l will list what's currently out there:
# auditctl -l
-w /root/output -p rwxa
That's the short version. auditctl has a different set of parameters that are a little bit more verbose, but have more options. The equivalent of the above would be: auditctl -a always,exit -F dir=/root/output -F perm=rwxa ... with options for additional rule fields/filters on uid, gid, pid, whether or not the action was successful, and so on.
Comments

published by noreply@blogger.com (Neil Elliott) on 2015-04-20 19:58:00 in the "linux" category

As a high-performance video rendering appliance, the Liquid Galaxy requires really good video cards -- better than your typical on-board integrated video cards. Despite ongoing attempts by competitors to displace them, Nvidia remains the best choice for high-end video, if you use the proprietary Nvidia driver for Linux.

In addition to providing regular security and system updates, End Point typically provides advanced remote monitoring of our customers' systems for issues such as unanticipated application behavior, driver issues, and hardware errors. One particularly persistent issue presents as an error with an Nvidia kernel module.  Unfortunately, relying on proprietary Nvidia drivers so as to maintain an acceptable performance level limits the available diagnostic information and options for resolution.

The issue presents when the system ceases all video output functions as Xorg crashes. The kernel log contains the following error message:

2015-04-14T19:59:00.000083+00:00 lg2 kernel: [  719.850677] NVRM: Xid (0000:01:00): 32, Channel ID 00000003 intr 02000000

The message is repeated approximately 11000 times every second until the disk fills and the ability to log in to the system is lost. The only known resolution at this time is to power-cycle the affected machine. In the error state, the module cannot be removed from the kernel, which also prevents Linux from shutting down properly. All affected systems were running some version of Ubuntu x86-64. The issue seems to be independent of driver version, but is at least present in 343.36 and 340.65, and affects all Geforce cards. Quadro cards seem unaffected.

The Xid message in the kernel log contains an error code that provides a little more information. The Nvidia docs list the error as "Invalid or corrupted push buffer stream". Possible causes listed include driver error, system memory corruption, bus error, thermal error, or frame buffer error. All affected systems were equipped with ECC RAM and were within normal operating temperature range when the issue presented.

Dealing with bugs like these can be arduous, but until they can be fixed, we cope by monitoring and responding to problems as quickly as possible.


Comments

published by noreply@blogger.com (Jon Jensen) on 2013-09-18 15:21:00 in the "linux" category

Logging website visitor traffic is an interesting thing: Which details should be logged? How long and in what form should you keep log data afterward? That includes questions of log rotation frequency, file naming, and compression. And how do you analyze the data later, if at all?

Allow me to tell a little story that illustrates a few limited areas around these questions.

Reverse DNS PTR records

System administrators may want to make more sense of visitor IP addresses they see in the logs, and one way to do that is with a reverse DNS lookup on the IP address. The network administrators for the netblock that the IP address is part of have the ability to set up a PTR (pointer) record, or not. You can find out what it is, if anything.

For example, let's look at DNS for End Point's main website at www.endpoint.com using the standard Unix tool "host":

% host www.endpoint.com
www.endpoint.com has address 208.43.132.31
www.endpoint.com has IPv6 address 2607:f0d0:2001:103::31
% host 208.43.132.31
31.132.43.208.in-addr.arpa domain name pointer 208.43.132.31-static.reverse.softlayer.com.
% host 2607:f0d0:2001:103::31
1.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.3.0.1.0.1.0.0.2.0.d.0.f.7.0.6.2.ip6.arpa domain name pointer 2607.f0d0.2001.0103.0000.0000.0000.0031-static.v6reverse.softlayer.com.

The www.endpoint.com name points to both an IPv4 and an IPv6 address, so there are two answers. When each of those IP addresses is looked up, each shows a PTR record pointing to a subdomain of softlayer.com, which gives a clue about where our site is hosted.

(As an aside: Why don't we use a prettier or more specific PTR record? We could set it to almost whatever we want. Well, there are dozens of websites hosted on those IP addresses, so which one should be in the PTR record? There's no obvious choice, and it doesn't matter for normal network functioning, so we just left it the way it was.)

So, is a PTR record like these useful to know about visitors to your website? Sometimes. Let's take a look at a random sample of visitors to a different website we manage. How much can you tell about each of the visitors based on their reverse DNS PTR records? Is it a bot, someone at home or the office, in which country, and who is their Internet provider? How common is it for a visitor's IP address to have no PTR record? And keep in mind that most of the visitors have no idea what their IP address or its PTR record is.

% host 93.137.189.55
55.189.137.93.in-addr.arpa domain name pointer 93-137-189-55.adsl.net.t-com.hr.
% host 66.249.73.121
121.73.249.66.in-addr.arpa domain name pointer crawl-66-249-73-121.googlebot.com.
% host 88.134.68.31
31.68.134.88.in-addr.arpa domain name pointer 88-134-68-31-dynip.superkabel.de.
% host 67.49.156.20
20.156.49.67.in-addr.arpa domain name pointer cpe-67-49-156-20.hawaii.res.rr.com.
% host 123.211.36.234
234.36.211.123.in-addr.arpa domain name pointer CPE-123-211-36-234.lnse3.cha.bigpond.net.au.
% host 91.75.70.162 
Host 162.70.75.91.in-addr.arpa. not found: 3(NXDOMAIN)
% host 209.82.97.10
10.97.82.209.in-addr.arpa domain name pointer mail02.westjet.com.
% host 76.70.117.223
223.117.70.76.in-addr.arpa domain name pointer bas1-toronto26-1279686111.dsl.bell.ca.
% host 101.160.207.115 
Host 115.207.160.101.in-addr.arpa. not found: 3(NXDOMAIN)
% host 184.198.177.214
214.177.198.184.in-addr.arpa domain name pointer 184-198-177-214.pools.spcsdns.net.
% host 84.199.97.130
130.97.199.84.in-addr.arpa domain name pointer 84-199-97-130.iFiber.telenet-ops.be.
% host 182.19.87.24 
Host 24.87.19.182.in-addr.arpa. not found: 3(NXDOMAIN)
% host 62.34.219.216
216.219.34.62.in-addr.arpa domain name pointer i01v-62-34-219-216.d4.club-internet.fr.
216.219.34.62.in-addr.arpa domain name pointer lns-c10k01-v-62-34-219-216.dsl.sta.abo.bbox.fr.
% host 187.86.213.190
Host 190.213.86.187.in-addr.arpa. not found: 3(NXDOMAIN)
% host 15.211.201.84
84.201.211.15.in-addr.arpa domain name pointer zccy01cs104.houston.hp.com.
% host 161.69.46.150
150.46.69.161.in-addr.arpa domain name pointer miv-scan015.scanalert.com.
% host 77.182.146.99
99.146.182.77.in-addr.arpa domain name pointer essn-4db69263.pool.mediaWays.net.
% host 107.0.160.152 
152.160.0.107.in-addr.arpa domain name pointer 107-0-160-152-ip-static.hfc.comcastbusiness.net.

Did you notice that one IP address returned two different PTR records? That is allowed, though uncommon, as I mentioned in my blog post Multiple reverse DNS pointers per IP address a few years back. Many reverse DNS control panels provided by commodity hosting providers won't allow you to assign multiple PTR records, but if you get your reverse DNS delegated to a real nameserver you control, you can do it.

Finding the IP address owner: whois

The reverse DNS PTR can be set misleadingly, such that a forward lookup on the name does not point back to the same IP address. In the end the way to really know who controls that IP address (or at least a network provider who supplies the ultimately responsible person) is with a "whois" lookup. We can check that the 208.43.132.31 IP address really is hosted at SoftLayer, and for which customer, like this:

% whois 208.43.132.31
[Querying whois.arin.net]
[Redirected to rwhois.softlayer.com:4321]
[Querying rwhois.softlayer.com]
[rwhois.softlayer.com]
%rwhois V-1.5:003fff:00 rwhois.softlayer.com (by Network Solutions, Inc. V-1.5.9.5)
network:Class-Name:network
network:ID:NETBLK-SOFTLAYER.208.43.128.0/19
network:Auth-Area:208.43.128.0/19
network:Network-Name:SOFTLAYER-208.43.128.0
network:IP-Network:208.43.132.0/27
network:IP-Network-Block:208.43.132.0-208.43.132.31
network:Organization;I:End Point Corporation
network:Street-Address:920 Broadway, Suite 701
network:City:New York
network:State:NY
network:Postal-Code:10010
network:Country-Code:US
network:Tech-Contact;I:sysadmins@softlayer.com
network:Abuse-Contact;I:abuse@endpoint.com
network:Admin-Contact;I:IPADM258-ARIN
network:Created:2007-06-18 12:15:54
network:Updated:2010-11-21 18:59:43
network:Updated-By:ipadmin@softlayer.com

%referral rwhois://root.rwhois.net:4321/auth-area=.
%ok

So you can see that's really End Point's IP address, at SoftLayer.

Use your local whois tool or search for a web-based one and look up a few of the IP addresses that didn't have reverse DNS PTR records in our log cross-section above. The results are interesting.

Reverse lookups in Apache httpd

Now let's say that as a system administrator you would like to see the PTR records for visitor IP addresses on your Apache httpd website. It may be tempting to use the HostnameLookups configuration directive to do real-time lookups and put them in the log alongside the IP address. It's easy but not wise to put the PTR record instead of the IP address, because it may not point back to the IP address, and even if it does, it can change over time, and will not provide a complete picture of the connection later on.

However, if you read the HostnameLookups documentation, you'll see the authors recommend it not be enabled on busy production servers because of the extra network traffic and delay for visitors, especially for any netblocks with slow DNS servers (and there are many out there). This is important! It really should almost never be enabled for any public site.

Most web server administrators learn this early on and wouldn't dream of enabling HostnameLookups.

However, I recently came across a situation where we inadvertently were doing the equivalent without explicitly enabling HostnameLookups. How? By limiting access based on the remote hostname! Read the documentation on the Allow directive, under the section "A (partial) domain-name":

This configuration will cause Apache to perform a double reverse DNS lookup on the client IP address, regardless of the setting of the HostnameLookups directive. It will do a reverse DNS lookup on the IP address to find the associated hostname, and then do a forward lookup on the hostname to assure that it matches the original IP address. Only if the forward and reverse DNS are consistent and the hostname matches will access be allowed.

This makes perfect sense, but it is a pretty big likely unexpected side effect to using something like:

Allow from .example.com

In our case it was an even less obvious case that didn't make us think of hostnames at all:

Allow from localhost

Here localhost was written, perhaps to save some effort or maybe increase clarity vs. writing out 127.0.0.1 (IPv4) and ::1 (IPv6). Mentally it's so easy to view "localhost" is a direct alias for 127.0.0.1 and ::1 that we can forget that the name "localhost" is just a convention, and requires a lookup like any other name. Those familiar with the MySQL database may know that it actually assigns special confusing meaning to the word "localhost" to make a UNIX socket connection instead of a TCP connection to 127.0.0.1 or whatever "localhost" is defined as on the system!

You may also be thinking that looking up 127.0.0.1 is fast because that is usually mapped to "localhost" in /etc/hosts. True, but every other visitor who is not in /etc/hosts gets the slow DNS PTR lookup instead! And depending on the operating system, you may see "ip6-localhost" or "ip6-loopback" (Debian 7, Ubuntu 12.04), "localhost6" (RHEL 5/6, Fedora 19) in /etc/hosts, or something else. So it's important to spell out the addresses:

Allow from 127.0.0.1
Allow from ::1

Doing so immediately stops the implicit HostnameLookups behavior and speeds up the website. In this case it wasn't a problem, since it was for a private, internal website that couldn't be visited at all by anyone not first allowed through a firewall, so traffic levels were relatively low. That access control is part of why localhost needed to be allowed in the first place. But it would have been very bad on a public production system due to the slowdown in serving traffic.

The right way

If you really want to see PTR records for every visitor IP address, you can use Apache's logresolve log post-processing program. Or you can let an analytics package do it for you.

So, lesson learned: It's not just HostnameLookups you need to keep turned off. Also watch out for the Apache Allow directive and don't use it with anything other than numeric IP addresses!


Comments

published by noreply@blogger.com (Brian Buchalter) on 2012-03-01 13:53:00 in the "linux" category

As part of End Point's preparation for World IPv6 Launch Day, I was asked to get my IPv6 certification from Hurricane Electric. It's a fun little game-based learning program which had me setup a IPv6 tunnel. IPv6 tunnels are used to provide IPv6 for those whose folks whose ISP or hosting provider don't currently support IPv6, by "tunneling" it over IPv4. The process for creating a tunnel is straight forward enough, but there were a few configuration steps I felt could be better explained.

After creating a tunnel, Hurricane Electric kindly provides a summary of your configuration and offers example configurations for several different operating systems and routers. Below is my configuration summary and the example generated by Hurricane Electric.

However, entering these commands change won't survive a restart. For Debian/Ubuntu users an update in /etc/network/interfaces does the trick.

#/etc/network/interfaces
auto he-ipv6
iface he-ipv6 inet6 v4tunnel
  address 2001:470:4:9ae::2
  netmask 64
  endpoint 209.51.161.58
  local 204.8.67.188
  ttl 225 
  gateway 2001:470:4:9ae::1

Firewall Configuration

If you're running UFW the updates to /etc/default/ufw are very straightforward. Simply change the IPV6 directive to yes. Restart the firewall and your network interfaces and you should be able to ping6 ipv6.google.com. I also recommend hitting http://test-ipv6.com/ for a detailed configuration test.

Behind NAT

If you're behind a NAT, the configuration needs to be tweaked a bit. First, you'll want to setup a static IP address behind your router. If you're router supports configuration of forwarding more than just TCP/UDP, you'll want to forward protocol 41 (aka IPv6) (NOT PORT 41), which is responsible for IPv6 tunneling over IPv4, to your static address. If you've got a consumer grade router that doesn't support this, you'll just have to put your machine in the DMZ, thus putting your computer "in front" of your router's firewall. Please make sure you are running a local software firewall if you chose this option.

After handling the routing of protocol 41, there is one small configuration change to /etc/network/interfaces. You must change your tunnel's local address from your public IP address, to your private NATed address. Here is an example configuration including both the static IP configuration and the updated tunnel configuration.

#/etc/network/interfaces
auto eth0
iface eth0 inet static
  address 192.168.0.50
  netmask 255.255.255.0
  gateway 192.168.0.1 

auto he-ipv6
iface he-ipv6 inet6 v4tunnel
  address 2001:470:4:9ae::2
  netmask 64
  endpoint 209.51.161.58
  local 192.168.0.50
  ttl 225 
  gateway 2001:470:4:9ae::1

Don't forget to restart your networking interfaces after these changes. I found a good ol' restart was helpful as well, but of course, we don't have this luxury in production, so be careful!

Checking IPv6

If you're reading this article, you're probably responsible for several hosts. For a gentle reminder which of your sites you've not yet setup IPv6, I recommend checking out IPvFoo for Chrome or 4or6 for Firefox. These tools make it easy for you to see which of your sites are ready for World IPv6 Launch Day!

Getting Help

Hurricane Electric provides really great support for their IPv6 tunnel services (which is completely free). Simply email ipv6@he.net and provide them with some useful information such as:

cat /etc/network/interfaces
cat netstat -nrA inet6  (these are your IPv6 routing tables)
cat /etc/default/ufw
relevant router configurations
I was very impressed to get a response from a competent person in 15 minutes! Sadly, there is one downside to using this tunnel; IRC is not an allowed.
Due to an increase in IRC abuse, new non-BGP tunnels now have IRC blocked by default. If you are a Sage, you can re-enable IRC by visiting the tunnel details page for that specific tunnel and selecting the 'Unblock IRC' option. Existing tunnels have not been filtered.
I guess ya gotta earn it to use IRC over your tunnel. Good luck!


Comments

published by noreply@blogger.com (Greg Sabino Mullane) on 2011-10-17 19:00:00 in the "linux" category

Image by Flickr user p886

My beloved mutt/imapfilter combo recently stopped working after an operating system switch. (tl;dr: that combo rocks; use ipairs instead of pairs) When my laptop wireless stopped working, and after spending some time fighting with it, I decided to simply install a new OS. As all of my important data is on a separate partition, this was not that big a deal. I ended up using Scientific Linux, as I'd heard good things about it, and it was one of the few distros that actually would install on my laptop (failures for one reason or another: Fedora, FreeBSD, Ubuntu, and OpenBSD). After the install, I simply copied my ~/.mutt directory and ~/.muttrc file into place, and similarly copied my ~/.imapfilter directory, which contained the all important config.lua file. The imapfilter program itself was not available via the normal yum repositories, so I simply grabbed the latest and greatest and did a manual install:


$ git clone https://github.com/lefcha/imapfilter.git
$ cd imapfilter
$ sudo yum install gcc lua-devel openssl-devel pcre-devel
$ make
$ sudo make install

I've used a lot of email clients over the years (and may have been using email longer than most people reading this). I started out (because that's all there was) with non-graphical clients such as mail, pine, elm, and mutt. Over the years I also tried out many graphical clients, such as Evolution, Kmail, Eudora, Thunderbird, and Claws Mail. However, nothing ever worked quite right, so I eventually ended up back with mutt, and have been happy with it ever since. The one drawback (or strength) of mutt is its single-mindedness. It does email very well, but lets other tools handle the ancillary tasks. One of those tasks is filtering, and that's where imapfilter comes in. I like to view all email that comes in, so mutt generally runs with my INBOX open. I scan through the items, marking them urgent if I need to keep them around, and deleting them if they are obvious trash. As needed, I'll kick off a imapfilter run, which then puts all my read, non-urgent, non-deleted email into the appropriate IMAP folders for me (mutt is even smart enough to realize that the folder was externally changed by imapfilter).

So I tried running imapfilter per usual on my new system and noticed an odd thing: each item in my filter was getting a minimum of 66 'hits', even when there was not even 66 total emails in my inbox! I output the number of matches to each filter I use, so instead of seeing what I was usually did:


Mediawiki emails moved:    1
Backcountry emails moved:  10
Perl QA messages moved:    0
...
Wiki alerts deleted:       0
Bucardo emails moved:      5
Maatkit emails moved:      0
Mail filtering complete

I saw everything at N+66 instead:


Mediawiki emails moved:   67
Backcountry emails moved: 76
Perl QA messages moved:   66
...
Wiki alerts deleted:      66
Bucardo emails moved:     71
Maatkit emails moved:     66
Mail filtering complete

Obviously, something was wonky. Glancing at the release notes showed that version 2.2 changed the format of the search results:

Since version 2.2, a different format is used for the returned structures of the searching methods, due to the introduction of multiple mailbox searching and meta-searching, and thus any configuration files that rely on them should be updated

Okay, but where was the 66 coming from? I created a ~/.imapfilter/test.lua file to show me exactly what was happening inside the loop over the results table. (imapfilter is written in a nice language called Lua, which calls its main data structures "tables". Probably to the chagrin of those using Lua/database crossover tools like Pl/Lua :) The test.lua file looked like this:


myaccount = IMAP {
  server   = 'mail.example.com',
  username = 'greg',
  password = 'secret',
  ssl      = 'tls1'
}
inbox = myaccount['INBOX']
result = inbox:contain_subject('bats')

count = 0
for k,v in pairs(result) do 
  count = count + 1 
  if count < 10 then
    print(count, "Call to pairs:",k,v)
  end
end
print("Total count for pairs: " .. count);

count = 0
for k,v in ipairs(result) do 
  count = count + 1 
  if count < 10 then
    print(count, "Call to ipairs:",k,v)
  end
end
print("Total count for ipairs: " .. count);

I downloaded and compiled version 2.0 of imapfilter and ran the above code, knowing that there were exactly two emails in my inbox that had a subject containing the string 'bats':

[~/code/imapfilter-2.0] ./imapfilter -c ~/.imapfilter/test.lua
  1      Call to pairs:    9         true
  2      Call to pairs:    32        true
Total count for pairs: 2
Total count for ipairs: 0

So it looked like the results table simply contained two entries, with keys of 9 and 32 (which correspond to where those emails happened to appear in my inbox). Calling ipairs yielded zero matches, which makes sense: there is no key of 1 (which is what Lua tables start with by convention, rather than 0 like almost everything else in the computer world :). The ipairs function goes through each key in order starting with 1 until a nil (undefined) key is found. In this case, 1 itself is nil. The output looks much different when I ran it using the new version (2.3) of imapfilter:

[~/code] imapfilter -c ~/.imapfilter/test.lua
  1      Call to pairs:    1              table: 0x82b0bd8
  2      Call to pairs:    2              table: 0x82b0c48
  3      Call to pairs:    _union         function: 0x81d48d0
  4      Call to pairs:    _mt            table: 0x82a32d0
  5      Call to pairs:    mark_answered  function: 0x81cefe0
  6      Call to pairs:    send_query     function: 0x81d8180
  7      Call to pairs:    is_flagged     function: 0x81c1878
  8      Call to pairs:    unmark_deleted function: 0x81bd890
  9      Call to pairs:    match_message  function: 0x81cd7f8
Total count for pairs: 68
  1      Call to ipairs:    1        table: 0x82b0bd8
  2      Call to ipairs:    2        table: 0x82b0c48
Total count for ipairs: 2

This tells us a quite a few things, and solves the mystery of the 66, which represents some meta-data stored in the results table. So rather than treating results as a simple key/value hash with one entry per match, the results table is now a dual-purpose table where the hash part of it contains some meta-data, while the actual matches are stored in the array (indexed) part of the table. Note how the counting of the matches now starts at 1 and increments, rather than using the position in the inbox, as it did before. Which means we must use ipairs to iterate through the table and get our matching entries, in this case with keys 1 and 2.

(If the "table" structure in Lua looks odd to you, that's because it is. I don't think I would have designed things that way myself - while it's clever to have a single structure that behaves as both an array with indices and a btree hash, it can lead to confusion and some ugly corner cases).

The next step was to get my filters working again - this was simply a matter of a global search and replace (M-x query-replace-regexp) from "pairs" to "ipairs".This is a good a point as any to explain what my file looks like (stored as ~/.imapfilter/config.lua). The first part simply sets some common options - for details on what they do, check out the manpage for imapfilter_config.


options.cache        = true
options.certificates = true
options.create       = false
options.info         = false
options.close        = true
options.expunge      = false

Next, a new table is created with the IMAP function. After that, we exclude all messages that are already marked as deleted, that have not yet been read, and have not been flagged. In other words, everything in my inbox I've already seen, but not flagged as urgent or deleted. The '*' in this case is a logical 'AND', and the output is the search result table we saw in the above code.


myaccount = IMAP {
  server   = 'mail.example.com',
  username = 'greg',
  password = 'secret,
  ssl      = 'tls1'
}

baseresult = inbox:is_seen() * inbox:is_unflagged() * inbox:is_undeleted()

Now that we have a search result, we simply start looking for things of interest and handling them. For example, to move messages to an existing IMAP folder:


-- Put Mediawiki messages into their folder
result = baseresult
  * (
    inbox:contain_to('@lists.wikimedia.org')
    + inbox:contain_to('@lists.wikimedia.org')
  )
count = 0 for k,v in ipairs(result) do count = count + 1 end
if count > 0 then
  inbox:move_messages(myaccount['INBOX/mediawiki'], result)
end
print('Mediawiki emails moved:        ' .. count)

Searches can be applied to an existing search result to create a new table. In the code above, a new table named 'result' is created that is based off of our 'baseresult' table, with the condition that only entries matching a specific "To" or "Cc" field are added.The '+' acts as as a logical 'OR'.

Deletion is handled in a similar way:


-- Delete wiki alerts
result = baseresult
  * inbox:contain_from('WikiAdmin ')
  * inbox:contain_subject('has been')
count = 0 for k,v in ipairs(result) do count = count + 1 end
if count > 0 then
  inbox:delete_messages(result)
end
print('Wiki alerts deleted:           ' .. count)

The rest of my config.lua file is more filtering sections, similar to the above. Adding a new filter is as easy as creating a new section similar to the above by editing the ~/.imapfilter/config.lua file. While that's not as automated as it could be, filter adjustment happens so rarely I have never been bothered by that step.

If you are not using imapfilter, you should check it out, even if you are not using mutt; imapfilter is completely independent of your email reading program, and can be run from anywhere, as it doesn't save or read anything locally. I find that imapfilter is very fast, so even when I used mail programs with built-in filters, I still employed imapfilter from time to time for bulk deletes and moves. Plus, it's a great way to dip your toe into Lua if you are not familiar with it (although without using some of its more interesting features, such as coroutines).


Comments

published by noreply@blogger.com (Greg Sabino Mullane) on 2011-01-07 16:55:00 in the "linux" category

The SSH config file has some nice features that help me to keep my sanity among a wide variety of servers spread across many different clients. Nearly all of my Postgres work is done by using SSH to connect to remote client sites, so the ability to connect to the various servers easily and intuitively is important. I'll go over an example of how a ssh config file might progress as you deal with an ever‑expanding client.

Some quick background: the ssh config file is a per‑user configuration file for the SSH program. It typically exists as ~/.ssh/config. It has two main purposes: setting global configuration items (such as ForwardX11 no), and setting things on a host‑by‑host basis. We'll be focusing on the latter.

Inside the ssh config file, you can create Host sections which specify options that apply only to one or more matching hosts. The sections are applied if the host name you type in as the argument to the ssh command matches what is after the word "Host". As we'll see, this also allows for wildcards, which can be very useful.

I'm going to walk through a hypothetical client, Acme Corporation, and show how the ssh config can grow as the client does, until the final example mirrors an actual section of my ssh config section file.

So, you've just got a new Postgres client called Acme Corporation, and they are using Amazon Web Services (AWS) to host their server. We're coming in as the postgres user, and have our public ssh keys already in place inside ~postgres/.ssh/authorized_keys on their server. The hostname is ec2‑456‑55‑123‑45.compute‑1.amazonaws.com. So, generally, we would connect by running:


$ ssh postgres@ec2‑456‑55‑123‑45.compute‑1.amazonaws.com

That's a lot to type each time! We could create a bash alias to handle this, but it's better to use the ssh config file instead. We'll add this to the end of our ssh config:


##
## Client: Acme Corporation
##

Host  acmecorp
User postgres
Hostname  ec2-456-55-123-45.compute-1.amazonaws.com

Now we can simply use 'acmecorp' in place of that ugly string:


$ ssh acmecorp

Notice that we don't need to specify the user anymore: ssh config plugs that in for us. We can still override it if we need to connect as someone else:


$ ssh greg@acmecorp

The next week, Acme Corporation decides that rather than allow anyone to SSH to their servers, they will use iptables or something similar to restrict access to select known hosts. Because different people with different IPs at End Point may need to access Acme, and because we don't want to have Acme have to open a new hole each time we connect from a different place, we will connect from a shared company box. In this case, the box is vp.endpoint.com. Acme arranges to allow SSH from that box to their servers, and each End Point employee has a login on the vp.endpoint.com box. What we need to do now is create a SSH tunnel. Inside of the ssh config file, we add a new line to the entry for 'acmecorp':


Host  acmecorp
User  postgres
Hostname  ec2-456-55-123-45.compute-1.amazonaws.com
ProxyCommand  ssh -q greg@vp.endpoint.com nc -w 180 %h %p

Now, when we run this:


$ ssh acmecorp

...everything looks the same to us, but what we are really doing is connecting to vp.endpoint.com, running the nc (netcat) command, and then connecting to the amazonaws.com box over the new netcat connection. (The arguments to netcat specify that the connection should be closed if there is the connection goes away for 180 seconds, and the host and port should be echoed along). As far as amazonaws.com is concerned, we are connecting from vp.endpoint.com. As far as we are concerned, we are going directly to amazonaws.com. A nice side effect, and a big reason why we don't simply use bash aliases, is that the scp program will use these aliases as well. So we can now do something like this:


$ scp check_postgres.pl acmecorp:

This will copy the check_postgres.pl program from our computer to the Acme one, going through the tunnel at vp.endpoint.com.

Business has been good for Acme lately and they finally have conceded to your strong suggestion to set up a warm standby server (using Postgres' Point In Time Recovery system). This new server is located at ec2‑456‑55‑123‑99.compute‑1.amazonaws.com, and the internal host name they give it is maindb‑replica (the original box is known as maindb‑db). This new server requires another host entry to ssh config. Rather than copy over the same ProxyCommand, we'll refactor the information out into a separate host entry. What we end up with is this:


Host  acmetunnel
User  greg
Hostname  vp.endpoint.com

Host  acmedb
User  postgres
Hostname  ec2-456-55-123-45.compute-1.amazonaws.com
ProxyCommand  ssh -q acmetunnel nc -w 180 %h %p

Host  acmereplica
User  postgres
Hostname  ec2-456-55-123-99.compute-1.amazonaws.com
ProxyCommand  ssh -q acmetunnel nc -w 180 %h %p

We also changed the name from acmecorp to just "acme" as that's enough to uniquely identify among our clients, and who wants to type more than they have to?

Next, the company adds a QA box they want End Point to help setup. This box, however, is *not* reachable from outside their network; it can be reached only from other hosts in their network. Luckily, we already have access to some of those. What we'll do is extend our tunnel by one more host, so that the path we travel from us to the Acme QA box is:

Local box → vp.endpoint.com → acreplica → acqa

Here's the section of the ssh config after we've added in the QA box:


Host  acmetunnel
User  greg
Hostname  vp.endpoint.com

Host  acmedb
User  postgres
Hostname  ec2-456-55-123-45.compute-1.amazonaws.com
ProxyCommand  ssh -q acmetunnel nc -w 180 %h %p

Host  acmereplica
User  postgres
Hostname  ec2-456-55-123-99.compute-1.amazonaws.com
ProxyCommand  ssh -q acmetunnel nc -w 180 %h %p

Host  acmeqa
User  postgres
Hostname  qa
ProxyCommand  ssh -q acreplica nc -w 180 %h %p

Note that we don't need the full hostname at this point for the "acmeqa" Hostname, as we can simply say 'qa' and the acreplica box knows how to get there.

There is still some unwanted repetition in the file, so let's take advantage of the fact that the "Host" item inside the ssh config file will take wildcards as well. It's not really apparent until you use wildcards, but a ssh host can match more than one "Host" section in the ssh config file, and thus you can achieve a form of inheritance. (However, once something has been set, it cannot be changed, so you always want to set the more specific items first). Here's what the file looks like after adding a wildcard section:


Host  acme*
User  postgres
ProxyCommand  ssh -q greg@vp.endpoint.com nc -w 180 %h %p

Host  acmedb
Hostname  ec2-456-55-123-45.compute-1.amazonaws.com

Host  acmereplica
Hostname  ec2-456-55-123-99.compute-1.amazonaws.com

Host  acmeqa
User  root
Hostname  qa
ProxyCommand  ssh -q acreplica nc -w 180 %h %p

Notice that the file is now simplified quite a bit. If we run this command:


$ ssh acmereplica

...then the Host acme* section sets up both the User and the ProxyCommand. It then also matches on the Host acmereplica section and applies the Hostname there.

Note that we have removed the "acmetunnel" section. Now that all the ProxyCommands are in a single place, we can simply go back to the original ProxyCommand and specify the exact user and host.

All of the above presumes we want to login as the postgres user, but there are also times when we need to login as a different user (e.g. 'root'). We can again use wildcards, this time to match the end of the host, to specify which user we want. Anything ending in the letter "r" means we log in as user root, and anything ending in the letter "p" means we log in as user postgres. Our final ssh config section for Acme is now:


##
## Client: Acme Corporation
##

Host  acme*
ProxyCommand  ssh -q greg@vp.endpoint.com nc -w 180 %h %p
Host  acme*r
User  root
Host  acme*p
User  postgres

Host  acmedb*
Hostname  ec2-456-55-123-45.compute-1.amazonaws.com

Host  acmereplica*
Hostname  ec2-456-55-123-99.compute-1.amazonaws.com

Host  acmeqa*
Hostname  qa
ProxyCommand  ssh -q acreplica nc -w 180 %h %p

From this point on, if Acme decides to add a new server, adding it into our ssh config is as simple as adding two lines:


Host  acmedev*
Hostname  ec2-456-55-999-45.compute-1.amazonaws.com

This automatically sets up two hosts for us, "acmedevr" and "acmedevp". What if we leave out the ending "r" or "p" and just ssh to "acmedev"? Then we'll connect as the default user, or $ENV{USER} (in my case, "greg").

Have fun configuring your ssh config file, don't be afraid to leave lots of comments inside of it, and of course keep it in version control!


Comments

published by Thom Holwerda on 2009-04-01 11:52:35 in the "Linux" category
Thom Holwerda

To all the Mono whiners:

THE WORLD DOESN’T CARE.

That is all.


Comments

published by noreply@blogger.com (Ron Phipps) on 2009-03-02 22:52:00 in the "passenger mod_rails selinux PassengerTempDir" category

We recently ran into an issue when launching a client's site using Phusion Passenger where it would not function with SELinux enabled. It ended up being an issue with Apache having the ability to read/write the Passenger sockets. In researching the issue we found another engineer had reported the problem and there was discussion about having the ability to configure where the sockets could be placed. This solution would allow someone to place the sockets in a directory other than /tmp and set the context on the directory so that sockets created within it have the same context and then grant httpd the ability to read/write to sockets with that specific context. This is a win over granting httpd the ability to read/write to all sockets in /tmp since many other services place their sockets there and you may not want httpd to be able to read/write to those sockets.

End Point had planned to take on the task of patching passenger and submitting the patch. While collecting information about the issue this morning to pass to Max I found this in the issue tracker for Passenger:

Comment 4 by honglilai, Feb 21, 2009 Implemented.

Status: Fixed
Labels: Milestone-2.1.0

Excellent! We'll be testing this internally soon and will post a new blog entry with our solution for Passenger + SELinux. Thanks to the Passenger engineers for taking the request seriously and working on an update with the PassengerTempDir configuration directive included.


Comments

published by Thom Holwerda on 2009-02-13 18:54:00 in the "Linux" category
Thom Holwerda

So, my parents bought this brand new multifunction uber-printer that can do anything, including making coffee. Great stuff.

I took their old printer home with me, because I don’t have one and was on the verge of buying one anyway. This is when I entered the twilight zone.

The HP Deskjet 840c has no Vista drivers. HP doesn’t make them. I need my Linux machine to print.

And now the world will end.


Comments

published by Thom Holwerda on 2009-01-28 08:48:39 in the "Linux" category
Thom Holwerda

I find it rather pathetic that the only entry on Planet GNOME so far regarding KDE 4.2 is some guy complaining about the release announcement using Flash for the screencasts. From a project that embraces such a controversial technology like Mono.

Jealous much?


Comments

published by noreply@blogger.com (Jon Jensen) on 2008-09-30 14:03:00 in the "Red Hat Enterprise Linux" category

At OSNews.com the article Windows x64 Watch List describes some of the key differences between 64-bit and 32-bit Windows. It's pretty interesting, and mostly pretty reasonable. But this one caught my eye:

There are now separate system file sections for both 32-bit and 64-bit code

Windows x64's architecture keeps all 32-bit system files in a directory named "C:WINDOWSSysWOW64", and 64-bit system files are place in the the oddly-named "C:WINDOWSsystem32" directory. For most applications, this doesn't matter, as Windows will re-direct all 32-bit files to use "SysWOW64" automatically to avoid conflicts.

However, anyone (like us system admins) who depend on VBScripts to accomplish tasks, may have to directly reference "SysWOW64" files if needed, since re-direction doesn't apply as smoothly.

I've been using 64-bit Linux since 2005 and found there to be some learning curve there, with distributors taking different approaches to supporting 32-bit libraries and applications on a 64-bit operating system.

The Debian Etch approach is to treat the 64-bit architecture as "normal", for lack of a better word, with 64-bit libraries residing in /lib and /usr/lib as always. It's recommended to run a 32-bit chroot with important libraries in the ia32-libs package going into /emul/ia32-linux. Ubuntu is similar, but its ia32-libs puts its ia32-libs files into /usr/lib32.

The Red Hat approach called "multilib" keeps 32-bit libraries in /lib and /usr/lib with new 64-bit libraries living in /lib64 and /usr/lib64. (I mentioned this a while back while discussing building a custom Perl on 64-bit Red Hat OSes.)

Each way has its tradeoffs, and causes a bit of trouble. That's just the cost of dealing with multiple architectures in a single running OS, where no such support was previously needed.

But the Windows way? Putting your 32-bit libraries in C:WINDOWSSysWOW64 and your 64-bit libraries in C:WINDOWSsystem32? It hurts to see the names be exactly backwards. That's really tops for confusion.


Comments

published by noreply@blogger.com (Jon Jensen) on 2008-08-09 22:30:00 in the "Red Hat Enterprise Linux" category

The other day I wanted to see a list of all RPMs that came from a source other than Red Hat, which were installed on a Red Hat Enterprise Linux (RHEL) 5 server. This is straightforward with the rpm --queryformat (short form --qf) option:

rpm -qa --qf '%{NAME} %{VENDOR}n' | grep -v 'Red Hat, Inc.' | sort

That instructs rpm to output each package's name and vendor, then we exclude those from "Red Hat, Inc." (which is the exact string Red Hat conveniently uses in the "vendor" field of all RPMs they pacakge).

By default, rpm -qa uses the format '%{NAME}-%{VERSION}-%{RELEASE}', and it's nice to see version and release, and on 64-bit systems, it's also nice to see the architecture since both 32- and 64-bit packages are often installed. Here's how I did that:

rpm -qa --qf '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR}n' | grep -v 'Red Hat, Inc.' | sort

With that I'll see output such as:

fping-2.4-1.b2.2.el5.rf.x86_64 Dag Apt Repository, http://dag.wieers.com/apt/
git-1.5.6.5-1.x86_64 End Point Corporation
iftop-0.17-1.el5.x86_64 (none)

There we see the fping package from the excellent DAG RPM repository, along with a few others.

To see a list of all symbols that can be used:

rpm --querytags

Comments

published by Thom Holwerda on 2008-07-13 17:24:05 in the "Linux" category
Thom Holwerda

So, like, GNOME was on its way to irrelevance, simply because they had no plan for the future, no direction, no leadership, no vision, no nothing.

Then came GUADEC, and at GUADEC, they came up with a vision for GNOME 3.0. And it’s all about tabs.

This basically means no more GNOME for me. Tabs suck balls in just about every implementation except settings panels. Tabs are bad because they constrain you. Tabbing is all the shizzle in web browsers, but all it does is stop you from having differently sized windows, having websites side by side - and to make it all even worse, tabbed programs introduce a new place to manage windows: the application window itself. So, users have to think about where to switch to a certain window - do we switch using the panel, or via an application? Wait, we have to switch to the application via the panel first, and then switch to the particular window we want inside the application? And what about closing documents versus closing windows? What about having 15 scientific .pdf’s loaded in a tabbed Evince? Can I still read the tabs, or are they shortened to only the first few letters of the filename?

What do you mean, pointless clicks?

We have been trying for ages now to move away from an application-centric world, towards a document-centric world, and Mac OS X is doing really, really well in that regard (Quick Look!), and GNOME itself was not doing bad either. By focussing efforts on tabbed applications, all that work has been in vain. They are setting the clock back, I don’t know, 15 years?

I’m happy that GNOME has a vision, but sadly, it’s one step forward, three billion steps back. I mean, vertical damn tabs? Why don’t you start eating babies while you’re at it?


Comments

published by Thom Holwerda on 2008-06-25 20:50:12 in the "Linux" category
Thom Holwerda

All I want to say here is that this is a pretty darn sad state of affairs.

Us OSNews folk deal with that stuff every day, but we don’t get to shut anything off. At one point, Aaron, you just get used to it. I’m happy you have the option to just turn it all off.

Happy coding!


Comments

published by Thom Holwerda on 2008-06-17 08:46:53 in the "Linux" category
Thom Holwerda

Can someone explain to me what’s so funny about this latest intertubes hype? I mean, the kid makes some very good points regarding the inherent crappiness of Linux, but it’s written like he’s a 12 year old.

Wait. 12 Years old?

I guess I just answered my own question. Of course teh intertubes loves him.


Comments