All opinions expressed are those of the authors and not necessarily those of, our sponsors, or our affiliates.
  Add to My Yahoo!  Subscribe with Bloglines  Subscribe in NewsGator Online

published by (Marco Matarazzo) on 2017-04-07 18:06:00 in the "CentOS" category

During a recent CentOS 7 update, among other packages, we updated our Percona 5.7 installation to version 5.7.17-13.

Quickly after that, we discovered that mysqldump stopped working, thus breaking our local mysql backup script (that complained loudly).

What happened?

The error we received was:

mysqldump: Couldn't execute 'SELECT COUNT(*) FROM INFORMATION_SCHEMA.SESSION_VARIABLES WHERE VARIABLE_NAME LIKE 'rocksdb_skip_fill_cache'': The 'INFORMATION_SCHEMA.SESSION_VARIABLES' feature is disabled; see the documentation for 'show_compatibility_56' (3167)

After a bit of investigation, we discovered this was caused by this regression bug, apparently already fixed but not yet available on CentOS:

Everything revolves around INFORMATION_SCHEMA being deprecated in version 5.7.6, when Performance Schema tables has been added as a replacement.

Basically, a regression caused mysqldump to try and use deprecated INFORMATION_SCHEMA tables instead of the new Performance Schema.

How to fix it?

Immediate workaround is to add this line to /etc/my.cnf or (more likely) /etc/percona-server.conf.d/mysqld.cnf, depending on how your configuration files are organized:


This flag was both introduced and deprecated in 5.7.6. It will be there for some time to help with the transition.

It seems safe and, probably, good to keep if you have anything still actively using INFORMATION_SCHEMA tables, that would obviously be broken if not updated to the new Performance Schema since 5.7.6.

With this flag, it is possible to preserve the old behavior and keep your old code in a working state, while you upgrade it. Also, according to the documentation, it should not impact or turn off the new behavior with Performance Schema.

More information on how to migrate to the new Performance Schema can be found here.


published by (Marco Matarazzo) on 2017-04-04 13:45:00 in the "CentOS" category

In End Point, we use different hosting providers based on the specific task needs. One provider we use extensively with good results is Linode.

During a routine CentOS 7 system update, we noticed a very strange behavior where our IPv6 assigned server address was wrong after restarting the server.

IPv6 on Linode and SLAAC

Linode is offering IPv6 on all their VPS, and IPv6 dynamic addresses are assigned to servers using SLAAC.

In the provided CentOS 7 server image, this is managed by NetworkManager by default. After some troubleshooting, we noticed that during the update the NetworkManager package was upgraded from 1.0.6 to 1.4.0.

This was a major update, and it turned out that the problem was a change in the configuration defaults between the two version.

Privacy stable addressing

Since 1.2, NetworkManager added the Stable Privacy Addressing feature. This allows for some form of tracking prevention, with the IPv6 address to be stable on a network but changing when entering another network, and still remain unique.

This new interesting feature has apparently become the default after the update, with the ipv6.addr-gen-mode property set to "stable-privacy". Setting it to ?eui64? maintains the old default behavior.

Privacy Extension

Another feature apparently also caused some problems on our VPS: the Privacy Extension. This is a simple mechanism that somewhat randomizes the network hardware?s (MAC) address, to add another layer of privacy. Alas, this is used in address generation, and that randomization seemed to be part of the problem we were seeing.

This too has become the default, with the ipv6.ip6-privacy property set to 1. Setting it to 0 turns off the feature.

To sum it up

In the end, after the update, we could restore the old behavior and resolve our issues by running, in a root shell:

nmcli connection modify "Wired connection 1" ipv6.ip6-privacy 0
nmcli connection modify "Wired connection 1" ipv6.addr-gen-mode eui64

After a reboot, the IPv6 address finally matched the one actually assigned by Linode, and everything was working ok again.

If you want to know more on Privacy Extensions and Privacy Stable Addressing, this great blog post by Lubomir Rintel helped us a lot understanding what was going on.


published by (Marco Matarazzo) on 2015-12-29 12:23:00 in the "CentOS" category

We recently had to move a git repository from an old CentOS 5 to a new CentOS 7 server.

On the old CentOS 5 we had a recent, custom compiled version of git while on the new server we are using the system default old 1.8 version, shipped by the official CentOS repositories. And, as usual when you tell yourself "What could possibly go wrong?", something did: every push began to return the dreaded "fatal: The remote end hung up unexpectedly" error.

After some time spent trying to debug the problem, we managed to isolate the problem to the pre-receive hook, specifically active on that repository. The script was very simple:

 read_only_users="alice bob"
 for user in $read_only_users
     if [ $USER == $user ]; then
         echo "User $USER has read-only access, push blocked."
         exit 1

... which apparently had no visible mistakes. On top of the lack of errors, this very same script used to work perfectly for years on the old server. Unfortunately, and quite disappointingly, even changing it to a simple:

 echo "These are not the droids you are looking for. Move along."

...did not help and the error still persisted.

Searching for clues around forums and wikis, we found this blog post talking about parameters passed through stdin.

On Git docs, we read that pre-receive hooks takes no arguments, but for each ref to be updated it receives on standard input a line of the format: <old-value> SP <new-value> SP <ref-name> LF.

At that point, we tried with a sample script that actually reads and does something with stdin:

 while read oldrev newrev refname
   echo "OLDREV: $oldrev - NEWREV: $newrev - REFNAME: $refname"

...and voilĂ : pushes started working again. Lesson learned: never ignore stdin.


published by (Emanuele 'Lele' Calo') on 2015-01-30 00:55:00 in the "CentOS" category

If you happen to have reached this page because you're trying to shrink an XFS filesystem let's put things straight: sorry, that's not possible.

But before you go away you should know there's still hope and I'd like to show you how I used a little workaround to avoid reinstalling a Red Hat Enterprise Linux 7 or CentOS 7 VM using XFS dump/restore on-the-fly and LVM capabilities, both standard choices for the regular RHEL/CentOS 7 setup.

First of all let's clarify the situation I found myself in. For various reasons I had a CentOS 7 VM with everything already configured and working, installed not many days ago to test new software we're evaluating.

The VM itself is hosted on a dedicated server we manage on our own, so I had a certain degree of freedom to what I could do without the need of paying any additional fee. You may not be in this same situation, but you can probably try some similar solution for little money if you're using an "hourly-billed" VPS provider.

The problem was that, even if everything was working and configured, the virtual hard disk device attached to the machine was too big and on the wrong storage area of the virtualization hosting server.

There was also another minor glitch: The VM itself was using an old virtualization device driver (IDE-based) instead of the new VIRTIO one. Since I knew that the virtualized OS CentOS 7 is capable of using VIRTIO based devices I also took the chance to fix this.

Unfortunately, XFS is not capable of being shrunk at the moment (and for the foreseeable future) so what I needed to do was to:

  1. add a new virtual device correctly using VIRTIO and stored in the right storage area of the virtualization host server
  2. migrate all the filesystems to the new virtual device
  3. set the VM OS to be working from the newly-built partition
  4. dismiss the old virtual device

In my specific case this translated to connect to the virtualization hosting server, create a new LVM logical volume to host the virtual disk device for the VM and then add the new virtual device to the VM configuration. Unfortunately in order to have the VM see the new virtual device I had to shut it down.

While connected to the virtualization host server I also downloaded and attached to the VM the latest ISO of SysRescueCD which is a data rescue specialized Linux distribution. I'm specifically using this distro since it's one of the few which offers the XFS dump/restore tools on the live ISO.

Now the VM was ready to be booted with the SysRescueCD Live OS and then I could start working my way through all the needed fixes. If you're doing something similar, of course please make sure you have offsite backups and have double-checked that they're readable before doing anything else.

First of all inspect your dmesg output and find what is the source virtual device and what's the new target virtual device. In my case the source was /dev/sda and the target was /dev/vda

dmesg | less

Then create a partition on the new device (eg: /dev/vda1) as Linux type for the /boot partition; this should be of the same size as the source /boot partition (eg: /dev/sda1) and dedicate all the remaining space to a new LVM type partition (eg: /dev/vda2)

fdisk /dev/vda
# [create /boot and LVM partitions]

You could also mount and copy the /boot files or re-create them entirely if you need to change the /boot partition size. Since I kept /boot exactly the same so I could use ddrescue (a more verbose version of classic Unix dd).

ddrescue /dev/sda1 /dev/vda1

The next step is supposed to migrate the MBR and should be working but in my case the boot phase kept failing so I also needed to reinstall the bootloader via the CentOS 7 rescue system (not covered in this tutorial but briefly mentioned near the end of the article).

ddrescue -i0 -s512 /dev/sda /dev/vda

Then create the target LVM volumes.

pvcreate /dev/vda2
vgcreate fixed_centos_VG /dev/vda2
lvcreate -L 1G -n swap fixed_centos_VG
lvcreate -l 100%FREE -n root fixed_centos_VG
vgchange -a y fixed_centos_VG

Create the target XFS filesystem.

mkfs.xfs -L root /dev/fixed_centos_VG/root

And then create the swap partition.

mkfs.swap /dev/fixed_centos_VG/swap

Next create the needed mountpoints and mount the old source and the new empty filesystems.

mkdir /mnt/disk_{wrong,fixed}
mount /dev/fixed_centos_VG/root /mnt/disk_fixed
vgchange -a y wrong_centos_VG
mount /dev/centos_VG/root /mnt/disk_wrong

Now here's the real XFS magic. We'll use xfsdump and xfsrestore to copy the filesystem content (files, directory, special files) without having to care about files permission, type, extended ACLs or anything else. Plus since it's only moving the content of the filesystem we won't need to have a partition of the same size and it won't take as long as copying the entire block device as the process will just have to go through real used space.

xfs_dump -J - /mnt/disk_wrong | xfs_restore -J - /mnt/disk_fixed

If you want a more verbose output, leave out the -J option. After the process is done, be sure to carefully verify that everything is in place in the new partition.

ls -lhtra /mnt/disk_fixed/

Then unmount the disks and deactivate the LVM VGs.

umount /mnt/disk_{old,fixed}
vgchange -a n centos_VG
vgchange -a n fixed_centos_VG

At this point in order to avoid changing anything inside the virtualized OS (fstab, grub and so on), let's remove the old VG and rename the newer one with the same name the old one had.

vgremove centos_VG
pvremove /dev/sda2
vgrename {fixed_,}centos_VG

You should be now able to shutdown the VM again, detach the old disk and start the new VM which will be using the new smaller virtual device.

If the boot phase keeps failing, boot the CentOS installation media in rescue mode and after chroot-ing inside your installation run grub-install /dev/vda (targeting your new main device) to reinstall grub.

Only after everything is working as expected, proceed to remove the old unneeded device and remove it from the virtualization host server.


published by (Spencer Christensen) on 2015-01-22 22:35:00 in the "CentOS" category

We use a variety of hosting providers for ourselves and our clients, including Hetzner. They provide good servers for a great price, have decent support, and we've been happy with them for our needs.

Recently I was given the task of building out a new development server for one of our clients, and we wanted it to be set up identically to another one of their servers but with CentOS 7. I placed the order for the hardware with Hetzner and then began the procedure for installing the OS.

Hetzner provides a scripted install process that you can kick off after booting the machine into rescue mode. I followed this process and selected CentOS 7 and proceeded through the whole process without a problem. After rebooting the server and logging in to verify everything, I noticed that the disk space was capped at 2 TB, even though the machine had two 3 TB drives in it (in hardware RAID 1). I looked at the partitions and found the partition table was "msdos". Ah ha!

At this point painful memories of running into this problem before hit me. I reviewed our notes of what we had done last time, and felt like it was worth a shot even though this time I'm dealing with CentOS 7. I went through the steps up to patching anaconda and then found that anaconda for CentOS 7 is newer and the files are different. I couldn't find any files that care about the partition table type, so I didn't patch anything.

I then tried to run the CentOS 7 install as-is. This only got me so far because I then ran into trouble with NetworkManager timing out and not starting.

screen shot of CentOS 7 installer failing
A screenshot of the CentOS 7 installer failing (anaconda) similar to what I was seeing.

Baffled, I looked into what may have been causing the trouble and discovered that the network was not set up at all and it looked as if no network interfaces existed. WHAT?? At this point I dug through dmesg and found that the network interfaces did indeed exist but udevd had renamed them. Ugh!

Many new Linux distributions are naming network interfaces based on their physical connection to the system: those embedded on the motherboard get named em1, em2, etc. Apparently I missed the memo on this one, as I was still expecting eth0, eth1, etc. And from all indications, so was NetworkManager because it could not find the network interfaces!

Rather than spend more time going down this route, I decided to change gears and look to see if there was any way to patch the Hetzner install scripts to use a GPT partition table with my install instead of msdos. I found and read through the source code for their scripts and soon stumbled on something that just might solve my problem. In the file /root/.oldroot/nfs/install/ I found mention of a config variable FORCE_GPT. If this is set to "1" then it will try to use a GPT partition table unless it thinks the OS won't like it, and it thinks that CentOS won't like it (no matter the version). But if you set FORCE_GPT to "2" it will use a GPT partition table no matter what. This config setting just needs to be added to the file you edit where you list out your partitions and LVM volumes.

FORCE_GPT 2                                                                                                      

PART /boot ext3 512M                                                                                             
PART lvm   vg0  all                                                                                              
LV  vg0  swap swap   swap  32G                                                                                   
LV  vg0  root  /     ext4 100G                                                                                   
LV  vg0  home  /home ext4 400G                                                                                   

I then ran the installer script and added the secret config option and... Bingo! It worked perfectly! No need to manually patch anything or install manually. And now we have a CentOS 7 server with full 3 TB of disk space usable.

(parted) print                                                            
Model: DELL PERC H710 (scsi)
Disk /dev/sda: 3000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: pmbr_boot

Number  Start   End     Size    File system  Name  Flags
 3      1049kB  2097kB  1049kB                     bios_grub
 1      2097kB  539MB   537MB   ext3
 2      539MB   3000GB  2999GB                     lvm


published by (Spencer Christensen) on 2014-11-07 18:18:00 in the "CentOS" category

When installing PostgreSQL 9.3 onto a CentOS 6 system, you may notice that some postgres commands appear to be missing (like pg_ctl, initdb, and pg_config). However, they actually are on your system but just not in your path. You should be able to find them in /usr/pgsql-9.3/bin/. This can be frustrating if you don't know that they are there.

To solve the problem, you could just use full paths to these commands, like /usr/pgsql-9.3/bin/initdb, but that may get ugly quick depending on how you are calling these commands. Instead, we can add them to the path.

You could just copy them to /usr/bin/ or create symlinks for them, but both of these methods are hack-ish and could have unintended consequences. Another option is to add /usr/pgsql-9.3/bin/ to your path manually, or to the path for all users by adding it to /etc/bashrc. But again, that seems hack-ish and when you upgrade postgres to 9.4 down the road things will break again.

So instead, let's look at how Postgres' other commands get installed when you install the rpm. When you run "yum install postgresql93" the rpm contains not only all the files for that package but it also includes some scripts to run (in this case one at install time and another at uninstall time). To view everything that is getting installed and to see the scripts that are run use this command: "rpm -qilv --scripts postgresql93". There will be a lot of output, but in there you will see:

postinstall scriptlet (using /bin/sh):
/usr/sbin/update-alternatives --install /usr/bin/psql pgsql-psql /usr/pgsql-9.3/bin/psql 930
/usr/sbin/update-alternatives --install /usr/bin/clusterdb  pgsql-clusterdb  /usr/pgsql-9.3/bin/clusterdb 930
/usr/sbin/update-alternatives --install /usr/bin/createdb   pgsql-createdb   /usr/pgsql-9.3/bin/createdb 930

That line "postinstall scriptlet (using /bin/sh):" marks the beginning of the list of commands that are run at install time. Ah ha! It runs update-alternatives! If you're not familiar with alternatives, the short description is that it keeps track of different versions of things installed on your system and automatically manages symlinks to the version you want to run.

Now, not all the commands we're interested in are installed by the package "postgresql93"- you can search through the output and see that pg_config gets installed but is not set up in alternatives. The commands initdb and pg_ctl are part of the package "postgresql93-server". If we run the same command to view its files and scripts we'll see something interesting- it doesn't set up any of its commands using alternatives! Grrr. :-(

In the postgresql93-server package the preinstall and postinstall scripts only set up the postgres user on the system, set up /var/log/pgsql, add postgres to the init scripts, and set up the postgres user's .bash_profile. That's it. But, now that we know what commands are run for getting psql, clusterdb, and createdb into the path, we can manually run the same commands for the postgres commands that we need. Like this:

/usr/sbin/update-alternatives --install /usr/bin/initdb pgsql-initdb /usr/pgsql-9.3/bin/initdb 930
/usr/sbin/update-alternatives --install /usr/bin/pg_ctl pgsql-pg_ctl /usr/pgsql-9.3/bin/pg_ctl 930
/usr/sbin/update-alternatives --install /usr/bin/pg_config pgsql-pg_config /usr/pgsql-9.3/bin/pg_config 930

These commands should be available in your path now and are set up the same as all your other postgres commands. Now, the question about why these commands are not added to alternatives like all the others is a good one. I don't know. If you have an idea, please leave it in the comments. But at least now you have a decent work-around.


published by (Jon Jensen) on 2014-01-13 21:40:00 in the "CentOS" category

I've had interesting tech news items piling up lately and it's time to mention some of those that relate to our work at End Point. In no particular order:

  • FreeOTP is a relatively new open source 2-factor auth app for Android by Red Hat. It can be used instead of Google Authenticator which last year became proprietary as noted in this Reddit thread. The Google Authenticator open source project now states: "This open source project allows you to download the code that powered version 2.21 of the application. Subsequent versions contain Google-specific workflows that are not part of the project." Whatever the reason for that change was, it seems unnecessary to go along with it, and to sweeten the deal, FreeOTP is quite a bit smaller. It's been working well for me over the past month or more.
  • Perl 5.18.2 was released.
  • Ruby 2.1.0 was released.
  • Ruby 1.9.3 end of life set for February 2015, and the formerly end-of-life Ruby 1.8.7 & 1.9.2 have had their maintenance extended for security updates until June 2014 (thanks, Heroku!).
  • Red Hat and CentOS have joined forces, in an unexpected but exciting move. Several CentOS board members will be working for Red Hat, but not in the Red Hat Enterprise Linux part of the business. Red Hat's press release and the CentOS announcement give more details.
  • Red Hat Enterprise Linux 7 is available in beta release. The closer working with CentOS makes me even more eagerly anticipate the RHEL 7 final release.
  • Docker has been getting a lot of well-deserved attention. Back in September, Red Hat announced it would help modify Docker to work on RHEL. To date only very recent Ubuntu versions have been supported by Docker because it relies on AUFS, a kernel patch never expected to be accepted in the mainline Linux kernel, and deprecated by the Debian project. Now Linode announced their latest kernels support Docker, making it easier to experiment with various Linux distros.
  • New maintenance releases of PostgreSQL, PHP, Python 2.7, and Python 3.3 are also out. Not to take all the small steps for granted!
  • Finally, for anyone involved in web development or system/network administration I can recommend a nice reference project called Know Your HTTP * Well. I've looked most closely at the headers section. It helpfully groups headers, has summaries, and links to relevant RFC sections.

And we're already two weeks into January!


published by (Emanuele 'Lele' Calo') on 2013-11-26 23:16:00 in the "CentOS" category

You may think that you already know what's the opposite of "DISABLED", but with MySQL Event Scheduler you'll be wrong.

In fact MySQL Event Scheduler may have three different states[1][2]:

DISABLED -  The Event Scheduler thread does not run [1]. In addition, the Event Scheduler state cannot be changed at runtime.
OFF (default) - The Event Scheduler thread does not run [1]. When the Event Scheduler is OFF it can be started by setting the value of event_scheduler to ON.
ON - The Event Scheduler is started; the event scheduler thread runs and executes all scheduled events.

So if you're going to find it in the DISABLED state and instinctively set it to ENABLED you'll end up with a non-starting MySQL daemon.
Be warned and stay safe out there!

[2]: When the Event Scheduler is not running does not appear in the output of SHOW PROCESSLIST


published by (Jon Jensen) on 2013-11-20 16:01:00 in the "CentOS" category

I was just reading my co-worker Lele's blog post about making SELinux dontaudit AVC denial messages visible and realized it was likely the solution to a mystery I ran into a few days ago.

As Lele explains, the SELinux dontaudit flag suppresses certain very common SELinux AVC denials to keep the audit logs from bloating beyond belief and being too hard to use. But sometimes a commonly harmless denial can be the cause of further errors. You can tell this is the case if temporarily disabling SELinux enforcing (setenforce 0) makes the problem go away, but /var/log/audit/audit.log still doesn't show any AVC denial actions being allowed through.

In my somewhat unusual case there is an Apache CGI shell script that calls sudo to invoke another program as a different user without using setuid or suEXEC. Everything works fine with SELinux enforcing, but there are some strange errors in the logs. In /var/log/secure:

sudo: PAM audit_log_acct_message() failed: Permission denied

And in the Apache error_log is the apparently strangely unbuffered output:

[error] sudo
[error] : 
[error] unable to send audit message
[error] : 
[error] Permission denied

To show the dontaudit AVC denials, I ran semodule -DB as Lele explained, and then I saw in /var/log/audit/audit.log:

type=AVC msg=audit(1384959223.974:4192): avc:  denied  { write } for  pid=14836 comm="sudo" scontext=system_u:system_r:httpd_sys_script_t:s0 tcontext=system_u:system_r:httpd_sys_script_t:s0 tclass=netlink_audit_socket
type=AVC msg=audit(1384959223.975:4194): avc:  denied  { read } for  pid=14836 comm="sudo" scontext=system_u:system_r:httpd_sys_script_t:s0 tcontext=system_u:system_r:httpd_sys_script_t:s0 tclass=netlink_audit_socket
type=AVC msg=audit(1384959223.999:4196): avc:  denied  { write } for  pid=14836 comm="sudo" scontext=system_u:system_r:httpd_sys_script_t:s0 tcontext=system_u:system_r:httpd_sys_script_t:s0 tclass=key

Somewhat as expected from the error messages we saw before, sudo is being denied permission to send a message into the kernel. Now that we have the AVC errors from the audit log it's easy to make a local SELinux policy module to allow this.

The spurious error messages go away, and we can run semodule -B to re-suppress the dontaudit messages.

Thanks, Lele. :)


published by (Cas Rusnov) on 2013-11-06 12:00:00 in the "CentOS" category

In collaboration with Spencer Christensen and Lele Calo.

The everyday problem: Set up a remotely-hosted machine with a 3 TB drive.
The bigger problem: It must be CentOS 5.

While this would be a trivial task with a newer OS, CentOS 5 only supports MBR style partitioning, which itself only supports drives less than 2 TB in size; well let us be clear, the installer and GRUB shipped with the installation disk only support MBR normally, the kernel supports the GPT format. GPT is a newer partition format that was introduced by EFI standard, which can support booting from large devices. From various documents and postings on the internet it seemed possible to still use MBR with more than 2TB, but in practice this turned out to be completely unsuccessful. So we moved on with a plan to use GPT.

Since the CentOS 5 installer cannot work with GPT partition tables, we needed to use something else to create the partitions we wanted. We did this by using a rescue CD, like SystemRescue CD from here

  • Boot into the rescue CD
  • Use gdisk to first delete the old partition table to make sure you start cleanly.
    • gdisk, then x (for extended commands), then z to delete.
  • Then go through the process to create the partitions as desired. We wanted:
    • /boot 500M
    • 1.5TB LVM physical disk
    • remaining space LVM physical disk
  • Save the partition table and quit gdisk
  • Create LVM group and volumes as desired. Here's what we did:
    • pvcreate /dev/sda2, then pvcreate /dev/sda3
    • vgcreate vg0 /dev/sda2 /dev/sda3
    • lvcreate -L 32G -n swap vg0
    • lvcreate -L 100G -n root vg0
  • Then make the file systems for those volumes.
    • mkfs.ext3 /dev/sda1 (the /boot partition)
    • mkswap /dev/vg0/swap
    • mkfs.ext4 /dev/vg0/root

Once the partitioning is set up as required, we boot the CentOS 5 rescue CD for the installation process. The installation disc also incidentally contains a rescue mode which can be used by typing linux rescue at the boot prompt. Follow the instructions to get to the rescue prompt. If the network install CD image is used, follow the menus until a choice as to how to load the rescue image is given, then select the appropriate method. We used the HTTP method, and specified as the server name, and /centos/5.10/os/x86_64 as the path (use your favorite mirror as needed); this step involves loading the rescue image which may take some time, at the end of which you will be prompted to find your existing OS, since there is none, select Skip and this will result in being dropped to the rescue prompt. Once at the rescue prompt, we can proceed to the OS installation step.

The first installation step at this stage is to modify anaconda so that it doesn't produce an unskippable error due to there being an "unsupported" GPT. First create a ramdisk to contain a copy of the anaconda libraries:

mkdir /mnt/anacondalib
mount -t tmpfs none /mnt/anacondalib
cp -R /mnt/runtime/usr/lib/anaconda/* /mnt/anacondalib/

Now edit the python file at /mnt/anacondalib/, and on line 1082 (vi and nano are present in the rescue image for editing), change the word "errors" to the word "warnings" - this little change allows anaconda to install despite the fact that we've setup the partitions using GPT for the /boot partition, which is what will normally cause the install to fail.

Now we mount the changed library directory over the read-only version from the installation media:

mount -o bind /mnt/anacondalib/ /mnt/runtime/usr/lib/anaconda/

Now we have to move /sbin out of way otherwise anaconda will fail complaining that /sbin already exists:

export PATH=$PATH:/sbin.bak
mv /sbin /sbin.bak
mkdir /sbin

Now we can start anaconda:

anaconda --dmraid --selinux -T -m $centos_mirror

You may of course replace $centos_mirror with your preferred mirror.

You may then walk through the Anaconda installation menus, proceeding until you get to the "Partitioning Type" step at which point the Create custom layout should be selected. This will take you to a Partitioning screen showing the partition scheme created during the GPT partition creation steps above. After setting your large main logical volume to mount as / (root) and your boot partition to mount as /boot, you should visually confirm the layout and proceed. After accepting the warning about using unsupported GPT partitioning, you will be prompted for several screens about grub options, all of which should be correct so may be accepted at their defaults. After this, the installation should be able to proceed as normal.

Once the OS installation is complete, you will be prompted to eject any media and reboot the machine. You can go ahead and try (we did), but you should run into an error similar to "No bootable media found." and the system is unable to boot. This is because the version of grub that is installed doesn't know how to deal with GPT partition tables. So the next step is to install a newer version of grub. We found some instructions at the bottom of this page: We didn't follow those exactly, so here is what we did:

  • Download SystemRescue CD from here:
  • Boot the SystemRescue CD


mkdir /mnt/boot
mount /dev/sda1 /mnt/boot   # mount your /boot partition to /mnt/boot
cp /lib/grub/x86_64/* /mnt/boot/grub/
umount /mnt/boot
grub  # no arguments, entered grub shell
root (hd0,0)
setup (hd0,0)
^D # exit grub shell

Now reboot the machine (without the SystemRescue CD)

At this point the machine successfully booted for us. Yay! Problem solved.