HOWTO Replace RedHat with Gentoo, Remotely over SSH

Author: Jordan Ritter <jpr5@darkridge.com>

Introduction

What is this?

This is a HOWTO that describes how to take a stock RedHat9 system and convert it to Gentoo, remotely over ssh and while it is running. These instructions have been tested with RedHat9, but very likely apply to everything from RedHat9 down to RedHat 7.2, and include everything from advice and gotchas around how to get another distribution installed over RedHat while it is running, to strategies for cleaning up the system of unused files once the transformation is complete.

Why do that?

While everyone has their own tastes in Linux distributions and installed software, there are frequently times when one doesn't have a real choice, and only the most popular distributions prevail. For example, remote ISPs that offer leasable colocated systems will provide only a short list of the most common Linux distros for pre-installation, and while it's almost a sure thing that RedHat is on that list, there is no guarantee that your favorite one is.

I had a remotely colocated server installed with RedHat9 that I had no physical access to. I love Gentoo, though, and so through much trial and tribulation (and about 80 vmware checkpoints) I figured out how to fulfull the Gentoo dream without needing to access the physical console.

Does it really work?

Yes, it really works! It probably seems a bit scary, but I have tried this process now a dozen times and the only problems I've ever had were the result of typo mistakes that can easily be avoided with more careful attention. No more RedHat9 on my systems -- all are 100% pure Gentoo now, baby!

Wait... I have questions about Gentoo..

This HOWTO assumes a basic knowledge of Gentoo. Some time is taken to explain the finer points and possible pitfalls and gotchas, but to successfully install Gentoo Linux really requires ready access to the Gentoo Installation Documentation, in addition to this document. Print it out in advance or have it handy before doing this.

OK, I'm ready. I can sue if this doesn't work, right?

Nope.

THESE INSTRUCTIONS ARE PROVIDED AS-IS AND WITHOUT WARRANTY. ALL USERS ACCEPT FULL RESPONSIBILITY AND LIABILITY IN THE EVENT OF SYSTEM FAILURE, DATA LOSS, OR ANY OTHER FAILURE OR LOSS RESULTING FROM THESE INSTRUCTIONS.

Phew, ok! Here we go!

The General Strategy

In summary, the transformation process is actually fairly simple. Since Gentoo, once unpacked, will run its own binaries from its own libraries, all we're really doing is unpacking Gentoo on top of RedHat without wiping out our remote access mechanism (sshd), then removing the old RedHat files.

The entire process can be distilled down into the following simple steps:

  1. Preparation -- Make copies of important files and configuration items, Download Gentoo tarballs, Get sshd running without PAM, and Tag old files for later removal;
  2. Installation -- Unpack Gentoo tarballs, Re-add important files and configuration items, Remove the old RedHat libc and related files, Synchronize Portage, and proceed with a normal Gentoo install;
  3. Clean-up -- Remove old RedHat files, and Secure the system.
See? Easy!

Step 1: Preparation

There are several things that need to be prepared before actually starting the transformation process. Create a directory on a partition you have plenty of space on and don't intend to touch (like ``/tmp/gentoo''), and use that to store temporary files and such, as explained below.

Again, please have handy in some form the Gentoo Installation Documentation, as this HOWTO won't take the time to explain the various Gentoo-specific concepts in detail.

All instructions assume the user is logged in as ``root''.

  1. Compile a kernel with relevant driver support compiled-in.

    Just to be on the safe side, you should compile a fully-static kernel. In other words, we want a kernel that will run the system that has no externel module dependencies. When we start removing libraries and old files and such, it is quite possible we might accidentially rm something that those exceedingly tedious RedHat kernels rely on. Just compile your own and make sure your system boots and runs from it before you even start this process. I suggest sticking with 2.4.x for now so as not to complicate things.

    NOTE: I suggest not enabling devfs support just yet. Gentoo can still boot without it, and it's better to have a kernel that can boot both systems RedHat + Gentoo than one that may only be able to boot under Gentoo (dunno how well RedHat supports devfs out-of-the-box).

    Also, just to be safe, copy the kernel (again, after you've tested it) to the safe directory where you're keeping copies of things.

  2. Turn off PAM in OpenSSH's sshd_config file.

    Just like with the kernel, we have to be careful about the differences between the PAM setup in RedHat and the PAM setup in Gentoo. You don't need PAM in order for the thing to function, so regardless of whether you like it or not just disable it so as to ensure you'll be able to login again.

    Best way to do this is:

    1. Modify ``/etc/ssh/sshd_config'', set ``UsePAM'' to ``no'', save the file.
    2. Run ``netstat -tanp | grep LISTEN | grep sshd'', and note the sshd PID.
    3. Kill sshd with ``kill PID'', where you got the PID from the previous step. This will only kill the listening sshd, and will not affect any already-connected sessions.
    4. Without logging out, run ``/usr/sbin/sshd''.
    5. Separate from your current logged-in session, try to re-ssh back into the box.

    If you get back in, you're fine on this step.

    Most folks will have no problem with this. If you do experience problems, consult ``/var/log/messages'', and in the worst-case scenario run sshd in the foreground with debugging turned on. That should give you enough detail about what's going on to figure out how to get things running fine.

  3. Download the latest snapshots/stage3 tarball.

    The only thing you need is the latest stage3 tarball. Make sure to decide ahead of time what platform architecture you are using (x86, pentium3, pentium4, etc), as that's an obvious benefit of Gentoo that you'll want to take advantage of. For the purposes of this HOWTO, we assume the safe ``x86'' architecture.

    You can find a full list of Gentoo mirrors at http://www.gentoo.org/main/en/mirrors.xml. As of this writing, the most recent stage3 tarball is: 

       ftp://distro.ibiblio.org/pub/linux/distributions/gentoo/releases/x86/2004.2/stages/x86/stage3-x86-2004.2.tar.bz2

  4. Make copies of important files and information.

    Copy the following files to your safe location:

    Additionally, take a quick snapshot of what modules are loaded, for later inspection, with ``/sbin/lsmod > /tmp/gentoo/modules.txt''.

  5. Mark all current files with a special date.

    Assuming the author isn't on crack, how would one know what files to delete after the Gentoo installation overlays the RedHat one? Easy! Set the timestamp on all files to a special date before proceeding. I used my birthday!

    First of all, we need to be careful about touching all files, because the kernel will detect accesses to ``/dev'' and try to autoload modules for things that you may not have present or installed, which will make things take a lot longer. So first we temporarily disable modprobe by moving it out of the way, and then move it back when we're done. Also, some filenames and directories have spaces in them, so we need to be careful of that also. 

            mv /sbin/modprobe /sbin/.modprobe
        find / | perl -wpe 's/ /\\ /g' | xargs touch -d "Feb 1 1978" -m -a
        mv /sbin/.modprobe /sbin/modprobe

Step 2: Installation

OK! So now it's time to take the dive. Yes, it's a little scary, but follow these instructions and you should be fine.

First, the most effective way to conduct this installation is with two (2) shells, both at or su'd to the root user. We'll call the first shell the "setup" shell and the second shell the "repair" shell. Let's assume that you used "/tmp/gentoo" as the safe directory to store copies of downloaded files and important files and configuration items.

  1. From the "setup" shell: Unpack the stage3 tarball onto the root partition. 

    cd / && tar jxvf /tmp/gentoo/stage3-*.tar.bz2

  2. Synchronize Portage. 
    emerge sync

  3. While the portage synchronization is happening, continue on by completing the following operations from the "repair" shell:

    IMPORTANT: At this point, test that you can ssh back into the box. You may have to restart the listening sshd server, but that's unlikely.

    By this point, you will be completely off of RedHat's system binaries and libraries and on Gentoo's. Now we must install some basic system components before we reboot and clean up.

  4. (Re-)Add basic system components:

    Install any other utilities you would like to have before rebooting. At the very least, you should (re-)install the minimum of things you don't want deleted:

And that's it! Now you're ready to reboot and come back up on Gentoo. At this point the best thing to do is go over your configuration and make sure you have the important things covered before you reboot:
  1. That the bootloader is properly configured to point to your preferred, working kernel.
  2. That your kernel supports the necessary drivers to properly boot the network interface.
  3. That the Gentoo network configuration looks correct and doesn't have any typos.
  4. That your fstab is correct and the right file systems will be mounted at boot time.
  5. That you can login to the system remotely, and that you can su to root.
  6. That the remote access mechanism, openssh, is configured properly to load at boot time. This can be via ``/etc/conf.d/local.start'' or the ``rc-update'' mechanism.

When you're ready, reboot your system: ``shutdown -r now''.

Step 3: Clean-up

The remaining tasks revolve around cleaning up the old RedHat files that Gentoo is not going to use (to get that disk space back), and a few OS security hardening items that can't hurt.

If you're like me, you install the minimum amount of software components necessary to run the system. If however you like installing lots of other things, then keep in mind that you need to consider the spirit of the following instructions rather than the literal directions, as you may have other components you'll want to isolate or protect.

  1. Protect certain files and directories.

    Before we install any more software, we need to take advantage of the timestamp magic from the Preparation Step and nuke all the old files. But before we do that, there are some directories and files we want to make sure are protected no matter what. The following is a command line that touches the timestamps of various files, and it should be good enough if you haven't done anything beyond the scope of these instructions. 

    for dir in /lib/modules /etc/ssh ; do \
find $dir | perl -wpe 's/ /\\ /g' | xargs touch -m -a ; \
done

for file in /etc/resolv.conf ; do \
touch -m -a $file; \
done

  2. Remove old RedHat files

    Recall that we ``touch''ed the entire file system back in the Preparation Step. This was to make it easy to find them after the installation -- the Gentoo steps would overwrite whatever files were in common with RedHat, thereby updating their timestamps, while the files that were untouched would retain the old timestamp. Now we'll run a ``find'' looking for directories and files older than a certain number of days, and then ``rm'' them.

    NOTE: the ``9125'' number below references the number of days since my birthday, which again was the date I chose in the Preparation Step. If you chose a different date, you need to adjust this number accordingly. 

    find /bin /sbin /etc /lib /usr /var -type f -mtime +9125 | perl -wpe 's/ /\\ /g' | xargs rm -v
find /bin /sbin /etc /lib /usr /var -type d -mtime +9125 | perl -wpe 's/ /\\ /g' | xargs rm -vrf

    Doing this will screw up the installation of groff (man) and emacs. I'm not sure how, but the solution is simple and easy enough so instead of drilling down to figure this out, I just provide you with the solution: 

    emerge groff
emerge emacs
emerge vi

    See? Simple.

  3. Optional: Remove old RedHat devices and special files.

    In the interest of being complete, we also want to consider wiping out any and all "special" files (devices, pipes, sockets, etc) from RedHat that Gentoo doesn't use. Again, remember that the timestamps will be recent if the Gentoo Installation overwrote any of the files, so you should be safe.

    NOTE: I did not do this step. I don't know if it works, but there won't be much disk space to recover from this step and it just wasn't important to me in light of the fact that the Gentoo system prefers to use DevFS and will take over the ``/dev'' directory. 

    find /dev -type b -mtime +9125  | perl -wpe 's/ /\\ /g' | xargs rm -v
find /dev -type c -mtime +9125  | perl -wpe 's/ /\\ /g' | xargs rm -v
find /dev -type p -mtime +9125  | perl -wpe 's/ /\\ /g' | xargs rm -v
find /dev -type s -mtime +9125  | perl -wpe 's/ /\\ /g' | xargs rm -v

  4. Remove old RedHat directories

    If they are empty, you should nuke all the other RedHat directories. 

    cd / && rm -rf opt misc tftpboot

    Once you're off of the RedHat kernel, you should ``rmdir /initrd'' as well.

    IMPORTANT: don't nuke the initrd directory, even if it's empty, if you're still on the RedHat kernel -- the RedHat kernel boot will fail and hang when it goes to mount the initrd image if the mount point is missing.

  5. Secure the system.

    As a general practice, I don't like my users doing much on a system beyond what I have specifically provided for them. To that end, I favor limiting the permissions of standard system components, because you never know what's going to become a security risk.

    1. Limit permissions of SUID/SGID binaries. 

      for file in /usr/bin/suidperl /usr/bin/sperl5.* /usr/bin/chage /usr/bin/gpasswd                 \
            /usr/bin/crontab /usr/bin/chfn /usr/bin/chsh /usr/bin/newgrp /usr/sbin/traceroute   \
            /usr/bin/traceroute6 /bin/ping /bin/ping6 /bin/mount /bin/umount /usr/bin/write; do \
chmod o-rwx $file; chgrp wheel $file; chmod g-r $file ; \
done

    2. Limit permissions of system components that users shouldn't need to run. 

      for file in /usr/bin/*cc* /usr/bin/*86* /usr/bin/tracepath* /usr/bin/sudo /usr/sbin/*del \
            /usr/bin/i386-pc-linux-gnu-* /usr/bin/?++ /usr/i386-pc-linux-gnu/bin/*       \
            /usr/i386-pc-linux-gnu/gcc-bin/3.*/* /usr/sbin/*add /usr/sbin/*mod ; do      \
chmod o-rwx $file; chmod g-rw $file; chgrp wheel $file; \
done

If you want to recompile parts of the OS, now's the time! Install those missing components, emerge whatever you want. You're done!