You may have read blogs about using pivot root to make EBS persistent boots, and creating AMIs from EBS-backed instances, yet still be wondering, “How the hell?” I hope this post answers that exact question.
Normally, AMI images can only boot from EBS snapshots. The following steps will allow you to create an AMI image which will use any EBS volume as root (/), specified on instance creation, of course.
This is how it works:
The AMI boots as per usual, but we’ve replaced /sbin/init with a clever script that runs as soon as the kernel loads. This clever script uses the /opt/aws/bin/ec2-attach-volume API tool to attach our persistent EBS volume–specified in your instance User Data field. It then reboots the system.
On the second boot, our init script mounts the now attached /dev/sdf (or /dev/xvdf) and runs pivot_root and chroot. At this point, the actual init runs and we have booted to a persistent root drive.
This is how to do it:
Anything in {}’s should be updated for your needs/situation.
1) Creating the Workbench Instance
a) Create a tempoaray instance, using the AMI you want your system based off of.
b) Log in with you SSH private key as ec2-user.
c) Change root password:
sudo passwd
d) Become super user:
su
e) Install Java:
yum install java
f) Create, or use an existing, AWS access key.
g) Copy X.509 certificate to /root/cert.pem
h) Copy private key to /root/pk.pem
2) Create Boostrapping System
a) Create and attach 1G EBS volume as /dev/sdl
Note: Must be in same Availability Zone as the current instance.
b) Format new partition:
/sbin/mkfs.ext3 /dev/sdl
c) Label new partition:
/sbin/e2label /dev/sdl /
d) Make mounting directory:
mkdir /mnt/sdl
e) Mount parition:
mount /dev/sdl /mnt/sdl
f) Copy root file system:
rsync -avHx / /mnt/sdl
g) Make devices:
/sbin/MAKEDEV -d /mnt/sdl/dev -x sdf /sbin/MAKEDEV -d /mnt/sdl/dev -x xvdf /sbin/MAKEDEV -d /mnt/sdl/dev -x console /sbin/MAKEDEV -d /mnt/sdl/dev -x null /sbin/MAKEDEV -d /mnt/sdl/dev -x zero
h) Create pivot directory:
mkdir /mnt/sdl/mnt/new-root/
i) Remove old init:
mv /mnt/sdl/sbin/init /mnt/sdl/sbin/init.orig
j) Create new init with the following code:
vim /mnt/sdl/sbin/init
#!/bin/sh
PATH=/bin:/usr/bin:/sbin:/usr/sbin
NEWDEV="/dev/sdf"
NEWDEV2="/dev/xvdf"
NEWTYP="ext3"
NEWMNT="/mnt/new-root"
OLDMNT="/mnt/old-root"
OPTIONS="noatime,ro"
attachEc2Volume()
{
echo "Could not mount. Starting network."
declare -x AWS_AUTO_SCALING_HOME="/opt/aws/apitools/as"
declare -x AWS_CLOUDWATCH_HOME="/opt/aws/apitools/mon"
declare -x AWS_ELB_HOME="/opt/aws/apitools/elb"
declare -x AWS_IAM_HOME="/opt/aws/apitools/iam"
declare -x AWS_PATH="/opt/aws"
declare -x AWS_RDS_HOME="/opt/aws/apitools/rds"
declare -x EC2_AMITOOL_HOME="/opt/aws/amitools/ec2"
declare -x EC2_CERT="/root/cert.pem"
declare -x EC2_HOME="/opt/aws/apitools/ec2"
declare -x EC2_PRIVATE_KEY="/root/pk.pem"
declare -x JAVA_HOME="/usr/lib/jvm/jre"
declare -x PATH="/usr/local/bin:/bin:/usr/bin:/opt/aws/bin"
echo "Path is $PATH"
/etc/init.d/network start
VOL_ID=`curl http://169.254.169.254/1.0/user-data`
echo "Detaching volume $VOL_ID"
ec2-detach-volume $VOL_ID
INSTANCE_ID=`curl http://169.254.169.254/1.0/meta-data/instance-id`
echo "Attaching volume $VOL_ID to this instance $INSTANCE_ID as $NEWDEV"
ec2-attach-volume -v $VOL_ID --instance $INSTANCE_ID --device $NEWDEV
echo "Rebooting"
/sbin/reboot -f
exit
}
echo "Remounting writable."
mount -o remount,rw /
[ ! -d $NEWMNT ] && echo "Creating directory $NEWMNT." && mkdir -p $NEWMNT
echo "Trying to mount $NEWDEV or $NEWDEV2 writable."
mount -t $NEWTYP -o rw $NEWDEV $NEWMNT || mount -t $NEWTYP -o rw $NEWDEV2 $NEWMNT || attachEc2Volume
echo "Mounted."
[ ! -d $NEWMNT/$OLDMNT ] && echo "Creating directory $NEWMNT/$OLDMNT." && mkdir -p $NEWMNT/$OLDMNT
echo "Remounting $NEWMNT $OPTIONS."
mount -o remount,$OPTIONS $NEWMNT
echo "Trying to pivot."
cd $NEWMNT
pivot_root . ./$OLDMNT
for dir in /dev /proc /sys; do
echo "Moving mounted file system ${OLDMNT}${dir} to $dir."
mount --move ./${OLDMNT}${dir} ${dir}
done
echo "Trying to chroot."
exec chroot . /bin/sh -c "umount ./$OLDMNT; exec /sbin/init $*" < /dev/console > /dev/console 2>&1
k) Make init executable:
chmod 755 /mnt/sdl/sbin/init
l) Unmout partition:
umount /mnt/sdl
3) Create AMI
a) Create a snapshot of the 1G EBS:
ec2-create-snapshot {vol-7f678417} --description {bootstrap-10}
Note: {vol-…} must be the EBS volume you create in step 2.
b) Create an AMI from the snapshot:
ec2-register --kernel {aki-427d952b} --architecture {x86_64} --name {ebs-bootstrap-v10} -s {snap-b9086fd3}
Note: {snap-…} must be the snapshot just created.
Note: Kernel and Architecture must be same as the current instance
4) Create Root EBS Volume
a) Create and attach {10G} EBS volume as /dev/sdf
Note: Must be in same Availability Zone as the current instance.
b) Format new partition:
/sbin/mkfs.ext3 /dev/sdfa
c) Label new partition:
/sbin/e2label /dev/sdf /
d) Make mounting directory:
mkdir /mnt/sdf
e) Mount parition:
mount /dev/sdf /mnt/sdf
f) Copy root file system:
rsync -avHx / /mnt/sdf
g) Unmout partition:
umount /mnt/sdf
h) Detach the EBS volume from instance.
5) Start your New AMI
a) Start an instance:
ec2-run-instances --kernel {aki-427d952b} --availability-zone {us-east-1c} --instance-type {t1.micro} \
--user-data {vol-49678421} {ami-a034c3c9}
Note: {ami-…} must be the AMI you created in step 3.
Note: {vol-…} must be the EBS volume you created in step 4.
Note: Kernel and and Availability Zone must be same as the original instance.
b) Monitor startup:
ec2-get-console-output {i-aea939c3}
Note: {i-…} must be the instance just create.
Linux and Lessons Learned 