How To Make Your Xen-PAE Kernel Work With More Than 4GB RAM (Debian Etch With GRUB)

Version 1.0
Author: Falko Timme <ft [at] falkotimme [dot] com>
Last edited 12/13/2007

If you have a server with more than 4GB RAM and want to install a 32bit Debian Etch on it (following this tutorial: Debian Etch And Xen From The Debian Repository), you'd expect the Xen-PAE kernel to see all your RAM because the Xen-PAE kernel supports up to 64GB RAM. In fact, it recognizes only about 3.3GB RAM due to a bug in the GRUB bootloader. This article explains how you can fix GRUB so that all your RAM gets recognized.

I do not issue any guarantee that this will work for you!

1 Preliminary Note

This bug exists on 64bit Debian systems as well - the default Debian Xen kernel also recognizes only 3.3GB of RAM. It's possible that the following procedure works for 64bit Debian Etch systems as well, although I have tested it only on a 32bit Debian Etch system.

I assume that you have already set up Xen on your system according to this tutorial: Debian Etch And Xen From The Debian Repository

I have tested this on a system with 6GB RAM.

Don't follow this tutorial if you're using another bootloader than GRUB (e.g. lilo)!

2 Check Your Memory

You can run

cat /proc/meminfo

to see how much memory your system recognizes:

Debian-40-etch-32-minimal:~# cat /proc/meminfo
MemTotal:      3468288 kB
MemFree:       3237948 kB
Buffers:         55884 kB
Cached:          46124 kB
SwapCached:          0 kB
Active:          68820 kB
Inactive:        41412 kB
HighTotal:     2731012 kB
HighFree:      2667548 kB
LowTotal:       737276 kB
LowFree:        570400 kB
SwapTotal:     2104440 kB
SwapFree:      2104440 kB
Dirty:            3524 kB
Writeback:           0 kB
AnonPages:        8196 kB
Mapped:           3932 kB
Slab:            10140 kB
PageTables:        304 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   3838584 kB
Committed_AS:    50368 kB
VmallocTotal:   116728 kB
VmallocUsed:      3776 kB
VmallocChunk:   112612 kB
Debian-40-etch-32-minimal:~#

As you see, only about 3.3GB are recognized by the system...

3 Rebuild GRUB

Now let's rebuild the GRUB bootloader. We need the package dpkg-dev for it:

apt-get install dpkg-dev

Then we download the GRUB sources...

apt-get source grub

... and install all requirements to rebuild GRUB:

apt-get build-dep grub

We must modify the stage2/common.c file in the GRUB source directory. First we make a copy of that file:

cd grub-0.97/
cp -a stage2/common.c stage2/common.c.original

Then we open it:

vi stage2/common.c

Around line 143, you should find this section:

[...]
#ifndef STAGE1_5
  unsigned long cont, memtmp, addr;
  int drive;
#endif
[...]

Modify it as follows:

[...]
#ifndef STAGE1_5
  unsigned long memtmp, addr;
  volatile unsigned long cont;
  int drive;
#endif
[...]

Save the file, then build the new GRUB package:

debian/rules binary

Afterwards we can install the new GRUB package as follows:

dpkg -i ../grub_0.97-27_i386.deb

Now we must install GRUB on our hard drives. I'm using two hard drives, /dev/sda and /dev/sdb (for software RAID1), so I run

grub-install /dev/sda
grub-install /dev/sdb

If you only got one hard drive, then run grub-install only once. Make sure that you use the correct device name for your hard drive (for example, instead of /dev/sda, yours might be /dev/hda - you can find out about the hard drive names by running

fdisk -l

).

Finally, it's time to reboot:

reboot

4 Check Your Memory Again

After the reboot, it's time to check the memory again. If everything went well, the system should now recognize the full amount of RAM:

cat /proc/meminfo

Debian-40-etch-32-minimal:~# cat /proc/meminfo
MemTotal:      6067200 kB
MemFree:       5902700 kB
Buffers:          1596 kB
Cached:          13964 kB
SwapCached:          0 kB
Active:          19316 kB
Inactive:         4408 kB
HighTotal:     5329732 kB
HighFree:      5299600 kB
LowTotal:       737468 kB
LowFree:        603100 kB
SwapTotal:     2104440 kB
SwapFree:      2104440 kB
Dirty:             784 kB
Writeback:           0 kB
AnonPages:        8160 kB
Mapped:           3932 kB
Slab:             8616 kB
PageTables:        324 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   5138040 kB
Committed_AS:    50388 kB
VmallocTotal:   116728 kB
VmallocUsed:      3776 kB
VmallocChunk:   112612 kB
Debian-40-etch-32-minimal:~#

As you see, my system is now recognizing my full 6GB of RAM.

5 Links

• Debian: http://www.debian.org