Programmatically Building A FAT File System In Java
Overview
I was in the process of building a Gradle plugin for my bare-metal embedded Java on Raspberry Pi project. One of my requirements, and one of my gripes with the Pi, is the opaque “image” that has to be written to the SD card to boot the Pi. It has always seemed to me that there is a bit of “voodoo” involved in getting this right. The reason for my project is to bundle up all the necessary assembler, Java VM, class loader, and boot minutiae in one place, so that all one had to do is write Java code, build, and deploy. I wanted deployment to occur without having to alter the bootable image each time, and I wanted the plugin to be able to construct a bootable image for the Pi with a simple serial bootloader to facilitate. That way, one could code embedded apps without having to re-burn a new image to the CD card each time. Having all the necessary pieces self-contained within the plugin reduces friction.
What Is A Raspberry Pi Bootable Image?
At its most basic, it is a boot partition with a FAT32 filesystem containing a few binary and text configuration files. So what are those files? Below is applicable for the Pi Zero…other models have differing requirements (which leads to the complexity).
- Second stage bootloader (bootcode.bin) - This is used to retrieve the GPU firmware from the SD card, program the firmware, then start the GPU.
- Configuration (config.txt) - Firmware loader and the firmware itself can be configured with name value pairs contained within this file.
- GPU firmware (start.elf and fixup.dat) - Once loaded, this allows the GPU to start up the CPU. An additional file, fixup.dat, is used to configure the SDRAM partition between the GPU and the CPU. RAM is shared between devices. At this point, the CPU is released from reset and execution is transferred over to user code.
- User code (kernel.img) - This can be one of any number of binaries. Typically, this is the Linux kernel (usually named kernel.img), but it can also be another bootloader (e.g. U-Boot), or a bare-metal application (a serial bootloader in my case).
So How Do We Build A Raspberry Pi Image In Code?
The requirements are to format a default FAT32 boot partition on an SD card then place the files listed above on that partition. Creating and formatting a FAT32 partition on a local storage device is usually left to a host operating system. There are many reasons for that, but I want to make this process portable, such that if you use the plugin on any host, the image can be built without dealing with the nuances of the host. My Gradle plugin is built using Java, and so I went looking for a native Java library that could create partitions, format them as FAT32, and copy files to them, and do so to a local file on the host’s file system. This is so the image can then be bit copied to the SD card.
It turns out that there are not many libraries out there that do this. Only the brave would delve into the arcane world of FAT internals with its 40+ years of historical baggage.
But I did find one.
fat32-lib by Matthias Treydte is a very complete, native Java implementation of the various FAT file systems. It is a great work but lacks documentation. So below, I’ll describe how to code up creation of a Pi image.
The Code
package com.chuckbenedict.gradle.plugin.internal;
import de.waldheinz.fs.fat.FatFile;
import de.waldheinz.fs.fat.FatFileSystem;
import de.waldheinz.fs.fat.FatLfnDirectory;
import de.waldheinz.fs.fat.FatLfnDirectoryEntry;
import de.waldheinz.fs.fat.FatType;
import de.waldheinz.fs.fat.SuperFloppyFormatter;
import de.waldheinz.fs.util.FileDisk;
import java.io.File;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
/**
*
* @author Matthias Treydte <mt at waldheinz.de>
*/
public final class ImageBuilder {
public static ImageBuilder of(File rootDir) throws IOException {
if (!rootDir.isDirectory()) {
throw new IOException("root must be a directory");
}
return new ImageBuilder(rootDir);
}
private void copyContents(File f, FatFile file) throws IOException {
final RandomAccessFile raf = new RandomAccessFile(f, "r");
try {
final FileChannel fc = raf.getChannel();
long dstOffset = 0;
while (true) {
final int read = fc.read(this.buffer);
if (read >= 0) {
this.buffer.flip();
file.write(dstOffset, this.buffer);
this.buffer.clear();
dstOffset += read;
} else {
break;
}
}
} finally {
this.buffer.clear();
raf.close();
}
}
private final File imageRoot;
private final ByteBuffer buffer;
private ImageBuilder(File imageRoot) {
this.imageRoot = imageRoot;
this.buffer = ByteBuffer.allocate(1024 * 1024);
}
public void createDiskImage(File outFile) throws IOException {
final FileDisk fd = FileDisk.create(outFile, 33l * 1024 * 1024);
final FatFileSystem fs = SuperFloppyFormatter
.get(fd).setFatType(FatType.FAT32).setVolumeLabel("RPIBOOT").format();
try {
this.copyRec(this.imageRoot, fs.getRoot());
} finally {
fs.close();
fd.close();
}
}
private void copyRec(File src, FatLfnDirectory dst) throws IOException {
for (File f : src.listFiles()) {
System.out.println("-> " + f);
if (f.isDirectory()) {
final FatLfnDirectoryEntry de = dst.addDirectory(f.getName());
copyRec(f, de.getDirectory());
} else if (f.isFile()) {
final FatLfnDirectoryEntry de = dst.addFile(f.getName());
final FatFile file = de.getFile();
copyContents(f, file);
}
}
}
}
The Code Discussion
Usage
Bottom line on top…how do we use this class?
ImageBuilder.of(new File("/home/auser/pifiles"))
.createDiskImage(new File("/home/auser/bootable.img"));
Assume /home/auser/pifiles is a directory containing one or more files that you want to copy to the root of a new FAT32 file system. So in my case, pifiles would contain
bootcode.bin
config.txt
fixup.dat
kernel.img
start.elf
/home/auser/bootable.img is the resulting image file containing the new file system.
The static of method, given a directory reference of a File containing your image files, returns an ImageBuilder. The createDiskImage method of the ImageBuilder instance takes a reference to a File containing the name of the new FAT32 file system image you want to create on the local file system.
How ImageBuilder Does What It Does
public final class ImageBuilder {
public static ImageBuilder of(File rootDir) throws IOException {
if (!rootDir.isDirectory()) {
throw new IOException("root must be a directory");
}
return new ImageBuilder(rootDir);
}
private ImageBuilder(File imageRoot) {
this.imageRoot = imageRoot;
this.buffer = ByteBuffer.allocate(1024 * 1024);
}
}
The static of method simply delegates to the ImageBuilder constructor passing in the directory of files to contain the root of the new file system. The ImageBuilder constructor stores off the File and allocates a ByteBuffer that will later be used for copying file data to the image.
public void createDiskImage(File outFile) throws IOException {
final FileDisk fd = FileDisk.create(outFile, 33l * 1024 * 1024);
final FatFileSystem fs = SuperFloppyFormatter
.get(fd).setFatType(FatType.FAT32).setVolumeLabel("RPIBOOT").format();
try {
this.copyRec(this.imageRoot, fs.getRoot());
} finally {
fs.close();
fd.close();
}
}
The createDiskImage instance method first creates a FileDisk instance using the image outFile File reference passed into the method. A FileDisk is a block device that uses a file as a backing store. A block device is simply a blob of binary data that can be read/written in arbitrary blocks of data. This is the secret sauce that enables a FAT file system to be written into an image file.
Why is the size roughly 33MB and not smaller given the small sizes of the files in question? Because the specifications say a FAT32 image has a minimum size defined by (assuming 512 bytes per sector, which is typical):
FAT32 minimum :
1 sector per cluster × 65,525 clusters =
33,548,800 bytes (32,762.5 KB)
Next, the SuperFloppyFormatter class is used to create a FatFileSystem instance. What is a super floppy? Microsoft’s Specification says that a super floppy is removable media that does not contain a partition table. It represents one file system on the media. A Raspberry Pi image usually does contain a partition table with multiple file systems - the first file system being the bootloader and the second (or more) being dedicated to the operating system (like Linux). In my case, I only need the bootloader files and the Pi firmware understands the super floppy spec.
Finally, the copyRec instance method is called and is given the root of the source files and the root of the new FAT file system of type FatLfnDirectory. Lfn means long file name.
private void copyRec(File src, FatLfnDirectory dst) throws IOException {
for (File f : src.listFiles()) {
System.out.println("-> " + f);
if (f.isDirectory()) {
final FatLfnDirectoryEntry de = dst.addDirectory(f.getName());
copyRec(f, de.getDirectory());
} else if (f.isFile()) {
final FatLfnDirectoryEntry de = dst.addFile(f.getName());
final FatFile file = de.getFile();
copyContents(f, file);
}
}
}
The copyRec instance method iterates over all src files. If any of those files are directories, it creates a new directory of type FatLfnDirectoryEntry in the destination and recursively calls itself with files in the subdirectory. If an actual file, it creates a directory entry for the file, gets a reference to the new file of type FatFile and then calls copyContents passing in the original file reference and the new FatFile.
private void copyContents(File f, FatFile file) throws IOException {
final RandomAccessFile raf = new RandomAccessFile(f, "r");
try {
final FileChannel fc = raf.getChannel();
long dstOffset = 0;
while (true) {
final int read = fc.read(this.buffer);
if (read >= 0) {
this.buffer.flip();
file.write(dstOffset, this.buffer);
this.buffer.clear();
dstOffset += read;
} else {
break;
}
}
} finally {
this.buffer.clear();
raf.close();
}
}
copyContents takes the f File reference, which is only a reference to the location of the file, and creates a RandomAccessFile, which can be used to read the content of the file. Through a FileChannel, the source file is read into a buffer. The buffer pointer is reset to the beginning of the buffer with the flip method, and the buffer is written to the destination FatFile.
Copy Image To SD Card
Once the bootable.img is created, it must be copied in raw form to a blank SD card. The way to do this does vary between operating systems, but on a Mac, Linux, or Windows (with Windows Subsystem For Linux installed) host, one can use dd:
sudo dd if=bootable.img of=/dev/diskx bs=512
But It Didn’t Work!
Yeah, once I did all this, the Pi would not boot as expected. What was more puzzling is that I could reinsert the card into my Mac host and see all the files as expected on the card. Further, I could md5 the file contents and compare the checksum to the originals, and they all matched. This was a puzzle!
Troubleshooting
Without putting the uart_2ndstage=1 debug option in the config.txt file, I would have never solved this problem. This option will cause bootcode.bin and start.elf to output diagnostics to UART0.
When I copied the original source files from the Mac Finder on top of the dd copied SD card, I could get the image to boot. This was even more puzzling given that the original files md5 checksum matched.
I then copied each file in turn, and noticed that each bootloader stage would boot. I also noticed that it did not matter if kernel.img was copied. If start.elf started, then the kernel would run.
Observing this behavior, it became clear to me that the Pi first and second stage bootloaders could not read the FAT directory structures correctly when built by the FAT library. But why not? The Mac OS could, and further re-wrote them correctly.
I resorted to comparing the constructed FAT image file with a good bootable SD card using Hex Fiend. Eventually what I noticed was that the FAT32 library was writing all file directory entries as long file names, unless the files were named with the older 8.3 syntax in upper case. Microsoft in their infinite wisdom enabled lowercase 8.3 file names to be supported as short file names (this originally was not the case) using a custom flag on the directory structure as part of the VFAT specification starting with Windows NT and later versions of Windows XP. The library was not doing this.
Different FAT file support between the first and second stage bootloaders and the kernel loader was confirmed by a Raspberry Pi software engineer in Image file will not boot. He also confirmed short file name only FAT support in the first and second stage bootloaders. What I’d like to suggest is that those loaders have a hybrid-type of support, not supporting long file names, but including the lowercase support of 8.3 names. I am not sure a revision of FAT ever supported this combination.
Regardless, I added lowercase short file name support in this fork of fat32-lib.
Success
My Pi now boots with this programmatically constructed image!
Spread the Word
If you found this information helpful, I would appreciate a kind mention.
Click to Tweet (you may edit before posting.)
XRPTipBot tips appreciated too!