January 20, 2023

Kyu networking -- Build NetBSD from source

Why you ask? And what does this have to do with Kyu networking?

I am working with my Orange Pi PC plus board and now want to look at the NetBSD sources to see how it handles MMU initialization. I have found in working with U-Boot that doing a build is very useful in a study of the source. The ".o" files serve as markers that indicate exactly which source files were used in the build.

Besides that, I haven't build NetBSD in over a decade. It will be interesting to see what is new and to do it just for "old times sake".

I used Git to fetch the latest sources. I have an ARM toolchain on my Fedora linux machine. In an ideal world I would just type "make", but some kind of configuration step will be needed to tell the NetBSD build system what board I want to build for.

The above page caught my eye, and upon reading it it becomes even more interesting as the writer says that he uses linux on an x86 machine as his build host. This page is pretty old (no date anywhere) as he talks about Fedora 5.

Build the toolchain

Rather than try to use the gcc/binutils toolchain I already have, I follow the recommended path and do the following:
cd /u1/NetBSD/Git/src
./build.sh -u -m evbarm tools
I get scolded immediately.
ERROR: No MACHINE_ARCH provided. Use 'build.sh -m evbarm list-arch' to show options
./build.sh -m evbarm list-arch
MACHINE=evbarm		MACHINE_ARCH=		NO_DEFAULT
MACHINE=evbarm		MACHINE_ARCH=earmv4	ALIAS=evbearmv4-el	ALIAS=evbarmv4-el
MACHINE=evbarm		MACHINE_ARCH=earmv4eb	ALIAS=evbearmv4-eb	ALIAS=evbarmv4-eb
MACHINE=evbarm		MACHINE_ARCH=earmv5	ALIAS=evbearmv5-el	ALIAS=evbarmv5-el
MACHINE=evbarm		MACHINE_ARCH=earmv5hf	ALIAS=evbearmv5hf-el	ALIAS=evbarmv5hf-el
MACHINE=evbarm		MACHINE_ARCH=earmv5eb	ALIAS=evbearmv5-eb	ALIAS=evbarmv5-eb
MACHINE=evbarm		MACHINE_ARCH=earmv5hfeb	ALIAS=evbearmv5hf-eb	ALIAS=evbarmv5hf-eb
MACHINE=evbarm		MACHINE_ARCH=earmv6	ALIAS=evbearmv6-el	ALIAS=evbarmv6-el
MACHINE=evbarm		MACHINE_ARCH=earmv6hf	ALIAS=evbearmv6hf-el	ALIAS=evbarmv6hf-el
MACHINE=evbarm		MACHINE_ARCH=earmv6eb	ALIAS=evbearmv6-eb	ALIAS=evbarmv6-eb
MACHINE=evbarm		MACHINE_ARCH=earmv6hfeb	ALIAS=evbearmv6hf-eb	ALIAS=evbarmv6hf-eb
MACHINE=evbarm		MACHINE_ARCH=earmv7	ALIAS=evbearmv7-el	ALIAS=evbarmv7-el
MACHINE=evbarm		MACHINE_ARCH=earmv7eb	ALIAS=evbearmv7-eb	ALIAS=evbarmv7-eb
MACHINE=evbarm		MACHINE_ARCH=earmv7hf	ALIAS=evbearmv7hf-el	ALIAS=evbarmv7hf-el
MACHINE=evbarm		MACHINE_ARCH=earmv7hfeb	ALIAS=evbearmv7hf-eb	ALIAS=evbarmv7hf-eb
MACHINE=evbarm		MACHINE_ARCH=aarch64	ALIAS=evbarm64-el	ALIAS=evbarm64
MACHINE=evbarm		MACHINE_ARCH=aarch64eb	ALIAS=evbarm64-eb
Some searches lead me to this more recent guide: Based on this, I try:
cd /u1/NetBSD/Git/src
#./build.sh -u -m evbarm tools
#./build.sh -U -O ~/obj -j2 -m evbarm -a aarch64 tools
./build.sh -U -O ~/obj -m evbarm -a earmv7hf tools
Away it goes! I could have used the -j option to encourage parallel action in make, but I am in no big hurry and would rather keep the build process as simple as possible.

While the build is grinding away, I do some reading. Apparently the "ALIAS" columns above show alternate expressions that combine MACHINE and MACHINE_ARCH, so for example I could have typed:

./build.sh -U -O ~/obj -m evbarmv7-el tools
This is indeed taking a long time. The encouraging part is that it is smoothly grinding its way along like it knows what it is doing. I see it is putting things into /home/tom/obj/tooldir... which makes perfect sense given the -O option I have it.

It finishes with this cheery summary:

===> Tools built to /home/tom/obj/tooldir.Linux-6.0.16-300.fc37.x86_64-x86_64
===> build.sh ended:      Fri Jan 20 09:22:45 MST 2023
===> Summary of results:
	 build.sh command:    ./build.sh -U -O /home/tom/obj -m evbarm -a earmv7hf tools
	 build.sh started:    Fri Jan 20 09:02:13 MST 2023
	 NetBSD version:      10.99.2
	 MACHINE:             evbarm
	 MACHINE_ARCH:        earmv7hf
	 Build platform:      Linux 6.0.16-300.fc37.x86_64 x86_64
	 HOST_SH:             /usr/bin/sh
	 No $TOOLDIR/bin/nbmake, needs building.
	 Bootstrapping nbmake
	 MAKECONF file:       /etc/mk.conf (File not found)
	 TOOLDIR path:        /home/tom/obj/tooldir.Linux-6.0.16-300.fc37.x86_64-x86_64
	 DESTDIR path:        /home/tom/obj/destdir.evbarm
	 RELEASEDIR path:     /home/tom/obj/releasedir
	 Created /home/tom/obj/tooldir.Linux-6.0.16-300.fc37.x86_64-x86_64/bin/nbmake
	 Updated makewrapper: /home/tom/obj/tooldir.Linux-6.0.16-300.fc37.x86_64-x86_64/bin/nbmake-evbarm
	 Tools built to /home/tom/obj/tooldir.Linux-6.0.16-300.fc37.x86_64-x86_64
	 build.sh ended:      Fri Jan 20 09:22:45 MST 2023
===> .

The config file

cd sys/arch/evbarm/conf
The file "GENERIC" has the line:
options 	SOC_SUN8I_H3
So, knowing nothing really, I do this (making no edits whatsoever to the file:
cp GENERIC ROTGUT
/home/tom/obj/tooldir.Linux-6.0.16-300.fc37.x86_64-x86_64/bin/nbconfig ROTGUT
cd ../compile/ROTGUT
/home/tom/obj/tooldir.Linux-6.0.16-300.fc37.x86_64-x86_64/bin/nbmake-evbarm depend
/home/tom/obj/tooldir.Linux-6.0.16-300.fc37.x86_64-x86_64/bin/nbmake-evbarm
Running mbconfig takes only seconds. The depend step takes a while, maybe a full minute. The build itself finishes in much less time than I expected:
/home/tom/obj/tooldir.Linux-6.0.16-300.fc37.x86_64-x86_64/bin/armv7--netbsdelf-eabihf-objcopy -S -O binary netbsd netbsd.img
/home/tom/obj/tooldir.Linux-6.0.16-300.fc37.x86_64-x86_64/bin/nbmkubootimage -A arm -T kernel_noload -O linux -C none -e 0 -n NetBSD/earmv7hf 10.99.2 netbsd.img netbsd.ub
 image type:  uimg
 magic:       0x27051956
 time:        Fri Jan 20 11:49:22 2023
 size:        11681440
 load addr:   0x00000000
 entry point: 0x00000000
 data crc:    0x8a2e844a
 os:          5 (linux)
 arch:        2 (arm)
 type:        14 (kernel_noload)
 comp:        0 (none)
 name:        NetBSD/earmv7hf 10.99.2
 header crc:  0x37dbb888
Here is the log from the build (5800 lines of it!) All of the .o files and such are in the "ROTGUT" directory, 5618 of them. What we want looks to be:
-rw-r--r-- 1 tom tom  7207584 Jan 20 11:49 netbsd.map
-rwxr-xr-x 1 tom tom 59060116 Jan 20 11:49 netbsd-ROTGUT.debug
-rwxr-xr-x 1 tom tom 13696280 Jan 20 11:49 netbsd
-rwxr-xr-x 1 tom tom 70807036 Jan 20 11:49 netbsd.gdb
-rwxr-xr-x 1 tom tom 11681440 Jan 20 11:49 netbsd.img
-rw-r--r-- 1 tom tom 11681504 Jan 20 11:49 netbsd.ub

What can we learn from the build log?

I wrote a python script to extract and reformat (one compact line per file) a list of all the source files involved in the build -- 1909 of them! Having this list allows searching to find files of interest for study. For example I can search on "sunxi" and find all kinds of things unique to the Allwinner chips. I can search on "*.S" to find all the assembler files, and so on. Some examples:
sys/arch/arm/sunxi/sunxi_mc_mpstart.S
sys/arch/arm/sunxi/sunxi_mc_smp.c
common/lib/libc/arch/arm/string/memcpy.S
common/lib/libc/arch/arm/string/memmove.S
common/lib/libc/arch/arm/string/strcpy.S
sys/arch/arm/arm/armv6_start.S
sys/arch/arm/arm32/locore.S
sys/arch/arm/arm/psci_arm.S
sys/arch/arm/xilinx/zynq7000_uart.c
And remember, these are only files used in a 32 bit ARM build.

A quick side note. How can this "generic kernel" work. Well, with the help of the device tree, that's how.

Boot and run this thing

I recently made a bootable SD card for NetBSD 9.3 -- so maybe we can just mount this card, copy the kernel I just built onto it, and boot it up. I actually redo the process and make a fresh new 16G card just for this:
dd if=u-boot-sunxi-with-spl.bin of=/dev/sdg bs=1k seek=8 conv=sync
dd if=armv7.img of=/dev/sdg bs=1M conv=sync
Looking at this card, I see a FAT partition (presumably for EFI) and a bigger partition that linux will not mount. In the FAT partition I see 3 big files:
-rw-r--r-- 1 tom tom 8887632 Aug  4 12:04 kernel7.img
-rw-r--r-- 1 tom tom 8887640 Aug  4 12:04 netbsd-GENERIC.ub
-rw-r--r-- 1 tom tom 2846852 Aug  4 12:04 start.elf
I have some odd trouble booting this card at first, but after mounting it on linux to look at it, it boots OK. Maybe linux put a partition table on it for me? Clearly the two "dd" commands will not do that. I don't really have a clue, but I'll just try something. Linux mounts the FAT partition as /run/media/tom/NETBSD/
cd /run/media/tom/NETBSD/
mv kernel7.img ORIG.img
mv netbsd-GENERIC.ub ORIG.ub
cd /u1/NetBSD/Git/src/sys/arch/evbarm/compile/ROTGUT
cp netbsd.ub /run/media/tom/NETBSD/netbsd-GENERIC.ub
cp netbsd.img /run/media/tom/NETBSD/kernel7.img
sync
Well, this does not work. It still boots NetBSD 9.3 from the flash card (not the freshly compiled xxx) and 9.3 still does this shortly after boot:
[  26.1506681] uvm_fault(0x913a71d8, c000, 2) -> e
[  26.1506681] Fatal kernel mode data abort: 'Translation Fault (S)'
[  26.1606687] trapframe: 0xbd73fc38
[  26.1606687] FSR=00000805, FAR=0000c110, spsr=200c0013
[  26.1706686] r0 =0000c100, r1 =00000000, r2 =901d6000, r3 =00000000
[  26.1806685] r4 =90b7f910, r5 =00000000, r6 =00000000, r7 =809582f0
[  26.1806685] r8 =00000000, r9 =04526000, r10=0000000c, r11=bd73fcac
[  26.1906685] r12=901d6000, ssp=bd73fc88, slr=90b7f91c, pc =8037b8d4

Stopped in pid 144.1 (dhcpcd) at        netbsd:uvm_pagefree+0x200:      strne
lr, [r0, #0x010]


Have any comments? Questions? Drop me a line!

Kyu / [email protected]