January 14, 2021

EBAZ4205 Bitcoin miner board - XC7Z010 pins CPU cores

The part on my board is a Zynq-XC7Z010. It has two ARM Cortex-A9 cores.

The following gem of an application note gives the elusive details about how to gain access to the second core:

How it works

Unlike some other SoCs the Zynq starts both cores on reset (no need to enable clocks and fiddle with reset control bits). Core0 of course gets busy and runs the boot code, ultimately U-Boot and then our code. Core1 gets parked running in a loop in OCM (on chip memory).

OCM is 256K of ram that ends up mapped to the very end of the addess space. Core1 gets parked in a loop that lives in OCM at 0xfffffe00. (OCM itself extends from 0xfffc0000 to 0xffffffff.) The loop has the cpu execute WFE, where it sits and waits for an exception. When one comes along, it pops out of the WFE, inspects location 0xfffffff0 and if it finds it non-zero, branches there. If it is zero, it loops back and executes the WFE again and sits waiting for another exception.

I played around putting addresses into 0xfffffff0 and got no action. What was required was to put an address there and then execute a SEV instruction. This got results! The SEV instruction sends an exception to all cores in a multicore system (or else is a NOP if not implemented). Hence it is just what we require.

When the core starts up, nothing has been initialized. In particular the SP is zero, which works out OK since the first push onto the stack will wrap back to the end of OCM. This is fortunate behavior, and allows me to aim the second core directly at C code without bouncing through some assembler setup in a file like start.S. Ultimately of course I will want to do a considerable amount of assembler setup to get the second core running.

That loop in OCM

Here it is dumped in hex: 
FFFFFF20  F57FF04F E320F002 EAFFFFFC F57FF04F
FFFFFF30  E320F002 E3E0000F E590E000 E37E00D4
FFFFFF40  0AFFFFF9 EE070F15 EE070FD5 EE080F17
FFFFFF50  E3A04000 EE014F10 E12FFF1E 00000000
And after some fooling around, I obtained this disassembly: 
  20:   f57ff04f        dsb     sy
  24:   e320f002        wfe
  28:   eafffffc        b       20 ??

  2c:   f57ff04f        dsb     sy
  30:   e320f002        wfe
  34:   e3e0000f        mvn     r0, #15         ; 0xfffffff0
  38:   e590e000        ldr     lr, [r0]
  3c:   e37e00d4        cmn     lr, #212        ; 0xd4
  40:   0afffff9        beq     2c

  44:   ee070f15        mcr     15, 0, r0, cr7, cr5, {0}
  48:   ee070fd5        mcr     15, 0, r0, cr7, cr5, {6}
  4c:   ee080f17        mcr     15, 0, r0, cr8, cr7, {0}
  50:   e3a04000        mov     r4, #0
  50:   ee014f10        mcr     15, 0, r4, cr1, cr0, {0}
  50:   e12fff1e        bx      lr
  50:   00000000
We have two loops. The first does not seem of interest. The second does what has been described. Note that it does not compare against 0, but against a specific value, the negative of 0xd4. So the magic location must have been preloaded with this value. And in fact, an experiment shows the magic location has been preloaded with 0xFFFFFF2C -- the very address of the loop we are examining. In a way this is nice because we could specify a jump to address 0 if we had code there.

Last we have a gang of these mcr instructions which are placing values into registers in the system control coprocessor. Here is a short summary of that. 

mcr     15, 0, r0, cr7, cr5, {0}
15 is the coprocessor number (the system coprocessor)
0 is "opcode 1" (always zero for the 4 mcr instructions here.
Rd is the register we are writing from (either r0 or r4 here).
crn is the primary destination
crm is the secondary register name.
{x} is opcode 2

So what do these do? 

cr7,cr5,0 - invalidate all instruction caches to PoU
cr7,cr5,6 - invalidate all branch predictors
cr8,cr7,0 - invalidate entire unified TLB
cr1,cr0,0 - system control register (SCTLR) set to zero
The first 3 don't care what is in the processor register (r0), they invalidate entire objects. The last one writes the contents of r4 (which is zero) to the SCTLR.
Feedback? Questions? Drop me a line!

Tom's Computer Info / [email protected]