During the last SoC IG group meeting, I’ve presented (slides) an alternative approach to boot a general purpose OS on SoC based systems. The overall idea is to split the booting process in two decoupled parts:
setting up the hardware: done by early stage booting phases and U-Boot (or Barebox)
booting the OS: done by GRUB
This approach allows to decouple the OS development from the hardware platform one. The touchpoint between the two sides is the execution of GRUB as an EFI application by U-boot. GRUB and the OS will then boot under the assumption that all the hardware configuration were done before by U-Boot and the other early stage booting phases.
In a few words, using an analogy, this is the same approach used by all PCs where the BIOS or the UEFI firmware is in charge of setting up the hardware and executing the bootloader.
This is clearly not exhaustive as a description, but it should be enough to trigger thoughts and questions.
The official documentation we are building for our project do not yet describe this approach, however we have a dedicated chapter on how to use GRUB within the scope of building the next Linux image for Front-End Computers in the ATS sector.
Hi, I have been following this development and it is of great interest to me. Could you please elaborate on the process for the whole stack up for the split-boot? I have been following the workshop and haven’t really found a clear step by step guide on how the RPU FSBL is configured.
I will be really thankful if you have any additional information in this matter.
There are no particular FSBL configurations to be applied to implement the split I described with GRUB booting the OS, and FSBL + U-boot to handle the lower levels. The only thing that technically matters in this split is that U-Boot must be built with the capability of starting EFI applications (The command bootefi), and the device tree needs to be available in U-Boot. This is also the beauty of this split: it does not have strong requirements nor dependencies.
Unfortunately, we did not find the time yet to describe step by step of to reproduce this setup
I know that this has been done before. It would be interesting to do it again with proper documentation. I have limited time, but I am happy to contribute if I can.
Thanks for the proposal which I see as going in the direction to simplify via separation of concerns.
I think that we should be clear as to what is meant with separating hw and sw in the sense that out of a design compilation come two items that have drastically different lifetimes:
the FSBL PS configuration that needs to take place before booting (i.e., it’s like a BIOS), and
the bitfile+dtsi PL configuration that can be loaded and reloaded ad libitum after the OS boots.
In this sense, I think that we should be careful with the assumption that
the OS will then boot under the assumption that all the hardware configuration [was] done
Basically, what has been configured is the PS; the PL can be loaded and reloaded afterwards.
In this context, I would think that the important point is for there to be automated (CI/CD) ways to go from a firmware design to an “updated BIOS version”. If, in particular, there would be a way to “just” drop the FSBL into an otherwise “generic BIOS”, that would be even better, providing almost perfect factorisation. By perfect, I mean that if I have a new design that requires a new FSBL, I’d place that one file in place, reboot, and be done.
The main message remains that the assumption highlighted above should be considered in light of the ability to reload the PL from a running OS.
@adavid We were also thinking about this and what we came up with is to basically cut everything that is depending on the configuration of the PS out of the FSBL and perform those configurations as a later step in the boot process. This way the boot firmware is basically generic, like a “generic BIOS”, and the actual configuration of the PS can be fetched at runtime via U-Boot.
I think this might be a great addition for booting SoCs with the approach presented by @fvaga. The combined procedure would look something like this: There are generic versions of the FSBL, the PMU Firmware, the ATF and U-Boot on the board, e.g. in the QSPI memory. The SoC boots into U-Boot, fetches the configuration for the PS e.g. from the network and applies it. After that, U-Boot could fetch GRUB and start it. U-Boot could basically also be used to fetch and apply the bitfile.
I am looking forward to hear your thoughts on this.
Yes, you are hitting the nail in the head and this process would fulfil the point I raised, the critical step being step no. 2.
I think that, echoing comments from the meeting in early May, it would be awesome if the two efforts (yours and @fvaga’s) could find a way to merge their functionality into a single stack.
Thanks for your comments. Seems to me that we are progressing on something fundamental. I would also agree with @adavid, that it would be great if the the two efforts of the split-boot first part, and the GRUB-boot second part could be joined and provide a single boot stack
There is, however, one thing that I have not yet fully understood; for the first part to work, you need U-Boot being able to communicate via Ethernet. And for that you need some part of the PS already being configured, or additionally also some part of the PL, depending on how the actual hardware accesses the Ethernet. Is this right? It just came to my mind when I read @fuchsm 's sequence of boot steps again …
There is actually one more question that came to my mind: how often do projects realistically need to change the configuration of the PS part? I mean in the scenario that @adavid described, how often would the FSBL need to be changed?
We recently implemented a change to the default FSBL that came with PetaLinux and the ZCU102 BSP where we needed a flag to be set (FSBL_A53_TCM_ECC_EXCLUDE_VAL=0) in order for the R5 cores on our MpSoC to get access to the full TCM memory. I would consider this a “set-once-never-change-again” thing unless we discover other things in the future that needs to be changed in the FSBL.
In our case, not often. And we are the ones who have gone for PL reconfiguration on a running Linux system, which is something many do not even consider.
That said, the main point for me is to have the ability to easily use CI pipelines to create artefacts from running Vivado and drop them into a running system when needed, not just for convenience but also for reproducibility.
That’s correct. You need some kind of a base configuration that fits the hardware of the board. It is however possible to do a partly configuration of the PS in the FSBL and later on to overwrite this configuration in U-boot. The same applies to the PL. On the serenity boards I have been working with we need AXI-I2C IP cores in the PL for some configuration tasks in the FSBL. So we use a temporary bitfile that is loaded in the FSBL and overwrite this with the final one that is loaded in U-Boot.
With help from @fvaga I got GRUB running in combination with our “Split Boot” on the Kria. However, I am currently a bit limited in time, so I could not get the rest of the boot process after GRUB to work.
I am also working on getting our tools for the “Split Boot” and the documentation ready to share, so you can try it yourself.
A full boot sequence using “split boot” and GRUB sounds absolutely great!
Please let us know when you have will have achieved this completely! And when you will have documentation available …
I am still concerned about the base configuration for “split boot” and the reconfiguration during the U-Boot phase. How will this work if the board uses the SoC Ethernet through the PL, e.g. for 10 GBit/s Ethernet? So, the question is, what are the limitations for the base configuration, and what can be re-configured later? Or to turn it another differently: are there restrictions or recommendations to the designers when they develop the board the SoC or SoM is on? In other words, in order to define a common boot sequence, do we have to provide guidelines for the designers of future boards
As for the PS configuration I also think that it is not expected to change very frequently, at least not nearly as often as the PL bitstream file … But I agree with André David that it is important to have a “drop-in” mechanism for reproducibility and for early detection of problems, i.e. to make use of CI. Maybe that is another case for guidelines how to build the complete set of files required for the common boot sequence
Using Ethernet through the PL shouldn’t be a problem. One could, for example, download all files needed in U-Boot before one modifies the configuration of the PS. In this case, it would not be a problem if U-Boot could not use Ethernet with the modified configuration of the PS anymore.
The only thing I noticed so far is that I have to reload the Ethernet driver in U-Boot after overwriting the configuration in the PS (because overwriting the PS includes a reset of the GEM). But this is just one more step and not really a limitation.
As far as I can tell, there is nothing special to consider when designing the boards in terms of Split-Boot. Besides the need of an Ethernet interface and a non-volatile memory of course. However, we have not tested too many boards and FPGA projects with Split-Boot yet, so maybe I am taking something for granted that is actually not. For example, there might be limitations if a board aims for booting the APU while other parts of the Zynq are still unpowered.
