Gem5 Arm Fullsystem仿真

处理器与AI芯片 2023-01-04 2169

描述

gem5 system emulation 模式，内部实现了对system call的模拟，使用了一段时间后，有一些发现:

如果使用spec2017 X86编译，那么会存在对intel比较新的指令不支持的问题；后来使用gcc march K6 m32来解决，即使用amd的k6 32bit编译，但是这也只是权宜之计 ;
gem5的开发人员在邮件list中介绍intel对gem5的译码支持比较差，最开始gem5的x86也是基于amd的 ;
很多论文中使用了ARM架构，gem5的开发人员也和ARM合作比较紧密，所以gem5对arm架构的指令支持比较好 ;
即使使用上面的方法，se模式还是会遇到gem5没有模拟的system call函数的问题
同时还会遇到环境的问题，比如549.fotonik3d，需要手动将input输入的压缩文件OBJ.dat.xz手动解压之后才能运行。

因为上述的问题，在邮件list中可以看出gem5的开发人员比较倾向于使用full system模式，用他的话就是"works magically"。

X86 full system

如果是基于X86 Ubuntu系统模拟gem5 arch，制作disk image比较简单，可以git clone gem5 resource, 在spec2017文件夹下放入spec2017.iso，调用build.sh自动调用packer将spec2017装入ubuntu的disk image生成spec-2017。

 ./build/X86/gem5.fast 
 --outdir=./m5out/ 
 configs/example/gem5_library/x86-spec-cpu2017-benchmarks.py
 --image=path/spec-2017
 --benchmark=505.mcf_r
 --size=ref
 --partition=1

这里需要强调的是，gem5中介绍了多次qemu-kvm, 但是qemu-kvm并不是在X86架构中生成disk image和真正运行gem5时必须的工具。

在gem5中引入它的主要作用就是在boot阶段使用qemu，在真实的cpu上运行boot 阶段，进行加速，实际上如果我们在云服务器上跑，如果没有qemu软件或者权限，只是使用atomic cpu跑boot也是比较快的。

就是不要被qemu迷惑，fullsystem gem5可以理解成在gem5上跑app，不过这个app是os，单纯的用atomic cpu跑也没有任何问题。

ARM full system

X86 full system的问题是memory最大支持3GB，目前看gem5的设置是不支持5GB 6GB这样的设置。

我们介绍一下如何生成gem5的disk image。

首先看一下最终成功运行full system 使用的指令:

./build/ARM/gem5.fast 
 -d ./m5out/ARM/fullsystem64/spec2017 
./configs/example/fs.py  
 --kernel 2022/binaries/vmlinux.arm64 
 --disk-image path/expanded-aarch64-ubuntu-trusty-headless.img 
 --bootloader 2022/binaries/boot.arm64 --mem-type=DDR4_2400_4x16 
 --param 'system.highest_el_is_64 = True' 
 --script=./m5out/ARM/fullsystem64/spec2017_restore/spec2017.rcS

kernel 我的理解就是os内核程序，disk image则是装载了benchmark的磁盘镜像。

内核程序与我们无关，我们可以直接使用，disk image 则需要我们手动装载。

gem5官方提供的kernel和disk image

https://www.gem5.org/documentation/general_docs/fullsystem/guest_binaries

script这里指定的是一个script

#!/bin/bash
source /root/.bashrc
/sbin/m5 checkpoint 1
echo "A real multi node workload might start here ..."
cd /home/gem5/spec2017
source shrc
echo "Reset stats"
/sbin/m5 resetstats
runcpu --size test --iterations 1 --config myconfig.aarch64.cfg  --nobuild 605.mcf_s
/sbin/m5 exit 1

通过指定这个script，gem5在boot成功后，运行这个script，就调用了脚本内的runcpu，自动运行了spec2017对应的app。

如果我们不指定这个script，并且不对disk image进行任何修改，那么boot成功后，要求输入用户名和密码，输入root可以进入，不过这个操作比较麻烦，还是建议指定script。

现在唯一需要的工作就是实现disk image，这里帮助对我很大的是这篇博客。https://www.eecg.utoronto.ca/~elsayed9/website/blog/gem5_fs_arm_flow.php

首先遇到的问题就是官方提供的image 1GB或者2GB，然而spec2017有4GB我们需要对image进行扩容。按照博客的操作如下

```bash
$ # Backup the original disk image if needed
$ cp aarch64-ubuntu-trusty-headless.img expanded-aarch64-ubuntu-trusty-headless.img
$ # Increase disk image by 2G
$ dd if=/dev/zero bs=1G count=2 >> expanded-aarch64-ubuntu-trusty-headless.img
$ sudo parted expanded-aarch64-ubuntu-trusty-headless.img resizepart 1 100%
$ # Parse some info for 'losetup' and 'mount' later
$ name=$(sudo fdisk -l expanded-aarch64-ubuntu-trusty-headless.img | tail -1 | awk -F: '{ print $1 }' | awk -F" " '{ print $1 }')
$ start_sector=$(sudo fdisk -l expanded-aarch64-ubuntu-trusty-headless.img | grep $name | awk -F" " '{ print $2 }')
$ units=$(sudo fdisk -l expanded-aarch64-ubuntu-trusty-headless.img | grep Units | awk -F" " '{ print $8 }')
$ # Attach to device and record output, to me it was /dev/loop18                                                               
$ sudo losetup -f --show expanded-aarch64-ubuntu-trusty-headless.img -o $(($start_sector*$units))                         
$ sudo e2fsck -f /dev/loop18  # Fix potential errors, press Y for all fixes
$ sudo resize2fs /dev/loop18  # Actual resizing step
$ sudo e2fsck -f /dev/loop18  # Double check there are no error
$ sudo losetup -d /dev/loop18 # Detach from the loop device
$ # Mount image and check new size
$ mkdir disk_mnt
$ sudo mount -o loop,offset=$(($start_sector*$units)) expanded-aarch64-ubuntu-trusty-headless.img disk_mnt
$ df -h # Should show the new expanded image size with the Used and Avail for disk_mnt
$ sudo umount disk_mnt

扩容之后mount image,就可以安装spec2017到这个disk image了。

这里建议看一下 gem5-resources/src/spec-2017/disk-image/spec-2017/install-spec2017.sh 这个是装载spec2017到x86 os的过程，我们装载spec2017到arm，可以按照这个流程来。

gem5 resources的路径https://gem5.googlesource.com/public/gem5-resources

sudo chroot . #将当前mount目录切换为主目录
创建/home/gem5/文件夹将cpu_spec2017.iso拷贝到这个文件夹
按照cpu_spec2017.iso的install流程，mount cpu_spec2017.iso 然后install.sh
install 之后，我们可以build，生成spec2017的可执行文件等。建议参考install-spec2017.sh

disk image中已经有gcc aarch64的编译工具，因此我们不需要额外再安装gcc aarch64，还是比较方便的。

我们将spec2017安装到了/home/gem5/，再结合一下刚才介绍的spec2017.rcS，就能看出来这个script的作用实际上就是进入文件夹，然后runcpu。

我们看一下gem5成功boot后运行spec2017的os系统界面，这个界面通过./util/term/m5term 得到，后面会介绍。

[  0.345190] sd 00 [sda] Attached SCSI disk
[  0.352995] EXT4-fs (sda1): mounted filesystem without journal. Opts: (null)
[  0.353004] VFS: Mounted root (ext4 filesystem) on device 8:1.
[  0.353626] devtmpfs: mounted
[  0.353684] Freeing unused kernel memory: 448K
[  0.359059] random: fast init done
Mount failed for selinuxfs on /sys/fs/selinux:  No such file or directory
[  0.372646] random: init: uninitialized urandom read (12 bytes read)
[  0.399519] random: mountall: uninitialized urandom read (12 bytes read)
Boot Success Reset stats
Run 602.gcc_s test
SPEC CPU(r) 2017 Benchmark Suites
Copyright 1995-2017 Standard Performance Evaluation Corporation (SPEC)
runcpu v5825
Using 'linux-aarch64' tools
Reading file manifests... read 32270 entries from 2 files in 0.44s (72680 files/s)
Loading runcpu modules.................
Locating benchmarks...found 47 benchmarks in 53 benchsets.
Reading config file '/home/gem5/spec2017/config/myconfig.aarch64.cfg'
1 configuration selected:
 Action  Run Mode   Workload    Report Type    Benchmarks
--------   --------   --------   -----------------   --------------------------
validate   speed    test     SPECspeed2017_int   602.gcc_s               
-------------------------------------------------------------------------------
Setting up environment for running

大约1Billion指令之后，大约半小时，完成boot。再执行3.5Billion的指令进入真实的runcpu仿真。gem5仿真显示的界面：

**** REAL SIMULATION ****
build/ARM/dev/arm/rv_ctrl.cc:176: warn: SCReg: Access to unknown device dcc0pos0dev0
build/ARM/arch/arm/insts/pseudo.cc:172: warn:     instruction 'csdb' unimplemented
build/ARM/dev/arm/gic_v2.cc:683: warn: GIC APRn write ignored because not implemented: 0xd0
build/ARM/dev/arm/gic_v2.cc:683: warn: GIC APRn write ignored because not implemented: 0xd4
build/ARM/dev/arm/gic_v2.cc:683: warn: GIC APRn write ignored because not implemented: 0xd8
build/ARM/dev/arm/gic_v2.cc:683: warn: GIC APRn write ignored because not implemented: 0xdc
AtomicCPU 0 At 103419026000 Tid[0] 100000000 instructions are executed.
build/ARM/dev/arm/rv_ctrl.cc:122: warn: Tried to read RealView I/O at offset 0x60 that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:122: warn: Tried to read RealView I/O at offset 0x48 that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:198: warn: Tried to write RVIO at offset 0xa8 (data 0) that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:198: warn: Tried to write RVIO at offset 0xa8 (data 0) that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:198: warn: Tried to write RVIO at offset 0xa8 (data 0) that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:198: warn: Tried to write RVIO at offset 0xa8 (data 0) that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:198: warn: Tried to write RVIO at offset 0xa8 (data 0) that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:198: warn: Tried to write RVIO at offset 0xa8 (data 0) that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:198: warn: Tried to write RVIO at offset 0xa8 (data 0) that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:198: warn: Tried to write RVIO at offset 0xa8 (data 0) that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:198: warn: Tried to write RVIO at offset 0xa8 (data 0) that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:198: warn: Tried to write RVIO at offset 0xa8 (data 0) that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:122: warn: Tried to read RealView I/O at offset 0x8 that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:122: warn: Tried to read RealView I/O at offset 0x48 that doesn't exist
build/ARM/dev/arm/energy_ctrl.cc:77: warn: EnergyCtrl: Disabled handler, ignoring read from reg 0
AtomicCPU 0 At 437185231000 Tid[0] 200000000 instructions are executed.
AtomicCPU 0 At 499727384000 Tid[0] 300000000 instructions are executed.
AtomicCPU 0 At 582010000000 Tid[0] 400000000 instructions are executed.
AtomicCPU 0 At 641077500000 Tid[0] 500000000 instructions are executed.
AtomicCPU 0 At 700009321500 Tid[0] 600000000 instructions are executed.
AtomicCPU 0 At 759169539000 Tid[0] 700000000 instructions are executed.
AtomicCPU 0 At 818393124500 Tid[0] 800000000 instructions are executed.
AtomicCPU 0 At 877446054000 Tid[0] 900000000 instructions are executed.
showInstNum AtomicCPU 0 At 907078901500 Tid[0] 950264835 instructions are executed.
Writing checkpoint
build/ARM/sim/simulate.cc:194: info: Entering event queue @ 907078901500.  Starting simulation...
AtomicCPU 0 At 936518571000 Tid[0] 1000000000 instructions are executed.
AtomicCPU 0 At 1019589630500 Tid[0] 100000000 instructions are executed.
AtomicCPU 0 At 1075531936500 Tid[0] 200000000 instructions are executed.
AtomicCPU 0 At 1133110363000 Tid[0] 300000000 instructions are executed.
AtomicCPU 0 At 1190919530500 Tid[0] 400000000 instructions are executed.
AtomicCPU 0 At 1248468781000 Tid[0] 500000000 instructions are executed.
AtomicCPU 0 At 1305429062000 Tid[0] 600000000 instructions are executed.
AtomicCPU 0 At 1363401303000 Tid[0] 700000000 instructions are executed.
AtomicCPU 0 At 1421469390000 Tid[0] 800000000 instructions are executed.
AtomicCPU 0 At 1479484997500 Tid[0] 900000000 instructions are executed.
AtomicCPU 0 At 1537414678500 Tid[0] 1000000000 instructions are executed.
AtomicCPU 0 At 1595390184500 Tid[0] 1100000000 instructions are executed.
AtomicCPU 0 At 1646996227500 Tid[0] 1200000000 instructions are executed.
AtomicCPU 0 At 1698272492000 Tid[0] 1300000000 instructions are executed.
AtomicCPU 0 At 1754533327500 Tid[0] 1400000000 instructions are executed.
AtomicCPU 0 At 1814630599500 Tid[0] 1500000000 instructions are executed.
AtomicCPU 0 At 1866048774500 Tid[0] 1600000000 instructions are executed.
AtomicCPU 0 At 1922046022000 Tid[0] 1700000000 instructions are executed.
AtomicCPU 0 At 1978814167500 Tid[0] 1800000000 instructions are executed.
AtomicCPU 0 At 2036107971000 Tid[0] 1900000000 instructions are executed.
AtomicCPU 0 At 2093257147500 Tid[0] 2000000000 instructions are executed.
AtomicCPU 0 At 2150632827000 Tid[0] 2100000000 instructions are executed.
AtomicCPU 0 At 2206964371500 Tid[0] 2200000000 instructions are executed.
AtomicCPU 0 At 2264055743500 Tid[0] 2300000000 instructions are executed.
AtomicCPU 0 At 2324544549000 Tid[0] 2400000000 instructions are executed.
AtomicCPU 0 At 2381492086000 Tid[0] 2500000000 instructions are executed.
AtomicCPU 0 At 2439386832000 Tid[0] 2600000000 instructions are executed.
AtomicCPU 0 At 2497622146000 Tid[0] 2700000000 instructions are executed.
AtomicCPU 0 At 2556071830000 Tid[0] 2800000000 instructions are executed.
AtomicCPU 0 At 2613942974500 Tid[0] 2900000000 instructions are executed.
AtomicCPU 0 At 2672323657000 Tid[0] 3000000000 instructions are executed.
AtomicCPU 0 At 2730096081000 Tid[0] 3100000000 instructions are executed.
AtomicCPU 0 At 2788185618000 Tid[0] 3200000000 instructions are executed.
AtomicCPU 0 At 2846567200000 Tid[0] 3300000000 instructions are executed.
AtomicCPU 0 At 2906211195500 Tid[0] 3400000000 instructions are executed.
AtomicCPU 0 At 2965004517500 Tid[0] 3500000000 instructions are executed.

这里的Atomic****CPUinstructions are executed.是我自己增加的打印，不必关注，主要为了显示各个阶段的指令数目。

最后再介绍一下./util/term/m5term 3460。

在gem5开始运行后，会显示system.terminal listening for connections on port NUMBER.

这时我们另开一个terminal，输入./util/term/m5term NUMBER，即可观察到当前os具体运行到哪一步，而上面说的，如果不指定script，需要手动输入root也是在这里。

对我帮助很大的两篇博客:

https://www.eecg.utoronto.ca/~elsayed9/website/blog/gem5_fs_arm_flow.php

https://lucian.run/2021/10/03/gem5%20FS/

有小伙伴后台私信我申请读博士的事，有意向的小伙伴欢迎私信。

审核编辑：李倩

打开APP阅读更多精彩内容