Life without Magisk

[Renate]

Magisk is a great and clever program. Still, it has to go through a lot of machinations to make its magic work. Try doing a "mount" command and see all the stuff that it has to jimmy up to make things work. Maybe the performance hit is not bad, but still...

It is possible to just make the mods that you want without Magisk. This will require you to make a full, binary backup of /vendor and /system so that when Onyx releases an incremental update it will recognize those two partitions as "virgin". Of course, retaining the last non-incremental update and fudging the script or build dates will allow you to downgrade then upgrade.

The system partition is verified against a hash tree when loading, so we want to get rid of verification. The system partition is loaded from /vendor/etc/fstab.qcom That is also verified against a hash tree when loading so we want to get rid of that verification. The vendor partition is loaded from the dtb that is stuck on the kernel in the boot partition.

So, we have to:

• Modify the dtb in boot so that the flags "wait,verify" are changed to "wait"
• Modify /vendor/etc/fstab.qcom so that the flags "wait,verify" are changed to "wait"

Are we there yet? Almost.
The two partitions vendor and system are protected by error correction. Android will attempt to correct any changes you make to those two partitions. We have to get rid of that too. The last 4096 block in each partition contains two copies of the FEC header. We can just zero those out.

So now you have a system where these two partitions are normally only mounted read-only, but could be modified externally. If you have a rooted recovery you can make your desired changes and reboot to the main system.

To be continued...

[mergen3107]

So, basically this looks like CWM?

[Renate]

Originally Posted by mergen3107

So, basically this looks like CWM?

Mmm, not really. It's complicated.

The first part is moving your Magisk modules to hard modifications in /system.
Magisk does disable some? verifications but doesn't? get rid of the error correction.

So you have to do a full (i.e. partition) backup of the system and vendor partitions and then disable error correction (FEC). You can then copy over your modifications to the system partition, then you can delete the Magisk modules. You're still running Magisk, but without modules. You still have access to "su".

When Magisk is running, you can't modify the system partition because it's all part of the stuff that Magisk is doing. Maybe you can mount it separately, but I'm still figuring stuff out and I don't need that uncertainty. You need a recovery. For recovery I don't need menus and "push this button", I only need a recovery with a rooted, permissive ADB. I do have one.

# mkdir fake
# mount -t ext4 /dev/block/by-name/system /fake
^D
C:\>adb push whatever /fake/system/bin/
C:\>adb shell
# chmod 755 /fake/system/bin/whatever

Of course all this stuff can be put into a makefile or a batch file to redo it all instantly when the next Onyx update comes along.

The second step would be to remove Magisk entirely. I have that working now and all my modifications work correctly. I have normal ADB access but currently it's not running rooted as I've run into some problems with permissions and I'd have to run it SELinux permissive.

Yeah, there is still work to do, but the bottom line is that I can reboot, hit "Last Read" and I'm back to my book where I was and my clicker works correctly.

[Renate]

Interim progress report:
I'm running without Magisk, with enforcing and I have a rooted ADB accessible.

I have to say, if you run the system in SELinux permissive, it's a train wreck.
All this goofy stuff writes all sorts of stuff that it shouldn't.
If you see navigation icons at the bottom of the screen you might be running permissive.

I've still a ways to go to make this a usable product.

[Renate]

I'm pretty happy with things as they are now. I've done a bunch of work on fixing the USB connection so that Onyx can't screw it up. It's always insisting on enabling MTP. My solution won't appeal to many. I just want my rooted ADB.

For those who want to mod their system without using Magisk, the information above is correct, disable verify on /vendor, disable verify on /system, disable error correction on both. To modify DTBs (without "round-tripping" them through dtc disassemble, dtc assemble) you can use my latest released dtbview.exe (in the sig). To extract/replace the dtb in images you can use imgutil.exe

Just extract, dump, hex edit, replace:

C:\>imgutil /x boot.img dtb
C:\>dtbview dtb > dtb.lst
C:\>whatever-hexedit dtb
C:\>imgutil /r boot.img dtb

041b40              vendor {
041b4c                 compatible = "android,vendor";
041b68                 dev = "/dev/block/platform/soc/c0c4000.sdhci/by-name/vendor";
041bac                 type = "ext4";
041bc0                 mnt_flags = "ro,barrier=1,discard";
041be4                 fsmgr_flags = "wait,verify";
041bfc                 status = "ok";
041c0c              }

Then (in this case) look a bit past 41be4 for the comma in "wait,verify" and change it to a null character.

Edit: You can blank the last bit of a partition easily (and without arithmetic):

For systems with eMMC (i.e. 8 x 512 blocks):

C:\>edl /e /pvendor /s-8
C:\>edl /e /psystem /s-8

For systems with UFS (i.e. 1 x 4096 block):

C:\>edl /e /u /pvendor /s-1
C:\>edl /e /u /psystem /s-1

[Renate]

So, that was the Poke3. The Poke3 did not have a super partition. Things are different now on the Poke5 (and similar generation).

Since super is early mount the fstab for it is not the one that you see in /system/vendor/etc/fstab.emmc but the one in the ramdisk for the boot image, /fstab.emmc. Basically you text edit it to get rid of things that say "avb". Now you don't have to worry that you are modifying system.

You can get your super partition with EDL. You can extract the system logical partition from that with lputil.exe (not quite released yet). By putting the system image on a Linux or Android system you can 1) resize it to 50% bigger or so, 2) revert the "shared blocks" to unshared blocks, 3) delete something big and stupid that you don't use (maybe kreader?), 4) add things useful to you, 5) resize the partition to the same size it was before. Now replace the logical partition in the super partition. It will fit exactly. Then flash super.

[Anghirrim]

Hey, super interesting thanks!!

What do you mean by reverting shared blocks on system partition?

[Renate]

Originally Posted by Anghirrim

What do you mean by reverting shared blocks on system partition?

The logical partitions inside the physical super partition are just snapshots of ext4 physical partitions. You can see inside them with 7zip or mount them as a loop device. A file consists of some number of chained 4 kilobyte blocks. As some of these blocks from different files are all the same they use just one copy. Of course this makes changing anything difficult. So we use the command "e2fsck -E unshare_blocks my.img" to make individual copies of all the shared blocks. This sharing doesn't save you all that much space, but unsharing makes it too big to fit where it was. We could do stuff like resizing the logical partition inside super, but it's easier to find some stuff that we don't need and just delete that. That will give us enough room.

Edit: Also, I just learned why killing apex for adbd bootloops.
Dependencies. You need these in both lib and lib64.

libadbconnection_client.so
libadb_pairing_auth.so
libadb_pairing_connection.so
libadb_pairing_server.so

I have root without Magisk, but I have to recompile sepolicy because I'm running permissive.

[Anghirrim]

Thanks for the explanation!

[Renate]

Ok, it's running better:

C:\>adb shell
Poke5P:/ # id
uid=0(root) gid=0(root) groups=0(root) context=u:r:su:s0
Poke5P:/ # getenforce
Enforcing

It took a bunch of compiling sepolicy and patching super.
I've got to get this down to a simpler procedure.

Edit: I deleted all that useless USB config stuff.
Now the device is always in ADB mode and nothing else.
I can lock it down to specific machines if I like without the chance to authorize.

[Renate]

If you're doing your desktop work on a Windows platform there is something to make life easier.

If you want to manipulate super.img you'll need two different things:
Something to extract the logical partitions.
Something to manipulate the ext4 logical partitions.

Extracting logical partitions is simple, for Windows just use my lputil.exe

Modifying ext4 logical partitons is another thing. For just extracting, 7zip can do that.
But to modify an ext4 partition that is using shared blocks requires Linux. You can send your system partition over to a Linux system (like a Raspberry Pi) and manipulate it there. But that gets tedious transfering 2 GB files back and forth.

(I actually did something similar once with a Linux desktop and a Windows auxilliary system using a USB flash drive that was switched between the two systems.)

Windows has WSL, Windows subsystem for Linux. You can run a regular distribution of Debian or even a desktop Ubuntu. And the good thing is, you can mix all your Windows command and your Linux commands in one makefile (or batch file).

doitall:
# extract
	lputil super.img /x /s0 /psystem sys.img
# mount
	wsl -- e2fsck -f -y sys.img
	wsl -- resize2fs sys.img 550000
	wsl -- e2fsck -E unshare_blocks -y sys.img
	wsl -- e2fsck -f -y sys.img
	wsl -- sudo mount -t ext4 -o loop sys.img mnt
# do your changes here
	wsl -- sudo cp fonts.xml mnt/system/etc/fonts.xml
# unmount
	wsl -- sudo umount mnt
	wsl -- e2fsck -f -y sys.img
	wsl -- resize2fs sys.img 524113
	wsl -- e2fsck -f -y sys.img
# replace
	lputil super.img /r /s0 /psystem sys.img

(550000 and 524113 are numbers specific to my Go6.)

Ok, that seems a bit complicated, but the good part is that you just need to run it. When you find something else to modify, just add another line and rerun it. It beats the heck out of typing individual commands.

In actual use, once you have your system image extracted, unshared and resized you can just mount/mod/umount without going through as many steps. Of course if you screw up you'll need to redo everything. It's mostly the unshare operation that takes the most time.