Talk:MIPS64

Please keep

Hi,

please, please, please don't delete the page or the archived binaries etc.

Hardware

Availability

There's 100000's of SmartNICs based on this CPU that are being obsoleted at various cloud providers. They're very cheap to obtain, I've seen them as low as $6 for a 25G model with 16Cores/16GB Ram, and even the more expensive offers are a good value. There's even an OpenWRT user who just was silly about it and bought 50 for $2 each.

So the availability has changed.

Good models

CN6640 (Dual/Quad 10G, 12 Core?, 8/16GB via DDR3 SODIMM)
CN2300/CN7322 (Dualport 10G / Dualport 25G, 12 / 16 Core, 8GB)
CN7890 (Dualport 40G, 24/32 Core, 32GB)

Some even have 2x SATA 3.0, some USB 3.0 :-) Others have PCIe, even host support, so you don't need a PC around them to run.

There's also

embedded boards

by Oracle for a "boarder network controller" (4x10G has 2x32 Core/64GB)
by Kontron (32Core also), as AdvTCA / PMC industry standard formats
There's official dev boards, where you'd want the ones between CN68xx and CN78xx one mostly
There's boards that are more custom with undocumented connectors but come with insane performance capabilities (AdvanTech Packetarium, usually 2x10g or 4x10g but also often with FPGAs onboard etc.)

Shiny Boxes

A notable option is the old UniFi USG-X-8, which has been overtaken by the UDM-Pro in many setups. In those old gateways you'll find... a 16-core CN7360 and 16GB ram. It has been tested to hold 32GB. So those are also fancy build hosts.

Almost all older UniFi devices and 'security gateways' are MIPS based, but only specific models have so many cores that they'll be really interesting. All Ubiquiti kernel modules are proprietary (/lib/modules/<kernel>/extra), that includes the fan control.

Untested

SonicWall also used MIPS/Octeon CPUs. It appears the high-end line could be cheap and useful. It's unclear whether these are OcteonII or OcteonIII

Model: 9200 9400 9600 E10200 E10400 E10800 Processing cores: 24 32 32 24 48 96

The 9200-9600 series are 1U (black) The E10k series are 4U or 5U (silver, so in fact shiny)

Not so good models

Many other network devices come with those chips, but are not expected to be trivial to turn into a dev system. Notable examples are i.e. the 3U Palo Alto boxes that have at least 3 Octeon processors along with an x86_64 control plane node.

Complexity

It's complicated

The complexity of course hasn't changed, much. it's still a large undertaking to get the cards to run your own OS, and their OSS history is certainly not a shining light. There's no public user manual for the cards, a SDK is NDAed and you need to pick up scraps to get it working. I'm failing at making it work. But I fancy to think I'm failing forward.

Much of the complicated stuff is just toil/churn of kicking out the vendor toolchain - and they're even helping on that end. But it is much nonetheless.

Accordingly, it makes sense to not have the architecture supported by default.

MII

There's a very bad interface into ethtool, meaning you can pretty much forget setting link speeds etc. That ball has been dropped a decade ago. Interface stats are available via /sys

FPU

The Cavium Octeon III stuff generally has 'hardfloat' aka FPUs, but Alpine doesn't yet use that. I'm working on rebuilding everything in a mips64-octeon-musl-hf target or so. It's not horribly bad and there's no actual technical blockers to it. The wiki says that alpine didn't manage to do link in hardfloat mode, but I think this can be solved by just throwing many more CI toolchains at the problem.

Endianness

There's a constant forth-and-back in the communities around this. The cards can boot in both little and big endian modes and the SDK allows building both ways. I feel there's no stable result until both modes build well, but I'm gonna stick with big endian as they is the alpine 3.14 setup and the demo toolchains default to it.

SATA

The SATA interfaces are unstable with many modern SATA 3/3.1 disks. Intel SSDs have worked like a charm for me. (I tested S3500-S3520) At least per the kernel config, SATA port multipliers are supported. (untested)

LBU / diskless mode

The Octeon II and better totally support using named memory blocks, as such boot to RAM is very feasible. There's discrepancies in how the memory blocks are passed in examples and in how alpine had expected to use them. This is only a problem of semantics as far I could see, but will need some help Alpine devs to solve. LBU mode is perfect for those devices that already have an eMMC / NAND / PCIe SSD. The eMMC does not read super fast, reading a compressed apkovl would improve matters for this. Up to a certain size, it could also be loaded via PCIe and simplify the boot process a lot.

architecture revival feasible

Once enough of them have leaked out, it will be very feasible for the community to contribute and revive the port. At extremely slow pace, the vendor has upstreamed most parts for both Linux and U-Boot and as such, the situtation is less horrible than a few years ago. The gaps are getting smaller and easier to step over. As such it is a really nice tinkering platform, and in some sense it's even more open, or at least more open to be investigated than the easier to obtain alternatives like Ubiquiti routers.

The only holdup is the sparse documentation and if we just have a clear warning then it should be hopefully OK to keep all the traces of this architecture support around?

leftover work

From what I read on the page, the situation around Go is very bad; some more things seem to involve autotools (a nightmare solo but tolerable for many) and, lastly, the FPU. I would guess only the FPU part is really something where it matters if it gets attention or not. I gotta check what the situation on Debian look(s|ed) like, but it's also something that could become less of a problem as progress happens.

Other uses

To be clear, the cards work in Alpine as normal 25G NICs, but it would be a lot more fun to also run Alpine ON them.