Shouldn't all ActivityPub servers implement all objects and leave it to the client?

I've been working with some smart people on something to hopefully become a "federated account" that can be used with any service, and is 100% compatible with OIDC, so its easy for systems to implement as the authentication vehicle: https://fedid.me

Just presented it at IIW and interest is building thus far, so my hopes are high 😉

There have been some platforms that combined form factors like Friendica and mbin

For clarity, it's not Lemmy that uses 'Article'. I can't remember what does, friendica maybe?
Lemmy uses 'Page' for posts, and 'Note' for comments.
Mastodon uses 'Note' for both, with 'inReplyTo' used to distinguish whether Lemmy would call it a 'Page'. It uses 'Question' for polls.
Pixelfed also uses 'Note', with an 'Image' type attachment (I thinks Loops is similar, just with a 'Video' type attachment).
PeerTube uses 'Video'.
Funkwhale uses 'Album' and 'Playlist'
CastoPod uses 'PodcastEpisode'

There's no one universal ActivityPub server because the Fediverse is based on a broken promise: i.e. that you should be able to use whatever service, to interact with whatever other one. You very often can't, because ActivityPub hasn't been implemented by each platform as some universal thing, it's been co-opted by each to serve it's own purposes. Lemmy best federates with other Lemmy instances, following Lemmy's way of doing stuff, but a good chuck of the Fediverse follows a different model, and is receiving their activity and quietly discarding it because it doesn't know what to do with it. If all the Lemmy instances suddenly chose to use a different protocol than ActivityPub, most people wouldn't notice the difference.

The only problem that I see with this is that images and especially videos take up vastly more space than plain text.

And does ActivityPub include client APIs? I haven't looked at it, but my completely unfounded impression has been that it only handled server to server communication.

One (small) issue is that it assumes there *is* a client outside of the server. Most server apps have both a web UI and an API, and it's even possible to build an ActivityPub server with no client API at all (microblog.pub was like that, IIRC). I think the ActivityPub C2S API was intended for servers that worked just like you're describing, but IDK if anyone ever really implemented it properly.

No. The various Fediverse server applications are too different in how they work.

First of all, it isn't all just about object types. The architecture of various Fediverse server applications is vastly different, including how they handle objects, and how they distribute them.

For example, on Mastodon, a thread is just a loose string of posts and more posts which, technically, are identical in properties. Mastodon doesn't know conversations, and Mastodon doesn't know groups. You receive the posts from those whom you follow plus, by default, the posts that mention you.

Friendica does know conversations, and it knows groups because it has them implemented. On Friendica, a thread is one (1) post plus comments, just like on Facebook or on blogs. You receive the posts from those whom you're connected with, but not their comments on other people's posts. Plus, you receive all comments on posts from those whom you're connected with. Receiving posts from those whom you've mentioned is optional but off by default AFAIK.

Forte is like Friendica, but with nomadic identity. That obviously isn't a client thing.

Hubzilla and (streams) are like Forte, but with wholly different protocols that were made for nomadic identity in the first place and with ActivityPub as an optional extra.

Lemmy, Mbin and PieFed are all about conversations and groups. You literally can't follow Lemmy users (something that Mastodon users will never understand), you can only follow Lemmy communities (something that's totally alien to many Mastodon users).

There are many more differences.

Mastodon's HTML sanitiser that rips out most text formatting is on the server side AFAIK. If you make Mastodon the gold standard, say buh-bye to numbered lists, horizontal lines, tables etc. (And I'm not kidding, there are places in the Fediverse that support these. In posts.)

Character limits are server-side. Since the huge majority of Fediverse users and many Fediverse devs think the Fediverse was made as a Twitter replacement, they also think that there has to be an arbitrary character limit, otherwise it wouldn't be microblogging, right? Welll, then Friendica, Hubzilla, (streams) and Nomad can wave good-bye to their unlimited character counts and 100,000+-character posts.

Filters are server-side. And they work vastly differently on different Fediverse server apps. Some import filtered content and then delete it. Others reject it.

Permissions are server-side. Permissions are absolutely essential and integral parts of Hubzilla, (streams) and Forte, but the entire rest of the Fediverse doesn't even know they exist. Of course, it'd be great if everything down to mastodon.social implemented the (streams)/Forte permissions system, but it'd completely overwhelm those who came to mastodon.social in search of Twitter without Musk.

Another feature that Friendica and Hubzilla could kiss good-bye if there was only one unified server backend are multiple profiles per account. Speaking of which, it's farewell to multiple channels (identities, like accounts everywhere else) on one account/login for Hubzilla, (streams) and Forte. Unless everything else is willing to implement both.

Lastly, Hubzilla has absolute literal shit-tons of features on top of even Friendica. Both have built-in file spaces, but Hubzilla has one with WebDAV connectivity (as do (streams) and Forte). Both have federated event calendars, but Hubzilla also uses it as a frontend for its built-in CalDAV calendar server (which is headless on (streams) and Forte). Hubzilla, (streams) and Forte have an optional CardDAV addressbook server. Hubzilla also has optional stuff like non-federating long-form articles, "cards" that work similarly, a simple built-in wiki engine for multiple wikis per channel with multiple pages each, support for simple webpages (the official Hubzilla website is on a Hubzilla channel) and so forth. I'm not even remotely kidding with any of this.

If you want to unify Fediverse servers, they'd all have to become Hubzilla, but with nomadic ActivityPub.

While I think fragmentation can grow into being a problem, trying to standardize things too much can be problematic too, as the developers would be bloating the software for features that the community may use very little, as well as, by consequence of the bloating, the devs being either limited to a design that needs to take into account the quirks of all object formats, or to make some frankenstein monster design to include those different formats.

A more reliable path, I think, is what Kbin (RIP) and its successor Mbin do, to have a section for articles and one for notes. While it's still more load on the developers and the servers, at least it shouldn't be as much as having to make sense of multiple formats together, since the two sections don't directly interfere with each other. This, on a final point, is, to my understanding, and with their respective proportions, what happens with the Linux family of operating systems, where it's also pretty fragmented, but every once in a while a way to put two different environments together appear, like Wine and Xfce translating Windows and QT5 programs, or AppImage and Flatpak trying to be as universal as possible by depending on as little default dependencies from the host system as possible.

This is what I also think, would make the #fediverse much more viable this way.

Imho, ActivityPub is a bad protocol that tries to accomplish everything, and ends up being bad at all of it. The spec is also ambiguous in a lot of areas. And major implementations don’t always follow the spec. All in all, it’s a miracle the fediverse even works as well as it does.