Recent searches
Search options
Administered by:
Is YaCy.net as a p2p Search Engine part of the #Fediverse? I would guess so. Would be nice to be invited to that party! What do you think?
@orbiterlab
It's been a while since I had a deeper look into the architecture of #YaCy.
To the best of my knowledge, services don't belong to the #Fediverse by virtue of their distribution model, but by the protocols used.
Some services form a connected graph (using OStatus, DFRN, Zot or ActivityPub), while others have no way of connecting to it despite being distributed (e.g. #Matrix or #IPFS).
Do you see possibilities of connecting #YaCy at some future point?
https://upload.wikimedia.org/wikipedia/commons/d/d0/How-the-Fediverse-connects.png
@schmidt_fu I admit that I did not know that a certain distribution model is required to be part of the fediverse, but it is a sufficient strict definition to find out which software can be connected with another. So far YaCy is mainly about index distribution, not content sharing as i.e. ActivityPub is designed. There is also no use of mis-using the protocol for a different purpose, so maybe the answer is "no" so far as we cannot see how a search engine is a producer of content in that sense :(
@orbiterlab
I didn't mean to discourage you, just clarify some concepts.
I do see some possibilities to connect #YaCy to the #Fediverse (though I'm not an expert).
YaCy is strong in search, which is a weakness of Mastodon at least, and it can be leveraged in more than one way, as you've shown with the #SusiAI.
@orbiterlab
Imagine people running their own search engine, along with their own social network on the same node. All the local posts could be immediately indexed, and maybe search could be triggered by ActivityPub direct messages.
There could be #SusiAI chatbots on the #Fediverse for some #YaCy instances as well, at the same time answering queries and enriching the data pool by concise questions.
On a lower level, YaCy peer stats could be posted via ActivityPub, or different event logs.
@schmidt_fu Yes! I also made the loklak project where we scraped tweets from twitter in large quantities avoiding the API, I made a search index for 3 billion tweets already. SusiAI is currently in "deep sleep" but I do not want to give up the project completely but maybe we make a re-start using latest AI tools and cherries from what we did so far. Maybe everything fits together so far and we can do something good for mastodon.
I’m not however sure if activitypub is the correct protocol for doing it. Happy to be convinced otherwise though.
Where I see issues is that I would as a node want to rerank results that others returned, meaning I would want the ranking algorithm sent too, and things like keyword frequencies so I could at least redo TF/IDF or BM25 on top.
When I finish off my experiments with consuming content over activitypub I am going to give this a decent amount of thought.
If you or anyone else is aware of proposals to have search in activitypub please let me know, I’d love to assist.
@boyter @schmidt_fu I dived into the activitypub whitepaper yesterday and found out that it is beautiful and straightforward on one hand, but completely missing a search service/api on the other hand. However, Mastodon has built-in search using elasticsearch as attached service which couldn't be better. I just did not found out where (what api) the search ability is exposed to fediverse clients for distributed search. Can someone please guide me?
@orbiterlab
There are several standards that provide the full functionality only together. A crucial part in AP interop is the #WebFinger protocol, also some services have their own client-server protocol, like Mastodon.
You can read a little about that in the Mastodon Docs, from section "Spec compliance":
https://docs.joinmastodon.org/spec/activitypub/
@boyter
@orbiterlab
While we're talking, I could imagine that @smallcircles would be interested in the integration of distributed social media and distributed web search (maybe with smart agents) - maybe with some further thoughts about #LinkedData and #SOLID apps.
@boyter
@orbiterlab
Ah, I read your question again now, AFAIK the search function of #Mastodon is not exposed via ActivityPub. That why I initially thought about using #YaCy for a distributed search for all consenting instances (there are some who are opposed to being searchable). This would be a unique service at the moment.
Other than that, making YaCy an ActivityPub Relay could be a great way (again, opt-in per instance) to provide the crawler with current URLs.
@boyter
I would love to see multiple search engines peering together over a common set of API’s to produce a federated distributed search. Have yacy work with yioop, and any other random search out there. It could be amazing.