cross-posted from: https://lemmy.world/post/6301281
I’m excited to announce the first alpha preview of this project that I’ve been working on for the past 4 months. I’m initially posting about this in a few small communities, and hoping to get some input from early adopters and beta testers.
What is a DHT crawler?
The DHT crawler is Bitmagnet’s killer feature that (currently) makes it unique. Well, almost unique, read on…
So what is it? You might be aware that you can enable DHT in your BitTorrent client, and that this allows you find peers who are announcing a torrent’s hash to a Distributed Hash Table (DHT), rather than to a centralized tracker. DHT’s lesser known feature is that it allows you to crawl the info hashes it knows about. This is how Bitmagnet’s DHT crawler works works - it crawls the DHT network, requesting metadata about each info hash it discovers. It then further enriches this metadata by attempting to classify it and associate it with known pieces of content, such as movies and TV shows. It then allows you to search everything it has indexed.
This means that Bitmagnet is not reliant on any external trackers or torrent indexers. It’s a self-contained, self-hosted torrent indexer, connected via the DHT to a global network of peers and constantly discovering new content.
The DHT crawler is not quite unique to Bitmagnet; another open-source project, magnetico was first (as far as I know) to implement a usable DHT crawler, and was a crucial reference point for implementing this feature. However that project is no longer maintained, and does not provide the other features such as content classification, and integration with other software in the ecosystem, that greatly improve usability.
Currently implemented features of Bitmagnet:
- A DHT crawler
- A generic BitTorrent indexer: Bitmagnet can index torrents from any source, not only the DHT network - currently this is only possible via the /import endpoint; more user-friendly methods are in the pipeline
- A content classifier that can currently identify movie and television content, along with key related attributes such as language, resolution, source (BluRay, webrip etc.) and enriches this with data from The Movie Database
- An import facility for ingesting torrents from any source, for example the RARBG backup
- A torrent search engine
- A GraphQL API: currently this provides a single search query; there is also an embedded GraphQL playground at /graphql
- A web user interface implemented in Angular: currently this is a simple single-page application providing a user interface for search queries via the GraphQL API
- A Torznab-compatible endpoint for integration with the Serverr stack
Interested?
If this project interests you then I’d really appreciate your input:
- How did you get along with following the documentation and installation instructions? Were there any pain points?
- There’s a roadmap of high-priority features on the website - what do you see as the highest priority for near-term development?
- If you’re a developer, are you interested in contributing to the project?
Thanks for your attention. If you’re interested in this project and would like to help it gain momentum then please give it a star on GitHub, and expect further updates soon!
You must log in or register to comment.
This looks rad. Sadly I don’t have time to pay around with it right now but will def come back to this.
It’s been 2 hours, come back to this.
Slight off-topic but do people not know about Tixati ? It’s a torrent client but it has searchable DHT (by torrent name) and full decentralised forums (with audio streaming, for whatever reason lol) for a long time now.
Except for being closed source it’s pretty interesting. A bit baffled there isn’t more talk about it, or at least about the features.As a people; no, I did not.
Great client but unfortunately for those of us on private trackers it isn’t going to be whitelisted on many of them, or at least the most prominent ones. At least that was the case last I checked, and I gave up on it after.
Until I was reading about this project I had never heard of it. And I would consider myself pretty plugged into torrent news.
deleted by creator
Proprietary?
This is dope! There are a lot more important things on the roadmap, but a qBitTorrent plugin or compatible endpoint would be amazing.
The included docker compose was very easy to use. I was up and running in just a few minutes.
DHT crawling started immediately which was pretty cool to see.
Sometime later this week I’ll try integrating it into my arr stack.
What kind of storage usage can I expect from running this? I would also assume that the database would heavily prefer solid state storage?
Edit: the author answered that question here already: https://lemmy.world/comment/4148270
Is it safe to run this without a VPN if I am just using it to index?
Very nice work, and thanks for sharing it!
I haven’t spent much time digging through the code yet but noted that you’re running your application in the
ENTRYPOINTrather than theCMD. This can make overriding or extending parts of your container difficult, so you may wanna change that.I may try setting this on my k8s cluster… if it can build on aarm64 ;-)
Please, with sugar on top: make this a plugin that can be integrated to qbittorrent or rtorrent.
Try to make your idea reach the seedboxes. This would be great for open-tracker seedboxes!
Looks like a great replacement for rats on the boat.
