Home
bitmagnet.io

cross-posted from: https://lemmy.world/post/6301281

I’m excited to announce the first alpha preview of this project that I’ve been working on for the past 4 months. I’m initially posting about this in a few small communities, and hoping to get some input from early adopters and beta testers.

What is a DHT crawler?

The DHT crawler is Bitmagnet’s killer feature that (currently) makes it unique. Well, almost unique, read on…

So what is it? You might be aware that you can enable DHT in your BitTorrent client, and that this allows you find peers who are announcing a torrent’s hash to a Distributed Hash Table (DHT), rather than to a centralized tracker. DHT’s lesser known feature is that it allows you to crawl the info hashes it knows about. This is how Bitmagnet’s DHT crawler works works - it crawls the DHT network, requesting metadata about each info hash it discovers. It then further enriches this metadata by attempting to classify it and associate it with known pieces of content, such as movies and TV shows. It then allows you to search everything it has indexed.

This means that Bitmagnet is not reliant on any external trackers or torrent indexers. It’s a self-contained, self-hosted torrent indexer, connected via the DHT to a global network of peers and constantly discovering new content.

The DHT crawler is not quite unique to Bitmagnet; another open-source project, magnetico was first (as far as I know) to implement a usable DHT crawler, and was a crucial reference point for implementing this feature. However that project is no longer maintained, and does not provide the other features such as content classification, and integration with other software in the ecosystem, that greatly improve usability.

Currently implemented features of Bitmagnet:

  • A DHT crawler
  • A generic BitTorrent indexer: Bitmagnet can index torrents from any source, not only the DHT network - currently this is only possible via the /import endpoint; more user-friendly methods are in the pipeline
  • A content classifier that can currently identify movie and television content, along with key related attributes such as language, resolution, source (BluRay, webrip etc.) and enriches this with data from The Movie Database
  • An import facility for ingesting torrents from any source, for example the RARBG backup
  • A torrent search engine
  • A GraphQL API: currently this provides a single search query; there is also an embedded GraphQL playground at /graphql
  • A web user interface implemented in Angular: currently this is a simple single-page application providing a user interface for search queries via the GraphQL API
  • A Torznab-compatible endpoint for integration with the Serverr stack

Interested?

If this project interests you then I’d really appreciate your input:

  • How did you get along with following the documentation and installation instructions? Were there any pain points?
  • There’s a roadmap of high-priority features on the website - what do you see as the highest priority for near-term development?
  • If you’re a developer, are you interested in contributing to the project?

Thanks for your attention. If you’re interested in this project and would like to help it gain momentum then please give it a star on GitHub, and expect further updates soon!

  • taaz@biglemmowski.winEnglish
    131·
    1 year ago

    Slight off-topic but do people not know about Tixati ? It’s a torrent client but it has searchable DHT (by torrent name) and full decentralised forums (with audio streaming, for whatever reason lol) for a long time now.
    Except for being closed source it’s pretty interesting. A bit baffled there isn’t more talk about it, or at least about the features.

  • TheOneCurly@lemmy.theonecurly.pageEnglish
    7·
    1 year ago

    The included docker compose was very easy to use. I was up and running in just a few minutes.

    DHT crawling started immediately which was pretty cool to see.

    Sometime later this week I’ll try integrating it into my arr stack.

  • frozen@lemmy.frozeninferno.xyzEnglish
    7·
    1 year ago

    This is dope! There are a lot more important things on the roadmap, but a qBitTorrent plugin or compatible endpoint would be amazing.

  • Daniel Quinn@lemmy.caEnglish
    4·
    1 year ago

    Very nice work, and thanks for sharing it!

    I haven’t spent much time digging through the code yet but noted that you’re running your application in the ENTRYPOINT rather than the CMD. This can make overriding or extending parts of your container difficult, so you may wanna change that.

    I may try setting this on my k8s cluster… if it can build on aarm64 ;-)

  • privadesco@feddit.nlEnglish
    3·
    1 year ago

    Please, with sugar on top: make this a plugin that can be integrated to qbittorrent or rtorrent.

    Try to make your idea reach the seedboxes. This would be great for open-tracker seedboxes!