Obviously there’s not a lot of love for OpenAI and other corporate API generative AI here, but how does the community feel about self hosted models? Especially stuff like the Linux Foundation’s Open Model Initiative?

I feel like a lot of people just don’t know there are Apache/CC-BY-NC licensed “AI” they can run on sane desktops, right now, that are incredible. I’m thinking of the most recent Command-R, specifically. I can run it on one GPU, and it blows expensive API models away, and it’s mine to use.

And there are efforts to kill the power cost of inference and training with stuff like matrix-multiplication free models, open source and legally licensed datasets, cheap training… and OpenAI and such want to shut down all of this because it breaks their monopoly, where they can just outspend everyone scaling , stealiing data and destroying the planet. And it’s actually a threat to them.

Again, I feel like corporate social media vs fediverse is a good anology, where one is kinda destroying the planet and the other, while still niche, problematic and a WIP, kills a lot of the downsides.

  • fruitycoder@sh.itjust.works
    link
    fedilink
    arrow-up
    1
    ·
    1 month ago

    The splitting is 80% of the cool factor for me. Rather than bog down the one node that can handle those cooler models, and have more contribution opportunities.

    I wonder honestly if a petals network could be a target host on horde lol

    • brucethemoose@lemmy.worldOP
      link
      fedilink
      arrow-up
      1
      ·
      edit-2
      1 month ago

      The problem is that splitting models up over a network, even over LAN, is not super efficient. The entire weights need to be run through for every half word.

      And the other problem is that petals just can’t keep up with the crazy dev pace of the LLM community. Honestly they should dump it and fork or contribute to llama.cpp or exllama, as TBH no one wants to split up LLAMA 2 (or even llama 3) 70B, and be a generation or two behind for a base instruct model instead of a finetune.

      Even the horde has very few hosts relative to users, even though hosting a small model on a 6GB GPU would get you lots of karma.

      The diffusion community is very different, as the output is one image and even the largest open models are much smaller. Lora usage is also standardized there, while it is not on LLM land.

      • fruitycoder@sh.itjust.works
        link
        fedilink
        arrow-up
        1
        ·
        edit-2
        1 month ago

        I guess to me be able to serve the 408b model even though I’m on a laptop is just awesome to me.

        Also I saw Lora was an option for Petals but I haven’t messed with it at all.