• 10 Posts
  • 2.83K Comments
Joined 2 years ago
cake
Cake day: March 22nd, 2024

help-circle

  • I think this is what “want an EV, research it heavily” folks tend to end up with, where Teslas leans more “want an EV, don’t research it quite as heavily” crowd.

    Not that either brand makes bad cars.

    I just mean the Hyundai/Kia EVs are less “sexy” on the surface. Their charge and range is lower, but if you look into it, it’s an engineering compromise for faster charging. Hyundai isn’t a very “glamorous” brand, but their recent cars have been excellent, and if you actually look into Tesla, well… gestures at X. Lower end Teslas are priced somewhat attractively and tend to have better resale value than Hyundai at the moment, and it’s quite rational to assume “well, they’ve been making EVs longer.” But if you research it, Teslas have a more hidden costs over time, and even though some systems like the heatpumps are less refined than Teslas, Hyundai engineered the heck out of the car.



  • Chromium can’t do absolutely anything they want though. Anything drastic would break compatibility with Safari and FF.

    …But if they do, if they actually get effectively 100% market share, think about it:

    • They could close Chromium’s source, and kill off 3rd party forks as a “security risk.”

    • They could literally disable adblocking, not just “softly” like the Manifest V3 switch did.

    • Similarly, they could implement custom ad and tracking APIs that are effectively unblockable.

    • They could arbitrarily block web pages, or redirect them, purely for Alphabet’s interests.

    • They could mandate that sites they use have to use Chrome, to shut out other browser efforts.

    • They could encourage similar behavior from pages depending on Google ad revenue.


    If all this sounds tin foil hat-ish (and I know it does), consider Google’s past behavior over the decade.

    …Is it that different from what they’ve already been doing? To virtually no public pushback?


    And this is why Safari and FF are so important. As long as they exist, that cant happen, but they’re getting dangerously close to market irrelevance.





  • Actually one addendum/correction:

    You can use your integrated graphics to render the desktop. Plug your monitor into the UHD graphics instead of your Nvidia card.

    This saves a notable about of VRAM you can use to fit more context, and speeds up inference a bit too. It also lets you configure models closer to your VRAM limit, as you no longer have the variability of apps randomly taking it.


  • The best you could run on that system, I believe, is Step 3.7 via ik-llama.cpp.

    You don’t want, nor need, the iGPU. Unfortunately nothing is really optimized to use it for hybrid inference, and it’s not very powerful anyway. You’re better off with a CUDA/CPU backend.

    How it would work in practice is:

    • The “dense” parts of a MoE model like Step stay on the GPU.

    • The big, sparse parts stay in CPU RAM.

    • For prompt processing, everything is done on the GPU since the penalty for going over the PCIe bus isn’t too bad.

    • For text generation (generating the actual output), your CPU cores run the sparse part. This is fine. It’s not nearly as slow as you’d think, as this part is really compute lite and AVX accelerated.


    The tricky part is finding appropriate models and quantizations for your setup. You’re looking at something in the 100-170B range, basically, with the experts quantized to 3-4 bits and the dense weights at 6-bit or higher.

    I can look around later, if you’re interested, but I haven’t kept up with model releases in this range other than the Stepfun series.


    On a separate note, I wouldn’t containerize ik_llama.cpp.

    It just wastes precious RAM and compute. It has basically no dependencies, so it’s not really a security risk, and there isn’t much good that bundles models with it.

    Now, the UIs. Containerize those. Those have tons of dependencies, so it’s easier and safer.




  • People never really cared about information hygiene.

    They tried to teach it at the “dawn of the internet for the average person” back then. The lesson was supposed to be to follow Wikipedia’s links, which do cite primary sources.

    This is true.

    But it turns out, no one cares. Schools failed. People fundamentally want validation of what feels good, and social media took full advantage of this.