Видео

Great video and a good explanation. Thanks for your hard work on this amazing video!!

Very clear 👏👏

Had to go back and rewatch a section after I realized I'd been spacing out staring at the coffee bean's reactions.

Thanks!

Most insightful explanation I have found on this subject so far. I was looking for it for days... Thank you! Keep going, you rock!

Thanks!

Hey letitia. I think you should consider changing the logo. It kinda looks like a turd

Super informative and helpful! Thanks a lot!

Excellent video

BEST of BEST Explanation. 1) Visually, 2) intuitively, 3) by numerical examples. And your English is better than native for Foreigners to listen.

Awesome! Thanks for sharing!

but is the benefit worth the potential negatives? surely, there are tradeoffs here.

Thanks for the video. So, is Weaviate a database service that is leveraging the LLM to generate natural language queries to access the data? So, basically rather than hiring an SQL programmer, you just feed Weaviate the data (somehow), and use the LLM to gen the queries, which are then fed back to Weaviate so it can provide the query results?

nice video. I am from India. I am working in the AI field. do you have any remote opportunities for me?

Thanks, first video on probing that makes sense to me. But just wondering if probing is just for diagnostics or it's actual option for fine tuning in production?

☺

Great video! 🤌

Awesome!!

Cool! I love these hands-on videos with code examples! Thanks for sharing the notebook!

One of the selling points of Lora is to be able to mix and match the A and B matrices for different fine-tuning runs, without having to keep the weights of the full model if they are available elsewhere. Here it seems you have to save the entire model, so this is a big tradeoff compared to Lora and derivatives.

I am afraid you did not fully understood the mechanism of Information geometry behind UMAP and how the KL-divergence acts as the "spring-dampener" mechanism. Keenan Crane and Melvin Leok have great educational materials on the topic.

yes

8:39 What I meant to say, is VQ-VAE, not VQGAN! Thanks to Luis Cunha for spotting this!

Concept of rank of a matrix,tauught in such an effective way

I love this paper. The solution is elegant.

thanks! also very cute merch!

I ran into trouble trying to Fine Tune a Swin model with LoRA. That type of model isn’t supported yet for LoRA. I wonder if it’ll be the same for GaLoRA

OMG, such a amazing video to explain Positional Embedding....

Thank you so much! Super helpful.

excellent explanation!

congrats leititia!!

The size of a LoRA (on disk) is a fraction of the size of the model it's applied to. GaLore looses this benefit by updating the weights. If you have use for many fine tunes of a single model, two GaLore fine-tunes will take more space than ten LoRAs (depending on rank, to be fair). I assume they don't mention this very significant tradeoff, as you don't mention it in the video. That seems like a dishonest comparison, if that's the case.

Great video!

Can info shed during the lossy compression process be set aside in a non memory fashion for retrieval? Thinking out loud. Not always helpful. But state mapping would be interesting in the training process as well as post.

Great video! Got a follow up question: what kind of finetuning is the finetuning provided by the openai API, where it finetunes a model based on a training set of Q&A pairs provided by the user?

Congratulations Frau Doktor.

😘 LOL! 礼 Stay for the EPIC lipstick! M4D L0V3!

Holy frick what a perfectly concise video on galore! There is a GitHub implementation of the research from the paper, they are currently working on a multigpu implementation. I too am curious how well things scale up to modern and larger llms, and have a multigpu rig I want to test it out on.

Congratulations

As far as i get it, i determine the gradients G and and also a low rank component P. That component allows me to „shrink“ the gradient matrix G i calculate at every step to down to R before applying it to W. So i do not save compute while calculating the gradients, but while applying and saving (as momentum or such) them?

I never thought I'd say this, but I'm actually excited to learn about efficient training methods for deep learning models on consumer GPUs. Who knew running out of GPU memory could be so... enlightening? Thanks for explaining LoRA and Galore in a way that doesn't make my brain hurt (too much). Now, if you'll excuse me, I have some large language models to train or at least, try not to run out of GPU memory

Thank you a lot for your explain! may i know which software you using for making animation in your video? thanks!

Congrats on the PhD! :D

Do I understand it correctly that it can't work with quantization and the model must fit in memory in 16bit?

One important thing to note is that while this technique is more memory efficient it's also 17% slower in the setup the authors use. That's a pretty big deal, especially for pretraining.

I always liked Galor(e), though I might be biased.

Congrats on your PhD! Could you use 'r' as a hyperparameter during pretraining as well? e.g. start pretraining with low r and gradually increase it as more precision is needed? I don't think it could do that much since gains are already very high at the start.