MoRA replaces low-rank matrices with a square matrix that gives fine-tuned LLMs higher capacity for learning new knowledge
Amazing to see how clever maths are continuously optimising LLMs. I wish though we were able to leapfrog by moving at larger "learning steps" in our own human mental models.
Amazing to see how clever maths are continuously optimising LLMs. I wish though we were able to leapfrog by moving at larger "learning steps" in our own human mental models.