Learning in Public
Learning in public is advised by many; more so in this AI age where built things are cheap(er) but true understanding is hard(er) to come by. Consider the excellent advice in Vlad's blogpost on how to break into Frontier labs and one of them is to outwardly signal that you truly understand the things that you need to. To make sure of that is to learn publicly, show your trials and tribulations publicly or at-least have a public record of it so that people can readily look at that and discern that you have actually done your homework.
I have long been sympathetic of this idea. Learning in public is also an act of creation, an outward generating act that transcends the inward act of learning or consumption. Therefore, I am trying to stream my learning sessions on youtube. Of course, it's very informal, messy and cringey but I am persevering through it. I haven't told anyone of this explicitly and I don't intend to. I have streamed 2 sessions as of writing and they have been to a cumulative audience of 0. It's a bit weird speaking to myself (and some fictional audience that may or may not exist in the future) in my room as I work through the problems on TensorTonic and implement the "Attention is all you need" Paper.
Today, I had a difficult time trying to wrap my head around the positional encoding section. I had imagined the positional encoding to be a more simple injection of some positional sense of the sequence ordering but the theory behind how it was proposed in the paper is a lot neater and more profound. I want to study it more carefully and then write a longer article on it.