u/lovelacedeconstruct
Why is LayerNorm needed at inference in transformers ?
Everything I read online mentioned training benefits, but I dont really get why is it done at inference , it takes a negligible portion of the running time so it doesnt matter but I really want to know if there is a reason
Insane memory leak when using coded workflow
Understanding Matrix representation of linear maps
I am having a hard time understanding the matrix representation of linear maps
suppose I want to find a transformation that rotates the bases vectors in R2 90 degrees
such that T(1,0) = (0,1) and T(0,1) = (-1,0)
so we can T(a,b) = T(a(1,0) + b(0,1)) and we know its a linear map so
T(a,b) = aT((1,0)) + bT((0,1)) = aT(1,0) + bT(0,1) = a(0,1) + b(-1,0)
= (-b, a)
alright I am ok till this point, then matrix multiplication is pulled out of nowhere
I don't get whether this defines matrix multiplication or matrix multiplication defines it
what motivated defining the matrix multiplication as applying the linear map to a vector and mapping the basis-images as the matrix's columns
like there was a sneaky failed attempt at the beginning of the course to just throw away matrix multiplication as this mysterious thing and have this be an aha moment where everything falls together but I dont really get it
What is the best way to represent hierarchical data ?
Like for example making a file system where each folder can either have arbitrary number of files and/or folders, and you need to be able to move subtrees around and delete them frequently
I read about hierarchyid in sql server but I dont think its portable or a good idea.
Do I really have to implement the tree operations myself ?
What is the right way to fuzzy search across large tables ?
The way I understand fuzzy search is you have a substring that you want to search for.
instead of using the traditional way of searching for an exact match , you basically assign scores based on certain criterias like for example :
1- matched leading letter +10
2- consecutive match +5
3- unmatched letter -1
4- complete match +20
and so on
(besided the point but quick question can you think of a way to abstract the scoring system so that a user can supply his own ?)
its obviously slower than exact match but very useful when the user have a vague idea of what he wants but dont remember or if there is a spelling mistake.
How virtual functions work !
From what I read online the idea is for each class we create a vtable which in simple terms is an array of function pointers, one entry per virtual function.
Every object carries a hidden pointer (vptr) as its first member pointing to its class's vtable.
Derived classes also get their own vtable with the same layout as the base, but with their overriding implementations swapped in. Since a derived class is a superset of the base, it's always safe to treat a derived object as a base object the memory layout is compatible. So if we point the vptr to the derived class's vtable instead of the base's, any code working through a base pointer will transparently call the derived implementation.
I tried to implement the same idea in C (please its for demonstration this is not production code and nobody should do it I know) and I managed to get the assembly output close
but I have few questions:
1- what is this +16 to the vtable address in the c++ assembly
c -version
mov QWORD PTR [rsp+24], OFFSET FLAT:"dog_vtable"
mov QWORD PTR [rsp+16], OFFSET FLAT:"cat_vtable"
c++ version
mov QWORD PTR [rsp+24], OFFSET FLAT:"vtable for Dog"+16
mov QWORD PTR [rsp+16], OFFSET FLAT:"vtable for Cat"+16
I guess its relevant to this (what does typeinfo here denote?)
"vtable for Dog":
.quad 0
.quad "typeinfo for Dog"
.quad "Dog::speak()"
"vtable for Cat":
.quad 0
.quad "typeinfo for Cat"
.quad "Cat::speak()"
Reflection in C# is amazing !
I just learned about C# reflection, which as I understood it a way to access metadata about your code itself at runtime, like inspect the types and properties
Assembly.GetExecutingAssembly()
.GetTypes()
.Where(t => t.Namespace == "MyNamespace");
you can for example get the current assembly, return every single type defined filtered by required namespace ( classes, interfaces, arrays, values, enumeration, etc..)
now you can walk up the tree, and for each type pick its kind (interface / abstract class / class) and then use GetProperties and GetMethods to obtain the rest of information
You can do alot of things using this information, I made an attempt to translate the information to plantuml syntax and get automatic class diagrams of my code, Its really fun and powerful to mess around and find ways to visualize this information and change it from one state to another
I tried to implement xiaolin wu anti aliased circle algorithm, the way I understood it is the genius part was the Lemma
which basically simultaneously tell you how bright to paint the inner and outer pixel and when to step inward to the next row all in one.
which makes the algorithm incredibly simple you can do a precomputed table with the max possible radius and just query it in a loop which would look something like
while (i > j)
{
j++;
if (D[r][j] > T)
i--; // step inward
float outer = D[r][j] / 255.0f;
float inner = (255 - D[r][j]) / 255.0f;
plot8(..., i, j, inner, color);
plot8(..., i + 1, j, outer, color);
T = D[r][j];
}
now I don't get how to fill this circle ? do I neglect the inner pixel and draw horizontal lines ? it doesnt look right