Why is LayerNorm needed at inference in transformers ?

Everything I read online mentioned training benefits, but I dont really get why is it done at inference , it takes a negligible portion of the running time so it doesnt matter but I really want to know if there is a reason

reddit.com
u/lovelacedeconstruct — 4 days ago

Understanding Matrix representation of linear maps

I am having a hard time understanding the matrix representation of linear maps

suppose I want to find a transformation that rotates the bases vectors in R2 90 degrees

such that T(1,0) = (0,1) and T(0,1) = (-1,0)

so we can T(a,b) = T(a(1,0) + b(0,1)) and we know its a linear map so

T(a,b) = aT((1,0)) + bT((0,1)) = aT(1,0) + bT(0,1) = a(0,1) + b(-1,0)

= (-b, a)

alright I am ok till this point, then matrix multiplication is pulled out of nowhere

I don't get whether this defines matrix multiplication or matrix multiplication defines it

what motivated defining the matrix multiplication as applying the linear map to a vector and mapping the basis-images as the matrix's columns

like there was a sneaky failed attempt at the beginning of the course to just throw away matrix multiplication as this mysterious thing and have this be an aha moment where everything falls together but I dont really get it

reddit.com
u/lovelacedeconstruct — 1 month ago
▲ 14 r/dotnet

What is the best way to represent hierarchical data ?

Like for example making a file system where each folder can either have arbitrary number of files and/or folders, and you need to be able to move subtrees around and delete them frequently

I read about  hierarchyid in sql server but I dont think its portable or a good idea.

Do I really have to implement the tree operations myself ?

u/lovelacedeconstruct — 2 months ago
▲ 1 r/csharp

What is the right way to fuzzy search across large tables ?

The way I understand fuzzy search is you have a substring that you want to search for.

instead of using the traditional way of searching for an exact match , you basically assign scores based on certain criterias like for example :

1- matched leading letter +10
2- consecutive match +5
3- unmatched letter -1
4- complete match +20
and so on
(besided the point but quick question can you think of a way to abstract the scoring system so that a user can supply his own ?)

its obviously slower than exact match but very useful when the user have a vague idea of what he wants but dont remember or if there is a spelling mistake.

reddit.com
u/lovelacedeconstruct — 2 months ago

How virtual functions work !

From what I read online the idea is for each class we create a vtable which in simple terms is an array of function pointers, one entry per virtual function.

Every object carries a hidden pointer (vptr) as its first member pointing to its class's vtable.

Derived classes also get their own vtable with the same layout as the base, but with their overriding implementations swapped in. Since a derived class is a superset of the base, it's always safe to treat a derived object as a base object the memory layout is compatible. So if we point the vptr to the derived class's vtable instead of the base's, any code working through a base pointer will transparently call the derived implementation.

Illustration

I tried to implement the same idea in C (please its for demonstration this is not production code and nobody should do it I know) and I managed to get the assembly output close

Compiler Explorer

but I have few questions:
1- what is this +16 to the vtable address in the c++ assembly

c -version

        mov     QWORD PTR [rsp+24], OFFSET FLAT:"dog_vtable"
        mov     QWORD PTR [rsp+16], OFFSET FLAT:"cat_vtable"

c++ version

        mov     QWORD PTR [rsp+24], OFFSET FLAT:"vtable for Dog"+16
        mov     QWORD PTR [rsp+16], OFFSET FLAT:"vtable for Cat"+16

I guess its relevant to this (what does typeinfo here denote?)

"vtable for Dog":
        .quad   0
        .quad   "typeinfo for Dog"
        .quad   "Dog::speak()"
"vtable for Cat":
        .quad   0
        .quad   "typeinfo for Cat"
        .quad   "Cat::speak()"
u/lovelacedeconstruct — 2 months ago
▲ 240 r/csharp

Reflection in C# is amazing !

I just learned about C# reflection, which as I understood it a way to access metadata about your code itself at runtime, like inspect the types and properties

Assembly.GetExecutingAssembly()
                .GetTypes()
                .Where(t => t.Namespace == "MyNamespace");

you can for example get the current assembly, return every single type defined filtered by required namespace ( classes, interfaces, arrays, values, enumeration, etc..)

now you can walk up the tree, and for each type pick its kind (interface / abstract class / class) and then use GetProperties and GetMethods to obtain the rest of information

You can do alot of things using this information, I made an attempt to translate the information to plantuml syntax and get automatic class diagrams of my code, Its really fun and powerful to mess around and find ways to visualize this information and change it from one state to another

https://preview.redd.it/tknpd06nea4h1.png?width=1340&format=png&auto=webp&s=4237bd4184e8910f9de107254c7ddf3010cfe978

reddit.com
u/lovelacedeconstruct — 3 months ago

I tried to implement xiaolin wu anti aliased circle algorithm, the way I understood it is the genius part was the Lemma

https://preview.redd.it/pjn53n68bqzg1.png?width=1488&format=png&auto=webp&s=064c6b488103dc38bb42be78125ac0a9e41ab5de

which basically simultaneously tell you how bright to paint the inner and outer pixel and when to step inward to the next row all in one.
which makes the algorithm incredibly simple you can do a precomputed table with the max possible radius and just query it in a loop which would look something like

while (i > j)
{
    j++;                 

    if (D[r][j] > T)       
        i--;                 //    step inward

    float outer = D[r][j]         / 255.0f;   
    float inner = (255 - D[r][j]) / 255.0f;   

    plot8(..., i,     j, inner, color);   
    plot8(..., i + 1, j, outer, color);   

    T = D[r][j];            
}

now I don't get how to fill this circle ? do I neglect the inner pixel and draw horizontal lines ? it doesnt look right

reddit.com
u/lovelacedeconstruct — 4 months ago