I’ve been working with real-time vision/VLM inference lately and I’m curious what stacks people here are using.
We’ve been building around the Overshoot API, especially for applications where latency matters ( < 200 ms). Would be interested to hear what others are using for hosted VLM inference vs self-hosting.
Play chess against frontier VLMs. Real-time vision powered by Overshoot.
Play now: VLM Chess
Play chess against frontier VLMs. Real-time vision powered by Overshoot.
Play now: VLM Chess
CLIP vs SigLIP
Before Vision Language Models can perform tasks such as classification or video question and answer, the image or video being passed to the model has to be converted into a representation that the model can ‘understand’ or process.
To do this, VLMs usually use a pretrained vision encoder.
Although the underlying architecture of modern vision encoders is primarily transformer-based, the actual objective the model is learning can vary significantly.
What are encoders?
A vision encoder is responsible for converting images into a numerical representation that VLMs can understand.
Typically, most vision encoders today are built on transformer architecture, in which the model divides an image into patches and transforms each of those patches into a vectorized visual embedding.
After this, many VLMs pass the embeddings to a projector, usually a linear layer or MLP, to map the dimensions of the image to those expected by an LLM.
If most vision encoders share the same model design, what actually makes them different? Rather than model architecture, the significance is in how they are trained.
CLIP
CLIP, or Contrastive Language-Image Pre-training, learns to understand images through pairs of images and text. Its objective is to match similar images and captions by ‘pulling them closer together’, while simultaneously repelling incorrect image-caption pairs.
Training mainly relies on a ‘two tower’ system. CLIP will typically have a pretrained vision encoder, such as a ViT, as well as a pretrained text encoder. The model passes an image through the ViT and produces an associated vector embedding, while the caption is passed to the text encoder to get a corresponding text embedding.
Given these pairings, the model therefore creates a similarity matrix which compares every image embedding with every text embedding.
Each cell within this matrix contains a cosine similarity between the image and text pairing. Mathematically, cosine similarity is the dot product of two vectors divided by the product of their lengths. More simply, it measures the cosine of the angle between two vectors in a high-dimensional embedding space. Vectors that are more semantically aligned will be ‘closer together’, have a more acute angle between them, and consequently have a higher cosine similarity.
CLIP then applies contrastive learning across this matrix. At a high level, contrastive learning here is similar to categorical cross entropy across both the rows and columns of the matrix. Using softmax, the model looks to assign the highest probability to the matching image-text pair, as well as the matching text-image pair.
CLIP is powerful because it shifts learning from simple labels toward greater semantic understanding and allows for zero-shot classification, including on classes it was not explicitly trained to classify.
At the same time, though, CLIP also introduces a particular structural problem. Examples compete against one another within the training batch. What if there are multiple captions within a batch that also reasonably match the image?
SigLIP
SigLIP, or Sigmoid Loss for Language-Image Pre-training, retains many similar characteristics to CLIP. Similar to CLIP, SigLIP has both an image and text encoder, embedded representations of both text and image, and similarity scores mapped to a similarity matrix.
However, the difference between the two lies in the loss function.
CLIP learns similarities between images and texts by applying softmax across a batch, causing potential matches to compete with one another. For SigLIP, instead of having this global normalization, it examines each image-caption pair as an independent binary prediction.
By applying a sigmoid function to each pair’s score, the model estimates whether the image and text match.
Rather than phrasing the objective as:
Out of these options, which specific text describes this visual?
SigLIP effectively poses a different question:
Is this particular image-text pairing a valid match: true or false?
While this shift in perspective might seem marginal, it fundamentally redefines the nature of the optimization task.
Because SigLIP does not require the softmax normalization used by CLIP, its training objective can scale more efficiently across large distributed systems. It also removes the requirement that every example participate in one shared normalization operation.
A lot of modern VLMs still rely on pretrained CLIP-style vision encoders, which are primarily trained to align images with text descriptions.
That seems like a strong foundation for semantic recognition, but I wonder how sufficient it is for tasks that require precise counting, spatial relationships, fine-grained attributes, or other forms of visual reasoning that caption matching may not explicitly encourage.
Do you think the vision encoder is becoming a bottleneck for modern VLMs, or is the limitation mostly elsewhere in the system?
Hey! Check out this blog I wrote on VLM mirages. Would love to hear any thoughts or feedback!
I wrote an introductory overview of visual token pruning in vision-language models, covering why visual tokens can be redundant and how approaches such as PuMer and FastV reduce computation.
I’d really appreciate feedback on the technical explanations or anything important I may have missed. Thank you!