u/WillingnessQuick5074

r/vectordatabase r/Rag r/WordpressPlugins r/HostingReport r/u_WillingnessQuick5074 r/LangChain r/BlackboxAI_ r/drupal r/semanticweb r/Chatbots r/mcp r/apache r/nginx r/PHP

▲ 14 r/nginx+1 crossposts

Open source traffic analyser that measures engaged time properly and separates bots from people, with the evidence attached.

I run Opensolr. This is a thing we built for ourselves and then open sourced, so the disclosure is up front.

We had a problem we could not answer with any analytics product we were paying for. The dashboards said traffic was healthy. The logs said something else. So I wrote the thing that settles it.

What we found on our own site

Roughly half of the sessions our analytics reported carried a hard bot signature.

Dozens of addresses on unrelated consumer ISPs, all sharing one byte identical header fingerprint. One page each, never two. Every single one reporting exactly ten seconds on site.

From inside any one plane of observation that is indistinguishable from a crowd of real visitors. That is the whole problem. A User-Agent blocklist catches what announces itself. It does not catch headless Chrome on rotating residential proxies, which is what scraping actually looks like now.

How it works

Three independent planes, cross checked against each other.

Transport, what the access log records
Behaviour, how the session moves across requests
Execution, what the browser can actually run

Each one alone is defeatable and essentially every existing tool uses exactly one of them. Every verdict comes with the evidence that produced it, never a bare score you have to trust.

Time on site, measured honestly

Conventional analytics pings while the page is merely loaded, so a tab left open in a background window reports as an hour of engagement. Loghound keeps three separate clocks and never mixes them. Wall time, visible time, engaged time. Kept apart those numbers say something. Averaged into one they say whatever you want to hear.

Where the data goes

It installs on your own server and reads your access log read only. What it learns lives in two Solr indexes on your own Opensolr account. Everything except the address is hashed. No telemetry, no vendor copy of your traffic.

An Opensolr account is required because it needs to create and shape its own indexes, which it cannot do against a Solr server it does not manage. The free plan is enough to start. That is the honest catch and I would rather say it here than have somebody find it in the installer.

Stack

PHP 8.1, no Composer, no npm, no build step. Apache, nginx and Caddy log formats. systemd units for the tailer and the scoring job.

Source: https://github.com/phpcip/loghound Docs: https://opensolr.com/loghound-docs What it does: https://opensolr.com/loghound

Happy to answer anything, including the awkward questions about false positives. There is a whole documentation page on what it does not catch and why, because a bot detector that oversells itself is worse than none.

u/WillingnessQuick5074 — 1 month ago

▲ 6 r/HostingReport+2 crossposts

Opensolr now has a free forever plan

Opensolr now has a free forever plan

Not a trial. No countdown, no expiry date, no credit card field anywhere. You sign up with a name and an email, you get a fully managed Opensolr Index in a few seconds, and you keep it for as long as you use it.

What you get on the free plan:

1 fully managed Opensolr Index
48.8 MB disk space
512 MB traffic per month
Full Apache Solr API, so any Solr client library works unchanged
Our WordPress and Drupal search plugins, with facets and autocomplete

What is not included: Web Crawler, backups, search analytics, AI and hybrid search. Those stay on the paid plans.

One rule worth knowing: the plan never expires, but an index that sees nothing at all for 30 days can be cleaned up. Logging in, indexing a document, changing the config or running a single query all count as activity, so normal use keeps it alive.

Full details on the free plan: https://opensolr.com/learn/billing/34/solr-cloud-free-membership

Paid plans, if you outgrow it: https://opensolr.com/pricing

u/WillingnessQuick5074 — 7 days ago

▲ 7 r/vectordatabase+2 crossposts

Everyone keeps telling me Solr can't do modern AI search. Fine. Here is a live Solr index with real vectors and RAG. Build one in a click and go look at the embeddings yourself

Disclosure: I run Opensolr. This is my product.

I got tired of hearing "Solr is fine for keywords, use something else for vectors". Easier to show than argue:

https://opensolr.com/rag-in-60-seconds

One click creates a real Apache Solr 9.6 index. Paste JSON, or give it your sitemap and the crawler indexes your site. Embeddings happen server-side — no OpenAI key, no Docker, no model download. Then ask questions and get answers grounded in your own documents, with sources.

You also get the index credentials, so you can open the raw Solr index and look at the 1024-dimension vectors yourself instead of trusting a demo.

No signup. Deletes itself after 3 days.

Two things I learned building it: pure vector search kept missing exact tokens (product codes, names), pure keyword search kept missing paraphrases — you need both. And what you put in the context window matters more than which model you use.

Go break it and tell me where it falls over.

reddit.com

u/WillingnessQuick5074 — 2 months ago

▲ 2 r/Rag

Everyone keeps telling me Solr can't do modern AI search. Fine. Here is a live Solr index with real vectors and RAG — build one in a click and go look

Disclosure: I run Opensolr. This is my product, so judge accordingly. This is a Completely Free Sandbox, which leads to a full Tutorial.

I got tired of hearing "Solr is fine for keywords, use something else for vectors". Easier to show than argue:

https://opensolr.com/rag-in-60-seconds

You also get the index credentials, so you can open the raw Solr index and look at the 1024-dimension vectors yourself instead of trusting a demo.

No signup. Deletes itself after 24 hours.

Go break it and tell me where it falls over.

reddit.com

u/WillingnessQuick5074 — 2 months ago

▲ 0 r/WordpressPlugins

[FREEMIUM] Opensolr Search — free plugin that replaces WP search with hybrid keyword + semantic/vector search (WooCommerce, facets, autocomplete). 15-day trial, then €17/mo.

I run a small hosted Solr company (Opensolr) and a few months ago we released a free WordPress plugin that replaces the default WP search with a proper search engine — and I mean proper: keyword search and semantic/vector search combined in one query (hybrid), not just a fancier LIKE query on wp_posts.

What you get out of the box, no code:

Hybrid search: exact keyword matches + semantic matches, so "cheap laptop for students" finds "affordable notebooks for college" too
Autocomplete / typeahead
Facets (categories, tags, custom taxonomies, WooCommerce attributes & price)
WooCommerce products indexed with the rest of your content
Multilingual sites (locale-aware)
Result elevation (pin a page to the top for a given query)
Search analytics — what people actually search for and what gets zero results
Two ways to get content in: push from WordPress (posts/pages/products) or let a crawler index the live site

How it works: the plugin ships the content to a hosted Solr index we run, queries it, and renders results on a search page / shortcode / embeddable widget. Vector embeddings are generated on our side, so no GPU or extra infra on your host. Works fine on cheap shared hosting because all the heavy lifting happens off your server.

Cost: create an account, you get a 15-day free trial with everything enabled (vector search included). After that the smallest plan is €17/month, billed yearly — 1 index, 100 MB of index storage, 20 GB search traffic/month. That's comfortably enough for a blog, a docs site or a small-to-medium shop (a few thousand pages). There's also a 20% discount for WordPress/Drupal users on the pricing page.

Honest limitations:

It's a hosted service, so if you need everything on-prem this isn't for you
No free-forever tier — 15 days, then paid
The smallest plan is one index, so one site (multisite needs more)

Links:

Plugin: https://wordpress.org/plugins/opensolr-search/
What it looks like / how hybrid search works: https://opensolr.com/wordpress-ai-search
Docs: https://opensolr.com/wordpress-search-docs

Happy to answer anything about relevance tuning, Solr, or why WP's built-in search is the way it is. And if you try it and something sucks, tell me — I'm the one fixing it.

reddit.com

u/WillingnessQuick5074 — 2 months ago

▲ 0 r/semanticweb

We built a custom {!hybrid} query parser fusing BM25 + kNN per document — now exposed through LangChain/LlamaIndex/Haystack/Scout/MCP

For the Solr folks: instead of RRF-ing two separate requests client-side, we wrote a query parser plugin that runs edismax and kNN as sub-queries and fuses normalized scores per document — mode=union|intersection|keywords_required|meaning_required, with a tunable semantic-vs-lexical alpha.

The interesting engineering bit was score normalization: BM25 is unbounded while cosine similarity lives in [0..1], so naive weighted sums let lexical outliers swamp the vector signal. We ended up reshaping BM25 through a sigmoid with a tunable k before fusing — the alpha slider then behaves predictably across corpora.

On top of the parser: server-side E5 embeddings at index and query time (a knn_vector field populated by the ingestion pipeline), and this week we shipped integration packages so the same pipeline is reachable from Python, PHP, and MCP agents without touching Solr syntax.

Live demo of the parser in action (a real news index): https://search.opensolr.com/news__dense?q=how+can+I+save+money%3F

Disclosure: founder of the service. AMA about the parser internals.

reddit.com

u/WillingnessQuick5074 — 2 months ago

▲ 1 r/vectordatabase

A LangChain vector store where you never run an embedding model and never send vectors

I kept seeing the same RAG stack: sentence-transformers locally (or OpenAI embeddings billed per token) + a vector DB + BM25 bolted on with client-side score fusion. I run a managed Apache Solr service, so I built a vector store integration that pushes all of that server-side.

pip install langchain-opensolr

from langchain_opensolr import OpensolrVectorStore

vs = OpensolrVectorStore(index="mysite__dense", email="you@example.com",
                         api_key="...", create_if_missing=True)
vs.add_texts(["Hybrid search fuses BM25 with vector similarity"])

docs = vs.similarity_search("how do keyword and meaning combine?", hybrid=True)
answer = vs.ai_answer("what does the doc say about score fusion?")  # grounded RAG, no LLM key

What's different from most entries in the integrations directory: embeddings (multilingual E5, 1024-dim) are computed on the server at both index and query time, hybrid fusion happens inside Solr via a custom {!hybrid} query parser (not client-side score juggling), and underneath it's plain Apache Solr 9 — facets, highlighting, the whole /select API still there.

Disclosure: founder. It's in the official LangChain integrations directory. Live demo of the same pipeline (our own site's index): https://search.opensolr.com/news__dense?q=how+am+I+supposed+to+save+money%3F — GitHub: https://github.com/phpcip/langchain-opensolr

reddit.com

u/WillingnessQuick5074 — 2 months ago

▲ 1 r/BlackboxAI_

A LangChain vector store where you never run an embedding model and never send vectors

pip install langchain-opensolr

from langchain_opensolr import OpensolrVectorStore

vs = OpensolrVectorStore(index="mysite__dense", email="you@example.com",
                         api_key="...", create_if_missing=True)
vs.add_texts(["Hybrid search fuses BM25 with vector similarity"])

docs = vs.similarity_search("how do keyword and meaning combine?", hybrid=True)
answer = vs.ai_answer("what does the doc say about score fusion?")  # grounded RAG, no LLM key

reddit.com

u/WillingnessQuick5074 — 2 months ago

▲ 1 r/mcp

opensolr-mcp: give Claude/Cursor a managed search engine — index, search, and grounded answers as tools

Published our MCP server in the official registry (com.opensolr/opensolr-mcp, domain-verified). One config entry and any MCP agent gets 9 tools: hybrid + semantic search, document ingestion (async, embeddings computed server-side), grounded RAG answers, and index management.

{ "mcpServers": { "opensolr": {
    "command": "uvx", "args": ["opensolr-mcp"],
    "env": { "OPENSOLR_EMAIL": "...", "OPENSOLR_API_KEY": "..." } } } }

The agent never touches a vector — plain text in, ranked results out. Fun test we ran: ingested a PDF (server-side text extraction), then asked "which funny British films with zombies could I watch tonight?" — zero keyword overlap with the document, found it semantically, answered from the PDF's content.

Ingestion is async with a per-job status board in the control panel, so agents can fire-and-forget or poll the ingest status tool.

Live demo of the same search pipeline the tools use (a real news index): https://search.opensolr.com/news__dense?q=which+ancient+trade+routes+are+becoming+tourist+attractions%3F

Disclosure: founder. The free trial covers the whole flow (no card). GitHub: https://github.com/phpcip/opensolr-mcp

reddit.com

u/WillingnessQuick5074 — 2 months ago

▲ 10 r/Rag

Lesson learned building RAG on top of our own search product: retrieval quality was 90% of the problem

We run a search hosting service and recently shipped RAG across our client packages (LangChain / LlamaIndex / Haystack / MCP / Laravel Scout). The thing that actually moved answer quality wasn't the LLM — it was retrieval.

What failed first: our answer endpoint originally did its own lexical retrieval (edismax, strict minimum-match) — paraphrased questions returned "no information" while semantic search found the document instantly. Classic vocabulary mismatch.

What fixed it: two-step RAG where retrieval is a first-class citizen — hybrid (BM25 + kNN fused per document) picks the top N docs, and their content becomes the LLM context. Crucially, the same relevance tuning the search UI uses (field weights, minimum match, stopword-cleaned lexical query BUT the full phrase to the embedder) applies to RAG retrieval too. Tuning retrieval in one place improved both search results AND answers.

Second lesson: cache answers keyed on (query + retrieved context), not query alone — otherwise stale answers survive reindexing.

Third lesson: the prompt matters less than people think, but one clause matters a lot: "answer ONLY based on the content below" turned meandering summaries into grounded, sectioned answers.

Live demo of the retrieval + answer pipeline (a real news index): https://search.opensolr.com/news__dense?q=why+did+scientists+change+the+el+nino+labelling%3F

We ship this as packages if anyone wants to poke at the approach (disclosure: founder): https://pypi.org/project/langchain-opensolr/ — happy to answer questions about the hybrid fusion details.

reddit.com

u/WillingnessQuick5074 — 2 months ago

▲ 2 r/LangChain

A LangChain vector store where you never run an embedding model and never send vectors

pip install langchain-opensolr

from langchain_opensolr import OpensolrVectorStore

vs = OpensolrVectorStore(index="mysite__dense", email="you@example.com",
                         api_key="...", create_if_missing=True)
vs.add_texts(["Hybrid search fuses BM25 with vector similarity"])

docs = vs.similarity_search("how do keyword and meaning combine?", hybrid=True)
answer = vs.ai_answer("what does the doc say about score fusion?")  # grounded RAG, no LLM key

reddit.com

u/WillingnessQuick5074 — 2 months ago

▲ 0 r/PHP

Hybrid search (BM25 + vectors) and grounded RAG for PHP — embeddings computed server-side, zero ML dependencies

I run a managed Apache Solr service (Opensolr, 15 years now) and just shipped a PHP package for it: hybrid search (BM25 + kNN vectors fused per document) and grounded RAG answers, with embeddings computed server-side — so no Python sidecar, no OpenAI key, no ML dependencies in composer.json.

composer require opensolr/laravel-scout-opensolr

Despite the name, the core client is framework-agnostic (plain Guzzle) — you can use it from any PHP app:

use Opensolr\ScoutOpensolr\OpensolrClient;

$client = new OpensolrClient('you@example.com', 'your-api-key');

// hybrid search: the platform's tuned pipeline (field weights, minimum match,
// semantic-vs-lexical balance), overridable per call
$results = $client-&gt;embedAndSearch('myapp__dense', 'budget dining spots', 10, [
    'search_mode' =&gt; 'keywords_required',
]);

// grounded RAG: top hybrid hits become the LLM context, one call
$answer = $client-&gt;aiAnswer('myapp__dense', 'what does our refund policy say?');

If you ARE on Laravel, it registers as a Scout engine: Post::search('budget dining')->paginate(15) gets hybrid semantic relevance, and all your models share one index (scoped per model automatically), so a single plan covers everything.

Writes go through an async ingestion queue — documents get embeddings, language detection and derived fields computed server-side, with a per-job status board in the control panel.

Live demo of the exact search pipeline (a real news index): https://search.opensolr.com/news__dense?q=how+am+I+supposed+to+save+money%3F

Disclosure: I'm the founder. Free 15-day trial, no card. Code: https://github.com/phpcip/laravel-scout-opensolr — feedback welcome, especially on the client API design.

u/WillingnessQuick5074 — 2 months ago

▲ 2 r/u_WillingnessQuick5074+1 crossposts

Custom hybrid (BM25 + kNN) query parser for Solr 9.x — score-based blend instead of RRF. Live demos to poke at.

We built a custom {!hybrid} QParser plugin for Solr 9.x that fuses BM25 and kNN per document. Solr's native RRF combiner (9.11/10.1) does rank-based fusion; we went a different route — a score-based blend with in-candidate-set normalization — and I'm curious what people think.

The problem: BM25 is unbounded, and cosine similarity is anisotropic — modern embedders pack it into a razor-thin high band (e.g. ~0.92–0.94 for basically everything). So summing raw scores means one side dominates, and the search-mode toggles end up cosmetic. We normalize BOTH signals within the candidate set (BM25 max-norm; vector min-max to spread that thin cosine band), then blend with a tunable semantic↔lexical alpha. The mode (union / intersection / keywords-required / meaning-required) actually controls the candidate set now, not just the score.

We A/B'd it against RRF on a real product catalog and the score-based blend ranked cleaner for us (RRF mixed editorial/blog content in among the products), but I'd genuinely like other eyes on where it falls over.

Live demos — change the `q=` in the URL, and hit the "{ } Debug" toggle in the bottom-right for debugQuery + the full params sent to Solr + per-doc lexical/vector score components:

- "see heat through walls" → https://search.opensolr.com/fluke?q=see+heat+through+walls&locale=en_us&in=web

- "how do I check current without touching wires" → https://search.opensolr.com/fluke?q=how+do+I+check+current+without+touching+wires&locale=en_us&in=web

- EN query → Romanian catalog, "tyres that work both in winter and summer" → https://search.opensolr.com/dedeman?q=tyres+that+work+both+in+winter+and+summer&in=web

- EN query → Swedish news, "what's going on with migration these days?" → https://search.opensolr.com/nyheter?q=what%27s+going+on+with+migration+these+days%3F&sort_field=creation_date&sort_dir=desc

- "what is going on with climate change" → https://search.opensolr.com/vector?q=what+is+going+on+with+climate+change&in=web

Read-only Solr access if you want to run your own queries (basic auth 123:123):

- https://123:123@fi.solrcluster.com/solr/fluke/select?wt=json&indent=true&q=*:*&rows=2

- https://123:123@de9.solrcluster.com/solr/vector/select?wt=json&indent=true&q=*:*&rows=2

- https://123:123@de9.solrcluster.com/solr/nyheter/select?wt=json&indent=true&q=*:*&rows=2

Write-up on how it works: https://opensolr.com/opensolr-platform-user-documentation/hybrid-search

Free trial if you want to index your own data: https://opensolr.com/register

Roast it — observations, critiques, and queries that break it are exactly what I'm after.

reddit.com

u/WillingnessQuick5074 — 3 months ago