u/siakshit

I'm using ghcr.io/openzim/zimit with Zimit 3.1.3 and Browsertrix Crawler 1.14.1.

I want to create a ZIM starting from this page:

https://www.geeksforgeeks.org/python/python-programming-language-tutorial/

My intended crawl is:

• Depth 0: the seed page
• Depth 1: every geeksforgeeks.org page linked from the seed
• Depth 2: every geeksforgeeks.org page linked from those depth-1 pages
• Do not crawl external domains
• I do NOT want the scope restricted to the seed URL's path

--depth 2 appears to work correctly. Browsertrix shows:

"maxDepth":2

The problem is the scope. With my normal seed, Browsertrix shows:

"scopeType":"prefix"

and generates an include regex for:

https://www.geeksforgeeks.org/python/python-programming-language-tutorial/

As a result, links such as:

https://www.geeksforgeeks.org/python/python-data-types/
https://www.geeksforgeeks.org/python/python-functions/
https://www.geeksforgeeks.org/python/python-interview-questions/

are skipped as:

Skipping Link; not in scope

I tried these:

--include-domains geeksforgeeks.org
-i geeksforgeeks.org
--include "^https://(www\.)?geeksforgeeks\.org/.*"

but in my Zimit 3.1.3 setup they appear to be passed to warc2zim, rather than configuring Browsertrix's crawl scope.

I also tried:

--scopeType custom

which successfully reached Browsertrix, but then Browsertrix had an empty include list and excluded even the seed.

So my question is:

What is the correct Zimit 3.1.3 command/option to make Browsertrix crawl the entire geeksforgeeks.org domain while still using --depth 2 from the seed?

Is it simply:

--scopeType domain

or is there another Browsertrix scope option that needs to be passed through Zimit?

Example of the kind of command I'm trying to achieve:

docker run --rm \
  -v "$PWD/output:/output" \
  ghcr.io/openzim/zimit \
  zimit \
  --seeds "https://www.geeksforgeeks.org/python/python-programming-language-tutorial/" \
  --name gfg_python \
  --title "GFG Python Tutorial" \
  --description "Offline archive of GeeksforGeeks Python tutorials." \
  --publisher "Akshit Kamboj (si4k)" \
  --creator "Akshit Kamboj (si4k)" \
  --lang en \
  --output /output \
  --workers 3 \
  --depth 2

I specifically want domain-wide scope + depth 2, not prefix scope.

Thanks!

I've been building Si4k Search, an open-source search engine designed for people running large offline knowledge servers with Kiwix/ZIM collections.

The idea is simple: instead of manually searching individual ZIMs, Si4k searches across the whole local Kiwix library and provides one unified search interface.

Current features:

• Searches across multiple ZIM sources through Kiwix
• Progressive result streaming using SSE
• Priority-based ZIM search, while still searching the rest of the library
• Pagination
• Automatic ZIM categorization based on metadata, names and parent folders
• Keyword-based category prioritization
• Configurable search concurrency and worker limits
• Configurable LRU caching
• Docker deployment
• Works on relatively modest hardware
• Designed to keep the actual ZIM library outside the Docker image

I currently have it running on an old-ish home server with 133 ZIM sources.

The architecture is roughly:

User
  ↓
Si4k Search
  ↓
Kiwix
  ↓
Local ZIM collection

The Docker container only contains the search engine. The ZIM collection stays mounted from the host, so adding hundreds of GB of knowledge doesn't mean rebuilding the Docker image.

I've also tested the search engine under concurrent workloads and added resource controls because I want this to be practical for home/offline knowledge servers rather than requiring a powerful machine.

Future ideas:

• Offline dictionaries
• Query translation and multilingual ZIM routing
• Local voice-to-text search
• Optional local LLM summaries grounded in search results
• OpenStreetMap/offline map search
• More offline document providers

I'm particularly interested in feedback from people running Kiwix/ZIM servers:

What would you want a unified search layer like this to do?

Are there problems with searching large ZIM collections that I should be thinking about?

GitHub: https://github.com/akshitkamboz13/search.si4k

Demo: https://search.si4k.online

I was analysing my game suddenly engine says +M389 I know its a glitch or something as I move Kc7 that +M389 becomes +M15 but still curious what would be +M389 /s