The Principle of Less
The standard playbook for modern software insists that more data equals better product capability. Collect the raw payloads, store every user action, accumulate giant vector spaces, and figure out the value later.
For a massive enterprise with dedicated compliance, infrastructure, and security teams, that approach is manageable, and often profitable. Large organizations have the leverage to monetize massive data lakes, build proprietary training sets, and absorb the overhead of cross-region replication and regulatory audits. They can afford to store everything because they have entire departments dedicated to extracting signal from the noise.
The Asymmetry of Scale
For smaller teams, however, copying the enterprise playbook is an existential trap. The calculus of data collection changes entirely when your engineering bandwidth is measured in single digits.
When you harvest everything, you inherit everything: sprawling compliance frameworks, ballooning infrastructure bills, and an expanding surface area of technical debt. Big teams trade capital and headcount to manage data complexity; small teams simply drown in it. The counterintuitive answer for lean teams is to aggressively reduce scope. By refusing to collect what you don’t strictly need to process, you instantly simplify your architecture, eliminate compliance overhead, and keep system complexity down to something a single engineer can reason about.
The Hidden Cost of Over Harvesting
When smaller teams try to collect everything, they inherit massive operational drag driven by three fundamental forces:
- Risk: Compliance nightmares from storing PII or sensitive user text, requiring complex retention schedules, redaction pipelines, and encryption-at-rest policies that pull developers away from core product work.
- Cost: Storage and networking bloat from transferring, indexing, and backing up raw payloads instead of compact, lossy representations.
- Complexity: Architectural bloat from turning lean, purposeful services into sprawling monoliths just to handle speculative feature requirements and data migrations.
In Practice
While building Atlas, our primary architectural bet is simple: don’t make the search algorithm do work the system can eliminate earlier. Instead of holding raw documents in a heavy vector database or building a massive proprietary data warehouse, we ship a client library that handles content hashing on the user’s side.
Atlas never gets to see the raw content; it only sees the keyed hashes generated locally. Atlas automatically derives a collection from the hashes you provide for a document, then derives LSH bands from those collections. All of those design choices mean we can search through millions of hashes and get results in tens of milliseconds, while keeping compliance overhead and networking costs near zero.
Privacy isn’t just a marketing term for us, it is inherent to our design. Even if our servers are breached, your indexes are secure because we never own your documents, see their text or touch their contents, only the search representation generated by the client.
Conclusion
By letting the user own the raw data and shipping only lossy, compact fingerprints; as seen in Glyph Collections and Glyph Query; smaller teams can punch above their weight. Less data collection doesn’t just protect user privacy; it protects your team from drowning in its own infrastructure.