dsfx/docs/discovery.md

# Encrypted File Storage System – Initial Discovery Document

## 1. Overview

This document outlines the discovery and design considerations for a secure, encrypted file storage system intended for self-hosting enthusiasts. The system is designed not only to securely store and version files with client-side encryption and an immutable manifest but also to extend into a peer-to-peer (P2P) network for redundancy, backup, and collaborative sharing between trusted nodes.

## 2. Goals & High-Level Vision

### 2.1. Security & Privacy

- **End-to-End Encryption:**

  - Files are encrypted on the client before being uploaded, ensuring that the server or any peer only ever handles ciphertext.
  - Decryption occurs solely on the client, preserving data privacy even in a distributed network.

- **Robust Key Management:**
  - Encryption keys are derived client-side using strong key derivation functions (e.g., Argon2, scrypt, or PBKDF2).
  - Integration with OS-level or hardware-based key management solutions is considered for enhanced security.

### 2.2. Efficient Versioning & Storage

- **File Partitioning & Chunking:**

  - Files are divided into fixed-size or content-defined chunks (e.g., 1 MB per chunk).
  - Only updated chunks are re-uploaded which reduces upload times and storage consumption.

- **Content Addressable Storage & Deduplication:**

  - Each chunk is encrypted separately and indexed by its SHA-256 hash.
  - This method allows for deduplication, reducing redundant uploads across versions and even across files.

- **Verifiable Append-Only Manifest:**
  - An append-only log (referred to as the manifest) maintains a complete history of chunk operations and file versions.
  - Periodic snapshots (that include a hash of the log state) allow for pruning of older logs while ensuring verifiability of data integrity.

### 2.3. P2P Node Network & Redundancy

- **Peer-to-Peer (P2P) Connectivity:**

  - Users can connect their nodes with those of friends using an IP address plus public key combination for secure identification and mutual authentication.
  - Nodes establish encrypted channels (e.g., via TLS or secure Diffie-Hellman exchanges) to share encrypted file blobs.

- **Data Redundancy and Backup:**

  - Shared file blobs are replicated across nodes to provide data redundancy and backup.
  - Users can define replication policies, choosing which peers should store copies.

- **Dynamic Trust and Access Control:**
  - Nodes employ certificate or public-key-based validation mechanisms to grant or restrict access.
  - A decentralized trust framework ensures that only authorized nodes participate in file sharing.

## 3. Architecture & Component Overview

### 3.1. Client Application

- **User Interaction:**

  - Initially a command-line interface (CLI) is used for file management; a web interface is planned for future development.

- **Processing & Encryption:**

  - Files are partitioned into chunks which are each individually encrypted using authenticated encryption algorithms (e.g., AES-GCM or ChaCha20-Poly1305).
  - The client maintains an immutable, append-only manifest that records all operations such as additions, modifications, and deletions.

- **Synchronization:**
  - The manifest is used for multi-device synchronization, allowing devices to merge their change logs seamlessly.
  - The manifest incorporates periodic snapshots to prune old entries while keeping a cryptographic chain for integrity verification.

### 3.2. Server/P2P Node

- **Self-Hosting Friendly:**

  - Designed to run on user-managed servers with simple configuration.
  - Containerized deployment (e.g., Docker) ensures consistent and easy installation.

- **API & Data Handling:**

  - The node provides a secure, stateless API (RESTful or gRPC) for interactions with clients.
  - Encrypted blobs (chunks) are stored and indexed using content addressing to support deduplication and versioning.

- **P2P Functionality & Node Connectivity:**

  - Nodes establish direct secure connections using an IP address and public key identifier.
  - Support for NAT traversal techniques (e.g., UPnP, STUN, TURN) is built in to facilitate peer connections.
  - A decentralized discovery mechanism (potentially via a DHT or bootstrap nodes) helps nodes locate one another.

- **Redundancy & Data Synchronization:**
  - Each node maintains information in the manifest regarding which peers store which file chunks.
  - Health checks and heartbeat signals maintain up-to-date replication and rebalancing across the network.

## 4. Detailed Design Considerations

### 4.1. File Partitioning and Deduplication

- **Chunking Strategy:**

  - Fixed-size chunks provide predictability, while content-defined chunking (e.g., using Rabin fingerprinting) may help reduce unnecessary re-uploads when file content shifts.

- **Content Addressing:**
  - SHA-256 is used to hash each chunk before storage, allowing duplicate chunks to be recognized and only stored once across nodes.

### 4.2. Manifest (Append-Only Log)

- **Structure & Integrity:**

  - The manifest is an append-only log where each entry records a discrete change (addition, deletion, modification of chunks), along with metadata such as timestamps, device identifiers, and operation details.
  - Cryptographic chaining (each log entry containing a hash of the previous one) ensures tamper-evident history.

- **Snapshot Mechanism:**
  - Periodic snapshots capture the full state of the manifest up to that point by incorporating an aggregate hash.
  - Future log entries reference the last snapshot, allowing previous data to be pruned while maintaining verifiability.

### 4.3. Multi-Device & P2P Synchronization

- **Manifest Merging:**

  - Devices (including those on separate nodes) merge their manifest logs using ordering methods such as vector clocks or Lamport timestamps to resolve concurrent updates.

- **Node-to-Node Sharing:**
  - Secure node connections based on IP address and public key identifiers facilitate the sharing of encrypted blobs between trusted peers.
  - A decentralized model ensures that file updates, redundancy, and replication policies are distributed and maintained across the network.

### 4.4. Security and Access Control

- **Connection Security:**

  - All node-to-node communications use strong encryption (TLS, Diffie-Hellman key exchange, or equivalent) to protect data in transit.

- **Trust & Authentication:**
  - Nodes exchange public keys during initial handshake for mutual authentication.
  - Certificate-based or signed permission systems safeguard against unauthorized access, ensuring only trusted peers participate.

### 4.5. Scalability, Resilience, and Management

- **NAT Traversal & Discovery:**

  - Implementation of NAT traversal techniques (UPnP, STUN, TURN) and decentralized peer discovery ensures reliable connectivity, even behind firewalls.

- **Monitoring & Conflict Resolution:**
  - Systems to monitor node availability (heartbeats, health checks) are essential for maintaining replication and redundancy.
  - Conflict resolution protocols are implemented in the manifest for consistent state management across devices and nodes.

## 5. Threat Model & Security Audit

- **Threat Model Perspective:**

  - Protect against potential vulnerabilities including unauthorized access, tampering with the manifest, compromised nodes, and interception of communications.
  - Emphasize that all sensitive operations (encryption, key management, and manifest maintenance) are performed client-side or within a trusted node environment.

- **Audit & Transparency:**
  - An immutable, cryptographically chained manifest offers auditability.
  - The design encourages regular security audits and community reviews to validate the security framework.

## 6. Roadmap & Future Enhancements

- **Initial Development:**

  - Build a robust CLI client for file operations, including adding files, updating versions, and restoring files, backed by an append-only manifest.
  - Develop the server/P2P node with secure API endpoints and replication functionality.

- **P2P Network Expansion:**

  - Implement automated peer discovery, NAT traversal support, and dynamic trust mechanisms.
  - Create user-friendly tools for configuring peer connections and managing replication policies.

- **Advanced Features:**
  - Extend the system with a web interface.
  - Explore integration with existing distributed systems or version control systems.
  - Consider plugins or additional tools to automate backup management and network health monitoring.

## 7. Conclusion

This discovery document establishes the fundamental framework for a secure, self-hosted encrypted file storage system that integrates efficient versioning, a verifiable append-only manifest, and a P2P network for file sharing and redundancy. By combining client-side encryption, content-addressable storage, robust manifest management, and secure node connectivity, the product aims to deliver high security, privacy, and resilience in a distributed self-hosting environment.