dsfx/docs/adr/0006-manifest-design.md

# ADR-0006: Manifest Design (Append-Only Log)

## Status

Proposed

## Context

Our system requires robust support for file versioning, change tracking, and auditability while maintaining data integrity. To achieve this, the manifest will serve as an append-only log that records every file operation. This design is essential for ensuring that changes to file data and metadata can be traced over time, which is core to our mission of transparency and user control.

The manifest must be tamper-evident through cryptographic chaining of log entries and include periodic snapshots that summarize the state up to a given point. These snapshots help balance performance with auditability while keeping the historical log verifiable. Additionally, this design must support deterministic simulation testing to validate its behavior under conditions such as simulated network and disk delays or failures (NFR-2, NFR-6).

## Decision

We will implement the manifest as an append-only log structured with the following characteristics:

- **Append-Only Structure and Cryptographic Chaining:**
  Each log entry will record a discrete change (e.g., file addition, modification, or deletion) along with associated metadata (such as timestamps and file or chunk identifiers). Every entry will include a cryptographic hash that chains it to the previous entry, ensuring that any tampering is immediately detectable. This design supports auditability and transparency (FR-6, NFR-5).

- **Snapshot Mechanism:**
  To avoid performance degradation as the log grows, the manifest will incorporate periodic snapshots. A snapshot captures the current state of the manifest along with an aggregate hash referencing all previous entries. Future log entries will reference the most recent snapshot, enabling efficient verification of the complete history while pruning old log details if needed.

- **Versioning and Conflict Management:**
  The manifest will serve as the authoritative source for file versioning across multiple devices and nodes. Strategies to handle version merging and conflict resolution will be embedded within the manifest design, ensuring that concurrent updates can be reconciled consistently (FR-10).

- **Deterministic Simulation Integration:**
  The manifest design will allow for deterministic simulation of its append-only behavior. This means that in testing, we can simulate network and disk latency or failures and ensure that log entries are recorded, chained, and snapshotted correctly. This approach is critical to validating our performance and robustness requirements under adverse conditions (NFR-2, NFR-6).

## Consequences

- **Advantages:**

  - **Auditability and Transparency:** The chained, append-only log design provides clear, verifiable evidence of every operation, which is central to our commitment to user control and data integrity (FR-6, NFR-5).
  - **Resilience and Consistency:** With proper snapshots and cryptographic chaining, the manifest provides a resilient mechanism for tracking file changes, aiding in conflict detection and resolution.
  - **Efficient Verification:** Periodic snapshots reduce the burden of verifying a long history of entries, helping maintain performance while still offering complete audit trails.

- **Trade-offs:**
  - **Increased Storage and Processing Overhead:** Maintaining and validating a constantly growing log and producing periodic snapshots will add processing and storage overhead. This trade-off is justified by the need for thorough auditability and data integrity.
  - **Complexity in Conflict Management:** As multiple devices interact and update the manifest concurrently, embedded versioning and conflict resolution mechanisms add to system complexity. This complexity is managed internally and remains transparent to the user.
  - **Testing Considerations:** To ensure the manifest performs correctly under simulated failure conditions, the deterministic testing framework must robustly emulate various latencies and failures, which increases testing effort.

## References to Requirements

- **Functional Requirements:**

  - **Versioning and Change Tracking (FR-3 and FR-10):** The manifest is essential for recording file versions and managing conflict resolution across devices.
  - **Auditability (FR-6):** The tamper-evident, chained log design provides verifiable audit trails for all file operations.

- **Non-Functional Requirements:**
  - **Performance & Responsiveness (NFR-2):** The use of snapshots and efficient verification techniques is critical to maintaining performance as the manifest grows.
  - **Deployability & Maintainability (NFR-6):** The design supports a deterministic simulation framework that verifies behavior during network and disk latency or failure conditions.
  - **Auditability & Transparency (NFR-5):** Ensuring that every log entry’s integrity can be cryptographically verified directly supports these principles.

## Conclusion

Implementing the manifest as an append-only, cryptographically chained log with periodic snapshots ensures a robust, transparent audit trail critical for file versioning and user-controlled data management. This design not only satisfies our functional and non-functional requirements but also aligns with our mission of providing a secure, resilient, and fully verifiable storage system.