feat(docs): beef out docs

docs/README.md

@@ -4,10 +4,9 @@ This is the documentation for DSFX, the next generation decentralized file excha
 
 ## Overview
 
-- [Axioms](./axioms.md) is our style guide, which outlines our design philosophy and coding standards.
-- [Hosting](./hosting.md) gets you started with your own DSFX server.
-- [Administration](./administration.md) shows you how to manage your DSFX server and its users.
-- [Operating](./operating.md) covers the usage of the dsfx client to securely upload and download files.
+- [Start](./start.md) gets you running with DSFX.
+- [Concepts](./concepts/README.md) explains the basic concepts of DSFX.
+- [Operating](./operating/README.md) covers the installation, usage, and maintenance of DSFX.
 
 These docs are targeted at the users of DSFX. If you want to understand how it
 works under the hood, check out the [internals docs](./internals/README.md).

docs/administration.md

@@ -1,77 +0,0 @@
-# Administration
-
-## Overview
-
-This document provides an overview of the administration tasks and responsibilities for managing the
-system. It includes guidelines for user management, system configuration, and maintenance tasks.
-
-Interactions with the server are done through the `dsfxctl` command line tool, which allows administrators
-to perform various operations with their system.
-
----
-
-## dsfxctl
-
-### Installation
-
-To install the `dsfxctl` command line tool, we recommend using the standard go toolchain to install
-the latest version of the tool. You can do this by running the following command:
-
-```bash
-go install koti.casa/numenor-labs/dsfx/cmd/dsfxctl@latest
-```
-
-### Configuration
-
-The first time you run the `dsfxctl` command, it will create a configuration directory to store
-the necessary files for the tool to function properly. By default, this directory is located in the
-`/etc/dsfxctl`, but this can be overridden by setting the **-configDir** cli flag to a different path.
-We recommend configuring a path in your home directory so that the command can be run without root
-privileges. For example, you can set the flag to point to a directory in your home
-directory, such as `~/.config/dsfxctl`.
-
-```bash
-go run koti.casa/numenor-labs/dsfx/cmd/dsfxctl@main -configDir ~/.config/dsfxctl
-```
-
-### Identity
-
-The `dsfxctl` command requires a user identity to perform operations. This identity is specified
-by the `ed25519.key` file in the configuration directory. The key file should contain a valid
-ed25519 private key encoded in base64. The server application will use this key to sign requests and
-authenticate the user. If the key file is not present, which is the case when you first run the command,
-the server will generate a new key pair and store it in the `ed25519.key` file. This file is stored
-unencrypted, so it is important to ensure that the permissions on this file are set correctly to
-prevent unauthorized access. We recommend setting the permissions to `0600` to restrict access
-to the owner only. You can do this by running the following command:
-
-```bash
-chmod 0600 ~/.config/dsfxctl/ed25519.key
-```
-
-In the future we will explore options for encrypting the key at rest, or using a hardware security
-module (HSM) to manage the key securely.
-
-To view the identity associated with the `dsfxctl` command, you can run the following command:
-
-```bash
-dsfxctl identity
-```
-
-This will display the base64 encoded public key associated with the private key in the `ed25519.key` file.
-
-### Admin Registration
-
-Admin registration is performed similar to authorized ssh users. The server application initializes a
-file in the config directory called `admins`. This file contains a list of base64 encoded public keys
-for each admin user. The server will check this file to determine if a user is authorized to perform
-administrative tasks. Keys are separated by newlines, and the server will ignore any empty lines.
-
-You must have access to the host machine that you are deploying the server on to perform the
-initial registration of the admin user. You can get the public key of your `dsfxctl` client by running
-the `dsfxctl identity` command. You should then copy the base64 encoded public key and paste it into the
-`admins` file in the configuration directory.
-
-## User Management
-
-**todo**

docs/concepts/README.md

@@ -0,0 +1,9 @@
+# Concepts
+
+This section is for anyone evaluating DSFX, eager to learn about it, or curious. It focuses on
+the big picture and problems that DSFX solves.
+
+- [Decentralized](./decentralized.md) goes over the decentralized nature of DSFX.
+- [Security](./security.md) explains the security model of DSFX and how it works.
+- [Privacy](./privacy.md) see how we handle user privacy and data protection.
+- [Blobs](./blobs.md) explains how DSFX handles files and blobs.

docs/concepts/blobs.md

@@ -0,0 +1,50 @@
+# Blobs
+
+_This document is under construction. We are actively refining its content, so please check back for updates._
+
+## Overview
+
+In DSFX, file data is managed through a system of "blobs." A blob is a content-addressable, encrypted chunk of data that forms the basic unit of file transfer and storage within the DSFX network. By decoupling file management from the underlying data, DSFX can efficiently distribute files, eliminate duplicates, and maintain privacy, as servers never access the raw underlying file content.
+
+## How Blobs Work
+
+1. **Partitioning and Padding:**
+
+   - When a user uploads a file, the client application partitions the file into 1 MB chunks.
+   - If a file is smaller than 1 MB, it is padded to reach the standard chunk size.
+   - This uniform chunk size supports simpler handling on both the client and server sides.
+
+2. **Encryption:**
+
+   - Each 1 MB chunk is encrypted using appropriate cryptographic methods.
+   - Encryption ensures that even if a server handles the blob, it remains oblivious to its underlying content.
+
+3. **Content-Addressable Storage:**
+
+   - Once encrypted, every blob is assigned a unique identifier, which is its SHA-256 hash.
+   - Because the blob’s identifier is computed from its content, any change in the underlying data results in a new identifier.
+   - This design supports both integrity verification and de-duplication: identical data results in the same blob hash, allowing multiple files with overlapping content to share the same underlying blobs.
+
+4. **Server Role:**
+
+   - DSFX servers operate only in terms of these blobs. They are responsible solely for receiving, storing, and transferring blobs as units of data.
+   - The servers do not hold or understand metadata about the file or its meaning. Their responsibilities are limited to blob distribution and management.
+
+5. **Client Responsibilities:**
+   - The client application is in charge of managing the file manifest. This manifest maps files to the specific ordered list of blob identifiers.
+   - Clients track what blobs comprise a file, ensuring that files can be reconstructed accurately when needed.
+   - This model also enables advanced storage operations, such as de-duplication across different files or even accounts.
+
+## Future Directions
+
+- **Reference Counting:**
+
+  - In the future, we plan to explore mechanisms such as reference counting. This would allow multiple user accounts to reference the same underlying blob.
+  - A reference counting system can enable efficient storage utilization and simplify garbage collection of unused data.
+
+- **Enhanced Distribution:**
+
+  - As the DSFX network evolves, we envision more sophisticated techniques for blob distribution and caching, reducing network overhead and speeding up file retrieval.
+
+- **Extended Metadata:**
+  - Further refinements to the file manifest management may include additional metadata, enhancing the client’s ability to manage, search, and organize content more effectively.

docs/concepts/decentralized.md

@@ -0,0 +1,32 @@
+# Decentralized
+
+_This document is under construction. We are actively working on refining the content and will update it in the near future._
+
+## Overview
+
+DSFX is built on a fundamentally decentralized architecture. This design choice eliminates any single point of failure, fosters enhanced privacy, and distributes trust across the participating nodes. In DSFX, every node functions both as a client and a server, exchanging data and verifying transactions independently from any centralized authority.
+
+## Key Principles
+
+- **Trustless Communication:**
+  Every peer in the network establishes secure channels using cryptographic handshakes. By leveraging robust key exchange mechanisms and digital signatures, DSFX enables nodes to securely communicate and verify each other’s identities without relying on a central server.
+
+- **Distributed Data Exchange:**
+  Files and blobs are exchanged directly between nodes. The system is designed to handle data distribution in a way that prevents data bottlenecks while maintaining integrity and confidentiality.
+
+- **Resilience and Fault Tolerance:**
+  As nodes operate independently, the network remains resilient—even if parts of it fail or become compromised. Data exchange is dynamically rerouted through alternative nodes ensuring continuous operation.
+
+- **Decentralized Governance:**
+  DSFX’s decentralized setup leaves the decisions regarding network upgrades, protocol changes, and other critical parameters to the majority of the network participants rather than a central authority.
+
+## Future Directions
+
+In upcoming revisions, we plan to delve deeper into:
+
+- Specific protocols and algorithms that enable decentralized key management.
+- The methodology behind node discovery and dynamic routing.
+- Comparative benefits over centralized file exchange systems.
+- Use cases and scenarios that highlight the importance of decentralization in DSFX.
+
+_Stay tuned – this section is actively evolving as we refine and expand the decentralized concepts underlying DSFX._

docs/concepts/privacy.md

@@ -0,0 +1,29 @@
+# Privacy
+
+_This document is under construction. We are actively refining its content, so please check back for updates._
+
+## Overview
+
+DSFX is designed with privacy at its core. In an age where data breaches and surveillance are increasingly common, DSFX strives to empower users by protecting their personal information and file transfers from unwanted exposure. Our approach integrates advanced cryptographic techniques, decentralized data handling, and user-centric controls to ensure that privacy is maintained at every level of the system.
+
+## Key Principles
+
+- **Data Encryption End-to-End:**
+  All files, messages, and handshakes in DSFX are encrypted using state-of-the-art algorithms. This means that data is protected not only during transit but also while at rest on different nodes, ensuring that only authorized parties can access it.
+
+- **Decentralized Data Management:**
+  By distributing file storage and exchange across multiple nodes, DSFX minimizes the risk of mass data breaches. No single node holds the entirety of a user's data, reducing the impact of potential compromises.
+
+- **User-Controlled Data Sharing:**
+  Privacy is further reinforced by giving users full control over what data is shared, with whom, and under what circumstances. Access permissions, controlled sharing links, and selective disclosure ensure that personal information remains private unless explicitly authorized for disclosure.
+
+## Future Directions
+
+In future revisions, we plan to expand on:
+
+- Detailed breakdowns of the encryption algorithms and key management systems used.
+- Privacy-preserving strategies in network design, including metadata minimization and connection obfuscation.
+- Consent and access policies that empower users to define and manage their privacy preferences.
+- Comparative studies of DSFX’s approach against traditional centralized systems, underscoring the benefits in protecting user data.
+
+_This section is actively evolving as we continue to integrate best practices in privacy preservation and adapt to new challenges._

docs/concepts/security.md

@@ -0,0 +1,35 @@
+# Security
+
+_This document is under construction. We are actively refining its content, so please check back for updates._
+
+## Overview
+
+Security is a foundational pillar of DSFX. Our goal is to protect user data and network communications against potential adversaries by integrating advanced cryptographic algorithms, rigorous protocol designs, and best practices in secure coding. In DSFX, every interaction—from file exchanges to authentication handshakes—is carefully engineered to minimize risks and uphold the integrity and confidentiality of the system.
+
+## Key Principles
+
+- **Robust Cryptography:**
+  DSFX leverages industry-standard symmetric and asymmetric encryption techniques. Files and messages are encrypted using AES-GCM for authenticated encryption, ensuring data remains confidential and tamper-proof during storage and transit.
+
+- **Mutual Authentication:**
+  Using a combination of ephemeral Diffie–Hellman key exchange and digital signatures based on ED25519, DSFX executes secure handshakes where both parties validate each other’s identity. This mutual authentication strategy prevents man-in-the-middle attacks and ensures that only legitimate nodes can participate in the network.
+
+- **Integrity and Non-Repudiation:**
+  By signing every exchange with strong cryptographic signatures, DSFX not only protects the integrity of data but also guarantees non-repudiation. Any alteration or tampering with the data during transit is immediately detectable via failed signature verifications.
+
+- **Defense-in-Depth:**
+  Security in DSFX is implemented in multiple layers—from low-level cryptographic routines and length-prefixed framing to high-level decentralized key management. This layered approach ensures that even if one mechanism fails, other safeguards remain to protect the system.
+
+- **Adherence to Secure Coding Practices:**
+  Inspired by our internal axioms, our implementation enforces strict safety and correctness guidelines. We use assertions liberally, integrate explicit error handling, and maintain rigorous controls over resource usage to fend off potential vulnerabilities such as buffer overflows or memory corruptions.
+
+## Future Directions
+
+In upcoming revisions, we plan to delve deeper into:
+
+- Detailed technical explanations of the cryptographic routines used (e.g., AES-GCM, ED25519, ECDH key exchanges).
+- An in-depth look at our secure handshake protocol, discussing its design, potential threats, and mitigation strategies.
+- Comparisons with other security models to highlight the advantages provided by DSFX’s layered defense mechanism.
+- Practical considerations and best practices for maintaining a secure environment when deploying DSFX.
+
+_This section is actively evolving as we continue to assess security challenges and integrate improvements in our design and implementation._

docs/internals/README.md

@@ -5,7 +5,7 @@ DSFX, you don't need to read this and could head straight to our user-level docs
 
 If you want to learn how DSFX works inside, here's what we got:
 
-- [Axioms](../axioms.md) is _the_ style guide, and more. This is the philosophy underlining
+- [Axioms](./axioms.md) is _the_ style guide, and more. This is the philosophy underlining
   all the code here!
 - [commit strategy](./commits.md) is the guide for writing commit messages that follow the Conventional Commits
   specification.

docs/operating.md

@@ -1,3 +0,0 @@
-# Operating
-
-Work in progress.

docs/operating/README.md

@@ -0,0 +1,8 @@
+# Operating
+
+This section is for anyone who wants to run their own instance of the DSFX server.
+
+- [Installing](./installing.md) will help you install dsfx.
+- [Hardware](./hardware.md) will help you choose the right hardware for your needs.
+- [Deploying](./deploying.md) will help you deploy dsfx to your server.
+- [Upgrading](./upgrading.md) will help you upgrade your instance of dsfx.

docs/operating/deploying.md

@@ -0,0 +1,33 @@
+# Deploying
+
+Please follow the [installation instructions](./installing.md) to install the
+server on your deployment machine.
+
+## Starting
+
+If you opted to _not_ install dsfx locally, then you can run it directly with the go toolchain.
+
+```bash
+go run koti.casa/numenor-labs/dsfx/cmd/dsfx@main
+```
+
+Otherwise, you can run the server with the `dsfx` command. For those who built
+directly from source, refer to the binary location that you specified in the
+build command.
+
+## Configuration
+
+The application accepts the following cli flags:
+
+- **-h**: Show help message
+- **-host**: The host on which the DSFX server will run (default: `0.0.0.0`)
+- **-port**: The port on which the DSFX server will listen (default: `8000`)
+- **-logLevel**: The log level to use. One of (error, warn, info, debug) (default: `info`)
+- **-dataDir**: The directory where the DSFX data files are stored (default: `/etc/dsfx/data`)
+- **-configDir**: The directory where the DSFX configuration files are stored (default: `/etc/dsfx/config`)
+
+## Local Files
+
+The DSFX server uses local files for configuration and storage. The default directories for these
+files are specified in the **-configDir** and **-dataDir** cli flags. You can
+change these directories by specifying the flags when you run the server.

docs/operating/hardware.md

@@ -0,0 +1,6 @@
+# Hardware
+
+Testing is currently happing on a Raspberry Pi 3 with 4 cores, 1 gb of ram, and
+a 32gb sd card. We plan on releasing charts to show what sort of performance and
+reliability you can expect from various hardware configurations, but for now our
+target is the Pi. This forces us to be small, avoid waste, and be efficient.

docs/operating/installing.md

@@ -1,18 +1,17 @@
-# Hosting
+# Installing
 
-This document describes how to set up and run your own DSFX server. It is
-intended for users who want to run their own instance of DSFX, either for
-testing or for production use. If you are looking for the DSFX client, please
-see the [operating docs](./operating.md). If you are looking for the DSFX
-administration docs, please see the [administration docs](./administration.md).
+This document describes how to install the dsfx server on your machine. The
+dsfx server is a decentralized file exchange server that allows you to share
+files with other users in a secure and private way. The server is designed to
+be easy to install and configure, and it can be run on a variety of platforms.
+The server is written in Go, and it uses the Go toolchain for building and
+installing. The server is designed to be run as a standalone application, and
+it does not require any external dependencies.
 
 - [Setup](#setup)
   - [Dependencies](#dependencies)
   - [Git Setup](#git-setup)
-  - [Binary Installation](#binary-installation)
-  - [Running](#running)
-- [Configuration](#configuration)
-- [Local Files](#local-files)
+  - [Installation](#installation)
 
 ---
 
@@ -62,7 +61,13 @@ export GOPRIVATE=koti.casa
 If you want to persist this setting across sessions, you can add the above line to your
 `~/.bashrc` or `~/.bash_profile`.
 
-### Binary Installation
+### Stopping Point
+
+At this point, you should have the go toolchain installed and configured. If you
+only want to run the server, and you don't care to actually install it, you can
+you can skip directly the [deploying docs](./deploying.md).
+
+### Installation
 
 Now that you have the go toolchain set up with access to our repository, you can simply install the
 project by using the `go install` command.
@@ -71,30 +76,6 @@ project by using the `go install` command.
 go install koti.casa/numenorlabs/dsfx/cmd/dsfx@main
 ```
 
-### Running
-
-If you are familiar with the go toolchain, you can run the executables in the
-`cmd` directory directly. For other users, we recommend using the `go run`
-command to run the project. This will automatically download and install the
-dependencies for you, and run the project in a single command.
-
-```bash
-go run koti.casa/numenorlabs/dsfx/cmd/dsfx@main
-```
-
-## Configuration
-
-The application accepts the following cli flags:
-
-- **-h**: Show help message
-- **-host**: The host on which the DSFX server will run (default: `0.0.0.0`)
-- **-port**: The port on which the DSFX server will listen (default: `8000`)
-- **-logLevel**: The log level to use. One of (error, warn, info, debug) (default: `info`)
-- **-dataDir**: The directory where the DSFX data files are stored (default: `/etc/dsfx/data`)
-- **-configDir**: The directory where the DSFX configuration files are stored (default: `/etc/dsfx/config`)
-
-## Local Files
-
-The DSFX server uses local files for configuration and storage. The default directories for these
-files are specified in the **-configDir** and **-dataDir** cli flags. You can
-change these directories by specifying the flags when you run the server.
+Assuming that you have the `GOPATH` environment variable set up correctly, you will now be able to
+run the `dsfx` command from the command line. If you are not able to, then you must configure your
+`GOPATH` environment variable.

docs/operating/upgrading.md

@@ -0,0 +1,3 @@
+# Upgrading
+
+Under construction.

docs/start.md

@@ -0,0 +1,3 @@
+# Start
+
+Under construction.