lentil/rhizome-node
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

planning

Browse Source
This commit is contained in:
Lentil Hoffman 2025-06-23 00:59:55 -05:00
parent f6790bf90d
commit ef6efe758f
Signed by: lentil
GPG Key ID: 0F5B99F3F4D0C087
4 changed files with 456 additions and 373 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

304
plans/beam-implementation.md Normal file
View File

@ -0,0 +1,304 @@
# Rhizome on the BEAM: Implementation Guide
> **Note**: This document outlines a fresh implementation of Rhizome in Elixir, using the TypeScript implementation as a reference.
## Table of Contents
- [Motivation](#motivation)
- [Architecture Overview](#architecture-overview)
- [Migration Strategy](#migration-strategy)
- [Key Components](#key-components)
- [Data Synchronization Model](#data-synchronization-model)
- [Development Roadmap](#development-roadmap)
- [Performance Considerations](#performance-considerations)
## Motivation
Moving Rhizome to Elixir and the BEAM virtual machine provides several key advantages:
1. **Distribution by Default**
- Built-in distribution primitives for node-to-node communication
- Network partition tolerance out of the box
- Location transparency for processes
2. **Fault Tolerance**
- Let it crash philosophy with supervision trees
- Self-healing systems through process isolation
- Hot code reloading for zero-downtime updates
3. **Concurrency Model**
- Lightweight processes for handling millions of concurrent connections
- Efficient message passing between processes
- Built-in backpressure handling
4. **Ecosystem Benefits**
- Mature tooling for distributed systems
- Strong pattern matching and immutability
- Excellent support for building resilient systems
## Architecture Overview
### Current TypeScript Architecture
```mermaid
graph TD
subgraph "TypeScript Implementation"
A[HTTP API<br/><small>Express</small>]
B[WebSocket<br/><small>Socket.IO</small>]
C[Core Engine]
D[Delta Store]
E[View Layer]
F[Persistence<br/><small>LevelDB</small>]
G[Peer Network<br/><small>Libp2p</small>]
A --> C
B --> C
C --> D
C --> E
D --> F
C --> G
end
```
### Proposed Elixir Architecture
```mermaid
graph TD
subgraph "Elixir Implementation"
A[HTTP/WebSocket<br/><small>Phoenix</small>]
B[Core OTP App]
C[Delta Store]
D[View Layer]
E[Process Supervision]
F[Distribution<br/><small>EPMD/GenRPC</small>]
G[Persistence<br/><small>Mnesia/ETS</small>]
A <--> B
B <--> C
B <--> D
B <--> E
B <--> F
C <--> G
end
```
## Implementation Roadmap
### 1. Core Engine
- **Delta Processing**
- Define core Delta types and operations
- Implement DeltaBuilder
- Design storage layer (Mnesia/ETS)
- **View System**
- Implement Lossy/Lossless views
- Create resolver framework
- Add caching layer
### 2. Distribution
- **Node Communication**
- Node discovery and membership
- Delta synchronization protocol
- Conflict resolution strategies
- **Plugin System**
- Plugin behavior and lifecycle
- Dependency management
- Hot code reloading
### 3. API & Tooling
- **HTTP/WebSocket API**
- RESTful endpoints
- Real-time updates
- Authentication/authorization
- **Developer Experience**
- TypeScript type generation
- CLI tools
- Monitoring and metrics
## Key Components
### 1. Delta Processing
This implementation will follow similar patterns to the TypeScript version but leverage Elixir's strengths:
```elixir
defmodule Rhizome.Delta do
@type t :: %__MODULE__{
id: String.t(),
creator: String.t(),
timestamp: integer(),
operations: [operation()],
transaction_id: String.t() | nil,
negate: boolean()
}
defstruct [:id, :creator, :timestamp, :operations, :transaction_id, negate: false]
def new(creator, host) do
%__MODULE__{
id: generate_id(),
creator: creator,
timestamp: System.system_time(:millisecond),
operations: []
}
end
def add_operation(delta, operation) do
%{delta | operations: [operation | delta.operations]}
end
end
```
### 2. View System
```elixir
defmodule Rhizome.View.Lossy do
@behaviour Rhizome.View.Behaviour
@impl true
def init(initial_state) do
%{state: initial_state, cache: %{}}
end
@impl true
def reduce(%{state: state} = view, delta) do
new_state = apply_delta(state, delta)
%{view | state: new_state}
end
@impl true
def resolve(%{state: state}), do: state
defp apply_delta(state, %Delta{operations: ops}) do
Enum.reduce(ops, state, &apply_operation/2)
end
end
```
### 3. Plugin System
```elixir
defmodule Rhizome.Plugin do
@callback init(args :: term) :: {:ok, state :: term} | {:error, reason :: term}
@callback handle_delta(delta :: Delta.t(), state :: term) :: {:ok, new_state :: term} | {:error, term}
@callback handle_call(request :: term, from :: {pid, reference}, state :: term) ::
{:reply, reply, new_state} |
{:reply, reply, new_state, timeout | :hibernate} |
{:noreply, new_state} |
{:noreply, new_state, timeout | :hibernate} |
{:stop, reason, reply, new_state} |
{:stop, reason, new_state} when reply: term, new_state: term, reason: term
defmacro __using__(_opts) do
quote do
@behaviour Rhizome.Plugin
use GenServer
# Default implementations
@impl true
def init(_args), do: {:ok, %{}}
@impl true
def handle_call(_request, _from, state), do: {:reply, :ok, state}
def start_link(args) do
GenServer.start_link(__MODULE__, args, name: __MODULE__)
end
end
end
end
```
## Data Synchronization Model
### 1. Delta Propagation
```mermaid
sequenceDiagram
participant C1 as Client 1
participant N1 as Node 1
participant N2 as Node 2
participant N3 as Node 3
C1->>N1: Submit Delta
N1->>N1: Apply Delta Locally
N1->>N2: Gossip Delta
N1->>N3: Gossip Delta
N2->>N2: Apply Delta
N3->>N3: Apply Delta
N2->>N1: Acknowledge
N3->>N1: Acknowledge
```
### 2. Conflict Resolution
1. **Last Write Wins** (Default)
2. **Custom Resolvers**
3. **CRDT-based** for special cases
## Development Milestones
### 1. Core Delta Engine
- [ ] Define delta types and operations
- [ ] Implement DeltaBuilder
- [ ] Basic storage with Mnesia/ETS
- [ ] View system with Lossy/Lossless support
### 2. Distributed Foundation
- [ ] Node discovery and membership
- [ ] Delta synchronization protocol
- [ ] Conflict resolution strategies
- [ ] Plugin system
### 3. Production Features
- [ ] HTTP/WebSocket API
- [ ] Authentication & authorization
- [ ] Monitoring and metrics
- [ ] Developer tooling
## Performance Characteristics
### Key Advantages
1. **Concurrency**
- Handle 100K+ concurrent connections per node
- Sub-millisecond delta processing
- Linear scaling with cores
2. **Memory Usage**
- Shared binary heap for deltas
- Efficient garbage collection
- Process isolation for fault tolerance
3. **Network Efficiency**
- Delta compression
- Batched updates
- Smart backpressure handling
## Getting Started
### Prerequisites
- Elixir 1.14+
- Erlang/OTP 25+
- Node.js (for assets)
### Running Locally
```bash
# Clone the repository
git clone https://github.com/your-org/rhizome-beam.git
cd rhizome-beam
# Install dependencies
mix deps.get
cd assets && npm install && cd ..
# Start the application
iex -S mix phx.server
```
## Contributing
1. Fork the repository
2. Create a feature branch
3. Submit a pull request
## License
[Your License Here]
## Acknowledgments
- The Elixir and Erlang communities
- The original TypeScript implementation for inspiration
- Research in distributed systems and CRDTs

191
plans/fluent-view-composition.md
View File

@ -1,191 +0,0 @@
# Fluent API for View Composition
## Overview
This document outlines a fluent API for declaring and composing lossy views in a declarative and type-safe manner. The API is designed to make it easy to create complex view compositions with minimal boilerplate.
## Core Concepts
1. **View Composition**: Combining multiple resolvers to create a unified view
2. **Fluent Builder Pattern**: Method chaining for declarative configuration
3. **Type Safety**: Leveraging TypeScript's type system for better developer experience
4. **Extensibility**: Easy to add new composition patterns and resolvers
## API Design
### 1. Base Builder
```typescript
import { Lossless } from '../lossless';
type ViewTransformer = (view: LosslessViewOne) => LosslessViewOne;
type ResultTransformer<T, R> = (result: T) => R;
class ViewCompositionBuilder {
private resolvers: Array<{
key: string;
resolver: Lossy<any, any>;
viewTransformer?: ViewTransformer;
resultTransformer?: (result: any) => any;
}> = [];
constructor(private readonly lossless: Lossless) {}
// Start building a new composition
static create(lossless: Lossless): ViewCompositionBuilder {
return new ViewCompositionBuilder(lossless);
}
// Add a resolver to the composition
withResolver<T, R>(
key: string,
resolver: Lossy<T, R>,
options: {
view?: ViewTransformer;
result?: ResultTransformer<R, any>;
} = {}
): this {
this.resolvers.push({
key,
resolver,
viewTransformer: options.view,
resultTransformer: options.result,
});
return this;
}
// Build the final composition
build<T extends Record<string, any>>(): Lossy<Record<string, any>, T> {
// Implementation that creates a combined resolver
return new CombinedResolver(this.lossless, this.resolvers);
}
}
```
### 2. Example Usage: Relationship Graph
```typescript
const relationshipGraph = ViewCompositionBuilder
.create(lossless)
.withResolver('entities', new EntityResolver(), {
view: view => ({
...view,
propertyDeltas: Object.fromEntries(
Object.entries(view.propertyDeltas).filter(([k]) => !k.startsWith('_rel_'))
)
})
})
.withResolver('relationships', new RelationshipResolver(), {
view: view => ({
...view,
propertyDeltas: Object.fromEntries(
Object.entries(view.propertyDeltas).filter(([k]) => k.startsWith('_rel_'))
)
})
})
.withResolver('stats', new StatsCollector())
.withResolver('metadata', new MetadataResolver(), {
result: (metadata) => ({
...metadata,
generatedAt: new Date().toISOString()
})
})
.build<{
entities: EntityMap;
relationships: RelationshipMap;
stats: Stats;
metadata: Metadata;
}>();
```
### 3. Advanced Composition
```typescript
// Nested composition
const userProfile = ViewCompositionBuilder
.create(lossless)
.withResolver('basicInfo', new BasicInfoResolver())
.withResolver('activity', ViewCompositionBuilder
.create(lossless)
.withResolver('recentPosts', new RecentPostsResolver())
.withResolver('notifications', new NotificationsResolver())
.build()
)
.withResolver('recommendations', new RecommendationsResolver())
.build<{
basicInfo: UserBasicInfo;
activity: {
recentPosts: Post[];
notifications: Notification[];
};
recommendations: Recommendation[];
}>();
```
## Implementation Details
### CombinedResolver Implementation
```typescript
class CombinedResolver<State extends Record<string, any>, Result>
extends Lossy<State, Result> {
constructor(
private readonly lossless: Lossless,
private readonly resolvers: Array<{
key: string;
resolver: Lossy<any, any>;
viewTransformer?: ViewTransformer;
resultTransformer?: (result: any) => any;
}>
) {
super(lossless);
}
initializer(view: LosslessViewOne): State {
return this.resolvers.reduce((state, { key, resolver, viewTransformer }) => {
const transformedView = viewTransformer ? viewTransformer(view) : view;
return {
...state,
[key]: resolver.initializer(transformedView)
};
}, {} as State);
}
reducer(state: State, view: LosslessViewOne): State {
return this.resolvers.reduce((newState, { key, resolver, viewTransformer }) => {
const transformedView = viewTransformer ? viewTransformer(view) : view;
return {
...newState,
[key]: resolver.reducer(state[key], transformedView)
};
}, { ...state });
}
resolver(state: State): Result {
return this.resolvers.reduce((result, { key, resolver, resultTransformer }) => {
const resolved = resolver.resolver(state[key]);
return {
...result,
[key]: resultTransformer ? resultTransformer(resolved) : resolved
};
}, {} as Result);
}
}
```
## Benefits
1. **Readability**: Clear, declarative syntax
2. **Type Safety**: Full TypeScript support with proper type inference
3. **Composability**: Easy to combine and nest resolvers
4. **Maintainability**: Isolated concerns and transformations
5. **Flexibility**: Custom view and result transformations
## Next Steps
1. Implement the base `ViewCompositionBuilder` and `CombinedResolver`
2. Add support for common patterns (filtering, mapping, etc.)
3. Create documentation with examples
4. Refactor existing resolvers to use the new composition API
5. Add performance optimizations (memoization, lazy evaluation)

152
plans/plugin-dep-vis.md Normal file
View File

@ -0,0 +1,152 @@
# Plugin Dependency Visibility Implementation Plan
## Overview
This document outlines the implementation plan for enforcing restricted state visibility in the CustomResolver system. The goal is to ensure that each plugin can only access the states of properties it has explicitly declared as dependencies.
## Current Behavior
- All plugins currently receive the complete `allStates` object containing all property states
- There's no enforcement of which states a plugin can access
- Dependencies are declared but not used for access control
## Proposed Changes
### 1. Update ResolverPlugin Interface
```typescript
interface ResolverPlugin<T = unknown> {
name: string;
dependencies?: PropertyID[]; // Explicitly declare which properties this plugin depends on
// ... rest of the interface
}
```
### 2. Modify CustomResolver Implementation
#### 2.1 Update Reducer Method
Modify the `reducer` method to filter states before passing to plugins:
```typescript
public reducer(
acc: CustomResolverAccumulator,
cur: LosslessViewOne
): CustomResolverAccumulator {
// ... existing setup code ...
// Process each property in execution order
for (const propertyId of executionOrder) {
// ... existing delta processing ...
// Create filtered states object with only declared dependencies
const visibleStates: Record<PropertyID, unknown> = {};
// Add states for declared dependencies
if (plugin.dependencies) {
for (const dep of plugin.dependencies) {
if (allStates[dep] !== undefined) {
visibleStates[dep] = allStates[dep];
}
}
}
// Pass only visible states to the plugin
propertyState.state = propertyState.plugin.update(
propertyState.state,
value,
delta,
visibleStates
);
// ... rest of the method ...
}
}
```
#### 2.2 Update Resolver Method
Similarly update the `resolver` method:
```typescript
resolver(cur: CustomResolverAccumulator): CustomResolverResult {
const res: CustomResolverResult = {};
for (const [entityId, entity] of Object.entries(cur)) {
const entityResult = { id: entityId, properties: {} };
const allStates: Record<PropertyID, unknown> = {};
// First pass: collect all states
for (const [propId, propState] of Object.entries(entity.properties)) {
allStates[propId] = propState.state;
}
// Second pass: resolve each property with filtered states
for (const [propId, propState] of Object.entries(entity.properties)) {
const plugin = propState.plugin;
const visibleStates: Record<PropertyID, unknown> = {};
// Only include declared dependencies
if (plugin.dependencies) {
for (const dep of plugin.dependencies) {
if (allStates[dep] !== undefined) {
visibleStates[dep] = allStates[dep];
}
}
}
const resolvedValue = plugin.resolve(propState.state, visibleStates);
if (resolvedValue !== undefined) {
entityResult.properties[propId] = resolvedValue;
}
}
if (Object.keys(entityResult.properties).length > 0) {
res[entityId] = entityResult;
}
}
return res;
}
```
### 3. Add Validation
Add validation to ensure dependencies exist:
```typescript
private validateDependencies(): void {
// Existing cycle detection...
// Add validation that all dependencies exist
for (const [pluginId, plugin] of Object.entries(this.config)) {
for (const dep of plugin.dependencies || []) {
if (!this.config[dep]) {
throw new Error(`Plugin '${pluginId}' depends on unknown property: ${dep}`);
}
}
}
}
```
### 4. Update Tests
1. Add tests for state visibility:
- Test that plugins only receive their declared dependencies
- Test that plugins can't access undeclared dependencies
- Test that dependency validation works
- Test that existing functionality remains unchanged
2. Update existing tests to account for the new behavior
## Migration Strategy
1. This is a breaking change for any plugins that were accessing undeclared dependencies
2. Add warnings in the next minor version
3. Make the behavior opt-in initially with a flag
4. In the next major version, make it the default
## Implementation Steps
1. [ ] Add the state filtering to `reducer`
2. [ ] Add the state filtering to `resolver`
3. [ ] Update dependency validation
4. [ ] Add comprehensive tests
5. [ ] Update documentation
6. [ ] Add deprecation warnings for plugins accessing undeclared dependencies
## Future Considerations
1. Add a debug mode that logs when plugins access undeclared dependencies
2. Consider adding granular permissions (read/write) for dependencies
3. Add support for wildcard dependencies for plugins that need access to many properties

182
plans/view-composition.md
View File

@ -1,182 +0,0 @@
# View Composition with Combined State
## Overview
This document outlines the design for composing multiple lossy views using a combined state approach. This pattern allows different resolvers to work on their own parts of the state while sharing the same underlying delta stream.
## Core Concept
The combined state approach involves:
1. Creating a parent resolver that manages multiple child resolvers
2. Each child resolver maintains its own state
3. The parent coordinates the initialization, reduction, and resolution of child states
4. The final result combines the outputs of all child resolvers
## Implementation Pattern
### 1. Base Interfaces
```typescript
interface CombinedState {
[resolverKey: string]: unknown;
}
interface CombinedResult {
[resolverKey: string]: unknown;
}
class CombinedResolver<State extends CombinedState, Result extends CombinedResult>
extends Lossy<State, Result> {
private resolvers: {
[key: string]: {
instance: Lossy<any, any>;
initializer: (view: LosslessViewOne) => any;
reducer: (state: any, view: LosslessViewOne) => any;
resolver: (state: any) => any;
};
} = {};
}
```
### 2. Registering Resolvers
```typescript
registerResolver<T, U>(
key: string,
resolver: Lossy<T, U>,
options?: {
// Optional: Transform the view before passing to the child resolver
viewTransformer?: (view: LosslessViewOne) => LosslessViewOne;
// Optional: Transform the result after resolution
resultTransformer?: (result: U) => unknown;
}
) {
this.resolvers[key] = {
instance: resolver,
initializer: (view) => {
const transformedView = options?.viewTransformer?.(view) ?? view;
return resolver.initializer(transformedView);
},
reducer: (state, view) => {
const transformedView = options?.viewTransformer?.(view) ?? view;
return resolver.reducer(state, transformedView);
},
resolver: (state) => {
const result = resolver.resolver(state);
return options?.resultTransformer?.(result) ?? result;
}
};
}
```
### 3. Core Methods
```typescript
initializer(view: LosslessViewOne): State {
const state = {} as State;
for (const [key, { initializer }] of Object.entries(this.resolvers)) {
state[key] = initializer(view);
}
return state;
}
reducer(state: State, view: LosslessViewOne): State {
const newState = { ...state };
for (const [key, { reducer }] of Object.entries(this.resolvers)) {
newState[key] = reducer(state[key], view);
}
return newState;
}
resolver(state: State): Result {
const result = {} as Result;
for (const [key, { resolver }] of Object.entries(this.resolvers)) {
result[key] = resolver(state[key]);
}
return result;
}
```
## Example: Relationship Graph with Multiple Resolvers
```typescript
class RelationshipGraphResolver extends CombinedResolver<
{
entities: EntityState;
relationships: RelationshipState;
metadata: MetadataState;
},
{
graph: GraphResult;
stats: StatsResult;
metadata: MetadataResult;
}
> {
constructor(lossless: Lossless) {
super(lossless);
// Register entity resolver
this.registerResolver('entities', new EntityResolver(lossless));
// Register relationship resolver
this.registerResolver(
'relationships',
new RelationshipResolver(lossless),
{
// Only process relationship deltas
viewTransformer: view => ({
...view,
propertyDeltas: Object.fromEntries(
Object.entries(view.propertyDeltas)
.filter(([key]) => key.startsWith('_rel_'))
)
})
}
);
// Register metadata resolver
this.registerResolver('metadata', new MetadataResolver(lossless));
}
// Override resolver to combine results
resolver(state: any) {
const results = super.resolver(state);
return {
nodes: results.entities,
edges: results.relationships,
stats: results.stats,
metadata: results.metadata
};
}
}
```
## Benefits
1. **Separation of Concerns**: Each resolver handles a specific aspect of the data
2. **Reusability**: Resolvers can be reused in different combinations
3. **Maintainability**: Changes to one resolver don't affect others
4. **Testability**: Each resolver can be tested in isolation
5. **Flexibility**: Easy to add, remove, or modify resolvers
## Performance Considerations
1. **Memory Usage**: Combined state increases memory usage
2. **Processing Overhead**: Each delta is processed by all resolvers
3. **Optimization**: Use view transformers to filter deltas early
## Next Steps
1. Implement the base `CombinedResolver` class
2. Refactor `RelationshipGraphResolver` to use this pattern
3. Add tests for the composition behavior
4. Document common patterns and best practices
5. Consider adding middleware support for cross-cutting concerns