cleanup and levelDB integration

This commit is contained in:
Mykola Bilokonsky 2025-06-09 22:12:49 -04:00
parent c8488843d2
commit 1c55e95a7b
38 changed files with 443 additions and 218 deletions

View File

@ -1,4 +1,4 @@
import * as RhizomeImports from "../src";
import * as _RhizomeImports from "../src";
/**
* Tests for lossless view compose() and decompose() bidirectional conversion
* Ensures that deltas can be composed into lossless views and decomposed back

View File

@ -1,4 +1,4 @@
import * as RhizomeImports from "../src";
import * as _RhizomeImports from "../src";
import { Delta } from '../src/core';
import { NegationHelper } from '../src/features';
import { RhizomeNode } from '../src/node';

View File

@ -1,13 +0,0 @@
describe('Relational', () => {
it.skip('Allows expressing a domain ontology as a relational schema', async () => {});
// Deltas can be filtered at time of view resolution, and
// excluded if they violate schema constraints;
// Ideally the sender minimizes this by locally validating against the constraints.
// For cases where deltas conflict, there can be a resolution process,
// with configurable parameters such as duration, quorum, and so on;
// or a deterministic algorithm can be applied.
it.skip('Can validate a delta against a relational constraint', async () => {});
it.skip('Can validate a delta against a set of relational constraints', async () => {});
});

View File

@ -50,12 +50,13 @@ describe('Delta Storage', () => {
runStorageTests(() => storage as DeltaQueryStorage);
});
describe.skip('LevelDB Storage', () => {
describe('LevelDB Storage', () => {
let storage: DeltaQueryStorage;
beforeEach(async () => {
storage = new LevelDBDeltaStorage('./test-data/leveldb-test');
await (storage as LevelDBDeltaStorage).open();
await (storage as LevelDBDeltaStorage).clearAll();
});
afterEach(async () => {
@ -81,7 +82,7 @@ describe('Delta Storage', () => {
it('throws on unknown storage type', () => {
expect(() => {
StorageFactory.create({ type: 'unknown' as any });
StorageFactory.create({ type: 'unknown' as 'memory' | 'leveldb' });
}).toThrow('Unknown storage type: unknown');
});
});

View File

@ -1,4 +1,4 @@
import * as RhizomeImports from "../src";
import * as _RhizomeImports from "../src";
import { Delta } from '../src/core';
import { Lossless } from '../src/views';
import { RhizomeNode } from '../src/node';

View File

@ -1 +1 @@
MANIFEST-000012
MANIFEST-000030

Binary file not shown.

Binary file not shown.

View File

@ -1 +1 @@
MANIFEST-000012
MANIFEST-000030

Binary file not shown.

Binary file not shown.

View File

@ -1,143 +1,311 @@
# Next Steps - LevelDB Storage Tests & Cleanup
# Phase 4: Delta Patterns & Query Traversal - Implementation Plan
This document provides context and instructions for completing the storage system implementation in the next Claude Code session.
## Overview
## Current Status ✅
Phase 4 recognizes that in Rhizome, **deltas ARE relationships**. Instead of adding a relationship layer on top of deltas, we're creating tools to work with delta patterns more effectively. This phase focuses on formalizing common delta patterns, building query conveniences for traversing these patterns, and creating specialized resolvers that interpret deltas as familiar relational concepts.
- **Directory reorganization**: COMPLETE ✅
- **Storage abstraction**: COMPLETE ✅
- **Memory storage**: COMPLETE ✅ (9/9 tests passing)
- **LevelDB storage**: CODE COMPLETE ✅ (tests need fixing)
- **Query engines**: COMPLETE ✅ (both lossless and storage-based)
- **RhizomeNode integration**: COMPLETE ✅
- **Build system**: COMPLETE ✅ (clean compilation)
- **Test suite**: 21/22 suites passing, 174/186 tests passing
## Core Insights
## Immediate Tasks 🔧
1. **Deltas are relationships**: Every delta with pointers already expresses relationships
2. **Patterns, not structure**: We're recognizing patterns in how deltas connect entities
3. **Perspective-driven**: Different views/resolvers can interpret the same deltas differently
4. **No single truth**: Competing deltas are resolved by application-level lossy resolvers
5. **Time-aware**: All queries are inherently temporal, showing different relationships at different times
### 1. Fix LevelDB Storage Tests (Priority: HIGH)
## Current State ✅
**Issue**: LevelDB tests fail with "Database is not open" error
- **All tests passing**: 21/21 suites, 183/183 tests (100%)
- **Delta system**: Fully functional with pointers expressing relationships
- **Negation system**: Can invalidate deltas (and thus relationships)
- **Query system**: Basic traversal of lossless views
- **Schema system**: Can describe entity structures
- **Resolver system**: Application-level interpretation of deltas
**Location**: `__tests__/storage.ts` (currently skipped on line 53)
## Implementation Plan
**Root Cause**: LevelDB requires explicit opening in newer versions
### Step 1: Delta Pattern Recognition
**Solution Strategy**:
```typescript
// In LevelDBDeltaStorage constructor or storeDelta method:
async ensureOpen() {
if (this.db.status !== 'open') {
await this.db.open();
}
}
**Goal**: Formalize common patterns of deltas that represent familiar relationships
// Call before any operation:
await this.ensureOpen();
**Tasks**:
1. Create `src/patterns/delta-patterns.ts`:
- Define patterns for common relationship types
- Create pattern matching utilities
- Document pattern conventions
2. Common patterns to recognize:
```typescript
// One-to-one: A delta pointing from A to B with unique constraint
const AuthorshipPattern = {
name: 'authorship',
match: (delta) =>
delta.pointers.some(p => p.targetContext === 'author') &&
delta.pointers.some(p => p.targetContext === 'post'),
interpret: (delta) => ({
post: delta.pointers.find(p => p.targetContext === 'post').target,
author: delta.pointers.find(p => p.targetContext === 'author').target
})
};
// One-to-many: Multiple deltas pointing from many Bs to one A
const PostsByAuthorPattern = {
name: 'posts-by-author',
query: (authorId) => ({
pointers: {
some: {
target: authorId,
targetContext: 'author'
}
}
})
};
```
3. Pattern validation:
- Ensure deltas match expected patterns
- Provide clear feedback when patterns are violated
- Allow flexible pattern definitions
### Step 2: Query Pattern Traversal
**Goal**: Make it easy to traverse delta patterns in queries
**Tasks**:
1. Extend `QueryEngine` with pattern-aware methods:
```typescript
// Find all deltas that establish a certain relationship
queryEngine.findRelationships('authorship', {
author: 'user-123'
});
// Traverse relationships in time
queryEngine.findRelationships('authorship', {
author: 'user-123',
asOf: timestamp // Time-travel query
});
```
2. Create traversal helpers:
```typescript
// Follow a chain of relationships
queryEngine.traverse({
start: 'user-123',
follow: [
{ pattern: 'authorship', direction: 'from' },
{ pattern: 'comments', direction: 'to' }
],
includeNegated: false // Perspective choice
});
```
3. Multi-perspective queries:
```typescript
// Different views of the same deltas
queryEngine.query('Post', {}, {
perspectives: {
published: { includeNegated: false },
draft: { includeNegated: true },
historical: { asOf: timestamp }
}
});
```
### Step 3: Pattern-Aware Resolvers
**Goal**: Create resolvers that interpret delta patterns as familiar concepts
**Tasks**:
1. Create `src/views/resolvers/pattern-resolver.ts`:
```typescript
class PatternResolver {
// Interpret deltas matching certain patterns
resolveWithPatterns(entityId, patterns) {
const deltas = this.lossless.getDeltasForEntity(entityId);
return {
entity: entityId,
relationships: patterns.map(pattern => ({
type: pattern.name,
targets: deltas
.filter(pattern.match)
.map(pattern.interpret)
}))
};
}
}
```
2. Specialized pattern resolvers:
- `ReferenceResolver`: Follows pointer patterns
- `TemporalResolver`: Shows relationships over time
- `CompetingValueResolver`: Handles multiple values for same relationship
3. Resolver composition:
```typescript
// Stack resolvers for different perspectives
const publishedView = new ResolverStack([
new NegationFilter(),
new TemporalResolver({ until: now }),
new LastWriteWins()
]);
```
### Step 4: Delta Pattern Validation
**Goal**: Validate that deltas follow expected patterns (without enforcing)
**Tasks**:
1. Create `src/features/pattern-validation.ts`:
```typescript
// Validate but don't enforce
validateDeltaPattern(delta, pattern) {
const result = pattern.validate(delta);
if (!result.valid) {
// Emit warning, but still accept delta
this.emit('pattern-warning', {
delta,
pattern: pattern.name,
issues: result.issues
});
}
return result;
}
```
2. Pattern constraints as guidance:
- Required pointer contexts
- Expected value types
- Cardinality suggestions
- Temporal constraints
3. Missing information detection:
```typescript
// Detect incomplete patterns
detectMissingRelationships(entity, expectedPatterns) {
return expectedPatterns.filter(pattern =>
!this.hasMatchingDelta(entity, pattern)
);
}
```
### Step 5: Collection Pattern Helpers
**Goal**: Make collections work naturally with delta patterns
**Tasks**:
1. Extend collections with pattern methods:
```typescript
class PatternAwareCollection extends Collection {
// Create deltas that match patterns
relate(from, to, pattern) {
const delta = pattern.createDelta(from, to);
return this.rhizomeNode.acceptDelta(delta);
}
// Query using patterns
findRelated(entity, pattern) {
return this.queryEngine.findRelationships(pattern, {
[pattern.fromContext]: entity
});
}
}
```
2. Pattern-based operations:
- Batch relationship creation
- Relationship negation helpers
- Pattern-based cascades
### Step 6: Temporal Pattern Queries
**Goal**: Leverage time-travel for relationship history
**Tasks**:
1. Time-aware pattern queries:
```typescript
// Show relationship changes over time
queryEngine.relationshipHistory('authorship', {
post: 'post-123',
timeRange: { from: t1, to: t2 }
});
// Find when relationships were established/negated
queryEngine.relationshipTimeline(entityId);
```
2. Temporal pattern analysis:
- Relationship duration
- Relationship conflicts over time
- Pattern evolution
## File Structure
**New files to create**:
```
src/
├── patterns/
│ ├── delta-patterns.ts # Pattern definitions
│ ├── pattern-matcher.ts # Pattern matching utilities
│ └── pattern-validators.ts # Pattern validation
├── query/
│ └── pattern-query-engine.ts # Pattern-aware queries
├── views/
│ └── resolvers/
│ ├── pattern-resolver.ts # Pattern interpretation
│ └── temporal-resolver.ts # Time-aware resolution
└── features/
└── pattern-validation.ts # Soft validation
```
**Files to modify**:
- `src/storage/leveldb.ts` - Add auto-opening logic
- `__tests__/storage.ts` - Remove `.skip` from line 53
- `src/query/query-engine.ts` - Add pattern methods
- `src/collections/collection-abstract.ts` - Add pattern helpers
- `src/node.ts` - Wire up pattern features
**Test command**: `npm test -- __tests__/storage.ts`
## Testing Strategy
### 2. Complete Linting Cleanup (Priority: MEDIUM)
**New test files**:
- `__tests__/delta-patterns.ts` - Pattern definition and matching
- `__tests__/pattern-queries.ts` - Pattern-based traversal
- `__tests__/pattern-validation.ts` - Soft validation behavior
- `__tests__/temporal-patterns.ts` - Time-travel relationship queries
- `__tests__/competing-relationships.ts` - Multiple relationship handling
**Current lint issues**: 45 errors (mostly unused vars and `any` types)
**Test scenarios**:
1. Define and match delta patterns
2. Query relationships using patterns
3. Validate deltas against patterns (warnings only)
4. Time-travel through relationship history
5. Handle competing relationship deltas
6. Detect missing relationships
7. Test pattern-based cascading negations
**Key files needing attention**:
- `src/query/query-engine.ts` - Remove unused imports, fix `any` types
- `src/query/storage-query-engine.ts` - Fix `any` types in JsonLogic
- `src/storage/leveldb.ts` - Remove unused loop variables (prefix with `_`)
- Various test files - Remove unused `RhizomeImports`
## Success Criteria
**Quick fixes**:
```typescript
// Instead of: for (const [key, value] of iterator)
// Use: for (const [_key, value] of iterator)
- [ ] Delta patterns are well-defined and matchable
- [ ] Queries can traverse relationships via delta patterns
- [ ] Pattern validation provides guidance without enforcement
- [ ] Time-travel queries work with relationships
- [ ] Competing relationships are handled gracefully
- [ ] Missing relationships are detectable
- [ ] Performance scales with pattern complexity
- [ ] Developers find patterns intuitive to use
// Instead of: JsonLogic = Record<string, any>
// Use: JsonLogic = Record<string, unknown>
```
## Key Principles to Maintain
### 3. Enable Relational Tests (Priority: LOW)
1. **Deltas are relationships** - Never create a separate relationship system
2. **Patterns are recognition** - We're recognizing what's already there
3. **Perspective matters** - Same deltas, different interpretations
4. **No enforcement** - Validation guides but doesn't restrict
5. **Time is first-class** - All relationships exist in time
6. **Conflicts are natural** - Multiple truths coexist until resolved by views
**Currently skipped**: `__tests__/relational.ts`
## Next Session Tasks
**Check**: Whether relational collection tests work with new directory structure
1. Define core delta patterns in `delta-patterns.ts`
2. Create pattern matching utilities
3. Extend QueryEngine with pattern-aware methods
4. Write tests for pattern recognition
5. Document the delta-as-relationship philosophy
## Context for Next Session 📝
### Storage Architecture Overview
The storage system now supports pluggable backends:
```
RhizomeNode
├── lossless (in-memory views)
├── deltaStorage (configurable backend)
├── queryEngine (lossless-based, backward compatible)
└── storageQueryEngine (storage-based, new)
```
**Configuration via environment**:
- `RHIZOME_STORAGE_TYPE=memory|leveldb`
- `RHIZOME_STORAGE_PATH=./data/rhizome`
### Key Files & Their Purposes
```
src/
├── storage/
│ ├── interface.ts # DeltaStorage + DeltaQueryStorage interfaces
│ ├── memory.ts # MemoryDeltaStorage (working ✅)
│ ├── leveldb.ts # LevelDBDeltaStorage (needs open() fix)
│ ├── factory.ts # StorageFactory for backend switching
│ └── store.ts # Legacy store (kept for compatibility)
├── query/
│ ├── query-engine.ts # Original lossless-based (working ✅)
│ └── storage-query-engine.ts # New storage-based (working ✅)
└── node.ts # Integrates both storage & query engines
```
### Test Strategy
1. **Memory storage**: Fully working, use as reference
2. **LevelDB storage**: Same interface, just needs DB opening
3. **Storage factory**: Already tested and working
4. **Query engines**: Both working with reorganized imports
## Success Criteria 🎯
**When complete, you should have**:
- [ ] All storage tests passing (both memory and LevelDB)
- [ ] Lint errors reduced to <10 (from current 45)
- [ ] Documentation updated for storage backends
- [ ] Optional: Relational tests re-enabled
**Test command for validation**:
```bash
npm test # Should be 22/22 suites passing
npm run lint # Should have <10 errors
npm run build # Should compile cleanly (already working)
```
## Notes & Gotchas ⚠️
1. **LevelDB opening**: The Level library changed APIs - databases need explicit opening
2. **Import paths**: All fixed, but watch for any remaining `../` vs `./` issues
3. **TypeScript**: Using ES modules (`"type": "module"`) - imports must include file extensions if needed
4. **Test isolation**: LevelDB tests should use unique DB paths to avoid conflicts
5. **Cleanup**: LevelDB creates real files - tests should clean up temp directories
## Phase 4 Readiness
Once this storage work is complete, the codebase will be ready for **Phase 4: Relational Features** with:
- ✅ Clean, organized directory structure
- ✅ Pluggable storage backends (memory + persistent)
- ✅ Dual query engines (lossless + storage-based)
- ✅ Comprehensive test coverage
- ✅ Solid architecture for relational schema expressions
The storage abstraction provides the foundation needed for advanced relational features like foreign key constraints, join operations, and complex queries across collections.
This approach embraces Rhizome's fundamental architecture where deltas ARE the relationships, making it easier to work with these patterns while respecting the system's perspective-driven, temporal nature.

View File

@ -1,8 +1,7 @@
import express, {Router} from "express";
import {Collection} from "../collections";
import {Delta} from "../core";
import {Delta, DeltaFilter} from "../core";
import {RhizomeNode} from "../node";
import {StorageJsonLogic} from "../query";
export class HttpApi {
router = Router();
@ -158,7 +157,7 @@ export class HttpApi {
const { schemaId } = req.params;
const { filter, maxResults, deltaFilter } = req.body;
const options: { maxResults?: number; deltaFilter?: any } = {};
const options: { maxResults?: number; deltaFilter?: DeltaFilter } = {};
if (maxResults) options.maxResults = maxResults;
if (deltaFilter) {
// Note: deltaFilter would need to be serialized/deserialized properly in a real implementation

View File

@ -1,13 +1,13 @@
import { apply } from 'json-logic-js';
import Debug from 'debug';
import { SchemaRegistry, SchemaID, ObjectSchema } from '../schema/schema';
import { Lossless, LosslessViewOne, LosslessViewMany } from '../views/lossless';
import { Lossless, LosslessViewOne, LosslessViewMany, CollapsedDelta } from '../views/lossless';
import { DomainEntityID } from '../core/types';
import { Delta, DeltaFilter } from '../core/delta';
import { DeltaFilter } from '../core/delta';
const debug = Debug('rz:query');
export type JsonLogic = Record<string, any>;
export type JsonLogic = Record<string, unknown>;
export interface QueryOptions {
maxResults?: number;
@ -182,8 +182,8 @@ export class QueryEngine {
* Convert a lossless view to a queryable object based on schema
* Uses simple resolution strategies for now
*/
private losslessViewToQueryableObject(view: LosslessViewOne, schema: ObjectSchema): Record<string, any> {
const obj: Record<string, any> = {
private losslessViewToQueryableObject(view: LosslessViewOne, schema: ObjectSchema): Record<string, unknown> {
const obj: Record<string, unknown> = {
id: view.id,
_referencedAs: view.referencedAs
};
@ -199,28 +199,31 @@ export class QueryEngine {
// Apply simple resolution strategy based on property schema type
switch (propertySchema.type) {
case 'primitive':
case 'primitive': {
// Use last-write-wins for primitives
const lastDelta = deltas.sort((a, b) => b.timeCreated - a.timeCreated)[0];
const primitiveValue = this.extractPrimitiveValue(lastDelta, propertyId);
obj[propertyId] = primitiveValue;
break;
}
case 'array':
case 'array': {
// Collect all values as array
const arrayValues = deltas
.map(delta => this.extractPrimitiveValue(delta, propertyId))
.filter(value => value !== null);
obj[propertyId] = arrayValues;
break;
}
case 'reference':
case 'reference': {
// For references, include the target IDs
const refValues = deltas
.map(delta => this.extractReferenceValue(delta, propertyId))
.filter(value => value !== null);
obj[propertyId] = refValues;
break;
}
default:
obj[propertyId] = deltas.length;
@ -234,12 +237,12 @@ export class QueryEngine {
/**
* Extract primitive value from a delta for a given property
*/
private extractPrimitiveValue(delta: any, propertyId: string): any {
// Look for the value in deltas that target this property
// The delta should have a 'value' pointer containing the actual value
private extractPrimitiveValue(delta: CollapsedDelta, _propertyId: string): unknown {
// Look for the value in collapsed pointers
// CollapsedPointer is {[key: PropertyID]: PropertyTypes}
for (const pointer of delta.pointers) {
if (pointer['value'] !== undefined) {
return pointer['value'];
if (pointer.value !== undefined) {
return pointer.value;
}
}
return null;
@ -248,11 +251,11 @@ export class QueryEngine {
/**
* Extract reference value (target ID) from a delta for a given property
*/
private extractReferenceValue(delta: any, propertyId: string): string | null {
private extractReferenceValue(delta: CollapsedDelta, _propertyId: string): string | null {
// For references, we want the value pointer that contains the reference ID
for (const pointer of delta.pointers) {
if (pointer['value'] !== undefined && typeof pointer['value'] === 'string') {
return pointer['value'];
if (pointer.value !== undefined && typeof pointer.value === 'string') {
return pointer.value;
}
}
return null;

View File

@ -7,7 +7,7 @@ import { Delta, DeltaFilter } from '../core/delta';
const debug = Debug('rz:storage-query');
export type JsonLogic = Record<string, any>;
export type JsonLogic = Record<string, unknown>;
export interface StorageQueryOptions {
maxResults?: number;
@ -25,7 +25,7 @@ export interface StorageQueryResult {
export interface StorageEntityResult {
entityId: DomainEntityID;
deltas: Delta[];
properties: Record<string, any>; // Resolved properties for filtering
properties: Record<string, unknown>; // Resolved properties for filtering
}
/**
@ -206,8 +206,8 @@ export class StorageQueryEngine {
/**
* Resolve entity properties from deltas for query filtering
*/
private resolveEntityProperties(deltas: Delta[], schema: ObjectSchema): Record<string, any> {
const properties: Record<string, any> = {};
private resolveEntityProperties(deltas: Delta[], schema: ObjectSchema): Record<string, unknown> {
const properties: Record<string, unknown> = {};
// Group deltas by property context
const propertyDeltas = new Map<string, Delta[]>();
@ -234,27 +234,30 @@ export class StorageQueryEngine {
// Apply simple resolution strategy based on property schema type
switch (propertySchema.type) {
case 'primitive':
case 'primitive': {
// Use last-write-wins for primitives
const lastDelta = propDeltas.sort((a, b) => b.timeCreated - a.timeCreated)[0];
properties[propertyId] = this.extractPrimitiveValue(lastDelta, propertyId);
break;
}
case 'array':
case 'array': {
// Collect all values as array
const arrayValues = propDeltas
.map(delta => this.extractPrimitiveValue(delta, propertyId))
.filter(value => value !== null);
properties[propertyId] = arrayValues;
break;
}
case 'reference':
case 'reference': {
// For references, include the target IDs
const refValues = propDeltas
.map(delta => this.extractReferenceValue(delta, propertyId))
.filter(value => value !== null);
properties[propertyId] = refValues;
break;
}
default:
properties[propertyId] = propDeltas.length;
@ -267,7 +270,7 @@ export class StorageQueryEngine {
/**
* Extract primitive value from a delta for a given property
*/
private extractPrimitiveValue(delta: Delta, propertyId: string): any {
private extractPrimitiveValue(delta: Delta, _propertyId: string): unknown {
for (const pointer of delta.pointers) {
if (pointer.localContext === 'value') {
return pointer.target;
@ -279,7 +282,7 @@ export class StorageQueryEngine {
/**
* Extract reference value (target ID) from a delta for a given property
*/
private extractReferenceValue(delta: Delta, propertyId: string): string | null {
private extractReferenceValue(delta: Delta, _propertyId: string): string | null {
for (const pointer of delta.pointers) {
if (pointer.localContext === 'value' && typeof pointer.target === 'string') {
return pointer.target;
@ -306,7 +309,7 @@ export class StorageQueryEngine {
/**
* Check if an entity matches a schema (basic validation)
*/
private entityMatchesSchema(properties: Record<string, any>, schema: ObjectSchema): boolean {
private entityMatchesSchema(properties: Record<string, unknown>, schema: ObjectSchema): boolean {
const requiredProperties = schema.requiredProperties || [];
for (const propertyId of requiredProperties) {

View File

@ -87,5 +87,5 @@ export interface DeltaQuery {
export interface StorageConfig {
type: 'memory' | 'leveldb' | 'sqlite' | 'postgres';
path?: string; // for file-based storage
options?: Record<string, any>;
options?: Record<string, unknown>;
}

View File

@ -2,7 +2,7 @@ import Debug from 'debug';
import { Level } from 'level';
import { Delta, DeltaID, DeltaFilter } from '../core/delta';
import { DomainEntityID } from '../core/types';
import { DeltaStorage, DeltaQueryStorage, DeltaQuery, StorageStats } from './interface';
import { DeltaQueryStorage, DeltaQuery, StorageStats } from './interface';
const debug = Debug('rz:storage:leveldb');
@ -26,7 +26,14 @@ export class LevelDBDeltaStorage implements DeltaQueryStorage {
}
}
private async ensureOpen(): Promise<void> {
if (this.db.status !== 'open') {
await this.db.open();
}
}
async storeDelta(delta: Delta): Promise<void> {
await this.ensureOpen();
debug(`Storing delta ${delta.id} to LevelDB`);
const batch = this.db.batch();
@ -63,11 +70,18 @@ export class LevelDBDeltaStorage implements DeltaQueryStorage {
}
async getDelta(id: DeltaID): Promise<Delta | null> {
await this.ensureOpen();
try {
const deltaJson = await this.db.get(`delta:${id}`);
// Handle case where LevelDB returns string "undefined" for missing keys
if (deltaJson === 'undefined' || deltaJson === undefined) {
return null;
}
return JSON.parse(deltaJson);
} catch (error) {
if ((error as any).code === 'LEVEL_NOT_FOUND') {
if ((error as { code?: string }).code === 'LEVEL_NOT_FOUND') {
return null;
}
throw error;
@ -75,10 +89,11 @@ export class LevelDBDeltaStorage implements DeltaQueryStorage {
}
async getAllDeltas(filter?: DeltaFilter): Promise<Delta[]> {
await this.ensureOpen();
const deltas: Delta[] = [];
// Iterate through all delta records
for await (const [key, value] of this.db.iterator({
for await (const [_key, value] of this.db.iterator({
gte: 'delta:',
lt: 'delta:\xFF'
})) {
@ -90,7 +105,7 @@ export class LevelDBDeltaStorage implements DeltaQueryStorage {
deltas.push(delta);
}
} catch (error) {
debug(`Error parsing delta from key ${key}:`, error);
debug(`Error parsing delta from key ${_key}:`, error);
}
}
@ -98,10 +113,11 @@ export class LevelDBDeltaStorage implements DeltaQueryStorage {
}
async getDeltasForEntity(entityId: DomainEntityID): Promise<Delta[]> {
await this.ensureOpen();
const deltaIds: string[] = [];
// Use entity index to find all deltas for this entity
for await (const [key, deltaId] of this.db.iterator({
for await (const [_key, deltaId] of this.db.iterator({
gte: `entity:${entityId}:`,
lt: `entity:${entityId}:\xFF`
})) {
@ -121,10 +137,11 @@ export class LevelDBDeltaStorage implements DeltaQueryStorage {
}
async getDeltasByContext(entityId: DomainEntityID, context: string): Promise<Delta[]> {
await this.ensureOpen();
const deltaIds: string[] = [];
// Use context index to find deltas for this specific entity+context
for await (const [key, deltaId] of this.db.iterator({
for await (const [_key, deltaId] of this.db.iterator({
gte: `context:${entityId}:${context}:`,
lt: `context:${entityId}:${context}:\xFF`
})) {
@ -144,13 +161,14 @@ export class LevelDBDeltaStorage implements DeltaQueryStorage {
}
async queryDeltas(query: DeltaQuery): Promise<Delta[]> {
await this.ensureOpen();
let candidateDeltaIds: Set<string> | null = null;
// Use indexes to narrow down candidates efficiently
if (query.creator) {
const creatorDeltaIds = new Set<string>();
for await (const [key, deltaId] of this.db.iterator({
for await (const [_key, deltaId] of this.db.iterator({
gte: `creator:${query.creator}:`,
lt: `creator:${query.creator}:\xFF`
})) {
@ -161,7 +179,7 @@ export class LevelDBDeltaStorage implements DeltaQueryStorage {
if (query.host) {
const hostDeltaIds = new Set<string>();
for await (const [key, deltaId] of this.db.iterator({
for await (const [_key, deltaId] of this.db.iterator({
gte: `host:${query.host}:`,
lt: `host:${query.host}:\xFF`
})) {
@ -173,7 +191,7 @@ export class LevelDBDeltaStorage implements DeltaQueryStorage {
if (query.targetEntities && query.targetEntities.length > 0) {
const entityDeltaIds = new Set<string>();
for (const entityId of query.targetEntities) {
for await (const [key, deltaId] of this.db.iterator({
for await (const [_key, deltaId] of this.db.iterator({
gte: `entity:${entityId}:`,
lt: `entity:${entityId}:\xFF`
})) {
@ -186,7 +204,7 @@ export class LevelDBDeltaStorage implements DeltaQueryStorage {
// If no index queries were used, scan all deltas
if (candidateDeltaIds === null) {
candidateDeltaIds = new Set<string>();
for await (const [key, value] of this.db.iterator({
for await (const [key, _value] of this.db.iterator({
gte: 'delta:',
lt: 'delta:\xFF'
})) {
@ -237,13 +255,14 @@ export class LevelDBDeltaStorage implements DeltaQueryStorage {
}
async getStats(): Promise<StorageStats> {
await this.ensureOpen();
let totalDeltas = 0;
const entities = new Set<DomainEntityID>();
let oldestDelta: number | undefined;
let newestDelta: number | undefined;
// Count deltas and track entities
for await (const [key, value] of this.db.iterator({
for await (const [_key, value] of this.db.iterator({
gte: 'delta:',
lt: 'delta:\xFF'
})) {
@ -267,7 +286,7 @@ export class LevelDBDeltaStorage implements DeltaQueryStorage {
newestDelta = delta.timeCreated;
}
} catch (error) {
debug(`Error parsing delta for stats from key ${key}:`, error);
debug(`Error parsing delta for stats from key ${_key}:`, error);
}
}
@ -300,15 +319,17 @@ export class LevelDBDeltaStorage implements DeltaQueryStorage {
// LevelDB-specific methods
async clearAll(): Promise<void> {
await this.ensureOpen();
debug('Clearing all data from LevelDB');
await this.db.clear();
}
async compact(): Promise<void> {
await this.ensureOpen();
debug('Compacting LevelDB');
// LevelDB compaction happens automatically, but we can trigger it
// by iterating through all keys (this is a simple approach)
for await (const [key] of this.db.iterator()) {
for await (const [_key] of this.db.iterator()) {
// Just iterating triggers compaction
}
}

View File

@ -1,7 +1,7 @@
import Debug from 'debug';
import { Delta, DeltaID, DeltaFilter } from '../core/delta';
import { DomainEntityID } from '../core/types';
import { DeltaStorage, DeltaQueryStorage, DeltaQuery, StorageStats } from './interface';
import { DeltaQueryStorage, DeltaQuery, StorageStats } from './interface';
const debug = Debug('rz:storage:memory');

View File

@ -1 +1 @@
MANIFEST-000006
MANIFEST-000020

View File

@ -1,3 +1,3 @@
2025/06/09-21:50:47.185401 7177213fe640 Recovering log #5
2025/06/09-21:50:47.301447 7177213fe640 Delete type=0 #5
2025/06/09-21:50:47.301483 7177213fe640 Delete type=3 #4
2025/06/09-22:08:03.381417 7d18dafbe640 Recovering log #19
2025/06/09-22:08:03.388931 7d18dafbe640 Delete type=3 #18
2025/06/09-22:08:03.389041 7d18dafbe640 Delete type=0 #19

View File

@ -1,3 +1,3 @@
2025/06/09-21:50:17.946302 7189167bf640 Recovering log #3
2025/06/09-21:50:17.971267 7189167bf640 Delete type=3 #2
2025/06/09-21:50:17.971333 7189167bf640 Delete type=0 #3
2025/06/09-22:06:22.826797 7d82d53fe640 Recovering log #17
2025/06/09-22:06:22.833921 7d82d53fe640 Delete type=0 #17
2025/06/09-22:06:22.833954 7d82d53fe640 Delete type=3 #16

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

View File

@ -1 +1 @@
MANIFEST-000036
MANIFEST-000241

View File

@ -1,3 +1,5 @@
2025/06/09-21:50:17.827319 7189167bf640 Recovering log #35
2025/06/09-21:50:17.847669 7189167bf640 Delete type=0 #35
2025/06/09-21:50:17.847721 7189167bf640 Delete type=3 #34
2025/06/09-22:08:03.351430 7d18da7bd640 Recovering log #240
2025/06/09-22:08:03.351481 7d18da7bd640 Level-0 table #242: started
2025/06/09-22:08:03.353466 7d18da7bd640 Level-0 table #242: 877 bytes OK
2025/06/09-22:08:03.359635 7d18da7bd640 Delete type=0 #240
2025/06/09-22:08:03.359683 7d18da7bd640 Delete type=3 #238

View File

@ -1,3 +1,5 @@
2025/06/09-21:50:17.802741 7189167bf640 Recovering log #33
2025/06/09-21:50:17.820142 7189167bf640 Delete type=3 #32
2025/06/09-21:50:17.820212 7189167bf640 Delete type=0 #33
2025/06/09-22:08:03.334848 7d18da7bd640 Recovering log #237
2025/06/09-22:08:03.334894 7d18da7bd640 Level-0 table #239: started
2025/06/09-22:08:03.337138 7d18da7bd640 Level-0 table #239: 855 bytes OK
2025/06/09-22:08:03.344340 7d18da7bd640 Delete type=0 #237
2025/06/09-22:08:03.344389 7d18da7bd640 Delete type=3 #235

Binary file not shown.

79
todo.md
View File

@ -64,56 +64,95 @@ This document tracks work needed to achieve full specification compliance, organ
- [ ] Test query performance at scale
- [ ] Add query result caching with invalidation
## Phase 4: Relational Features
## Phase 4: Delta Patterns & Query Traversal
### 4.1 Relational Schema Expression
- [ ] Design relational schema DSL
- [ ] Implement foreign key constraints
- [ ] Add relationship traversal in queries
- [ ] Implement join operations in lossy views
- [ ] Enable the skipped relational tests
### 4.1 Delta Pattern Recognition
- [ ] Define common delta patterns (authorship, membership, etc.)
- [ ] Create pattern matching utilities
- [ ] Build pattern validation (guidance, not enforcement)
- [ ] Document delta-as-relationship philosophy
### 4.2 Constraint Validation
- [ ] Add unique constraints
- [ ] Implement required field validation
- [ ] Add custom constraint functions
- [ ] Test constraint violations and error handling
### 4.2 Pattern-Aware Queries
- [ ] Extend QueryEngine with pattern traversal methods
- [ ] Add multi-perspective query support
- [ ] Implement temporal relationship queries
- [ ] Create relationship history and timeline queries
## Phase 5: Advanced Features
### 4.3 Pattern-Based Resolvers
- [ ] Build pattern-aware resolvers for common relationships
- [ ] Implement competing relationship resolution
- [ ] Add missing relationship detection
- [ ] Create resolver composition utilities
### 5.1 View Optimizations
### 4.4 Schema-as-Deltas (Meta-Schema System)
- [ ] Define schema entities that are stored as deltas in the system
- [ ] Implement schema queries that return schema instances from lossless views
- [ ] Create schema evolution through delta mutations
- [ ] Add temporal schema queries (schema time-travel)
- [ ] Build schema conflict resolution for competing schema definitions
- [ ] Test runtime schema updates and their effects on existing data
## Phase 5: GraphQL API Layer
### 5.1 GraphQL Schema Generation
- [ ] Generate GraphQL schemas from Rhizome schemas
- [ ] Map delta patterns to GraphQL relationships
- [ ] Support multiple schema perspectives (published, draft, etc.)
- [ ] Add GraphQL directives for perspective control
### 5.2 GraphQL Resolvers
- [ ] Implement resolvers that traverse delta patterns
- [ ] Add support for nested relationship queries
- [ ] Handle temporal queries (time-travel via arguments)
- [ ] Implement competing value resolution in GraphQL context
### 5.3 GraphQL Mutations
- [ ] Create mutations that generate appropriate deltas
- [ ] Handle relationship creation/updates via delta generation
- [ ] Implement negation mutations for "deletes"
- [ ] Add transaction support for multi-delta mutations
### 5.4 GraphQL Subscriptions
- [ ] Stream delta updates as GraphQL subscriptions
- [ ] Filter subscriptions by pattern/entity
- [ ] Support real-time relationship updates
- [ ] Add perspective-aware subscription filtering
## Phase 6: Performance & Optimization
### 6.1 View Optimizations
- [ ] Implement incremental view updates
- [ ] Add view materialization strategies
- [ ] Create view caching layer
- [ ] Add partial view generation
### 5.2 Network Resilience
### 6.2 Network Resilience
- [ ] Add network partition handling
- [ ] Implement delta retry mechanisms
- [ ] Add peer health monitoring
- [ ] Test split-brain scenarios
### 5.3 Performance & Scale
### 6.3 Performance & Scale
- [ ] Add benchmarks for large datasets
- [ ] Implement delta pruning strategies
- [ ] Add memory-efficient view generation
- [ ] Create performance regression tests
## Phase 6: Developer Experience
## Phase 7: Developer Experience
### 6.1 Better TypeScript Support
### 7.1 Better TypeScript Support
- [ ] Improve TypedCollection type inference
- [ ] Add stricter schema typing
- [ ] Create type guards for delta operations
- [ ] Add better IDE autocomplete support
### 6.2 Debugging & Monitoring
### 7.2 Debugging & Monitoring
- [ ] Add delta stream visualization
- [ ] Create conflict resolution debugger
- [ ] Add performance profiling hooks
- [ ] Implement comprehensive logging
### 6.3 Documentation
### 7.3 Documentation
- [ ] Document schema definition format
- [ ] Create resolver implementation guide
- [ ] Add query language documentation