Structured Data Roadmap¶

This document outlines the roadmap for implementing first-class support for structured data and system interaction within the Endo shell. It builds upon the F# syntax extensions, particularly records and union types.

Vision¶

The goal is to evolve Endo from a shell that primarily deals with streams of text to one that can natively understand and manipulate streams of structured objects. This enables more powerful, reliable, and composable automation, akin to PowerShell but with a functional approach.

Instead of parsing text with awk, sed, and grep:

# Instead of: ps aux | grep 'endo' | awk '{print $2}'
ps |> filter (_.command == "endo") |> map _.pid

Phase 6.1: Core Infrastructure -- Complete¶

CoreVM Support for Records and Unions¶

Discriminated unions and record types are fully implemented in the CoreVM, supporting efficient creation, manipulation, and reference counting.

Structured Command Interface¶

An internal C++ interface (StructuredCommand) allows commands to declare that they produce structured data and advertise the type of their output. Platform-abstracted via ProcessProvider interface for cross-platform support.

Structured Data Wrapper¶

Four data source commands for ad-hoc structured parsing:

• open-json / open-csv -- read from files
• from-json / from-csv -- read from pipe input

These support inline record type definitions and named type references:

# Inline type definition
open-json "file.json" as { name: string; age: int }

# Named type reference
type Person = { name: str; age: int }
open-csv "people.csv" as Person

# Pipe-based source
curl api/users | from-json as { name: string; id: int } |> map _.name

Phase 6.2: Built-in Structured Commands -- Complete¶

ls¶

Returns a stream of FileInfo records.

ls |> filter (_.size > 1000000) |> sortBy _.size |> map _.name

ps¶

Returns a stream of ProcessInfo records.

ps |> filter (_.cpu > 5.0) |> sortBy _.cpu

jobs¶

Returns a stream of JobInfo records.

jobs |> filter (_.state == "Running") |> map _.command

Phase 6.3: Extensible Command Discovery¶

Phase 6.3a: Output Recognition Files -- Complete¶

Declarative YAML definitions teach Endo how to parse external command output without modifying the tools themselves.

• YAML definition file format with JSON and fields parser types
• Variant matching by command arguments with priority system
• command_to_run override to redirect commands to structured-output flags
• Definition file search paths: ~/.config/endo/definitions/, system-wide, and bundled
• Pipeline integration via StructuredPipelineSourceExpr AST node
• Bundled definitions for docker ps, docker images, git log, git status

command: "docker"
variants:
  - name: "ps-json"
    matches:
      - ["ps"]
      - ["ps", "-a"]
    priority: 10
    command_to_run: "docker ps --format json {args}"
    parser:
      type: "json"
      format: "lines"

docker ps |> filter (_.status |> contains "Up") |> map _.names

See Structured Output Recognition for the full specification.

Phase 6.3b: Self-Describing Commands -- Planned¶

For new commands that opt into structured output:

• Discovery via my-command --endo-schema convention
• Libraries for C++, Rust, Go, Python to simplify writing structured commands
• Schema caching for performance

Phase 6.4: Structured Data Pipeline Integration -- Complete¶

Record-Aware List Operations¶

Standard F# higher-order functions work directly with record-typed lists. No special-purpose verbs are needed:

Placeholder Lambda Sugar¶

Parser-level sugar for concise field access:

• _.field desugars to fun __x -> __x.field
• _.field == value desugars to fun __x -> __x.field == value
• _ + 1 desugars to fun __x -> __x + 1

Table Rendering¶

Lists of records are automatically rendered as tables when displayed:

• Auto-detect column widths with terminal-width-aware shrinking
• Three styles: Bordered (Unicode box-drawing), Compact, Plain
• Auto-style selection: Bordered with color for terminals, Plain for pipes

Suggested Future Structured Commands¶

Candidates for built-in structured output: