tag) html "article" --inner # Get inner HTML (excludes

tag) html "li:nth-child(2)" # Select second list item html "p:not(.intro)" # Select paragraphs without .intro class html "a[href$='.pdf']" --attr href --all # All PDF links html "a" --html --base=https://example.com/ # Extract links with absolute URLs ``` ### `jq` - JSON Processor **Usage:** `jq [filter]` Processes JSON input using the full [jq](https://jqlang.github.io/jq/) filter language (powered by jq-web/WASM). Supports pipes, functions, conditionals, object construction, array slicing, and more. Output is pretty-printed by default. **Examples:** ``` jq . jq ".users[0].name" jq ".users | length" jq "[.[] | .name]" jq ".items | map(select(.active))" jq ".users | sort_by(.age) | reverse" jq '{name: .first, age: .years}' ``` ### `ls` - List Temp Files **Usage:** `ls [pattern] [flags]` **Flags:** - `-l`, `--long` - Show detailed info (size, created, expires) - `-S`, `--sort=` - Sort by `name`, `size`, `time`, or `expires` (default: time) - `-r`, `--reverse` - Reverse sort order - `-j`, `--json` - Output as JSON array **Behavior:** - Lists temp files in `/tmp/` storage (scoped to your API key) - If pattern provided: pattern is date-expanded (same as tmp/tee; use `X-User-Time` header) then matched (supports `*` and `?` wildcards) - If no pattern: lists all temp files - Default: one file path per line (format: `/tmp/`) - Long format: path, size, created, expires (tab-separated) - JSON format: array of objects with path, size, created_at, expires_at **Wildcards:** - `*` - Matches any sequence of characters - `?` - Matches a single character **Examples:** ``` ls ls --long ls --sort=size --reverse ls --json ls "*.txt" ls "/tmp/test?.log" ls "/tmp/backup-*" ``` **Note:** Only works with `/tmp/` files (scoped to your API key). Pattern can include `/tmp/` prefix (stripped). Wildcards on filename only. Empty output if no match. ### `mv` - Rename Temp File or Saved Pipeline **Usage:** `mv ` **Behavior:** - **Resolution:** First argument is resolved in order: 1) temp file (wildcards `*`, `?` and date expansion supported; exactly one match required for mv), 2) saved pipeline by name (exact, case-sensitive), else fail. Temp file wins when both a temp file and a saved pipeline match the same name. - **Temp branch:** Dest = temp path (date expansion only; overwrites if exists). - **Saved-pipeline branch:** Dest = new pipeline name (validated like save; renames the saved pipeline). **Examples:** ``` mv draft.txt report-%Y-%m-%d.txt mv report-%Y-%m-%d.txt archive-%Y-%m-%d.txt mv report-%Y-%m-%d-*.log archive-%Y-%m-%d.log mv MyPipeline RenamedPipeline ``` **API:** `POST /mv` - dedicated endpoint (source/dest as query params; temp files and saved pipelines). For temp-only: `PATCH /tmp/:path` with **text/plain** body = new filename (date expansion via `X-User-Time`). No body = extend expiration. Add **`?cmd=cp`** to copy instead of rename (source remains). Pipeline form (e.g. `mv foo bar` in a pipeline) supports both temp files and saved pipelines by name. ### `cp` - Copy Temp File or Saved Pipeline **Usage:** `cp ` **Behavior:** Same resolution as `mv`: temp file first (wildcards, date expansion; exactly one match), then saved pipeline by name. Source file or pipeline remains; dest is temp path or new pipeline name. Copying a saved pipeline creates a new pipeline (new ULID) with the same content. **Examples:** ``` cp draft.txt report-%Y-%m-%d.txt cp report-%Y-%m-%d.txt archive-%Y-%m-%d.txt cp report-%Y-%m-%d-*.log archive-%Y-%m-%d.log cp MyPipeline MyPipelineBackup ``` **API:** `POST /cp` - dedicated endpoint (source/dest as query params; temp files and saved pipelines). For temp-only: `PATCH /tmp/:path?cmd=cp` with **text/plain** body = new filename (copy; source remains). Pipeline form supports both temp files and saved pipelines by name. ### `rm` - Remove Temp File or Saved Pipeline **Usage:** `rm ` **Behavior:** - **Resolution:** First argument is resolved in order: 1) temp file (wildcards `*`, `?` and date expansion supported; deletes all matches), 2) saved pipeline by name (exact, case-sensitive), else fail. Temp file wins when both exist. - Returns the deleted path(s) or name as plaintext; 404 with JSON error if neither temp nor saved pipeline matches. **Examples:** ``` rm oldfile.txt rm backup-%Y-%m-%d.log rm "*.tmp" rm MySavedPipeline ``` **API:** `POST /rm?path=` - dedicated endpoint (temp files and saved pipelines; supports wildcards and date expansion for temp). Pipeline form supports both. For temp-only: `DELETE /tmp/`. **Note:** Pipeline form supports both temp files and saved pipelines. For bulk delete of temp files, use `ls` + `xargs rm` or multiple `rm` commands. ### `mail` - Send Email via SMTP **Usage:** `mail -s "subject" ` or `mail --subject "subject" --to ` **Flags:** - `-s`, `--subject` - Email subject - `--to ` - Recipient email address (required) - `--html` - Transform piped input to HTML (detects markdown/text/HTML and converts markdown to HTML) - `--test` - Output raw SMTP message (RFC 5322 format) instead of sending (for debugging) **Behavior:** - Takes piped input as email body - Sends email via SMTP (configured on the API token) - With `--html` flag: Automatically detects if input is text, markdown, or HTML - If markdown: Converts to HTML - If HTML: Leaves as-is - If text: Leaves as-is (plain text email) - With `--test` flag: Outputs the raw email message with all headers (From, To, Subject, Date, Message-ID, MIME-Version, Content-Type) without sending **SMTP Configuration:** SMTP settings are stored on each API token, allowing each customer to configure their own email service: - `smtp_host` - SMTP server hostname (required) - `smtp_port` - SMTP server port (default: 25) - `smtp_user` - SMTP username (optional) - `smtp_pass` - SMTP password (optional) - `smtp_from` - From email address (default: piped@localhost) **Examples:** ``` echo "Alert: System is down" | mail -s "System Alert" admin@example.com cat report.txt | mail --subject "Daily Report" --to team@example.com echo "# Hello\nThis is **markdown**" | mail --to=user@example.com --subject="Report" --html echo "Test body" | mail --to=user@example.com --subject="Test" --test ``` **Note:** Requires SMTP configuration to be set on the API token (except with `--test` flag). Email body comes from piped input. ### `project` - Export/Import Project as YAML **Usage:** `project [name] [--export=]` (export) or `cat project.yaml | project [flags]` (import) **Flags:** - `[name]` - Project name to include in export (optional) - `--export=` - Export only specified sections. Values: `all`, `pipelines` (or `saved`), `exports`, `aliases`, `tools`, or comma-separated (e.g., `tools,exports`) - `--replace=` - Clear before import. Values: `all`, `pipelines` (or `saved`), `exports`, `aliases`, `tools`, or comma-separated (e.g., `exports,aliases`) **Behavior:** - Without input: Exports current project (pipelines, exports, aliases, tools) as YAML - With `--export`: Exports only the specified sections - Without input + `--replace`: Clears specified data (fresh blank environment) - With YAML input: Imports project data (additive by default, merges with existing) - With YAML input + `--replace`: Clears specified data types before import **YAML Format:** ```yaml project: name: optional-name pipelines: - name: my-pipeline pipeline: echo hello | grep h exports: - name: MY_VAR value: my-value aliases: - alias: greet command: echo hello tools: - name: piped type: mcp spec: '{"mcpServers":{"piped":{"type":"http","url":"https://piped.sh/mcp","headers":{"X-API-Key":"$PIPED_API_KEY"}}}}' ``` **Examples:** ``` # Export project as YAML project project "My Project" # Export only specific sections project --export=tools project --export=pipelines,exports # Import project (additive) cat backup.yaml | project # Import with replace cat backup.yaml | project --replace=all cat backup.yaml | project --replace=pipelines cat backup.yaml | project --replace=saved,aliases # Clear all data (fresh blank environment) project --replace=all # Clear only tools project --replace=tools # Export to file project | tee /tmp/backup.yaml ``` **Note:** Useful for backup/restore, sharing configurations, and programmatic project management. The `replace` and `export` options support `saved` as an alias for `pipelines`. ### `awk` - Text Processing **Usage:** `awk [program]` or `awk -F [program]` Process text line-by-line using awk programs. Supports field splitting, pattern matching, built-in variables (NR, NF, $0, $1, etc.), and text transformations. **Flags:** - `-F `, `--field-separator=` - Set field separator (default: whitespace) **Examples:** ``` awk '{print $1}' awk -F, '{print $2}' awk '/^error/' awk '{print NR, $0}' awk -F: '{print $1, $3}' echo "alice,30\nbob,25" | awk -F, '$2 > 27 {print $1}' ``` ### `deno` - Execute TypeScript/JavaScript **Usage:** `deno /tmp/` or `deno ` or `deno /tmp/ --allow-net=` Execute a TypeScript or JavaScript script stored in temp file storage or as a saved pipeline using Deno. The script receives piped input via stdin and returns output via stdout. `deno.land` is always accessible so scripts can import from the Deno module registry. No filesystem access or subprocess execution. **Parameters:** - `/tmp/` or `` - Temp file path or saved pipeline name of the script to execute (required). Resolution: temp file first, then saved pipeline by name. - `--allow-net=` - Comma-separated additional hosts to allow outbound network (e.g. `api.openai.com`). `deno.land` is always included. Localhost and private IPs are always blocked. **Script pattern:** ```ts const input = await new Response(Deno.stdin.readable).text(); // transform input... await Deno.stdout.write(new TextEncoder().encode(result)); ``` **Examples:** ``` echo 'hello world' | deno /tmp/transform.ts cat /tmp/data.json | deno /tmp/process.ts cat /tmp/data.json | deno /tmp/fetch.ts --allow-net=api.openai.com echo 'hello world' | deno my-transform.ts ``` **Workflow:** ``` # 1. Write your script to temp storage echo 'const input = await new Response(Deno.stdin.readable).text(); console.log(input.toUpperCase())' | tee /tmp/upper.ts # 2. Run it echo "hello world" | deno /tmp/upper.ts # Output: HELLO WORLD ``` ### `sed` - Stream Editor **Usage:** `sed [expression]` (expression required, e.g. `s/old/new/` or `/pattern/d`) **Expressions:** - `s/pattern/replacement/flags` - Substitution (g = global, i = case-insensitive, p = print) - `/pattern/d` - Delete lines matching pattern - `Nd` - Delete line N (e.g., `1d` deletes line 1) - `N,Md` - Delete lines N through M inclusive (e.g., `1,3d` deletes lines 1-3) - `/pattern/p` - Print lines matching pattern (use with `-n` flag) - `Np` - Print line N - `N, Mp` - Print lines N through M - `Na/text` - Append text after line N - `N,Ma/text` - Append text after lines N through M - `Ni/text` - Insert text before line N - `N,Mi/text` - Insert text before lines N through M - `Nc/text` - Change/replace line N with text - `N,Mc/text` - Change/replace lines N through M with text **Flags:** - `-n`, `--quiet` - Suppress default output (only print explicitly requested lines) **Substitution Special Characters:** - `&` - Represents the matched text - `\1`, `\2`, etc. - Backreferences to capture groups - `\n` - Newline - `\t` - Tab **Examples:** ``` sed "s/old/new/g" sed "s/\\d+/NUMBER/g" sed "/error/d" sed "1d" # Delete first line sed "1,3d" # Delete lines 1-3 sed "s/hello/[&]/g" # Wrap matches in brackets sed "s/(\\w+) (\\w+)/\\2, \\1/" # Swap two words using backreferences sed "s/old/new/g;s/foo/bar/g" # Multiple expressions sed "2a/INSERTED" # Append text after line 2 sed "2i/INSERTED" # Insert text before line 2 sed "2c/CHANGED" # Replace line 2 with text sed "/pattern/p" --quiet # Print only matching lines ``` ### `sha256` - Compute SHA-256 Hash **Usage:** `sha256` **Behavior:** - Computes SHA-256 hash of input - Outputs hexadecimal hash string - No arguments needed **Examples:** ``` cat data.txt | sha256 echo "hello world" | sha256 cat url | sha256 | tee /tmp/hash ``` **Note:** Useful for change detection, checksums, and monitoring. Compare hashes to detect when content changes. ### `md5` - Compute MD5 Hash **Usage:** `md5` **Behavior:** - Computes MD5 hash of input - Outputs hexadecimal hash string - No arguments needed **Examples:** ``` cat data.txt | md5 echo "hello world" | md5 cat url | md5 | tee /tmp/hash ``` **Note:** Use for compatibility or speed; for integrity/security use `sha256`. ### `sleep` - Delay (Pipeline Only) **Usage:** `sleep ` **Arguments:** - `` (required): Delay in seconds. Integer or decimal (e.g. `2`, `0.5`). Maximum 10 seconds. **Behavior:** - Waits for the given number of seconds (non-blocking), then passes input through unchanged. - Pipeline-only (no standalone HTTP endpoint). - Useful for rate limiting or adding delay between stages. **Examples:** ``` echo hello | sleep 1 | cat cat data | sleep 0.5 | curl https://example.com/webhook --method POST --data "$(cat)" ``` ### `tool` - Manage Tools (OpenAPI Specs, MCP Configs) **Usage:** `tool [name] [--delete] [--type=]` Manage registered tools (OpenAPI specs and MCP server configs) used by the `mcp` command. **Behavior:** - `tool` (no args) — list all tools (tab-separated: name, type) - `tool ` (no stdin) — show tool spec - `cat spec | tool ` (with stdin) — create or update tool (auto-detects type from spec, or use `--type=openapi` / `--type=mcp`) - `tool --delete` — delete tool **Flags:** - `--type=` - Explicit tool type: `openapi` or `mcp`. Auto-detected from spec content if omitted. - `--delete` - Delete the named tool. **Examples:** ``` tool tool my-api cat /tmp/spec.yaml | tool my-api cat /tmp/spec.yaml | tool --type=openapi my-api tool my-api --delete ``` **Note:** Tools are token-scoped and encrypted at rest. Not available in the public playground. Use `mcp` command to call tools registered as MCP servers. ### `mcp` - Call Tools on Remote MCP Servers **Usage:** `mcp

[json-args]` **Arguments:** - `` (required): Registered tool name (e.g. `piped`) - `` (required): MCP tool name from the server's tool list (e.g. `execute_pipeline`) - `[json-args]` (optional): JSON object of arguments. If omitted and stdin has content, stdin is passed as the `input` argument. **Behavior:** - Calls a tool on a registered MCP (Model Context Protocol) server - The tool must be registered with `type: "mcp"` via the tools API - Auth credentials (`$EXPORT` references in the MCP server config headers) are resolved internally — they never appear in pipeline text - Uses JSON-RPC protocol internally (`tools/call`) - Returns the tool's text output - On error, returns the MCP server's error message with HTTP 502 **Registering an MCP tool:** ``` POST /tools/piped Content-Type: application/json X-API-Key: { "type": "mcp", "spec": "{ \"mcpServers\": { \"piped\": { \"type\": \"http\", \"url\": \"https://piped.sh/mcp\", \"headers\": { \"X-API-Key\": \"$PIPED_API_KEY\" } } } }" } ``` **Examples:** ``` # Call execute_pipeline on the "piped" MCP server mcp piped execute_pipeline '{"pipeline":"ls"}' # Read a temp file mcp piped read_file '{"path":"/tmp/data.csv"}' # Pipe output to other commands mcp piped execute_pipeline '{"pipeline":"echo hello"}' | grep hello # Pass stdin as the "input" argument (when no json-args given) echo "hello world" | mcp piped execute_pipeline ``` **Note:** Requires authentication. Not available in the public playground. The `mcp` command does not consume AI tokens — it's a direct HTTP call to the MCP server. Only HTTP-based MCP servers are supported (not stdio). Use `$EXPORT` variables for API keys in the server config headers. ### `man` - Show Command Help **Usage:** `man [command]` **Behavior:** - With command: Show manual/help for that pipeline command (content from config/man.yaml). - Without command: List available commands. - **GET** `/man?cmd=` or **POST** `/man` with plaintext body = command name. **Examples:** ``` man sleep man cat man ``` ### `sort` - Sort Lines **Usage:** `sort [flags]` **Flags:** - `-n`, `--numeric` - Numeric sort - `-r`, `--reverse` - Reverse order - `-u`, `--unique` - Remove duplicates **Examples:** ``` sort sort -n sort -r -u ``` ### `source` - Execute Pipeline from Temp Storage or Saved Pipeline **Usage:** `source /tmp/` or `source ` **Arguments:** - `/tmp/` or `` (required): Temp file path or saved pipeline name. Resolution: temp → saved pipeline by name. Paths starting with `/tmp/` are temp-only. Saved pipeline names cannot start with `/tmp` (reserved). **Behavior:** - Reads pipeline definition from a temp file or saved pipeline - Executes the fetched pipeline with stdin as input - Returns the output of the pipeline execution - **JS/TS auto-delegation:** If the path ends in `.ts` or `.js`, `source` automatically delegates to the `deno` command instead of executing the content as a pipeline. This means you can `source my-script.ts` to run TypeScript/JavaScript directly — no hashbangs or special syntax needed. - To execute a pipeline from a URL: `curl | tee /tmp/pipeline.txt && source /tmp/pipeline.txt` **Pipeline Execution:** - The fetched content is treated exactly as if posted directly to the root `/` endpoint - Supports all pipeline features: multiple commands chained with `|`, sequential statements, nested `source` commands - Stdin from previous command becomes the initial input for the fetched pipeline **Examples:** ``` # Execute pipeline from temp storage cat data.txt | source /tmp/my-pipeline.txt # Execute saved pipeline by name cat data.txt | source MyPipeline # Run a TypeScript saved pipeline (auto-delegates to deno) echo "hello" | source my-transform.ts # Run a JS temp file (auto-delegates to deno) echo "hello" | source /tmp/process.js # Store pipeline and reuse it echo "cat | grep pattern | sort" | tee /tmp/my-pipeline.txt cat data.txt | source /tmp/my-pipeline.txt # Fetch pipeline from URL, store it, then execute curl https://example.com/pipelines/process.txt | tee /tmp/process.txt cat data.txt | source /tmp/process.txt ``` **Use Cases:** - **Reusable pipelines**: Store common pipelines in temp storage or as saved pipelines - **Pipeline composition**: Build complex pipelines from simpler ones - **Maintainability**: Update a saved pipeline in one place, affects all uses - **TypeScript/JavaScript scripts**: Save a `.ts` or `.js` script and `source` it — runs via Deno automatically **Note:** Use `curl` to fetch pipelines from URLs, then `tee` to store them, and `source` to execute. ### `tail` - Output Last N Lines or Bytes **Usage:** `tail [-n N]` or `tail -N` or `tail --lines=N` (works on stdin, no filename argument) **Flags:** - `-n N`, `--lines=N` - Number of lines to output (default: 10) - `-n +N` - Start from line N onwards (e.g., `+5` means from line 5 to end) - `-c N`, `--bytes=N` - Number of bytes instead of lines - `-c +N` - Start from byte N onwards **Examples:** ``` cat file.txt | tail cat file.txt | tail -5 cat file.txt | tail -n 10 cat file.txt | tail +5 cat file.txt | tail -c 100 cat file.txt | tail -c +50 cat file.txt | tail --lines=20 ``` **Note:** Works on piped input only (no filename argument). Outputs last N lines. Default is 10 lines if not specified. ### `tee` - Write to Temp Storage **Usage:** `tee /tmp/ [ /tmp/ ... ]` **Flags:** - `--append`, `-a` - Append to existing file (default: overwrite) - `--quiet`, `-q` - Don't output file contents to stdout (quiet mode) - `--expire `, `-e ` - Set expiration time. Supports suffixes: `s` (seconds), `m` (minutes), `h` (hours), `d` (days). A bare number defaults to hours. Max: 168h (7d). Default: 36h. **Examples:** ``` cat data | tee /tmp/step1 cat data | tee /tmp/step1 /tmp/backup cat data | tee /tmp/step1 | grep pattern cat data | tee /tmp/log --append cat data | tee -e 2h /tmp/short-lived cat data | tee --expire 7d /tmp/week-long cat data | tee -e 30m /tmp/brief cat data | tee -q /tmp/quiet # Write to file without outputting to stdout ``` **Note:** - Writes to temp storage AND outputs to stdout (like Unix `tee`) - Use `-q` or `--quiet` flag to suppress stdout output (file is still written) - Files are scoped to your API key (private to your account) - Files are compressed (gzip) and encrypted (AES-256-GCM) before storage using your API key - Files expire after 36 hours by default (configurable per token or per-file with `--expire`) - Maximum expiration is 168 hours (7 days) - Files persist even if you regenerate your API key (ULID-based storage) ### `touch` - Create or Refresh Temp File Expiry **Usage:** `touch /tmp/ [path2 ...] [--expire 36]` Like Unix `touch`: creates an empty temp file if the path does not exist, or refreshes its expiration if it does. Content of existing files is unchanged. **Flags:** - `--expire `, `-e ` - Expiration time. Same format as `tee`: bare number (hours) or suffix `s`/`m`/`h`/`d`. Default: 36h. Max: 168h (7d). **Examples:** ``` touch /tmp/step1 touch /tmp/a /tmp/b --expire 2h touch /tmp/marker --expire 7d touch -e 30m /tmp/brief ``` **Note:** - Path(s) support date expansion (e.g. `/tmp/log-%Y-%m-%d.txt`); use `X-User-Time` header for expansion. - Default expire is 36 hours (same as `tee`). Use `--expire` to set a different TTL for new or refreshed files. ### `find` - Find Temp Files Matching Criteria **Usage:** `find /tmp/ [-name | -regex | -iregex ] [-mtime ]` Find temp files matching specified criteria. Similar to Unix `find` command. **Options:** - `path` - Starting directory path (e.g., `/tmp/blah/`). Finds files under this "directory" (temp files can have `/` in names to simulate folders). If omitted, searches all temp files. - `-name ` - Filter by filename pattern using glob matching (`*` and `?` wildcards). Pattern matches against filename or full path. - `-regex ` - Filter by regex pattern (case-sensitive). Pattern is anchored to start and end (like Unix find). Regex patterns are validated for ReDoS safety. - `-iregex ` - Filter by regex pattern (case-insensitive). Pattern is anchored to start and end (like Unix find). Regex patterns are validated for ReDoS safety. - `-mtime ` - Filter by modification time (uses `created_at` field, which is updated by `touch` command). **Note:** Only one of `-name`, `-regex`, or `-iregex` can be specified. **mtime format:** - `+N` - Files modified more than N time units ago (older than N) - `-N` - Files modified less than N time units ago (newer than N, within last N) - `N` - Files modified within the last N time units - Units: `d` (days), `h` (hours), `m` (minutes), `s` (seconds) **Examples:** ``` find /tmp/blah/ find /tmp/blah/ -name "*.txt" find /tmp/blah/ -regex ".*\.txt$" find /tmp/blah/ -iregex ".*\.TXT$" find /tmp/ -mtime +1d find /tmp/blah/ -name "*.txt" -mtime +1d find /tmp/blah/ -regex ".*\.txt$" -mtime +1d find /tmp/ -mtime -2h find /tmp/logs/ -mtime 1d ``` **Note:** - Path supports date expansion (e.g. `/tmp/log-%Y-%m-%d/`); use `X-User-Time` header for expansion. - Output: one file path per line (format: `/tmp/`), sorted alphabetically. - Modification time uses `created_at` field, which is updated when files are touched. - Regex patterns are validated for ReDoS (Regular Expression Denial of Service) safety. Unsafe patterns will be rejected with an error. ### `test` - Test File Properties, String Comparisons, and Integer Comparisons **Usage:** `test [path]` or `test ` (pipeline mode) or `test -n ` / `test -o ` or `test ` or `test ` or `test ` or `[[ expr ]]` **File Conditions:** - `-f`, `--file` - File exists and is a regular file (file mode only) - `-e`, `--exists` - File exists (any type, file mode only) - `-s`, `--size` - Non-empty (file exists and size > 0, OR piped input has content) - `-z`, `--zero` - Empty (file exists and size == 0, OR piped input is empty) - `-n`, `--newer` - Path is newer than ref (path created_at > ref). Requires `ref` and `path`. Ref = temp file path (exactly one) or date description (e.g. "1 hour ago", "yesterday"). - `-o`, `--older` - Path is older than ref (path created_at < ref). Same ref/path semantics as `-n`. **String Tests:** - `string` - True if string is not the null (empty) string - `s1 = s2` - True if strings s1 and s2 are identical - `s1 != s2` - True if strings s1 and s2 are not identical - `s1 < s2` - True if s1 comes before s2 based on binary value of characters - `s1 > s2` - True if s1 comes after s2 based on binary value of characters **Integer Tests:** - `n1 -eq n2` - True if integers n1 and n2 are algebraically equal - `n1 -ne n2` - True if integers n1 and n2 are not algebraically equal - `n1 -gt n2` - True if n1 is algebraically greater than n2 - `n1 -ge n2` - True if n1 is algebraically greater than or equal to n2 - `n1 -lt n2` - True if n1 is algebraically less than n2 - `n1 -le n2` - True if n1 is algebraically less than or equal to n2 **File Mode:** (with path argument) ``` test -f /tmp/file.txt test -s /tmp/data.txt test -z /tmp/empty.txt test -n "1 hour ago" /tmp/out test -o /tmp/ref /tmp/file ``` **Pipeline Mode:** (no path argument, tests piped input) ``` echo hello | test -s && echo "has content" echo | test -z && echo "is empty" ``` **String/Integer Comparison Mode:** ``` test hello # non-null string → true test foo = foo && echo "match" # string equality test foo != bar && echo "different" # string inequality test abc < def && echo "before" # string ordering test 10 -gt 5 && echo "bigger" # integer greater than test 5 -le 10 && echo "ok" # integer less or equal ``` **Behavior:** - Returns HTTP 200 with `"true"` output if condition is true - Returns HTTP 200 with empty output if condition is false - Returns HTTP 400 for invalid syntax/path/operator - Works seamlessly with `&&` and `||` operators (non-empty output = success, empty output = failure) **Examples:** ``` # File existence checks test -f /tmp/file.txt && cat /tmp/file.txt test -f /tmp/file.txt || echo "File not found" # File size checks test -s /tmp/data.txt && echo "File has content" test -z /tmp/empty.txt && echo "File is empty" # Pipeline mode (test piped input) echo hello | test -s && echo "has content" echo | test -z && echo "is empty" # String comparisons test hello # true (non-null) test foo = foo && echo "match" test foo != bar && echo "different" test abc < def && echo "abc before def" # Integer comparisons test 10 -gt 5 && echo "10 > 5" test 5 -eq 5 && echo "equal" test 3 -le 5 && echo "3 <= 5" # Combined with other commands cat /tmp/data.txt | test -s && grep pattern /tmp/data.txt || echo "File is empty" # Time comparisons (-n newer, -o older; ref = file or date string) test -n "1 hour ago" /tmp/out && echo "recent" test -o /tmp/ref /tmp/file && echo "file is older than ref" ``` **`[[ ]]` Syntax Sugar:** `[[ expr ]]` is equivalent to `test expr`. No quoting of arguments is required. Examples: ``` [[ -f /tmp/file.txt ]] && cat /tmp/file.txt [[ foo = foo ]] && echo "match" [[ 10 -gt 5 ]] && echo "bigger" echo hello | [[ -s ]] && echo "has content" ``` **Note:** The `test` command is designed for conditional logic with `&&` and `||`. It returns `"true"` for success (non-empty output) and empty string for failure, making it work naturally with the output-based success/failure model. For `-n`/`-o`, ref must resolve to exactly one file when it is a path; invalid date ref returns 400. Integer operators require integer operands (returns 400 otherwise). ### `tr` - Translate Characters **Usage:** `tr --from= --to=` or `tr -d ` or `tr -s ` or `tr -c ` **Flags:** - `--from`, `--to` - Character sets for translation - `-d`, `--delete` - Delete characters - `-s`, `--squeeze` - Squeeze repeated characters (optionally specify which char) - `-c`, `--complement` - Translate/delete all except specified set - `-t`, `--truncate` - Truncate 'to' set to length of 'from' set **Character Classes:** - `[:lower:]` - Lowercase letters (a-z) - `[:upper:]` - Uppercase letters (A-Z) - `[:alpha:]` - Letters (a-z, A-Z) - `[:digit:]` - Digits (0-9) - `[:alnum:]` - Alphanumeric (letters + digits) - `[:space:]` - Whitespace characters - `[:punct:]` - Punctuation characters - `[:graph:]` - Printable characters except space - `[:print:]` - Printable characters including space - `[:cntrl:]` - Control characters - `[:xdigit:]` - Hexadecimal digits (0-9, A-F, a-f) **Character Ranges:** - `a-z` - Lowercase letters a through z - `A-Z` - Uppercase letters A through Z - `0-9` - Digits 0 through 9 - Can combine: `a-z0-9` **Escape Sequences:** - `\n` - Newline - `\t` - Tab - `\r` - Carriage return - `\f` - Form feed - `\v` - Vertical tab - `\NNN` - Octal character (e.g., `\141` for 'a') - `\xHH` - Hexadecimal character (e.g., `\x61` for 'a') **Examples:** ``` tr "[:lower:]" "[:upper:]" tr -d "abc" tr -s " " tr "a-z" "A-Z" # Character range tr "[:digit:]" "x" # Replace all digits with x tr -d "[:punct:]" # Delete punctuation tr -c "[:alnum:]" " " # Replace non-alphanumeric with space tr -s "[:space:]" # Squeeze all whitespace tr "\\n" " " # Replace newlines with spaces ``` ### `uniq` - Remove Duplicates **Usage:** `uniq [flags]` **Flags:** - `-c`, `--count` - Prefix with occurrence count - `-d`, `--duplicates` - Show only duplicate lines - `-u`, `--unique` - Show only unique lines (appear exactly once) - `-i`, `--ignore-case` - Case-insensitive comparison - `-f N`, `--fields=N` - Skip first N fields when comparing - `-s N`, `--skip-chars=N` - Skip first N characters when comparing - `-w N`, `--check-chars=N` - Compare only first N characters - `--all` - Remove ALL duplicates (not just consecutive) **Examples:** ``` uniq uniq -c uniq -d uniq -u # Show only lines that appear exactly once uniq -i # Case-insensitive comparison uniq -f 2 # Skip first 2 fields when comparing uniq -s 3 # Skip first 3 characters when comparing uniq -w 5 # Compare only first 5 characters uniq --all # Remove all duplicates regardless of position uniq --all -c # Count all occurrences (not just consecutive) ``` ### `wc` - Word Count **Usage:** `wc [-l|-w|-c]` or `wc [mode=lines|words|chars]` **Modes:** Default is words when none specified. - `-l`, `mode=lines` - Count lines - `-w`, `mode=words` - Count words (default) - `-c`, `mode=chars` - Count characters **Examples:** ``` wc wc -l wc --mode=chars ``` ### `xargs` - Execute Command for Each Input Item **Usage:** `xargs ` or `xargs [flags] ` **Flags:** - `-n`, `--max-args=` - Maximum number of arguments per invocation (default: 1) - `-I`, `--replace=` - Replace placeholder in command with input item (default: `{}`) - `-d`, `--delimiter=` - Input delimiter (default: `\n`, supports `\t`, `\r`, `\0`) - `-0`, `--null` - Use null delimiter (`\0`) - `-a`, `--arg-file=` - Read arguments from `/tmp/` file instead of stdin - `-t`, `--verbose` - Print each command before execution - `-P`, `--parallel=` - Run up to N commands in parallel (default: 1) **Parameters:** - `command` (required): Command or pipeline to execute for each input item - Can be unquoted: `xargs echo` - Can be quoted: `xargs "echo {}"` or `xargs 'echo {}'` - Can be a pipeline: `xargs "cat {} | grep pattern"` **Behavior:** - Reads input items from stdin (or from `/tmp/` file if `-a` specified) - Splits input by delimiter (default: newline) - For each item (or batch with `-n`): - If `-I` is used: Replace placeholder in command string with the item(s) - If `-I` is not used: Append items as arguments to the command - Execute the command/pipeline - Collect output - Concatenates all outputs with newlines **Examples:** ``` # Execute cat for each file listed ls /tmp/*.txt | xargs cat # Output: contents of all files concatenated # With placeholder replacement echo -e "file1.txt\nfile2.txt" | xargs -I {} echo "Processing: {}" # Output: Processing: file1.txt\nProcessing: file2.txt # Batch processing (2 items at a time) echo -e "a\nb\nc\nd" | xargs -n 2 echo # Executes: echo a b, then echo c d # Read from temp file xargs -a urls.txt cat # Reads items from /tmp/urls.txt, executes cat for each # Complex pipeline echo -e "https://api1.com\nhttps://api2.com" | xargs -I {} "cat {} | jq .data" # For each URL: fetches, parses JSON, extracts .data field # Custom delimiter echo "apple,banana,cherry" | xargs -d ',' echo # Output: apple\nbanana\ncherry ``` **Note:** This command is pipeline-only (no standalone HTTP endpoint). When `-I` is not used, items are appended as arguments to the command (e.g., `ls | xargs cat` → `cat file1.txt file2.txt`). When `-I` is used, the placeholder is replaced in the command string before execution. ### `xpath` - XPath Parser **Usage:** `xpath [query] [flags]` or `xpath --query= [flags]` **Flags:** - `--query=` - XPath expression to evaluate (required) - `--all` - Return all matches (default: first match only) - `--text` - Normalize whitespace in extracted text - `--node` - Return serialized XML/HTML of matched nodes - `--count` - Return count of matches instead of content - `--delimiter=` - Output delimiter between results (default: newline) **Supported XPath Functions:** - `text()` - Get text content - `contains(str, substr)` - Check if string contains substring - `starts-with(str, prefix)` - Check if string starts with prefix - `normalize-space(str)` - Normalize whitespace - `string-length(str)` - Get string length - `count(nodes)` - Count nodes - `position()` - Get position in node set - `last()` - Get last position - `concat(str1, str2, ...)` - Concatenate strings **Supported Namespaces (auto-detected):** - `atom:` - Atom feeds - `rss:` - RSS 1.0 feeds - `dc:` - Dublin Core - `media:` - Media RSS - `itunes:` - iTunes podcasts **Examples:** ``` xpath "//h1/text()" xpath "//a/@href" --all xpath "//div[@class='content']" --all xpath "//li" --count xpath "//item/title" --all --delimiter="," xpath "//a[contains(@href, 'example')]" --all xpath "//atom:entry/atom:title" --all ``` ## Conditional Operators (`&&` and `||`) **Usage:** `command1 && command2` or `command1 || command2` **Behavior:** - `&&` (AND): Execute next command only if previous command succeeds - `||` (OR): Execute next command only if previous command fails - Success = non-empty output (or HTTP 200/204 with non-empty body) - Failure = empty output OR HTTP 4xx/5xx error **Precedence:** Parentheses `()` > Pipes `|` > AND `&&` > OR `||` > Semicolons `;` or newlines **Examples:** ``` # AND operator - chain commands that depend on success grep error logs.txt && echo "Found errors" test -f /tmp/file.txt && cat /tmp/file.txt echo hello | grep hello && echo "match found" # OR operator - provide fallback behavior grep error logs.txt || echo "No errors found" test -f /tmp/file.txt || echo "File not found" cat /tmp/data.txt || echo "Failed to read file" # Combined operators test -f /tmp/file.txt && cat /tmp/file.txt || echo "File not found or empty" grep error logs.txt && tee /tmp/errors.txt || echo "No errors" # With pipes cat data.txt | grep pattern && echo "found" || echo "not found" test -s /tmp/data.txt && cat /tmp/data.txt | grep result || echo "no data" # Complex chains test -f /tmp/file1.txt && test -f /tmp/file2.txt && diff /tmp/file1.txt /tmp/file2.txt grep error logs.txt && tee /tmp/errors.txt | wc -l || echo "No errors" ``` **Note:** The success/failure model is output-based: commands that produce output are considered successful, commands that produce empty output are considered failed. HTTP errors (4xx/5xx) always indicate failure. This matches shell behavior and is intuitive for web developers. ## Common Pipeline Patterns ### Data Processing Workflows **Extract and filter CSV data:** ``` cat data.csv | csv --select "name,age" | csv --filter "age > 18" | sort -n cat data.csv | csv --select "1-3" | csv --filter "age > 18 AND status == 'active'" | csv --to-json ``` **Monitor website changes:** ``` curl https://example.com | sha256 | tee /tmp/last-hash curl https://example.com | sha256 | diff /tmp/last-hash ``` **Process logs:** ``` cat /tmp/logs.txt | grep "error" | head -20 | tee /tmp/errors cat /tmp/errors | wc -l ``` **Extract and transform data:** ``` cat data.txt | cut -d "," -f 1,3 | sort | uniq | head -10 cat data.txt | cut -d "," -f "-2" --output-delimiter="|" | sort | uniq --all ``` **Encode/decode data:** ``` cat secret.txt | base64 | tee /tmp/encoded cat /tmp/encoded | base64 -d ``` **Multi-step processing:** ``` curl https://example.com | grep pattern | tee /tmp/step1 | head -5 | tail -3 | sort ``` **Compare two sources:** ``` curl https://url1.com | sha256 | tee /tmp/hash1 curl https://url2.com | sha256 | tee /tmp/hash2 cat /tmp/hash1 | diff /tmp/hash2 ``` **Send webhooks/API calls:** ``` echo '{"event": "deploy"}' | curl https://webhook.example.com/events --method POST --data "$(cat)" cat data.json | curl https://api.example.com/users --method POST --bearer token --data "$(cat)" cat data.json | curl https://api.example.com/users --method POST --header "Authorization: Bearer token" --data "$(cat)" cat logs.txt | grep ERROR | curl https://logs.example.com/api/errors --method POST --data "$(cat)" ``` **Send notifications:** ``` echo "Build succeeded" | mail -s "Build Status" team@example.com cat report.txt | mail --subject "Daily Report" --to admin@example.com ``` **Reusable pipeline scripts:** ``` # Store pipeline in temp storage echo "cat | csv --select 'name,email' | csv --filter 'age > 18'" | tee /tmp/adults.txt # Use it multiple times cat /tmp/users1.csv | source /tmp/adults.txt cat /tmp/users2.csv | source /tmp/adults.txt # Fetch pipeline from URL, store it, then execute curl https://example.com/pipelines/process.txt | tee /tmp/process.txt cat /tmp/data.txt | source /tmp/process.txt | grep result ``` **Extract specific lines:** ``` cat file.txt | head -100 | tail -50 # Lines 51-100 ``` **Process JSON data:** ``` cat api.json | jq ".users[0].name" | base64 ``` **Filter and count:** ``` cat logs.txt | grep "error" | wc -l cat data.csv | csv --filter "status == 'active'" | wc -l cat data.csv | csv --aggregate "count()" --filter "age > 18" --to-json cat data.csv | csv --aggregate "sum(price),avg(age)" --group-by "category" --to-json ``` ### Error Handling Patterns **Check if pattern exists (output-based):** ``` cat file.txt | grep "pattern" && echo "Found" || echo "Not found" ``` **Conditional processing with test:** ``` test -f /tmp/file.txt && cat /tmp/file.txt || echo "File not found" test -s /tmp/data.txt && process /tmp/data.txt || echo "No data" ``` **Guard expensive operations:** ``` test -f /tmp/cache.txt && cat /tmp/cache.txt || (fetch data | tee /tmp/cache.txt) ``` ### Performance Tips - Use `/tmp/` storage to cache expensive operations - Chain filters early to reduce data size - Use `head`/`tail` to limit processing scope - Store intermediate results with `tee` for reuse ### Input Size Limits - **Maximum input size:** 10MB (configurable via `MAX_INPUT_SIZE` env var) - **Maximum lines:** 1,000,000 lines (configurable via `MAX_LINES` env var) - Commands will return 400 error if limits are exceeded ## Request Construction ### Basic Pipeline Request **Method 1: Pipeline in POST body (recommended)** ```http POST https://piped.sh/ HTTP/1.1 Content-Type: text/plain X-API-Key: cat | grep hello | wc -l ``` **Method 2: Pipeline in query parameter** **Pipeline string:** `cat | grep hello | wc -l` **URL-encoded:** `cat%20%7C%20grep%20hello%20%7C%20wc%20-l` ```http POST https://piped.sh/?pipeline=cat%20%7C%20grep%20hello%20%7C%20wc%20-l HTTP/1.1 Content-Type: text/plain X-API-Key: hello world hello there goodbye ``` **Alternative: GET request** ```http GET https://piped.sh/?pipeline=cat%20%7C%20grep%20hello%20%7C%20wc%20-l HTTP/1.1 X-API-Key: ``` **Response:** - Status: `200 OK` - Content-Type: `text/plain` - Body: `2` (count of lines containing "hello") **Note:** When using query parameter, the POST body becomes the initial input. When using POST body for pipeline, there's no initial input (or use `cat` with URLs/POST body). ### Pipeline with URL Fetching **Method 1: Pipeline in POST body** ```http POST https://piped.sh/ HTTP/1.1 Content-Type: text/plain X-API-Key: curl https://api.example.com/data --header "Authorization: Bearer token" | jq .data | grep active ``` **Method 2: Pipeline in query parameter** **Pipeline string:** `curl https://api.example.com/data --header "Authorization: Bearer token" | jq .data | grep active` **URL-encoded:** `curl%20https://api.example.com/data%20--header%20%22Authorization:%20Bearer%20token%22%20%7C%20jq%20.data%20%7C%20grep%20active` ```http POST https://piped.sh/?pipeline=curl%20https://api.example.com/data%20--header%20%22Authorization:%20Bearer%20token%22%20%7C%20jq%20.data%20%7C%20grep%20active HTTP/1.1 Content-Type: text/plain X-API-Key: ``` **Note:** Use `curl` to fetch from URLs; the URL content becomes the pipeline input. ### Pipeline with POST Request **Pipeline string:** `curl https://api.example.com/endpoint --method POST --data "query=test" | jq .result` **URL-encoded:** `curl%20https://api.example.com/endpoint%20--method%20POST%20--data%20%22query%3Dtest%22%20%7C%20jq%20.result` ```http POST https://piped.sh/?pipeline=curl%20https://api.example.com/endpoint%20--method%20POST%20--data%20%22query%3Dtest%22%20%7C%20jq%20.result HTTP/1.1 Content-Type: text/plain X-API-Key: ``` **Note:** The `--data` flag provides the POST body for the HTTP request, not the pipeline input. ### Complete cURL Examples **Pipeline in POST body (simpler):** ```bash curl -X POST "https://piped.sh/" \ -H "Content-Type: text/plain" \ -H "X-API-Key: " \ -d "cat | grep hello | wc -l" ``` **Pipeline in query parameter (with initial input):** ```bash curl -X POST "https://piped.sh/?pipeline=cat%20%7C%20grep%20hello%20%7C%20wc%20-l" \ -H "Content-Type: text/plain" \ -H "X-API-Key: " \ -d "hello world hello there goodbye" ``` **GET request (pipeline in query parameter):** ```bash curl -X GET "https://piped.sh/?pipeline=cat%20%7C%20grep%20hello%20%7C%20wc%20-l" \ -H "X-API-Key: " ``` **Posting a script file:** ```bash curl -X POST "https://piped.sh/" \ -H "Content-Type: text/plain" \ -H "X-API-Key: " \ --data-binary @pipeline.txt ``` ### POST Body Encoding When POSTing pipelines or data with special characters, you can use encoding to handle them safely: **Supported encodings:** - `urlencoded` - URL encoding (e.g., `hello%20world`) - `base64` - Base64 encoding - `quoted-printable` or `qp` - Quoted-printable encoding (RFC 2045) **Methods (priority order):** 1. **Content-Transfer-Encoding header** (highest priority): ```bash curl -X POST "https://piped.sh/" \ -H "Content-Type: text/plain" \ -H "Content-Transfer-Encoding: urlencoded" \ -H "X-API-Key: " \ -d "echo%20'hello%20world'" ``` 2. **Content-Type header parameter**: ```bash curl -X POST "https://piped.sh/" \ -H "Content-Type: text/plain; encoding=urlencoded" \ -H "X-API-Key: " \ -d "echo%20'hello%20world'" ``` 3. **Query parameter**: ```bash curl -X POST "https://piped.sh/?encoding=urlencoded" \ -H "Content-Type: text/plain" \ -H "X-API-Key: " \ -d "echo%20'hello%20world'" ``` **Base64 example:** ```bash curl -X POST "https://piped.sh/?encoding=base64" \ -H "Content-Type: text/plain" \ -H "X-API-Key: " \ -d "ZWNobyAnaGVsbG8gd29ybGQn" ``` **Quoted-printable example:** ```bash curl -X POST "https://piped.sh/?encoding=quoted-printable" \ -H "Content-Type: text/plain" \ -H "X-API-Key: " \ -d "echo=20'hello=20world'" ``` ## Common Patterns ### Web Scraping **Pipeline:** `curl https://example.com | html "div.content" --all | grep "keyword"` - Fetches webpage (redirects followed by default), extracts all matching HTML elements, filters by keyword ### API Data Processing **Pipeline:** `curl https://api.example.com/data --header "Authorization: Bearer token" | jq ".items[*]" | csv --to-json` - Fetches API with auth header, extracts JSON array, converts to CSV then JSON ### Text Processing Chain **Pipeline:** `cat | grep "error" -i | sed "s/ERROR/Error/g" | sort | uniq -c` **Enhanced:** `cat | grep "error" -i | sed "s/ERROR/Error/g;s/CRITICAL/Critical/g" | sort | uniq --all -c` - Filters case-insensitive, normalizes text, sorts, counts duplicates - **Note:** Input comes from POST body (no URL specified) ### Data Extraction & Filtering **Pipeline:** `cat | cut -d "," -f 1,3 | csv --filter "age > 18" --to-json | jq ".name"` **Enhanced:** `cat | cut -d "," -f "1-3" --complement | csv --filter "age > 18 AND contains(name, 'John')" --to-json` - Extracts CSV columns, filters rows, converts to JSON, extracts field - **Note:** CSV data comes from POST body ### Change Detection **Pipeline:** `curl https://example.com/page --header "User-Agent: Bot" | html "#content" | sed "s/\\s+/ /g" | tr "[:upper:]" "[:lower:]"` **Enhanced:** `curl https://example.com/page | html "#content" | sed "s/<[^>]*>//g" | tr -s "[:space:]" | tr "a-z" "A-Z"` - Fetches page with custom header, extracts content, normalizes whitespace and case ## Response Format ### Success Response - **Status:** `200 OK` - **Content-Type:** `text/plain` - **Body:** Command output as plain text ### Error Responses **401 Unauthorized:** Missing or invalid API key ```json { "error": "API key required. Provide X-API-Key header or Authorization: Bearer " } ``` **400 Bad Request:** Invalid pipeline or command error ```json { "error": "Error in command 'grep': Query parameter 'match' is required" } ``` **400 Bad Request:** Invalid pipeline syntax ```json { "error": "Unknown command: unknowncmd" } ``` **400 Bad Request:** URL fetch failure ```json { "error": "Error fetching https://example.com: Failed to fetch https://example.com: 404 NOT FOUND" } ``` ## URL Encoding Notes When constructing pipeline strings, URL-encode special characters: - Space: `%20` or `+` - Pipe (`|`): `%7C` - Quotes (`"`): `%22` - Colon (`:`): `%3A` - Equals (`=`): `%3D` - Hash (`#`): `%23` - Question mark (`?`): `%3F` - Ampersand (`&`): `%26` **Example:** - Original: `cat | grep hello | wc -l` - Encoded: `cat%20%7C%20grep%20hello%20%7C%20wc%20-l` **Example with quotes:** - Original: `cat --header "User-Agent: Bot"` - Encoded: `cat%20--header%20%22User-Agent:%20Bot%22` **Important:** Always URL-encode the entire pipeline string before adding it to the query parameter. ## Best Practices for AI Agents 1. **Prefer POST body for pipeline:** Put pipeline string directly in POST body (simpler, no URL encoding needed) 2. **Use query parameter when needed:** Only use `?pipeline=` when you also need initial input in POST body 3. **Chain commands logically:** Each command receives output from previous command as input 4. **Handle errors:** Check response status (200 = success, 400/401 = error) and parse JSON error messages 5. **Use appropriate content types:** Request with `Content-Type: text/plain`, response is `text/plain` 6. **Test incrementally:** Build pipelines step-by-step for complex operations (test each command separately first) 7. **Use `curl` for URL fetching:** Use `curl` with URLs to fetch external data (supports headers, POST, retry, etc.) 8. **Use `cat` for local data:** Use `cat` for temp files and saved pipelines; use `curl` for HTTP(S) URLs 9. **Quote arguments with spaces:** Use quotes for header values, data, etc. that contain spaces 10. **Multiple URLs:** `curl` can fetch multiple URLs - they're concatenated with newlines 11. **Pipeline input flow:** - With query param: POST body → first command → second command → ... → final output - Without query param: Pipeline in POST body, no initial input (use `curl` for URL input or `cat` for temp files) 12. **Reusable pipelines:** Use `source` command to execute pipelines from temp files or saved pipelines; fetch from URLs with `curl | tee /tmp/x && source /tmp/x` 13. **Conditional execution:** Use `&&` (AND) and `||` (OR) operators for conditional logic - success is based on non-empty output, failure is empty output or HTTP errors 14. **Test command:** Use `test` command for file property checks (`-f`, `-e`, `-s`, `-z`) - works seamlessly with `&&`/`||` operators 15. **Output-based success/failure:** Commands that produce output are successful, commands with empty output are failed (for `&&`/`||`). HTTP 4xx/5xx errors always indicate failure. ## Quick Reference | Task | Pipeline Example | |------|------------------| | Count lines | `cat \| wc -l` | | Filter by pattern | `cat \| grep "pattern"` | | Extract JSON field | `cat \| jq ".field"` | | Extract HTML element | `curl https://example.com \| html "div.content"` | | Parse CSV | `cat \| csv --select "col1,col2" --to-json` | | Sort and deduplicate | `cat \| sort \| uniq` | | Extract columns | `cat \| cut -d "," -f 1,3` | | Extract columns by range | `cat \| cut -d "," -f 1-3` | | Extract with complement | `cat \| cut -d "," -f 1,3 --complement` | | Replace text | `cat \| sed "s/old/new/g"` | | Multiple sed expressions | `cat \| sed "s/old/new/g;s/foo/bar/g"` | | Delete lines | `cat \| sed "/pattern/d"` | | Append/insert text | `cat \| sed "2a/INSERTED"` or `sed "2i/INSERTED"` | | Transform case | `cat \| tr "[:lower:]" "[:upper:]"` | | Transform with ranges | `cat \| tr "a-z" "A-Z"` | | Delete with complement | `cat \| tr -c "[:alnum:]" -d` | | Remove all duplicates | `cat \| uniq --all` | | Show only unique lines | `cat \| uniq -u` | | Case-insensitive uniq | `cat \| uniq -i` | | Fetch with auth | `curl https://api.com --header "Authorization: Bearer token"` | | POST request | `curl https://api.com --method POST --data "key=value"` | | Fetch multiple URLs | `curl https://api1.com https://api2.com` | | Send webhook | `echo '{"event": "test"}' \| curl https://webhook.com/events --method POST --data "$(cat)"` | | Send email | `echo "message" \| mail -s "subject" user@example.com` | | Check API key info | `api` | | Disable redirects | `curl https://example.com --no-redirects` | | Retry on failure | `curl https://api.com --retry 3 --retry-delay 1000` | | Write to temp storage | `cat \| tee /tmp/step1` | | Create or refresh temp file expiry | `touch /tmp/step1` or `touch /tmp/a /tmp/b --expire 2h` | | Read from temp storage | `cat /tmp/step1` | | List temp files | `ls` | | List temp files with pattern | `ls "/tmp/*.txt"` or `ls "/tmp/test?"` | | Delete temp file | `rm /tmp/oldfile.txt` | | Multi-step with temp files | `cat \| tee /tmp/step1 \| grep pattern` | | Conditional execution | `cat \| grep error && echo "error found" \|\| echo "no errors"` | | String comparison | `test foo = foo && echo "match"` or `[[ foo = foo ]] && echo "match"` | | Integer comparison | `test 10 -gt 5 && echo "bigger"` or `[[ 10 -gt 5 ]] && echo "bigger"` | | String non-null test | `test hello && echo "non-empty"` | | `[[ ]]` syntax sugar | `[[ -f /tmp/file.txt ]] && cat /tmp/file.txt` (equivalent to `test`) | | Output text | `echo "hello world"` | | Pass through input | `cat \| echo` | | AI analysis | `cat data.csv \| ai "Find the top 5 records"` | | Cut with attached flags | `cat \| cut -d',' -f4` | | Cut with ranges | `cat \| cut -d "," -f "1-3"` | | Cut with complement | `cat \| cut -d "," -f 1,3 --complement` | | CSV with quoted select | `cat \| csv --select "name,email"` | | CSV with index select | `cat \| csv --select "1,3"` | | CSV with range select | `cat \| csv --select "1-3"` | | CSV with AND/OR filter | `cat \| csv --filter "age > 18 AND status == 'active'"` | | CSV with string functions | `cat \| csv --filter "contains(name, 'John')"` | | CSV aggregation | `cat \| csv --aggregate "sum(price),avg(age)"` | | CSV group by | `cat \| csv --aggregate "sum(price)" --group-by "category"` | | CSV pagination | `cat \| csv --limit 10 --offset 5` | | CSV sorting | `cat \| csv --sort "age:desc"` | | CSV type conversion | `cat \| csv --types "age:int,price:float" --to-json` | | Multiple sed expressions | `cat \| sed "s/old/new/g;s/foo/bar/g"` | | Sed with backreferences | `cat \| sed "s/(\\w+) (\\w+)/\\2, \\1/"` | | Sed append/insert | `cat \| sed "2a/INSERTED"` or `sed "2i/INSERTED"` | | Sed print lines | `cat \| sed "/pattern/p" --quiet` | | Delete first line | `cat \| sed "1d"` | | Delete line range | `cat \| sed "1,3d"` | | Tr with ranges | `cat \| tr "a-z" "A-Z"` | | Tr with complement | `cat \| tr -c "[:alnum:]" -d` | | Tr with escape sequences | `cat \| tr "\\n" " "` | | Uniq show unique only | `cat \| uniq -u` | | Uniq case-insensitive | `cat \| uniq -i` | | Uniq remove all duplicates | `cat \| uniq --all` | | Uniq skip fields | `cat \| uniq -f 2` | | Compute SHA-256 hash | `cat \| sha256` | | Compute MD5 hash | `cat \| md5` | | Change detection | `curl https://example.com \| sha256 \| tee /tmp/hash` | | Base64 encode | `cat \| base64` | | Base64 decode | `cat \| base64 -d` | | First N lines | `cat \| head -5` | | Last N lines | `cat \| tail -5` | | First then last | `cat \| head -10 \| tail -5` | | Execute pipeline from URL | `curl https://example.com/pipeline.txt \| tee /tmp/p.txt && source /tmp/p.txt` | | Execute pipeline from temp storage | `cat /tmp/data.txt \| source /tmp/my-pipeline.txt` | | Execute saved pipeline by name | `cat /tmp/data.txt \| source MySavedPipeline` | | Run saved TS/JS script | `echo "data" \| source my-transform.ts` (auto-delegates to deno) | | Store and reuse pipeline | `echo "cat \| grep pattern" \| tee /tmp/pipeline.txt` then `cat /tmp/data \| source /tmp/pipeline.txt` | | List cron schedules (named pipelines) | `crontab -l` or `crontab --list` | | Remove all cron schedules | `crontab -r` or `crontab --remove` | | List timezones | `crontab -tz` or `crontab --timezone` | | Filter timezones by pattern | `crontab -tz americas` or `crontab --timezone europe` | | Install cron from stdin | `echo "5 * * * * Europe/London Prayer" \| crontab` or `cat /tmp/crontab.txt \| crontab` | | Install cron from file | `crontab /tmp/crontab.txt` or `crontab MyCrontabPipeline` | | Find files by path prefix | `find /tmp/blah/` | | Find files by name pattern | `find /tmp/ -name "*.txt"` | | Find files by modification time | `find /tmp/ -mtime +1d` | | Find files with multiple criteria | `find /tmp/blah/ -name "*.txt" -mtime +1d` | | Export project as YAML | `project` or `project "My Project"` | | Export only tools | `project --export=tools` | | Export specific sections | `project --export=pipelines,exports` | | Import project (additive) | `cat backup.yaml \| project` | | Import with replace all | `cat backup.yaml \| project --replace=all` | | Import replacing pipelines | `cat backup.yaml \| project --replace=pipelines` | | Import with selective replace | `cat backup.yaml \| project --replace=saved,aliases` | | Backup project to file | `project \| tee /tmp/backup.yaml` | | Call MCP tool | `mcp piped execute_pipeline '{"pipeline":"ls"}'` | | MCP tool with pipe | `mcp piped execute_pipeline '{"pipeline":"echo hi"}' \| grep hi` | | MCP stdin as input | `echo "data" \| mcp piped execute_pipeline` | **Note:** `\|` in table represents actual `|` character in pipeline string. ## Temporary Storage (`/tmp/`) ### Writing to Temp Storage Use `tee` command to write to temp storage: ``` cat data | tee /tmp/step1 ``` - Writes to `/tmp/step1` (scoped to your API key) - Also outputs data to stdout (for chaining) - Use `-q` or `--quiet` flag to suppress stdout output: `cat data | tee -q /tmp/step1` - Multiple paths: `tee /tmp/step1 /tmp/backup` **Date expansion in paths** Temp file paths support date expansion so users can use time-based filenames (e.g. daily logs) without hardcoding dates. The time used for expansion is the **`X-User-Time`** request header when present (ISO 8601); if omitted, UTC is used. The same expansion applies wherever a temp path is accepted: **tmp** (read/write/delete/PATCH), **tee**, **touch**, **cat**, **ls**, **mv**, **cp**, **test** (file path), and **source** (when the URL is a `/tmp/` path). **Date Format Specifiers:** Temp file paths support Unix `date`-style format specifiers for automatic timestamp inclusion: ``` cat data | tee /tmp/backup-%Y%m%d.log cat data | tee /tmp/data-%Y-%m-%d-%H%M%S.txt ``` Supported format specifiers: - `%Y` - Full year (4 digits, e.g., 2024) - `%y` - Year (2 digits, e.g., 24) - `%m` - Month (01-12) - `%d` - Day of month (01-31) - `%H` - Hour (00-23) - `%I` - Hour (01-12) for 12-hour format - `%M` - Minute (00-59) - `%S` - Second (00-59) - `%j` - Day of year (001-366) - `%w` - Day of week (0-6, Sunday=0) - `%W` - Week number (00-53) - `%V` - ISO week number (01-53) - `%a` - Abbreviated weekday name (Sun, Mon, Tue, etc.) - `%A` - Full weekday name (Sunday, Monday, Tuesday, etc.) - `%b` - Abbreviated month name (Jan, Feb, Mar, etc.) - `%B` - Full month name (January, February, March, etc.) - `%z` - Timezone offset (e.g., +0500, -0800) - `%Z` - Timezone name/abbreviation (UTC or UTC+offset) - `%p` - AM/PM - `%r` - 12-hour time format (%I:%M:%S %p) - `%R` - 24-hour time format (%H:%M) - `%T` - 24-hour time format (%H:%M:%S) - `%D` - Date format (%m/%d/%y) - `%F` - ISO date format (%Y-%m-%d) - `%s` - Unix timestamp (seconds since epoch) - `%n` - Newline character - `%t` - Tab character - `%%` - Literal % character - `%V` - ISO week number (01-53) Examples: - `backup-%Y%m%d.log` → `backup-20240115.log` (daily backups) - `data-%Y-%m-%d-%H%M%S.txt` → `data-2024-01-15-143022.txt` (timestamped files) - `logs-%Y-%j.log` → `logs-2024-015.log` (year and day of year) ### Reading from Temp Storage Use `cat` with `/tmp/` paths: ``` cat /tmp/step1 cat /tmp/backup-%Y%m%d.log # Date formats work when reading too ``` - Reads from temp storage (scoped to your API key) - Date format specifiers are expanded when reading (useful for daily backups) - Can concatenate multiple temp files: `cat /tmp/step1 /tmp/step2` - **All temp files are private** - API key authentication required for access **Using `source` with temp storage:** The `source` command can also read pipelines from `/tmp/` paths: ``` # Store a reusable pipeline echo "cat | grep pattern | sort" | tee /tmp/my-pipeline.txt # Execute it later cat data.txt | source /tmp/my-pipeline.txt ``` - Works exactly like `cat /tmp/` but executes the content as a pipeline - Supports all `cat` flags (though not needed for `/tmp/` paths) - Perfect for storing reusable pipeline scripts ## Important Notes 1. **Input Sources:** - POST body: Use when you have data to process - URLs: Use `cat ` to fetch external data - Temp files: Use `cat /tmp/` to read from temp storage - All can be combined: `cat /tmp/step1 https://api.com` (concatenated) 2. **Command Execution:** - Commands execute left-to-right in pipeline - Each command receives stdout from previous command as stdin - Output is always `text/plain` (even for JSON/CSV commands) 3. **Temp Storage:** - Files are scoped to API key (no collisions between users) - Default expiration: 36 hours (configurable per token via `temp_file_ttl` config) - All files are private to the API key (authentication required) - Files persist even if API key is regenerated (ULID-based storage) 4. **Quoting:** - Quotes in pipeline strings should be URL-encoded as `%22` (when using query param) - Or put pipeline in POST body (no encoding needed!) - Headers with spaces: `--header "key: value"` → `--header%20%22key:%20value%22` 5. **Special Characters:** - Regex patterns in `grep`/`sed`: Escape special chars or URL-encode - JSONPath in `json`: Use dot notation (`.field`) or bracket notation (`["field"]`) - CSS selectors in `html`: Basic selectors supported (tag, class `#id`, `.class`, `[attr]`) 6. **Error Recovery:** - Use `--retry` and `--retry-delay` for transient failures - Check error messages for specific command failures - Invalid pipelines return 400 with error message --- EOF