Annotations & Fragments

Given a file or entity, what annotations, fragments, and fields will the pipeline produce?
This guide is for consumers of pipeline output — developers querying entities from OpenSearch, building UIs on top of annotations, or integrating downstream systems. It documents the contract: what you can expect on an entity after processing, by input type.

Quick Start: What Happens to My File?
Entity Structure (Output Shape)
Output by Input Type
- Documents
- Images
- Audio
- Video
- Generic / Other
Annotation Dictionary
Fragment Dictionary
Shared Data Formats
Cheat Sheet

Quick Start: What Happens to My File?

You upload a file
        │
        ▼
┌──────────────────────────────────────────────────┐
│  The pipeline inspects os_item_content_type      │
│  (MIME type) and routes accordingly:             │
│                                                  │
│  application/pdf ──────────► Document path       │
│  application/vnd.*office*──► Document path       │
│  application/vnd.*present*─► Document path       │
│  application/msword ───────► Document path       │
│  image/* ──────────────────► Image path          │
│  audio/* ──────────────────► Audio path          │
│  video/* ──────────────────► Video path          │
│  text/csv ─────────────────► Spreadsheet path    │
│  application/vnd.ms-excel──► Spreadsheet path    │
│  application/vnd.*sheet*───► Spreadsheet path    │
│  application/vnd.*ods──────► Spreadsheet path    │
│  text/tab-separated-values─► Spreadsheet path    │
└──────────────────────────────────────────────────┘
        │
        ▼
  Entity comes back with:
    • annotations   (text, tags, NER, summary, embeddings…)
    • fragments     (pages, keyframes, audio segments…)
    • each fragment also carries its own annotations

Entity Structure in OpenSearch

After processing, entities are indexed into OpenSearch. The document _id equals the entity's os_entity_uid. Fields marked with ◆ are always present; ◇ are conditional.

Document Shape

OpenSearch document (_id = os_entity_uid)
│
├── document_fields               # Ontology record fields (flat dict)
│   ├── os_entity_uid             ◆      # Unique identifier (= _id = entity_id)
│   ├── entity_id                 ◆      # Same as os_entity_uid
│   ├── entity_type               ◆      # Ontology type (e.g. "os_document", "os_fragment")
│   ├── os_concept                ◆      # Same as entity_type
│   ├── entity_label              ◆      # Human-readable name
│   ├── os_workspace              ◆      # Workspace ID
│   ├── os_item_content_type      ◆      # MIME type
│   ├── os_has_attachment         ◆      # true if binary content exists
│   ├── os_parent_folder          ◇      # Parent folder UID (for files)
│   ├── source_entity_uid         ◇      # Root entity UID — same for ALL fragments of a file
│   ├── os_parent_uid             ◇      # Immediate parent UID — for tree reconstruction
│   ├── fragment_index            ◇      # 0-based position among siblings (under same parent)
│   ├── fragment_count            ◇      # Total sibling count
│   ├── fragment_collection_type  ◇      # Fragment category (e.g. "document_pages")
│   └── ... (any ontology-specific fields)
│
├── annotations                   # Enrichment data from pipeline
│   ├── extract:txt               ◇      # Extracted / transcribed text
│   ├── extract:txtLang           ◇      # Language of extracted text
│   ├── extract:metadata          ◇      # File metadata (ExifTool)
│   ├── summary:txt               ◇      # LLM summary
│   ├── ner:entities              ◇      # Named entities
│   ├── image:annotations         ◇      # Image tags & detections
│   ├── rag:chunks                ◇      # RAG text chunks + embeddings
│   ├── rag:fulltext              ◇      # Concatenated RAG text
│   └── materialized              ◇      # Rolled-up child fragment data
│       ├── ner:entities          ◇
│       └── ...
│
└── processing_status             # Pipeline status
    ├── status_code               ◆      # RUNNING | COMPLETED | FAILED
    ├── reason                    ◆      # Status description
    └── timestamp                 ◆      # When status was set

`source_entity_uid` vs `os_parent_uid`

These two fields serve different purposes and it is important to understand the distinction:

Field	Points To	Same Across Tree?	Purpose
`source_entity_uid`	The root file/entity that was split	Yes — identical on every fragment in the tree	Flat lookup: "give me all fragments of this file"
`os_parent_uid`	The immediate parent (root entity or another fragment)	No — varies per tree level	Tree reconstruction: "who is my direct parent?"

                 ┌───────────────────┐
                 │    Root File      │
                 │ _id = "file-001"  │
                 └────────┬──────────┘
                          │
        ┌─────────────────┼─────────────────┐
        ▼                 ▼                 ▼
   ┌──────────┐    ┌──────────┐      ┌──────────┐
   │ Page 1   │    │ Page 2   │      │ Page 3   │
   │          │    │          │      │          │
   │ source_  │    │ source_  │      │ source_  │
   │ entity_  │    │ entity_  │      │ entity_  │
   │ uid =    │    │ uid =    │      │ uid =    │
   │ file-001 │    │ file-001 │      │ file-001 │  ← all same: root file
   │          │    │          │      │          │
   │ os_      │    │ os_      │      │ os_      │
   │ parent_  │    │ parent_  │      │ parent_  │
   │ uid =    │    │ uid =    │      │ uid =    │
   │ file-001 │    │ file-001 │      │ file-001 │  ← parent is root file
   └────┬─────┘    └──────────┘      └──────────┘
        │
   ┌────┴────────┐
   ▼             ▼
┌────────┐  ┌────────┐
│ Img A  │  │ Img B  │
│        │  │        │
│source_ │  │source_ │
│entity_ │  │entity_ │
│uid =   │  │uid =   │
│file-001│  │file-001│  ← still the root file
│        │  │        │
│os_     │  │os_     │
│parent_ │  │parent_ │
│uid =   │  │uid =   │
│page-1  │  │page-1  │  ← parent is Page 1 (not root)
└────────┘  └────────┘

Querying Annotations

In python using the octostar-python-client library, use get_entity_annotations() to fetch annotations for entities. The function supports automatic fragment fetching and merging via the recurse and merge parameters:

from octostar.utils.search import get_entity_annotations

# Parent annotations only (no fragments)
result = get_entity_annotations.sync(
    os_entity_uids=["uid-1", "uid-2"],
    recurse=False,
    client=client,
)
# {"uid-1": {"extract:txt": "...", ...}, "uid-2": {...}}

# Parent + fragment annotations merged together (default)
result = get_entity_annotations.sync(
    os_entity_uids=["uid-1"],
    recurse=True,     # fetch fragments via source_entity_uid
    merge=True,        # merge fragment annotations into parent
    client=client,
)
# {"uid-1": {"extract:txt": "full merged text...", ...}}

# Parent + fragment annotations kept separate, in DFS tree order
result = get_entity_annotations.sync(
    os_entity_uids=["uid-1"],
    recurse=True,
    merge=False,       # return each fragment's annotations separately
    client=client,
)
# {"uid-1": {"uid-1": {...parent...}, "frag-1": {...}, "frag-2": {...}, ...}}

# Fetch specific fields only
result = get_entity_annotations.sync(
    os_entity_uids=["uid-1"],
    fields=["extract:txt", "ner:entities"],
    client=client,
)

When recurse=True, the function internally:

Queries parent annotations by _id
Queries all fragments by source_entity_uid (flat fetch — gets the entire tree at once)
Rebuilds the tree using os_parent_uid for correct DFS ordering
Merges (or returns) annotations in tree order

Merge strategies used when merge=True:

Annotation Key	Strategy	Behavior
`extract:txt`	`strconcat`	Concatenate with `\n`
`rag:fulltext`	`strconcat`	Concatenate with `\n`
`image:annotations`	`lstconcat`	Concatenate lists
`image:annotations-key`	`lstconcat`	Concatenate lists
`image:annotations-sensitive`	`lstconcat`	Concatenate lists
`image:annotations-settings`	`lstconcat`	Concatenate lists
`ner:entities`	`lstconcat`	Concatenate lists
`rag:chunks`	`lstconcat`	Concatenate lists
`summary:txt`	`first`	Keep first (parent) value
`extract:metadata`	`first`	Keep first (parent) value
(everything else)	`first`	Keep first non-null value

Querying Fragments Directly

Get all fragments of a file (flat)

Use source_entity_uid to find every fragment that belongs to a root file, regardless of tree depth — this returns a flat list:

{
  "query": {
    "bool": {
      "must": [
        {"term": {"document_fields.os_workspace.keyword": "<workspace_id>"}},
        {"term": {"document_fields.source_entity_uid.keyword": "<root_file_uid>"}}
      ]
    }
  },
  "size": 10000,
  "sort": [{"document_fields.fragment_index": {"order": "asc"}}],
  "_source": ["document_fields", "annotations"]
}

This returns pages, images embedded in those pages, audio tracks, etc. — all in one query.

Timbr SQL equivalent:

SELECT entity_id, entity_type, entity_label, fragment_index,
       fragment_collection_type, os_parent_uid
FROM timbr.`os_fragment`
WHERE os_workspace = :workspace_id
  AND source_entity_uid = :root_file_uid

Reconstruct the tree

Once you have the flat list, use os_parent_uid to build the parent-child hierarchy:

Every fragment whose os_parent_uid = <root_file_uid> is a direct child of the root (e.g. pages)
Every fragment whose os_parent_uid = <page_fragment_uid> is a child of that page (e.g. extracted images)
Sort children at each level by fragment_index

# Pseudocode: build tree from flat fragment list
root_id = "file-001"
children_by_parent = defaultdict(list)

for frag in all_fragments:
    children_by_parent[frag["os_parent_uid"]].append(frag)

for kids in children_by_parent.values():
    kids.sort(key=lambda f: f["fragment_index"])

# children_by_parent[root_id]         → [Page 1, Page 2, Page 3]
# children_by_parent["page-1-uid"]    → [Image A, Image B]

Get only direct children of a specific node

To get just one level (e.g. only pages of the root, or only images of page 1), filter by os_parent_uid:

{
  "query": {
    "bool": {
      "must": [
        {"term": {"document_fields.os_workspace.keyword": "<workspace_id>"}},
        {"term": {"document_fields.os_parent_uid.keyword": "<parent_node_uid>"}}
      ]
    }
  },
  "sort": [{"document_fields.fragment_index": {"order": "asc"}}],
  "_source": ["document_fields", "annotations"]
}

Fragment Hierarchy Diagram

┌──────────────────────────────────────────────────────────────────────┐
│  Root File                                                           │
│  _id = "file-001"                                                    │
│  entity_type = "os_file"                                             │
└───────────────────────────────┬──────────────────────────────────────┘
                                │
         All fragments below have source_entity_uid = "file-001"
                                │
        ┌───────────────────────┼───────────────────────┐
        │                       │                       │
        ▼                       ▼                       ▼
┌───────────────┐     ┌───────────────┐       ┌───────────────┐
│ Page Frag 1   │     │ Page Frag 2   │       │ Page Frag 3   │
│ _id="pg-1"    │     │ _id="pg-2"    │       │ _id="pg-3"    │
│               │     │               │       │               │
│ os_parent_uid │     │ os_parent_uid │       │ os_parent_uid │
│  = "file-001" │     │  = "file-001" │       │  = "file-001" │
│ fragment_idx=0│     │ fragment_idx=1│       │ fragment_idx=2│
│ collection_   │     │ collection_   │       │ collection_   │
│ type=         │     │ type=         │       │ type=         │
│ "document_    │     │ "document_    │       │ "document_    │
│  pages"       │     │  pages"       │       │  pages"       │
└───────┬───────┘     └───────────────┘       └───────────────┘
        │
        │  Children of Page 1 (os_parent_uid = "pg-1")
        │
   ┌────┴──────────┐
   ▼               ▼
┌───────────┐ ┌───────────┐
│ Image A   │ │ Image B   │
│ _id=      │ │ _id=      │
│ "img-a"   │ │ "img-b"   │
│           │ │           │
│os_parent_ │ │os_parent_ │
│uid="pg-1" │ │uid="pg-1" │  ← immediate parent is page, not root
│           │ │           │
│source_    │ │source_    │
│entity_uid │ │entity_uid │
│="file-001"│ │="file-001"│  ← root is still the original file
│           │ │           │
│collection_│ │collection_│
│type=      │ │type=      │
│"document_ │ │"document_ │
│ images"   │ │ images"   │
└───────────┘ └───────────┘

Summary: Query by source_entity_uid to get all fragments of a file in one flat query. Use os_parent_uid to reconstruct the tree. Use os_parent_uid to query only direct children of a specific node.

Missing fragments: If a fragment failed to process or was not indexed, tree traversal via os_parent_uid may hit a dead end — children of the missing fragment become unreachable because their parent node is absent. A source_entity_uid query is resilient to this: it always returns every indexed fragment regardless of gaps in the tree, so you can still read annotations from all surviving fragments even if the hierarchy is incomplete. Prefer source_entity_uid when completeness matters more than tree structure.

Output by Input Type

Documents

Applies to: application/pdf, application/vnd.openxmlformats-officedocument.*, application/msword, application/vnd.oasis.opendocument.*, and other Tika-supported MIME types.

Splitting Behavior by MIME Type

Not all document types are split the same way. Splitting depends on whether a splitter plugin exists for the MIME type and what unit it operates on, as well as splitting parameters (max_pages, max_rows, extract_images):

MIME Type	Split Unit	Splitting Behavior
`application/pdf`	page	Native PDF page extraction
`application/vnd.openxmlformats-officedocument.wordprocessingml.document`, `application/msword`	page	Converted to PDF via LibreOffice, then page-split
`application/vnd.oasis.opendocument.text`	page	Converted to PDF via LibreOffice, then page-split
`application/vnd.openxmlformats-officedocument.presentationml.presentation`, `application/vnd.ms-powerpoint`	slide	Converted to PDF, each slide becomes a page
`application/vnd.oasis.opendocument.presentation`	slide	Same as above
`application/vnd.openxmlformats-officedocument.spreadsheetml.sheet`, `application/vnd.ms-excel`	row	Chunked by data rows (controlled by `max_rows`, not `max_pages`)
`text/csv`, `text/tab-separated-values`	row	Same row-based chunking
`application/vnd.oasis.opendocument.spreadsheet`	row	Same row-based chunking
`text/plain` and other types	—	No splitter plugin. Text is extracted as a single unit, no page/row fragments are produced.

Key: max_pages only affects page/slide-based types. max_rows only affects row-based types. If neither limit applies to a given MIME type, the file passes through as a single entity with no fragments.

Annotations on the Parent Entity

Annotation	Type	Presence	Description
`extract:txt`	`str`	Sometimes	Full extracted text (Tika parser; OCR fallback for scanned PDFs); only present if document has not been split
`extract:txtLang`	`str`	Sometimes (when text is non-empty)	Detected language of extracted text; only present if document has not been split
`extract:metadata`	`dict`	Always	ExifTool file metadata. Keys vary per format (see extract:metadata).
`ner:entities`	`dict`	Usually	Named entities extracted from summary. Structure: `{ label: [value, ...] }`
`summary:txt`	`str`	Always for PDF, DOCX, DOC, ODT. Not produced for PPTX, XLSX, CSV, and others.	LLM-generated summary; Hierarchical when the document has fragments.
`extract:frontmatter`	`dict`	Optional	Document front-matter — keys: `toc`, `abstract`, `glossary`, `introduction`
`rag:chunks`	`list[dict]`	Always	RAG text chunks with embeddings (see RAG Chunks)
`rag:fulltext`	`str`	Usually	All RAG chunk texts concatenated
`materialized.ner:entities`	When fragmented	Child annotations accumulated on parent

summary:txt MIME allowlist: By default, only application/pdf, DOCX (application/vnd.openxmlformats-officedocument.wordprocessingml.document), DOC (application/msword), and ODT (application/vnd.oasis.opendocument.text) are eligible for summarization. Presentations, spreadsheets, CSV, and plain text are not summarized.

Fragments Produced

Collection Type	Content	When Produced	Record Fields
`document_pages`	One entity per page or slide	Page/slide-based types (PDF, DOCX, DOC, ODT, PPTX, PPT, ODP) when `max_pages` is set	`fragment_page_number`, `fragment_start`, `fragment_end`, `fragment_index`, `fragment_count`
`document_page_groups`	Groups of consecutive pages (e.g. pages 1–20, 21–40)	When the document has more pages than `max_pages` — individual pages are grouped under an intermediate node	Same as above
`table_rows`	One entity per row range	Row-based types (XLSX, XLS, CSV, TSV, ODS) when `max_rows` is set	`fragment_start`, `fragment_end`, `fragment_index`, `fragment_count`
`table_row_groups`	Groups of consecutive row ranges	When the spreadsheet has more rows than `max_rows` and multiple grouping levels apply	Same as above
`document_images`	Embedded images extracted from the document	When `extract_images` is enabled (applies to page-based and spreadsheet types)	`fragment_index`, `fragment_count`, `os_item_content_type: "image/*"`

Row-based types (XLSX, XLS, CSV, TSV, ODS) produce table_rows fragments (or table_row_groups when grouping) when max_rows is set. Each fragment represents a row range — fragment_start/fragment_end refer to row indices.

Types without a splitter plugin (text/plain, etc.) produce no fragments — only the parent entity is enriched.

Annotations on the main document

Annotation	Presence	Notes
`extract:txt`	No	See the individual pages
`extract:txtLang`	No	See the individual pages
`ner:entities`	Usually	NER for the document summary
`materialized.ner:entities`	Usually	NER for the whole document
`summary:txt`	Always	LLM summary of the entire document

Annotations on `document_page_groups` Fragments

Annotation	Presence	Notes
`extract:txt`	No	See the individual pages
`extract:txtLang`	No	See the individual pages
`ner:entities`	Usually	NER for the group summary
`materialized.ner:entities`	Usually	NER for all children pages
`summary:txt`	Always	LLM summary of the group

Annotations on `document_pages` (Leaf) Fragments

Annotation	Presence	Notes
`extract:txt`	Always	Text of that individual page/slide/row-range
`extract:txtLang`	Always (if text)	Text language
`ner:entities`	Usually	NER for the page's text
`summary:txt`	Sometimes	Only produced if the page has sub-fragments (e.g. images). Summary of the page contents
`section:type`	Always	One of: `textual`, `image`, `tabular`, `list`, `mixed`

summary:txt on fragments: The parent entity and document_page_groups are prioritized for LLM summarization budget. Individual document_pages leaves only get summary:txt when the remaining LLM call budget (governed by max_llm_calls) allows it. For large documents with many pages, most leaf pages will not have summary:txt.

Annotations on image fragments (extracted document images): Same as Images below.

Images

Applies to: image/jpeg, image/png, image/gif, image/webp, image/tiff, image/bmp, image/heic, and other image MIME types.

Annotations on the Entity

Annotation	Type	Presence	Description
`extract:txt`	`str`	Always	Two-paragraph description: concise + detailed, generated by vision LLM
`extract:txtLang`	`str`	Always	Always `"english"` (LLM output language)
`extract:metadata`	`dict`	Always	ExifTool metadata (EXIF, XMP, IPTC — varies by image format)
`image:annotations`	`list[dict]`	Always	All detected labels. Each: `{"label": "..."}`. Does not include face embeddings (they exist as separate fragments)
`image:annotations-key`	`list[dict]`	Always	Investigation-relevant labels (e.g. `firearms`, `cash`)
`image:annotations-sensitive`	`list[dict]`	Always (may be empty list)	Sensitive content flags (e.g. `nudity_pornography`, `violence`)
`image:annotations-setting`	`list[dict]`	Always	Image general setting labels (e.g. `Photo`, `Indoor`, `Morning`)
`ner:entities`	`dict`	Usually	Named entities from the image description
`rag:chunks`	`list[dict]`	Always	RAG chunks from description text and image annotations
`rag:fulltext`	`str`	Always	Concatenated RAG text

Fragments Produced

Collection Type	Content	Presence	Record Fields
`face_crops`	Cropped face regions as separate image entities	When multiple faces detected (or single face covering < 70% of image area)	`fragment_index`, `fragment_count`, `fragment_image_bbox`

Annotations on face crop fragments:

Annotation	Presence	Notes
`image:annotations`	Always	Single detection: `{"label": "face", "bbox": [...], "confidence": ..., "embedding": {...}}`

Note: When exactly one face is detected and it covers ≥ 70% of the image area, no face crop fragment is created — the detection is written directly onto the parent entity's image:annotations.

Possible `image:annotations` Label Values

Labels come from a structured analysis model. These are the possible values:

Image type: Drawing, Photo, Screenshot, Scan, Security Camera Footage, Satellite Image

Setting: Indoor, Outdoor, plus a free-form category string

Conditions: Morning, Noon, Afternoon, Evening, Night, Sunny, Cloudy, Rainy, Snowy, Foggy, Spring, Summer, Autumn, Winter

Object detection (boolean flags): cars, motorcycles, trucks, planes, helicopters, boats, flags, drugs_pills, drugs_powder, drugs_other, credit_cards, id_cards, other_cards, luxury_items, cash, alcohol, tattoos, firearms, blood_blood_stains, plus free-form entries in other_objects

Sensitive categories (boolean flags): nudity_pornography, violence, terrorism, war, body_injury

Audio

Applies to: audio/mpeg, audio/wav, audio/ogg, audio/flac, audio/aac, and other audio MIME types.

Annotations on the Parent Entity

Annotation	Type	Presence	Description
`extract:txt`	`str`	Sometimes	Full transcript, if the audio has not been split
`extract:txtLang`	`str`	Sometimes	Detected language of transcript, if the audio has not been split
`extract:metadata`	`dict`	Always	ExifTool audio metadata (duration, bitrate, channels, etc.)
`ner:entities`	`dict`	Usually	Named entities from transcript text
`rag:chunks`	`list[dict]`	Usually	RAG chunks from transcript
`rag:fulltext`	`str`	Usually	Concatenated RAG text

Fragments Produced

Collection Type	Content	Presence	Record Fields
`audio_split`	Time-based or size-based audio segments	Always	`fragment_index`, `fragment_count`, `fragment_start`, `fragment_end`, `fragment_start_offset`, `fragment_end_offset`

Annotations on audio segment fragments: Each segment is individually transcribed, so carries its own extract:txt and extract:txtLang.

Video

Applies to: video/mp4, video/avi, video/quicktime, video/x-matroska, video/webm, and other video MIME types.

Video is the most complex input type — it produces two kinds of sub-products, each processed through media-specific paths.

Annotations on the Parent Entity

Annotation	Type	Presence	Description
`extract:metadata`	`dict`	Always	ExifTool video metadata (duration, resolution, codec, etc.)
`ner:entities`	`dict`	Usually	Named entities from transcript + descriptions
`summary:txt`	`str`	Optional	Summary of transcribed content
`rag:chunks`	`list[dict]`	Usually	RAG chunks
`rag:fulltext`	`str`	Usually	Concatenated RAG text
`materialized.ner:entities`	Usually	NER for all children elements

Fragments Produced

Collection Type	Content	Presence	Record Fields
`video_keyframes`	Keyframe images, filtered by saliency and hash similarity	Always (when video has visual stream)	`fragment_index`, `fragment_count`, `fragment_keyframe_number`, `fragment_start` (seconds), `fragment_end` (seconds)
`video_audio`	Audio track extracted from video (single entity)	Always (when video has audio stream)	`fragment_index: 0`, `fragment_count: 1`, `fragment_start: 0`, `fragment_end` (duration in seconds)

Annotations on keyframe fragments: Same as Images — each keyframe is processed as a standalone image.

Annotations on the audio fragment: Same as Audio — transcribed, NER'd.

Archives / Executables / Generic / Other

Entities that do not match a specific media type still receive baseline processing:

Annotation	Presence	Description
`extract:metadata`	Always (if attachment present)	ExifTool metadata
`extract:txt`	If Tika can parse it	Extracted text; archives are always skipped
`extract:txtLang`	If text extracted	Detected language
`ner:entities`	If text available	Named entities from extracted text (also parses record fields like `description` and `contents`)
`rag:chunks`	If text available	RAG chunks built from `extract:txt`, record fields (`description`, `contents`), and other annotations
`rag:fulltext`	If text available	Concatenated RAG text

Annotation Dictionary

Detailed schema for each annotation key.

`extract:txt`

Type: str
Content: Plain text. Multi-paragraph. Line breaks separate logical sections (pages, segments, sentences).
Source: Tika (documents), Whisper/AssemblyAI (audio), vision LLM (images).
On fragments: Contains only the fragment's portion of text.
On parent after merge: Contains the concatenated text of all children.

`extract:txtLang`

Type: str
Content: Language name string (e.g. "english", "german", "french").
Note: Detected from the first 100k characters of extract:txt. For images, always "english" (the captioning language).

`extract:metadata`

Type: dict
Content: Keys are ExifTool field names with namespace prefixes stripped (File:, ExifTool:, SourceFile removed). Values are strings, numbers, or nested structures depending on the metadata field.
Examples:

{
  "ImageWidth": 1920,
  "ImageHeight": 1080,
  "MIMEType": "image/jpeg",
  "DateTimeOriginal": "2024:01:15 14:30:00",
  "GPSLatitude": "48.8566",
  "GPSLongitude": "2.3522",
  "Duration": "00:05:32",
  "AudioBitrate": "128000"
}

`extract:frontmatter`

Type: dict
Content: Optional keys — only present when detected:
- toc — table of contents
- abstract — document abstract
- glossary — glossary section
- introduction — introduction text

`ner:entities`

Type: dict
Content: Keys are lowercase entity labels, values are deduplicated lists of entity strings.

{
  "person": ["Alice Johnson", "Bob Smith"],
  "org": ["Acme Corp", "United Nations"],
  "loc": ["New York", "Paris"],
  "date": ["January 2024", "2023-12-01"],
}

`summary:txt`

Type: str
Content: LLM-generated natural language summary. For multi-page documents with fragments, this is a hierarchical summary that considers section types.
Typical length: Proportional to input document and inversely proportional to fragment depth, up to around 4096 tokens.

`image:annotations`

Type: list[dict]
Content: Each item has at minimum {"label": "..."}. Items from face/object detection may also include:

[
  {"label": "Photo"},
  {"label": "Outdoor"},
  {"label": "cars"},
  {"label": "face", "bbox": [120, 45, 280, 210], "confidence": 0.97,
   "embedding": {"#type": "VECTOR", "data": [...], "metadata": {...}}},
  {"label": "image embedding",
   "embedding": {"#type": "VECTOR", "data": [...], "metadata": {...}}}
]

Sub-keys for filtered views:
- image:annotations-key — object detection labels only
- image:annotations-sensitive — sensitive content flags only
- image:annotations-setting — image type + setting + conditions only These fields never contain embeddings.

`section:type`

Type: str
Values: textual, image, tabular, list, mixed
Present on: Fragment entities only (pages, sections).

`rag:chunks`

Type: list[dict]
Content: Array of chunk objects. Max count governed by rag_max_chunks (default: 100).

[
  {
    "metadata": {
      "source": "entity-uid-here",
      "entity_type": "os_document",
      "chunk_index": 0,
      "text": "The actual text content of this chunk...",
      "embedding": {
        "#type": "VECTOR",
        "data": [0.012, -0.034, 0.056, ...],
        "metadata": {
          "model_name": "sentence-transformers/all-MiniLM-L6-v2",
          "dim": 384
        }
      }
    }
  }
]

`rag:fulltext`

Type: str
Content: All rag:chunks[*].metadata.text values concatenated.

`materialized.*`

Type: Varies (mirrors the type of the source annotation)
Content: Accumulated values from child fragment annotations, written to the parent entity in the search index. The key mirrors the original annotation key prefixed with materialized..
Example: materialized.ner:entities contains the combined NER results from all children fragments.

Fragment Dictionary

Common Record Fields on All Fragments

Field	Type	Always Present	Description
`fragment_index`	`int`	✓	0-based position among siblings
`fragment_count`	`int`	✓	Total sibling count
`fragment_collection_type`	`str`	✓	Category (see per-type tables below)
`os_item_content_type`	`str`	✓	MIME type of the fragment content
`os_has_attachment`	`bool`	✓	Whether binary content is stored
`fragment_start`	`int`	✓	Start position (unit depends on type)
`fragment_end`	`int`	✓	End position (unit depends on type)

`document_pages` / `document_page_groups`

For page/slide-based types (PDF, DOCX, DOC, ODT, PPTX, PPT, ODP).

Field	Type	Always	Description
`fragment_page_number`	`int`	✓	Starting page number
`fragment_start`	`int`	✓	Start page
`fragment_end`	`int`	✓	End page
`fragment_start_offset`	`int`	◇	Context overlap (chars before this range)
`fragment_end_offset`	`int`	◇	Context overlap (chars after this range)
`os_item_name`	`str`	◇	Section heading (if detected)
`section:type` (annotation)	`str`	✓	`textual`, `image`, `tabular`, `list`, `mixed`

`table_rows` / `table_row_groups`

For row-based types (XLSX, XLS, CSV, TSV, ODS).

Field	Type	Always	Description
`fragment_start`	`int`	✓	Start row index
`fragment_end`	`int`	✓	End row index
`os_item_name`	`str`	◇	Sheet name or heading
`section:type` (annotation)	`str`	✓	Always `tabular`

`document_images`

Field	Type	Always	Description
`os_item_content_type`	`str`	✓	Image MIME type (e.g. `image/png`)
`fragment_index`	`int`	✓	Image index within the document
Annotations			Same as standalone Image processing

`video_keyframes`

Field	Type	Always	Description
`fragment_keyframe_number`	`int`	✓	Keyframe sequence number
`fragment_start`	`int`	✓	Timestamp in seconds
`fragment_end`	`int`	✓	Timestamp in seconds
`os_item_content_type`	`str`	✓	`image/png` or `image/jpeg`
Annotations			Same as standalone Image processing

`video_audio`

Field	Type	Always	Description
`fragment_start`	`int`	✓	Always `0`
`fragment_end`	`int`	✓	Duration in seconds (ceil)
`os_item_content_type`	`str`	✓	Audio MIME type
Annotations			Same as standalone Audio processing

`audio_split`

Field	Type	Always	Description
`fragment_start`	`int`	✓	Start time in seconds
`fragment_end`	`int`	✓	End time in seconds
`fragment_start_offset`	`int`	◇	Overlap with previous segment (seconds)
`fragment_end_offset`	`int`	◇	Overlap with next segment (seconds)
Annotations			`extract:txt`, `extract:txtLang` for the segment

`face_crops`

Field	Type	Always	Description
`fragment_image_bbox`	`list[int]`	✓	`[x1, y1, x2, y2]` bounding box in pixels
`os_item_content_type`	`str`	✓	`image/png`
Annotations			Single-item `image:annotations` with `label` (= `face`), `bbox`, `confidence`, `embedding`

Shared Data Formats

Embedding Vectors

All embeddings in annotations share the same envelope:

{
  "#type": "VECTOR",
  "data": [0.012, -0.034, 0.056, ...],
  "metadata": {
    "model_name": "model-identifier",
    "model_version": "version-string",
    "dim": 512
  }
}

Context	`model_name`	Typical `dim`
Face embedding	`facenet_pytorch_inception_resnet_v1`	512
Image embedding (CLIP)	`vitb32`	512
RAG chunk embedding	(configurable sentence-transformer)	384+

Processing Status

Entities carry a processing_status in the search index:

Status	Meaning
`RUNNING`	Pipeline is actively processing this entity
`COMPLETED`	Processing finished successfully
`FAILED`	Processing encountered an error

Cheat Sheet

What annotations will my parent entity have?

This table shows what is on the parent entity in OpenSearch. When an entity is split into fragments, some annotations (like extract:txt) live only on fragments and are not directly on the parent — use get_entity_annotations with recurse=True to get the merged view.

	Documents	Presentations	Images	Audio	Video	Spreadsheets	Archives / Other
`extract:txt`	◇ ^1^	◇ ^1^	✓	◇ ^1^	— ^2^	◇ ^1^	◇
`extract:txtLang`	◇ ^1^	◇ ^1^	✓	◇ ^1^	— ^2^	◇ ^1^	◇
`extract:metadata`	✓	✓	✓	✓	✓	✓	✓
`ner:entities`	◇	◇	◇	◇	◇	◇	◇
`summary:txt`	✓	—	—	◇	◇	—	—
`extract:frontmatter`	◇	—	—	—	—	—	—
`image:annotations`	—	—	✓	—	—	—	—
`image:annotations-key`	—	—	✓	—	—	—	—
`image:annotations-sensitive`	—	—	✓	—	—	—	—
`image:annotations-setting`	—	—	✓	—	—	—	—
`translate:txt:{lang}`	—	—	—	◇	◇	—	—
`translate:src`	—	—	—	◇	◇	—	—
`rag:chunks`	✓	✓	✓	◇	◇	◇	◇
`rag:fulltext`	◇	◇	✓	◇	◇	◇	◇
`materialized.*`	◇ ^3^	◇ ^3^	◇	◇	◇ ^3^	—	—

✓ = always ◇ = conditional — = not applicable

^1^ Only on the parent when the entity was not split into fragments. When split, extract:txt and extract:txtLang live on the leaf fragments instead. ^2^ Video parent never has extract:txt directly — text lives on keyframe and audio fragments. ^3^ Present when the entity was fragmented; contains rolled-up child annotations (e.g. materialized.ner:entities).

What fragments will my entity produce?

Input Type	Fragment Types	Count
Document (PDF, DOCX, DOC, ODT)	`document_pages` + `document_page_groups` + `document_images`	Pages: 1 per page (plus groups by `max_pages`). Images: varies.
Presentation (PPTX, PPT, ODP)	`document_pages` + `document_page_groups` + `document_images`	1 per slide (plus groups). Images: varies.
Spreadsheet (XLSX, XLS, CSV, TSV, ODS)	`table_rows` (+ `table_row_groups`)	Row-range chunks (controlled by `max_rows`).
Image	`face_crops`	0 (single/large face) or N (multi-face)
Audio	`audio_split`	Depends on duration and segment config
Video	`video_keyframes` + `video_audio`	Keyframes: varies (filtered). Audio: exactly 1.
Archives / Executables / Other	(none)	Metadata-only — no fragments produced.

Table of Contents​

Quick Start: What Happens to My File?​

Entity Structure in OpenSearch​

Document Shape​

source_entity_uid vs os_parent_uid​

Querying Annotations​

Querying Fragments Directly​

Get all fragments of a file (flat)​

Reconstruct the tree​

Get only direct children of a specific node​

Fragment Hierarchy Diagram​

Output by Input Type​

Documents​

Splitting Behavior by MIME Type​

Annotations on the Parent Entity​

Fragments Produced​

Annotations on the main document​

Annotations on document_page_groups Fragments​

Annotations on document_pages (Leaf) Fragments​

Images​

Annotations on the Entity​

Fragments Produced​

Possible image:annotations Label Values​

Audio​

Annotations on the Parent Entity​

Fragments Produced​

Video​

Annotations on the Parent Entity​

Fragments Produced​

Archives / Executables / Generic / Other​

Annotation Dictionary​

extract:txt​

extract:txtLang​

extract:metadata​

extract:frontmatter​

ner:entities​

summary:txt​

image:annotations​

section:type​

rag:chunks​

rag:fulltext​

materialized.*​

Fragment Dictionary​

Common Record Fields on All Fragments​

document_pages / document_page_groups​

table_rows / table_row_groups​

document_images​

video_keyframes​

video_audio​

audio_split​

face_crops​

Shared Data Formats​

Embedding Vectors​

Processing Status​

Cheat Sheet​

What annotations will my parent entity have?​

What fragments will my entity produce?​

Table of Contents

Quick Start: What Happens to My File?

Entity Structure in OpenSearch

Document Shape

`source_entity_uid` vs `os_parent_uid`

Querying Annotations

Querying Fragments Directly

Get all fragments of a file (flat)

Reconstruct the tree

Get only direct children of a specific node

Fragment Hierarchy Diagram

Output by Input Type

Documents

Splitting Behavior by MIME Type

Annotations on the Parent Entity

Fragments Produced

Annotations on the main document

Annotations on `document_page_groups` Fragments

Annotations on `document_pages` (Leaf) Fragments

Images

Annotations on the Entity

Fragments Produced

Possible `image:annotations` Label Values

Audio

Annotations on the Parent Entity

Fragments Produced

Video

Annotations on the Parent Entity

Fragments Produced

Archives / Executables / Generic / Other

Annotation Dictionary

`extract:txt`

`extract:txtLang`

`extract:metadata`

`extract:frontmatter`

`ner:entities`

`summary:txt`

`image:annotations`

`section:type`

`rag:chunks`

`rag:fulltext`

`materialized.*`

Fragment Dictionary

Common Record Fields on All Fragments

`document_pages` / `document_page_groups`

`table_rows` / `table_row_groups`

`document_images`

`video_keyframes`

`video_audio`

`audio_split`

`face_crops`

Shared Data Formats

Embedding Vectors

Processing Status

Cheat Sheet

What annotations will my parent entity have?

What fragments will my entity produce?