# NeuronPlant / AI platform

Serving, RAG services, NIM / runtime integration, and the data path from sources through ingest, train or RAG, Docker/Kubernetes, KV / prefix cache, operate / observe, handover.

Canonical HTML: https://neuronplant.com/applications
Collection file: https://neuronplant.com/llms/pipeline.txt

NeuronPlant is not an NVIDIA Cloud Partner. NVOnline packs are named, not reprinted. Customer names and customer specifications publish only after written approval.

# NeuronPlant applications / platform

Accelerators without a platform are inventory. The factory includes serving, RAG services, NIM / runtime integration, and the path from data to workload.

NeuronPlant delivers AI factories. After power, cooling, racks and accelerators exist, there is still a platform: serving, RAG services, NIM / runtime integration, plus a data path from sources through ingest, storage, train/fine-tune or RAG, accelerator runtime (Docker images, Kubernetes cell), inference serving (NVIDIA NIM from public docs), KV / prefix cache on large serving, applications, then operate / observe.

NeuronPlant owns that platform path as part of the factory delivery: architecture, integration onto the plant (network planes, storage tiers, NIM/profile sizing from NVIDIA public documentation, KV / prefix cache as a serving-plane constraint), and handover. NeuronPlant does not claim to write every model, is not a generic software ISV, and is not an NVIDIA Cloud Partner.

## Stages

### 01 Data sources
Where bytes originate: operational systems, files, streams, instruments, model registries.
Plant: Lands on the North-South / front-end plane, not on the East-West compute rails.

### 02 Ingest
Controlled intake: validation, classification, residency, and a path that can be audited.
Plant: Sized to nightly rebuilds or continuous feed. Must not contend with checkpoint burst.

### 03 Storage
Tiers, not a single NAS. Documents, datasets, checkpoints, indexes and weights have different machines.
Plant: Storage/data fabric off the East-West compute rails when dedicated. Object, parallel FS, and node-local NVMe are different jobs. Do not auto-label this North-South or East-West without that context.

### 04A Train / fine-tune
Distributed training or adaptation on a coupled accelerator domain. Checkpoints are a storage event.
Plant: East-West compute fabric (Quantum-X or Spectrum-X) plus HPS sequential GB/s. See SuperPOD public RAs.

### 04B RAG / retrieval
Document store plus a vector index. Ingest and query are IOPS and metadata, not training scratch.
Plant: Keep the index off the training filesystem. North-South / front-end Ethernet is usually enough if the index is close. That is not the East-West compute fabric.

### 05 Accelerator runtime
The cluster that actually runs the job. Workloads ship as Docker images. Kubernetes schedules, restarts, and places them. Placement, isolation, and a scheduler that knows the fabric.
Plant: A starting cell is three masters and three workers. Masters may be VMs. Workers that hold accelerators are usually bare metal. Observe sits on this plane. Same accelerators as the hall.

### 06 Inference platform
Serving plane. NVIDIA NIM is the cited runtime for profiled LLM serving. Precision, TP size and LoRA come from NVIDIA's public support matrix. On a large job, KV and prefix cache are a memory hierarchy, not the weight disk.
Plant: Local NVMe for images and weights. KV lives in HBM first, then host overflow, then a shared prefix cache across the serving plane. Size that hierarchy separately from the profile disk.

### 07 Applications
Where the factory meets the enterprise: assistants, tools, batch, APIs. Ownership of the app stays with the operator.
Plant: North-South front-end path, identity, and a handover that names interfaces. Not a NeuronPlant SaaS. Not management/OOB.

## KV cache / large serving

KV cache is a serving constraint, not a storage array.

On transformer inference, each new token attends to the keys and values of every prior token. Those K/V tensors are cached so the runtime does not recompute the whole prefix on every decode step.

A small job keeps KV in accelerator HBM. GPU is the usual home. That is enough.

A large workload is different. Long context, many concurrent sessions, and shared system prompts or RAG prefixes fill HBM. The plant then needs a memory hierarchy: HBM → RAM → NVMe → remote, plus a prefix cache and KV-aware routing so the fleet does not recompute the same context.

Prefill writes the cache. Decode reads it. That split changes how you size HBM, host memory, the inference fabric, and sometimes a fast tier next to the accelerators.

This is not a storage array. It is not a training checkpoint pool. The constraint is the job: serving plane, prefix cache, prefill versus decode, HBM versus host.

The question at scale: do not recompute context across a growing fleet. Hierarchy is HBM → RAM → NVMe → remote. Too heavy for the homepage. It lives here.

- On-device HBM: First home for active KV. Fine until context length and concurrency fill the device.
- Host overflow / RAM: When HBM fills, KV spills to host memory. That is a memory and fabric size, not a new LUN on the training filesystem.
- NVMe: Next overflow when host RAM is not enough. Still a serving-plane tier, still not the training filesystem.
- Remote: A shared cache off the node when the fleet should not keep rebuilding the same prefix. Latency and fabric become the spec.
- Local vs prefix: Per-request KV stays local. Shared system prompts and RAG prefixes belong in a prefix cache so the fleet does not recompute the same context.
- KV-aware routing: Send the next turn to the replica that already holds the cache. Round-robin across a growing fleet is how you pay prefill again.

## RAG is a data architecture, not a vector database.

Retrieval is a path with named data states. Source → landing → parse → process → chunk → embed → index → retrieve → LLM. A vector engine is one station. It is not the plant.

- 01 Source (Authoritative corpus. Systems of record, files, streams.) Name where truth lives. Size storage from this TB, not from the index.
- 02 Landing (As-received bytes. Copy or reference, written down.) Controlled intake onto the plant. Must not contend with checkpoint burst.
- 03 Parse (Structure extracted. Layout, tables, attachments.) CPU workers, not accelerator nodes. Failures here poison every downstream state.
- 04 Process (Cleaned, classified, residency applied.) ACL and retention start here, not at the chatbot.
- 05 Chunk (Retrieval units. Overlap and size are a design.) Changing chunking is a re-embed. Treat it as a plant event.
- 06 Embed (Vectors for those chunks, versioned with the model.) An embedding-model change rebuilds this state and the index. GPU is the common accelerator.
- 07 Index (Queryable ANN / keyword / hybrid.) IOPS and metadata. Keep off the training filesystem.
- 08 Retrieve (Candidates for this query, under ACL.) Latency to the serving plane. Delete and ACL must propagate here, not only in the source.
- 09 LLM (Generated answer. Not a source of truth.) Inference plus KV. The corpus does not live in the prompt.

## If you cannot rebuild it, you do not own it.

For every state: where it lives, how long, what is authoritative, how it rebuilds, how it is backed up, how it is deleted, and who is allowed to delete it.

- Where: Which tier and which plane. Landing is not the index. The index is not the LLM.
- How long: Retention per state. Hot retrieve, legal hold, and disposable rebuilds are different clocks.
- Authoritative: Source of record. Embeddings are derived. Chat transcripts are not the corpus.
- Rebuild: From which upstream state, in what window, with what compute. Write it before the first index.
- Backup: What is cheaper to protect than to rebuild. Derived vectors are often the latter.
- Delete / who deletes: A named owner. Source delete is not index delete until it has propagated.
- Embedding-model change: A plant event: re-chunk or re-embed, new index, dual-run window. Not a config toggle on the chatbot.
- ACL / delete propagation: Permissions and erasures have to reach retrieve, not only the system of record. Otherwise the factory leaks.

## Runtime / Operate

Canonical page: /runtime

Docker: the workload ships as an image. Same artifact from lab to factory.

Kubernetes: schedule, restart, and place those containers. A fleet that is ssh-and-excel will not stay up.

Starting cell: three masters (VM-capable) and three workers (workers that hold accelerators are usually bare metal). Fluent Bit is the example log shipper.

## Fork

Train / fine-tune and RAG are different I/O machines that can share a hall. Mixing them on one filesystem is how jobs stall.

## In scope

- Architecture of the data-to-workload path
- Integration onto the plant: network planes, storage tiers, rack placement
- NVIDIA NIM / profile sizing from NVIDIA public documentation
- KV / prefix cache as a serving-plane constraint on large inference, including HBM → RAM → NVMe → remote and KV-aware routing
- RAG as a data path with named states, sized from source data, with a rebuildable lifecycle
- Docker images and a Kubernetes cell (3 masters, 3 workers) as the factory runtime
- Observe / monitor as part of that plane, with Fluent Bit as the example shipper
- Orchestration and MLOps as interfaces, not as a NeuronPlant product
- Handover: runbooks, boundaries, and who operates which plane

## Out of scope

- Writing every model the customer will run
- A generic software ISV or SaaS application suite
- NVIDIA partnership, Cloud Partner, or NVOnline status
- Named customer logos or invented case studies

## Related factory templates

- Private AI (/ai-factories/private-ai): Enterprise sources, controlled ingest, RAG, NIM serving, internal applications.
- Training cluster (/ai-factories/training): Datasets, staging, coupled fabric, checkpoints, scheduler, experiment platform.
- AI-as-a-Service (/ai-factories/ai-as-a-service): Onboarding, tenancy, accelerator pools, shared fabric, control plane, customer workloads.

NVIDIA NIM support matrix: https://docs.nvidia.com/nim/large-language-models/latest/support-matrix.html


Canonical: https://neuronplant.com/applications

# KV cache is a serving constraint, not a storage array.

On transformer inference, each new token attends to the keys and values of every prior token. Those K/V tensors are cached so the runtime does not recompute the whole prefix on every decode step.

A small job keeps KV in accelerator HBM. GPU is the usual home. That is enough.

A large workload is different. Long context, many concurrent sessions, and shared system prompts or RAG prefixes fill HBM. The plant then needs a memory hierarchy: HBM → RAM → NVMe → remote, plus a prefix cache and KV-aware routing so the fleet does not recompute the same context.

Prefill writes the cache. Decode reads it. That split changes how you size HBM, host memory, the inference fabric, and sometimes a fast tier next to the accelerators.

This is not a storage array. It is not a training checkpoint pool. The constraint is the job: serving plane, prefix cache, prefill versus decode, HBM versus host.

The question at scale: do not recompute context across a growing fleet. Hierarchy is HBM → RAM → NVMe → remote. Too heavy for the homepage. It lives here.

- On-device HBM: First home for active KV. Fine until context length and concurrency fill the device.
- Host overflow / RAM: When HBM fills, KV spills to host memory. That is a memory and fabric size, not a new LUN on the training filesystem.
- NVMe: Next overflow when host RAM is not enough. Still a serving-plane tier, still not the training filesystem.
- Remote: A shared cache off the node when the fleet should not keep rebuilding the same prefix. Latency and fabric become the spec.
- Local vs prefix: Per-request KV stays local. Shared system prompts and RAG prefixes belong in a prefix cache so the fleet does not recompute the same context.
- KV-aware routing: Send the next turn to the replica that already holds the cache. Round-robin across a growing fleet is how you pay prefill again.

Canonical: https://neuronplant.com/applications#kv-cache

# RAG is a data architecture, not a vector database.

Retrieval is a path with named data states. Source → landing → parse → process → chunk → embed → index → retrieve → LLM. A vector engine is one station. It is not the plant.

# If you cannot rebuild it, you do not own it.

For every state: where it lives, how long, what is authoritative, how it rebuilds, how it is backed up, how it is deleted, and who is allowed to delete it.

- Where: Which tier and which plane. Landing is not the index. The index is not the LLM.
- How long: Retention per state. Hot retrieve, legal hold, and disposable rebuilds are different clocks.
- Authoritative: Source of record. Embeddings are derived. Chat transcripts are not the corpus.
- Rebuild: From which upstream state, in what window, with what compute. Write it before the first index.
- Backup: What is cheaper to protect than to rebuild. Derived vectors are often the latter.
- Delete / who deletes: A named owner. Source delete is not index delete until it has propagated.
- Embedding-model change: A plant event: re-chunk or re-embed, new index, dual-run window. Not a config toggle on the chatbot.
- ACL / delete propagation: Permissions and erasures have to reach retrieve, not only the system of record. Otherwise the factory leaks.

Canonical: https://neuronplant.com/applications#rag

# NeuronPlant runtime / operate

The factory runs as a fleet of images, not as a snowflake hall.

After the accelerators exist, the workload still has to ship, schedule, and stay up. Docker is the artifact. Kubernetes is the plant's control plane. That plane rides management, not application North-South. A starting cell is three masters and three workers. Observability is part of that plane.

## Why Docker

The workload ships as an image.

Serving, ingest, RAG, NIM runtime, plant services. The hall is not a unique snowflake OS. Same artifact from lab to factory.

## Why Kubernetes

Schedule, restart, and place those containers across the plant.

Accelerator nodes, CPU nodes, and plant services in one control plane. Not because cloud native is a slogan. An AI factory is a fleet. A fleet that is ssh-and-excel will not stay up.

## What Kubernetes requires

A starting cell: three masters and three workers. That is a plant constraint, not a certification syllabus.

- Three masters: Three masters so etcd and the API stay up if one dies. That is why 3, not 1.
- Three workers: Three workers as a starting factory cell. The cell can grow. Accelerator workers are not the same machine class as the masters.
- Masters may be VMs: Masters may be virtual machines. Workers that hold accelerators are usually bare metal. The plant does not require three physical master servers.
- Plant jobs, not a SKU list: Network, storage classes, and an accelerator device plugin are jobs on the plane. They are not a product catalog we refresh every week.

## Observability

Logs, metrics, and traces are part of the plane.

Fluent Bit is the example log shipper. A DaemonSet at the edge of every node, forwarding to the plant's log store. That is the note. Not a weekly shopping list of dashboards.

## Kubernetes as production infrastructure

Not K8s plus vLLM as a slogan. A factory control plane with CPU workers, accelerator workers, and the platform services that must not sit on the expensive nodes.

- Control plane: API, etcd, schedulers. Masters may be VMs. This plane stays up if a GPU node dies.
- CPU workers: Ingest, parse, RAG services, registries, identity, plant jobs. Sized as a cell, grown as a gate. See /expansion.
- Accelerator workers: GPU is the common case. TPU and other ASICs when that is the job. Bare metal. Device plugin. Not a place to park Prometheus.
- Registry: The images the factory actually runs. On the management plane, not on an HBM node.
- Secrets: A named store and a rotation path. Not files on the GPU worker.
- Identity: Who may schedule, who may retrieve, who may delete. RAG ACL starts here and has to reach the index.
- RAG services: Parse, embed jobs, retrievers. CPU first. Accelerators only for the embed/infer step that needs them.
- Keep platform off accelerator nodes: Do not park registry, logging, identity, or the control plane on expensive accelerator workers. That is how a fleet starves itself.

## In scope

- Docker as the image that moves serving, ingest, RAG, NIM, and plant services from lab to factory
- Kubernetes as the control plane that schedules that fleet across accelerator, CPU, and plant nodes
- A starting cell of three masters and three workers, with masters allowed to be VMs
- Observability as part of the plane, with Fluent Bit as the example log shipper
- Production Kubernetes: control plane, CPU workers vs accelerator workers, registry, secrets, identity, RAG services
- Platform services kept off expensive accelerator nodes

## Out of scope

- A CNCF certification syllabus or weekly operator-version pin list
- A CNI, service-mesh, or dashboard catalog we would have to refresh every week
- Requiring three physical master servers when VMs will hold quorum
- Treating ssh-and-excel as the runbook for an accelerator hall
- K8s plus vLLM as the whole platform story
- Named TTFT / ITL numbers as a NeuronPlant product

Canonical page: /runtime. Also on /what-we-build#operate and /applications#runtime.


Canonical: https://neuronplant.com/runtime

# Kubernetes as production infrastructure

Not K8s plus vLLM as a slogan. A factory control plane with CPU workers, accelerator workers, and the platform services that must not sit on the expensive nodes.

- Control plane: API, etcd, schedulers. Masters may be VMs. This plane stays up if a GPU node dies.
- CPU workers: Ingest, parse, RAG services, registries, identity, plant jobs. Sized as a cell, grown as a gate. See /expansion.
- Accelerator workers: GPU is the common case. TPU and other ASICs when that is the job. Bare metal. Device plugin. Not a place to park Prometheus.
- Registry: The images the factory actually runs. On the management plane, not on an HBM node.
- Secrets: A named store and a rotation path. Not files on the GPU worker.
- Identity: Who may schedule, who may retrieve, who may delete. RAG ACL starts here and has to reach the index.
- RAG services: Parse, embed jobs, retrievers. CPU first. Accelerators only for the embed/infer step that needs them.
- Keep platform off accelerator nodes: Do not park registry, logging, identity, or the control plane on expensive accelerator workers. That is how a fleet starves itself.

Canonical: https://neuronplant.com/runtime#plane