01 / Data sources
Where bytes originate: operational systems, files, streams, instruments, model registries.
Plant: Lands on the North-South / front-end plane, not on the East-West compute rails.
NP-APP/Applications / Platform
Accelerators without a platform are inventory.
After the hall exists, supplied by a colo partner or a custom-system builder, there is still a path from data to workload. NeuronPlant engineers that path onto the accelerator plant. We do not write every model, and this is not a generic software product.
Accelerators without a platform are inventory. The factory includes serving, RAG services, NIM / runtime integration, and the path from data to workload.
NP-PIPE / Data to workload
Schematic · not a product flowchart
Stages
02 / Ingest
Controlled intake: validation, classification, residency, and a path that can be audited.
Plant: Sized to nightly rebuilds or continuous feed. Must not contend with checkpoint burst.
03 / Storage
Tiers, not a single NAS. Documents, datasets, checkpoints, indexes and weights have different machines.
Plant: Storage/data fabric off the East-West compute rails when dedicated. Object, parallel FS, and node-local NVMe are different jobs. Do not auto-label this North-South or East-West without that context.
04A / Train / fine-tune
Distributed training or adaptation on a coupled accelerator domain. Checkpoints are a storage event.
Plant: East-West compute fabric (Quantum-X or Spectrum-X) plus HPS sequential GB/s. See SuperPOD public RAs.
04B / RAG / retrieval
Document store plus a vector index. Ingest and query are IOPS and metadata, not training scratch.
Plant: Keep the index off the training filesystem. North-South / front-end Ethernet is usually enough if the index is close. That is not the East-West compute fabric.
05 / Accelerator runtime
The cluster that actually runs the job. Workloads ship as Docker images. Kubernetes schedules, restarts, and places them. Placement, isolation, and a scheduler that knows the fabric.
Plant: A starting cell is three masters and three workers. Masters may be VMs. Workers that hold accelerators are usually bare metal. Observe sits on this plane. Same accelerators as the hall.
06 / Inference platform
Serving plane. NVIDIA NIM is the cited runtime for profiled LLM serving. Precision, TP size and LoRA come from NVIDIA's public support matrix. On a large job, KV and prefix cache are a memory hierarchy, not the weight disk.
Plant: Local NVMe for images and weights. KV lives in HBM first, then host overflow, then a shared prefix cache across the serving plane. Size that hierarchy separately from the profile disk.
07 / Applications
Where the factory meets the enterprise: assistants, tools, batch, APIs. Ownership of the app stays with the operator.
Plant: North-South front-end path, identity, and a handover that names interfaces. Not a NeuronPlant SaaS. Not management/OOB.
NP-APP-KV/Serving plane / large workload
KV cache is a serving constraint, not a storage array.
On transformer inference, each new token attends to the keys and values of every prior token. Those K/V tensors are cached so the runtime does not recompute the whole prefix on every decode step.
A small job keeps KV in accelerator HBM. GPU is the usual home. That is enough.
A large workload is different. Long context, many concurrent sessions, and shared system prompts or RAG prefixes fill HBM. The plant then needs a memory hierarchy: HBM → RAM → NVMe → remote, plus a prefix cache and KV-aware routing so the fleet does not recompute the same context.
Prefill writes the cache. Decode reads it. That split changes how you size HBM, host memory, the inference fabric, and sometimes a fast tier next to the accelerators.
This is not a storage array. It is not a training checkpoint pool. The constraint is the job: serving plane, prefix cache, prefill versus decode, HBM versus host.
The question at scale: do not recompute context across a growing fleet. Hierarchy is HBM → RAM → NVMe → remote. Too heavy for the homepage. It lives here.
On-device HBM
First home for active KV. Fine until context length and concurrency fill the device.
Host overflow / RAM
When HBM fills, KV spills to host memory. That is a memory and fabric size, not a new LUN on the training filesystem.
NVMe
Next overflow when host RAM is not enough. Still a serving-plane tier, still not the training filesystem.
Remote
A shared cache off the node when the fleet should not keep rebuilding the same prefix. Latency and fabric become the spec.
Local vs prefix
Per-request KV stays local. Shared system prompts and RAG prefixes belong in a prefix cache so the fleet does not recompute the same context.
KV-aware routing
Send the next turn to the replica that already holds the cache. Round-robin across a growing fleet is how you pay prefill again.
NP-APP-RAG/Applications / RAG
RAG is a data architecture, not a vector database.
Retrieval is a path with named data states. Source → landing → parse → process → chunk → embed → index → retrieve → LLM. A vector engine is one station. It is not the plant.
NP-APP / RAG path
Nine states. One corpus path.
Source to LLM. Each box is a data state, not a vendor. The vector index is station 07.
Source → landing → parse → process → chunk → embed → index → retrieve → LLM
01 / Source
Authoritative corpus. Systems of record, files, streams.
Name where truth lives. Size storage from this TB, not from the index.
02 / Landing
As-received bytes. Copy or reference, written down.
Controlled intake onto the plant. Must not contend with checkpoint burst.
03 / Parse
Structure extracted. Layout, tables, attachments.
CPU workers, not accelerator nodes. Failures here poison every downstream state.
04 / Process
Cleaned, classified, residency applied.
ACL and retention start here, not at the chatbot.
05 / Chunk
Retrieval units. Overlap and size are a design.
Changing chunking is a re-embed. Treat it as a plant event.
06 / Embed
Vectors for those chunks, versioned with the model.
An embedding-model change rebuilds this state and the index. GPU is the common accelerator.
07 / Index
Queryable ANN / keyword / hybrid.
IOPS and metadata. Keep off the training filesystem.
08 / Retrieve
Candidates for this query, under ACL.
Latency to the serving plane. Delete and ACL must propagate here, not only in the source.
09 / LLM
Generated answer. Not a source of truth.
Inference plus KV. The corpus does not live in the prompt.
RAG storage is sized from source data, not vector capacity. Capabilities / RAG storage.
NP-APP-LIFE/Applications / data lifecycle
If you cannot rebuild it, you do not own it.
For every state: where it lives, how long, what is authoritative, how it rebuilds, how it is backed up, how it is deleted, and who is allowed to delete it.
Where
Which tier and which plane. Landing is not the index. The index is not the LLM.
How long
Retention per state. Hot retrieve, legal hold, and disposable rebuilds are different clocks.
Authoritative
Source of record. Embeddings are derived. Chat transcripts are not the corpus.
Rebuild
From which upstream state, in what window, with what compute. Write it before the first index.
Backup
What is cheaper to protect than to rebuild. Derived vectors are often the latter.
Delete / who deletes
A named owner. Source delete is not index delete until it has propagated.
Embedding-model change
A plant event: re-chunk or re-embed, new index, dual-run window. Not a config toggle on the chatbot.
ACL / delete propagation
Permissions and erasures have to reach retrieve, not only the system of record. Otherwise the factory leaks.
NP-RT/Runtime / Operate
The factory runs as a fleet of images, not as a snowflake hall.
After the accelerators exist, the workload still has to ship, schedule, and stay up. Docker is the artifact. Kubernetes is the plant's control plane. That plane rides management, not application North-South. A starting cell is three masters and three workers. Observability is part of that plane.
DWG-RT-01 · laying out
Why Docker
The workload ships as an image.
Serving, ingest, RAG, NIM runtime, plant services. The hall is not a unique snowflake OS. Same artifact from lab to factory.
Why Kubernetes
Schedule, restart, and place those containers across the plant.
Accelerator nodes, CPU nodes, and plant services in one control plane. Not because cloud native is a slogan. An AI factory is a fleet. A fleet that is ssh-and-excel will not stay up.
3 masters · 3 workers
Three masters so etcd and the API stay up if one dies. That is why 3, not 1. Three workers as a starting factory cell. The cell can grow. Accelerator workers are not the same machine class as the masters. Masters may be virtual machines. Workers that hold accelerators are usually bare metal. The plant does not require three physical master servers.
Fluent Bit
Logs, metrics, and traces are part of the plane.
Fluent Bit is the example log shipper. A DaemonSet at the edge of every node, forwarding to the plant's log store. That is the note. Not a weekly shopping list of dashboards.
In scope
- Architecture of the data-to-workload path
- Integration onto the plant: network planes, storage tiers, rack placement
- NVIDIA NIM / profile sizing from NVIDIA public documentation
- KV / prefix cache as a serving-plane constraint on large inference, including HBM → RAM → NVMe → remote and KV-aware routing
- RAG as a data path with named states, sized from source data, with a rebuildable lifecycle
- Docker images and a Kubernetes cell (3 masters, 3 workers) as the factory runtime
- Observe / monitor as part of that plane, with Fluent Bit as the example shipper
- Orchestration and MLOps as interfaces, not as a NeuronPlant product
- Handover: runbooks, boundaries, and who operates which plane
Not in scope
- Writing every model the customer will run
- A generic software ISV or SaaS application suite
- NVIDIA partnership, Cloud Partner, or NVOnline status
- Named customer logos or invented case studies
NP-APP-PLANT/On the plant
The pipeline has to land on planes that already exist.
Network, storage and serving are not a separate software project. They are the same factory, continued.
Structured cabling
The platform rides the same structured cabling and passive infrastructure. East-West compute, storage/data, North-South front-end, and management/OOB are separate plants. Cabling is how those plants are built, not a fifth fabric. A serving path on the training trunks is how you create an outage.
Open →
Network planes
Ingest and applications ride North-South / front-end. Datasets and checkpoints ride the storage/data fabric. Collectives ride the East-West compute fabric. Management/OOB is operational control, never application North-South. Converged versus dedicated is a compute/storage decision. Collapsing planes without naming it is how a 'model problem' becomes an outage.
Open →
Storage tiers
Object or file for source documents. Parallel FS for training scratch and checkpoints. Local NVMe for NIM weights. The index is its own machine. KV cache is a serving-plane memory hierarchy, not another LUN on those arrays.
Open →
Runtime / Operate
Docker is the artifact. Kubernetes is the control plane. A starting cell is three masters and three workers. Masters may be VMs. Fluent Bit ships logs from every node. Durable names, not a SKU catalog.
Open →
NIM / profile sizing
NVIDIA NIM selects a profile (precision, tensor-parallel size, LoRA) from its public support matrix. We use that document to choose node vs rack, local disk, and concurrency headroom. We do not publish our own model cards.
Open →
Expansion / Day-1
The pipeline has gates too. Ingest, index, KV, and serving each have a measurement that opens the next investment. Additive vs transition lives on Expansion.
Open →
Handover
The factory is not done at first token. Documentation, operating procedures, observe / monitor, and a support model that names who owns ingest, the index, the serving plane and the application.
Open →
NIM profiles: NVIDIA NIM for LLMs support matrix. Cited as a public NVIDIA document, not as a partnership.
NP-APP-REF/Factory templates
The same path, different jobs.
Private AI
Enterprise sources, controlled ingest, RAG, NIM serving, internal applications.
Open →Training cluster
Datasets, staging, coupled fabric, checkpoints, scheduler, experiment platform.
Open →AI-as-a-Service
Onboarding, tenancy, accelerator pools, shared fabric, control plane, customer workloads.
Open →NP / Intake
Tell us what you need to achieve with AI.
One accountable delivery partner from AI requirement to a delivered factory: site, power, cooling, compute, fabric, structured cabling & passive infrastructure, storage, platform, procurement, and handover. You do not need a vendor list or a hall name first. The hall is a supplier.