FactorySemantics MES · a measured plant

Sixty million ids a day, every one observed

A disposable-cutlery manufacturer makes ten million forks, ten million spoons and ten million knives a day, stacks them, wraps each stack onto a plate and palletizes the wraps. Every piece, stack, plate and wrap carries its own id; a vision station judges every piece and every stack on four attributes; quality measures five dimensions every fifteen minutes; and every pallet leaves with a certificate of analysis. This is what the MES did with that plant, measured: for the people who run the line and its quality, and for the people who run the systems.

At the real rate550groups / s1,981,758 inspection groups in an hour, 99.94% of what the stations published, 0 partial
At 5x the rate2,787groups / severy group recorded in both runs; the agent at 72.9% of one core, PostgreSQL at 25.5%
One agent process's ceiling≈3,490groups / sabout 6.0 times the customer's rate; the 10x run fell behind at 68.8% CPU
A pallet's certificate37.3msCpk on fifteen characteristics and the listing of 1,440 units, from the containment record

One desktop: Intel i5-12600K, 16 GB, NVMe. PostgreSQL 18 as a user service on the same machine, one database per run; the API one uvicorn process. Speed is the load multiplier — 1x is the plant's real rate. Each speed was run at least twice. Every figure below is computed from the runs' own databases, logs and results files by labs/cutlery/analysis/cutlery_run_analysis.ipynb; nothing is typed in.

The plant, and what flows through it

Nine moulding lines — three each of forks, spoons and knives — each ending in a vision station that judges every piece. Thirty-two stackers take a fork, a spoon and a knife from the lines' output and close a stack that its own vision station judges; two stackers feed every wrapper because a stack takes twice as long as a wrap; sixteen wrappers seal a stack onto a plate; four palletizers take 240 wraps to a pallet. The customer's arithmetic is 60,041,666 ids a day: 30,000,000 pieces, 10,000,000 stacks, 10,000,000 plates, 10,000,000 wraps and 41,666 pallets.

StreamPer dayPer secondHow it enters the MES
forks, spoons, knives30,000,000347one OPC UA group per piece: serial, four vision attributes, a pass word
stacks10,000,000116one group per stack: serial, its three members, four attributes
plates10,000,000116the wrap's group names the plate
wraps10,000,000116one group per wrap: serial, the stack and the plate, four attributes
pallets41,6660.5one group per pallet: serial and its 240 wraps
ids60,041,66669550,000,000 inspection groups a day
The ISA-95 hierarchy from the equipment table: one enterprise, one site, four areas (Moulding, Stacking, Wrapping, Palletizing), a work centre per line and a work unit per machine, each with its rated cycle.

How an id enters: one group notify, one event, one record

Nothing is minted from a count. The station is the source of truth: its vision system decides whether the piece is good, stamps the serial and the four readings with one source time, and publishes them as one OPC UA group. The agent subscribes to every inspection tag at full rate, assembles the tags that share a source time into one event, and writes it in bulk — the unit, its inspection row and, for a stack or a wrap, the containment of its members — with no HTTP in the path. A piece that fails never reaches a stack. OPC UA only notifies a value that changed, so a tag that repeats from one event to the next is filled from what the station last sent; a group whose serial never arrived is recorded as partial and counted, never dropped.

The processes that stood in for a plant and the measured traffic between them at the real rate: the stations' OPC UA groups into the agent, the agent's bulk writes into PostgreSQL, and the floor and the screens through the API.
The processes that stood in for a plant and the measured traffic between them at the real rate: the stations' OPC UA groups into the agent, the agent's bulk writes into PostgreSQL, and the floor and the screens through the API.
The agent's own reports of the inspection path at each speed: groups taken per second of wall clock, the largest batch it wrote, the running mean of milliseconds per batch, and the groups waiting for their remaining tags when it reported. Dotted lines are the repeat run.
The agent's own reports of the inspection path at each speed: groups taken per second of wall clock, the largest batch it wrote, the running mean of milliseconds per batch, and the groups waiting for their remaining tags when it reported. Dotted lines are the repeat run.
Inspection groups the stations published (the replay's last periodic tally) against groups the agent recorded, per run. Coverage is read from the record: every station numbers its groups, so a gap in the sequence is a published group that was not recorded, and the hover states the gaps with the partial groups, duplicates and unknown members.

What the vision stations judged, and where the pieces went

Every piece carries four readings and a verdict; every stack and every wrap the same. The verdict is a pass word whose bits name the attribute that failed, so why a piece was scrapped is a query. The containment records say which marker's pieces went into which stacker's stacks, which stacker fed which wrapper, and what reached which pallet — and because a failing piece is never offered to a stacker, the number of stacks containing a failed piece has one right answer.

Left: the share of pieces each moulding line's vision station failed, against the scrap rate the line declares. Right: which of the four attributes the failed pieces failed on, from the pass word's bits.
Left: the share of pieces each moulding line's vision station failed, against the scrap rate the line declares. Right: which of the four attributes the failed pieces failed on, from the pass word's bits.
The four readings the vision station records on every piece, for the first sixty thousand pieces of one line of each kind, with the station's limits. A reading outside a limit is a failed piece that never reached a stack.
The four readings the vision station records on every piece, for the first sixty thousand pieces of one line of each kind, with the station's limits. A reading outside a limit is a failed piece that never reached a stack.
Each marker's pieces into each stacker's stacks, each stacker into its wrapper, each wrapper onto its palletizer, and what was still in the buffers when the hour ended, in pieces, from the containment records.
How long a judged fork waited to be stacked, a stack to be wrapped, and a wrap to be palletized, from the timestamps the stations put on the events. The pallet wait is the palletizer collecting 240 wraps.
How long a judged fork waited to be stacked, a stack to be wrapped, and a wrap to be palletized, from the timestamps the stations put on the events. The pallet wait is the palletizer collecting 240 wraps.

The dimensional checks, and the certificate a pallet leaves with

Every fifteen minutes an inspector measures five characteristics of each utensil through the same API a browser uses. A pallet's certificate of analysis states, for each characteristic, the checks inside the pallet's production window, the Cpk over them (sigma from the mean moving range) and whether the process was stable — and says no capability outright when the window holds fewer than the twelve points the method needs. Under that it lists every wrap, the stack and the plate in each, and every piece in each stack, rendered from the containment record and issued as an immutable document: 1,440 units under one pallet serial, gathered in 37.3 ms.

The five characteristics measured on each utensil every fifteen minutes of line time through the hour, against their specification limits, from the quality checks the floor recorded through the API.
The five characteristics measured on each utensil every fifteen minutes of line time through the hour, against their specification limits, from the quality checks the floor recorded through the API.
One pallet from the real-time hour as the certificate lists it: the pallet at the centre, its wraps, in each wrap a stack and a plate, in each stack a fork, a spoon and a knife. Colour is what the unit is.

What the line did

The bookings say what each station made, the state intervals say what it was doing, and the scorecard says whether the MES saw the breakdowns the script put in — with a fault it recorded after its window withheld rather than scored missed.

Good production booked per station in five-minute buckets across the real-time hour, one line per station. The dips are the scripted breakdowns, micro-stops and the plant-wide changeover.
Good production booked per station in five-minute buckets across the real-time hour, one line per station. The dips are the scripted breakdowns, micro-stops and the plant-wide changeover.
Left: the share of the hour each kind of station spent running, idle, in setup or down, from the state intervals the agent recorded. Right: per run, the faults the script put in, the ones the MES recorded inside their window, and the ones it recorded after the window and so withheld from the score. A run whose own pipeline lagged its sample has its verdict withheld outright, which is what the 5x, 8x and 10x runs show.
Left: the share of the hour each kind of station spent running, idle, in setup or down, from the state intervals the agent recorded. Right: per run, the faults the script put in, the ones the MES recorded inside their window, and the ones it recorded after the window and so withheld from the score. A run whose own pipeline lagged its sample has its verdict withheld outright, which is what the 5x, 8x and 10x runs show.

For IT: the systems, the loads, and where they bend

Five processes and a database server stood in for a plant: an OPC UA server replaying the line's per-second truth and publishing the inspection groups; the OPC agent taking the groups, writing units and inspections in bulk, booking from counter deltas and recording states; PostgreSQL holding one database for the run; the API serving the floor, the screens and the certificates; and the operations loop being the people. Storage is a module: the same models and migrations run on one SQLite file for a small plant, and on PostgreSQL when a plant like this one switches it on.

RunGroups / sCoveragePartialms / batchAgent CPU %PostgreSQL CPU %Replay CPU %API CPU %MB after the hourVerdict
1x552100.00%0128.623.25.012.81.31,254kept up
1x551100.03%0128.222.95.512.81.21,248kept up
1x55099.94%0129.622.55.512.51.21,237kept up
3x1,66899.68%0128.956.415.737.62.61,177kept up
3x1,669100.17%0129.756.515.737.62.61,165kept up
10x2,14948.24%0728.571.822.892.95.1593fell behind
10x2,18748.11%0638.668.822.193.05.2431fell behind
5x2,80899.84%0277.172.924.862.43.11,112kept up
5x2,78799.42%0314.772.925.562.33.91,106kept up
8x1,34930.27%0650.564.319.193.95.3333fell behind
8x71015.98%0650.246.615.793.94.2190fell behind
CPU and resident memory of each process through each run, the PostgreSQL backends serving the run summed as one (their resident figures each count the shared buffers, so the sum overstates what the server holds). The agent and the database grow with the inspection rate; the replay grows with the tag rate and is the first thing at its limit past 5x; the API and the floor are flat.
CPU and resident memory of each process through each run, the PostgreSQL backends serving the run summed as one (their resident figures each count the shared buffers, so the sum overstates what the server holds). The agent and the database grow with the inspection rate; the replay grows with the tag rate and is the first thing at its limit past 5x; the API and the floor are flat.
Agent CPU against inspection groups per second for every run, with a line fitted through the origin over the runs that kept up. Where the line meets 100% is one agent process's ceiling.
Agent CPU against inspection groups per second for every run, with a line fitted through the origin over the runs that kept up. Where the line meets 100% is one agent process's ceiling.
What the large tables and their indexes weigh after each run's hour, from PostgreSQL's own accounting, and the projection of each run's rate to a day. Units, inspections and their indexes are the day; everything else is master data and bookings.
What the large tables and their indexes weigh after each run's hour, from PostgreSQL's own accounting, and the projection of each run's rate to a day. Units, inspections and their indexes are the day; everything else is master data and bookings.
Milliseconds for each recall question against each run's database: what is in a pallet, its trace back, where a resin lot and a plate lot went, the certificate's data, and quarantining a pallet with everything under it.
Milliseconds for each recall question against each run's database: what is in a pallet, its trace back, where a resin lot and a plate lot went, the certificate's data, and quarantining a pallet with everything under it.

Findings

Operations and quality

  • Every id is an observed event: the stations published 1,982,899 inspection groups in the real-time hour and the agent recorded 1,981,758 of them — 0 partial, 0 duplicate, 0 naming a member it had never seen.
  • A piece that failed vision is in the record with the attribute it failed on, and no stack contains one.
  • The dimensional checks are the process's own record — 60 in the hour, five characteristics per utensil every fifteen minutes — and a pallet's certificate computes its Cpk over the checks inside that pallet's window, with the sample size stated.
  • The certificate lists everything on the pallet, each wrap with its stack and plate, each stack with its fork, spoon and knife, and is issued immutable.

IT

  • At the real rate the plant costs one desktop 22.5% of a core on the agent, 5.5% on PostgreSQL, 12.5% on the replay and 1.2% on the API. Memory is flat.
  • The inspection path scales with rate until one agent process saturates at about 3,490 groups a second — roughly 6.0 times the customer's rate. The next step is a second agent process over a partition of the stations, not a bigger box.
  • Storage grows 1236.6 MB an hour at the real rate, 29.68 GB a day; units, inspections and their indexes are almost all of it. That is the large plant's tier — PostgreSQL, partitioned by day, closed days archived to plain files — and a small plant never needs it.
  • The recall questions do not grow with the database except the one that asks where a lot went that reached a third of everything made.

What the simulator cannot say

  • Its stations' vision readings are drawn around a nominal; a real station's distribution has the shape the process gives it. The method — limits, a pass word, Cpk over a window — does not depend on the shape.
  • The OPC UA server is a Python replay. It publishes what the plant would at 1x; at higher speeds the emitted count against the expected count says how much of the load it could produce, and only what was published can be scored.

Provenance

The notebook that produced every figure, the runner that produced the evidence, and the plant description are in the repository under labs/cutlery/. Re-running the notebook re-derives the page.

EvidenceWhereStamp (UTC)
scored run at 1xlabs/cutlery/out/tmp/fsmes-run-cutlery-jo5wii0c, database fsmes_run_20260906t204224z_1x20260906T204224Z
scored run at 1xlabs/cutlery/out/tmp/fsmes-run-cutlery-wl3ql8_g, database fsmes_run_20260906t214300z_1x20260906T214300Z
scored run at 1xlabs/cutlery/out/tmp/fsmes-run-cutlery-jf63pri8, database fsmes_run_20260906t233823z_1x20260906T233823Z
scored run at 3xlabs/cutlery/out/tmp/fsmes-run-cutlery-bwd8r0vm, database fsmes_run_20260907t003928z_3x20260907T003928Z
scored run at 3xlabs/cutlery/out/tmp/fsmes-run-cutlery-8wlfrc8w, database fsmes_run_20260907t010005z_3x20260907T010005Z
scored run at 10xlabs/cutlery/out/tmp/fsmes-run-cutlery-p0ir8zw1, database fsmes_run_20260907t014656z_10x20260907T014656Z
scored run at 10xlabs/cutlery/out/tmp/fsmes-run-cutlery-v07_u9rz, database fsmes_run_20260907t015329z_10x20260907T015329Z
scored run at 5xlabs/cutlery/out/tmp/fsmes-run-cutlery-a0jhqdei, database fsmes_run_20260907t020033z_5x20260907T020033Z
scored run at 5xlabs/cutlery/out/tmp/fsmes-run-cutlery-dlfl4b68, database fsmes_run_20260907t021309z_5x20260907T021309Z
scored run at 8xlabs/cutlery/out/tmp/fsmes-run-cutlery-10dkevce, database fsmes_run_20260907t022544z_8x20260907T022544Z
scored run at 8xlabs/cutlery/out/tmp/fsmes-run-cutlery-xmcsicm4, database fsmes_run_20260907t023347z_8x20260907T023347Z
simulator per-second truthlabs/cutlery/out/*.csvgenerated by run_cutlery.py