Skip to content
Follow one frame

Smart camera

A camera with its own small computer, on the wall above a factory's loading dock. It watches the dock doors and the paved area in front of them, and runs a detection model on each frame. When a person steps into the marked zone in front of a door, it keeps a clip, and it drops the rest. When someone asks, it shows the dock live.

Step 1 of 8: Capture

Step 01 of 08, on the machine · Sense

Capture

A person walks into the marked zone in front of the empty dock door, beside a trailer at the next door. Light from the apron comes through the window and the lens onto the image sensor, which reads the whole picture at once, 40 times a second.

What it costs

92 MB of pixels a second: 1,920 × 1,200 pixels, 40 times a second, if each pixel is kept in one byte. That is about 740 Mbit/s, most of a gigabit link, before anything is compressed.

Derived from the Basler ace acA1920-40gc catalogue row

Why it is hard, and what is at work

The camera uses one exposure setting for the whole frame. With the sun behind the trailer, the person beside it is in shadow. At night the infrared lights only reach so far, and rain and fog reflect their light back into the lens.

Parts working at this step
  • Lens, image sensor, and infrared lights

    A fixed lens focuses the dock onto a 2.3 megapixel sensor that reads every pixel at the same instant (a global shutter). A moving forklift keeps its shape, where a sensor that reads line by line would draw it leaning. Infrared lights around the lens light the area at night, when the camera switches to black and white.

  • Housing, window, and sunshield

    A sealed aluminium housing holds the camera, its computer, and its storage behind a flat glass window. A sunshield over the top keeps direct sun off the housing and the window, and rain off the glass.

  • Wall bracket

    A steel arm holds the camera out from the wall above the dock doors, high enough to see over a trailer and the whole apron, tipped down toward it.

Step 02 of 08, on the machine · Think

Detect

The processor scales the frame down and passes it to the detection model on the accelerator. The model returns a box round each thing it recognises, with a label and a score: here a trailer and a person.

What it costs

At most 25 ms for each frame if the model runs on all 40 frames a second. Detectors of this kind usually run at about 10 to 15 frames a second, so a person crossing quickly spends fewer frames in view.

Estimate

Why it is hard, and what is at work

A detector only finds things that look like its training pictures. A person half hidden by a trailer, a person pushing a cage that hides their legs, or a reflection on a wet surface all look different, and the model misses them, or finds a person where there is none.

Parts working at this step
  • Processor and AI accelerator

    A small computer runs the camera. It takes each frame from the sensor, scales it to the size the model expects, and runs a detection model on an accelerator card beside it. The model puts a box round each person, vehicle, or trailer it finds, with a score for how sure it is.

Step 03 of 08, on the machine · Think

Decide

The camera's own software decides what to keep. A person inside the marked zone in front of a dock door is what the site asked to see, so the camera keeps a clip: a few seconds either side of this frame, taken from the recording of the last minute on its card. A frame with nobody in the zone is dropped.

What it costs

About 43 GB a day to send every frame as a 4 Mbit/s video. About 0.5 GB a day if the camera keeps 100 clips of 10 seconds at the same rate.

Estimate

Why it is hard, and what is at work

The camera can't send every frame, and a clip it doesn't keep is lost for good. If the rule for keeping clips is too loose, forklift clips fill the card and the link. If it is too strict, the camera drops the near miss the site wanted to see. The site also decides what may leave it at all (for example, workers' faces and truck number plates may have to be blurred or cut first).

Parts working at this step
  • Processor and AI accelerator

    A small computer runs the camera. It takes each frame from the sensor, scales it to the size the model expects, and runs a detection model on an accelerator card beside it. The model puts a box round each person, vehicle, or trailer it finds, with a score for how sure it is.

  • Clip storage

    A memory card at the back of the board holds a recording of the last minute or so, and the clips the camera keeps, until they are sent or overwritten.

Step 04 of 08, on the machine · Act

Stream

In the site office, someone asks to see the dock now. The camera's device agent has already dialled out to the server, so the camera needs no inbound port and the office needs no VPN (a private tunnel into the site network). The server sends the camera a command to open a live session. The camera publishes its video over WHIP, a standard way to send video over WebRTC, and the browser watches it over WHEP, its counterpart for watching. The video passes through the server, which sends it on to each person watching. The server records none of it: recording is the device agent's capture, at step 05.

What it costs

About 2 Mbit/s up the camera's link for as long as someone watches, for 640 × 480 pictures at 30 frames a second, the size, rate, and bitrate the solution's demo script sets when it encodes on the processor.

Estimate

Device to Cloud Flywheel solution: Cameras and vision devices

Why it is hard, and what is at work

Live video is the heaviest thing the camera sends, so it runs only while someone watches, and it shares the cable and the uplink with the clips. In the device agent, recording and live view each open the video source on their own. A network camera's stream can be opened twice; a camera on a USB port can be opened by only one program at a time, so recording and live view cannot both run on it.

Parts working at this step
  • PoE and network link

    One Ethernet cable carries power to the camera (Power over Ethernet, PoE) and its data to the site network. A board at the back of the housing takes the power off the cable for the computer and the lights, and the cable runs along the arm into a junction box on the wall.

  • Processor and AI accelerator

    A small computer runs the camera. It takes each frame from the sensor, scales it to the size the model expects, and runs a detection model on an accelerator card beside it. The model puts a box round each person, vehicle, or trailer it finds, with a score for how sure it is.

Where Device to Cloud Flywheel fits: Cameras and vision devices

Cameras in the field, fixed, on a machine, or a browser on a laptop. You want to see them live and pull frames back for training.

Demo

examples/browser-camera/demo.sh

Needs a running stack, a USB camera on the machine that runs it, Go, GStreamer, jq, curl, v4l2-ctl, and ffmpeg or GStreamer's H.264 decoder.

The demo script drives a real USB camera on the machine that runs it. It onboards the camera as a device, advertises it as a media source, opens a live session, publishes the camera's video over WHIP, pulls it back over WHEP the way a viewer would, and saves one decoded frame as a picture. With --synthetic it publishes a GStreamer test pattern instead and says so on every line. It shows live view only: it records no segments and deploys no model. The solution catalogue lists only GStreamer for a synthetic camera as what the demo needs; this page follows the script. The solution page describes a browser page driven headlessly with a fake camera, and a smart-camera driver whose only transport is a mock; neither is what the script runs. The solution page also calls the server the recording point; in the code the server only relays live video, and recording happens on the device. The solution page also says a rollout pauses when a new detector does worse, a check the rollout engine does not make.

The repository is private while the first release lands, so this demo cannot be run from this page. Write to hello@devicetocloud.ai and we will tell you when it opens.

From the solution page, punctuation adjusted. What the demo needs is read from its script.

Step 05 of 08, across the fleet

Log

The kept clip waits on the camera's card. Across 1,000 cameras, clips pile up faster than a team can look at them. With capture turned on, Device to Cloud Flywheel's device agent records the camera's video in segments of about six seconds, keeps each on disk with the SHA-256 checksum of its bytes, and goes on recording while the link is down. It uploads every segment it records and has no keep-or-drop rule of its own. The choice made at step 03 lives in the video pipeline you give the agent, or in your own software. An operation you add can write kept clips to a file when the team, or a script of its own, calls it: the agent uploads the file and returns a reference in that command's result, outside the segment list.

What it costs

About 0.5 TB a day across 1,000 cameras if each keeps 100 clips of 10 seconds at about 4 Mbit/s. With capture uploading every segment of a 4 Mbit/s stream, it is about 43 TB a day.

Estimate

Device to Cloud Flywheel solution: Cameras and vision devices

The fleet view shows 1,000 cameras to show the problem at that size; Device to Cloud Flywheel has been run on fleets of single digits to low tens of devices.

Why it is hard, and what is at work

A clip is only useful with the model version, the settings, and the time it was recorded under. A camera whose clock drifts files a near miss under the wrong minute. The code has a capture policy with a setting to upload only on an event, which the agent does not read.

Parts working at this step
  • Clip storage

    A memory card at the back of the board holds a recording of the last minute or so, and the clips the camera keeps, until they are sent or overwritten.

  • Processor and AI accelerator

    A small computer runs the camera. It takes each frame from the sensor, scales it to the size the model expects, and runs a detection model on an accelerator card beside it. The model puts a box round each person, vehicle, or trailer it finds, with a score for how sure it is.

Where Device to Cloud Flywheel fits: Cameras and vision devices

Cameras in the field, fixed, on a machine, or a browser on a laptop. You want to see them live and pull frames back for training.

Demo

examples/browser-camera/demo.sh

Needs a running stack, a USB camera on the machine that runs it, Go, GStreamer, jq, curl, v4l2-ctl, and ffmpeg or GStreamer's H.264 decoder.

The demo script drives a real USB camera on the machine that runs it. It onboards the camera as a device, advertises it as a media source, opens a live session, publishes the camera's video over WHIP, pulls it back over WHEP the way a viewer would, and saves one decoded frame as a picture. With --synthetic it publishes a GStreamer test pattern instead and says so on every line. It shows live view only: it records no segments and deploys no model. The solution catalogue lists only GStreamer for a synthetic camera as what the demo needs; this page follows the script. The solution page describes a browser page driven headlessly with a fake camera, and a smart-camera driver whose only transport is a mock; neither is what the script runs. The solution page also calls the server the recording point; in the code the server only relays live video, and recording happens on the device. The solution page also says a rollout pauses when a new detector does worse, a check the rollout engine does not make.

The repository is private while the first release lands, so this demo cannot be run from this page. Write to hello@devicetocloud.ai and we will tell you when it opens.

From the solution page, punctuation adjusted. What the demo needs is read from its script.

Step 06 of 08, across the fleet

Return

Each segment goes up under the camera's own device credential: its bytes to the blob store first, then its times, size, and checksum to the server. A segment the server already holds from that camera, by checksum, is recorded once. The device agent dials out, so the camera's network needs no inbound port. The team lists a camera's segments on its Operations tab in the console, or through the API, and downloads the ones it wants with dcfctl device cp --store.

What it costs

About 7 minutes to send a day's 0.5 GB of clips if the camera gets about 10 Mbit/s of the site's uplink.

Estimate

Device to Cloud Flywheel solution: Cameras and vision devices

The fleet view shows 1,000 cameras to show the problem at that size; Device to Cloud Flywheel has been run on fleets of single digits to low tens of devices.

Why it is hard, and what is at work

The camera shares the site's uplink with the office, so clips go up slowly, at night, or both. What leaves the site is the site's decision. Device to Cloud Flywheel moves the bytes it is given and blurs nothing. The reference camera driver takes named privacy zones as desired state, and its only transport is a mock, so blurring on a real camera is your own software's job.

Parts working at this step
  • PoE and network link

    One Ethernet cable carries power to the camera (Power over Ethernet, PoE) and its data to the site network. A board at the back of the housing takes the power off the cable for the computer and the lights, and the cable runs along the arm into a junction box on the wall.

Where Device to Cloud Flywheel fits: Cameras and vision devices

Cameras in the field, fixed, on a machine, or a browser on a laptop. You want to see them live and pull frames back for training.

Demo

examples/browser-camera/demo.sh

Needs a running stack, a USB camera on the machine that runs it, Go, GStreamer, jq, curl, v4l2-ctl, and ffmpeg or GStreamer's H.264 decoder.

The demo script drives a real USB camera on the machine that runs it. It onboards the camera as a device, advertises it as a media source, opens a live session, publishes the camera's video over WHIP, pulls it back over WHEP the way a viewer would, and saves one decoded frame as a picture. With --synthetic it publishes a GStreamer test pattern instead and says so on every line. It shows live view only: it records no segments and deploys no model. The solution catalogue lists only GStreamer for a synthetic camera as what the demo needs; this page follows the script. The solution page describes a browser page driven headlessly with a fake camera, and a smart-camera driver whose only transport is a mock; neither is what the script runs. The solution page also calls the server the recording point; in the code the server only relays live video, and recording happens on the device. The solution page also says a rollout pauses when a new detector does worse, a check the rollout engine does not make.

The repository is private while the first release lands, so this demo cannot be run from this page. Write to hello@devicetocloud.ai and we will tell you when it opens.

From the solution page, punctuation adjusted. What the demo needs is read from its script.

Step 07 of 08, across the fleet

Learn

The team sees that the detector at a newly added site misses people in dark overalls beside dark trailers. From that site's returned clips they label a few thousand frames and train a new detector with them. In Device to Cloud Flywheel, a segment comes back by its blob id, and a trained model goes up as a file in the blob store with its checksum. Device to Cloud Flywheel chooses no clips, labels nothing, and schedules no training.

What it costs

About 12 hours of labelling for 2,000 frames if each frame needs about 3 boxes and each box takes about 7 seconds.

Estimate

Device to Cloud Flywheel solution: Training a policy and shipping it to the fleet

A second product, Device to Cloud Engineer, is an engineer agent that writes the device code a machine like this runs. It is in development, and there is nothing to try yet.

The fleet view shows 1,000 cameras to show the problem at that size; Device to Cloud Flywheel has been run on fleets of single digits to low tens of devices.

Why it is hard, and what is at work

A detector that works better at the new site can work worse at an old one. The team has to test it on clips from every kind of site, by day and by night. They also need to know which model and settings recorded each clip. Otherwise they compare pictures that differ in more than the one thing they changed.

Where Device to Cloud Flywheel fits: Training a policy and shipping it to the fleet

Field data coming back from machines, and a model you want to improve and redeploy without a bespoke pipeline each time. It runs across the other solutions rather than describing one kind of device.

Demo

embodiments/train/demo.sh

Needs a running stack, Go, uv, jq, and curl.

The demo collects data from a simulated excavator (a MuJoCo simulation), trains a policy on it, and ships it to a two-machine fleet. No camera is in it. The solution catalogue says the training and deploy halves do not depend on the hardware: a detector travels the same way, as a file in the blob store named in desired state by its checksum.

The repository is private while the first release lands, so this demo cannot be run from this page. Write to hello@devicetocloud.ai and we will tell you when it opens.

From the solution page, wording adjusted. What the demo needs is read from its script.

Step 08 of 08, across the fleet

Ship

The new detector goes out as desired state: a model file in the blob store, named by its id and SHA-256 checksum, with the pipeline and its zones. The reference smart-camera driver pulls the file, checks the checksum, and refuses a file that does not match. That driver runs against a mock camera, and the repository has no transport for a real one, so on a real camera your own code loads the model, the way the training demo's machine loads its policy. A rollout sends the change to one group first, here 4 of the 5 cameras at this one site, and releases the next group only when someone advances it and every camera in the groups before has the change in its desired state. It can be paused, resumed, and rolled back.

What it costs

About 10 GB to move if the detector is about 10 MB and goes to 1,000 cameras, over site uplinks shared with everything else.

Estimate

Device to Cloud Flywheel solution: Training a policy and shipping it to the fleet

The fleet view shows 1,000 cameras to show the problem at that size; Device to Cloud Flywheel has been run on fleets of single digits to low tens of devices.

Why it is hard, and what is at work

A camera stops detecting while it loads a new model. Update 1,000 cameras at once and they all stop detecting at the same time. A staged rollout limits that to one group at a time. A camera that keeps its old model running until the new one has loaded never stops detecting. The solution page says a rollout pauses when a new detector does worse, but the rollout engine has no such check. Whether the new detector is better at each site is the team's call, from its own measures. The rollout makes every group the same size, so groups of 4 cameras across 1,000 take 250 advances.

Parts working at this step
  • Processor and AI accelerator

    A small computer runs the camera. It takes each frame from the sensor, scales it to the size the model expects, and runs a detection model on an accelerator card beside it. The model puts a box round each person, vehicle, or trailer it finds, with a score for how sure it is.

  • PoE and network link

    One Ethernet cable carries power to the camera (Power over Ethernet, PoE) and its data to the site network. A board at the back of the housing takes the power off the cable for the computer and the lights, and the cable runs along the arm into a junction box on the wall.

Where Device to Cloud Flywheel fits: Training a policy and shipping it to the fleet

Field data coming back from machines, and a model you want to improve and redeploy without a bespoke pipeline each time. It runs across the other solutions rather than describing one kind of device.

Demo

embodiments/train/demo.sh

Needs a running stack, Go, uv, jq, and curl.

The demo collects data from a simulated excavator (a MuJoCo simulation), trains a policy on it, and ships it to a two-machine fleet. No camera is in it. The solution catalogue says the training and deploy halves do not depend on the hardware: a detector travels the same way, as a file in the blob store named in desired state by its checksum.

The repository is private while the first release lands, so this demo cannot be run from this page. Write to hello@devicetocloud.ai and we will tell you when it opens.

From the solution page, wording adjusted. What the demo needs is read from its script.

And back

Back to 01 Capture, with the new detector

The next person to cross the dock is captured and goes through the same steps, this time judged by the new detector. The clips it keeps from the new site show whether it now finds people in dark overalls beside a dark trailer.

On the machine
Across the fleet
And back
Every part in the cycle

Each subsystem has its own way to fail.

The catalogue examples are representative parts taken from vendor datasheets and product pages, each linked to its source. They show the kind of component each subsystem uses and the numbers that matter for it. They are not a bill of materials, and nobody has checked that they work together.

PartWhat it doesWhy it is hardCatalogue examples
Housing, window, and sunshieldWorking at 01 captureWhat it does. A sealed aluminium housing holds the camera, its computer, and its storage behind a flat glass window. A sunshield over the top keeps direct sun off the housing and the window, and rain off the glass.Why it is hard. A box on a sunny wall gets far hotter than the air around it, and the processor inside adds its own heat. There is no fan, because dust would clog it, so the heat has to leave through the metal. The window fogs on cold mornings and collects dust and spider webs. A drop of water on the glass blurs the picture, and the model can't fix that.Catalogue examples

The catalogue has no enclosure row.

Lens, image sensor, and infrared lightsWorking at 01 captureWhat it does. A fixed lens focuses the dock onto a 2.3 megapixel sensor that reads every pixel at the same instant (a global shutter). A moving forklift keeps its shape, where a sensor that reads line by line would draw it leaning. Infrared lights around the lens light the area at night, when the camera switches to black and white.Why it is hard. The light at a dock door changes all day: sun on the apron, a dark trailer, headlights at night, and a low evening sun straight into the lens. A sensor like this one sets one exposure for the whole picture, so a person in shadow beside a sunlit trailer can come out too dark to detect. Under infrared, clothes and paint look different from the daytime colour pictures a model learned from.Catalogue examples
resolution 1,920 × 1,200 · frame rate 40 Hz · sensor size 1/1.2 in

The Basler ace is a machine-vision camera that takes its power over its network cable. It stands in here for the sensor: a smart camera puts a sensor of about this size and resolution behind its own lens, often one that combines two exposures of each frame to cope with bright and dark in one picture.

Processor and AI acceleratorWorking at 02 detect, 03 decide, 04 stream, 05 log, 08 shipWhat it does. A small computer runs the camera. It takes each frame from the sensor, scales it to the size the model expects, and runs a detection model on an accelerator card beside it. The model puts a box round each person, vehicle, or trailer it finds, with a score for how sure it is.Why it is hard. The model has to keep up with the camera using only the power the cable carries, without making more heat than the housing can shed. A model trained at one site learns that site's paint, trailers and light. At a new site it misses people in overalls of another colour, or sees a person in a stack of tyres.Catalogue examples
AI throughput 13 trillion operations/s · length 80 mm · width 22 mm
core 4 · core clock 2,400 MHz · memory 8 GB

A Raspberry Pi 5 with a Hailo-8L accelerator on an M.2 card is one way to build it. Commercial smart cameras put the processor and the accelerator in one chip.

Clip storageWorking at 03 decide, 05 logWhat it does. A memory card at the back of the board holds a recording of the last minute or so, and the clips the camera keeps, until they are sent or overwritten.Why it is hard. Memory cards wear out when they are written all day, and a card that fails loses the clip that mattered. When the link is down for a day and the card fills, the camera has to choose what to overwrite.Catalogue examples

The catalogue has no storage row.

PoE and network linkWorking at 04 stream, 06 return, 08 shipWhat it does. One Ethernet cable carries power to the camera (Power over Ethernet, PoE) and its data to the site network. A board at the back of the housing takes the power off the cable for the computer and the lights, and the cable runs along the arm into a junction box on the wall.Why it is hard. The cable sets the power budget: a PoE+ port delivers about 25 W at the camera, and the computer, the accelerator, the infrared lights, and a heater in winter all share it. The site network belongs to the site, whose IT team rarely opens an inbound port or gives a camera a public address. A site with no wired link sends over a cellular modem, which is paid for by the gigabyte.Catalogue examples
downlink rate max 300 Mbit/s · uplink rate max 50 Mbit/s · transmit power 23 dBm

The catalogue has no PoE switch or injector row. The LTE (4G cellular) modem listed is what a camera uses where no cable reaches; its 50 Mbit/s uplink is a best case, shared with everyone else on the same cell.

Wall bracketWorking at 01 captureWhat it does. A steel arm holds the camera out from the wall above the dock doors, high enough to see over a trailer and the whole apron, tipped down toward it.Why it is hard. Wind and passing trucks shake a camera on an arm. A shaking picture blurs, and moves every box the model draws. If a ladder knocks the camera, every zone drawn on the picture ends up in the wrong place. At 5 m up, fixing it takes a lift and a closed dock door.Catalogue examples

The catalogue has no bracket or fastener row.

The drawing is illustrative, not a validated design. Costs marked estimate are estimates; the others are worked out from the catalogue part named under them.

Where to start

The fleet half of this machine, as Device to Cloud Flywheel solutions.

  • Cameras and vision devices

    Cameras in the field, fixed, on a machine, or a browser on a laptop. You want to see them live and pull frames back for training.

    docs/solutions/cameras.md (opens when the code is published)

    In the cycle at 04 stream, 05 log, 06 return. Quoted from the solution page, punctuation adjusted.

  • Training a policy and shipping it to the fleet

    Field data coming back from machines, and a model you want to improve and redeploy without a bespoke pipeline each time. It runs across the other solutions rather than describing one kind of device.

    docs/solutions/train-and-deploy.md (opens when the code is published)

    In the cycle at 07 learn, 08 ship. Quoted from the solution page, wording adjusted.

Neither of the two solution pages has a section that separates measured results from claims, so nothing on this page is quoted as measured.

These pages live in the Device to Cloud Flywheel repository, which is private while the first release lands. Write to hello@devicetocloud.ai for access, and we will tell you when it opens.

Where we help

What your team builds, and what Device to Cloud Flywheel does for you

You keep building
  • The detection model
  • Which clips to keep
  • Loading the model on a real camera
You don't build
  • Records video in short segments and uploads each under the camera's own credential
  • Sends a new detector by checksum, one group at a time, with pause and rollback
  • Lets the team list and download any camera's segments