> For the complete documentation index, see [llms.txt](https://robopipe.gitbook.io/doc/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://robopipe.gitbook.io/doc/api/websocket-api-reference/nn-inference.md).

# NN Inference

## Running inference

The neural network starts at the point it is uploaded to the camera. The output from the model can be read by subscribing to a websocket endpoint `ws://host:port/cameras/{mxid}/streams/{sensor_name}/nn`. The outputted data will be a json in the format:

```json
{
    "detections": ...
}
```

where detections will contain whetever you is your model's output.

## Output format

The data format received from the websocket endpoint depends on the type of neural network you have running on the camera.

### Generic neural network

In case you used a generic neural network, the output data will be a one dimensional array containing raw data from the last layer of the neural network.

```json
{
    "detections": [
        0.3,
        0.02,
        0.05,
        ...
    ]
}
```

### YOLO & MobilenetSSD

In case you have chosen either a YOLO or MobilenetSSD model, the output format will be an array containing detection objects. Each detection object contains:

* label - number indicating the label it detected
* confidence - float in the range \[0, 1] indicating the confidence level of the detection
* coords - four element tuple, containing the coordinates of the detection in the image \[x\_min, y\_min, x\_max, y\_max], these values are [*normalized*](#user-content-fn-1)[^1], this means that if you want to get the real bounding box, you need to multiply both **X** roodinates by image width and both **Y** coordinates by image height

```json
{
    "detections": [
        {"label": 0, "confidence": 0.765, "coords": [0.5, 0.1, 0.6, 0.23]},
        ...
    ]
}
```

#### Depth

In case a depth stream is active and running at the same time a neural network is deployed, each detection object will also contain information regarding its depth position (in millimeters) in addition to the information above.

```json
{
  "detections": [
    {
      "label": 0,
      "confidence": 0.765,
      "coords": [0.5, 0.1, 0.6, 0.23],
      "spatial_coords": {
        "x": 250,
        "y": 100,
        "z": 2500
      }
    }
  ]
}

```

[^1]: The value is in the range \[0, 1].


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://robopipe.gitbook.io/doc/api/websocket-api-reference/nn-inference.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.