Drone Telemetry Pipelines: From PX4 and MAVLink to Searchable Flight Data

Every flight produces a detailed record of what the vehicle sensed, decided and did, yet on most drone programs that record lives on an SD card until someone needs it, and then it cannot be found. A good drone telemetry pipeline turns PX4 logs, MAVLink streams and companion computer data into a searchable flight history that supports maintenance, fleet analytics and incident investigation. This guide walks through the formats, the collection paths and a reference architecture you can build on.

What data a drone actually produces

A PX4-based drone with a companion computer and a camera payload generates four categories of data, each with different rates, sizes and uses.

The common mistake is to treat these as separate systems. The fix is a shared flight identifier, created at arming and stamped on every artifact, so one query returns the log, video, bag and operator notes for a flight.

PX4 logs: the ULog format and logging profiles

How ULog is structured

ULog is PX4's self-describing binary log format. A file starts with a header and a definitions section containing message formats, information messages (such as hardware, software version and system name) and the initial parameter values. The data section then contains subscriptions to uORB topics, the logged samples for each topic (with microsecond timestamps), text log messages from the system, parameter changes, and dropout markers when the logger could not keep up. Because formats are embedded, parsers handle mixed firmware versions without the source.

Multi-instance topics, such as several batteries or IMUs, carry an instance ID. Track dropout markers: frequent dropouts usually point to a slow or worn SD card.

Logging modes and profiles

PX4's logger writes to the SD card by default. Two parameter groups control what you get:

• When to log (SDLOG_MODE): for example, from arming until disarming, or from boot, which captures pre-arm problems such as sensor calibration issues and failed arming checks.
• What to log (SDLOG_PROFILE): a bitmask that adds topic sets on top of the default, such as estimator replay, thermal calibration, system identification, high-rate, debug, sensor comparison, vision and avoidance, and raw high-rate IMU data.

For full control, a custom topic list file on the SD card can define exactly which topics are logged and at which intervals. A practical fleet policy is a lean default profile for production flights, plus richer profiles enabled for test flights, tuning and vehicles under investigation.

MAVLink v2 telemetry and message signing

MAVLink is the lightweight messaging protocol between the autopilot, ground stations and companion computers. MAVLink v2 adds extensible messages, larger message IDs and optional message signing. For a telemetry pipeline, a small set of messages carries most of the operational value:

Note the integer scaling: positions in 1e-7 degrees, altitudes in millimeters, currents in centiamperes. Normalize units at ingest, once, and document the conversions.

Message signing

MAVLink v2 signing appends a 13-byte signature block to each packet: a link ID, a 48-bit timestamp, and a 48-bit signature derived from SHA-256 over a shared 32-byte secret key and the packet contents. It authenticates packets and protects against replay, but it does not encrypt them; anyone on the link can still read telemetry. Keys should be provisioned over a secure local connection, and support and configuration differ between autopilot firmware versions, so verify for yours. If confidentiality matters, carry MAVLink inside an encrypted transport such as a VPN or TLS tunnel on IP links.

The PX4 to ROS 2 bridge with uXRCE-DDS

Since PX4 v1.14, the supported bridge to ROS 2 is uXRCE-DDS. A client module on the flight controller connects over serial or UDP to the Micro XRCE-DDS Agent running on the companion computer, and the agent makes selected uORB topics appear as ROS 2 topics, typically under /fmu/out/ for data from the vehicle and /fmu/in/ for commands to it.

# On the companion computer: start the agent on the serial link to the flight controller
MicroXRCEAgent serial --dev /dev/ttyTHS1 -b 921600

# ...or over Ethernet/UDP
MicroXRCEAgent udp4 -p 8888

# Inspect bridged topics (requires px4_msgs built for your firmware version)
ros2 topic list | grep /fmu/out
ros2 topic echo /fmu/out/battery_status --qos-reliability best_effort

# Record vehicle topics alongside perception data in MCAP
ros2 bag record -s mcap -o flight_$(date +%Y%m%d_%H%M%S) \
  /fmu/out/vehicle_status /fmu/out/vehicle_global_position \
  /fmu/out/vehicle_odometry /fmu/out/battery_status /camera/image_raw/compressed

Three practical notes. First, the px4_msgs package must match the firmware's message definitions, so pin them together in your release process. Second, PX4 publishes with best-effort QoS; subscribers must use compatible settings such as the sensor-data profile. Third, the set of bridged topics is defined in a configuration file in the PX4 source, and recent releases introduced message versioning, so some topic names may carry a version suffix; check the file for your firmware rather than hard-coding names. Once the data is in ROS 2, you can reuse the recording, compression and upload design from our rosbag2 and MCAP logging pipeline guide.

Collecting telemetry and parsing ULog files

Live collection with pymavlink or MAVSDK

For live telemetry, MAVSDK offers a high-level, asynchronous API (telemetry streams for position, battery, flight mode and health) and suits mission applications. pymavlink gives raw access to every message and is convenient for a lightweight collector. A tool such as mavlink-router can split the autopilot's stream so the ground station and the collector both receive it.

import json, time
from pymavlink import mavutil

m = mavutil.mavlink_connection("udpin:0.0.0.0:14550")
m.wait_heartbeat()

def set_rate(msg_id, hz):
    m.mav.command_long_send(m.target_system, m.target_component,
        mavutil.mavlink.MAV_CMD_SET_MESSAGE_INTERVAL, 0,
        msg_id, int(1e6 / hz), 0, 0, 0, 0, 0)

set_rate(mavutil.mavlink.MAVLINK_MSG_ID_BATTERY_STATUS, 1)
set_rate(mavutil.mavlink.MAVLINK_MSG_ID_GLOBAL_POSITION_INT, 5)

TYPES = ["HEARTBEAT", "SYS_STATUS", "ATTITUDE", "GLOBAL_POSITION_INT", "BATTERY_STATUS"]
with open("/var/spool/telemetry/live.jsonl", "a") as spool:
    while True:
        msg = m.recv_match(type=TYPES, blocking=True, timeout=5)
        if msg is None:
            spool.write(json.dumps({"event": "link_timeout", "rx_time": time.time()}) + "\n")
            continue
        rec = msg.to_dict()
        rec["rx_time"] = time.time()
        spool.write(json.dumps(rec) + "\n")

Parsing a ULog with pyulog

pyulog, maintained by the PX4 project, reads ULog files in Python and ships command-line tools for quick inspection and CSV export. The script below extracts battery and position data into Parquet and computes per-flight metadata for the index. It needs pandas and pyarrow installed.

from pathlib import Path
import pandas as pd
from pyulog import ULog

def topic_frame(ulog, name, fields, instance=0):
    try:
        d = ulog.get_dataset(name, instance)
    except (IndexError, KeyError):
        return pd.DataFrame(columns=["timestamp_us", *fields])
    df = pd.DataFrame({"timestamp_us": d.data["timestamp"]})
    for f in fields:
        if f in d.data:
            df[f] = d.data[f]
    return df

def ulog_to_parquet(path: Path, out_dir: Path) -> dict:
    ulog = ULog(str(path), ["battery_status", "vehicle_global_position"])
    bat = topic_frame(ulog, "battery_status",
                      ["voltage_v", "current_a", "remaining", "discharged_mah"])
    pos = topic_frame(ulog, "vehicle_global_position", ["lat", "lon", "alt"])

    merged = pd.merge_asof(pos.sort_values("timestamp_us"),
                           bat.sort_values("timestamp_us"),
                           on="timestamp_us", direction="backward")
    merged["t_s"] = (merged["timestamp_us"] - ulog.start_timestamp) / 1e6
    out_dir.mkdir(parents=True, exist_ok=True)
    merged.to_parquet(out_dir / f"{path.stem}.parquet", index=False)

    errors = [m.message for m in ulog.logged_messages
              if m.log_level_str() in ("EMERGENCY", "ALERT", "CRITICAL", "ERROR")]
    return {
        "log_file": path.name,
        "duration_s": round((ulog.last_timestamp - ulog.start_timestamp) / 1e6, 1),
        "sys_name": ulog.msg_info_dict.get("sys_name"),
        "ver_sw": ulog.msg_info_dict.get("ver_sw"),
        "airframe": ulog.initial_parameters.get("SYS_AUTOSTART"),
        "max_alt_rel_m": float((merged["alt"] - merged["alt"].iloc[0]).max()) if len(merged) else None,
        "min_voltage_v": float(bat["voltage_v"].min()) if len(bat) else None,
        "dropouts": len(ulog.dropouts),
        "errors": errors[:20],
    }

Field names vary slightly between PX4 versions, which is why the helper tolerates missing fields. In production, run this as a containerized extractor triggered on upload, so every new log is converted and indexed automatically. MerkleBot runs such jobs as Docker-based compute and data extractors, alongside pre-built extractors for ROS bag files that pull out video and enrich logs.

Ground-to-cloud transfer and a reference architecture

Upload after landing vs. live LTE streaming

Downloading ULogs over a telemetry radio is possible through MAVLink's log transfer protocol but slow; at an illustrative 5 KB/s, a 100 MB log takes more than five hours. Pull logs over the companion computer's high-speed link or Wi-Fi instead.

Reference architecture

1. Flight controller: PX4 writes ULog to SD, sends MAVLink to the radio and companion, and runs the uXRCE-DDS client.
2. Companion computer: runs the Micro XRCE-DDS Agent, ROS 2 nodes and a rosbag2/MCAP recorder; mavlink-router splits telemetry; an agent assigns the flight ID at arming, spools live telemetry and, after disarm, fetches the ULog, hashes all artifacts and queues uploads.
3. Uplink: summary telemetry over LTE in flight; bulk upload over Wi-Fi or Ethernet at the dock, resumable and bandwidth-limited.
4. Ingest and processing: an API receives the manifest and files; extractors convert ULog to Parquet, cut video, run analysis and write metadata to the index.
5. Storage: recent flights in hot object storage; older logs and media tiered to cheaper archive storage. One MerkleBot mobile robotics customer saved up to 10x on storing large video datasets by backing up the archive to a decentralized storage network; see hybrid cloud storage for robotics.
6. Consumers: search, fleet health dashboards, Flight Review-style plots, maintenance systems and incident tools.

Flight indexing, search and automated post-flight analysis

Per-flight metadata

Searchable UAV data management depends on writing a metadata record for every flight at ingest. A practical schema:

{
  "flight_id": "fl_2026-09-30_x500-014_0007",
  "vehicle_id": "x500-014",
  "airframe": 4001,
  "ver_sw": "v1.15.4",
  "started_at": "2026-09-30T09:12:44Z",
  "duration_s": 1184.2,
  "max_alt_rel_m": 61.8,
  "distance_km": 7.4,
  "min_cell_v": 3.46,
  "battery_serial": "BAT-0193",
  "anomalies": ["vibration_high_z", "ekf_hgt_ratio_high"],
  "artifacts": ["ulog", "mcap", "video", "telemetry_jsonl"],
  "sha256_manifest": "9c1e...47ab"
}

With records stored as Parquet, questions such as "which flights on firmware 1.15 had low cell voltage and estimator warnings?" become one query, for example with DuckDB:

import duckdb
duckdb.sql("""
  SELECT flight_id, vehicle_id, started_at, min_cell_v
  FROM 'index/flights/*.parquet'
  WHERE ver_sw LIKE 'v1.15%' AND min_cell_v < 3.5
    AND list_contains(anomalies, 'ekf_hgt_ratio_high')
  ORDER BY started_at DESC
""").show()

Automated post-flight checks

• Vibration: track IMU vibration metrics and accelerometer clipping per flight. Rising vibration on one airframe often precedes a loose mount, damaged propeller or failing motor bearing.
• Battery sag: estimate internal resistance from voltage drop versus current between hover and climb (R ≈ ΔV / ΔI). An increasing trend across cycles, keyed by battery serial, flags packs for retirement before they cause a forced landing.
• EKF innovations: estimator test ratios for velocity, position, height and magnetometer show how well sensors agree with the filter. Ratios approaching or exceeding 1.0 mean measurements are being rejected; sustained elevated values deserve a look.
• Logger health: dropouts and high CPU load point to SD card or configuration problems.

Set thresholds per airframe from a baseline of known-good flights rather than universal constants, and roll results into fleet dashboards: flights per vehicle, anomaly rate by firmware, battery health distribution and hours to next maintenance.

Compliance, multi-domain fleets and a checklist

Compliance and incident investigation

Operational record-keeping and retention requirements vary by jurisdiction and operation type, so define retention with your compliance team. Technically, the pipeline should guarantee that original logs are preserved unmodified and that their integrity can be proven later. Hash every artifact on the companion computer immediately after disarm, sign the manifest with a device key, and anchor daily Merkle roots in an append-only log, as described in our article on Merkle trees for verifiable machine data. For an incident, freeze the flight's artifacts under a legal hold, export them with their manifest, and run analysis only on copies.

Rovers, boats and underwater ROVs

The same autopilot ecosystems run far beyond multicopters. PX4 supports fixed-wing, VTOL, rovers and boats, with experimental support for underwater vehicles, while ArduPilot's Rover and ArduSub firmware power many ground vehicles and ROVs. Because they all speak MAVLink, the live telemetry path is reusable as is. ArduPilot writes DataFlash .bin logs rather than ULog, which pymavlink can parse, so add one extractor and keep the index schema shared, with domain-specific fields such as depth for underwater vehicles.

Checklist

• Flight ID generated at arming and stamped on ULog, bags, media and telemetry.
• Logging from boot; profiles defined for production, test and investigation.
• SD card health and logger dropouts monitored.
• MAVLink rates set explicitly; units normalized at ingest.
• Signing enabled where supported; encrypted transport for sensitive links.
• px4_msgs pinned to firmware; uXRCE-DDS topics and QoS verified.
• Logs hashed and manifests signed at disarm; resumable upload at the dock.
• ULog extractor producing Parquet and per-flight metadata automatically.
• Battery serials and component IDs recorded per flight.
• Retention, legal hold and storage tiering policies documented.

Frequently asked questions

Conclusion: from SD cards to a searchable flight history

Drone telemetry becomes valuable when every flight is captured completely, linked across streams and indexed for search. Log ULogs from boot, collect MAVLink for live awareness, bridge to ROS 2 with uXRCE-DDS where autonomy lives, hash everything at landing, and let automated extractors turn raw files into Parquet, metadata and health signals. The MerkleBot developer docs show how the agent, CLI and API fit into this flow.

Building or scaling a drone or field robot fleet? Book a 30-minute demo to see PX4 and ROS 2 data flowing into searchable, verifiable storage.