Picogram Data Packet exchange
By: picotronix.dev
Using multiple Pico microcontrollers in a distributed system
Pico 2 Modules are quite powerful, but not really suited to running lots of software components at the same time. Usually a Pico will have hardware interfaces and the software will be busy running those modules. Micropython is good at running in a single thread and while the RP2350 has a second core which can be used for synchronising the various low level hardware blocks, multitasking is not really a good idea.
If we divide up our project into two or more discrete blocks (separate Pico 2 modules) we make the software easier to write, as each module has a dedicated purpose. However, we now have the problem of co-ordinating the processes running on each Pico. We need a comms link between the modules, and we need a datagram protocol.
Picogram: A Datagram Architecture for Distributed Processing
A Datagram is a structured block of information similar to a json file. The datagram is usually formed as a "dictionary" object in the software system, and serialised for transmission. When received, the serialized data is reformed into another dictionary which is functionally identical to the sender copy. For picograms, we want to use a very compact format, so instead of json, we use msgpack to construct a binary serialized key/value structure. Micropython natively supports dictionaries, and (u)msgpack is a standard module which works with Pico.
Using a high level language like python gives us a big advantage. Any software that supports msgpack will be able to read/write a picogram. The dictionary structure allows every picogram to be unique. The definition of fields is quite arbitrary, and as long as we have an agreed upon ID field, the software can use fields matching that ID. This means that we don’t have any strict structure for a picogram data object. The serializers can put key/value pairs anywhere in the binary blob that gets transmitted. As long as the binary data object conforms to msgpack spec, it will be unpacked into a dictionary which is functionally identical to the original sender copy.
Note that using python with msgpack makes programming easier than using a low level language like C. The C code would need a way to implement a "dictionary" with arbitrary data types – possible, but not as easy as using python.
Modular Design based on Datagrams
The picotronix software system is designed to be modular with a data driven architecture, comprising separate parts that only interact via structured “datagrams” (picograms), consisting of a msgpack serialized key/value table. In languages like (u)python the datagram is reconstructed as a “dictionary”. The dictionary has specifier fields and also relevant parameters that define exactly what the required response will be. The response will also be a Picogram.
In a data acquisition device this architecture makes sense. The operation of any such instrument ultimately requires data capture and retrieval. How this is achieved is not relevant to higher levels of the software system. By splitting the functionality between separate modules, the software design is partitioned and removes the complex interdependencies associated with monolithic architectures.
picotronix – a dual pico hardware implementation
A picotronix implementation consists of a hardware system based on (usually) two Raspberry Pi Pico modules using RP2040, RP2350, or future versions of these microcontrollers. A Pico 2 module functions as the Analog and Digital Data acquisition and processing unit (the Data Pico) and a Pico 2W provides Control/Display and Remote Interface functions (the Control Pico). In addition, the Data Pico may be controlled by an external program running on a PC or other device. In that case the picograms would come from the external device, and the Control Pico may just act as a message forwarder.
The software running on these two modules is very different. Mixing them on one processor would make software development much harder – the Data Pico is very time critical, and cannot allocate cpu cycles to LCD updates or wifi traffic. The Control Pico must be driven by the LCD Display refresh cycle and external wifi network events. These timing constraints are very different.
data packet transfer protocol
Sending a complete byte array across a comms link requires a robust protocol. The protocol must include recovery, identification of the packet start and the length of the payload. For our Pico modules, we use an ASYNC full duplex direct link running at 3Mb/s. This is very reliable and errors should be rare.
The protocol is structured as :-
SYNC (4 bytes) LENGTH (2 bytes) PAYLOAD (0- 64K)
SYNC
The 4 byte sync value is chosen as 0x35C6A95A. This is a high entropy pseudorandom number – a binary number where each bit is 1 or 0 with equal probability. The chances of this bit sequence appearing in any data is 1/2^32 – which is about 1 in 4 billion. As a bonus, this sequence uniquely identifies the packet as a picogram.
The receiver will be waiting for a valid picogram data packet. It will receive incoming bytes into a 4 byte list until the 0x35C6A95A sequence is matched. This indicates that the following bytes are the remainder of the data packet. If the receiving processor has been reset or otherwise derailed, it may be in the middle of the packet being sent. The sync sequence allows the receiver to discard any incoming data that is not a complete packet.
LENGTH
The first 2 bytes after the sync sequence is a 16 bit payload length. Due to the constraints of Pico, data packets must be less than 64K – which is a reasonable constraint. This 16 bit value will exactly match the length of the following byte array. This byte array is expected to be msgpack serialized output. The receiver can then do a read of N bytes and pass this to umsgpack.loads(bytes).
PAYLOAD
The bulk of the packet follows the first 6 bytes of sync + length. If the payload is truncated, the transfer will not be accepted. Corrupted bytes will not be detected unless msgpack detects a violation – if msgpack fails to decode the payload, the packet will be ignored. The sender is responsible for dealing with an ignored request. It is dealt with at the application layer of software. Normally, when a picogram is received and decoded as a dictionary structure, the receiver will send a short special acknowledge picogram so the sender knows the transmission has been successful.
EXAMPLE
| 35 | C6 | A9 | 5A | _______ | hi | lo | _____ | b0 b1 b2 ……. |
Example Pod Scan Picogram
In the case of the picotronix being used as a peripheral client, one data request would be a request for a Pod ID scan. This may look like a key/value map like this
“MSG_IDENTITY” -> 14328
“MSG_OPERATION” -> “ID_SCAN”
The Control Pico will receive this and start an ID scan operation for the PORTS (two Analog, one mixed) to identify any attached PODS. The Control Pico will construct a return Picogram to send back which will look something like this –
“MSG_IDENTITY” -> 23841
“MSG_OPERATION” -> “ID_SCAN_RESULT”
“ANALOG-A” -> 16
“ANALOG-B” -> 13
“DIGITAL” -> 34
This information is all that is required to define the options available to the end user. For example, if the DIGITAL ID indicates that a 100MS/s ADC is attached to the Digital POD, the user will be offered appropriate Time-Base settings. The user software should know how to correctly drive that hardware module.
Distributed operation across the internet
The Data Pico is a Pico 2 which is configured to capture digital and analog signals. All requests and responses to this Pico are via picogram datagrams. The source of these datagrams is not restricted to any particular system or location. As long as the picogram is a valid intact block, it can originate from a remote system. This means it should be possible to use the Data Pico as a remote data capture device located anywhere.
An internet transfer may be any protocol, providing that the picogram maintains its integrity. The transport protocol should be one that guarantees complete data and does not allow out of order data sequencing.