RFM95W transceiver convert packet to variables

[Javacat1]

I'm working on a radio-based rocket tracking ground camera, and have RFM95W transceivers for communication. I'm wanting to send multiple pieces of data (a time tag and 3d acceleration) in a single packet to reduce transmission losses, and I've gotten the Arduino code for simple transmit/receive (send string in, get string out) however I want to split the packet's data into 4 variables on the receiver end to be able to manipulate them individually. Does anyone have any guides/tips/places to look?

 

[waltr]

Yes, but it depends on which programming language you are using.

Since you mention Arduino I assume you are doing this in Arduino IDE which is C.
In this case google "c strings" and "C string parsing" to learn how.

 

[Javacat1]

Yes, but it depends on which programming language you are using.

Since you mention Arduino I assume you are doing this in Arduino IDE which is C.
In this case google "c strings" and "C string parsing" to learn how.

Yes, sorry I forgot to include that. Using an ESP8266, it looks like strtok() is the function I'm looking for

 

[Steve Shannon]

I'm working on a radio-based rocket tracking ground camera, and have RFM95W transceivers for communication. I'm wanting to send multiple pieces of data (a time tag and 3d acceleration) in a single packet to reduce transmission losses, and I've gotten the Arduino code for simple transmit/receive (send string in, get string out) however I want to split the packet's data into 4 variables on the receiver end to be able to manipulate them individually. Does anyone have any guides/tips/places to look?

What you’re talking about is called a protocol. The transmitter assembles a message, usually with a CRC or some kind of checksum for error checking. The receiver receives the message, verifies the checksum or CRC and, if they match, disassembles the message. If the receiver side gets a bad message it sends a nack (not acknowledged) which tells the transmitter side to repeat the message. Otherwise it sends an ack.
There are already many different protocols, but some of them are very complex. My advice is to build and test the protocol library first, and then incorporate it in the radio transceivers. If part of the protocol message includes a byte that indicates the version of protocol and a byte that reflects the size of the message then you can handle newer versions without failure even if versions are mismatched. (Unrecognized tokens from newer versions get discarded).

 

[waltr]

Yes, sorry I forgot to include that. Using an ESP8266, it looks like strtok() is the function I'm looking for

Was not familiar with that function. Should do what you want.

I also have been using ESP8266 modules. One reads Acc, Mag, Gyro & Alt then logs data to Flash.
After landing I download data to my phone then email to myself. Works very much like the Eggtimer Quantum.

Steve has good points.
You seem to have a very simple protocol. What happens if part of the 'packet' doesn't make it to the receiver?

 

[Javacat1]

Was not familiar with that function. Should do what you want.

I also have been using ESP8266 modules. One reads Acc, Mag, Gyro & Alt then logs data to Flash.
After landing I download data to my phone then email to myself. Works very much like the Eggtimer Quantum.

Steve has good points.
You seem to have a very simple protocol. What happens if part of the 'packet' doesn't make it to the receiver?

I actually have a payload built from an esp8266 that my mentor's company provides that does almost exactly what you described does. I haven't thought super far ahead into the protocol or partial packets, as this is my first time diving into radio systems, but my plan for missing packets was to take advantage of the time tag to account for dropped packets by just upping the timespan used for calculations. I'm still in the stage of making them chirp at each other, so as I'm learning more about it I'll be taking stuff into account. currently my plan is "hope it doesn't break or just ignore dropped data"

 

[curtisheisey]

You can include an int with a sequence number in the transmitted packet. Increase it by one for each packet sent. The receiver can easily tell if a packet is dropped, if there is a gap in the received numbers.

 

[Javacat1]

You can include an int with a sequence number in the transmitted packet. Increase it by one for each packet sent. The receiver can easily tell if a packet is dropped, if there is a gap in the received numbers.

Yeah that's the purpose of the ttag, since I'm sending acceleration data I can use the differences between previous tags as my delta t. I also got advice to make packed structs for more efficient data processing but I need to do more testing.

 

[SparkyVT]

I'm working on a radio-based rocket tracking ground camera, and have RFM95W transceivers for communication. I'm wanting to send multiple pieces of data (a time tag and 3d acceleration) in a single packet to reduce transmission losses, and I've gotten the Arduino code for simple transmit/receive (send string in, get string out) however I want to split the packet's data into 4 variables on the receiver end to be able to manipulate them individually. Does anyone have any guides/tips/places to look?

I've done this with my system using the RadioHead RFM95 library. My protocol sends 5 packets per second, with 4 samples of data in each packet, for an effective 20 samples per second across 7 parameters. I only send GPS coordinates once per packet since most GPS units can't update that fast. To save transmission time I convert all the parameters into 16-bit or 8-bit integers, except for the GPS coordinates where I use floats. You can look at the specific code here:

https://github.com/SparkyVT/HPR-Rocket-Flight-Computer/blob/master/Main Code/Telemetry.ino
Your data rate will be highly influenced by the LoRa settings you choose. If you stick with the default you will get good range and distance. If you want more redundancy, then you can increase the spreading factor or coding rate, but will lose bit rate. You could potentially compensate by increasing the bandwidth, but run a higher risk of interference. My code also has an optional FHSS implementation if you are curious.

 

[Javacat1]

I've done this with my system using the RadioHead RFM95 library. My protocol sends 5 packets per second, with 4 samples of data in each packet, for an effective 20 samples per second across 7 parameters. I only send GPS coordinates once per packet since most GPS units can't update that fast. To save transmission time I convert all the parameters into 16-bit or 8-bit integers, except for the GPS coordinates where I use floats. You can look at the specific code here:

https://github.com/SparkyVT/HPR-Rocket-Flight-Computer/blob/master/Main Code/Telemetry.ino
Your data rate will be highly influenced by the LoRa settings you choose. If you stick with the default you will get good range and distance. If you want more redundancy, then you can increase the spreading factor or coding rate, but will lose bit rate. You could potentially compensate by increasing the bandwidth, but run a higher risk of interference. My code also has an optional FHSS implementation if you are curious.

thank you for this! I'll be taking a look through this as I work on my stuff. Honestly not super worried about redundancy, but still very interesting. I also got advice from some others I was talking with to use packed structures as the actual data format to be more efficient.

 

[SparkyVT]

Interesting. I just looked up what a packed structure was and yes, it will be more efficient use of memory. I don't use this in my code and instead just use a plain old global variable that contains a bunch of bytes. I feed that to the radio and just parse the stream of bytes on the other end. A packed structure would certainly be more elegant and efficient, so I might give that a try.

 

[Javacat1]

Interesting. I just looked up what a packed structure was and yes, it will be more efficient use of memory. I don't use this in my code and instead just use a plain old global variable that contains a bunch of bytes. I feed that to the radio and just parse the stream of bytes on the other end. A packed structure would certainly be more elegant and efficient, so I might give that a try.

Here's my brand-new repo to put stuff I haven't gotten a chance to test the radio functionality with structures yet, but stuff has been compiling so far so hopefully it'll work

 

[SparkyVT]

What distance are you trying to send packets? I ask because you have your power set to 23, which is the maximum setting of 100mW. I use a power setting of 12 or 13, which is less than 30mW, and still get miles and miles of range on the default settings. I recommend reducing it to decrease the chance of intereference with other components like the flight computer.

I see that you are using a BMP280. Good luck. My LoRa radio iterfered with the BMP280 and I never could solve it. I would check to make sure this isn't happening with your system. It would alwasy give me erroneous pressure and temperature readings when the radio was transmitting.

I also see that you are trying to send 64 byte packets at what looks like 10ms intervals. This is probably not feasible because the radio will not be done transmitting by the time the next packet needs to go out. You will step on the transmission every time and no packets will complete.

 

[Steve Shannon]

Here's my brand-new repo to put stuff I haven't gotten a chance to test the radio functionality with structures yet, but stuff has been compiling so far so hopefully it'll work

I wish I had a nickel for every time i felt hopeful simply because something compiled!

 

[waltr]

Here's my brand-new repo to put stuff I haven't gotten a chance to test the radio functionality with structures yet, but stuff has been compiling so far so hopefully it'll work

question with this part of your code:

angleX = mpu6050.getAngleX();
angleY = mpu6050.getAngleY();
angleZ = mpu6050.getAngleZ();

why angle from accelerometer??
This is ok IF the rocket is sitting on the pad but meaningless once in the air.
On the pad there is a gravity vector reference but not in the air.

 

[thzero]

I wish I had a nickel for every time i felt hopeful simply because something compiled!

Ship it!

 

[Javacat1]

What distance are you trying to send packets? I ask because you have your power set to 23, which is the maximum setting of 100mW. I use a power setting of 12 or 13, which is less than 30mW, and still get miles and miles of range on the default settings. I recommend reducing it to decrease the chance of intereference with other components like the flight computer.

I see that you are using a BMP280. Good luck. My LoRa radio iterfered with the BMP280 and I never could solve it. I would check to make sure this isn't happening with your system. It would alwasy give me erroneous pressure and temperature readings when the radio was transmitting.

I also see that you are trying to send 64 byte packets at what looks like 10ms intervals. This is probably not feasible because the radio will not be done transmitting by the time the next packet needs to go out. You will step on the transmission every time and no packets will complete.

I'm talking with a radio enthusiast friend of mine, and they recommended the same thing about Tx power. I only had it on 23 because I have no idea what exactly I need currently, and just wanted to guarantee a connection. Also, I've gotten packed structures to send as packets, which are more efficient, and hopefully will be able to send more often given smaller data sizes as well. I'm still in the early testing stage, but I'll also keep that in mind for the BMP280

 

[SparkyVT]

I use the SX1272 LoRa calculator provided by Semtech to help figure out my expected on-air time and bitrate.

https://semtech.my.salesforce.com/s...K/6T9Vdb3_ldnElA8drIbPYjs1wBbhlWUXej8ZMXtZXOM

 

[benjaminjw]

Keeping information as text has a lot of advantages in readability and usability. Formats like CSV and JSON can solve a lot of problems. Their downside is size and performance. You will use CPU cycles converting the information to text, extra bandwidth to transmit the information, and extra CPU cycles in the receiver to convert from text into values a computer can use for calculations. But that overhead may not be an issue for you. It may be difficult to measure any speed difference and the space difference may not be a problem unless you are close to the data rate limit of your radio.

Binary protocols solve some of these issues. If you want to create a binary protocol, you’ll want to study up a little on how information is represented in computers. Computers store variable values in sequences of bytes. The number of bytes and the method used to interpret these bytes into information depends on the processor.

A common convention is to use “network” order. There are functions in the c standard library to serialize and deserialize common data types like into and out of network order. A lot of these functions have terse names like ntohf or htonf for the reverse.

Variable length field types like strings or arbitrary byte sequences can be encoded in two parts: a length indicator and a payload. The length indicator needs to be sized to represent the maximum size of the payload. There are more sophisticated techniques that use variable length encoding.

Many libraries and formats — Google Protocol Buffers, Thrift, Aveo, CBOR — have been created to address this kind of issue. I haven’t attempted to use them for embedded development, but they are common for programming in other contexts. This is an extremely heavy handed approach that is most often done to avoid programmers writing tedious protocol code and to enable cross language communication (e.g. C++ to Java) or communication across processor architectures. You might draw some inspiration by reading about how these libraries approach the problem.

I wouldn’t worry about the performance of text unless you’re really pushing the limits of your CPU or have some experience with binary formats. It isn’t a challenge that you can’t overcome without some studying and some work, but it might not be worth the effort. It can be super fun and gratifying so if that sounds fun to you … go for it!

I would +1 earlier suggestions about explicit sequence numbers since that would make it easy to see packet drops. I might also consider a protocol version number to help you evolve the protocol.

Good luck!

 

[benjaminjw]

https://github.com/Javacat1/Radio-Camera/blob/main/Logging_Transmit.ino
I think you could have a small bug that might avoid any data rate concerns. Line 108 sends the 64-byte value present in the packet variable. It sends 64 bytes where the beginning bytes are the contents of the packet, but the tail of the packet is probably uninitialized data. This is probably a mistake since the earlier line composes a variable length string. I suspect you're not noticing data corruption issues because each invocation of the snprintf on the buffer is likely producing a longer output string than previous invocations. If a future invocation could be shorter than a previous one you would transmit bits of the old string at the tail end. One way to check if this is true in in the setup(), store 64x 'A' characters into the packet variable. If my guess is correct, then I think you're going to be happy.

The good news is that sending 64 bytes each loop() is more than you need. The snprintf function returns the number of bytes produced in the output string. The function is designed to produce output of different lengths on each invocation, so the return value tells you how long the output string is each time it runs. You could capture the return value from snprintf to determine the length of the output string. Use that as the length of the packet to transmit instead of the hard coded 64 on line 108.

Earlier there was talk about binary/packed data formats vs text. I don't think this is necessary, but here are 3 ridiculous ideas to shrink the packet size and keep the protocol as text. Lines 107 produces the text-based protocol sending a 4-byte integer and 3x 4-byte floats truncated to 1 digits of precision. The 4 byte integer as text will turn into a variable length sequence of characters (e.g. "30", "40", "50", ...). Incrementing by 10 every 10ms will quickly reach 5-7 digits. The value will determine the number of digits. The floats will be fixed length because they are truncated (e.g. "0.1", "0.2", ...). The truncation for the transmit packet is 1 digit plus a decimal point. Plus there are delimiters between fields. In total, 3 floats with 2 characters each, 3 commas, and a variable length integer. Total: 15-20 bytes per packet. Not bad. Very readable. Less than the 64 bytes per packet you thought you might need. You might even consider adding the full number of variables to the radio transmission and you might consider increasing the precision from 1 digit to 3. It's good to have options!

• Reduce the number of bytes transmitted by using a +1 counter rather than transmitting ttag. Reduce the size of the counter to a 2-byte value (2^16 10ms intervals is hours). A 2-byte hex value written as text will consume 4 bytes regardless the size. The printf format specifier you want is something like %4x. The total packet size would be 4+1+2+1+2+1+2=13.
• Eliminate the commas. Now that each field is a fixed length, you can infer where each field begins and ends just by the position. e.g. "0001.1.3.2" is counter 1 with x, y, z values. That saves a further 3 bytes.
• Eliminate the decimal points. This is a bit tricker since it is built into printf. You'll have to write the float to a different variable and then copy the bytes from the tmp variable into the output string. Honestly this is probably more work than sending as binary, but it is possible. The net result would be a 4-character ttag counter and 3 characters (1 character per float). Down to 7 bytes per packet.

Binary works too. The rpacket struct contains an int and 3 floats. That's 12 bytes of data, assuming the compiler packs the fields densely. You can validate the packing using the sizeof( ) method to determine the size of an instance of the struct. The rf95 function wants send(uint8_t *, int). The same casting technique will work here. So 12 bytes of binary is bigger than 7. What gives? The binary protocol transmits the full floating point precision of the 3 variables! So a lot more information is conveyed by the binary protocol rather than the text protocol.

 

[Former Squirrel]

You might find these helpful. Using the union, user-defined data types gets you past the LoRa packet char to float thing.