Datalogging Part 4, Java Learning Exercise
This Part 4 describes the inner workings of a different, much simpler Datalogging class than the one used in Part 1 and Part 2.
It's provided here strictly as a Java learning exercise, and does not require familiarity with Parts 1 and 2.
Like the class in Parts 1 and 2, this one lets you store live robot data in a dedicated log file on the RC device, and download to a computer for spreadsheet evaluation and charting.
Datalogging has long been available in the Education version of LEGO Mindstorms software, used by FIRST LEGO League teams.
Advanced FTC teams sometimes develop custom data solutions with Android Studio. This tutorial series aims to make basic Datalogging available to less experienced teams using OnBot Java and FTC Blocks. It may also inspire a development effort to incorporate broader Datalogging capability into the FTC SDK.
Right-click this image to open a demonstration video in a new tab:
Datalogging demonstration on YouTube
An FTC robot can generate data from many sources:
- digital sensors (e.g. touch sensor)
- analog sensors (e.g. potentiometer)
- standalone I2C sensors (e.g. color sensor)
- built-in I2C Inertial Measurement Unit (IMU)
- encoders (DC motor, odometry)
- camera/navigation data from Vuforia
- camera/recognition data from TensorFlow Object Detection (TFOD)
- SDK/controller methods (e.g. motor current)
- user input (gamepad data)
- OpMode variables and index values
- Android/device data
- programmed timers
- etc.
Data can accumulate and change rapidly, often too fast for detailed evaluation from telemetry or human observation. Also, it can be hard to determine the linked relationships among various data sources.
Spreadsheets and graphing are common tools for evaluating data. This tutorial describes three steps:
- Write the data to a file, during the OpMode
- Download the completed log file to a computer
- Import and evaluate the data in a spreadsheet
To prepare, we create a Java program or class, containing commands or methods to be used by any OpMode for writing data to a log file. Create this program in OnBot Java (OBJ), to use it later in an OBJ OpMode or to support custom Blocks called myBlocks. MyBlocks are described in a separate tutorial at the FTC wiki.
Creating such a 'utility' class allows each OpMode to use simple commands, or methods, to implement Datalogging. Most of the preparation and work is done by the methods of this utility class, called Datalogger.
Ultimately the log file will contain multiple lines of text, where each line is a set of Comma Separated Values (CSV). To build this file, the OpMode uses one Datalogger method to append, or add, each data value to a single line of text. The OpMode uses another method to write that line to the log file, with an automatically included timestamp. This process is repeated to build up a record over time, then the OpMode uses a final method to close the datalog.
The Datalogger class' key lines of code are described here in six sections.
A Java constructor runs once, when the class is set up or instantiated in the user's OpMode. This constructor's only parameter is the user-specified name of the target datalog file.
public Datalogger (String fileName) {
// Build the path with the filename provided by the calling OpMode.
String directoryPath = "/sdcard/FIRST/java/src/Datalogs";
String filePath = directoryPath + "/" + fileName + ".txt";
new File(directoryPath).mkdir(); // create Datalogs folder if needed
// Set up the file writer and line buffer.
writer = new FileWriter(filePath);
lineBuffer = new StringBuffer(128); // initial length 128
timeBase = System.currentTimeMillis();
addField("Time"); // first/default column heading
} // end constructorNote: the FileWriter() and StringBuffer() tasks need exception handling, omitted here for clarity.
// The OpMode calls this method to complete the current line (row) of data
// and prepare for another.
public void newLine(){
long milliTime;
lineBuffer.append('\n'); // end-of-line character
writer.write(lineBuffer.toString()); // add line (row) to file
lineBuffer.setLength(0); // clear the line (row)
// Update and log the default first column (time of reading).
milliTime = System.currentTimeMillis();
// Divide milliseconds by 1,000 to log seconds, in field named "Time".
addField(String.format("%.3f",(milliTime - timeBase) / 1000.0));
// The 1000.0 decimal avoids a type error; the expression's variables are 'long'.
} // end newLine() methodNote: the writer.write() task needs exception handling, omitted here for clarity.
The Datalogger utility class builds a datalog file containing only text, separated by commas. Text can be of Java type String or char, handled by these two methods.
// These two (overloaded) methods add a text field to the line (row),
// preceded by a comma. This creates the comma-separated values (CSV).
public void addField(String s) {
if (lineBuffer.length()>0) {
lineBuffer.append(',');
}
lineBuffer.append(s);
}
public void addField(char c) {
if (lineBuffer.length()>0) {
lineBuffer.append(',');
}
lineBuffer.append(c);
}Checking the line length (before inserting a comma) is not needed when a default timestamp (and its comma) will be inserted before all data, as in the current example. The check is here in case the default timestamp is removed.
Each non-text data type needs its own method to accept the data, and convert it into Java type String or char. That method, in turn, uses one of the above text-only methods to append the field to the end of the current line.
Here is the method to process Boolean (true/false) data.
// The following (overloaded) method converts Boolean to text
// (Java type char) and adds it to the current line (row).
public void addField(boolean b) {
addField(b ? '1' : '0');
}If the parameter b is true, the character '1' is logged. Otherwise the character '0' is logged.
These (overloaded) methods accept various numeric types, all converted to type String for the method listed above. Not to worry! Spreadsheet programs typically interpret these correctly as numbers.
public void addField(byte b) {
addField(Byte.toString(b));
}
public void addField(short s) {
addField(Short.toString(s));
}
public void addField(long l) {
addField(Long.toString(l));
}
public void addField(float f) {
addField(Float.toString(f));
}
public void addField(double d) {
addField(Double.toString(d));
}Any int values are processed as long, through Java's implicit type casting or type promotion.
// The OpMode must call this method when finished logging data.
public void closeDataLogger() {
writer.close(); // close the file
}Note: the writer.close() task needs exception handling, omitted here for clarity.
The above 6 code sections are put together with their required import statements, member/variable declarations and exception handling. This gives a complete, basic Java class posted here as W_Datalogger_v05.java.
This example class also includes other minor features, omitted above for clarity.
-
second default column: time between data readings, to avoid later calculating "delta" between timestamps
-
resetTime()method to allow optional reset of main timer (first 2 columns) -
firstLine()method to close the first line containing only column labels -
privateclasses to meet Java programming guidelines for encapsulation -
finalize()method for Java garbage collection and improved exception handling
To create this Java file, connect your programming laptop to a Control Hub, or to an RC phone with Expansion Hub. The robot's active Configuration must include the built-in IMU, with its default name "imu". (This should already be present, under I2C Sensors, Bus 0.)
In OnBot Java, click the large plus-sign to create a new file named Datalogger.java. Empty its contents, then copy-and-paste the posted Example code into that file. Click the wrench icon "Build Everything" and wait for the acknowledgement "Build Successful".
Study the Java code and its comments. Add any comments you think will help you and your teammates later.
Optional to right-click and Download that Java file, to store it safely on the laptop.
For FTC Blocks users, myBlocks can be created to provide Datalogger's methods, separately and/or in various combinations. The FTC wiki contains a tutorial on myBlocks.
To get started, myBlocks should provide Datalogger's instantiation and its individual methods addField(), firstLine(), newLine(), resetTime() and closeDataLogger(). Use annotations with detailed comments/documentation describing how to use these myBlocks.
Next we describe a user's OpMode that could operate the robot and log the specified data, using methods provided in the Datalogger class.
For simplicity, this example uses a REV Control Hub or Expansion Hub with no connected devices (motors, servos or external sensors). It logs data only from the Inertial Measurement Unit (IMU) contained in the Hub.
Testing this OpMode can be done by simply rotating the Hub by hand, then looking at the line graph of the heading angle.
The OpMode's key lines of code are described here in seven sections.
String datalogFilename = "myDatalog_001"; // modify name for each run
// Instantiate Datalogger class.
imuDL = new Datalogger(datalogFilename);Edit the logfile name (in quotes) for each run. A repeated name will over-write the previous datalog. Optional to Rename the saved file after each run.
In this example, we gather only the Z-angle or "heading" from the IMU. We also create another piece of data, a simple index that counts each reading.
// Name the fields (column labels) generated by this OpMode.
imuDL.addField("Count");
imuDL.addField("IMU Heading Angle");
imuDL.firstLine(); // end first line (row)The firstLine() method is the same as newLine(), without adding a timestamp. It's not described above, for clarity.
The following code appears inside a loop, such as while (opModeIsActive()). The IMU provides angular orientation data in the order specified; here, the Z-angle is the first angle of "ZYX".
angles = imu.getAngularOrientation(AxesReference.INTRINSIC, AxesOrder.ZYX, AngleUnit.DEGREES);
myHeading = angles.firstAngle; // store Z-angle or heading
readCount ++; // increment the counter
These lines also appear in the same while() loop.
// Populate the fields to be logged. Must appear in the same order
// as the column labels, to ensure data is logged to the intended
// field name.
imuDL.addField(readCount);
imuDL.addField(myHeading);
imuDL.newLine(); // end the current set of readings
Driver Station (DS) telemetry allows the user to monitor the Z-angle (heading) while manually rotating the Control Hub or Expansion Hub.
// Show live telemetry on the Driver Station screen.
telemetry.addData("Count", readCount);
telemetry.addData("myHeading", "%.1f", myHeading);
telemetry.update();These optional telemetry commands send data only to the Driver Station (DS) screen, not to the datalog file.
Instead of looping 'forever', you could collect data for a specific amount of time by using a custom timer.
In a linear OpMode, the corresponding Java code can look like this:
// Put these 3 lines before waitForStart().
ElapsedTime logTimer;
logTimer = new ElapsedTime(ElapsedTime.Resolution.SECONDS);
double logDuration = 5; // time limit in seconds
logTimer.reset(); // put this immediately before the while() loop
while (logTimer.time() < logDuration && opModeIsActive()) {
// collect and log data here, optional to show DS telemetry
}Blocks programmers can refer to this loop timer example. It's part of an FTC Wiki tutorial on FTC Timers, useful for Blocks and Java programmers.
This housekeeping step must be performed when Datalogging is complete.
imuDL.closeDataLogger(); // close Datalogger when finished
Putting these pieces together, with the required import statements and member/variable declarations, gives the complete, basic Java OpMode posted here as W_DL_OpMode_IMU_v05.java.
This example OpMode includes two minor features, omitted above for clarity.
-
a custom timer is used to log data on a regular interval, rather than 'as fast as the main loop can cycle'
-
added telemetry of IMU status codes
To create this OpMode in OnBot Java, click the large plus-sign to create a new Autonomous OpMode named DL_OpMode_IMU_v01.java or your own preferred filename. Empty its contents, then copy-and-paste the posted Java code into your OpMode file.
Edit the W_Datalogger_v05 class name to agree with your saved name. This must be done in two places:
W_Datalogger_v05 imuDL; // edit name to Datalogger, or the name you used
// Instantiate Datalogger class. Edit name as needed.
imuDL = new W_Datalogger_v05(datalogFilename);Also edit the @Autonomous annotation, for the name and group you prefer to see on the Driver Station dropdown list of OpModes.
Click the wrench icon "Build Everything" and wait for the acknowledgement "Build Successful".
Study the Java code and its comments. Add any comments you think will help you and your teammates later. "Build" again after any changes.
This example OpMode does not include two optional features mentioned above:
-
"loop for time", to collect data for a specific number of seconds
-
call to
resetTime()method is commented out; it can restart the main timer (columns 1 & 2)
On the paired Driver Station (DS) device, select the new Autonomous OpMode.
Touch INIT, wait for acknowledgement (ready for Start).
After touching the Start button, slowly rotate the Control Hub or Expansion Hub counter-clockwise (to the left), by hand. When it has rotated about 90 degrees, touch the Stop button on the DS device.
The test is done! At this point there should be a new log file on the RC device, in the folder /sdcard/FIRST/java/src/Datalogs. You can see it listed in OnBot Java by refreshing the Chrome browser.
-
Run only as long as needed. At some point the RC device could "time out" and crash, possibly failing to close the datalog.
-
Do not run the OpMode while the RC device is connected via USB cable (in file transfer or MTP mode) to a Windows laptop. The datalog file will be created, but no data will be logged to the file.
-
It's OK to run the OpMode while the RC device is (optionally) connected wirelessly via Android Debug Bridge (adb), if used.
After being closed (by the OpMode), the log file can be downloaded or copied from the RC phone or Control Hub to your laptop.
The sample Java class "Datalogger" stores log files in the folder /sdcard/FIRST/java/src/Datalogs. This makes Datalogs and its log files appear in the OnBot Java screen, at the left side.
Refresh the Chrome browser, to update the folder listing. Click on a datalog filename to see its contents.
Right-click on a log file, and choose "Download". During the download dialog, navigate to your target folder on the laptop, and change the file extension from .txt to .csv. This change allows the file to be automatically recognized and imported by spreadsheet programs.
The downloaded CSV filename may appear at the bottom of the browser window (green oval in above image). Double-click or Open the file; it will automatically open in the laptop's default spreadsheet software, e.g. Microsoft Excel, if installed. Or right-click and choose "Show in folder", to open its location in Windows Explorer.
If the sample Java class "Datalogger" was modified to store the log files elsewhere, they can be transferred using instructions from Datalogging Part 3, RC File Transfer.
Now the CSV file is stored on the computer. The goal is to import this data to a spreadsheet program for evaluation. This tutorial covers Microsoft Excel and Google Sheets, two popular spreadsheets with built-in charting and graphing.
For the following example datalog, the OpMode was started, then the REV Control Hub was manually (slowly) rotated past 90 degrees, then the OpMode was stopped. This took about 15 seconds overall.
On the computer, double-clicking the CSV filename may automatically open it in the default application such as Microsoft Excel (if installed). Or, right-click and choose "Open with...", then choose Excel.
The file will open, showing several columns and many rows of data. Click on the column heading (letter C or D) over "IMU Heading Angle"; this is the data to be charted.
Click the top menu item "Insert", circled above in yellow. Then click on the icon for charts (see green arrow). Then click the basic 2-D Line style, circled in yellow.
This will insert a simple line graph of the IMU heading data, as shown below.
That's it! You have successfully stored and charted data from an FTC robot.
If you "Save As" Excel format, your original CSV file will not be affected.
Excel locks any open file. Close Excel, or the open CSV file, to allow an adb pull of another datalog with the same filename. Or, use the sample OpMode's datalogFilename variable to uniquely name each CSV file.
Excel offers many more data evaluation features, beyond the scope of this tutorial. You are encouraged to learn and explore those features, ultimately providing a better understanding of robot performance. This may help you with troubleshooting, optimization and robot design.
In the Chrome browser, type "sheets.google.com". Then click the large plus sign in the lower right corner, as shown below.
This creates and opens a new untitled spreadsheet. As shown below in yellow, click "File", then "Import". Then click "Upload".
Drag the CSV data file from Windows Explorer to the rectangle shown, or click the blue "Select a file.." box to manually find and select the CSV data file.
For this example, the default settings are good, as shown by the yellow check-marks below. Click "Import data".
The file will open, showing several columns and many rows of data. Click on the column heading (letter C or D) over "IMU Heading Angle"; this is the data to be charted.
As shown below, click "Insert", then "Chart". This will insert a simple line graph of the IMU heading data, as shown below.
That's it! You have successfully stored and charted data from an FTC robot.
At top left, type a filename to replace "Untitled spreadsheet". There's no "save" button; Sheets changes are automatically saved online.
Google Sheets offers many more data evaluation features, beyond the scope of this tutorial. You are encouraged to learn and explore those features, ultimately providing a better understanding of robot performance. This may help you with troubleshooting, optimization and robot design.
Datalogging demonstration on YouTube
After your basic tests are working well, consider adding some optional features mentioned above. Name your new files with version numbers, using OnBot Java's right-click commands Copy, Paste and Rename.
Then continue customizing the OpMode for your needs. Work step-by-step, adding one feature at a time.
Teams using FTC Blocks and OnBot Java can use Datalogging to solve problems and improve robot performance. Teams using Android Studio can use all the same Java code described above.
FTC students will benefit from this valuable skill, widely used in professional fields such as engineering, scientific research, medicine, social media and marketing, business management, and many more.
You are encouraged to submit and describe other examples that worked for you.
===========
This is Part 4 of a 4-part series on FTC Datalogging.
Part 1 shows how to run a sample OpMode and chart its data, using a robust and versatile Datalogging class.
Part 2 shows how to customize the sample OpMode for logging your own robot data of interest.
Part 3 shows Android Studio users how to transfer datalog files from the RC device to a computer. (This isn't needed with OnBot Java, where datalogs appear on-screen for easy Download.)
This Part 4 describes the inner workings of a different, much simpler Datalogging class. It's strictly a Java learning exercise, showing basic steps to log robot data on the RC device. It does not require familiarity with Parts 1 and 2.
Questions, comments and corrections to [email protected]
-
TensorFlow 2023-2024