IQ Measurement Automation Plugin User Guide

Owned by John Medalen
Last updated: May 05, 2021 by John Medalen
14 min read

IQ Measurement Automation - User Guide
Contents
List of Figures
Introduction
Main Page
Dark and Light Test Initial Conditions
Dark Tests
Dark Level Stability Test (DLS)
Dark Shading and Pixel Above Mean Test (DSH)
Light Tests
SNR and Structure Noise Test
Light Shading Test
Sensor Response Linearity Test
Sensor Gain Linearity Test
Running the Test- An Example
Light Tests
Dark Tests
Report Summary
Config File
Saving Config File
Loading Config File
Troubleshooting
Dependency Warning
Start Dark Tests button not enabled
Start Light Tests button not enabled
Data is the same for DLS test and Structure noise test

List of Figures

Figure 1 - Main Page
Figure 2 - Dark and light test initial conditions
Figure 3 - Dark Level Stability Test
Figure 4 - Dark Shading Test Input fields:
Figure 5 - Light Tests
Figure 6 - SNR Test
Figure 7 - Light Shading Test
Figure 8 - Sensor Response Linearity Test
Figure 9 - Sensor Gain Linearity Test
Figure 10 - MAIN Page Example
Figure 11 - Light Shading Test Example
Figure 12 - Temperature Setting Prompt
Figure 13 - Test Progress Dialog
Figure 14 - Test Completed Message
Figure 15 - Dark Tests Example
Figure 16 - Running Script Example
Figure 17 - Tests Completed
Figure 18 - Both MCR and XLSXWriter Missing
Figure 19 - MCR Missing
Figure 20 - XLSXWriter Missing
Figure 21 - Matlab Warning


Introduction

Image Quality Measurement Automation is a DevWare/DevWareX plugin to automate the image quality measurements required by the AEs and FAEs. It has a suite of tests that can be run in any combination, and the results of the test are beautifully packaged into a report spreadsheet, generated at the end of the test. The plugin is released as a part of DevWare, both internal and external versions. It is available from DevWare release version 4.5.30 onwards.This plugin has two sections of tests, namely, the Dark Tests and the Light Tests. The Dark tests are run with the setup in complete darkness, whereas the Light tests are run with a flat field setup.To run the IQ measurement Automation plugin:

  1. Connect the device under test (module/headboard + Demo3) to the laptop/PC
  2. Run DevWare. Probe the device / load sensor data as required.
  3. Load the INI file that has the presets to be used for testing.
  4. Then click Plugins -> Image Quality Measurement Automation to start the plugin.

When started, the plugin displays the main page. When running for the first time, you might get a couple of warnings. This is because the plugin requires a couple of components that are not available by default with DevWare. These are:

  1. XlsxWriter package for Python.
  2. Matlab compiler Runtime version 7.17

Follow the instructions in the warning dialogs displayed by the tool for download and installation.

Note: long Preset names may cause a "No such file or directory" error from XlsxWriter.  This is due to the Preset name being used to form the folders and files.
If this error occurs, the only solution is to rename the Preset name in the INI file to a length of 30 characters or less.

Main Page

When the plugin is started, the following page is shown. It gives an introduction to the plugin, and takes some basic inputs from the user to run all tests.
Figure 1 - Main Page
Input fields:

  1. Configuration File – Clicking the Load button allows the user to load a config file. This is a pre-defined config file, best saved from the GUI.
  2. Test Description – This will be the name of the project. Enter a string. This string will be a part of the report spreadsheet filename.
  3. Project Path – This will be the location where the test results will be stored. Browse button on the main page helps user navigate the folder structure to select the project path.
  4. Responsible Name – This will be the name of the Engineer running the test. It will be printed in the summary sheet of the report spreadsheet.
  5. Hardware Description – This is the name of the sensor/SOC for which the test is being run, and any other details the user wishes to add. This will be printed in the summary sheet of the report spreadsheet.
  6. Setup Description – this is a description of the setup, indicating illuminant used, etc.
  7. Tests – The main page also contains checkboxes for each test to be run. Special condition tests are currently not supported, and hence are greyed out. Select the tests you want to run, then click Next

Clicking "Exit" hides the plugin. It can be re-opened by clicking Plugins -> IQ Measurement Automation. When the Next button is clicked, the Tests page is shown. This page has one tab for each set of tests. Currently, there is one tab for Dark tests, and one for Light tests.

Dark and Light Test Initial Conditions

Both dark and light tests have "Initial Conditions". The settings in this section apply to all tests in that tab. The initial conditions tab in the GUI is shown below:
Figure 2 - Dark and light test initial conditions
It has the following inputs:

  1. Analog gain – This field accepts analog gains at which to run the tests. Multiple gain values are separated by commas. The values for the Analog gain should be between 1.00 and 8.00.
  2. Temperatures – This field accepts temperature(s) of the testing environment. For multiple temperatures, separate using a comma (,). At the start of each test, you will be prompted to adjust the setup temperature to match the requirement. Once the setup is completed, click OK to start the test.
  3. ROI – This is the ROI used to run the Dark Level Stability test. You can select full frame, Mouse Selection, or center ROIs.
  4. Presets – The plugin uses the last INI file loaded into DevWare. This name is displayed in the label as shown in the screenshot. The presets are executed in the order in which they are selected.

For example: In the above screenshot, if you select the "stream_1080p_30fps" before you select "stream_13m_30fps", then the tests for preset "stream_1080p_30fps" are executed before that for "stream_13m_30fps".The flowchart for all tests is as below, based on initial conditionsGain == LastGain?YesNoRes == LastRes? YesNoEndT == LastTemp?T = Temp[TempCount]Res = Resolutions[PresetCount]Gain = Gains[GainCount]TempCount += 1Export to excel and plot / generate reportRequires User Interaction to set temperatureRun testGainCount += 1PresetCount += 1StartNoYes

Dark Tests

Dark tests are run in "dark conditions". These tests are designed for light absence throughout the sensor. To run these tests, mount the sensor in a dark chamber. Do NOT change the light conditions until the measurement process is over.Two dark tests are supported in this plugin:

  • Dark Level Stability Test and Structure Noise Test
  • Dark Shading and Pixels Above Mean Test

Dark Level Stability Test (DLS)


Figure 3 - Dark Level Stability Test
The input fields for the Dark Level Stability test are as below:

  1. Image Data Type – This is the image data type to use for the noise exposure sweep.
  2. Frames Per Sample – This is the number of frames to capture per sample for the noise exposure sweep.
  3. Maximum exposure – This is the value of the exposure (in ms) at which the noise exposure sweep stops.
  4. Log Scale – Sweep through exposure in log scale, or linear scale
  5. Decades and Samples/Decade – When Log scale checkbox is checked, these fields decide the number of samples for the exposure sweep
  6. Min Exposure/Num Samples – These fields are valid when the exposure sweep is on linear scale.
  7. Structure Noise ROI – Use either full frame ROI, or the same ROI as for Dark Level Stability test.

If this is the only Dark Test selected, the "Start Dark Tests" button should be enabled once the Initial conditions and DLS test inputs are entered. If it is not enabled, check the troubleshooting section of this document.

Dark Shading and Pixel Above Mean Test (DSH)


Figure 4 - Dark Shading TestInput fields:

  1. Pixel Integration time - These are the integration times (in ms) for which the dark shading test is run. Enter multiple values by separating them with a comma (,).
  2. Color Correction and White Balance – This enables the color correction and white balance in DevWare.
  3. Digital Gain Stretching – This enables digital gain stretching. Enter a number between 1 and 100 in the box next to it, for a digital gain stretch value.
  4. Total Frames – This is the total number of frames to capture for the dark shading test.
  5. Average into one frame – When this check box is selected, the input frames are averaged into one frame before it is saved, and histogram data is collected.
  6. Output Formats – These are the additional formats of images required to be saved. RAW image will be saved by default.

Light Tests

Light tests are run in "flat field conditions". These tests are designed for uniform light conditions throughout the sensor. To run these tests, mount the sensor in a chamber with a flat scene. Do NOT change the light conditions until the measurement process is over.The initial conditions for the Light tests are exactly the same as for Dark Tests. Refer Dark Tests Section for the input fields in "Initial Conditions".
Figure 5 - Light Tests
Four Light tests are supported in this plugin:

  • Signal to Noise Ratio (SNR) and Structure Noise test
  • Light Shading Test
  • Sensor Response Linearity Test
  • Sensor Gain Linearity Test

SNR and Structure Noise Test


Figure 6 - SNR Test
Input fields:

  1. Image Data Type – This is the image data type to use for the noise exposure sweep.
  2. Maximum exposure – This is the value of the exposure (in ms) at which the noise exposure sweep stops. Clicking the "Auto Calc" button gets the software to automatically calculate this value as the exposure time at which all the channels saturate.
  3. Frames Per Sample – This is the number of frames to capture per sample for the noise exposure sweep.
  4. Log Scale – Sweep through exposure in log scale, or linear scale
  5. Decades and Samples/Decade – When Log scale checkbox is checked, these fields decide the number of samples for the exposure sweep
  6. Min Exposure/Num Samples – These fields are valid when the exposure sweep is on linear scale.
  7. Structure Noise ROI – Use either full frame ROI, or the same ROI as for Dark Level Stability test.

Light Shading Test


Figure 7 - Light Shading Test
This test has the following inputs:

  1. Color Correction and White Balance – This enables the color correction and white balance in DevWare.
  2. Pixel Integration times – This group of inputs has 2 check boxes.
    1. Manual Exp Time – If this check box is selected, then the numbers entered next to it are the integration times (in ms) for which the light shading test is run. Enter multiple values by separating them with a comma (,).
    2. Auto Exp Time – If this check box is selected, enter the target signal level percentages in the box next to it. Enter values greater than 100 to get over-saturated cases. Default value is 68.36% which corresponds to 700 signal level out of a full signal range of 1024. Enter multiple values by separating them with a comma (,).
  3. Total Frames – This is the total number of frames to capture for the dark shading test.
  4. Average into one frame – When this check box is selected, the input frames are averaged into one frame before it is saved, and histogram data is collected.
  5. Output Formats – These are the additional formats of images required to be saved. RAW image will be saved by default.

Sensor Response Linearity Test

This test runs the noise exposure sweep test for two sets of exposure ranges, with different number of samples for each exposure range. These are:

  1. Small signal level – This is the small signal range (default 1ms to 10ms).
  2. Medium signal level – This is the medium signal range (default 11ms to 200ms).

The rule is the maximum exposure for the small signal level should be 5% of the maximum exposure for the medium signal level range.
Figure 8 - Sensor Response Linearity Test
Inputs:

  1. Auto Calc Exposure Times – The "Auto calc max exposure time" can be used to use the plugin to calculate the max exposure time as the exposure time at which all the channels saturate. Clicking this will fill all the exposure time input fields for this test.
  2. Default Dark Offset – If this checkbox is checked, the plugin uses default dark offset values in the Linearity test calculations. If unchecked, the user will be prompted to create "dark" conditions, so that the dark offsets can be collected for this test.
  3. Image Data Type – This is the image data type to use for the noise exposure sweep.
  4. Frames Per Sample – This is the number of frames to capture per sample for the noise exposure sweep.
  5. Small Signal Level Sweep – Select the minimum and maximum exposure time for the small signal sweep, and the number of samples for this sweep. The sweep is linear, not log scale.
  6. Medium Signal Level Sweep – Select the maximum exposure time and number of samples for this sweep. The sweep is linear, not log scale. Also, the minimum exposure time for this sweep is decided by the maximum exposure time in the Small signal level sweep, to avoid overlaps in the sweeps.

Sensor Gain Linearity Test

This test runs a gain sweep for a given pixel integration time. It collects the noise exposure results into a text file, which is then processed by the python script to generate the report summary. The GUI input fields for this test are as shown below:
Figure 9 - Sensor Gain Linearity Test
Input Fields:

  1. Image Data Type – This is the image data type to use for the noise exposure sweep.
  2. Min Gain – this is the minimum value for the gain sweep. Must be between 1.00 and 8.00.
  3. Max Gain – This is the maximum gain value for the gain sweep. Must be between 1.00 and 8.00. and greater than Min Gain.
  4. Pixel Integration Time – The integration times at which the gain sweep is run. Multiple values are separated by a comma(,). The "Auto calc exposure" checkbox can be used to get the plugin to calculate the max exposure time as the exposure time at which all the channels saturate.
  5. Frames Per Sample – This is the number of frames to capture per sample for the gain sweep.
  6. Number of Samples – Number of samples for the gain sweep.
  7. Default Dark Offset – If this checkbox is checked, the plugin uses default dark offset values in the Linearity test calculations. If unchecked, the user will be prompted to create "dark" conditions, so that the dark offsets can be collected for this test.

Running the Test- An Example

In this section the user is guided through running the plugin from start to end for two dark tests and one light test. The main page GUI for the example is shown below. The results are chosen to be saved at "C:\Aptina Imaging Dev" folder.
Figure 10 - MAIN Page Example
On clicking "Next", the test page will be shown. Either the dark or the light tests can be run first. Order does not matter. However, the report summary spreadsheet is generated only when all selected tests have been run. The following sections describe the test progress.

Light Tests

Let us run the Light tests first. The screenshot below shows the Light tests tab in the Tests page.As you can see below, only the Light Shading test is enabled. The other tests are marked "(Disabled)". This is because only the Light Shading test was selected on the Main page.After entering all the inputs (refer Light Shading Test section), click the "Start Light Tests" button to start the light tests. If the "Start Light Tests" button is not enabled, it means you are missing an input required to run the tests. Look at the inputs carefully, and ensure none of the fields are empty.
Figure 11 - Light Shading Test Example
Clicking the "Start Light Tests" button starts the test. At the start of every test, the following message box is shown.
Figure 12 - Temperature Setting Prompt
When this box appears, the user must make sure they have the setup set at the temperature requested. Once the setup is ready, the user clicks "OK". This will start the test. Clicking "Cancel" will cancel the test at this point.Once the test is running, the following progress dialog is displayed.
Figure 13 - Test Progress Dialog
Both Dark tests and Light tests have a progress dialog. The dialog informs the user which test is currently running, and the conditions for the current test. The test can be cancelled at any time by clicking the "Cancel" button. The test might take a few seconds to cancel, depending on what operation is running at that time. Also, if a test is cancelled, the results folder is deleted.Once light tests are completed, the following message box is shown, indicating that the report spreadsheet will be generated after the selected dark tests are also completed.
Figure 14 - Test Completed Message

Dark Tests

The following screenshot shows Dark Tests. Both the Dark Level Stability and Dark Shading tests are selected, and both will be run, in order. A similar progress dialog as shown above will be displayed while the test is being run.
Figure 15 - Dark Tests Example
When all the tests selected have been completed, the python script generates a spreadsheet. This may take a while, especially when the shading tests are selected, and the following status appears next to the "Start Dark Tests" button, indicating that the plugin generating the spreadsheet.
Figure 16 - Running Script Example
When all the tests are completed, the following box is shown. Clicking the link will take you to the results folder.
Figure 17 - Tests Completed

Report Summary

The report summary spreadsheet is generated by a python script at the end of the test. The python scripts are in the same folder as the plugin DLL, and the report summary is saved to the results folder. The summary spreadsheet's file name is decided by the "Test Description" entered in the main page of the plugin. The spreadsheet has the following sheets

  1. Summary Sheet – This sheet provides a concise summary of all the tests that were run. It contains the important information in the form of tables and plots. It gives links to the individual test sheets as well.
  2. Mosaic Sheet – This sheet has all the plots generated for each test. It displays these plots side-by-side for easy comparison
  3. Individual Test Sheets – Each test has its own sheet in the report spreadsheet. These are linked to the summary sheet, and contain all the detailed information for each test.
  4. INI File sheet – this sheet contains the entire INI file used to run the test. Selecting column A and copying to a notepad will result in a copy of the original INI file.
  5. Config File sheet – This is the config file with all the test inputs used to run this test. When saved to a notepad, it can be used to load configuration into the plugin, to replicate the test.

Config File

The plugin provides a way to replicate the test elsewhere, by allowing the user to save the input test conditions into a file, and load it again later.

Saving Config File

The config file can be saved by clicking the Save Config file button on the Tests page. This opens up a dialog so that the user can select the destination folder, and give a meaningful name to the saved config file

Loading Config File

The config file can be loaded on the main page of the plugin by clicking the Load button. This opens up a dialog so that the user can select the config file to load. All the input fields will be populated according to the inputs in the file.WARNING: Do not try to modify the config file manually. To modify it, always load into the plugin, make the changes required, in the GUI, and save it again into the config file.

Troubleshooting

Dependency Warning

As mentioned in section 1, the plugin requires Matlab Compiler Runtime and XLSXWriter package for python. The warning messages for these are shown below. Warning is displayed only for the missing packages.
Figure 18 - Both MCR and XLSXWriter Missing

Figure 19 - MCR Missing

Figure 20 - XLSXWriter Missing
If you have installed MCR version 7.15 but still get the following warning message at the end of the test, you just need to restart DevWare.
Figure 21 - Matlab Warning

Start Dark Tests button not enabled

Check all the input fields for all the enabled tests. No input field should be empty.

Start Light Tests button not enabled

Check all the input fields for all the enabled tests. No input field should be empty.

Data is the same for DLS test and Structure noise test

If the selected ROI is the same, the DLS results are used as the Structure noise test results. The Structure noise test is not run separately. Verify the Structure Noise test ROI (in the DLS test tab).