ColorPipe User Guide
- John Medalen
- John Medalen
Introduction
The ApBase library includes a software color processing pipeline. The color pipe is used in DevWare and other Aptina applications, and is available to 3rd party applications through the ApBase function ap_ColorPipe(). This document lists the processing stages, and how to configure and control them.
Table of Contents
Block Diagram
Block Descriptions
DevWare Plug-in GrabFrame() Hook
Insert JPEG Header
Descramble
Deinterlace
YCbCr Column-wise Stereo Separation
Array Image Repacking
Bayer Decompression
DevWare Plug-in RawImageData() Hook
Multiple Exposure Combining
DevWare Plug-in LinearImageData() Hook
Normalize Bayer
Adaptive Local Tone Mapping
Clean JPEG
Optical Black
Auto-Exposure
Software Gain
Noise Removal
Lens Correction
Defect Correction
Local Green Balance
Noise Measurement
DevWare Plug-in ImageData() Hook
Auto-White Balance
Convert to RGB-48
Demosaic
Aperture Correction
Color Correction Matrix
Gamma Correction
Convert Original Pixel Type to RGB
ColorPipe Filter Plug-in
Digital Video Stabilization
Software Binning
Stereo Merge
Select Color Channel
Reorient
Noise Measurement
DevWare Plug-in ImageData() Hook
Auto-Focus
Calculate Focus Number
Embed Watch Data
Embed Overlays
Processing Images without a Device Connected
Block Diagram
The diagram below shows the image data path and each processing stage. The locations of DevWare plug-in hooks are also shown.
Block Descriptions
Below are descriptions of the blocks and which STATE variables control each one. See also the STATE Variables document.
DevWare Plug-in GrabFrame() Hook
Remarks
A DevWare plug-in implementing the GrabFrame() hook can process the image here.
Insert JPEG Header
Remarks
If this is JPEG data and there is no header, the header is added here. In ApBase, this is done in the ap_GrabFrame() call.
Descramble
Remarks
For AR1011, AR1411 or similar sensors, raw data, rearrange the pixels into the proper order and interpolate over special pixels.
STATE Variables
Descramble Mode | Enable this block. |
Special Pixel Mode | Enable interpolating over special pixels. |
Deinterlace
Remarks
Interleave the scan lines of a two-field interlaced image. Only relevant to SOC or ISP output.
STATE Variables
Deinterlace Mode | Enable and select the row order. |
YCbCr Column-wise Stereo Separation
Remarks
For YCbCr column-wise stereo images (alternate pixels left, right), separate the stereo images.
STATE Variables
Stereo Colwise | Enable and select the image order. |
Array Image Repacking
Remarks
For data from and array imager, arrange the data into a grid of images.
STATE Variables
AI Repack | Enable. |
Bayer Decompression
Remarks
Decompress or linearize (e.g. decompand companded) imager data if needed.
STATE Variables
Compress | Select the compression type, SMIA DPCM, SMIA packing, a-Law, companding. |
BitDepth Decomp | Pixel bit depth after decompression. |
DevWare Plug-in RawImageData() Hook
Remarks
A DevWare plug-in implementing the RawImageData() hook can process the image here. The image format can't be changed.
Multiple Exposure Combining
Remarks
If the image consists of multiple separate exposures, then this block can combine the exposures into a single, linear HDR image.
STATE Variables
Linearize HDR | Enable combining multiple exposures. |
Num Exposures HDR | Number of exposures, 1 ... 4 |
| Ratio T1_T2 | The intended exposure ratio between T1 and T2, as an integer. Usually a power of 2. |
| Ratio T2_T3 | The intended exposure ratio between T2 and T3, as an integer. Usually a power of 2. |
| Ratio T3_T4 | The intended exposure ratio between T3 and T4, as an integer. Usually a power of 2. |
| Actual Ratio T1_T2 | The actual exposure ratio between T1 and T2, as a floating point string. |
| Actual Ratio T2_T3 | The actual exposure ratio between T2 and T3, as a floating point string. |
| Actual Ratio T3_T4 | The actual exposure ratio between T3 and T4, as a floating point string. |
DevWare Plug-in LinearImageData() Hook
Remarks
A DevWare plug-in implementing the LinearImageData() hook can process the image here. If there was no multiple exposure combining, then this data is the same as the RawImageData() hook. The image format can't be changed.
Normalize Bayer
Remarks
Subtract black pedestal and rescale pixel data to 0…4095. (If raw data is less then 12 bits, then scale up to 12 bits. If raw data is more than 12 bits, then just use 12 of the bits and discard the rest, usually the 12 MSBs.) For all subsequent blocks, Bayer data is 12 bits with no pedestal.
STATE Variables
True Black Level | Data pedestal, proportional to True Black Scale (not to the image pixel depth). |
True Black Scale | Indicates the scale of True Black Level. For example 1024 indicates 10-bit black level value. |
| Dynamic Range MSB | Indicate the MSB of the raw data. Set to 31 for default processing. |
Adaptive Local Tone Mapping
Remarks
Use a sophisticated algorithm to reduce a deep HDR image (up to 20 bits per pixel) to 12 bits while preserving color bringing out dark areas. Subtract black pedestal. Pixel data range after this block is always 0…4095. For all subsequent blocks, Bayer data is 12 bits with no pedestal.
STATE Variables
ALTM Enable | Enable ALTM processing. |
Clean JPEG
Remarks
Remove CCP sync codes from JPEG data.
STATE Variables
Strip FSP | Enable. |
Optical Black
Remarks
Analyze dark rows and dark columns and correct the image rows and columns.
STATE Variables
Optical Black | Optical black processing options. |
Optical Black Row Filter | Enable low-pass filtering in the vertical direction. |
Optical Black 1 Row Start | Define optical black rows in the image. |
Optical Black 1 Column Start | Define optical black columns in the image. |
Auto-Exposure
Remarks
Analyze image brightness and adjust sensor exposure and gain. (No change to the image.)
STATE Variables
Auto Exposure | Enable AE. |
Auto Exposure Target | Target image brightness, percentage. |
Auto Exposure Stability | Allowed deviation from target. |
Auto Exposure Speed | Convergence speed |
Auto Exposure Minimum FPS | Exposure limit |
Auto Exposure Flicker Filter | Whether to limit exposure time to multiples of a flicker frequency. |
Auto Exposure Soft Limit | Keep exposure below this limit unless gain exceeds soft limit. |
Auto Exposure Soft Gain Limit | Keep gain below this limit if possible. |
Auto Exposure Gain Limit | Maximum gain. |
Auto Exposure Global Gain Limit | Maximum global gain, for sensors with global gain. |
Auto Exposure Software Gain Limit | Maximum software gain. |
Auto Exposure Freeze Gains | If set, don't adjust gain. |
Auto Exposure Fade Saturation | If set, reduce saturation in low light. |
Auto Exposure Fade Aperture | If set, reduce sharpening in low light. |
Auto Exposure Fade Target | If set, reduce target level in low light. |
Software Gain
Remarks
Multiply all pixels by a gain value.
STATE Variables
Software Gain | Gain value ×1000. |
Noise Removal
Remarks
Run the various noise removal algorithms. Each algorithm can be enabled independently. The algorithms run in this order: Temporal, High/Low, Flat Region Filter, Adapting Average-Near, Clarity+.
STATE Variables
Noise Removal | Select which algorithm(s) to use. |
Noise Removal Level | Aggressiveness of the noise removal. |
Noise Removal Depth | Depth for temporal algorithm. |
Lens Correction
Remarks
Apply position-dependent gain to correct lens shading. This block can also do calibration. Calibrating lens correction hardware should be done with the LensCalib tool or lenscal.dll library.
STATE Variables
Lens Correction Enable | Apply lens correction. |
Lens Correction Falloff | Allowed shading falloff for calibration. |
Lens Correction Lens Radius | Radius in pixels of the area to calibrate. |
Detect Lens Shading Req. | Do a software LC calibration. |
Lens Curve Red | Lens shading curvatures from the calibration. |
Defect Correction
Remarks
Correct single-pixel defects.
STATE Variables
Defect Enable | Enable correction. |
Defect Max | Size of defect buffer. |
Defect Auto Defect Correction | Enable periodic recalibration. |
Defect Detect Dirty | Trigger a new defect detection. |
Defect Count | Number of defects detected. |
Local Green Balance
Remarks
Correct localized imbalances between Gr and Gb signals in Bayer images.
STATE Variables
Green Balance Enable | Enable green correction. |
Green Balance Apos | Parameters for the algorithm. |
Noise Measurement
Remarks
The Bayer noise measurement happens here. (No change to the image.)
STATE Variables
Noise Image Type | Raw or RGB. |
Noise Frames | Number of frames to analyze. |
Begin Noise Measurement | Trigger noise measurements to begin. |
Noise Measurement Results | Text of the results. |
DevWare Plug-in ImageData() Hook
Remarks
A DevWare plug-in implementing the ImageData() callback can process data here.
Auto-White Balance
Remarks
Analyze the image and adjust the color correction matrix and possibly red and blue gains. (No change to the image.) Use the SensorTune tool to generate the AWB parameters.
STATE Variables
White Balance | Enable auto or manual white balance. |
WB Speed | Convergence speed. |
WB Adjust Gains | Whether to adjust gains on the device. |
WB Manual Position | Adjustment position, 0 to 100 |
WB Manual RedGreen | Red gain applied to CCM |
WB Manual BlueGreen | Blue gain applied to CCM |
WB Interpolate Saturation | Apply saturation to CCM |
WB Normalize Matrix | Normalize CCM |
Relative Red Gain | Current red gain relative to green. |
Relative Blue Gain | Current blue gain relative to green. |
Convert to RGB-48
Remarks
Conversion of the image to deep RGB is here if needed. The image is only saved to a file, not displayed, and no further processing is done to deep RGB images.
Demosaic
Remarks
For Bayer data, run the selected demosaic algorithm (followed by aperture correction, color correction and gamma correction).
A Filter Plug-in can substitute its processing for the Demosaic, Aperture Correction, CCM and Gamma Correction stages (the Bayer to RGB conversion).
A 5x5 non-linear matrix is used for conversion.
STATE Variables
Filter | Select the demosaic algorithm. |
Aperture Correction
Remarks
Sharpening for Bayer images.
STATE Variables
Aperture Enable | Enable aperture correction. |
Aperture | Degree of sharpening. |
Color Correction Matrix
Remarks
Multiply the interpolated RGB values by a 3x3 matrix. This mainly corrects for crosstalk.
STATE Variables
Color Correction | Apply the CCM. |
Gamma Correction
Remarks
Gamma table lookup on RGB data.
STATE Variables
Gamma Correction | The exponent of the correction curve to generate a gamma table. |
Black Correct | Flatten the bottom of the curve of the generated gamma table. |
Contrast | 'S' curve applied to gamma curve. |
| Gamma Table Reset | Reset custom gamma table. |
| Gamma Table | Add or change an entry in the custom gamma table. |
| Gamma Table Select | Select the custom gamma table or the generated table. |
Convert Original Pixel Type to RGB
Remarks
For non-Bayer data (YCbCr, JPEG, etc.), convert the pixels to RGB.
STATE Variables
Filter | Select the subroutine to convert the image from the camera pixel format to RGB. |
ColorPipe Filter Plug-in
Remarks
A DLL implementing the colorpipe Filter plug-in API can take over conversion of the raw data to RGB-32. This can be used for custom Demosaic, or other type of processing.
STATE Variables
Filter | Select the subroutine to convert the image from the camera pixel format to RGB. Plug-in filters will be among the choices. |
Digital Video Stabilization
Remarks
Perform the host-side portion of the digital video stabilization algorithm (shift the image). This relies on the hardware ISP to provide the shift amount.
STATE Variables
DVS Split Screen | Split screen demo. |
Software Binning
Remarks
Reduce the image size by an integer factor.
STATE Variables
Bin XBin Y | Bin factors. |
Stereo Merge
Remarks
Merge left and right images of a stereo pair into a single image.
STATE Variables
Stereo Merge | Merging mode. |
Stereo Shift X Stereo Shift Y | Alignment correction. |
Select Color Channel
Remarks
Zero out all but one color channel. For diagnostics.
STATE Variables
Output Channel | Select a single color channel. |
Output BwColor | Show in grayscale or color. |
Reorient
Remarks
Horizontally mirror and/or vertically flip the image. This can be used to correct an image file that was not captured in the upright orientation. On a live sensor you would normally set the sensor's flip and mirror controls.
STATE Variables
Orientation | Enable flip and/or mirror. |
Noise Measurement
Remarks
The RGB domain noise measurement is performed here. (No change to the image.)
STATE Variables
Noise Image Type | Raw or RGB.. |
Noise Frames | Number of frames to analyze. |
Begin Noise Measurement | Trigger noise measurements to begin. |
Noise Measurement Results | Text of the results. |
DevWare Plug-in ImageData() Hook
Remarks
DevWare plug-ins that implement the ImageData() hook can process the RGB image here.
Auto-Focus
Remarks
Analyze image sharpness and adjust lens position. (No change to image.)
STATE Variables
Auto Focus | Enable automatic refocus trigger. |
Auto Focus Window Size | Size of AF window as a percent of the whole image. |
Focuser Position | Current position. |
Calculate Focus Number
Remarks
Calculate a focus metric that can be used for manual focus adjustment. Optionally highlight edges in the image.
STATE Variables
Calculate Focus | Enable focus metric calculation. |
Focus | Current focus metric value. |
Embed Watch Data
Remarks
Embed the text of the Watch window data (register values being monitored) into the image.
Embed Overlays
Remarks
Several kinds of overlay indicators or text can be enabled. The text or graphics are embedded in the image data.
STATE Variables
Exclusion Overlay | Exclusion window for AP0101 and similar. |
AWB Zones Enabled | AWB zone overlay for many SOCs. |
WB Zone Overlay | Software AWB zone overlay for old software algorithm. |
Auto Exposure Overlay | AE zone overlay for many SOCs. |
AE350 Zones | AE zone overlay for MT9V022 and similar. |
Array Imager Color Overlay | Color-code array imager focal planes. |
Show Defects | Defect overlay for MT9M111. |
Lens Correction Overlay | Show knee points for old SOCs with piecewise quadradic lens correction blocks. |
Lens Center Overlay | Show lens correction center. |
Auto Focus Overlay | Auto-focus window for many SOCs. |
MAE Overlay | Motion Auto Exposure overlay for MT9T111 and similar. |
CRA Overlay | Bullseye overlay. |
Processing Images without a Device Connected
There are a couple of ways of processing an image without an actual device being attached;
- If you have a raw image file you want to run through the colorpipe:
ap_CreateFromImageFile()
ap_GrabFrame()
ap_ColorPipe() - If you have image data in memory imitating the output of a sensor:
ap_CreateVirtual(path_to_sensor_xsdat_file)
ap_SetImageFormat()
ap_ColorPipe()