Introduction
Camera is key component of Android multimedia subsystem. It supports various use cases like preview, image capture, video recording .This blog descries the Android camera architecture overview and various components in the user and kernel space modules.
Android Camera Architecture
Component Description:
Camera Application: Android Camera Default application in /packages/apps/camera
Camera JNI: This is the JNI interface for the Camera Application .It converts Java function Calls into native calls
File name: android_hardware_Camera.cpp
Camera Interface: The camera interface (camera.cpp) is proxy for remote object in camera service layer. It is the middle level interface between JNI and Camera Binder.
File name:camera.cpp
Camera Binder Interface: ICamera is the Binder interface for IPC call and exposes the API’s for Camera Interface. It binds with the Camera Service Layer for the Communication.
filename: Icamera.cpp
Camera service:
As part of system boot the camera service is started by the media server. It act as the mediator between Application and Hardware Layer to change the context. Application gets the Camera service handle and starts communicating. The camera service transforms the Application commands to the Hardware and sends the reply back. It interacts with Surface Flinger for delivering preview frames on display
From main_mediaserver.cpp , the camera service instantiated CameraService::instantiate()
As part of system boot the camera service is started by the media server. It act as the mediator between Application and Hardware Layer to change the context. Application gets the Camera service handle and starts communicating. The camera service transforms the Application commands to the Hardware and sends the reply back. It interacts with Surface Flinger for delivering preview frames on display
From main_mediaserver.cpp , the camera service instantiated CameraService::instantiate()
Camera HAL in Android system to interact with the Hardware Layer. It is required because not all the hardware Driver exposes the standard API’s for interaction. This makes easy porting with different Hardware this is the first level HAL implementation.
Boot Sequence:Enable the preview, and then it starts the preview. it registers new preview buffers for Display. Based on the Parameter and format the Surface layer creates a buffer source for this surface
Service Manager:
System server starts Service manager before starts the media server .It exposes service manager interfaces to other process. It must be started before any other service gets running. The interfaces are exposed via IServicemanager.
It open “/dev/binder” driver and then call BINDER_SET_CONTEXT_MGR ioctl to let
Binder kernel driver know it acts as a manager. Then it enters into a loop to wait for any data from
other process
File name: /frameworks/base/media/mediaserver/main_mediaserver.cpp
int main(int argc, char **argv)
{
struct binder_state *bs;
void *svcmgr = BINDER_SERVICE_MANAGER;
bs = binder_open(128*1024);
if (binder_become_context_manager(bs)) {
LOGE("cannot become context manager (%s)\n", strerror(errno));
return -1;
}
svcmgr_handle = svcmgr;
binder_loop(bs, svcmgr_handler);
return 0;
}
Media Server:
As part of Boot sequence the system starts the Media server process .It runs continuously in the background .Media Service starts Service Manager Module and initiates the basic service routines.
1. Media Player Service
2. Camera Service
3. Audio Flinger Service
4. Audio Policy Service
File name: /frameworks/base/media/mediaserver/main_mediaserver.cpp
int main(int argc, char** argv)
{
sp<ProcessState> proc(ProcessState::self());
sp<IServiceManager> sm = defaultServiceManager();
AudioFlinger::instantiate();
MediaPlayerService::instantiate();
CameraService::instantiate();
AudioPolicyService::instantiate();
}
IService manager:
File name: frameworks\base\camera\libcameraservice/IServiceManager.cpp
sp<IServiceManager> defaultServiceManager()
{
if (gDefaultServiceManager != NULL) return gDefaultServiceManager;
{
AutoMutex _l(gDefaultServiceManagerLock);
if (gDefaultServiceManager == NULL) {
gDefaultServiceManager = interface_cast<IServiceManager>(
ProcessState::self()->getContextObject(NULL));
}
}
return gDefaultServiceManager;
}
The first time gDefaultServiceManager is NULL, so it will first get a
Camera Service:
Singleton camera server, it handles multiple client arbitration, provides interface and control of camera hardware, provides interface to system. It attaches the camera service with the service manager. Other modules get access the Camera Service via service manager.
File name: frameworks\base\camera\libcameraservice/cameraservice.cpp
void CameraService::instantiate() {
defaultServiceManager()->addService(
String16("media.camera"), new CameraService());
}
IServiceManager::addService, which is a RPC call . The parameter passed to addService is the address of new Camera Service instance. It invokes BINDER_WRITE_READ to write the packet to binder kernel driver and adds the service with the service manager.
Camera Init Sequence:
Camera Application:
When user invokes Camera Application, Open() calls corresponding Camera JNI to create the instance of the Camera
Camera JNI and Camera Interface:
It establishes binder interface to camera service and calls Camera Connect to create new Client for the session
File name: camera.cpp (frameworks\base\libs\camera)
sp<Camera> Camera::connect()
{
sp<Camera> c = new Camera();
const sp<ICameraService>& cs = getCameraService();
if (cs != 0) {
c->mCamera = cs->connect(c);
}
}
const sp<ICameraService>& Camera::getCameraService()
{
Mutex::Autolock _l(mLock);
if (mCameraService.get() == 0) {
sp<IServiceManager> sm = defaultServiceManager();
sp<IBinder> binder;
do {
binder = sm->getService(String16("media.camera"));
}
Camera Service:
It invokes the new Client for the session. The Client calls HAL for creating Hardware instance then sets the callback functions for Data and Event Callback
File name: cameraservice.cpp (frameworks\base\camera\libcameraservice)
sp<ICamera> CameraService::connect(const sp<ICameraClient>& cameraClient)
{ // create a new Client object
client = new Client(this, cameraClient, callingPid);
mClient = client;
}
File name: cameraservice.cpp (frameworks\base\camera\libcameraservice)
CameraService::Client::Client(const sp<CameraService>& cameraService,
const sp<ICameraClient>& cameraClient, pid_t clientPid)
{
mHardware = openCameraHardware();
mUseOverlay = mHardware->useOverlay();
mHardware->setCallbacks(notifyCallback,
dataCallback,
dataCallbackTimestamp,
mCameraService.get());
// Enable zoom, error, and focus messages by default
mHardware->enableMsgType(CAMERA_MSG_ERROR |
CAMERA_MSG_ZOOM |
CAMERA_MSG_FOCUS);
}
Camera Hardware Interface (Android HAL):
The following operations are performed in the generic HAL Layer
- create the instance of hardware instance
- Start the process of initializing the hardware
- Set Actual preview buffer Count
- Checks whether it supports overlay
- Sets the call back to receive data and events
- Init the default camera Parameters such as Frame rate, width, height, zoom ,contrast ,etc…
Preview Sequence:
Application:
To enable the preview for the camera Application needs to pass the surface to the camera Service. Also application can get the camera parameter and set the camera parameter before actual preview start.
Application makes the following call sequence.
A. setPreviewDisplay ()
B. Getparameter()
C. Setparameter ()
D. startPreview()
Camera JNI/Interface:
Get the Surface from the application and pass it to the Camera Service
File Name: Camera.cpp (frameworks\base\libs\camera)
status_t Camera::setPreviewDisplay(const sp<ISurface>& surface)
{
if (surface == 0) {
LOGD("app passed NULL surface");
}
sp <ICamera> c = mCamera;
if (c == 0) return NO_INIT;
return c->setPreviewDisplay(surface);
}
Get Preview/Capture Parameters from HAL Layer. Application can evaluate or use these parameter for Display. It calls Setparameter to set the specific preview/capture param
File Name: Camera.cpp (frameworks\base\libs\camera)
String8 Camera::getParameters() const
{
LOGV("getParameters");
String8 params;
sp <ICamera> c = mCamera;
if (c != 0) params = mCamera->getParameters();
return params;
}
Set preview/Capture parameters.
File Name: Camera.cpp (frameworks\base\libs\camera)
status_t Camera::setParameters(const String8& params)
{
sp <ICamera> c = mCamera;
if (c == 0) return NO_INIT;
return c->setParameters(params);
}
Start preview mode
File Name: Camera.cpp (frameworks\base\libs\camera)
// start preview mode
status_t Camera::startPreview()
{
LOGV("startPreview");
sp <ICamera> c = mCamera;
if (c == 0) return NO_INIT;
return c->startPreview();
}
Camera Service:
Camera Service receives Set preview Display request from the Camera Interface layer. On receiving the request, camera service client clears old surface and buffers if it is set.If preview already started it registers the buffer
File Name: CameraService.cpp (frameworks\base\camera\libcameraservice)
status_t CameraService::Client::setPreviewDisplay(const sp<ISurface>& surface)
{
if (mSurface != 0) {
if ( !mUseOverlay)
{
mSurface->unregisterBuffers();
}
}
mSurface = surface; /*stores the Surface */
if (mHardware->previewEnabled()) {
if (mUseOverlay) {
result = setOverlay();
} else if (mSurface != 0) {
result = registerPreviewBuffers();
}
}
}
status_t CameraService::Client::registerPreviewBuffers()
{
int w, h;
CameraParameters params(mHardware->getParameters());
params.getPreviewSize(&w, &h);
ISurface::BufferHeap buffers(w, h, w, h,
HAL_PIXEL_FORMAT_YCrCb_420_SP,
mOrientation,
0,
mHardware->getPreviewHeap());
status_t ret = mSurface->registerBuffers(buffers);
}
Passing preview buffer:
It receives the buffer from the HAL and post the buffer to the surface
File Name: CameraService.cpp (frameworks\base\camera\libcameraservice)
void CameraService::Client::dataCallback(int32_t msgType, const sp<IMemory>& dataPtr, void* user)
{
switch (msgType) {
case CAMERA_MSG_PREVIEW_FRAME:
client->handlePreviewData(dataPtr);
break;
}
void CameraService::Client::handlePreviewData(const sp<IMemory>& mem)
{
if (!mUseOverlay)
{ Mutex::Autolock surfaceLock(mSurfaceLock);
if (mSurface != NULL) {
/*post the buffer for preview */
mSurface->postBuffer(offset);
}
}
}
The following sequence of actions are executed the Android HAL from the Camera Client.
1. Create Physical continuous memory for preview Heap
2. Send the command to the kernel for preview start
3. Receives the Frames.
Capture Picture:
Application:
The following functions are called to capture the Image from application
1. Initiate AutoFocus()
2. Initiate takepicture ()
3. Save the picture
Camera JNI/Interface:
Camera Interface routes the Auto focus call to the camera Service .Once Auto focus is completed Take picture is called .Data Callback is notifies the application back whenever frame is
available.Notifiy callback is notifies whenever there is a event or msg to the application
Data Callback Messages:
CAMERA_MSG_PREVIEW_FRAME
CAMERA_MSG_POSTVIEW_FRAME
CAMERA_MSG_RAW_IMAGE
CAMERA_MSG_COMPRESSED_IMAGE
Notify Events:
Notify Shutter Event
CAMERA_MSG_ERROR
/* Auto Focus .Call camera Service through Binder */
status_t Camera::autoFocus()
{
sp <ICamera> c = mCamera;
if (c == 0) return NO_INIT;
return c->autoFocus();
}
/* Take Picture .Call camera Service through Binder */
status_t Camera::takePicture()
{
sp <ICamera> c = mCamera;
if (c == 0) return NO_INIT;
return c->takePicture();
}
/* callback from camera service when frame or image is ready*/
void Camera::dataCallback(int32_t msgType, const sp<IMemory>& dataPtr)
{
sp<CameraListener> listener;
{
Mutex::Autolock _l(mLock);
listener = mListener;
}
if (listener != NULL) {
listener->postData(msgType, dataPtr);
}
}
/* callback from camera service*/
void Camera::notifyCallback(int32_t msgType, int32_t ext1, int32_t ext2)
{
sp<CameraListener> listener;
{
Mutex::Autolock _l(mLock);
listener = mListener;
}
if (listener != NULL) {
listener->notify(msgType, ext1, ext2);
}
}
Camera Service:
/*Auto Focus */
status_t CameraService::Client::autoFocus()
{
return mHardware->autoFocus();
}
/*Take Picutre .Frame is returned to the Data Callback */
status_t CameraService::Client::takePicture()
{
mHardware->enableMsgType(CAMERA_MSG_SHUTTER |
CAMERA_MSG_POSTVIEW_FRAME |
CAMERA_MSG_RAW_IMAGE |
CAMERA_MSG_COMPRESSED_IMAGE);
return mHardware->takePicture();
}
/* callback function from HAL to handle preview and snapshot frames*/
void CameraService::Client::dataCallback(int32_t msgType, const sp<IMemory>& dataPtr, void* user)
{
switch (msgType) {
case CAMERA_MSG_PREVIEW_FRAME:
client->handlePreviewData(dataPtr);
break;
case CAMERA_MSG_POSTVIEW_FRAME:
client->handlePostview(dataPtr);
break;
case CAMERA_MSG_RAW_IMAGE:
client->handleRawPicture(dataPtr);
break;
case CAMERA_MSG_COMPRESSED_IMAGE:
client->handleCompressedPicture(dataPtr);
break;
default:
if (c != NULL) {
c->dataCallback(msgType, dataPtr);
}
break;
}
void CameraService::Client::handleShutter( image_rect_type *size, bool
playShutterSoundOnly)
/* Notify callback for Shutter*/
if(playShutterSoundOnly) {
/*For playing Sound*/
c->notifyCallback(CAMERA_MSG_SHUTTER, 0, 0);
return;
}
/*Unregister the Buffer here*/
if (mSurface != 0 && !mUseOverlay) {
mSurface->unregisterBuffers();
}
Android HAL:
Take Picture:
1. Stop the preview
2. Set the picture param, creates and initializes memory for raw, compressed format and
thumbnail.
3. start snapshot process and receive snapshot from kernel
4 Notify to release the preview frame from Surface Flinger.
5. Pass the Picture to the Camera Service->Application.
Record Video:
Application:
When user changes from Preview mode to Record mode, video recorder is called to perform the recording operation. The following sequence of operations are performed in the camera Application.
File name:CameraApp.java,VideoRecorder.java ,mediarecorder.java
1. Change Preview to Record mode- MSG_CHANGE_VIDEO_PREVIEW
A. SetCameraParameters.
B. Disable auto Focus.
C. Restart the preview.
2. Start record - MSG_START_REC
A. Start the Recording Thread.
B. Open the Video File, update the Video Recorder size and Max Duration.
C. Init the Video Recorder (Creates Media Recorder)
D. Set the Camera Handle in the media recorder.
E. Set the Audio and Video Source.
F. Set Recording Output Format
G. Set setVideoSize, setVideoEncodingBitRate and setVideoFrameRate.
H. Set setAudioEncoder, Video Encoder and Preview surface.
I. Issue Media Recorder Prepare listener and Start the Video Recorder.
J. Set Info and Error Listener.
K. Start the Media Recorder.
Recorder Interface/JNI:
The media recorder application interacts with media Recorder JNI to interface with the native media recording layer. The JNI Layer interacts with media recorder interface for routing the call to Service layer.
Create Media Recorder:
Get the media player service handle from the service manager, call to create media Recorder.
Filename: Mediarecorder.cpp (frameworks\base\media\libmedia)
MediaRecorder::MediaRecorder()
{
/* Get the mediaplayer service */
const sp<IMediaPlayerService>& service(getMediaPlayerService());
if (service != NULL) {
mMediaRecorder = service->createMediaRecorder(getpid());
}
if (mMediaRecorder != NULL) {
mCurrentState = MEDIA_RECORDER_IDLE;
}
doCleanUp();
}
Set camera and Set preview surface:
Get the camera object handle and pass it to the native layer. Set the preview surface to view preview
Filename: Mediarecorder.cpp (frameworks\base\media\libmedia)
status_t MediaRecorder::setCamera(const sp<ICamera>& camera)
{
mMediaRecorder->setCamera(camera);
}
status_t MediaRecorder::setPreviewSurface(const sp<Surface>& surface)
{
mMediaRecorder->setPreviewSurface(surface->getISurface());
}
Audio, Video Source:
Set the Audio, video Source to construct media engine .Media recorder is initialized if it is not initialized before Audio/Video Set Source.
Audio –MIC, Video -CAMERA
Filename: Mediarecorder.cpp (frameworks\base\media\libmedia)
status_t MediaRecorder::setAudioSource(int as)
{
/*Call init if media recorder is in init state*/
if (mCurrentState & MEDIA_RECORDER_IDLE) ret = init();
mMediaRecorder->setAudioSource(as)
}
status_t MediaRecorder::setVideoSource(int vs)
{
/*Call init if media recorder is in init state*/
if (mCurrentState & MEDIA_RECORDER_IDLE) ret = init();
mMediaRecorder->setVideoSource(vs)
}
status_t MediaRecorder::init()
{
mMediaRecorder->init()
mCurrentState = MEDIA_RECORDER_INITIALIZED;
}
Different Audio source values specified in the Audio Source class
public static final int DEFAULT = 0;
/** Microphone audio source */
public static final int MIC = 1;
/** Voice call uplink (Tx) audio source */
public static final int VOICE_UPLINK = 2;
/** Voice call downlink (Rx) audio source */
public static final int VOICE_DOWNLINK = 3;
/** Voice call uplink + downlink audio source */
public static final int VOICE_CALL = 4;
/** Microphone audio source with same orientation as camera if available, the main
* device microphone otherwise */
public static final int CAMCORDER = 5;
/** Microphone audio source tuned for voice recognition if available, behaves like
* {@link #DEFAULT} otherwise. */
public static final int VOICE_RECOGNITION = 6;
Camera Source Values
public static final int DEFAULT = 0;
/** Camera video source */
public static final int CAMERA = 1;
Output format type, A/V encoder:
Set the output file format, Audio, video encoder to create media engine
The different file output formats are defined
private OutputFormat() {}
public static final int DEFAULT = 0;
/** 3GPP media file format*/
public static final int THREE_GPP = 1;
/** MPEG4 media file format*/
public static final int MPEG_4 = 2;
/** The following formats are audio only .aac or .amr formats **/
/*Raw AMR_NB */
public static final int RAW_AMR = 3;
/** @hide AMR NB
public static final int AMR_NB = 3;
/** @hide AMR WB file format */
public static final int AMR_WB = 4;
/** @hide AAC ADIF file format */
public static final int AAC_ADIF = 5;
/** @hide AAC ADTS file format */
public static final int AAC_ADTS = 6;
/** QCP file format */
public static final int QCP = 7;
/** 3GPP2 media file format*/
public static final int THREE_GPP2 = 8;
Audio encoder types
public static final int DEFAULT = 0;
/** AMR (Narrowband) audio codec */
public static final int AMR_NB = 1;
/** AMR (Wideband) audio codec */
public static final int AMR_WB = 2;
/** AAC audio codec */
public static final int AAC = 3;
/** enhanced AAC audio codec */
public static final int AAC_PLUS = 4;
/** enhanced AAC plus audio codec */
public static final int EAAC_PLUS = 5;
/** EVRC audio codec */
public static final int EVRC = 6;
/** QCELP audio codec */
public static final int QCELP =7;
Video Encoder types
public static final int DEFAULT = 0;
public static final int H263 = 1;
public static final int H264 = 2;
public static final int MPEG_4_SP = 3;
Filename: Mediarecorder.cpp (frameworks\base\media\libmedia)
status_t MediaRecorder::setOutputFormat(int of)
{
mMediaRecorder->setOutputFormat(of);
mCurrentState = MEDIA_RECORDER_DATASOURCE_CONFIGURED;
{
status_t MediaRecorder::setVideoEncoder(int ve)
{
mMediaRecorder->setVideoEncoder(ve)
}
status_t MediaRecorder::setAudioEncoder(int ae)
{
mMediaRecorder->setAudioEncoder(ae)
}
Media Recorder prepare and start:
Issue media recorder prepare then start media recorder command
Filename: Mediarecorder.cpp (frameworks\base\media\libmedia)
status_t MediaRecorder::prepare()
{
mMediaRecorder->prepare();
mCurrentState = MEDIA_RECORDER_PREPARED;
}
status_t MediaRecorder::start()
{
mMediaRecorder->start();
mCurrentState = MEDIA_RECORDER_RECORDING;
}
Media Player Service:
Media player service creates media recorder client for media recorder interaction. Client opens the instance of the stagefright recorder
Filename: MediaPlayerService.cpp (frameworks\base\media\libmediaplayerservice)
sp<IMediaRecorder> MediaPlayerService::createMediaRecorder(pid_t pid)
{
sp<MediaRecorderClient> recorder = new MediaRecorderClient(this, pid);
wp<MediaRecorderClient> w = recorder;
Mutex::Autolock lock(mLock);
mMediaRecorderClients.add(w);
return recorder;
}
FileName: frameworks\base\media/libmediaplayerservice/MediaRecorderClient.cpp
MediaRecorderClient::MediaRecorderClient(const sp<MediaPlayerService>& service, pid_t pid)
{
ALOGV("Client constructor");
mPid = pid;
mRecorder = new StagefrightRecorder;
mMediaPlayerService = service
}
It routes all the application calls (Set camera, Setpreviewsurface, Set A/V Encoder, output format, prepare and start) to the Mediarecorder for enabling/running the media graph for the Recorder.
Android Camera HAL Interface:
Thank you very much for sharing Android Camera implementation. It was very useful for me.
ReplyDeleteReally it is very good , very good explanation , if needed can you please explore it more in different aways like versions of Android
ReplyDelete