After refer to VideoCore project, I‘m able to use Apple‘s VideoToolbox framework to encode in hardware.

1. Start an VTCompressionSession:

   // Create compression session
   err = VTCompressionSessionCreate(kCFAllocatorDefault,
                                frameW,
                                frameH,
                                kCMVideoCodecType_H264,
                                encoderSpecifications,
                                NULL,
                                NULL,
                                (VTCompressionOutputCallback)vtCallback,
                                self,
                                &compression_session);
   
   if(err == noErr) {
       comp_session = session;
   }
   

2. push the raw frame to the VTCompressionSession

   // Delegate method from the AVCaptureVideoDataOutputSampleBufferDelegate
   - (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {
       // get pixelbuffer out from samplebuffer
       CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
       //set up all the info for the frame
       // call VT to encode frame
       VTCompressionSessionEncodeFrame(compression_session, pixelBuffer, pts, dur, NULL, NULL, NULL);
       }
   

3. Get the encoded frame in the VTCallback, this is a C method to be used as a parameter of VTCompressionSessionCreate()

   void vtCallback(void *outputCallbackRefCon,
           void *sourceFrameRefCon,
           OSStatus status,
           VTEncodeInfoFlags infoFlags,
           CMSampleBufferRef sampleBuffer ) {
       // Do whatever you want with the sampleBuffer
       CMBlockBufferRef blockBuffer = CMSampleBufferGetDataBuffer(sampleBuffer);
   
   }