OpenGL ES (GPUImage)

Posted on | Edited on | Comments:

从GPUImage开始学习OpenGL ES

OpenGL ES

  1. iOS使用的OpenGL ES版本是2.0
  2. GLKit的大部分方法已经被废弃掉，推荐使用Metal

坐标系

屏幕坐标系

  • OpenGL ES 的屏幕坐标系

  • 设备屏幕坐标系

参考

🌟🌟🌟🌟🌟（讲解的非常详细）：OpenGL ES 2.0 (iOS)
🌟🌟🌟🌟🌟（总结很全）：OpenGL ES， 初学者的自我总结
🌟🌟🌟🌟🌟（主要是实际运用）：Lyman’s Blog
🌟🌟🌟🌟🌟（OpenGL ES 2.0 文档）：OpenGL ES Software Development Kit

GPUImage

GLProgram

主要负责着色器程序的编译链接。

GPUImageInput

定义输入协议。

GPUImageOutput

定义输出对象，下面都是其子类。

  • GPUImageVideoCamera (for live video from an iOS camera)
  • GPUImageStillCamera (for taking photos with the camera)
  • GPUImagePicture (for still images)
  • GPUImageMovie (for movies)

GPUImageContext

直接使用单例，管理GPUImage上下文，核心使用了EAGLContext。

GPUImageVideoCamera

availableVideoCVPixelFormatTypes

Format YUV Discussion
kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange ✅ Bi-Planar Component Y’CbCr 8-bit 4:2:0
video-range (luma=[16,235] chroma=[16,240]).
baseAddr points to a big-endian CVPlanarPixelBufferInfo_YCbCrBiPlanar struct.
kCVPixelFormatType_420YpCbCr8BiPlanarFullRange ✅ Bi-Planar Component Y’CbCr 8-bit 4:2:0
full-range (luma=[0,255] chroma=[1,255]).
baseAddr points to a big-endian CVPlanarPixelBufferInfo_YCbCrBiPlanar struct.
kCVPixelFormatType_32BGRA ❌ 32 bit BGRA.

设置yuvConversionProgram

runSynchronouslyOnVideoProcessingQueue，由于切换了线程，使用useImageProcessingContext来重新设置当前上下文。

GPUImageVideoCameraDelegate

原始帧回调，在这里获取的是系统的原始帧回调，优先于target处理。

kCVImageBufferYCbCrMatrixKey

Describes the color matrix used for YCbCr to RGB conversion

CVBufferGetAttachment获取颜色模式

Type Detail
kCVImageBufferYCbCrMatrix_ITU_R_601_4 The conversion matrix for standard digital television images (following the ITU R 601 standard).

YUV

GPUImage默认获取的格式就是YUV的。

RGBA

OpenGL ES

GLProgram

Function Detail
glCreateProgram 创建一个着色器程序对象
glCreateShader 创建一个空着色器对象
glShaderSource 替换一个着色器对象的源程序
glCompileShader 编译一个着色器对象
glGetShaderiv 1. GL_COMPILE_STATUS检查编译状态
2. GL_INFO_LOG_LENGTH获取编译信息Log的长度
3. GL_LINK_STATUS 检查链接状态
glGetShaderInfoLog 获取编译信息Log
glAttachShader 关联一个着色器对象到一个着色器程序对象上面
glLinkProgram 链接一个着色器程序对象
glDeleteShader 删除一个着色器对象
glUseProgram 安装一个着色器程序对象
glDeleteProgram 删除一个着色器程序对象
Function Detail
glActiveTexture 选择一个活跃的纹理单元GL_TEXTURE[i]
glBindAttribLocation 关联一个顶点属性到顶点变量
glEnableVertexAttribArray enable一个定点属性数组
glGetUniformLocation 返回唯一变量的位置
glDeleteProgram 删除一个着色器程序对象
glDeleteProgram 删除一个着色器程序对象
glDeleteProgram 删除一个着色器程序对象
glDeleteProgram 删除一个着色器程序对象
glDeleteProgram 删除一个着色器程序对象
glDeleteProgram 删除一个着色器程序对象

OpenGL Shading Language

OpenGL着色器语言。

There are four main types: float, int, bool and sampler.

  • For the first three types, vector types are available:
Type Detail
vec2, vec3, vec4 2D, 3D and 4D floating point vector
ivec2, ivec3, ivec4 2D, 3D and 4D integer vector
bvec2, bvec3, bvec4 2D, 3D and 4D boolean vectors
  • For floats here are also matrix types:
Type Detail
mat2, mat3, mat4 2x2, 3x3, 4x4 floating point matrix
  • Samplers are types representing textures. They are used for texture sampling.
Type Detail
sampler1D, sampler2D, sampler3D 1D, 2D and 3D texture
samplerCube Cube Map texture
sampler1Dshadow, sampler2Dshadow 1D and 2D depth-component texture

Sampler types have to be uniform. They are not allowed to be declared as a non-uniform type. Here are the different sampler types

  • There are three types of inputs and outputs in a shader: uniforms, attributes and varyings.
Type Detail
uniform Uniforms are values which do not change during a rendering, for example the light position or the light color. Uniforms are available in vertex and fragment shaders. Uniforms are read-only.
attribute Attributes are only available in vertex shader and they are input values which change every vertex, for example the vertex position or normals. Attributes are read-only.
varying Varyings are used for passing data from a vertex shader to a fragment shader. Varyings are (perspective correct) interpolated across the primitive. Varyings are read-only in fragment shader but are read- and writeable in vertex shader (but be careful, reading a varying type before writing to it will return an undefined value). If you want to use varyings you have to declare the same varying in your vertex shader and in your fragment shader.

All uniform, attribute and varying types HAVE to be global. You are not allowed to specify a uniform/attribute/varying type in a function or a void.

  • Shader output:
Type Detail
gl_Position 4D vector representing the final processed vertex position. Only available in vertex shader.
gl_FragColor 4D vector representing the final color which is written in the frame buffer. Only available in fragment shader.
gl_FragDepth float representing the depth which is written in the depth buffer. Only available in fragment shader.
  • There are also many built-in function which can (and should) be used:
Type Detail
dot a simple dot product
cross a simple cross product
texture2D used for sampling a texture
normalize normalize a vector
clamp clamping a vector to a minimum and a maximum

VertexShader

🌰 YUV => BGRA && BGRA => YUV

1
2
3
4
5
6
7
8
9
10
11
/// kGPUImageVertexShaderString
attribute vec4 position;
attribute vec4 inputTextureCoordinate;

varying vec2 textureCoordinate;

void main()
{
    gl_Position = position;
    textureCoordinate = inputTextureCoordinate.xy;
}

转换的过程中在定点着色器上没有进行任何操作，只是将纹理坐标专递给片元着色器使用。

FrameShader

🌰 YUV => BGRA

  • kCVPixelFormatType_420YpCbCr8BiPlanarFullRange => kCVPixelFormatType_32BGRA
1
2
3
4
5
GLfloat kColorConversion601FullRangeDefault[] = {
    1.0,    1.0,    1.0,
    0.0,    -0.343, 1.765,
    1.4,    -0.711, 0.0,
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
varying highp vec2 textureCoordinate;

uniform sampler2D luminanceTexture;
uniform sampler2D chrominanceTexture;
uniform mediump mat3 colorConversionMatrix;

void main()
{
    mediump vec3 yuv;
    lowp vec3 rgb;
    
    yuv.x = texture2D(luminanceTexture, textureCoordinate).r;
    yuv.yz = texture2D(chrominanceTexture, textureCoordinate).ra - vec2(0.5, 0.5);
    rgb = colorConversionMatrix * yuv;
    
    gl_FragColor = vec4(rgb, 1);
}
  • kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange => kCVPixelFormatType_32BGRA
1
2
3
4
5
GLfloat kColorConversion601Default[] = {
    1.164,  1.164, 1.164,
    0.0, -0.392, 2.017,
    1.596, -0.813,   0.0,
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
varying highp vec2 textureCoordinate;

uniform sampler2D luminanceTexture;
uniform sampler2D chrominanceTexture;
uniform mediump mat3 colorConversionMatrix;

void main()
{
    mediump vec3 yuv;
    lowp vec3 rgb;
    
    yuv.x = texture2D(luminanceTexture, textureCoordinate).r;
    yuv.yz = texture2D(chrominanceTexture, textureCoordinate).rg - vec2(0.5, 0.5);
    rgb = colorConversionMatrix * yuv;
    
    gl_FragColor = vec4(rgb, 1);
}

🌰 BGRA => YUV

参考文章：

LearnOpenGL CN
GLSL Qualifiers

未完待续。。。。