作者：禮龍

声明：转发本文，请联系作者授权

Occlusion between people and rendered content

How It Works 它是如何工作的

Machine Learning

ARFrame

Composition 构成

RealityKit

• 提供了新的ARView
• 简单地在AR中使用超级现实主义的API
• 内置支持People Occlusion

override func viewDidLoad() {
super.viewDidLoad()
// Check If Supported
guard let config = arView.session.config as? ARWorldTrackingConfiguration,
ARWorldTrackingConfiguration.supportsFrameSemantics(.personSegmentationWithDepth) else {
return
}
 // Enable Frame Semantics
 config.frameSemantics = .personSegmentationWithDepth
 }

Update Your Configuration 更新你的配置

enum ARFrameSemantics {
 	case PersonSegmentation // Only People, No Depth
 	case PersonSegmentationWithDepth
}
class ARWorldTrackingConfiguration: ARConfiguration { 
	var frameSemantics: ARFrameSemantics { get set }
}

ARView

• 新的应用推荐方式
• 深度渲染整合
• 处理透明对象
• 旨在实现最佳性能

SceneKit

• 添加ARSCNView支持
• 仅开启框架语义
• 组合作为后期处理
• 透明度可能效果不佳

自定义合成

• 整合进入你自己的渲染器
• 合成完全可控
• 提供对所需功能的简单访问

• 产生matte新类
• 使用Metal提供纹理

func compositeFrame(_ frame : ARFrame!, commandBuffer : MTLCommandBuffer!) {
// Composition Part of the Rendering Code
guard ARWorldTrackingConfiguration.supportsFrameSemantics(.personSegmentationWithDepth) else {
return
}
// Schedule Matting
let matte = matte.generateMatte(from: frame, commandBuffer: commandBuffer)
let dilatedDepth = matte.generateDilatedDepth(from: frame, commandBuffer: commandBuffer)
// Custom composition code
 // Done
 commandBuffer.commit()
 }

class ARMatteGenerator: NSObject { func generateMatte(from: ARFrame, commandBuffer: MTLCommandBuffer) -> MTLTexutre
}

• Matte和深度分辨率不匹配
• 不能仅通过alpha解决
• 需要修改预估深度缓冲

fragment half4 customComposition(...)
{
	half4 camera = cameraTexture.sample(s, in.uv);
	half4 rendered = renderedTexture.sample(s, in.uv);
	float renderedDepth = renderedDepthTexture.sample(s, in.uv);
	half4 scene = mix(rendered, camera, rendered.a);
	half matte = matteTexture.sample(s, in.uv);
	float dilatedDepth = dilatedDepthTexture.sample(s, in.uv);
	if (dilatedDepth < renderedDepth) { // People in front of rendered
		return mix(scene, camera, matte);
	} else {
		return scene;
	}
}