Slide 1: Introduction to Active Shape Models (ASM)
Active Shape Models (ASM) are statistical models of shape that can deform to fit new examples. They are particularly useful in face recognition and image generation tasks. ASMs learn the patterns of shape variability from a training set of annotated images and can then be used to find and fit to new instances of the object in novel images.
import numpy as np
import matplotlib.pyplot as plt
# Define a simple face shape
face = np.array([(0, 0), (1, 0), (2, 1), (1, 2), (0, 2), (0, 0)])
# Plot the face shape
plt.figure(figsize=(6, 6))
plt.plot(face[:, 0], face[:, 1], 'b-')
plt.title('Simple Face Shape')
plt.axis('equal')
plt.show()Slide 2: Shape Representation
In ASM, shapes are typically represented as a set of landmark points. These points are chosen to represent key features of the object. For faces, landmarks might include the corners of the eyes, the tip of the nose, and the corners of the mouth.
import numpy as np
import matplotlib.pyplot as plt
# Define face landmarks
landmarks = np.array([
(0, 0), # Left eye
(2, 0), # Right eye
(1, 1), # Nose
(0, 2), # Left mouth corner
(2, 2) # Right mouth corner
])
# Plot the landmarks
plt.figure(figsize=(6, 6))
plt.scatter(landmarks[:, 0], landmarks[:, 1], c='r', s=50)
plt.plot(landmarks[:, 0], landmarks[:, 1], 'b-')
plt.title('Face Landmarks')
plt.axis('equal')
plt.show()Slide 3: Shape Alignment
Before building the model, we need to align the shapes in our training set. This typically involves translating, rotating, and scaling the shapes to minimize the differences between them that are not due to shape variation.
import numpy as np
from scipy.linalg import orthogonal_procrustes
def align_shapes(shapes):
# Compute mean shape
mean_shape = np.mean(shapes, axis=0)
# Align all shapes to the mean shape
aligned_shapes = []
for shape in shapes:
# Find optimal rotation and scale
R, _ = orthogonal_procrustes(shape, mean_shape)
# Apply transformation
aligned_shape = shape @ R
aligned_shapes.append(aligned_shape)
return np.array(aligned_shapes)
# Example usage
shapes = np.random.rand(10, 5, 2) # 10 shapes, each with 5 landmarks
aligned_shapes = align_shapes(shapes)
print("Original shape:", shapes[0])
print("Aligned shape:", aligned_shapes[0])Slide 4: Principal Component Analysis (PCA)
After alignment, we use Principal Component Analysis (PCA) to capture the main modes of shape variation. PCA reduces the dimensionality of the data while retaining the most important variations.
import numpy as np
from sklearn.decomposition import PCA
def perform_pca(aligned_shapes, n_components=5):
# Flatten the shapes
X = aligned_shapes.reshape(aligned_shapes.shape[0], -1)
# Perform PCA
pca = PCA(n_components=n_components)
pca.fit(X)
return pca
# Example usage
aligned_shapes = np.random.rand(100, 10, 2) # 100 shapes, each with 10 landmarks
pca = perform_pca(aligned_shapes)
print("Explained variance ratio:", pca.explained_variance_ratio_)
print("First principal component:", pca.components_[0])Slide 5: Shape Model
The shape model consists of the mean shape and the principal components. We can generate new shape instances by varying the weights of the principal components.
import numpy as np
class ShapeModel:
def __init__(self, mean_shape, components):
self.mean_shape = mean_shape
self.components = components
def generate_shape(self, weights):
shape = self.mean_shape.()
for i, w in enumerate(weights):
shape += w * self.components[i]
return shape.reshape(-1, 2)
# Example usage
mean_shape = np.random.rand(10, 2).flatten()
components = np.random.rand(5, 20) # 5 components, 20 = 10 landmarks * 2 coordinates
model = ShapeModel(mean_shape, components)
new_shape = model.generate_shape([0.1, -0.2, 0.3, 0, 0])
print("Generated shape:", new_shape)Slide 6: Image Sampling
To fit the model to a new image, we need to sample the image around each landmark. This is typically done by sampling along the normal to the shape boundary at each landmark.
import numpy as np
import matplotlib.pyplot as plt
def sample_around_landmark(image, landmark, normal, sample_points=5):
samples = []
for i in range(-sample_points, sample_points+1):
point = landmark + i * normal
x, y = int(point[0]), int(point[1])
if 0 <= x < image.shape[1] and 0 <= y < image.shape[0]:
samples.append(image[y, x])
return np.array(samples)
# Example usage
image = np.random.rand(100, 100)
landmark = np.array([50, 50])
normal = np.array([1, 0])
samples = sample_around_landmark(image, landmark, normal)
plt.figure(figsize=(10, 4))
plt.subplot(121)
plt.imshow(image, cmap='gray')
plt.plot(landmark[0], landmark[1], 'r.')
plt.title('Image with Landmark')
plt.subplot(122)
plt.plot(samples)
plt.title('Sampled Profile')
plt.show()Slide 7: Fitting the Model
Fitting the ASM to a new image involves iteratively adjusting the model parameters to better match the image data. This typically involves two steps: finding the best locations for each landmark, and then updating the model parameters to fit these locations.
import numpy as np
def fit_asm(model, image, initial_shape, max_iterations=50):
current_shape = initial_shape.()
for _ in range(max_iterations):
# Find best locations for landmarks
new_landmarks = find_best_landmarks(image, current_shape)
# Update model parameters
params = model.fit_to_landmarks(new_landmarks)
# Generate new shape
new_shape = model.generate_shape(params)
# Check for convergence
if np.allclose(current_shape, new_shape, atol=1e-6):
break
current_shape = new_shape
return current_shape
# Note: This is a simplified version. Real implementations would
# include more complex landmark finding and parameter fitting methods.Slide 8: Real-life Example: Face Detection
ASMs can be used for face detection in images. By fitting the model to different regions of an image, we can determine where faces are likely to be present.
import cv2
import numpy as np
def detect_faces(image, face_cascade):
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
faces = face_cascade.detectMultiScale(gray, 1.3, 5)
for (x, y, w, h) in faces:
cv2.rectangle(image, (x, y), (x+w, y+h), (255, 0, 0), 2)
return image
# Load pre-trained face detector
face_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_frontalface_default.xml')
# Load an image
image = cv2.imread('path_to_image.jpg')
# Detect faces
result = detect_faces(image, face_cascade)
# Display result
cv2.imshow('Detected Faces', result)
cv2.waitKey(0)
cv2.destroyAllWindows()Slide 9: Real-life Example: Facial Expression Analysis
ASMs can be used to analyze facial expressions by tracking the movement of facial landmarks over time.
import cv2
import dlib
import numpy as np
def analyze_expression(image, predictor):
# Detect face
detector = dlib.get_frontal_face_detector()
faces = detector(image)
if len(faces) == 0:
return "No face detected"
# Get facial landmarks
shape = predictor(image, faces[0])
landmarks = np.array([(shape.part(i).x, shape.part(i).y) for i in range(68)])
# Analyze expression (simplified)
mouth_width = np.linalg.norm(landmarks[54] - landmarks[48])
mouth_height = np.linalg.norm(landmarks[57] - landmarks[51])
if mouth_width > mouth_height * 1.5:
return "Smiling"
else:
return "Neutral"
# Load the predictor
predictor = dlib.shape_predictor("shape_predictor_68_face_landmarks.dat")
# Load an image
image = cv2.imread('path_to_image.jpg')
# Analyze expression
expression = analyze_expression(image, predictor)
print("Detected expression:", expression)Slide 10: Challenges and Limitations
While ASMs are powerful, they have some limitations:
- They require manual annotation of training images, which can be time-consuming.
- They may struggle with large variations in pose or lighting.
- They can be sensitive to initialization and may converge to local optima.
To address these issues, researchers have developed extensions like Active Appearance Models (AAMs) and Constrained Local Models (CLMs).
import numpy as np
import matplotlib.pyplot as plt
def visualize_asm_limitation(n_points=100):
# Generate a simple face shape
t = np.linspace(0, 2*np.pi, n_points)
x = 16 * np.sin(t)**3
y = 13 * np.cos(t) - 5 * np.cos(2*t) - 2 * np.cos(3*t) - np.cos(4*t)
# Add some random noise
x_noisy = x + np.random.normal(0, 1, n_points)
y_noisy = y + np.random.normal(0, 1, n_points)
plt.figure(figsize=(12, 6))
plt.subplot(121)
plt.plot(x, y, 'b-', label='True Shape')
plt.title('Ideal Face Shape')
plt.subplot(122)
plt.plot(x, y, 'b-', label='True Shape')
plt.plot(x_noisy, y_noisy, 'r.', label='Noisy Points')
plt.title('ASM Limitation: Sensitivity to Noise')
plt.legend()
plt.show()
visualize_asm_limitation()Slide 11: Extensions and Improvements
To overcome the limitations of basic ASMs, several extensions have been proposed:
- Active Appearance Models (AAMs): These incorporate texture information along with shape.
- Constrained Local Models (CLMs): These use local texture models around each landmark.
- 3D Morphable Models: These extend the concept to 3D face modeling.
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
def generate_3d_face(n_points=1000):
u = np.linspace(0, 2*np.pi, n_points)
v = np.linspace(0, np.pi, n_points)
x = 10 * np.outer(np.cos(u), np.sin(v))
y = 10 * np.outer(np.sin(u), np.sin(v))
z = 10 * np.outer(np.ones(np.size(u)), np.cos(v))
return x, y, z
fig = plt.figure(figsize=(10, 8))
ax = fig.add_subplot(111, projection='3d')
x, y, z = generate_3d_face()
ax.plot_surface(x, y, z, color='b', alpha=0.3)
ax.set_title('3D Morphable Model Concept')
plt.show()Slide 12: Future Directions
The field of face recognition and image generation is rapidly evolving. Some promising directions include:
- Deep learning-based approaches like Convolutional Neural Networks (CNNs) for facial landmark detection.
- Generative Adversarial Networks (GANs) for face image generation.
- Integration of ASMs with deep learning techniques for more robust face analysis.
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense
def create_cnn_landmark_detector():
model = Sequential([
Conv2D(32, (3, 3), activation='relu', input_shape=(224, 224, 3)),
MaxPooling2D((2, 2)),
Conv2D(64, (3, 3), activation='relu'),
MaxPooling2D((2, 2)),
Conv2D(64, (3, 3), activation='relu'),
Flatten(),
Dense(64, activation='relu'),
Dense(136) # 68 landmarks * 2 coordinates
])
return model
model = create_cnn_landmark_detector()
model.summary()Slide 13: Ethical Considerations
As face recognition technology becomes more prevalent, it's crucial to consider ethical implications:
- Privacy concerns: Facial data is sensitive personal information.
- Bias and fairness: Ensure models work equally well across different demographics.
- Consent: Consider whether individuals have consented to their facial data being used.
- Dual-use potential: Face recognition can be used for beneficial or harmful purposes.
Developers and researchers must prioritize these ethical considerations in their work.
def ethical_data_collection(image, face_cascade):
# Detect faces
faces = face_cascade.detectMultiScale(image, 1.3, 5)
# Anonymize faces
for (x, y, w, h) in faces:
image[y:y+h, x:x+w] = cv2.GaussianBlur(image[y:y+h, x:x+w], (99, 99), 30)
return image
# Note: This is a simplified example. Real-world applications would
# require more comprehensive privacy protection measures.Slide 14: Additional Resources
For those interested in diving deeper into Active Shape Models and related topics, here are some valuable resources:
- Cootes, T. F., et al. "Active Shape Models-Their Training and Application." Computer Vision and Image Understanding, 1995. (https://www.sciencedirect.com/science/article/abs/pii/S1077314285710041)
- Matthews, I., & Baker, S. "Active Appearance Models Revisited." International Journal of Computer Vision, 2004. (https://link.springer.com/article/10.1023/B:VISI.0000029666.37597.d3)
- Saragih, J. M., et al. "Deformable Model Fitting by Regularized Landmark Mean-Shift." International Journal of Computer Vision, 2011. (https://link.springer.com/article/10.1007/s11263-010-0380-4)
These papers provide in-depth explanations of the underlying principles and advanced techniques in facial modeling and analysis.