Abstract Independent Components Analysis (ICA) maximizes the statistical independence of the repr... more Abstract Independent Components Analysis (ICA) maximizes the statistical independence of the representational components of a training image ensemble, but it cannot distinguish between the different factors, or modes, inherent to image formation, including scene structure, illumination, and imaging. We introduce a nonlinear, multifactor model that generalizes ICA. Our Multilinear ICA (MICA) model of image ensembles learns the statistically independent components of multiple factors. Whereas ICA employs linear ( ...
Sensor Fusion III: 3D Perception and Recognition, 1991
This paper introduces a new approach to surface reconstruction motivated by concepts from numeric... more This paper introduces a new approach to surface reconstruction motivated by concepts from numerical grid generation. We develop adaptive mesh models that nonuniformly sample and reconstruct input shape data. Adaptive meshes are dynamic models assembled from nodal masses connected by adjustable springs. Acting as mobile sampling sites, the nodes observe interesting properties of the input data, such as depths, gradients, and curvatures. The springs automatically adjust their stiffnesses based on the locally sampled information in order to concentrate nodes near rapid shape variations. The representational power of an adaptive mesh is enhanced by its ability to optimally distribute the available degrees of freedom in accordance with the local complexity of the input data. Surface reconstruction using adaptive meshes runs at interactive rates with continuous 3D display on a graphics workstation.
A method, system, computer-readable medium and data structure are provided for processing image d... more A method, system, computer-readable medium and data structure are provided for processing image data in connection with image recognition. A response of an image (FIG. 6 element 210) to a basis tensor can be determined after the image is applied thereto. The image response can be flattened (FIG. 6 element 220). A coefficient vector may be extracted from the image response (FIG. 6 element 230). The extracted coefficient vector may be compared to a plurality of different parameters stored in coefficient rows of a matrix (FIG. 6 ...
A data structure, method, storage medium and logic arrangement are provided for use in collecting... more A data structure, method, storage medium and logic arrangement are provided for use in collecting and analyzing multilinear data describing various characteristics of different objects. In particular it is possible to recognize an unknown object or an unknown viewpoint of an object, as well as synthesize a known viewpoint never before recorded of an object, and reduce the amount of stored data describing an object or viewpoint by using dimensionality reduction techniques, and the like.
Proceedings IEEE Workshop on Detection and Recognition of Events in Video, 2001
A first step towards a n understanding of the semantic content in a video is the reliable detecti... more A first step towards a n understanding of the semantic content in a video is the reliable detection and recognition of actions performed by objects. This is a dificult problem due t o the enormous vaeability in a n action's appearance when seen from different viewpoints and/or at different times. In this paper we address the recognition of actions by taking a novel approach that models actions as special types of 3d objects. Specifically, we observe that any action can be represented as a generalized cylinder, called the action cylinder. Reliable recognition is achieved by recovering the viewpoint transformation between reference (model) and given action cylinders. A set of 8 corresponding points from time-wise corresponding cross-sections is shown t o be suficient t o align the two cylinders under perspective projection. A surprising conclusion from visualizing actions as objects i s that rigid, articulated, and nonrigid actions can all be modeled an a uniform framework.
Multilinear algebra, the algebra of higher-order tensors, offers a potent mathematical framework ... more Multilinear algebra, the algebra of higher-order tensors, offers a potent mathematical framework for analyzing ensembles of images resulting from the interaction of any number of underlying factors. We present a dimensionality reduction algorithm that enables subspace analysis within the multilinear framework. This N-mode orthogonal iteration algorithm is based on a tensor decomposition known as the N-mode SVD, the natural extension to tensors of the conventional matrix singular value decomposition (SVD). We demonstrate the power of multilinear subspace analysis in the context of facial image ensembles, where the relevant factors include different faces, expressions, viewpoints, and illuminations. In prior work we showed that our multilinear representation, called TensorFaces, yields superior facial recognition rates relative to standard, linear (PCA/eigenfaces) approaches. Here, we demonstrate factor-specific dimensionality reduction of facial image ensembles. For example, we can s...
Independent Components Analysis (ICA) maximizes the statistical independence of the representatio... more Independent Components Analysis (ICA) maximizes the statistical independence of the representational components of a training image ensemble, but it cannot distinguish between the different factors, or modes, inherent to image formation, including scene structure, illumination, and imaging. We introduce a nonlinear, multifactor model that generalizes ICA. Our Multilinear ICA (MICA) model of image ensembles learns the statistically independent components of multiple factors. Whereas ICA employs linear (matrix) algebra, MICA exploits multilinear (tensor) algebra. We furthermore introduce a multilinear projection algorithm which projects an unlabeled test image into the N constituent mode spaces to simultaneously infer its mode labels. In the context of facial image ensembles, where the mode labels are person, viewpoint, illumination, expression, etc., we demonstrate that the statistical regularities learned by MICA capture information that, in conjunction with our multilinear projecti...
Human motion is the composite consequence of multiple elements, including the action performed an... more Human motion is the composite consequence of multiple elements, including the action performed and a motion signature that captures the distinctive pattern of movement of a particular individual. We develop a new algorithm that is capable of extracting these motion elements and recombining them in novel ways. The algorithm analyzes motion data spanning multiple subjects performing different actions. The analysis yields a generative motion model that can synthesize new motions in the distinctive styles of these individuals. Our algorithms can also recognize people and actions from new motions by comparing motion signatures and action parameters.
This paper introduces a tensor framework for image-based rendering. In particular, we develop an ... more This paper introduces a tensor framework for image-based rendering. In particular, we develop an algorithm called TensorTextures that learns a parsimonious model of the bidirectional texture function (BTF) from observational data. Given an ensemble of images of a textured surface, our nonlinear, generative model explicitly represents the multifactor interaction implicit in the detailed appearance of the surface under varying photometric angles, including local (per-texel) reflectance, complex mesostructural self-occlusion, interreflection and self-shadowing, and other BTF-relevant phenomena. Mathematically, TensorTextures is based on multilinear algebra, the algebra of higher-order tensors, hence its name. It is computed through a decomposition known as the N-mode SVD, an extension to tensors of the conventional matrix singular value decomposition (SVD). We demonstrate the application of TensorTextures to the image-based rendering of natural and synthetic textured surfaces under con...
Natural images are the composite consequence of multiple factors related to scene structure, illu... more Natural images are the composite consequence of multiple factors related to scene structure, illumination, and imaging. For facial images, the factors include different facial geometries, expressions, head poses, and lighting conditions. We apply multilinear algebra, the algebra of higherorder tensors, to obtain a parsimonious representation of facial image ensembles which separates these factors. Our representation, called TensorFaces, yields improved facial recognition rates relative to standard eigenfaces.
The head-related transfer function (HRTF) captures most of the auditory cues used to discern soun... more The head-related transfer function (HRTF) captures most of the auditory cues used to discern sound source direction and therefore is essential for synthesizing convincing spatial audio. HRTFs are, however, difficult to measure and highly person-specific. This has created significant interest in alternative methods for generating custom HRTFs. We present a multilinear modeling framework for HRTFs which makes use of a tensor decomposition called the N -mode SVD [1], a multilinear extension of the conventional singular value decomposition (SVD). Regression is then used to map simple anatomical data to complex HRTF data. This mapping defines a data-driven model capable of producing entire sets of individualized HRTFs from easily obtained anatomical measurements. We show that our approach yields objectively superior results to those of a mapping based on principle components analysis (PCA).
Visual objects are composed of a recursive hierarchy of perceptual wholes and parts, whose proper... more Visual objects are composed of a recursive hierarchy of perceptual wholes and parts, whose properties, such as shape, reflectance, and color, constitute a hierarchy of intrinsic causal factors of object appearance. However, object appearance is the compositional consequence of both an object's intrinsic and extrinsic causal factors, where the extrinsic causal factors are related to illumination, and imaging conditions. Therefore, this paper proposes a unified tensor model of wholes and parts, and introduces a compositional hierarchical tensor factorization that disentangles the hierarchical causal structure of object image formation, and subsumes multilinear block tensor decomposition as a special case. The resulting object representation is an interpretable combinatorial choice of wholes' and parts' representations that renders object recognition robust to occlusion and reduces training data requirements. We demonstrate ourapproach in the context of face recognition by ...
By adhering to the dictum, “No causation without manipulation (treatment, intervention)”, cause a... more By adhering to the dictum, “No causation without manipulation (treatment, intervention)”, cause and effect data analysis represents changes in observed data in terms of changes in the causal factors. When causal factors are not amenable for active manipulation in the real world due to current technological limitations or ethical considerations, a counterfactual approach performs an intervention on the model of data formation. In the case of object representation or activity (temporal object) representation, varying object parts is generally unfeasible whether they be spatial and/or temporal. Multilinear algebra, the algebra of higher order tensors, is a suitable and transparent framework for disentangling the causal factors of data formation. Learning a part-based intrinsic causal factor representations in a multilinear framework requires applying a set of interventions on a part-based multilinear model. We propose a unified multilinear model of wholes and parts. We derive a hierarc...
This thesis introduces a multilinear algebraic framework for computer graphics, computer vision, ... more This thesis introduces a multilinear algebraic framework for computer graphics, computer vision, and machine learning, particularly for the fundamental purposes of image synthesis, analysis, and recognition. Natural images result from the multifactor interaction between the imaging process, the scene illumination, and the scene geometry. We assert that a principled mathematical approach to disentangling and explicitly representing these causal factors, which are essential to image formation, is through numerical multilinear algebra, the algebra of higher-order tensors. Our new image modeling framework is based on (i) a multilinear generalization of principal components analysis (PCA), (ii) a novel multilinear generalization of independent components analysis (ICA), and (iii) a multilinear projection for use in recognition that maps images to the multiple causal factor spaces associated with their formation. Multilinear PCA employs a tensor extension of the conventional matrix singul...
The adaptive mesh model is extended in several ways. Open adaptive meshes and closed adaptive she... more The adaptive mesh model is extended in several ways. Open adaptive meshes and closed adaptive shells based on triangular and rectangular elements are developed. A discontinuity detection and preservation algorithm suitable for the model is proposed. Techniques for adaptive hierarchical subdivision of adaptive meshes and shells are also developed. The extended model is applied to image and 3D surface data
Abstract Independent Components Analysis (ICA) maximizes the statistical independence of the repr... more Abstract Independent Components Analysis (ICA) maximizes the statistical independence of the representational components of a training image ensemble, but it cannot distinguish between the different factors, or modes, inherent to image formation, including scene structure, illumination, and imaging. We introduce a nonlinear, multifactor model that generalizes ICA. Our Multilinear ICA (MICA) model of image ensembles learns the statistically independent components of multiple factors. Whereas ICA employs linear ( ...
Sensor Fusion III: 3D Perception and Recognition, 1991
This paper introduces a new approach to surface reconstruction motivated by concepts from numeric... more This paper introduces a new approach to surface reconstruction motivated by concepts from numerical grid generation. We develop adaptive mesh models that nonuniformly sample and reconstruct input shape data. Adaptive meshes are dynamic models assembled from nodal masses connected by adjustable springs. Acting as mobile sampling sites, the nodes observe interesting properties of the input data, such as depths, gradients, and curvatures. The springs automatically adjust their stiffnesses based on the locally sampled information in order to concentrate nodes near rapid shape variations. The representational power of an adaptive mesh is enhanced by its ability to optimally distribute the available degrees of freedom in accordance with the local complexity of the input data. Surface reconstruction using adaptive meshes runs at interactive rates with continuous 3D display on a graphics workstation.
A method, system, computer-readable medium and data structure are provided for processing image d... more A method, system, computer-readable medium and data structure are provided for processing image data in connection with image recognition. A response of an image (FIG. 6 element 210) to a basis tensor can be determined after the image is applied thereto. The image response can be flattened (FIG. 6 element 220). A coefficient vector may be extracted from the image response (FIG. 6 element 230). The extracted coefficient vector may be compared to a plurality of different parameters stored in coefficient rows of a matrix (FIG. 6 ...
A data structure, method, storage medium and logic arrangement are provided for use in collecting... more A data structure, method, storage medium and logic arrangement are provided for use in collecting and analyzing multilinear data describing various characteristics of different objects. In particular it is possible to recognize an unknown object or an unknown viewpoint of an object, as well as synthesize a known viewpoint never before recorded of an object, and reduce the amount of stored data describing an object or viewpoint by using dimensionality reduction techniques, and the like.
Proceedings IEEE Workshop on Detection and Recognition of Events in Video, 2001
A first step towards a n understanding of the semantic content in a video is the reliable detecti... more A first step towards a n understanding of the semantic content in a video is the reliable detection and recognition of actions performed by objects. This is a dificult problem due t o the enormous vaeability in a n action's appearance when seen from different viewpoints and/or at different times. In this paper we address the recognition of actions by taking a novel approach that models actions as special types of 3d objects. Specifically, we observe that any action can be represented as a generalized cylinder, called the action cylinder. Reliable recognition is achieved by recovering the viewpoint transformation between reference (model) and given action cylinders. A set of 8 corresponding points from time-wise corresponding cross-sections is shown t o be suficient t o align the two cylinders under perspective projection. A surprising conclusion from visualizing actions as objects i s that rigid, articulated, and nonrigid actions can all be modeled an a uniform framework.
Multilinear algebra, the algebra of higher-order tensors, offers a potent mathematical framework ... more Multilinear algebra, the algebra of higher-order tensors, offers a potent mathematical framework for analyzing ensembles of images resulting from the interaction of any number of underlying factors. We present a dimensionality reduction algorithm that enables subspace analysis within the multilinear framework. This N-mode orthogonal iteration algorithm is based on a tensor decomposition known as the N-mode SVD, the natural extension to tensors of the conventional matrix singular value decomposition (SVD). We demonstrate the power of multilinear subspace analysis in the context of facial image ensembles, where the relevant factors include different faces, expressions, viewpoints, and illuminations. In prior work we showed that our multilinear representation, called TensorFaces, yields superior facial recognition rates relative to standard, linear (PCA/eigenfaces) approaches. Here, we demonstrate factor-specific dimensionality reduction of facial image ensembles. For example, we can s...
Independent Components Analysis (ICA) maximizes the statistical independence of the representatio... more Independent Components Analysis (ICA) maximizes the statistical independence of the representational components of a training image ensemble, but it cannot distinguish between the different factors, or modes, inherent to image formation, including scene structure, illumination, and imaging. We introduce a nonlinear, multifactor model that generalizes ICA. Our Multilinear ICA (MICA) model of image ensembles learns the statistically independent components of multiple factors. Whereas ICA employs linear (matrix) algebra, MICA exploits multilinear (tensor) algebra. We furthermore introduce a multilinear projection algorithm which projects an unlabeled test image into the N constituent mode spaces to simultaneously infer its mode labels. In the context of facial image ensembles, where the mode labels are person, viewpoint, illumination, expression, etc., we demonstrate that the statistical regularities learned by MICA capture information that, in conjunction with our multilinear projecti...
Human motion is the composite consequence of multiple elements, including the action performed an... more Human motion is the composite consequence of multiple elements, including the action performed and a motion signature that captures the distinctive pattern of movement of a particular individual. We develop a new algorithm that is capable of extracting these motion elements and recombining them in novel ways. The algorithm analyzes motion data spanning multiple subjects performing different actions. The analysis yields a generative motion model that can synthesize new motions in the distinctive styles of these individuals. Our algorithms can also recognize people and actions from new motions by comparing motion signatures and action parameters.
This paper introduces a tensor framework for image-based rendering. In particular, we develop an ... more This paper introduces a tensor framework for image-based rendering. In particular, we develop an algorithm called TensorTextures that learns a parsimonious model of the bidirectional texture function (BTF) from observational data. Given an ensemble of images of a textured surface, our nonlinear, generative model explicitly represents the multifactor interaction implicit in the detailed appearance of the surface under varying photometric angles, including local (per-texel) reflectance, complex mesostructural self-occlusion, interreflection and self-shadowing, and other BTF-relevant phenomena. Mathematically, TensorTextures is based on multilinear algebra, the algebra of higher-order tensors, hence its name. It is computed through a decomposition known as the N-mode SVD, an extension to tensors of the conventional matrix singular value decomposition (SVD). We demonstrate the application of TensorTextures to the image-based rendering of natural and synthetic textured surfaces under con...
Natural images are the composite consequence of multiple factors related to scene structure, illu... more Natural images are the composite consequence of multiple factors related to scene structure, illumination, and imaging. For facial images, the factors include different facial geometries, expressions, head poses, and lighting conditions. We apply multilinear algebra, the algebra of higherorder tensors, to obtain a parsimonious representation of facial image ensembles which separates these factors. Our representation, called TensorFaces, yields improved facial recognition rates relative to standard eigenfaces.
The head-related transfer function (HRTF) captures most of the auditory cues used to discern soun... more The head-related transfer function (HRTF) captures most of the auditory cues used to discern sound source direction and therefore is essential for synthesizing convincing spatial audio. HRTFs are, however, difficult to measure and highly person-specific. This has created significant interest in alternative methods for generating custom HRTFs. We present a multilinear modeling framework for HRTFs which makes use of a tensor decomposition called the N -mode SVD [1], a multilinear extension of the conventional singular value decomposition (SVD). Regression is then used to map simple anatomical data to complex HRTF data. This mapping defines a data-driven model capable of producing entire sets of individualized HRTFs from easily obtained anatomical measurements. We show that our approach yields objectively superior results to those of a mapping based on principle components analysis (PCA).
Visual objects are composed of a recursive hierarchy of perceptual wholes and parts, whose proper... more Visual objects are composed of a recursive hierarchy of perceptual wholes and parts, whose properties, such as shape, reflectance, and color, constitute a hierarchy of intrinsic causal factors of object appearance. However, object appearance is the compositional consequence of both an object's intrinsic and extrinsic causal factors, where the extrinsic causal factors are related to illumination, and imaging conditions. Therefore, this paper proposes a unified tensor model of wholes and parts, and introduces a compositional hierarchical tensor factorization that disentangles the hierarchical causal structure of object image formation, and subsumes multilinear block tensor decomposition as a special case. The resulting object representation is an interpretable combinatorial choice of wholes' and parts' representations that renders object recognition robust to occlusion and reduces training data requirements. We demonstrate ourapproach in the context of face recognition by ...
By adhering to the dictum, “No causation without manipulation (treatment, intervention)”, cause a... more By adhering to the dictum, “No causation without manipulation (treatment, intervention)”, cause and effect data analysis represents changes in observed data in terms of changes in the causal factors. When causal factors are not amenable for active manipulation in the real world due to current technological limitations or ethical considerations, a counterfactual approach performs an intervention on the model of data formation. In the case of object representation or activity (temporal object) representation, varying object parts is generally unfeasible whether they be spatial and/or temporal. Multilinear algebra, the algebra of higher order tensors, is a suitable and transparent framework for disentangling the causal factors of data formation. Learning a part-based intrinsic causal factor representations in a multilinear framework requires applying a set of interventions on a part-based multilinear model. We propose a unified multilinear model of wholes and parts. We derive a hierarc...
This thesis introduces a multilinear algebraic framework for computer graphics, computer vision, ... more This thesis introduces a multilinear algebraic framework for computer graphics, computer vision, and machine learning, particularly for the fundamental purposes of image synthesis, analysis, and recognition. Natural images result from the multifactor interaction between the imaging process, the scene illumination, and the scene geometry. We assert that a principled mathematical approach to disentangling and explicitly representing these causal factors, which are essential to image formation, is through numerical multilinear algebra, the algebra of higher-order tensors. Our new image modeling framework is based on (i) a multilinear generalization of principal components analysis (PCA), (ii) a novel multilinear generalization of independent components analysis (ICA), and (iii) a multilinear projection for use in recognition that maps images to the multiple causal factor spaces associated with their formation. Multilinear PCA employs a tensor extension of the conventional matrix singul...
The adaptive mesh model is extended in several ways. Open adaptive meshes and closed adaptive she... more The adaptive mesh model is extended in several ways. Open adaptive meshes and closed adaptive shells based on triangular and rectangular elements are developed. A discontinuity detection and preservation algorithm suitable for the model is proposed. Techniques for adaptive hierarchical subdivision of adaptive meshes and shells are also developed. The extended model is applied to image and 3D surface data
Uploads