Incomplete

Abstract In lifetesting, medical follow-up, and other fields the observation of the time of occurrence of the event of interest (called a death) may be prevented for some of the items of the sample by the previous occurrence of some other event (called a loss). Losses may be either accidental or controlled, the latter resulting from a decision to terminate certain observations. In either case it is usually assumed in this paper that the lifetime (age at death) is independent of the potential loss time; in practice this assumption deserves careful scrutiny. Despite the resulting incompleteness of the data, it is desired to estimate the proportion P(t) of items in the population whose lifetimes would exceed t (in the absence of such losses), without making any assumption about the form of the function P(t). The observation for each item of a suitable initial event, marking the beginning of its lifetime, is presupposed. For random samples of size N the product-limit (PL) estimate can be defined as follows: List and label the N observed lifetimes (whether to death or loss) in order of increasing magnitude, so that one has 0≤t 1ǐ≤t 2ǐ≤ … ≤t N ǐ. Then P(t)= II. [(N – r)/(N – r + 1)], where r assumes those values for which tr ≤t and for which tr ǐ measures the time to death. This estimate is the distribution, unrestricted as to form, which maximizes the likelihood of the observations. Other estimates that are discussed are the actuarial estimates (which are also products, but with the number of factors usually reduced by grouping); and reduced-sample (RS) estimates, which require that losses not be accidental, so that the limits of observation (potential loss times) are known even for those items whose deaths are observed. When no losses occur at ages less than t, the estimate of P(t) in all cases reduces to the usual binomial estimate, namely, the observed proportion of survivors.

@article{doi10108001621459195810501452,
    author = "Kaplan, Edward L. and Meier, Paul",
    title = "Nonparametric Estimation from Incomplete Observations",
    year = "1958",
    journal = "Journal of the American Statistical Association",
    abstract = "Abstract In lifetesting, medical follow-up, and other fields the observation of the time of occurrence of the event of interest (called a death) may be prevented for some of the items of the sample by the previous occurrence of some other event (called a loss). Losses may be either accidental or controlled, the latter resulting from a decision to terminate certain observations. In either case it is usually assumed in this paper that the lifetime (age at death) is independent of the potential loss time; in practice this assumption deserves careful scrutiny. Despite the resulting incompleteness of the data, it is desired to estimate the proportion P(t) of items in the population whose lifetimes would exceed t (in the absence of such losses), without making any assumption about the form of the function P(t). The observation for each item of a suitable initial event, marking the beginning of its lifetime, is presupposed. For random samples of size N the product-limit (PL) estimate can be defined as follows: List and label the N observed lifetimes (whether to death or loss) in order of increasing magnitude, so that one has 0≤t 1ǐ≤t 2ǐ≤ … ≤t N ǐ. Then P(t)= II. [(N – r)/(N – r + 1)], where r assumes those values for which tr ≤t and for which tr ǐ measures the time to death. This estimate is the distribution, unrestricted as to form, which maximizes the likelihood of the observations. Other estimates that are discussed are the actuarial estimates (which are also products, but with the number of factors usually reduced by grouping); and reduced-sample (RS) estimates, which require that losses not be accidental, so that the limits of observation (potential loss times) are known even for those items whose deaths are observed. When no losses occur at ages less than t, the estimate of P(t) in all cases reduces to the usual binomial estimate, namely, the observed proportion of survivors.",
    url = "https://doi.org/10.1080/01621459.1958.10501452",
    doi = "10.1080/01621459.1958.10501452",
    openalex = "W4293241248",
    references = "doi101016s0140673601265386, doi101017s0022172400014443, doi10108001621459194710501948, doi10108001621459195210501160, doi10108001621459195210501187, doi10108014786440308520304, doi101214aoms1177728793, doi101214aoms1177731170, doi101214aoms1177731566, doi1023072281004"
}

Abstract In lifetesting, medical follow-up, and other fields the observation of the time of occurrence of the event of interest (called a death) may be prevented for some of the items of the sample by the previous occurrence of some other event (called a loss). Losses may be either accidental or controlled, the latter resulting from a decision to terminate certain observations. In either case it is usually assumed in this paper that the lifetime (age at death) is independent of the potential loss time; in practice this assumption deserves careful scrutiny. Despite the resulting incompleteness of the data, it is desired to estimate the proportion P(t) of items in the population whose lifetimes would exceed t (in the absence of such losses), without making any assumption about the form of the function P(t). The observation for each item of a suitable initial event, marking the beginning of its lifetime, is presupposed. For random samples of size N the product-limit (PL) estimate can be defined as follows: List and label the N observed lifetimes (whether to death or loss) in order of increasing magnitude, so that one has 0≤t 1ǐ≤t 2ǐ≤ … ≤t N ǐ. Then P(t)= II. [(N – r)/(N – r + 1)], where r assumes those values for which tr ≤t and for which tr ǐ measures the time to death. This estimate is the distribution, unrestricted as to form, which maximizes the likelihood of the observations. Other estimates that are discussed are the actuarial estimates (which are also products, but with the number of factors usually reduced by grouping); and reduced-sample (RS) estimates, which require that losses not be accidental, so that the limits of observation (potential loss times) are known even for those items whose deaths are observed. When no losses occur at ages less than t, the estimate of P(t) in all cases reduces to the usual binomial estimate, namely, the observed proportion of survivors.

@article{doi1023072281868,
    author = "Kaplan, Edward L. and Meier, Paul",
    title = "Nonparametric Estimation from Incomplete Observations",
    year = "1958",
    journal = "Journal of the American Statistical Association",
    abstract = "Abstract In lifetesting, medical follow-up, and other fields the observation of the time of occurrence of the event of interest (called a death) may be prevented for some of the items of the sample by the previous occurrence of some other event (called a loss). Losses may be either accidental or controlled, the latter resulting from a decision to terminate certain observations. In either case it is usually assumed in this paper that the lifetime (age at death) is independent of the potential loss time; in practice this assumption deserves careful scrutiny. Despite the resulting incompleteness of the data, it is desired to estimate the proportion P(t) of items in the population whose lifetimes would exceed t (in the absence of such losses), without making any assumption about the form of the function P(t). The observation for each item of a suitable initial event, marking the beginning of its lifetime, is presupposed. For random samples of size N the product-limit (PL) estimate can be defined as follows: List and label the N observed lifetimes (whether to death or loss) in order of increasing magnitude, so that one has 0≤t 1ǐ≤t 2ǐ≤ … ≤t N ǐ. Then P(t)= II. [(N – r)/(N – r + 1)], where r assumes those values for which tr ≤t and for which tr ǐ measures the time to death. This estimate is the distribution, unrestricted as to form, which maximizes the likelihood of the observations. Other estimates that are discussed are the actuarial estimates (which are also products, but with the number of factors usually reduced by grouping); and reduced-sample (RS) estimates, which require that losses not be accidental, so that the limits of observation (potential loss times) are known even for those items whose deaths are observed. When no losses occur at ages less than t, the estimate of P(t) in all cases reduces to the usual binomial estimate, namely, the observed proportion of survivors.",
    url = "https://doi.org/10.2307/2281868",
    doi = "10.2307/2281868",
    openalex = "W3041047318",
    references = "doi101017s0022172400014443, doi10108001621459194710501948, doi10108001621459195210501160, doi10108001621459195210501187, doi10108001621459195310501206, doi10108014786440308520304, doi101214aoms1177728793, doi101214aoms1177731170, doi101214aoms1177731566, doi1023072281004"
}

The product-limit estimator for a distribution function, appropriate to observations which are variably censored, was introduced by Kaplan and Meier in 1958; it has provided a basis for study of more complex problems by Cox and by others. Its properties in the case of random censoring have been studied by Efron and later writers. The basic properties of the product-limit estimator are here shown to be closely parallel to the properties of the empirical distribution function in the general case of variably and arbitrarily censored observations.

@article{doi101017s0021900200047574,
    author = "Meier, Paul",
    title = "Estimation of a Distribution Function from Incomplete Observations",
    year = "1975",
    journal = "Journal of Applied Probability",
    abstract = "The product-limit estimator for a distribution function, appropriate to observations which are variably censored, was introduced by Kaplan and Meier in 1958; it has provided a basis for study of more complex problems by Cox and by others. Its properties in the case of random censoring have been studied by Efron and later writers. The basic properties of the product-limit estimator are here shown to be closely parallel to the properties of the empirical distribution function in the general case of variably and arbitrarily censored observations.",
    url = "https://doi.org/10.1017/s0021900200047574",
    doi = "10.1017/s0021900200047574",
    openalex = "W2790594745",
    references = "doi101214aoms1177731170"
}

Summary A broadly applicable algorithm for computing maximum likelihood estimates from incomplete data is presented at various levels of generality. Theory showing the monotone behaviour of the likelihood and convergence of the algorithm is derived. Many examples are sketched, including missing value situations, applications to grouped, censored or truncated data, finite mixture models, variance component estimation, hyperparameter estimation, iteratively reweighted least squares and factor analysis.

@article{doi101111j251761611977tb01600x,
    author = "Dempster, A. P. and Laird, N. M. and Rubin, Donald B.",
    title = "Maximum Likelihood from Incomplete Data Via the EM Algorithm",
    year = "1977",
    journal = "Journal of the Royal Statistical Society Series B (Statistical Methodology)",
    abstract = "Summary A broadly applicable algorithm for computing maximum likelihood estimates from incomplete data is presented at various levels of generality. Theory showing the monotone behaviour of the likelihood and convergence of the algorithm is derived. Many examples are sketched, including missing value situations, applications to grouped, censored or truncated data, finite mixture models, variance component estimation, hyperparameter estimation, iteratively reweighted least squares and factor analysis.",
    url = "https://doi.org/10.1111/j.2517-6161.1977.tb01600.x",
    doi = "10.1111/j.2517-6161.1977.tb01600.x",
    openalex = "W2049633694",
    references = "doi1010029780470316436, doi101007bf02289343, doi101017s0305004100009580, doi10108001621459197710480998, doi101093biomet583545, doi101093biomet633581, doi101214aoms1177697196, doi101214aoms1177703732, doi1023071909900, doi1023072063625"
}

The sphere of modern financial economics encompases finance, micro investment theory and much of the economics of uncertainty. As is evident from its influence on other branches of economics including public finance, industrial organization and monetary theory, the boundaries of this sphere are both permeable and flexible. The complex interactions of time and uncertainty guarantee intellectual challenge and intrinsic excitement to the study of financial economics. Indeed, the mathematics of the subject contain some of the most interesting applications of probability and optimization theory. But for all its mathematical refinement, the research has nevertheless had a direct and significant influence on practice. It was not always thus. Thirty years ago, finance theory was little more than a collection of anecdotes, rules of thumb, and manipulations of accounting data with an almost exclusive focus on corporate financial management. There is no need in this meeting of the guild to recount the subsequent evolution from this conceptual potpourri to a rigorous economic

@article{doi101111j154062611987tb04565x,
    author = "Merton, Robert C.",
    title = "A Simple Model of Capital Market Equilibrium with Incomplete Information",
    year = "1987",
    journal = "The Journal of Finance",
    abstract = "The sphere of modern financial economics encompases finance, micro investment theory and much of the economics of uncertainty. As is evident from its influence on other branches of economics including public finance, industrial organization and monetary theory, the boundaries of this sphere are both permeable and flexible. The complex interactions of time and uncertainty guarantee intellectual challenge and intrinsic excitement to the study of financial economics. Indeed, the mathematics of the subject contain some of the most interesting applications of probability and optimization theory. But for all its mathematical refinement, the research has nevertheless had a direct and significant influence on practice. It was not always thus. Thirty years ago, finance theory was little more than a collection of anecdotes, rules of thumb, and manipulations of accounting data with an almost exclusive focus on corporate financial management. There is no need in this meeting of the guild to recount the subsequent evolution from this conceptual potpourri to a rigorous economic",
    url = "https://doi.org/10.1111/j.1540-6261.1987.tb04565.x",
    doi = "10.1111/j.1540-6261.1987.tb04565.x",
    openalex = "W1999918735",
    references = "doi1010160022053176900466, doi1010160304393285900613, doi1010160304405x77900095, doi1010160304405x81900180, doi1010160304405x86900656, doi101086260061, doi101086295472, doi101111j154062611964tb02865x, doi101111j154062611977tb03317x, doi101126science7455683"
}

@incollection{doi101007978146124380925,
    author = "Kaplan, Edward L. and Meier, Paul",
    title = "Nonparametric Estimation from Incomplete Observations",
    year = "1992",
    booktitle = "Springer series in statistics",
    url = "https://doi.org/10.1007/978-1-4612-4380-9\_25",
    doi = "10.1007/978-1-4612-4380-9\_25",
    openalex = "W1979300931",
    references = "doi101016s0025619626044435, doi101017s0022172400014443, doi10108001621459195210501160, doi10108001621459195210501187, doi10108003461238194910419767, doi101136bmj23320266, doi101214aoms1177728793, doi101214aoms1177731170, doi1023072281318, openalexw2407759279"
}

The last two decades have seen enormous developments in statistical methods for incomplete data. The EM algorithm and its extensions, multiple imputation, and Markov Chain Monte Carlo provide a set of flexible and reliable tools from inference in large classes of missing-data problems. Yet, in practical terms, those developments have had surprisingly little impact on the way most data analysts handle missing values on a routine basis. Analysis of Incomplete Multivariate Data helps bridge the gap between theory and practice, making these missing-data tools accessible to a broad audience. It presents a unified, Bayesian approach to the analysis of incomplete multivariate data, covering datasets in which the variables are continuous, categorical, or both. The focus is applied, where necessary, to help readers thoroughly understand the statistical properties of those methods, and the behavior of the accompanying algorithms.All techniques are illustrated with real data examples, with extended discussion and practical advice. All of the algorithms described in this book have been implemented by the author for general use in the statistical languages S and S Plus. The software is available free of charge on the Internet.

@book{doi1012019781439821862,
    author = "Schafer, Joseph L.",
    title = "Analysis of Incomplete Multivariate Data",
    year = "1997",
    abstract = "The last two decades have seen enormous developments in statistical methods for incomplete data. The EM algorithm and its extensions, multiple imputation, and Markov Chain Monte Carlo provide a set of flexible and reliable tools from inference in large classes of missing-data problems. Yet, in practical terms, those developments have had surprisingly little impact on the way most data analysts handle missing values on a routine basis. Analysis of Incomplete Multivariate Data helps bridge the gap between theory and practice, making these missing-data tools accessible to a broad audience. It presents a unified, Bayesian approach to the analysis of incomplete multivariate data, covering datasets in which the variables are continuous, categorical, or both. The focus is applied, where necessary, to help readers thoroughly understand the statistical properties of those methods, and the behavior of the accompanying algorithms.All techniques are illustrated with real data examples, with extended discussion and practical advice. All of the algorithms described in this book have been implemented by the author for general use in the statistical languages S and S Plus. The software is available free of charge on the Internet.",
    url = "https://doi.org/10.1201/9781439821862",
    doi = "10.1201/9781439821862",
    openalex = "W4300187280"
}

Introduction Assumptions EM and Inference by Data Augmentation Methods for Normal Data More on the Normal Model Methods for Categorical Data Loglinear Models Methods for Mixed Data Further Topics Appendices References Index

@article{doi1023071271466,
    author = "Booth, David E. and Schafer, Joseph L.",
    title = "Analysis of Incomplete Multivariate Data",
    year = "2000",
    journal = "Technometrics",
    abstract = "Introduction Assumptions EM and Inference by Data Augmentation Methods for Normal Data More on the Normal Model Methods for Categorical Data Loglinear Models Methods for Mixed Data Further Topics Appendices References Index",
    url = "https://doi.org/10.2307/1271466",
    doi = "10.2307/1271466",
    openalex = "W1550443206"
}

Abstract Suppose we wish to recover a vector x 0 ∈ ℝ 𝓂 (e.g., a digital signal or image) from incomplete and contaminated observations y = A x 0 + e; A is an 𝓃 × 𝓂 matrix with far fewer rows than columns (𝓃 ≪ 𝓂) and e is an error term. Is it possible to recover x 0 accurately based on the data y? To recover x 0, we consider the solution x # to the 𝓁 1 ‐regularization problem where ϵ is the size of the error term e. We show that if A obeys a uniform uncertainty principle (with unit‐normed columns) and if the vector x 0 is sufficiently sparse, then the solution is within the noise level As a first example, suppose that A is a Gaussian random matrix; then stable recovery occurs for almost all such A 's provided that the number of nonzeros of x 0 is of about the same order as the number of observations. As a second instance, suppose one observes few Fourier samples of x 0; then stable recovery occurs for almost any set of 𝓃 coefficients provided that the number of nonzeros is of the order of 𝓃/(log 𝓂) 6. In the case where the error term vanishes, the recovery is of course exact, and this work actually provides novel insights into the exact recovery phenomenon discussed in earlier papers. The methodology also explains why one can also very nearly recover approximately sparse signals. © 2006 Wiley Periodicals, Inc.

@article{doi101002cpa20124,
    author = "Candès, Emmanuel J. and Romberg, Justin and Tao, Terence",
    title = "Stable signal recovery from incomplete and inaccurate measurements",
    year = "2006",
    journal = "Communications on Pure and Applied Mathematics",
    abstract = "Abstract Suppose we wish to recover a vector x 0 ∈ ℝ 𝓂 (e.g., a digital signal or image) from incomplete and contaminated observations y = A x 0 + e; A is an 𝓃 × 𝓂 matrix with far fewer rows than columns (𝓃 ≪ 𝓂) and e is an error term. Is it possible to recover x 0 accurately based on the data y? To recover x 0, we consider the solution x \# to the 𝓁 1 ‐regularization problem where ϵ is the size of the error term e. We show that if A obeys a uniform uncertainty principle (with unit‐normed columns) and if the vector x 0 is sufficiently sparse, then the solution is within the noise level As a first example, suppose that A is a Gaussian random matrix; then stable recovery occurs for almost all such A 's provided that the number of nonzeros of x 0 is of about the same order as the number of observations. As a second instance, suppose one observes few Fourier samples of x 0; then stable recovery occurs for almost any set of 𝓃 coefficients provided that the number of nonzeros is of the order of 𝓃/(log 𝓂) 6. In the case where the error term vanishes, the recovery is of course exact, and this work actually provides novel insights into the exact recovery phenomenon discussed in earlier papers. The methodology also explains why one can also very nearly recover approximately sparse signals. © 2006 Wiley Periodicals, Inc.",
    url = "https://doi.org/10.1002/cpa.20124",
    doi = "10.1002/cpa.20124",
    openalex = "W2164452299",
    references = "doi101002cpa20132, doi101016016727899290242f, doi101017cbo9780511804441, doi101073pnas0437847100, doi101109tit2005858979, doi101109tit2005862083, doi101109tit2006885507, doi101137s003614450037906x, doi101137s1064827596304010, openalexw2296616510"
}

This paper considers the model problem of reconstructing an object from incomplete frequency samples. Consider a discrete-time signal f/spl isin/C/sup N/ and a randomly chosen set of frequencies /spl Omega/. Is it possible to reconstruct f from the partial knowledge of its Fourier coefficients on the set /spl Omega/? A typical result of this paper is as follows. Suppose that f is a superposition of |T| spikes f(t)=/spl sigma//sub /spl tau//spl isin/T/f(/spl tau/)/spl delta/(t-/spl tau/) obeying |T|/spl les/C/sub M//spl middot/(log N)/sup -1/ /spl middot/ |/spl Omega/| for some constant C/sub M/>0. We do not know the locations of the spikes nor their amplitudes. Then with probability at least 1-O(N/sup -M/), f can be reconstructed exactly as the solution to the /spl lscr//sub 1/ minimization problem. In short, exact recovery may be obtained by solving a convex optimization problem. We give numerical values for C/sub M/ which depend on the desired probability of success. Our result may be interpreted as a novel kind of nonlinear sampling theorem. In effect, it says that any signal made out of |T| spikes may be recovered by convex programming from almost every set of frequencies of size O(|T|/spl middot/logN). Moreover, this is nearly optimal in the sense that any method succeeding with probability 1-O(N/sup -M/) would in general require a number of frequency samples at least proportional to |T|/spl middot/logN. The methodology extends to a variety of other situations and higher dimensions. For example, we show how one can reconstruct a piecewise constant (one- or two-dimensional) object from incomplete frequency samples - provided that the number of jumps (discontinuities) obeys the condition above - by minimizing other convex functionals such as the total variation of f.

@article{doi101109tit2005862083,
    author = "Candès, Emmanuel J. and Romberg, Justin and Tao, Terence",
    title = "Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information",
    year = "2006",
    journal = "IEEE Transactions on Information Theory",
    abstract = "This paper considers the model problem of reconstructing an object from incomplete frequency samples. Consider a discrete-time signal f/spl isin/C/sup N/ and a randomly chosen set of frequencies /spl Omega/. Is it possible to reconstruct f from the partial knowledge of its Fourier coefficients on the set /spl Omega/? A typical result of this paper is as follows. Suppose that f is a superposition of |T| spikes f(t)=/spl sigma//sub /spl tau//spl isin/T/f(/spl tau/)/spl delta/(t-/spl tau/) obeying |T|/spl les/C/sub M//spl middot/(log N)/sup -1/ /spl middot/ |/spl Omega/| for some constant C/sub M/>0. We do not know the locations of the spikes nor their amplitudes. Then with probability at least 1-O(N/sup -M/), f can be reconstructed exactly as the solution to the /spl lscr//sub 1/ minimization problem. In short, exact recovery may be obtained by solving a convex optimization problem. We give numerical values for C/sub M/ which depend on the desired probability of success. Our result may be interpreted as a novel kind of nonlinear sampling theorem. In effect, it says that any signal made out of |T| spikes may be recovered by convex programming from almost every set of frequencies of size O(|T|/spl middot/logN). Moreover, this is nearly optimal in the sense that any method succeeding with probability 1-O(N/sup -M/) would in general require a number of frequency samples at least proportional to |T|/spl middot/logN. The methodology extends to a variety of other situations and higher dimensions. For example, we show how one can reconstruct a piecewise constant (one- or two-dimensional) object from incomplete frequency samples - provided that the number of jumps (discontinuities) obeys the condition above - by minimizing other convex functionals such as the total variation of f.",
    url = "https://doi.org/10.1109/tit.2005.862083",
    doi = "10.1109/tit.2005.862083",
    openalex = "W2145096794",
    references = "doi101002j153873051961tb03976x, doi101007b98874, doi101016016727899290242f, doi1010179781108993456011, doi101073pnas0437847100, doi10110918959265, doi101109tit2004834793, doi101137s1064827596304010, doi10560219781421407944, openalexw2607928667"
}

@article{doi101016jacha200807002,
    author = "Needell, Deanna and Tropp, Joel A.",
    title = "CoSaMP: Iterative signal recovery from incomplete and inaccurate samples",
    year = "2008",
    journal = "Applied and Computational Harmonic Analysis",
    url = "https://doi.org/10.1016/j.acha.2008.07.002",
    doi = "10.1016/j.acha.2008.07.002",
    openalex = "W2289917018",
    references = "doi101002cpa20042, doi101002cpa20124, doi101016jcrma200803014, doi10110978258082, doi101109jstsp2007910281, doi101109tit2005858979, doi101109tit2005862083, doi101109tit2006871582, doi101109tit2006885507, doi101109tit2007909108, openalexw2296616510, openalexw3145128584"
}