Gradient Descent

The method of steepest-descent is re-visited in continuous time. It is shown that the continuous time version is a vector differential equation the solution of which is found by integration. Since numerical integration has many forms, we show an alternative to the conventional solution by using a Trapezoidal integration solution. This in turn gives a slightly modified least-mean squares (LMS) algorithm.

The many-to-many assignment problem (MMAP) is a recent topic of study in the field of combinatorial optimization. In this paper, a gradient-based interior-point method is proposed to solve MMAP. It is a deterministic method which assures an optimal solution. In this approach, the relaxation of the constraints is performed initially using the cardinality constraint detection operation. Then, the logarithmic barrier function (LBF) based gradient descent approach is executed to reach an accurate solution. Experiments have been performed to validate the practical implementation of the proposed algorithm. It also illustrates a significant improvement in convergence speed for the large MMAPs (i.e., if group size, α≥80) over state-of-the-art literature.

Recently, the Reconfigurable FSM has drawn the attention of the researchers for multistage signal processing applications. The optimal synthesis of Reconfigurable finite state machine with input multiplexing (Reconfigurable FSMIM) architecture is done by the iterative greedy heuristic based Hungarian algorithm (IGHA). The major problem concerning IGHA is the disintegration of a state encoding technique. This paper proposes the integration of IGHA with the state assignment using logarithmic barrier function based gradient descent approach to reduce the hardware consumption of Reconfigurable FSMIM. Experiments have been performed using MCNC FSM benchmarks which illustrate a significant area and speed improvement over other architectures during field programmable gate array (FPGA) implementation.

Electricity generation at the hydropower stations in Nigeria has been below the expected value. While the hydro stations have a capacity to generate up to 2,380 MW, the daily average energy generated in 2017 was estimated at around 846 MW. A factor responsible for this is the lack of a proper control system to manage the transfer of resources between the cascaded Kainji-Jebba Hydropower stations operating in tandem. This paper addressed the optimal regulation of the operating head of the Jebba hydropower reservoir in the presence of system constraints, flow requirement and environmental factors that are weather-related. The resulting two-point boundary value problem was solved using the progressive expansion of domain technique as against the shooting or multiple shooting techniques. The results provide the optimal inflow required to keep the operating head of the Jebba reservoir at a nominal level, hence ensuring that the maximum number of turbo-alternator units are operated.

In this paper, a matrix factorization recommendation algorithm is used to recommend items to the user by inculcating a hybrid optimization technique that combines Alternating Least Squares (ALS) and Stochastic Gradient Descent (SGD) in the advanced stage and compares the two individual algorithms with the hybrid model. This hybrid optimization algorithm can be easily implemented in the real world as a cold start can be easily reduced. The hybrid technique proposed is set side-by-side with the ALS and SGD algorithms individually to assess the pros and cons and the requirements to be met to choose a specific technique in a specific domain. The metric used for comparison and evaluation of this technique is Mean Squared Error (MSE).

Over the past decade, educational neuroscience research has increasingly identified the functional connectivity between the ventral striatum (VS) and the prefrontal cortex (PFC) as a significant biomarker for intrinsic motivation in adolescent students. Despite these findings, there remains a dearth of methods for utilizing such connectivity indices to effectively measure intrinsic motivation levels in educational settings. This article presents an overview of the most important neuroscientific research on intrinsic motivation in human youths together with a proposal whereby VS-PFC functional connectivity signals, extracted from functional magnetic resonance imaging (fMRI) data analysis, are used as predictors of intrinsic motivation through a machine learning (ML)-based linear regression model. By developing a robust linear regression model buttressed by ML techniques and a substantial sample of participants, our method aims to facilitate rapid and precise predictions of intrinsic motivation levels without the need for repeated assessments of intrinsic motivation, thereby saving time and resources in subsequent studies. To elucidate our model, we presented equations showing how regression parameters are computed using the conventional ordinary least squares (OLS) method and the ML-based gradient descent (GD) method, highlighting their differences in the process. Potential technical difficulties concerning the establishment and validation of our ML-based model are also discussed with concrete recommendations on how to resolve them. With the right implementation, we expect our method to benefit longitudinal fMRI studies examining developmental brain and behavioral changes in intrinsic motivation and educational intervention programs that require quick and accurate identification of students’ intrinsic motivation levels. Also noteworthy is that our proposed methodology is not limited to predicting intrinsic motivation alone and can be adapted for other functional connectivity and behavioral variables that may predict different outcome variables. The flexibility of our ML-based regression model will allow researchers to tailor the model by selecting alternative variables to suit their specific research needs.

A theoretical formulation of a fast learning method based on a pseudoinverse technique is presented. The efficiency and robustness of the method are verified with the help of an Exclusive OR problem and a dynamic system identification of a linear single degree of freedom mass–spring problem. It is observed that, compared with the conventional backpropagation method, the proposed method has a better convergence rate and a higher degree of learning accuracy with a lower equivalent learning coefficient. It is also found that unlike the steepest descent method, the learning capability of which is dependent on the value of the learning coefficient ν, the proposed pseudoinverse based backpropagation algorithm is comparatively robust with respect to its equivalent variable learning coefficient. A combination of the pseudoinverse method and the steepest descent method is proposed for a faster, more accurate learning capability.

We propose and analyze a stochastic Newton algorithm for homogeneous distributed stochastic convex optimization, where each machine can calculate stochastic gradients of the same population objective, as well as stochastic Hessian-vector products (products of an independent unbiased estimator of the Hessian of the population objective with arbitrary vectors), with many such stochastic computations performed between rounds of communication. We show that our method can reduce the number, and frequency, of required communication rounds compared to existing methods without hurting performance, by proving convergence guarantees for quasi-self-concordant objectives (e.g., logistic regression), alongside empirical evidence.

Adaptive regularization methods come in diagonal and full-matrix variants. However, only the former have enjoyed widespread adoption in training large-scale deep models. This is due to the computational overhead of manipulating a full matrix in high dimension. In this paper, we show how to make full-matrix adaptive regularization practical and useful. We present GGT, a truly scalable full-matrix adaptive optimizer. At the heart of our algorithm is an efficient method for computing the inverse square root of a low-rank matrix. We show that GGT converges to first-order local minima, providing the first rigorous theoretical analysis of adaptive regularization in non-convex optimization. In preliminary experiments, GGT trains faster across a variety of synthetic tasks and standard deep learning benchmarks.

Many real-world problems face the dilemma of choosing best $K$ out of $N$ options at a given time instant. This setup can be modelled as combinatorial bandit which chooses $K$ out of $N$ arms at each time, with an aim to achieve an efficient tradeoff between exploration and exploitation. This is the first work for combinatorial bandit where the reward received can be a non-linear function of the chosen $K$ arms. The direct use of multi-armed bandit requires choosing among $N$-choose-$K$ options making the state space large. In this paper, we present a novel algorithm which is computationally efficient and the storage is linear in $N$. The proposed algorithm is a divide-and-conquer based strategy, that we call CMAB-SM. Further, the proposed algorithm achieves a regret bound of $\tilde O(K^\frac{1}{2}N^\frac{1}{3}T^\frac{2}{3})$ for a time horizon $T$, which is sub-linear in all parameters $T$, $N$, and $K$. The evaluation results on different reward functions and arm distribution fun...

In this paper, we propose a new conjugate gradient-like algorithm. The step directions generated by the new algorithm satisfy a sufficient descent condition independent of the line search. The global convergence of the new algorithm, with the Armijo backtracking line search, is proved. Numerical experiments indicate the efficiency and robustness of the new algorithm in solving a collection of unconstrained optimization problems from CUTEst package.

A theoretical formulation of a fast learning method based on a pseudoinverse technique is presented. The efficiency and robustness of the method are verified with the help of an Exclusive OR problem and a dynamic system identification of a linear single degree of freedom mass–spring problem. It is observed that, compared with the conventional backpropagation method, the proposed method has a better convergence rate and a higher degree of learning accuracy with a lower equivalent learning coefficient. It is also found that unlike the steepest descent method, the learning capability of which is dependent on the value of the learning coefficient ν, the proposed pseudoinverse based backpropagation algorithm is comparatively robust with respect to its equivalent variable learning coefficient. A combination of the pseudoinverse method and the steepest descent method is proposed for a faster, more accurate learning capability.

In this work, vector autoregression and neural network approach to multivariate time series analysis is presented. A vector autoregressive model and multilayer perceptron network with back-propagation, gradient descent algorithm have been designed to model the monthly average exchange rates of three international currencies with respect to naira. The series span over the period of January, 2012 to August, 2017. The original series were preprocessed to smoothen the distribution and facilitate fast convergence in the network algorithm. In training the network to learn the combined series of the exchange rates, a remarkable achievement was made. Adding to the beauty of the network model is the fact that the number of units of the input layer was predetermined through the VAR model. Using some model performance measures (RMSE, MBE and R2), it was recorded that the neural network approach performs better than the VAR model as it yielded minimum error of prediction. 

Model calibration is a major challenge faced by the plethora of statistical analytics packages that are increasingly used in Big Data applications. Identifying the optimal model parameters is a time-consuming process that has to be executed from scratch for every dataset/model combination even by experienced data scientists. We argue that the lack of support to quickly identify sub-optimal configurations is the principal cause. In this paper, we apply parallel online aggregation to identify sub-optimal configurations early in the processing by incrementally sampling the training dataset and estimating the objective function corresponding to each configuration. We design concurrent online aggregation estimators and define halting conditions to accurately and timely stop the execution. The end-result is online approximate gradient descent—a novel optimization method for scalable model calibration. We show how online approximate gradient descent can be represented as generic database a...

Optimization methods, namely, gradient optimization methods, are a key part of neural network training. In this paper, we propose a new gradient optimization method using exponential decay and the adaptive learning rate using a discrete second-order derivative of gradients. The MAMGD optimizer uses an adaptive learning step, exponential smoothing and gradient accumulation, parameter correction, and some discrete analogies from classical mechanics. The experiments included minimization of multivariate real functions, function approximation using multilayer neural networks, and training neural networks on popular classification and regression datasets. The experimental results of the new optimization technology showed a high convergence speed, stability to fluctuations, and an accumulation of gradient accumulators. The research methodology is based on the quantitative performance analysis of the algorithm by conducting computational experiments on various optimization problems and comparing it with existing methods.

In this study, we introduce the classes of $\phi$-nearly contraction mappings, $\phi$-nearly nonexpansive mappings, $\phi$-nearly asymptotically nonexpansive mappings, $\phi$-nearly uniformly $k$-Lipschitzian mappings and $\phi$-nearly uniform $k$-contraction mappings. These classes include those classes studied by Sahu [1] as special cases. We study the existence of fixed points and the structure of their fixed point sets of mappings in Banach spaces. Our results improve and generalize many celebrated results of fixed point theory in the context of demicontinuity.

Cancer of the bone marrow, often known as Acute Lymphoblastic Leukemia (ALL), is characterized by the unchecked growth of lymphoid progenitor cells. It affects both children and adults and is the most prevalent form of childhood cancer. There have been considerable advances in the diagnosis and treatment of acute lymphoblastic leukemia in recent years. The ability to accurately assess risk and develop an appropriate treatment strategy relies on a diagnosis that takes into account all relevant clinical, morphological, cytogenetic, and molecular aspects. However, in order to enhance survival and quality of life for those afflicted by this aggressive hematological malignancy, more research and clinical trials are required to address the issues associated with resistance, relapse, and long-term toxicity. Therefore, in this research a deep optimized convolutional neural network is proposed for the early detection and diagnosis of ALL. The deep optimized CNN model architecture comprises of five convolutional blocks with 13 conv layers, 5 max pool layers. The proposed deep optimized CNN model is tuned using the hyperparameters such as epochs, batch size and optimizers namely Adam and Adamax. Out of the two optimizers, the proposed deep optimized CNN model has outperformed using Adam optimizer with the points of accuracy and precision as 0.96 and 0.95 respectively.