The Analytic Garden

The blockchain cannot be described just as a revolution. It is a tsunami-like phenomenon, slowly advancing and gradually enveloping everything along its way by the force of its progression.   - William Mougayar I think the whole narrative of blockchain without bitcoin will amount to very little.  - Fred Ehrsam We want a language that’s open source, with a liberal license. We want the speed of C with the dynamism of Ruby. We want a language that’s homoiconic, with true macros like Lisp, but with obvious, familiar mathematical notation like Matlab. We want something as usable for general programming as Python, as easy for statistics as R, as natural for string processing as Perl, as powerful for linear algebra as Matlab, as good at gluing programs together as the shell. Something that is dirt simple to learn, yet keeps the most serious hackers happy. We want it interactive and we want it compiled. Why We Created Julia - Jeff Bezanson, Stefan Karpinski, Viral Shah, ...

I spend most of my life feeling like I've been shot out of a cannon I spend most of my life feeling like I've been shot out of a cannon. - Molly Ivins The Kalman Filter I have been using this blog to post techniques that I have been trying to learn. I hope eventually to apply these to more serious data than the examples I have been  using. I'm going to continue and take a look at a very simple application of a Kalman filter . The Kalman filter allows us to use noisy measurements of a dynamic process to make reasonably accurate predictions of the process's state. Kalman filters are important for guidance, navigation, and control of vehicles and robots, aircraft and spacecraft. It is also widely used in time series analysis, signal processing, and econometrics. If you Google Kalman filter, you will find many web pages describing the method in varying levels of  mathematical difficulty. My favorite source for Kalman filter details is Kalman and Bayesian...

Doing data analysis requires quite a bit of thinking and we believe that when you’ve completed a good data analysis, you’ve spent more time thinking than doing. – Roger Peng I have been wanting to try TensorFlow for some time. Tensorflow is a library, developed by Google, for dataflow programming. I got interested in dataflow long ago in my misspent youth when I was a primary developer of something called KnowledgePro which among other things did a simplified version of dataflow. TensorFlow's killer app is deep learning . However, it has many other possible applications in areas such as machine learning and search. It is built around the notion of stateful data graphs. The idea is that you build a graph of a computation where the nodes are data, constants, or operations. The nodes are stateful in the sense that they maintain their values. The arcs link the nodes into a computational graph. Tensors flow along the arcs. Tensor are multidimensional arrays holding the cons...

We tell stories with maps about global warming, biodiversity; we can design more livable cities, track the spread of epidemics. That makes a difference.  Jack Dangermond A map is not the territory it represents, but, if correct, it has a similar structure to the territory, which accounts for its usefulness. Alfred Korzybski I wanted to plot sea surface temperature (SST). Actually, I wanted more than that. I wanted an easy way to plot global SST. I wanted a lazy way to make a nice plot. It turns out it's not hard, but more difficult than I naively expected. Get Some Data NOAA's National Centers for Environmental Information ( NCEI ) hosts public access archives for environmental data on Earth. Among the many data sets are high resolution satellite measurements of SST. The data, daily OISST , I'm interested in has a spatial grid resolution of 0.25° and temporal resolution of 1 day. It uses Advanced Very High Resolution Radiometer (AVHRR) infrared satellite SST da...

Turn and face the strange  Ch-ch-changes David Bowie - Changes Suppose you have some data that looks like this. It might be a time series or some other form of sequential data. It looks like there might be several different regions in the data.The count ranges seems to change somewhere around 25 and maybe again 75 or 80. Assuming there is some underlying process generating the data, we would like to know where do the parameters that control the process change and what are the values of the parameters in each region. This is toy data generated by this function: """ generateSeq - generate a sequence of Poisson distributed data with changes in mean at specified points arguments: length - the total sequence length cps - change point location lambdas - mean of data for each segment returns; seq - a numpy array of values at each position in the sequence Note: This function does no error checking. len(lambdas) must = len(cps) +1 ""...

In a previous post , I looked at Gaussian process regression. Gaussian processes are powerful.They give you useful error bounds. However, often the results look like curve fitting similar to fitting with splines and I suspect that people have a difficult time interpreting the covariance kernel. Often what people want is a simple interpretation of the data. This is relatively easy if it looks like there is a linear relationship among the  data or the data can be transformed to be somewhat approximately linear. Just crank up your favorite linear or  generalized linear modeling  (GLM) software. The software will fit the model to the data and give you the appropriate coefficients. Since the models are simple, interpretation is easy (well, maybe relatively easy). If the relationships among the data are non-linear, there's software to fit non-linear models, assuming you can come up with an appropriate set of functions to fit the data. Suppose you don't have strong feeling ...