{"id":9220,"date":"2020-08-05T07:53:33","date_gmt":"2020-08-05T07:53:33","guid":{"rendered":"https:\/\/www.experfy.com\/blog\/?p=9220"},"modified":"2023-11-22T12:08:16","modified_gmt":"2023-11-22T12:08:16","slug":"intro-to-probabilistic-programming","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/intro-to-probabilistic-programming\/","title":{"rendered":"Intro to probabilistic programming"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

What is Probabilistic Programming?<\/h1>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

The idea behind Probabilistic programming to bring the inference algorithms and theory from statistics combined with formal semantics, compilers, and other tools from programming languages to build efficient inference evaluators for models and applications from Machine Learning. In other words, probabilistic programming is\u00a0a tool for statistical modeling<\/em>. The idea is to borrow lessons from the world of programming languages and apply them to the problems of designing and using statistical models.<\/p>\n\n\n\n

\n

Probabilistic programming is about doing statistics using the tools of computer science.<\/em><\/p>\n<\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

In the above figure you can see a typical computer science programming pipeline: Write a program, specify the values of its arguments then evaluate the program to produce an output. The right-hand side illustrates the approach taken to modeling in statistics: Start with the output, the observations or data Y, then specify an abstract generative model p(X,Y), often denoted mathematically, and finally use algebra and inference techniques to characterize the posterior distribution, p(X | Y ), of the unknown quantities in the model given the observed quantities. Whereas in Probabilistic programming: a programming language for model definitions and statistical inference algorithms for computing the conditional distribution of the program inputs that could have given rise to the observed program output.<\/p>\n\n\n\n

\n

Note: Probabilistic programming is not about writing software that behaves probabilistically.<\/em><\/p>\n<\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

To implement such Ensemble Architecture Tensorflow introduces Tensorflow-probability.<\/p>\n\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Tensorflow-Probability<\/h1><\/h1>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

TensorFlow Probability is a library for probabilistic reasoning and statistical analysis in TensorFlow. As part of the TensorFlow ecosystem, TensorFlow Probability provides integration of probabilistic methods with deep networks, gradient-based inference via automatic differentiation, and scalability to large datasets and models via hardware acceleration (e.g., GPUs) and distributed computation.
Probabilistic Machine Learning tools in TensorFlow-probability are structured in different levels. In this blog, we\u2019ll discuss\u00a0Statistical building blocks<\/em>\u00a0and\u00a0Model Building\u00a0<\/em>using TensorFlow-probability.<\/p>\n\n\n\n

Let\u2019s start with importing necessary modules:<\/p>\n\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

\n

Statistical Functions in Tensorflow-probability<\/h1>\n<\/h1>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

\n

Distributions-<\/h1>\n<\/h1>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

A\u00a0tfp.distributions.Distribution<\/a><\/code>\u00a0is a class with two core methods:\u00a0sample<\/code>\u00a0and\u00a0log_prob<\/code>. This class contains many distributions which can be seen by writing:<\/p>\n\n\n\n

print_subclasses_from_module(tfp.distributions, tfp.distributions.Distribution)<\/pre>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n
Let\u2019s see how to sample a Normal Distribution which is a good starter in stat101 using tf-probability:<\/p>\n\n\n\n
# A standard normal
normal = tfd.Normal(loc=0., scale=1.) # mean=0, std=3samples = normal.sample(1000)
sns.distplot(samples)
plt.title(\"Samples from a standard Normal\")
plt.show()'''
log of the probability density\/mass function evaluated at the given sample value.
'''
print(\"log(PDF):\",normal.log_prob(0.))<\/pre>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n
Now in order to calculate other statistical parameters like cumulative distribution function and multiple distributions we can still utilize tf-probabilities native classes.<\/p>\n\n\n\n
# Define a single scalar Normal distribution.
dist = tfd.Normal(loc=0., scale=3.)  # mean=0, std=3# Evaluate the cdf at 1, returning a scalar.
dist.cdf(1.)# Define a batch of two scalar valued Normals.
# The first has mean 1 and standard deviation 11, the second 2 and 22.
dist = tfd.Normal(loc=[1, 2.], scale=[11, 22.])# Evaluate the pdf of the first distribution on 0, and the second on 1.5,
# returning a length two tensor.
dist.prob([0, 1.5])# Get 3 samples, returning a 3 x 2 tensor.
dist.sample([3])<\/pre>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n
Using the above code you can calculate CDFs and Multiple Normal Distributions on-the-fly.
While using statistical-tools in your project you might also require declaring Multivariate Distributions, tf-probability has got you covered here as well!<\/p>\n\n\n\n
mvn = tfd.MultivariateNormalDiag(loc=[0., 0.], scale_diag = [1., 1.])
print(\"Batch shape:\", mvn.batch_shape)
print(\"Event shape:\", mvn.event_shape)samples = mvn.sample(1000)
print(\"Samples shape:\", samples.shape)g = sns.jointplot(samples[:, 0], samples[:, 1], kind='scatter')
plt.show()<\/pre>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n
There are tons of distributions like these inside the tfp module, LogNormal, Logistic, LogitNormal Mixture, Multinomial, MultivariateNormalDiag, to name a few. Every distribution comes with a plethora of statistical inferences and functions.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
Bijectors-<\/h1><\/h1>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n
Bijectors represent invertible, smooth functions. They can be used to transform distributions, preserving the ability to take samples and compute log_probs. They can be accessed from the\u00a0tfp.bijectors<\/a><\/code>\u00a0module.<\/p>\n\n\n\n
Each bijector implements at least 3 methods:<\/p>\n\n\n\n