Davide Nunes

Python Autodocs with MkGenDocs

2020-12-21T00:00:00+00:00

mkgendocs

mkgendocs is a Python package for automatically generating documentation pages in markdown from Python source files, by parsing Google-style docstring.

If you search for automated documentation generators for Python (or any language really), half the Web will tell you that relying on auto-documentation tools instead of writing good useful documentation by hand, is a bad practice. They are partially right, but real projects are not a salty thread from Reddit, so take all the advice, like anything else, critically!

Open source project success can depend on how well you communicate about it with your community, and good documentation is an important component of this process. Nevertheless, there is something to be said about generating useful documentation from existing source files. In the case of Python, docstrings are at the frontline of your project documentation. Not every project will benefit from a publicly documented API, but some will (e.g. a library of re-usable components). Creating reference documentation is essencial when the number of components in a library or framework is extensive. Examples of such projects include:

These examples include API reference documentation that is generated and linked to the original source files (usually hosted on publicly available online repositories).

From Sphinx to MkDocs

The most feature-complete documentation generator for Python is without a doubt Sphinx. It is robust and feature rich, it can be made to work with other markdown languages besides reStructuredText, and different styles of docstrings with extensions like napoleon.

MkDocs on the other hand, is a static documentation website generator from Markdown. It has less features, but being more simple it also requires less extensions and configuration. Overall, it has a lower barrier of entry for someone wanting to start deploying documentation for a project. One of the downsides of MkDocs, is the fact that doesn’t include robust plugins for automatically building API documentation from docstrings. This led me to create mkgendocs.

Beautiful Documentation for Python Projects

I was leaning towards MkDocs because of themes like Material for MkDocs. The theme follows the spirit of Google’s material design guidelines. It is responsive, great on mobile devices, and highly customizable. It also offers a search bar with live updates, automated table of contents, among other features such as support for MathJax, Admonitions, etc. This makes for documentation that is not only good looking, but easy to navigate, comprehend, and generally pleasant to use.

mkgendocs provides a simple tool that takes existing Python source files and generates API documentation pages that work well with Material for MkDocs.

For now it only supports Google style docstrings.

Example of API documentation generation for TensorX library.

If a repository url is provided, mkgendocs uses this to automatically link the generated API documentation pages to the source files online.

For now I only added support for Github. Should not be difficult to add and test the source code linking feature for other providers.

Getting started with mkgendocs

Installation

Install mkgendocs from PyPI

pip install mkgendocs

Usage

gendocs --config mkgendocs.yml

A sources directory is created with the documentation that was automatically generated. Any examples in a “examples” directory are automatically copied over to the documentation, the module level docstrings of any example source files are also copied and converted to markdown.

Configuration Example

sources_dir: docs/sources
templates_dir: docs/templates
repo: https://github.com/davidenunes/tensorx  #link to sources on github
version: master                               #link to sources on github

pages:
  - page: "api/train/model.md"
    source: "tensorx/train/model.py"
    methods:
      - Model:
          - train
          - set_optimizer
  
  - page: "api/layers/core.md"
    source: 'tensorx/layers.py'
    classes:
      - Linear:
        - compute_shape
      - Module
  - page: "math.md"
    source: 'tensorx/math.py'
    functions:
      - sparse_multiply_dense

  # creates an index page based on everything from target source
  - page: "api/layers/index.md"
    source: "tensorx/layers.py"
    index: True

sources_dir: directory where the resulting markdown files are created
templates_dir: directory where template files can be stored. All the folders and files are copied to the sources_dir. Any markdown files are used as templates with the tag `` in the template files being replaced by the generated documentation.
repo: repository to create view source links automatically for each class, function, and method;
version: defaults to “master” to create link to sources in the form https://repo/blob/version/file.py/#L1425;
pages: list of pages to be automatically generated from the respective source files and templates:
- page: path for page template / sources dir for the resulting page;
- source: path to source file from which the page is to be generated;
- classes: list of classes to be fully documented; a list of method names can be passed for each class, the default is to generate all methods;
- functions: list of functions to be documented.
- index: if True creates an index page for the given sources, you can also specify classes and functions, but not methods

Thank you

Thank you for your interest. This is very much a work in progress, evolving according to my own needs, but if you find any of this useful, consider getting me some coffee, coffee is great!

Git Good

2020-09-01T00:00:00+00:00

Git is a free and open source distributed Version Control System (VCS). Git can be hard, especially for people discovering it for the first time, but there are a lot of things that make this an exceptional VCS worth learning. This post is not meant to be a git tutorial, but rather a compilation of things and tips I find interesting. I also wanted to give a perspective of git independent of services like GitHub. Hopefully you can find something useful and “git better at it” 🙃.

If you’re looking to learn how to use git, take a look at the Pro Git book, it’s free and it is a great resource!

Git is not GitHub 😮

Services like GitHub made git particularly popular, but the convenience of something like Github has its downsides for new users: it causes confusion about the differences between GitHub or GitLab, and the actual version control system, git. While the former are great to host git repositories and make them publicly available (among other things), git is a piece of software designed to allow for distributed version control and collaboration. This means that you can use it offline, without GitHub, you can collaborate with people via chat, email, and anything that allows you to send text to someone else. Yes, GitHub makes it convenient to host your project and make it discoverable; it provides an incentive for collaboration, but, at the end of the day, GitHub is not git, just as Gmail is not e-mail.

Git is distributed 🖧

For those used to GitHub, collaborating with git over e-mail might seem anachronistic, but, consider the scale of projects and what git was designed to do. Being able to send contributions as plain text allied with the threaded nature of e-mail, means that you can have multiple discussions around certain aspects of a contribution, which is a big plus. Linux kernel development, for example, doesn’t happen on GitHub or GitLab. From the linux kernel GitHub Pull requests (GitHub’s system for submitting contributions):

Thanks for your contribution to the Linux kernel!

Linux kernel development happens on mailing lists, rather than on GitHub - this GitHub repository is a read-only mirror that isn’t used for accepting contributions. So that your change can become part of Linux, please email it to us as a patch.

Sending patches isn’t quite as simple as sending a pull request, but fortunately it is a well documented process.

Here’s what to do:

Format your contribution according to kernel requirements

Decide who to send your contribution to

Set up your system to send your contribution as an email

Send your contribution and wait for feedback

The development happens on various mailing lists for multiple subsystems. It’s distributed development to an unprecedented scale. This scale has a price, discipline in your contributions, something that GitHub doesn’t really help reinforce –which seems to be the whole issue that the creator of git Linus Torvalds has with it.

Git daemon 🌐

Focusing the development on a centralised service like GitHub can feel like you’re using an improved version of Subversion client/server approach, but remember, git is truly a decentralized system and we can do better. Suppose you don’t want to use GitHub, but want to make your repository available for other people to clone, pull from, etc. Git supports Peer-to-Peer setup out of the box with git daemon. Collaboration then happens on each peer local copy of the source tree and some form of communication channel (e.g. e-mail, chat, etc).

From your git folder you can execute the following command

git daemon --export-all --base-path=. --reuseaddr

--enable=upload-pack: (enabled by default) allows for git fetch, git pull, and git clone.
--export-all: means that you don’t have to create a file named git-daemon-export-ok in each exported repository.
--base-path: allows people to clone projects without specifying the entire path. Example: if you start the daemon with --base-path=/srv/git and try to pull from git://example.com/hello.git, git daemon will interpret the path as /srv/git/hello.git.
--reuseaddr: allows the server to restart without waiting for old connections to time out.

Congratulations, your machine is now running a git server and anyone can do:

git clone git://192.168.1.42/  #Your IP
# or
git remote add Foo git://192.168.1.42/
git fetch Foo
git checkout develop
git push Foo develop

Git patches 📝

As we have discussed, git is a decentralized system, you can send contributions to anyone without the need of a centralized git repository. This is not only the default way of collaborating with git, it is particularly useful when a server is down, or if you don’t have permissions to write to a remote repository —but would still like to propose changes.

You can create a patch from your current changes without committing the code on your source tree:

# changes in the working tree not yet staged for commit
git diff > big-improvements.patch
# or changes between the index and your last commit;
# what you would be committing if you run "git commit"
# without "-a" option.
git diff --cached > big-improvements.patch

As an example suppose the change is the addition of a README file, the diff would look like this:

diff --git a/README b/README
new file mode 100644
index 0000000..e69de29

More often, what we would like to do is to propose a change we have made in your local source tree. To do this, we can use the git format-patch command:

git format-patch master

if we have commits ahead of the master branch, a diff file will be generated. We can also reference other commits in the same branch. Suppose we made changes in the current branch and want to reference the changes in relation to the last commit. We can do:

git format-patch HEAD~1

the patch file will contain something like:

From daf1010eb425a67ca6b0ba60f7cbec15bcff31f1 Mon Sep 17 00:00:00 2001
From: John Doe <heresjohnny@bestmail.com>
Date: Tue, 09 Sep 2020 15:42:00 +0100
Subject: [PATCH] Update README

---
 README | 1 +
 1 file changed, 1 insertion(+)

diff --git a/README b/README
index e69de29..d0fc019 100644
--- a/README
+++ b/README
@@ -0,0 +1 @@
+This is a README file, that is all.
--
2.28.0

to apply the patch we do:

# patch as unstaged changes in your branch
git apply 0001-Update-README.patch

# patch as commits
git am 0001-Update-README.patch

For more information, check the documentation for git diff, git format-patch, git apply, and git am.

Git tips 🔥

We are all bound to get stuck sometimes when things go wrong. A good starting point is this compilation. These are solutions to problems I often have to solve.

Premature commit 🔧

You pulled the trigger too fast on that commit and wish you could include additional changes? Make your changes, call git commit --amend and done. (Also useful to change the commit message.)

Go back ⌛

We messed up, go back to a previous commit.

# go back n commits n=1 in this case,
# --soft: optionally don't discard changes
# and put them on the staging area instead
git reset HEAD~1 --soft

Put changes on hold 🚧

So you want to get back to the state before you started making changes, but don’t want to throw these changes away:

# stash the changes
git stash
# get the changes back when needed
git stash pop

Where it went wrong 🔍

You have a problem and don’t know which commit introduced it, enter git bisect:

git bisect start          # start a bisect section
git bisect bad            # Current version is bad
git bisect good v2.2.1    # v2.2.1 is known to be good

bisect will now choose commits in the middle of the history and you can mark them as good or bad with the same commands. When no more revisions are available you’ll have a description of the commit that caused the problem. You can then reset the bisect state with git bisect reset.

Rebasing commits 💥

Sometimes we make two commits when in reality, we could have included all the changes in a single commit, and our history would be clearer. This is what is known as squashing. We can use git rebase to meld commits with previous commits. It can also be used instead of merging branches. Git’s rebase command temporarily rewinds the commits on your current branch, pulls in the commits from the other reference and reapplies the re-winded commits back on top.

Most people will advise you to always squash the commits and rebase them onto the parent branch (like master or develop) before you submit a pull request or send out a patch. Whether rebasing is preferred to merging really depends on the context.

If you want to read more about rebase vs merge, check out this post

Whatever you do DO NOT rebase commits in a upstream repository people can pull from. It will mess everyone’s history and lead to conflicts that all downstream peers will need to fix. Also, it’s never a good idea to rebase somebody else’s work, see this discussion.

# rebase last 2 commits
git rebase -i HEAD~2

The interactive system will open an editor where you can choose each commit in a list that are about to be changed. In this case, it will list 2 lines with the last 2 commits. This list reflects exactly how your branch will look like after the rebase:

pick c8175df added line
pick dc58443 added final line

# Rebase 65e38e0..dc58443 onto 65e38e0 (2 commands)
#
# Commands:
# p, pick <commit> = use commit
# r, reword <commit> = use commit, but edit the commit message
# r, reword <commit> = use commit, but edit the commit message
# s, squash <commit> = meld into previous commit
# f, fixup <commit> = like "squash", but discard log message
# ...

You can pick, reorder or squash any commits you want to make for a more readable history.

As we discussed, you can also rebase the current branch onto another

git checkout feature
git rebase -i master

For more information about rebase see the documentation

Good commit messages 🧐

Commit messages should be consistent across a project in terms of style, content, and metadata. But some good rules are as follows:

the minimum is a single short descriptive line (e.g. less than 50 characters);
separate subject, body, other data, with blank lines;
include metadata such as references to commits that introduce problems being solved.
wrap the body to 72 characters.

A good one liner can be something like this

Fix typos in the abstract

the form <Verb> <Target> <Description> is sometimes enough context to describe a simple change.

A great way to learn what good commit messages look like is to study repositories where this is done right. It is to no surprise that the Linux kernel and git itself are good examples. There are also plenty resources on the subject such as How to Write a Git Commit Message.

Good commit messages are an important collaboration tool. They are the best way to communicate context about a change to a fellow developer, or to our future selves. git diff will tell us what changed, the WHAT, but this added context documents the WHY.

Git for writing 👨‍💻

I have been experimenting with git for writing. The idea being that we can benefit from using version control with our papers, lecture notes, blog posts, etc. Suppose we are considering git to track changes in a scientific paper. We can mark submissions with tags, use patches to incorporate changes from collaborators, branches to work on revisions, git diff to visualize the changes, etc.

Remember that git cares about meaningful lines, so the first thing to take into account is that we should, at minimum hard wrap our lines at a given character limit, and/or write each sentence on a different line. By using Markdown or LaTex to write a document, we will be tempted to use the editor for soft word wrapping. This can lead to huge one-line paragraphs. The problem with this is that changing a word in that paragraph will be recorded as a change to the entire paragraph. Moving lines around has a similar effect.

As a workaround, to visualise changes in such cases, we can use the --color-words option with git diff.

Consider a LaTeX document, for example. If we move lines around and call the following command:

git diff --color-words

this is what we get:

But what is actually recorded in the diff is the following:

To generate a pdf to visualise the changes in our tracked latex document, check out the git latexdiff tool that wraps around git and latexdiff.

The result of the following command is an output pdf with the changes in relation with the previous commit

git latexdiff HEAD~1

Archiving projects 🏛️

This is not related to git specifically, but more with good open science practices. Online repository hosts like Github are not archival. Git itself lets us tag commits to mark versions and releases, but tags can be deleted, and storage is totally dependent on us. Remember, git distributed, and local. Git is meant to track changes and collaborate, not archive the state of projects.

Platforms like Zenodo, on the other hand, let you conveniently archive versions of GitHub repositories based on GitHub releases (tags).

If you already use GitHub as a public accessible mirror for your project tracked by git, this means you can easily archive certain versions of your project and automatically make them citable since Zenodo attributes a Digital Object Identifier (DOI) to its submissions.

With this said, we should discuss how to archive, backup, or share your entire git repository without depending on any specific platform. There are a couple of options.

You could zip it but… 🗄️

A zip of your project folder will include EVERYTHING, yes, including your .git folder with all the changes, branches, reflogs but this might not be what you actually want. A zip of the project folder will include untracked files, and you could accidentally share sensitive information, irrelevant temporary files or IDE and editor configuration folders, etc.

Mirror clone 🔗

A git clone with the option --mirror creates a bare repository (which contains only the stuff in the .git directory) and it maps all refs (including remote-tracking branches, etc.) to the target repository. This means that these refs can be updated by a git remote update.

git clone --mirror myrepo repo.git
# or some remote repository
git clone --mirror https://github.com/davidenunes/repo repo.git

You could make a backup of your repository by creating a mirror (bare) repository and archive it.

To restore the bare repository, you can do the following:

mkdir myrepo
mv repo.git myrepo/.git
cd myrepo
git init
git checkout -f

If you want to refresh the backup you just need to call git remote update from the clone location.

Bundling 📦

Git is capable of bundling its data into a single binary file. You need to list out every reference or specific range of commits that you want to be included. If you intend for this to be cloned somewhere else, you should add HEAD as a reference as well. Alternatively, you can use --all to include all refs.

Within the project folder do:

# this will include all info to recreate the master branch
git bundle create repo.bundle HEAD master
# optionally --all for all refs to be included
git bundle create repo.bundle --all

You can then send or store this file, and in another machine unbundle it into a repository:

git clone -b master repo.bundle repo

If you don’t include the HEAD reference (or --all) you will get the following error:warning: remote HEAD refers to non-existent ref, unable to checkout.

Using --all makes your bundle file match what you would get with git clone --mirror

Thank You

Thank you for reading ❤️ I would love to know what you think, if you do things differently, or have any other neat tips / suggestions. Let me know in the comment section. You can also follow me on Twitter, or subscribe to the RSS feed for more content.

Gumbel-Top Trick

2020-05-01T00:00:00+00:00

How to vectorize sampling from a discrete distribution

If you work with libraries such as NumPy, Jax Tensorflow, or PyTorch you (should) end-up writing a lot of vectorization code: instead of using control-flow operations (e.g. for loops), you write code that operates on an entire set of values at once. Inputs and outputs of your functions are multidimensional arrays or tensors. Lower-level libraries optimized for linear algebra operations (such as matrix multiplications) make dramatic performance improvements, especially when aided by modern hardware with direct support for vector-based instructions.

In libraries like NumPy or Tensorflow sampling from a discrete distribution without replacement is not vectorized because it requires bookkeeping. In other words, sampling from a population depends on the values we already sampled.

So some time ago, I came across a set of re-parametrization tricks that allow us to vectorize sampling from discrete distributions. This peaked my interest because I was looking for a way to build stochastic neural networks where neuron activations could be modelled with certain types of discrete distributions parametrized by unnormalized log-probabilities.

To get a probability distribution from unconstrained vectors, usually we use the softmax function:

\[ \sigma(y) = \frac{e^{y_i}}{\sum_{j=1}^N e^{y_j}} \]

We would then use the resulting distribution to sample classes from it, for example, using the inverse transform sampling: this takes uniform samples of a number \(u\ \in [0,1)\), interpreted as a probability, and then returns the largest number \(y\) from the domain of the distribution \(P(Y)\) such that \(P(-\infty < Y < y) \le u \). What we are doing is randomly choosing a proportion of the area under the curve and returning the number in the domain such that exactly this proportion of the area occurs to the left of that number.

The Gumbel-Max Trick

The Gumbel-Max trick can be used to sample from the previous discrete distribution without marginalizing all the unnormalized log probabilities (i.e, without \(\sum_{j=1}^N e^{y_j}\)). The procedure consists in taking the unnormalized log probabilities \(y_i\), adding noise \(z_i \sim~Gumbel(0,1) \) (i.i.d. from a Gumbel distribution) and taking arg max. In other words:

\[ y = \underset{ i \in K }{\operatorname{arg max}} x_i + z_i
\]

This eliminates the need for the marginalization (which can be expensive for high-dimensional vectors). Another consequence of doing away with the computation of a normalized probability distribution, is the fact that we don’t need to see all of the data before doing partial sampling, this means Gumbel-Max can be used for weighted sampling from a stream (see this). The Gumbel Distribution is used to model the distribution of the maximum (or the minimum) of a number of samples of various distributions and, as it turns out, \(z_i\) is distributed according to a softmax function \(\sigma(y)\).

Gumbel Probability Density Function (PDF) and Cumulative Distribution Function (CDF) respectively.

Gumbel distribution with location parameter \(\alpha\) and unit scale parameter has the following Cumulative Distribution Function (CDF):

\[ F(z;\alpha) = \exp \left[ -\exp\left[-(z-\alpha) \right]\right] \]

If \(z_k\) is the \(k^{th}\) element of the Gumbel distribution with location \(\alpha_k\), the probability that all of the other \(z_{k’\neq k}\) are less than \(z_k\) is:

\[ Pr(k > k’ | z_k, \{ \alpha_{k’}\}_{k’=1}^K) = \prod_{k’\neq k} \exp \left[ -\exp\left[-(z_k-\alpha_{k’}) \right] \right] \]

integrating the marginal distribution over \(z_k\) we have an integral which has the closed form:

\[ Pr(k > k’ | \{ \alpha_{k’}\}) = \frac{\exp\left [\alpha_k \right]}{\sum_{k’=1}^K \exp\left [\alpha_{k’} \right]} \]

which is exactly the softmax function.

The Gumbel-Top Trick

If we look at the Gumbel-Max trick as form of weighted reservoir sampling, we can see that if instead of arg max we take the top-k args, we are instead, sampling without replacement from the discrete categorical distribution. We can call this the Gumble-Top trick.

The Reparameterization Trick in Neural Networks

The reparameterization trick allows for the optimization of stochastic computation graphs via gradient descent. The essence of the trick is to refactor each stochastic node into a differentiable function of its parameters and a random variable with fixed distribution. As we have seen previously, some closed formed densities have a simple reparameterization. The choice of noise (e.g. Gumbel) gives the trick its name.

Generally speaking, this trick consists in sampling from \(p_\phi(x)\) by first sampling \(Z\) from some fixed distribution \(q(z)\) and then transforming the sample using some function \(g_\phi(z)\). This two step process is precisely what we call reparameterization trick, and it is what makes it possible to reduce the problem of estimating the gradient w.r.t. parameters of a distribution to the simpler problem of estimating the gradient w.r.t. parameters of a deterministic function. Once we reparameterized \(p_\phi(x)\), one can now express the objective as an expectation w.r.t.q(z):

\[ L(\theta, \phi)=\mathbb{E}_{X \sim p_{\phi}(x)} \left[ f_{\theta}(X) \right]=\mathbb{E}_{Z \sim q(z)} \left[ f_{\theta}\left(g_{\phi}(Z) \right) \right] \]

This trick was introduced in the context of variational inference independently by [Kingma & Welling 2014], [Rezende et al. 2014], and [Titsias & L ́azaro-Gredilla].

Implementation

Sampling from the Gumbel Distribution

We can sample \(z_i \sim \mathit{Gumbel(0,1)}\) as follows:

\[ \begin{eqnarray} x_i \sim \mathit{Uniform(0,1)} \nonumber \\
z_i = -\log(-\log(x_i)) \nonumber \end{eqnarray} \]

NumPy Gumbel-Top

To finish this post and get you an idea of how simple the vectorized procedure is, here’s an implementation using NumPy.

import numpy as np

def top_k(x, k):
    return np.argpartition(x, k)[..., -k:]

def sample_k(logits, k):
    u = np.random.uniform(size=np.shape(logits))
    z = -np.log(-np.log(u))
    return top_k(logits + z, k)

References

Blog Posts

Vieira, Tim. Gumbel-Max-Trick. 2014.
Ryans, Adam. The Gumbel-Max Trick for Discrete Distributions
Mena, Gonzalo. The Gumbel-Softmax Trick for Inference of Discrete Variables. 2017

Papers

Maddison, Chris J., Daniel Tarlow, and Tom Minka. A* sampling. Advances in Neural Information Processing Systems. 2014.
Kusner, Matt J., and José Miguel Hernández-Lobato. Gans for sequences of discrete elements with the gumbel-softmax distribution.
Jang, Eric, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax.
Efraimidis, Pavlos S., and Paul G. Spirakis. Weighted random sampling with a reservoir. Information Processing Letters 97.5 (2006): 181-185.
Kool, Wouter, Herke Van Hoof, and Max Welling. Stochastic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement. (2019).

EXP — A Tool for Hyperparameter Tuning

2020-02-29T00:00:00+00:00

exp

Experiment design, deployment, and optimization

Optimizing model hyperparameters is an ubiquitous task in Machine Learning research. Finding baselines that are useful as a ground truth for ablation studies implies that you want the comparison to be as fair as possible: we want to compare the best possible models for the given architectures. In the case of neural network models, this can mean adjusting learning rate, regularization weights, dropout probabilities, etc. Doing ablation studies for large models (or large datasets) can be particularly expensive, not just in terms of time, but (for GPU-accelerated models) of energy as well.

Writing code to run the experiments, take advantage of multiprocessing, or distributing model runs through different GPUs can be a pain. EXP separates model specification from model configuration, experiment deployment and result logging.

EXP is a tool / library for experiment design, model deployment, and hyperparameter optimization that performs 2 basic tasks:

model execution from parameter specifications;
model optimization given a parameter space.

Model Search with Bayesian Optimization

EXP wraps around scikit-optimize to find the best configuration for a model using global Bayesian optimization. For a tutorial on Bayesian Optimization see 1807.02811:

python -m exp.gopt -p basic.conf -m runnable.py -n 20 --workers 4

Step-by-step

With a single command, we can take any model and a parameter space specified in a configuration file and the optimization procedure will try to find the best possible model iteratively. Each model run informs the model about possible configuration candidates that might lead to better performance.

Suppose you want to optimize a simple function, say x², you want to find the parameter that minimizes this function, a parameter space file written in TOML might look like this:

[x]
type = "random"
bounds = [-10,10]
dtype = "int"
prior = "uniform"

The only thing you need now is a runnable model. A model here is just a python file runnable.py with a run function that takes a parameter dictionary as input:

def run(x=1, **kwargs):
    return x ** 2

Running the previous command with these two files will evaluate a number of initially random configurations in parallel (one for each available worker), generate a bayesian model of how the performance function might react to different configurations. The optimizer then suggests new configurations within the defined parameter value bounds. This continues iteratively until a number of specified runs n has passed.

Additionally, a gpu flag can be added to restrict the number of workers to the number of available GPU units in a machine. Each worker will only have access to 1 gpu unit (in this case using the CUDA_VISIBLE_DEVICES variable). This allows for a safe deployment of models using frameworks like TensorFlow or PyTorch that use every single GPU unit available to run their computational graphs.

Some caveats

The larger the parameter space, the harder the optimization problem, so this is not a silver bullet. I use Bayesian Optimization to complement my intuition about what parameters might yield a good model.

Gaussian optimization assumes that the parameter values are continuous, the optimizer does take care of corner cases, like integer or categorical values, but I recommend reading on the subject if you need to optimize discrete parameters, a better alternative for neural network architecture search might be the use of evolutionary computation techniques, but these are not yet available in the tool.

Thank You

For more documentation check out the project on Github. If you have any questions, drop me a line on twitter, or leave a comment bellow.

The Ignorant Schoolmaster

2020-02-19T00:00:00+00:00

The book The Ignorant Schoolmaster: Five Lessons in Intellectual Emancipation (1987), by the French philosopher Jacques Rancière, presents the notion of intellectual emancipation. He explores the idea of equality of minds while criticizing theories of pedagogical reform for failing to account for the fact that one cannot remove bias the education system using theories born from and designed to reinforce the very same bias they intend to remove.

Every time I find myself ranting about what makes something art, communication always comes to mind:

“The impossibility of our saying the truth, even when we feel it, makes us speak as poets, makes us tell the story of our mind’s adventures and verify that they are understood by other adventurers.”

and communication is so much about thinking as it is about doing:

“The virtue of our intelligence is less in knowing than in doing. Knowing is nothing, doing is everything, but this doing is fundamentally an act of communication.”

but cynic in me resonates especially with the following:

“There is no pride in saying out loud: Me too, I’m a painter! Pride consists in saying softly to others: You neither, you aren’t a painter! “Me too, I’m a painter” means: me too, I have a feelings to communicate to my fellow men.”

and this is exactly why sharing is so much more important —but less popular— than signalling (the two being often hard to distinguish):

“One must learn near those who have worked in the gap between feeling and expression , between the silent language of emotion and the arbitrariness of the spoken tongue, near those who have tried to give voice to the silent dialogue the soul has with itself, who have gambled all their credibility on the bet of the similarity of minds.”

The recurrent message of “he who knows how to remain true to himself in the middle of irrationality will triumph over the passions of others exactly as he triumphs over his own” takes me back to stoicism and Marcus Aurelius’ Meditations:

“The happiness of those who want to be popular depends on others; the happiness of those who seek pleasure fluctuates with moods outside their control; but the happiness of the wise grows out of their own free acts.”

Rancière’s emancipation, much like stoicism, falls back to reason above all else as a way to deal with our innate biases. Reason (and the awareness to reason) is perhaps the way to avoid “falling into the gravitational field of other minds”. But then again, as the economist Herbert Simon puts it, people do things for certain reasons, and those reasons need not be rational, we are after all being of bounded rationality.

The other recurrent theme in the book is a critique of theories of pedagogical reform. Theories that present themselves as an attempt to reform the social inequities of the school system without realizing that school is not a preparation for life, but a reflection that models it’s stratification.

For if science (theory) forms an enclave of freedom in a world of ideological enslavement, if science belongs to the intellectuals —the masters— and the critique of bourgeois content is reserved for those who already know, then there is only one way for students to criticize their masters’ knowledge from the point of view of class, and that is to become their peers.

Rancière’s critique of the educational theories shows them to have at least one thing in common: a lesson in inequality. Each, that is, by beginning with inequality, proves it, and by proving it, in the end, is obliged to rediscover it again and again. Erecting and maintaining the distance separating a future reconciliation from a present inequality —a distance discursively invented and reinvented so that it may never be abolished.

Hello World

2020-01-07T00:00:00+00:00

Welcome to my Web page. I enjoy maintaining control over my content somewhere, especially when other third-party “web-identity” services invite you to invest time and attention without matching the commitment with adequate levels of ownership and control. I will post about ongoing projects Artificial Intelligence in general, Natural Natural Language Processing, Machine Learning in particular along with musings on the philosophical aspects of intelligence, consciousness, and learning. More about me here.