Jekyll2021-07-08T17:07:31+00:00https://rmcomplexity.com/feed.xmlRemove ComplexityHelping removing complexity from software development through understanding.Josue Balandrano CoronelDeveloper secret news2021-07-08T00:00:00+00:002021-07-08T00:00:00+00:00https://rmcomplexity.com/article/2021/07/08/secret-dev-news<p>These are <strong>secret</strong> news, keep them in the down low and don’t share them with anyone.</p>
<ol>
<li>We write unnecessary code.</li>
<li>What we think the code will do is wrong.</li>
<li>Not everything is testable.</li>
<li>Not everyone will understand the code you write.</li>
<li>Code is not always extendible nor manageable.</li>
<li>The Software you build for a company will never be for the greater good.</li>
<li>You are being exploited, and your job exists because others get exploited.</li>
<li>You will never know everything there is about Software.</li>
<li>Technology moves in the direction of money, not progress.</li>
<li>Software will never keep up with real life.</li>
<li>AI will neither save us nor destroy us.</li>
<li>Things change around you; you don’t change things directly.</li>
<li>You don’t need everything in your tech stack.</li>
<li>Your architecture is unnecessarily complicated.</li>
<li>Your documentation is not good.</li>
<li>Every computer system is compromised.</li>
</ol>
<figure class="img center">
<img src="/assets/images/george-carlin-secret-news.gif" style="max-width:200px;" alt="George Carlin hiding because these are secret." class="img-responsive" />
<figcaption><em>Don't look at these for too long!</em></figcaption>
</figure>rmcomplexityThe secret news from the developer world.Everything you need to know about dataclasses2021-01-04T02:01:00+00:002021-01-04T02:01:00+00:00https://rmcomplexity.com/article/2021/01/04/everything-you-need-to-know-about-dataclasses<ul id="markdown-toc">
<li><a href="#how-to-create-a-data-class" id="markdown-toc-how-to-create-a-data-class">How to create a data class</a></li>
<li><a href="#field-definition" id="markdown-toc-field-definition">Field definition</a> <ul>
<li><a href="#specify-a-default-value" id="markdown-toc-specify-a-default-value">Specify a default value</a></li>
<li><a href="#include-or-exclude-fields-in-automatically-implemented-dunder-methods" id="markdown-toc-include-or-exclude-fields-in-automatically-implemented-dunder-methods">Include or exclude fields in automatically implemented dunder methods</a></li>
<li><a href="#add-field-specific-metadata" id="markdown-toc-add-field-specific-metadata">Add field-specific metadata</a></li>
</ul>
</li>
<li><a href="#customize-object-initialization-using-__post_init__" id="markdown-toc-customize-object-initialization-using-__post_init__">Customize object initialization using <code class="language-plaintext highlighter-rouge">__post_init__</code></a></li>
<li><a href="#data-classes-that-we-can-compare-and-order" id="markdown-toc-data-classes-that-we-can-compare-and-order">Data classes that we can compare and order</a></li>
<li><a href="#frozen-or-immutable-instances" id="markdown-toc-frozen-or-immutable-instances">Frozen (or immutable) instances</a></li>
<li><a href="#updating-an-object-instance-by-replacing-the-entire-object" id="markdown-toc-updating-an-object-instance-by-replacing-the-entire-object">Updating an object instance by replacing the entire object.</a></li>
<li><a href="#adding-class-attributes" id="markdown-toc-adding-class-attributes">Adding class attributes</a></li>
<li><a href="#inheritance-in-data-classes" id="markdown-toc-inheritance-in-data-classes">Inheritance in data classes</a></li>
<li><a href="#hash-able-object" id="markdown-toc-hash-able-object">Hash-able object</a></li>
<li><a href="#a-use-case-for-data-classes" id="markdown-toc-a-use-case-for-data-classes">A use case for data classes</a> <ul>
<li><a href="#benefits-of-using-data-classes" id="markdown-toc-benefits-of-using-data-classes">Benefits of using data classes</a></li>
<li><a href="#disadvantages-of-using-data-classes" id="markdown-toc-disadvantages-of-using-data-classes">Disadvantages of using data classes</a></li>
</ul>
</li>
</ul>
<p>Python <a href="https://docs.python.org/3/library/dataclasses.html">data classes</a> makes it super easy to write better classes by automatically implementing handy dunder methods like <code class="language-plaintext highlighter-rouge">__init__</code>, <code class="language-plaintext highlighter-rouge">__str__</code> (string representation) or <code class="language-plaintext highlighter-rouge">__eq__</code> (equals <code class="language-plaintext highlighter-rouge">==</code> operator). <a href="https://docs.python.org/3/library/dataclasses.html">Data classes</a> also make it easier to create frozen (immutable) instances, serialize instances and enforce type hints usage.</p>
<p>The main parts of a <a href="https://docs.python.org/3/library/dataclasses.html">data class</a> are:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">@dataclass</code> decorator which returns the same defined class but modified</li>
<li><code class="language-plaintext highlighter-rouge">field</code> function which allow for per-field customizations.</li>
</ul>
<blockquote>
<p><strong>Note</strong>: throughout this article we will be using different variations of this <code class="language-plaintext highlighter-rouge">Response</code> class. This class is meant to be a simplified representation of an HTTP response.</p>
</blockquote>
<h2 id="how-to-create-a-data-class">How to create a data class</h2>
<p>To create a data class all we need to do is use the <code class="language-plaintext highlighter-rouge">@dataclass</code> decorator on a custom class like this:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">fom</span> <span class="n">dataclasses</span> <span class="kn">import</span> <span class="nn">dataclass</span>
<span class="o">@</span><span class="n">dataclass</span>
<span class="k">class</span> <span class="nc">Response</span><span class="p">:</span>
<span class="n">status</span><span class="p">:</span> <span class="nb">int</span>
<span class="n">body</span><span class="p">:</span> <span class="nb">str</span>
</code></pre></div></div>
<p>The previous example creates a <code class="language-plaintext highlighter-rouge">Response</code> class with a <code class="language-plaintext highlighter-rouge">status</code> and <code class="language-plaintext highlighter-rouge">body</code> attributes. The <code class="language-plaintext highlighter-rouge">@dataclass</code> decorator by default gives us these benefits:</p>
<ul>
<li>Automatic creation of the following dunder methods:
<ol>
<li><code class="language-plaintext highlighter-rouge">__init__</code></li>
<li><code class="language-plaintext highlighter-rouge">__repr__</code></li>
<li><code class="language-plaintext highlighter-rouge">__eq__</code></li>
<li><code class="language-plaintext highlighter-rouge">__str__</code></li>
</ol>
</li>
<li>Enforcing type hints usage. If a field in a data class is defined without a type hint a <code class="language-plaintext highlighter-rouge">NameError</code> exception is raised.</li>
<li><code class="language-plaintext highlighter-rouge">@dataclass</code> does not create a new class, it returns the same defined class. This allows for anything you could do in a regular class to be valid within a data class.</li>
</ul>
<p>We can appreciate <a href="https://docs.python.org/3/library/dataclasses.html">data classes’</a> benefits by taking a look at the previously defined <code class="language-plaintext highlighter-rouge">Response</code> class.</p>
<p><strong>Instance initialization:</strong></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">resp</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="s">"OK"</span><span class="p">)</span>
</code></pre></div></div>
<p><strong>Correct representation of a class:</strong></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="kn">import</span> <span class="nn">logging</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="p">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">resp</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="s">"OK"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="n">resp</span><span class="p">)</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:</span><span class="n">Response</span><span class="p">(</span><span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="s">'OK'</span><span class="p">)</span>
</code></pre></div></div>
<p><strong>Instance equality</strong></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">resp_ok</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="s">"OK"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">resp_500</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">status</span><span class="o">=</span><span class="mi">500</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="s">"Error"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">resp_200</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="s">"OK"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">resp_ok</span> <span class="o">==</span> <span class="n">resp_500</span>
<span class="p">...</span> <span class="bp">False</span>
<span class="o">>>></span> <span class="n">resp_ok</span> <span class="o">==</span> <span class="n">resp_200</span>
<span class="p">...</span> <span class="bp">True</span>
</code></pre></div></div>
<blockquote>
<p><strong>Note</strong>: we can customize the implementation of each dunder method. We’ll see how later in this article.</p>
</blockquote>
<h2 id="field-definition">Field definition</h2>
<p>There are two ways of defining a field in a data class.</p>
<ol>
<li>Using type hints and an optional default value</li>
</ol>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dstaclass</span>
<span class="o">@</span><span class="n">dataclass</span>
<span class="k">class</span> <span class="nc">Response</span><span class="p">:</span>
<span class="n">body</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">status</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">200</span>
</code></pre></div></div>
<p>The previous class can be instantiated by passing only the <code class="language-plaintext highlighter-rouge">message</code> value or both <code class="language-plaintext highlighter-rouge">status</code> and <code class="language-plaintext highlighter-rouge">message</code></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="kn">import</span> <span class="nn">logging</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="p">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="o">>>></span> <span class="c1"># Create 200 response
</span><span class="o">>>></span> <span class="n">resp_ok</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">"OK"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="n">resp_ok</span><span class="p">)</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:</span><span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">'OK'</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">)</span>
<span class="o">>>></span> <span class="c1"># Create 500 response
</span><span class="o">>>></span> <span class="n">resp_error</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">status</span><span class="o">=</span><span class="mi">500</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="s">"error"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="n">resp_error</span><span class="p">)</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:</span><span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">'error'</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="mi">500</span><span class="p">)</span>
</code></pre></div></div>
<ol>
<li>Using the <code class="language-plaintext highlighter-rouge">field</code> method. This is recommended when there’s a need for more fine grained configuration on a field.</li>
</ol>
<p>By using the <code class="language-plaintext highlighter-rouge">field</code> method we can:</p>
<h3 id="specify-a-default-value">Specify a default value</h3>
<p>When using the <code class="language-plaintext highlighter-rouge">field</code> method we can specify a default value by passing a <code class="language-plaintext highlighter-rouge">default</code> parameter:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dataclass</span>
<span class="o">@</span><span class="n">dataclass</span>
<span class="k">class</span> <span class="nc">Response</span><span class="p">:</span>
<span class="n">body</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">status</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">default</span><span class="o">=</span><span class="mi">200</span><span class="p">)</span>
</code></pre></div></div>
<p>In Python it is not recommended to use <a href="https://docs.python-guide.org/writing/gotchas/#mutable-default-arguments">mutable values as argument defaults</a>. This means it’s not a good idea to define a data class like this (the following example is not valid)</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dataclass</span>
<span class="o">@</span><span class="n">dataclass</span>
<span class="k">class</span> <span class="nc">Response</span><span class="p">:</span>
<span class="n">status</span><span class="p">:</span> <span class="nb">int</span>
<span class="n">body</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">headers</span><span class="p">:</span> <span class="nb">dict</span> <span class="o">=</span> <span class="p">{}</span>
</code></pre></div></div>
<p>If we could use the previous code every instance of <code class="language-plaintext highlighter-rouge">response</code> would share the same <code class="language-plaintext highlighter-rouge">headers</code> object and that’s not good.</p>
<p>Fortunately data classes help us prevent this by raising an error when something like the example above is used. And if we need to add an immutable object as a default value we can use <code class="language-plaintext highlighter-rouge">default_factory</code>.</p>
<p>The <code class="language-plaintext highlighter-rouge">default_factory</code> value should be a function with no arguments. Commonly used functions include <code class="language-plaintext highlighter-rouge">dict</code> or <code class="language-plaintext highlighter-rouge">list</code> :</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dataclass</span><span class="p">,</span> <span class="n">field</span>
<span class="o">@</span><span class="n">dataclass</span>
<span class="k">class</span> <span class="nc">Response</span><span class="p">:</span>
<span class="n">status</span><span class="p">:</span> <span class="nb">int</span>
<span class="n">body</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">headers</span><span class="p">:</span> <span class="nb">dict</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">default_factory</span><span class="o">=</span><span class="nb">dict</span><span class="p">)</span>
</code></pre></div></div>
<p>We can then use this class like so:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="kn">import</span> <span class="nn">logging</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="p">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="o">>>></span> <span class="c1"># Create 200 response
</span><span class="o">>>></span> <span class="n">resp</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="s">"OK"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="n">resp</span><span class="p">)</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:</span><span class="n">Response</span><span class="p">(</span><span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="s">'OK'</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="p">{})</span>
</code></pre></div></div>
<blockquote>
<p><strong>Note</strong>: for mutable default values use <code class="language-plaintext highlighter-rouge">default_factory</code></p>
</blockquote>
<h3 id="include-or-exclude-fields-in-automatically-implemented-dunder-methods">Include or exclude fields in automatically implemented dunder methods</h3>
<p>By default every defined fields are used in <code class="language-plaintext highlighter-rouge">__init__</code>, <code class="language-plaintext highlighter-rouge">__str__</code>, <code class="language-plaintext highlighter-rouge">__repr__</code>, and <code class="language-plaintext highlighter-rouge">__eq__</code>. The <code class="language-plaintext highlighter-rouge">field</code> method allows to specify which fields are used when implementing the following dunder methods:</p>
<p><code class="language-plaintext highlighter-rouge">__init__</code></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dataclass</span>
<span class="o">@</span><span class="n">dataclass</span>
<span class="k">class</span> <span class="nc">Response</span><span class="p">:</span>
<span class="n">body</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">headers</span><span class="p">:</span> <span class="nb">dict</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">init</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">default_factory</span><span class="o">=</span><span class="nb">dict</span><span class="p">)</span>
<span class="n">status</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">200</span>
</code></pre></div></div>
<p>This data class will implement an <code class="language-plaintext highlighter-rouge">__init___</code> method like this one:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">body</span><span class="p">:</span><span class="nb">str</span><span class="p">,</span> <span class="n">status</span><span class="p">:</span> <span class="nb">int</span><span class="o">=</span><span class="mi">200</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">body</span> <span class="o">=</span> <span class="n">body</span>
<span class="bp">self</span><span class="p">.</span><span class="n">status</span> <span class="o">=</span> <span class="n">status</span>
<span class="bp">self</span><span class="p">.</span><span class="n">headers</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">()</span>
</code></pre></div></div>
<p>This version of the <code class="language-plaintext highlighter-rouge">Response</code> class will not allow for a <code class="language-plaintext highlighter-rouge">headers</code> value on initialization. Here’s how we could use it:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="kn">import</span> <span class="nn">logging</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="p">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="o">>>></span> <span class="c1"># Create 200 response
</span><span class="o">>>></span>
<span class="o">>>></span> <span class="n">resp</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">"Success"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="n">resp</span><span class="p">)</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:</span><span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">'Success'</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="p">{},</span> <span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">)</span>
<span class="o">>>></span>
<span class="o">>>></span> <span class="c1"># passing a headers param on initialization will raise an srgument error.
</span><span class="o">>>></span> <span class="n">resp</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">"Success"</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="p">{})</span>
<span class="p">...</span> <span class="nb">TypeError</span><span class="p">:</span> <span class="n">__init__</span><span class="p">()</span> <span class="n">got</span> <span class="n">an</span> <span class="n">unexpected</span> <span class="n">keyword</span> <span class="n">argument</span> <span class="s">'headers'</span>
<span class="o">>>></span>
<span class="o">>>></span> <span class="c1"># 'headers' is an instance attribute and can be used after initialization.
</span><span class="o">>>></span> <span class="n">resp</span><span class="p">.</span><span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="s">"Content-Type"</span><span class="p">:</span> <span class="s">"application/json"</span><span class="p">}</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="n">resp</span><span class="p">)</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:</span><span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">'Success'</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="s">'Content-Type'</span><span class="p">:</span> <span class="s">'application/json'</span><span class="p">},</span> <span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">)</span>
</code></pre></div></div>
<blockquote>
<p><strong>Note</strong>: Fields that are not used in <code class="language-plaintext highlighter-rouge">__init__</code> method can also be populated after init using <code class="language-plaintext highlighter-rouge">__post_init__</code>.</p>
</blockquote>
<p><code class="language-plaintext highlighter-rouge">__repr__</code> and <code class="language-plaintext highlighter-rouge">__str__</code></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dataclass</span>
<span class="o">@</span><span class="n">dataclass</span>
<span class="k">class</span> <span class="nc">Response</span><span class="p">:</span>
<span class="n">body</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">headers</span><span class="p">:</span> <span class="nb">dict</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="nb">repr</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">init</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">default_factory</span><span class="o">=</span><span class="nb">dict</span><span class="p">)</span>
<span class="n">status</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">200</span>
</code></pre></div></div>
<p>Now, the <code class="language-plaintext highlighter-rouge">Response</code> class will not print the value of <code class="language-plaintext highlighter-rouge">headers</code> when an instance is printed.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">resp</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">"Success"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="n">resp</span><span class="p">)</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:</span><span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">'Success'</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">)</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">__eq__</code></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dataclass</span><span class="p">,</span> <span class="n">field</span>
<span class="o">@</span><span class="n">dataclass</span>
<span class="k">class</span> <span class="nc">Response</span><span class="p">:</span>
<span class="n">body</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">headers</span><span class="p">:</span> <span class="nb">dict</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">compare</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">init</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="nb">repr</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">default_factory</span><span class="o">=</span><span class="nb">dict</span><span class="p">)</span>
<span class="n">status</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">200</span>
</code></pre></div></div>
<p>This version of the <code class="language-plaintext highlighter-rouge">Response</code> class will not take the <code class="language-plaintext highlighter-rouge">headers</code> value into consideration when comparing if an instance is equal to another.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">resp_json</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">"Success"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">resp_json</span><span class="p">.</span><span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="s">"Content-Type"</span><span class="p">:</span> <span class="s">"application/json"</span><span class="p">}</span>
<span class="o">>>></span> <span class="n">resp_xml</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">"Success"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">resp_xml</span><span class="p">.</span><span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="s">"Content-Type"</span><span class="p">:</span> <span class="s">"application/xml"</span><span class="p">}</span>
<span class="o">>>></span> <span class="n">resp_json</span> <span class="o">==</span> <span class="n">resp_xml</span>
<span class="p">...</span> <span class="bp">True</span>
</code></pre></div></div>
<p>Both objects are equal because only the <code class="language-plaintext highlighter-rouge">status</code> and <code class="language-plaintext highlighter-rouge">body</code> values are considered when checking for equality and not the <code class="language-plaintext highlighter-rouge">headers</code> value.</p>
<blockquote>
<p><strong>Note</strong>: when setting <code class="language-plaintext highlighter-rouge">compare</code> to <code class="language-plaintext highlighter-rouge">False</code> on a field it will not be used to automatically implement any comparable methods (<code class="language-plaintext highlighter-rouge">__lt__</code>, <code class="language-plaintext highlighter-rouge">__gt__</code>, etc…). More on comparisons later.</p>
</blockquote>
<h3 id="add-field-specific-metadata">Add field-specific metadata</h3>
<p>We can add metadata to a field. The metadata is a mapping and it’s meant to be used by 3rd party libraries. The data classes implementation does not use field metadata at all.</p>
<blockquote>
<p><strong>Note</strong>: If you decide to use field-specific metadata, be mindful, other 3rd party libraries could overwrite any value. It’s recommended to use a specific key to avoid collisions.</p>
</blockquote>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dataclass</span><span class="p">,</span> <span class="n">field</span>
<span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">Any</span>
<span class="o">@</span><span class="n">dataclass</span>
<span class="k">class</span> <span class="nc">Response</span><span class="p">:</span>
<span class="n">body</span><span class="p">:</span> <span class="n">Any</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">metadata</span><span class="o">=</span><span class="p">{</span><span class="s">"force_str"</span><span class="p">:</span> <span class="bp">True</span><span class="p">})</span>
<span class="n">headers</span><span class="p">:</span> <span class="nb">dict</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">init</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="nb">repr</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">default_factory</span><span class="o">=</span><span class="nb">dict</span><span class="p">)</span>
<span class="n">status</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">200</span>
</code></pre></div></div>
<p>This <code class="language-plaintext highlighter-rouge">Response</code> class assigns a mapping with the key <code class="language-plaintext highlighter-rouge">force_str</code> as metadata. The metadata mapping can be used as configuration to force using the string representation of whatever is passed as <code class="language-plaintext highlighter-rouge">body</code>.</p>
<p>To access a field’s metadata the <code class="language-plaintext highlighter-rouge">fields</code> method can be used.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">fields</span>
<span class="o">>>></span> <span class="n">resp</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">"Success"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">fields</span><span class="p">(</span><span class="n">resp</span><span class="p">)</span>
<span class="p">...(</span><span class="n">Field</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s">'body'</span><span class="p">,</span><span class="nb">type</span><span class="o">=</span><span class="n">typing</span><span class="p">.</span><span class="n">Any</span><span class="p">,</span><span class="n">default</span><span class="o">=<</span><span class="n">dataclasses</span><span class="p">.</span><span class="n">_MISSING_TYPE</span> <span class="nb">object</span> <span class="n">at</span> <span class="mh">0x7f955a0e97f0</span><span class="o">></span><span class="p">,</span><span class="n">default_factory</span><span class="o">=<</span><span class="n">dataclasses</span><span class="p">.</span><span class="n">_MISSING_TYPE</span> <span class="nb">object</span> <span class="n">at</span> <span class="mh">0x7f955a0e97f0</span><span class="o">></span><span class="p">,</span><span class="n">init</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span><span class="nb">repr</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span><span class="nb">hash</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span><span class="n">compare</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span><span class="n">metadata</span><span class="o">=</span><span class="n">mappingproxy</span><span class="p">({</span><span class="s">'force_str'</span><span class="p">:</span> <span class="bp">True</span><span class="p">}),</span><span class="n">_field_type</span><span class="o">=</span><span class="n">_FIELD</span><span class="p">),</span>
<span class="n">Field</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s">'headers'</span><span class="p">,</span><span class="nb">type</span><span class="o">=<</span><span class="k">class</span> <span class="err">'</span><span class="nc">dict</span><span class="s">'>,default=<dataclasses._MISSING_TYPE object at 0x7f955a0e97f0>,default_factory=<class '</span><span class="nb">dict</span><span class="s">'>,init=False,repr=False,hash=None,compare=True,metadata=mappingproxy({}),_field_type=_FIELD),
Field(name='</span><span class="n">status</span><span class="s">',type=<class '</span><span class="nb">int</span><span class="s">'>,default=200,default_factory=<dataclasses._MISSING_TYPE object at 0x7f955a0e97f0>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),_field_type=_FIELD))
</span></code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">fields</code> method returns a tuple of <code class="language-plaintext highlighter-rouge">Fields</code> objects. It can be used on an instance or a class.</p>
<p>To retrieve the <code class="language-plaintext highlighter-rouge">body</code> field we can use a comprehension and <code class="language-plaintext highlighter-rouge">next</code></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">body_field</span> <span class="o">=</span> <span class="nb">next</span><span class="p">(</span>
<span class="p">(</span><span class="n">field</span>
<span class="k">for</span> <span class="n">field</span> <span class="ow">in</span> <span class="n">fields</span><span class="p">(</span><span class="n">resp</span><span class="p">)</span>
<span class="k">if</span> <span class="n">field</span><span class="p">.</span><span class="n">name</span> <span class="o">==</span> <span class="s">"body"</span><span class="p">),</span>
<span class="bp">None</span>
<span class="p">)</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="n">body_field</span><span class="p">)</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:</span><span class="n">Field</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s">'body'</span><span class="p">,</span><span class="nb">type</span><span class="o">=</span><span class="n">typing</span><span class="p">.</span><span class="n">Any</span><span class="p">,</span><span class="n">default</span><span class="o">=<</span><span class="n">dataclasses</span><span class="p">.</span><span class="n">_MISSING_TYPE</span> <span class="nb">object</span> <span class="n">at</span> <span class="mh">0x7f955a0e97f0</span><span class="o">></span><span class="p">,</span><span class="n">default_factory</span><span class="o">=<</span><span class="n">dataclasses</span><span class="p">.</span><span class="n">_MISSING_TYPE</span> <span class="nb">object</span> <span class="n">at</span> <span class="mh">0x7f955a0e97f0</span><span class="o">></span><span class="p">,</span><span class="n">init</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span><span class="nb">repr</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span><span class="nb">hash</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span><span class="n">compare</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span><span class="n">metadata</span><span class="o">=</span><span class="n">mappingproxy</span><span class="p">({</span><span class="s">'force_str'</span><span class="p">:</span> <span class="bp">True</span><span class="p">}),</span><span class="n">_field_type</span><span class="o">=</span><span class="n">_FIELD</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="n">body_field</span><span class="p">.</span><span class="n">metadata</span><span class="p">)</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:{</span><span class="s">'force_str'</span><span class="p">:</span> <span class="bp">True</span><span class="p">}</span>
</code></pre></div></div>
<h2 id="customize-object-initialization-using-__post_init__">Customize object initialization using <code class="language-plaintext highlighter-rouge">__post_init__</code></h2>
<p>The <code class="language-plaintext highlighter-rouge">@dataclass</code> decorator automatically implements an <code class="language-plaintext highlighter-rouge">__init__</code> method. By using <code class="language-plaintext highlighter-rouge">__post_init__</code> we can add custom logic on initialization without having to re-implement <code class="language-plaintext highlighter-rouge">__init__</code>.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dataclass</span><span class="p">,</span> <span class="n">field</span>
<span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">Any</span>
<span class="kn">from</span> <span class="nn">sys</span> <span class="kn">import</span> <span class="n">getsizeof</span>
<span class="o">@</span><span class="n">dataclass</span>
<span class="k">class</span> <span class="nc">Response</span><span class="p">:</span>
<span class="n">body</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">headers</span><span class="p">:</span> <span class="nb">dict</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">init</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">compare</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">default_factory</span><span class="o">=</span><span class="nb">dict</span><span class="p">)</span>
<span class="n">status</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">200</span>
<span class="k">def</span> <span class="nf">__post_init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">"""Add a Content-Length header on init"""</span>
<span class="bp">self</span><span class="p">.</span><span class="n">headers</span><span class="p">[</span><span class="s">"Content-Length"</span><span class="p">]</span> <span class="o">=</span> <span class="n">getsizeof</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">body</span><span class="p">)</span>
</code></pre></div></div>
<p>When the previous class is instantiated the content length is automatically calculated.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="kn">import</span> <span class="nn">logging</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="p">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="o">>>></span> <span class="c1"># Create 200 response
</span><span class="o">>>></span> <span class="n">resp</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="s">"Success"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="n">resp</span><span class="p">)</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:</span><span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">'Success'</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="s">'Content-Length'</span><span class="p">:</span> <span class="mi">56</span><span class="p">},</span> <span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">)</span>
</code></pre></div></div>
<p>We can also access field specific metadata in <code class="language-plaintext highlighter-rouge">__post_init__</code></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dataclass</span><span class="p">,</span> <span class="n">is_dstaclass</span><span class="p">,</span> <span class="n">asdict</span>
<span class="o">@</span><span class="n">dataclass</span>
<span class="k">class</span> <span class="nc">Response</span><span class="p">:</span>
<span class="n">body</span><span class="p">:</span> <span class="n">Any</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">metadata</span><span class="o">=</span><span class="p">{</span><span class="s">"force_str"</span><span class="p">:</span> <span class="bp">True</span><span class="p">})</span>
<span class="n">headers</span><span class="p">:</span> <span class="nb">dict</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">init</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">compare</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">default_factory</span><span class="o">=</span><span class="nb">dict</span><span class="p">)</span>
<span class="n">status</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">200</span>
<span class="k">def</span> <span class="nf">stringify_body</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">"""Returns a string representation of the value in body"""</span>
<span class="n">body</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">body</span>
<span class="k">if</span> <span class="n">is_dataclass</span><span class="p">(</span><span class="n">body</span><span class="p">):</span>
<span class="n">body</span> <span class="o">=</span> <span class="n">asdict</span><span class="p">(</span><span class="n">body</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">body</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span>
<span class="k">return</span> <span class="n">json</span><span class="p">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">body</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">body</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">str</span><span class="p">(</span><span class="n">body</span><span class="p">)</span>
<span class="k">return</span> <span class="n">body</span>
<span class="k">def</span> <span class="nf">__post_init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">"""Custom int logic.
- Check if body is configured to force value as string
- Calculate body's length and add corresponding header.
"""</span>
<span class="n">body_field</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">__dataclass_fields__</span><span class="p">[</span><span class="s">"body"</span><span class="p">]</span>
<span class="k">if</span> <span class="n">body_field</span><span class="p">.</span><span class="n">metadata</span><span class="p">[</span><span class="s">"force_str"</span><span class="p">]:</span>
<span class="bp">self</span><span class="p">.</span><span class="n">body</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">stringify_body</span><span class="p">()</span>
<span class="bp">self</span><span class="p">.</span><span class="n">headers</span><span class="p">[</span><span class="s">"Content-Length"</span><span class="p">]</span> <span class="o">=</span> <span class="n">getsizeof</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">body</span><span class="p">)</span>
</code></pre></div></div>
<p>And we can use this class like this:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="kn">import</span> <span class="nn">logging</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="p">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="o">>>></span> <span class="c1"># Create 200 response
</span><span class="o">>>></span> <span class="n">resp</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="p">{</span><span class="s">"message"</span><span class="p">:</span> <span class="s">"Success"</span><span class="p">})</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="n">resp</span><span class="p">)</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:</span><span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">'{"message": "Success"}'</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="s">'Content-Length'</span><span class="p">:</span> <span class="mi">71</span><span class="p">},</span> <span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">)</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">body</code> value is automatically serialized into a string and stored in the calss on initialization.</p>
<p>The previous example is mainly to show custom initialization logic. In reality you might not want to store the string representation of a response body, instead it’s better to make the class serializable.</p>
<blockquote>
<p><strong>Note</strong>: you can use <code class="language-plaintext highlighter-rouge">asdict</code> to transform a data class into a dictionary, this is useful for string serialization.</p>
</blockquote>
<p>We can also specify fields which will not be attributes of an instance but will be passed onto the <code class="language-plaintext highlighter-rouge">__post_init__</code> hook by using <code class="language-plaintext highlighter-rouge">dataclasses.InitVar</code></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dataclass</span><span class="p">,</span> <span class="n">is_dstaclass</span><span class="p">,</span> <span class="n">asdict</span><span class="p">,</span> <span class="n">InitVar</span>
<span class="o">@</span><span class="n">dataclass</span>
<span class="k">class</span> <span class="nc">Response</span><span class="p">:</span>
<span class="n">body</span><span class="p">:</span> <span class="n">Any</span>
<span class="n">headers</span><span class="p">:</span> <span class="nb">dict</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">init</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">compare</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">default_factory</span><span class="o">=</span><span class="nb">dict</span><span class="p">)</span>
<span class="n">status</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">200</span>
<span class="n">force_body_str</span><span class="p">:</span> <span class="n">InitVar</span><span class="p">[</span><span class="nb">bool</span><span class="p">]</span> <span class="o">=</span> <span class="bp">True</span>
<span class="k">def</span> <span class="nf">stringify_body</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">"""Returns a string representation of the value in body"""</span>
<span class="n">body</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">body</span>
<span class="k">if</span> <span class="n">is_dataclass</span><span class="p">(</span><span class="n">body</span><span class="p">):</span>
<span class="n">body</span> <span class="o">=</span> <span class="n">asdict</span><span class="p">(</span><span class="n">body</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">body</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span>
<span class="k">return</span> <span class="n">json</span><span class="p">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">body</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">body</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">str</span><span class="p">(</span><span class="n">body</span><span class="p">)</span>
<span class="k">return</span> <span class="n">body</span>
<span class="k">def</span> <span class="nf">__post_init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">force_body_str</span><span class="p">):</span>
<span class="s">"""Custom int logic.
- Check if body is configured to force value as string
- Calculate body's length and add corresponding header.
"""</span>
<span class="k">if</span> <span class="n">force_body_str</span><span class="p">:</span>
<span class="bp">self</span><span class="p">.</span><span class="n">body</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">stringify_body</span><span class="p">()</span>
<span class="bp">self</span><span class="p">.</span><span class="n">headers</span><span class="p">[</span><span class="s">"Content-Length"</span><span class="p">]</span> <span class="o">=</span> <span class="n">getsizeof</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">body</span><span class="p">)</span>
</code></pre></div></div>
<p>We can easily configure if the value of <code class="language-plaintext highlighter-rouge">body</code> will be stored as string or not:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="kn">import</span> <span class="nn">logging</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="p">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="o">>>></span> <span class="c1"># Create 200 response where 'body' will be stored as a dict.
</span><span class="o">>>></span> <span class="n">resp</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="p">{</span><span class="s">"message"</span><span class="p">:</span> <span class="s">"Success"</span><span class="p">},</span> <span class="n">force_body_str</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="n">resp</span><span class="p">)</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:</span><span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="p">{</span><span class="s">'message'</span><span class="p">:</span> <span class="s">'Success'</span><span class="p">},</span> <span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="s">'Content-Length'</span><span class="p">:</span> <span class="mi">232</span><span class="p">},</span> <span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">)</span>
<span class="o">>>></span> <span class="c1"># Create 200 response where 'body' will be stored as a string.
</span><span class="o">>>></span> <span class="n">resp_str</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="p">{</span><span class="s">"message"</span><span class="p">:</span> <span class="s">"Success"</span><span class="p">})</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="n">resp</span><span class="p">)</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:</span><span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">'{"message": "Success"}'</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="s">'Content-Length'</span><span class="p">:</span> <span class="mi">71</span><span class="p">},</span> <span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">)</span>
</code></pre></div></div>
<h2 id="data-classes-that-we-can-compare-and-order">Data classes that we can compare and order</h2>
<p>By default a data class implements <code class="language-plaintext highlighter-rouge">__eq__</code>. We can pass an <code class="language-plaintext highlighter-rouge">order</code> boolean argument to the <code class="language-plaintext highlighter-rouge">@dataclass</code> decorator to also implement <code class="language-plaintext highlighter-rouge">__lt__</code> (less than), <code class="language-plaintext highlighter-rouge">__le__</code> (less or equal), <code class="language-plaintext highlighter-rouge">__gt__</code> (greater than) and <code class="language-plaintext highlighter-rouge">__ge__</code> (greater or equal).</p>
<p>The way these rich <a href="https://docs.python.org/3/reference/datamodel.html#object.__lt__">comparison methods</a> are implemented take every defined field and compare them in the order they are defined until there’s a value that’s not equal.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dataclass</span>
<span class="o">@</span><span class="n">dataclass</span><span class="p">(</span><span class="n">order</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Response</span><span class="p">:</span>
<span class="n">body</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">status</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">200</span>
</code></pre></div></div>
<p>The previous data class can now be compared using <code class="language-plaintext highlighter-rouge">>=</code>, <code class="language-plaintext highlighter-rouge"><=</code>, <code class="language-plaintext highlighter-rouge">></code> and <code class="language-plaintext highlighter-rouge"><</code> operands. The best use case for this is when sorting:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">resp_ok</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">"Success"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">resp_error</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">"Error"</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="mi">500</span><span class="p">)</span>
<span class="o">>>></span> <span class="nb">sorted</span><span class="p">([</span><span class="n">resp_ok</span><span class="p">,</span> <span class="n">resp_error</span><span class="p">])</span>
<span class="p">...</span> <span class="p">[</span><span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">'Error'</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="mi">500</span><span class="p">),</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">'Success'</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">)]</span>
</code></pre></div></div>
<p>In this example <code class="language-plaintext highlighter-rouge">resp_error</code> goes before <code class="language-plaintext highlighter-rouge">resp_ok</code> because the unicode value of <code class="language-plaintext highlighter-rouge">E</code> is less than the unicode value of <code class="language-plaintext highlighter-rouge">S</code>.</p>
<blockquote>
<p><strong>Note</strong>: implementing rich comparison methods allow us to easily sort objects.</p>
</blockquote>
<p>The implemented comparison methods will check the value of <code class="language-plaintext highlighter-rouge">body</code>, if both are equal it will continue to <code class="language-plaintext highlighter-rouge">status</code>. If the class had more fields the rest of the fields would be checked in order until a non-equal value is found.</p>
<p>The previous example is valid but it does not make much sense to sort <code class="language-plaintext highlighter-rouge">Response</code> objects based on the <code class="language-plaintext highlighter-rouge">body</code> and <code class="language-plaintext highlighter-rouge">status</code> values. It makes more sense to sort them on the length of the body. We can specify which fields to use in comparison by using the <code class="language-plaintext highlighter-rouge">field</code> method:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dataclass</span><span class="p">,</span> <span class="n">field</span>
<span class="kn">from</span> <span class="nn">sys</span> <span class="kn">import</span> <span class="n">getsizeof</span>
<span class="o">@</span><span class="n">dataclass</span><span class="p">(</span><span class="n">order</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Response</span><span class="p">:</span>
<span class="n">body</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">compare</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="n">status</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">compare</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">200</span><span class="p">)</span>
<span class="n">_content_length</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">compare</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">init</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__post_init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">"""Calculate and store content length on init"""</span>
<span class="bp">self</span><span class="p">.</span><span class="n">_content_length</span> <span class="o">=</span> <span class="n">getsizeof</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">body</span><span class="p">)</span>
</code></pre></div></div>
<p>In the previous example we specified which fields are used when implementing comparison methods by passing a boolean <code class="language-plaintext highlighter-rouge">compare</code> parameter to the <code class="language-plaintext highlighter-rouge">field</code> method.</p>
<p>This class will now be sorted by the size of the value of <code class="language-plaintext highlighter-rouge">body</code>. We can also judge if an instance is larger than another judging by the size of the value of <code class="language-plaintext highlighter-rouge">body</code>.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">resp_ok</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">"Success"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">resp_error</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">"Error"</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="mi">500</span><span class="p">)</span>
<span class="o">>>></span> <span class="nb">sorted</span><span class="p">([</span><span class="n">resp_ok</span><span class="p">,</span> <span class="n">re</span> <span class="n">sp_error</span><span class="p">])</span>
<span class="p">...[</span><span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">'Error'</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="mi">500</span><span class="p">,</span> <span class="n">_content_length</span><span class="o">=</span><span class="mi">54</span><span class="p">),</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">'Success'</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">,</span> <span class="n">_content_length</span><span class="o">=</span><span class="mi">56</span><span class="p">)]</span>
<span class="o">>>></span> <span class="c1"># resp_error is smaller than resp_ok because
</span><span class="o">>>></span> <span class="c1"># "Error" is smaller than "Success"
</span></code></pre></div></div>
<p>One downside of this implementation is that two given instances will be equal as long as the size of the <code class="language-plaintext highlighter-rouge">body</code> attribute is the same.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">resp_ok</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">"Success"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">resp_error</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">"Failure"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">resp_ok</span> <span class="o">==</span> <span class="n">resp_error</span>
<span class="p">...</span> <span class="bp">True</span>
<span class="o">>>></span> <span class="c1"># both instances are equal because Success and Failure have the same amounts of chars
</span><span class="o">>>></span> <span class="c1"># and getsizeof() returns the same size for both strings.
</span></code></pre></div></div>
<p>For equality it would be better to also check if the value of the <code class="language-plaintext highlighter-rouge">body</code> attribute is the same:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dataclass</span><span class="p">,</span> <span class="n">field</span>
<span class="kn">from</span> <span class="nn">sys</span> <span class="kn">import</span> <span class="n">getsizeof</span>
<span class="o">@</span><span class="n">dataclass</span><span class="p">(</span><span class="n">order</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Response</span><span class="p">:</span>
<span class="n">_content_length</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">compare</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">init</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="n">body</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">compare</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">status</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">compare</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">200</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__post_init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">"""Calculate and store content length on init"""</span>
<span class="bp">self</span><span class="p">.</span><span class="n">_content_length</span> <span class="o">=</span> <span class="n">getsizeof</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">body</span><span class="p">)</span>
</code></pre></div></div>
<p>By moving the <code class="language-plaintext highlighter-rouge">_content_length</code> field definition above <code class="language-plaintext highlighter-rouge">body</code> the length of the content will be used first for any comparisons. We also set the <code class="language-plaintext highlighter-rouge">body</code> field as a <code class="language-plaintext highlighter-rouge">compare</code> field. When checking for equality if the content length is the same the actual value of <code class="language-plaintext highlighter-rouge">body</code> will be checked, making for a better way to check for equality.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">resp_ok</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">"Success"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">resp_error</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">"Failure"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">resp_ok</span> <span class="o">==</span> <span class="n">resp_error</span>
<span class="p">...</span> <span class="bp">False</span>
</code></pre></div></div>
<p>This also works for sorting since response instances with the same content length will be sorted by the weight of the characters. Sorting will always yield the same order.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="nb">sorted</span><span class="p">([</span><span class="n">resp_ok</span><span class="p">,</span> <span class="n">resp_error</span><span class="p">])</span>
<span class="p">...</span> <span class="p">[</span><span class="n">Response</span><span class="p">(</span><span class="n">_content_length</span><span class="o">=</span><span class="mi">56</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="s">'Failure'</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">),</span> <span class="n">Response</span><span class="p">(</span><span class="n">_content_length</span><span class="o">=</span><span class="mi">56</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="s">'Success'</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">)]</span>
</code></pre></div></div>
<blockquote>
<p><strong>Note</strong>: the order in which fields are defined matter for comparisons.</p>
</blockquote>
<h2 id="frozen-or-immutable-instances">Frozen (or immutable) instances</h2>
<p>We can create frozen instances by passing <code class="language-plaintext highlighter-rouge">frozen=True</code> to the <code class="language-plaintext highlighter-rouge">@dataclass</code> decorator.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dataclass</span><span class="p">,</span> <span class="n">field</span>
<span class="o">@</span><span class="n">dataclass</span><span class="p">(</span><span class="n">frozen</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Response</span><span class="p">:</span>
<span class="n">body</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">status</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">200</span>
</code></pre></div></div>
<p>This is helpful when you want to make sure read-only data is not mistakenly modified by your code or 3rd party libraries. If you try to modify a value a <code class="language-plaintext highlighter-rouge">FrozenInstanceError</code> exception will be raised:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">resp_ok</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">"Success"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">resp_ok</span><span class="p">.</span><span class="n">body</span> <span class="o">=</span> <span class="s">"Done!"</span>
<span class="p">...</span> <span class="n">dataclasses</span><span class="p">.</span><span class="n">FrozenInstanceError</span> <span class="n">cannot</span> <span class="n">assign</span> <span class="n">to</span> <span class="n">field</span> <span class="s">'body'</span>
</code></pre></div></div>
<p>In Python we cannot really have <a href="https://discuss.python.org/t/immutability-in-python-is-really-hard/2536">immutable objects</a>. If you make an effort you can still modify a frozen data class instance:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="kn">import</span> <span class="nn">logging</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="p">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="o">>>></span> <span class="c1"># Check values of 'resp_ok'
</span><span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="n">resp_ok</span><span class="p">)</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:</span><span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">'Success'</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">)</span>
<span class="o">>>></span>
<span class="o">>>></span> <span class="nb">object</span><span class="p">.</span><span class="n">__setattr__</span><span class="p">(</span><span class="n">resp_ok</span><span class="p">,</span> <span class="s">"body"</span><span class="p">,</span> <span class="s">"Done!"</span><span class="p">)</span>
<span class="o">>>></span> <span class="c1"># We have modified a "frozen" instance
</span><span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="n">resp_ok</span><span class="p">)</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:</span><span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">'Done!'</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">)</span>
</code></pre></div></div>
<p>This is unlikely to happen but it’s worth knowing.</p>
<blockquote>
<p><strong>Note</strong>: use a frozen data class when using read-only data to avoid unwanted side-effects.</p>
</blockquote>
<blockquote>
<p><strong>Note</strong>: You cannot implement <code class="language-plaintext highlighter-rouge">__post_init__</code> hook in a frozen data class.</p>
</blockquote>
<h2 id="updating-an-object-instance-by-replacing-the-entire-object">Updating an object instance by replacing the entire object.</h2>
<p>The data classes module also offers a <code class="language-plaintext highlighter-rouge">replace</code> method which created a new instance using the same class. Any updates are passed as parameters:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dataclass</span><span class="p">,</span> <span class="n">replace</span>
<span class="o">@</span><span class="n">dataclass</span><span class="p">(</span><span class="n">frozen</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Response</span><span class="p">:</span>
<span class="n">body</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">status</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">200</span>
<span class="o">>>></span> <span class="kn">import</span> <span class="nn">logging</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="p">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="o">>>></span> <span class="c1"># Create 200 response
</span><span class="o">>>></span> <span class="n">resp_ok</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">"Success"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="n">resp_ok</span><span class="p">)</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:</span><span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">'Success'</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">)</span>
<span class="o">>>></span> <span class="c1"># Replace instance
</span><span class="o">>>></span> <span class="n">resp_ok</span> <span class="o">=</span> <span class="n">replace</span><span class="p">(</span><span class="n">resp_ok</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="s">"OK"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="n">resp_ok</span><span class="p">)</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:</span><span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">'OK'</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">)</span>
</code></pre></div></div>
<p>The value of <code class="language-plaintext highlighter-rouge">body</code> is updated and the value of <code class="language-plaintext highlighter-rouge">status</code> is copied over. Any reference to <code class="language-plaintext highlighter-rouge">resp_ok</code> is now pointing to the new, updated object.</p>
<blockquote>
<p><strong>Note</strong>: using <code class="language-plaintext highlighter-rouge">replace</code> ensures <code class="language-plaintext highlighter-rouge">_init_</code> and <code class="language-plaintext highlighter-rouge">__post_init__</code> are run with the updated values.</p>
</blockquote>
<h2 id="adding-class-attributes">Adding class attributes</h2>
<p>In Python a class can have a <a href="https://realpython.com/lessons/class-and-instance-attributes/">class attribute</a>, the difference from instance attributes are mainly these two:</p>
<ol>
<li>Class attribute are defined outside <code class="language-plaintext highlighter-rouge">__init__</code></li>
<li>Every instance of the class will share the same value of a class attribute.</li>
</ol>
<p>We can define class attributes in a data class by using the pseudo-field <code class="language-plaintext highlighter-rouge">typing.ClassVar</code></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dataclass</span>
<span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">ClassVar</span><span class="p">,</span> <span class="n">Any</span>
<span class="kn">from</span> <span class="nn">sys</span> <span class="kn">import</span> <span class="n">getsizeof</span>
<span class="kn">from</span> <span class="nn">collections.abc</span> <span class="kn">import</span> <span class="n">Callable</span>
<span class="o">@</span><span class="n">dataclass</span>
<span class="k">class</span> <span class="nc">Response</span><span class="p">:</span>
<span class="n">body</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">_content_length</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">default</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">init</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="n">status</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">200</span>
<span class="n">getsize_fun</span><span class="p">:</span> <span class="n">ClassVar</span><span class="p">[</span><span class="n">Callable</span><span class="p">[[</span><span class="n">Any</span><span class="p">],</span> <span class="nb">int</span><span class="p">]]</span> <span class="o">=</span> <span class="n">getsizeof</span>
<span class="k">def</span> <span class="nf">__post_init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">"""Calculate content length by using getsize_fun"""</span>
<span class="bp">self</span><span class="p">.</span><span class="n">_content_length</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">getsize_fun</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">body</span><span class="p">)</span>
</code></pre></div></div>
<p>In this version of <code class="language-plaintext highlighter-rouge">Response</code> we can specify a function used to calculate the content’s size. By default <code class="language-plaintext highlighter-rouge">sys.getsizeof</code> is used.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">functools</span> <span class="kn">import</span> <span class="nb">reduce</span>
<span class="k">def</span> <span class="nf">calc_str_unicode_weight</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">string</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
<span class="s">"""Calculates strn weight by adding each character's unicode value"""</span>
<span class="k">return</span> <span class="nb">reduce</span><span class="p">(</span><span class="k">lambda</span> <span class="n">weight</span><span class="p">,</span> <span class="n">char</span><span class="p">:</span> <span class="n">weight</span><span class="o">+</span><span class="nb">ord</span><span class="p">(</span><span class="n">char</span><span class="p">),</span> <span class="n">string</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
<span class="o">@</span><span class="n">dataclass</span>
<span class="k">class</span> <span class="nc">ResponseUnicode</span><span class="p">(</span><span class="n">Response</span><span class="p">):</span>
<span class="n">getsize_fun</span><span class="p">:</span> <span class="n">ClassVar</span><span class="p">[</span><span class="n">Callable</span><span class="p">[[</span><span class="n">Any</span><span class="p">],</span> <span class="nb">int</span><span class="p">]]</span> <span class="o">=</span> <span class="n">calc_str_unicode_weight</span>
<span class="o">>>></span> <span class="kn">import</span> <span class="nn">logging</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="p">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="o">>>></span> <span class="c1"># Create 200 response, using getsizeof to calculate content length
</span><span class="o">>>></span> <span class="n">resp_ok</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">"Success"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="n">resp_ok</span><span class="p">)</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:</span><span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">'Success'</span><span class="p">,</span> <span class="n">_content_length</span><span class="o">=</span><span class="mi">56</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">)</span>
<span class="o">>>></span> <span class="c1"># Override function to use when calculating content length
</span><span class="o">>>></span> <span class="n">resp_ok_unicode</span> <span class="o">=</span> <span class="n">ResponseUnicode</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">"Success"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="n">resp_ok_unicode</span><span class="p">)</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:</span><span class="n">ResponseUnicode</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">'Success'</span><span class="p">,</span> <span class="n">_content_length</span><span class="o">=</span><span class="mi">729</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">)</span>
</code></pre></div></div>
<p>To overwrite the functino used to calculate the content lenght we subclass <code class="language-plaintext highlighter-rouge">Response</code> and pass the function we want as <code class="language-plaintext highlighter-rouge">getsize_fun</code></p>
<blockquote>
<p><strong>Note</strong>: fields that use <code class="language-plaintext highlighter-rouge">ClassVar</code> are not used in data class mechanism like <code class="language-plaintext highlighter-rouge">__init__</code>, equality or comparison dunder methods.</p>
</blockquote>
<h2 id="inheritance-in-data-classes">Inheritance in data classes</h2>
<p>When using inheritance with data classes fields are merged, meaning child classes can overwrite field definitions. Everything else works the same since the <code class="language-plaintext highlighter-rouge">@dataclass</code> decorator returns an old regular Python class.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dataclasses</span>
<span class="o">@</span><span class="n">dataclass</span>
<span class="k">class</span> <span class="nc">Response</span><span class="p">:</span>
<span class="n">body</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">status</span><span class="p">:</span> <span class="nb">int</span>
<span class="n">headers</span><span class="p">:</span> <span class="nb">dict</span>
<span class="o">@</span><span class="n">dataclass</span>
<span class="k">class</span> <span class="nc">JSONResponse</span><span class="p">(</span><span class="n">Response</span><span class="p">):</span>
<span class="n">status</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">200</span>
<span class="n">headers</span><span class="p">:</span> <span class="nb">dict</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">default_factory</span><span class="o">=</span><span class="nb">dict</span><span class="p">,</span> <span class="n">init</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__post_init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">"""automatically add Content-Type header"""</span>
<span class="bp">self</span><span class="p">.</span><span class="n">headers</span><span class="p">[</span><span class="s">"Content-Type"</span><span class="p">]</span> <span class="o">=</span> <span class="s">"application/json"</span>
</code></pre></div></div>
<p>In the previous example the parent class <code class="language-plaintext highlighter-rouge">Response</code> defined the basic fields and the children class <code class="language-plaintext highlighter-rouge">JSONResponse</code> overwrites the <code class="language-plaintext highlighter-rouge">headers</code> field and sets a default value for the <code class="language-plaintext highlighter-rouge">status</code> field.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="kn">import</span> <span class="nn">logging</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="p">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="o">>>></span> <span class="c1"># Create 200 response
</span><span class="o">>>></span> <span class="n">resp_ok</span> <span class="o">=</span> <span class="n">JSONResponse</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="n">json</span><span class="p">.</span><span class="n">dumps</span><span class="p">({</span><span class="s">"message"</span><span class="p">:</span> <span class="s">"OK"</span><span class="p">}))</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="n">resp_ok</span><span class="p">)</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:</span><span class="n">JSONResponse</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">'{"message": "OK"}'</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="s">'Content-Type'</span><span class="p">:</span> <span class="s">'application/json'</span><span class="p">})</span>
</code></pre></div></div>
<h2 id="hash-able-object">Hash-able object</h2>
<p>The <code class="language-plaintext highlighter-rouge">@dataclass</code> decorator will automatically <a href="https://docs.python.org/3/reference/datamodel.html#object.__hash__">implement <code class="language-plaintext highlighter-rouge">__hash__</code> method</a> if the parameters <code class="language-plaintext highlighter-rouge">frozen</code> and <code class="language-plaintext highlighter-rouge">eq</code> are <code class="language-plaintext highlighter-rouge">True</code>. <code class="language-plaintext highlighter-rouge">frozen</code> is <code class="language-plaintext highlighter-rouge">False</code> by default and <code class="language-plaintext highlighter-rouge">eq</code> is <code class="language-plaintext highlighter-rouge">True</code> by default.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dataclass</span>
<span class="o">@</span><span class="n">dataclass</span><span class="p">(</span><span class="n">frozen</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Response</span><span class="p">:</span>
<span class="n">body</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">status</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">200</span>
</code></pre></div></div>
<p>We can now use any instance of this class as a key in a <code class="language-plaintext highlighter-rouge">dict</code> or in a <code class="language-plaintext highlighter-rouge">set</code>. For I stance, we can create a mapping of responses to users</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="kn">import</span> <span class="nn">logging</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="p">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="o">>>></span> <span class="c1"># Create 200 response
</span><span class="o">>>></span> <span class="n">resp_ok</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">"Success"</span><span class="p">)</span>
<span class="o">>>></span> <span class="c1"># Create 500 response
</span><span class="o">>>></span> <span class="n">resp_error</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="s">"Error"</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="mi">500</span><span class="p">)</span>
<span class="o">>>></span> <span class="c1"># Create a mapping of response -> usernames
</span><span class="o">>>></span> <span class="n">responses_to_users</span> <span class="o">=</span> <span class="p">{</span>
<span class="p">...</span> <span class="n">resp_ok</span><span class="p">:</span> <span class="p">[</span><span class="s">"j_mccain"</span><span class="p">,</span> <span class="s">"a_perez"</span><span class="p">],</span>
<span class="p">...</span> <span class="n">resp_error</span><span class="p">:</span> <span class="p">[</span><span class="s">"d_dane"</span><span class="p">,</span> <span class="s">"b_rodriguez"</span><span class="p">]</span>
<span class="p">...</span> <span class="p">}</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="n">responses_to_users</span><span class="p">[</span><span class="n">resp_ok</span><span class="p">])</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:[</span><span class="s">'j_mccain'</span><span class="p">,</span> <span class="s">'a_perez'</span><span class="p">]</span>
</code></pre></div></div>
<p>We can force a <code class="language-plaintext highlighter-rouge">__hash__</code> function implementation even if we don’t set <code class="language-plaintext highlighter-rouge">frozen</code> and <code class="language-plaintext highlighter-rouge">eq</code> to <code class="language-plaintext highlighter-rouge">True</code> by passing <code class="language-plaintext highlighter-rouge">force_hash=True</code> to the <code class="language-plaintext highlighter-rouge">@dataclass</code> decorator. This should only be used if you are 100% sure you need the functionality.</p>
<h2 id="a-use-case-for-data-classes">A use case for data classes</h2>
<p>Throughout this article we’ve made different updates to a <code class="language-plaintext highlighter-rouge">Response</code> class which represents a simplified HTTP response object. Let’s put everything together.</p>
<p>For simplicity we’re gonna write every class and function we’re going to use in the same file, in really these should be spread out into sensible modules.</p>
<blockquote>
<p><strong>Note</strong>: This is still not a 100% real HTTP response class, but it has enough information to see how we can use data classes</p>
</blockquote>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">from</span> <span class="nn">sys</span> <span class="kn">import</span> <span class="n">getsizeof</span>
<span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">ClassVar</span><span class="p">,</span> <span class="n">Any</span><span class="p">,</span> <span class="n">Optional</span><span class="p">,</span> <span class="n">cast</span>
<span class="kn">from</span> <span class="nn">collections.abc</span> <span class="kn">import</span> <span class="n">Callable</span>
<span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dataclass</span><span class="p">,</span> <span class="n">field</span><span class="p">,</span> <span class="n">InitVar</span><span class="p">,</span> <span class="n">asdict</span>
<span class="n">logging</span><span class="p">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="p">.</span><span class="n">DEBUG</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">APIException</span><span class="p">(</span><span class="nb">Exception</span><span class="p">):</span>
<span class="s">"""Custom API exception."""</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">message</span> <span class="o">=</span> <span class="n">message</span><span class="p">;</span>
<span class="bp">self</span><span class="p">.</span><span class="n">data</span><span class="p">:</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">:</span> <span class="n">Any</span><span class="p">]</span> <span class="o">=</span> <span class="n">kwargs</span>
<span class="k">class</span> <span class="nc">PositiveNumberValidator</span><span class="p">:</span>
<span class="s">"""Descriptor to make sure a value is a positive number"""</span>
<span class="k">def</span> <span class="nf">__set_name__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">owner</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">name</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"_</span><span class="si">{</span><span class="n">name</span> <span class="si">}</span><span class="s">"</span>
<span class="k">def</span> <span class="nf">__get__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">obj</span><span class="p">,</span> <span class="n">objtype</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">getattr</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span> <span class="bp">self</span><span class="p">.</span><span class="n">name</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__set__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">obj</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">validate</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
<span class="nb">setattr</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span> <span class="bp">self</span><span class="p">.</span><span class="n">name</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">validate</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="nb">int</span><span class="p">):</span>
<span class="k">raise</span> <span class="nb">AttributeError</span><span class="p">(</span><span class="sa">f</span><span class="s">"value of '</span><span class="si">{</span><span class="bp">self</span><span class="p">.</span><span class="n">name</span><span class="si">}</span><span class="s">' must be a number"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">value</span> <span class="o"><</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">raise</span> <span class="nb">AttributeError</span><span class="p">(</span><span class="sa">f</span><span class="s">"value of '</span><span class="si">{</span><span class="bp">self</span><span class="p">.</span><span class="n">name</span><span class="si">}</span><span class="s">'' must be a positive number"</span><span class="p">)</span>
<span class="o">@</span><span class="n">dataclass</span><span class="p">(</span><span class="n">eq</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Pager</span><span class="p">:</span>
<span class="s">"""Pager class.
This class is to hold any pager related data sent in the response.
The prev and next paramteres are meant to be links sent in the 'Link' header.
"""</span>
<span class="n">page</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="n">cast</span><span class="p">(</span><span class="nb">int</span><span class="p">,</span> <span class="n">PositiveNumberValidator</span><span class="p">())</span>
<span class="n">prev</span><span class="p">:</span> <span class="nb">str</span>
<span class="nb">next</span><span class="p">:</span> <span class="nb">str</span>
<span class="o">@</span><span class="n">dataclass</span><span class="p">(</span><span class="n">order</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">HTTPResponse</span><span class="p">:</span>
<span class="s">"""Parent HTTPResponse
This class can:
1. Ordered and compared by its content size and body using regular operators
2. Pass a 'content_type' string to be used as header value.
3. Update headers directly as a regular dictionary.
4. Customize the function to calculate content-length.
"""</span>
<span class="n">_content_length</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">init</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="n">body</span><span class="p">:</span> <span class="n">Any</span>
<span class="n">pager</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">Pager</span><span class="p">]</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">default</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">compare</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="n">headers</span><span class="p">:</span> <span class="nb">dict</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">default_factory</span><span class="o">=</span><span class="nb">dict</span><span class="p">,</span> <span class="n">init</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">compare</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="n">status</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">default</span><span class="o">=</span><span class="mi">200</span><span class="p">,</span> <span class="n">compare</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="n">content_type</span><span class="p">:</span> <span class="n">InitVar</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="s">"text/html"</span>
<span class="n">getsize_fun</span><span class="p">:</span> <span class="n">ClassVar</span><span class="p">[</span><span class="n">Callable</span><span class="p">[[</span><span class="n">Any</span><span class="p">],</span> <span class="nb">int</span><span class="p">]]</span> <span class="o">=</span> <span class="n">getsizeof</span>
<span class="k">def</span> <span class="nf">__post_init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">content_type</span><span class="p">):</span>
<span class="s">"""automatically calculate header values."""</span>
<span class="bp">self</span><span class="p">.</span><span class="n">_content_length</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">getsize_fun</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">body</span><span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">headers</span><span class="p">[</span><span class="s">"Content-Length"</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">_content_length</span>
<span class="bp">self</span><span class="p">.</span><span class="n">headers</span><span class="p">[</span><span class="s">"Content-Type"</span><span class="p">]</span> <span class="o">=</span> <span class="n">content_type</span>
<span class="o">@</span><span class="n">dataclass</span><span class="p">(</span><span class="n">frozen</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">JSONBody</span><span class="p">:</span>
<span class="s">"""Class to hold data sent in a JSON Response.
This class is immutable to avoid any unwanted modification of data.
"""</span>
<span class="n">message</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">data</span><span class="p">:</span> <span class="nb">dict</span>
<span class="o">@</span><span class="nb">classmethod</span>
<span class="k">def</span> <span class="nf">from_exc</span><span class="p">(</span><span class="n">cls</span><span class="p">:</span> <span class="nb">type</span><span class="p">,</span> <span class="n">exc</span><span class="p">:</span> <span class="n">APIException</span><span class="p">):</span>
<span class="s">"""Initializes a JSONBody object from an exception."""</span>
<span class="k">return</span> <span class="n">cls</span><span class="p">(</span><span class="n">message</span><span class="o">=</span><span class="nb">str</span><span class="p">(</span><span class="n">exc</span><span class="p">),</span> <span class="n">data</span><span class="o">=</span><span class="nb">getattr</span><span class="p">(</span><span class="n">exc</span><span class="p">,</span> <span class="s">"data"</span><span class="p">,</span> <span class="p">{}))</span>
<span class="o">@</span><span class="n">dataclass</span>
<span class="k">class</span> <span class="nc">JSONResponse</span><span class="p">(</span><span class="n">HTTPResponse</span><span class="p">):</span>
<span class="s">"""Class to represent a JSON Response. Child of HTTPResponse."""</span>
<span class="n">body</span><span class="p">:</span> <span class="n">JSONBody</span>
<span class="n">content_type</span><span class="p">:</span> <span class="n">InitVar</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="s">"application/json"</span>
</code></pre></div></div>
<p>Here, we created a parent class <code class="language-plaintext highlighter-rouge">HTTPResponse</code> which will hold the basic data needed to send an HTTP response. Then, we created a <code class="language-plaintext highlighter-rouge">JSONResponse</code> class which inherits from <code class="language-plaintext highlighter-rouge">HTTPResponse</code> and overwrites <code class="language-plaintext highlighter-rouge">body</code> and <code class="language-plaintext highlighter-rouge">content_type</code> attributes. Overwriting these attributes allow us to specify a different default content type and a different type for the <code class="language-plaintext highlighter-rouge">body</code>. There is also a <code class="language-plaintext highlighter-rouge">Pager</code> class which is used to hold any data related to pagination that’s sent in the response. The <code class="language-plaintext highlighter-rouge">Pager</code> class uses a descriptor to validate that <code class="language-plaintext highlighter-rouge">page</code> is always a positive number. And we also have a <code class="language-plaintext highlighter-rouge">JSONBody</code> which can be initialized by passing a <code class="language-plaintext highlighter-rouge">message</code> and a dictionary for <code class="language-plaintext highlighter-rouge">data</code> or can be initialized by passing an <code class="language-plaintext highlighter-rouge">APIException</code> instance. <code class="language-plaintext highlighter-rouge">APIException</code> is a custom exception we created to store an exception message and also some data related to said exception.</p>
<p><strong>Here’s some basic examples how we can use these classes</strong>:</p>
<ol>
<li>We can create a <code class="language-plaintext highlighter-rouge">JSONResponse</code> only by passing a <code class="language-plaintext highlighter-rouge">JSONBody</code> instance:
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="o">>>></span> <span class="kn">import</span> <span class="nn">logging</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="p">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">body</span> <span class="o">=</span> <span class="n">JSONBody</span><span class="p">(</span><span class="n">message</span><span class="o">=</span><span class="s">"Success"</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="p">{</span><span class="s">"values"</span><span class="p">:</span> <span class="p">[</span><span class="s">"value1"</span><span class="p">,</span> <span class="s">"value2"</span><span class="p">]})</span>
<span class="o">>>></span> <span class="n">resp</span> <span class="o">=</span> <span class="n">JSONResponse</span><span class="p">(</span><span class="n">body</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="s">"resp: %s"</span><span class="p">,</span> <span class="n">resp</span><span class="p">)</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:</span><span class="n">resp</span><span class="p">:</span> <span class="n">JSONResponse</span><span class="p">(</span><span class="n">_content_length</span><span class="o">=</span><span class="mi">48</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="n">JSONBody</span><span class="p">(</span><span class="n">message</span><span class="o">=</span><span class="s">'Success'</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="p">{</span><span class="s">'values'</span><span class="p">:</span> <span class="p">[</span><span class="s">'value1'</span><span class="p">,</span> <span class="s">'value2'</span><span class="p">]}),</span> <span class="n">pager</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="s">'Content-Length'</span><span class="p">:</span> <span class="mi">48</span><span class="p">,</span> <span class="s">'Content-Type'</span><span class="p">:</span> <span class="s">'application/json'</span><span class="p">},</span> <span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">)</span>
</code></pre></div> </div>
<p>This is already a powerful class and the code needed to implement it is quite short</p>
</li>
<li>We can also pass a <code class="language-plaintext highlighter-rouge">Pager</code> instance to make it a more robust response:
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="o">>>></span> <span class="n">pager</span> <span class="o">=</span> <span class="n">Pager</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">prev</span><span class="o">=</span><span class="s">"?prev=0"</span><span class="p">,</span> <span class="nb">next</span><span class="o">=</span><span class="s">"?next=2"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">resp</span> <span class="o">=</span> <span class="n">JSONResponse</span><span class="p">(</span><span class="n">body</span><span class="p">,</span> <span class="n">pager</span><span class="o">=</span><span class="n">pager</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="s">"resp: %s"</span><span class="p">,</span> <span class="n">resp</span><span class="p">)</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:</span><span class="n">resp</span><span class="p">:</span> <span class="n">JSONResponse</span><span class="p">(</span><span class="n">_content_length</span><span class="o">=</span><span class="mi">48</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="n">JSONBody</span><span class="p">(</span><span class="n">message</span><span class="o">=</span><span class="s">'Success'</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="p">{</span><span class="s">'values'</span><span class="p">:</span> <span class="p">[</span><span class="s">'value1'</span><span class="p">,</span> <span class="s">'value2'</span><span class="p">]}),</span> <span class="n">pager</span><span class="o">=</span><span class="n">Pager</span><span class="p">(</span><span class="n">prev</span><span class="o">=</span><span class="s">'?prev=0'</span><span class="p">,</span> <span class="nb">next</span><span class="o">=</span><span class="s">'?next=2'</span><span class="p">),</span> <span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="s">'Content-Length'</span><span class="p">:</span> <span class="mi">48</span><span class="p">,</span> <span class="s">'Content-Type'</span><span class="p">:</span> <span class="s">'application/json'</span><span class="p">},</span> <span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">)</span>
</code></pre></div> </div>
</li>
<li>We can easily conver data classes to dictionaries or tuples even when using nested data classes:
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="o">>>></span> <span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">asdict</span><span class="p">,</span> <span class="n">astuple</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="s">"serialized resp: %s"</span><span class="p">,</span> <span class="n">asdict</span><span class="p">(</span><span class="n">resp</span><span class="p">))</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:</span><span class="n">serialized</span> <span class="n">resp</span><span class="p">:</span> <span class="p">{</span><span class="s">'_content_length'</span><span class="p">:</span> <span class="mi">48</span><span class="p">,</span> <span class="s">'body'</span><span class="p">:</span> <span class="p">{</span><span class="s">'message'</span><span class="p">:</span> <span class="s">'Success'</span><span class="p">,</span> <span class="s">'data'</span><span class="p">:</span> <span class="p">{</span><span class="s">'values'</span><span class="p">:</span> <span class="p">[</span><span class="s">'value1'</span><span class="p">,</span> <span class="s">'value2'</span><span class="p">]}},</span> <span class="s">'pager'</span><span class="p">:</span> <span class="p">{</span><span class="s">'prev'</span><span class="p">:</span> <span class="s">'?prev=0'</span><span class="p">,</span> <span class="s">'next'</span><span class="p">:</span> <span class="s">'?next=2'</span><span class="p">},</span> <span class="s">'headers'</span><span class="p">:</span> <span class="p">{</span><span class="s">'Content-Length'</span><span class="p">:</span> <span class="mi">48</span><span class="p">,</span> <span class="s">'Content-Type'</span><span class="p">:</span> <span class="s">'application/json'</span><span class="p">},</span> <span class="s">'status'</span><span class="p">:</span> <span class="mi">200</span><span class="p">}</span>
<span class="o">>>></span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="s">"resp as tuple: %s"</span><span class="p">,</span> <span class="n">astuple</span><span class="p">(</span><span class="n">resp</span><span class="p">))</span>
<span class="p">...</span> <span class="n">INFO</span><span class="p">:</span><span class="n">root</span><span class="p">:</span><span class="n">resp</span> <span class="n">astuple</span><span class="p">:</span> <span class="p">(</span><span class="mi">48</span><span class="p">,</span> <span class="p">(</span><span class="s">'Success'</span><span class="p">,</span> <span class="p">{</span><span class="s">'values'</span><span class="p">:</span> <span class="p">[</span><span class="s">'value1'</span><span class="p">,</span> <span class="s">'value2'</span><span class="p">]}),</span> <span class="p">(</span><span class="s">'?prev=0'</span><span class="p">,</span> <span class="s">'?next=2'</span><span class="p">),</span> <span class="p">{</span><span class="s">'Content-Length'</span><span class="p">:</span> <span class="mi">48</span><span class="p">,</span> <span class="s">'Content-Type'</span><span class="p">:</span> <span class="s">'application/json'</span><span class="p">},</span> <span class="mi">200</span><span class="p">)</span>
</code></pre></div> </div>
</li>
</ol>
<p>With a more real example we can see the strengths and weaknesses of data classes.</p>
<h3 id="benefits-of-using-data-classes">Benefits of using data classes</h3>
<ol>
<li>We can create powerful classes with less code.</li>
<li>Type hints are enforced for every class and instance attribute.</li>
<li>We can customize how special dunder methods are implemented.</li>
<li>We can use data classes in the same way we use regular classes. In the previous example we used descriptors and class methods without an issue.</li>
<li>Inheritance can be used to make it easier to use data classes.</li>
<li>It’s esier to serialize instsances to dictionaries or tuples.</li>
<li>We can mix regular classes and data classes.</li>
</ol>
<h3 id="disadvantages-of-using-data-classes">Disadvantages of using data classes</h3>
<ol>
<li>When creating data classes that can be compared and ordered the order in which you define the fields matters. Read-ability can take a hit because of this. It is recommended to try and separate fields by type. In the <code class="language-plaintext highlighter-rouge">HTTPResponse</code> class we have first private attributes, then instance attributes, init only parameters and class attributes.</li>
<li>Field definition order also matters when using default values. Since <code class="language-plaintext highlighter-rouge">__init__</code> ‘s arguments are implemented using the same order the fields are defined, we have to first define attributes without default values and then attributes with default values.</li>
<li>When using <code class="language-plaintext highlighter-rouge">frozen=True</code> we cannot update values in <code class="language-plaintext highlighter-rouge">__post_init__</code></li>
<li>We have to manually optimize attribute access if needed. Meaning, adding <code class="language-plaintext highlighter-rouge">__slots__</code>. <a href="https://realpython.com/python-data-classes/#optimizing-data-classes">Real Python has a great example of this.</a></li>
</ol>
<p>I hope this article sheds some light on how and when to use data classes. If you like it, please follow this blog and make sure to follow me on <a href="http://twitter.com/rmcomplexity">twitter</a>.</p>rmcomplexityPython classes are powerful. Use dataclasses to create better classes with less code.Introduction to Python’s logging library2020-12-01T23:00:00+00:002020-12-01T23:00:00+00:00https://rmcomplexity.com/article/2020/12/01/introduction-to-python-logging<ul id="markdown-toc">
<li><a href="#the-basics" id="markdown-toc-the-basics">The basics</a> <ul>
<li><a href="#python-logger-structure" id="markdown-toc-python-logger-structure">Python logger structure</a></li>
</ul>
</li>
<li><a href="#what-is-basicconfig" id="markdown-toc-what-is-basicconfig">What is <code class="language-plaintext highlighter-rouge">basicConfig</code>?</a></li>
<li><a href="#custom-loggers" id="markdown-toc-custom-loggers">Custom Loggers</a></li>
<li><a href="#how-to-configure-loggers-formatters-filters-and-handlers" id="markdown-toc-how-to-configure-loggers-formatters-filters-and-handlers">How to configure loggers, formatters, filters and handlers</a> <ul>
<li><a href="#python-logging-flow-simplified" id="markdown-toc-python-logging-flow-simplified">Python logging flow simplified</a> <ul>
<li><a href="#every-log-event-created-will-be-processed-except-in-the-following-cases" id="markdown-toc-every-log-event-created-will-be-processed-except-in-the-following-cases">Every log event created will be processed <strong>except in the following cases</strong>:</a></li>
</ul>
</li>
<li><a href="#formatters" id="markdown-toc-formatters">Formatters</a></li>
<li><a href="#filters" id="markdown-toc-filters">Filters</a> <ul>
<li><a href="#custom-filter-example" id="markdown-toc-custom-filter-example">Custom filter example</a></li>
</ul>
</li>
<li><a href="#handlers" id="markdown-toc-handlers">Handlers</a> <ul>
<li><a href="#full-logging-configuration-example" id="markdown-toc-full-logging-configuration-example">Full logging configuration example</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#things-to-lookout-for-when-configuring-your-loggers" id="markdown-toc-things-to-lookout-for-when-configuring-your-loggers">Things to lookout for when configuring your loggers</a></li>
<li><a href="#recommendations-for-better-logging" id="markdown-toc-recommendations-for-better-logging">Recommendations for better logging</a></li>
</ul>
<p>Logging is one of the best way to keep track of what is going on inside your code while
it is running. An error message can tell you details about the state
of the application when an error happens. But proper logging will make it easier to see
the state of the application right before the error happened or the path the data took
before the error happened.</p>
<p><strong>What’s in this article?</strong></p>
<ul>
<li>Basic overview of Python’s logging system.</li>
<li>Explanation of how each logging module interact with each other.</li>
<li>Examples on how to use different types of logging configurations.</li>
<li>Things to look out for</li>
<li>Recommendations for better logging</li>
</ul>
<p>If you want you can jump to a <a href="#what-is-basicconfig">basic configuration</a> example or
a <a href="#full-logging-configuration-example">full fledged</a> example used in an application.</p>
<h2 id="the-basics">The basics</h2>
<p>At its simplest form a log message, in python is called a <a href="https://docs.python.org/3/library/logging.html#logrecord-attributes"><code class="language-plaintext highlighter-rouge">LogRecord</code></a>,
has a string (the message) and a <a href="https://docs.python.org/3/library/logging.html#logging-levels">level</a>. The level can be:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">CRITICAL</code>: Any error that makes the application stop running.</li>
<li><code class="language-plaintext highlighter-rouge">ERROR</code>: General error messages.</li>
<li><code class="language-plaintext highlighter-rouge">WARNING</code>: General warning messages.</li>
<li><code class="language-plaintext highlighter-rouge">INFO</code>: General informational messages.</li>
<li><code class="language-plaintext highlighter-rouge">DEBUG</code>: Any messages needed for debugging.</li>
<li><code class="language-plaintext highlighter-rouge">NOTSET</code>: Default level used in handlers to process any of the above types of messages.</li>
</ul>
<p>Python logging library has different modules. Not every module is necessary to start
logging messages.</p>
<ul>
<li><strong>Loggers:</strong> the object that implements the main logging API. We use a logger to create
logging events by using the corresponding level method, e.g. <code class="language-plaintext highlighter-rouge">my_logger.debug("my message")</code>.
By default there’s a root logger which can be configured using <a href=""><code class="language-plaintext highlighter-rouge">basicConfig</code></a> and we
can also create <a href="#custom-loggers">custom loggers</a>.</li>
<li><strong>Formatters:</strong> <a href="#formatters">these objects</a> are in charge of creating the correct representation of the logging
event. Most of the time we ant to create a human-readable representation but in some cases we
can output a specific format like a json object.</li>
<li><strong>Filters:</strong> We can use <a href="#filters">filters</a> to avoid printing logging events based on more complicated
criteria than logging levels. We can also use filters to modify logging events.</li>
<li><strong>Handlers:</strong> everything comes together in the <a href="#handlers">handlers</a>. A handler defines which formatters
and filter a logger is going to use. A handler is also in charge of sending the loging output
to the corresponding place,
this could be <code class="language-plaintext highlighter-rouge">stdout</code>, <code class="language-plaintext highlighter-rouge">email</code> or almost anythihg else we can think of.</li>
</ul>
<p>Loggers can have filters and handlers. Handlers can have filters and formatters.
We will dive into each one of these parts but it’s a good idea to keep in mind how
they relate with each other. Here’s a visual representation:</p>
<h5 id="python-logger-structure">Python logger structure</h5>
<figure class="img center">
<a href="/assets/images/python_logger_structure.png">
<img src="/assets/images/python_logger_structure.png" style="max-width:740px;" alt="Python logger structure" class="img-responsive" />
<figcaption><em>This reminds me of a cake, because of the layers.</em></figcaption>
</a>
</figure>
<p>Following Python’s philosophies the logging library can be easily used, for example:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">logging</span>
<span class="n">logging</span><span class="p">.</span><span class="n">warning</span><span class="p">(</span><span class="s">"Warning log message."</span><span class="p">)</span>
</code></pre></div></div>
<p>And that’s it. The example above will print a log message with <code class="language-plaintext highlighter-rouge">WARNING</code> level.
If you run the code above you will see this output:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>WARNING:root:Warning log message.
</code></pre></div></div>
<p>The first part of the output (<code class="language-plaintext highlighter-rouge">WARNING:root:</code>), comes from
the default logging configuration. The default configuration
will only print <a href="https://docs.python.org/3/library/logging.html#logging-levels"><code class="language-plaintext highlighter-rouge">WARNING</code> and above levels</a> and will prepend
the log level and the name of the logger – <em>since we haven’t created any loggers
the <code class="language-plaintext highlighter-rouge">root</code> logger is used. More on this later</em> –.</p>
<p>The logger in the example above is the <code class="language-plaintext highlighter-rouge">root</code> logger. This is a logger which is
the parent of every logger that’s created. Meaning, if you configure the <code class="language-plaintext highlighter-rouge">root</code>
logger then you are basically configuring every logger you create.</p>
<h2 id="what-is-basicconfig">What is <code class="language-plaintext highlighter-rouge">basicConfig</code>?</h2>
<p>To quickly configure any loggers used in your application you can use <code class="language-plaintext highlighter-rouge">basicConfig</code>.
This method will by default configure the root logger with a <code class="language-plaintext highlighter-rouge">StreamHandler</code> to print
log messages to <code class="language-plaintext highlighter-rouge">stdout</code> and a default formatter (like the one in the example above).</p>
<p>We can configure the logging format <a href="https://docs.python.org/3/library/logging.html#logging.basicConfig">using <code class="language-plaintext highlighter-rouge">basicConfig</code></a>.
For instance, if we would like to print only the log level, line number, log message
and log <code class="language-plaintext highlighter-rouge">DEBUG</code> events and up we can do this:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">logging</span>
<span class="n">logging</span><span class="p">.</span><span class="n">basicConfig</span><span class="p">(</span>
<span class="nb">format</span><span class="o">=</span><span class="s">"[%(levelname)s]:%(lineno)s - %(message)s"</span><span class="p">,</span>
<span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="p">.</span><span class="n">DEBUG</span>
<span class="p">)</span>
</code></pre></div></div>
<blockquote>
<p><i class="fas fa-bolt"> </i> <strong>Note:</strong> <br />
By using <code class="language-plaintext highlighter-rouge">logging.basicConfig</code> we are configuring the root logger</p>
</blockquote>
<p>For each log event there is an instance of <a href="https://docs.python.org/3/library/logging.html#logrecord-attributes"><code class="language-plaintext highlighter-rouge">LogRecord</code></a>.
We can set the format for our log messages using the <code class="language-plaintext highlighter-rouge">LogRecord</code>
<a href="https://docs.python.org/3/library/logging.html#logrecord-attributes">class’ attributes</a> and <code class="language-plaintext highlighter-rouge">%</code>-style formatting – <em><code class="language-plaintext highlighter-rouge">%</code>-style formatting is still
used to maintain backwards compatibility</em> –.
Since Python 3.2 we can also use <code class="language-plaintext highlighter-rouge">$</code> and <code class="language-plaintext highlighter-rouge">{}</code> style to format messages,
but we have to specify the style we’re using, by default <code class="language-plaintext highlighter-rouge">%</code>-style is used.
The style is configured by using the <code class="language-plaintext highlighter-rouge">style</code> parameter in <code class="language-plaintext highlighter-rouge">basicConfig</code>
(<code class="language-plaintext highlighter-rouge">logging.basicConfig(style='{')</code>).
With the updated configuration we can now see every log level message, for example:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="n">CRITICAL</span><span class="p">]:</span><span class="mi">9</span> <span class="o">-</span> <span class="n">Critical</span> <span class="n">log</span> <span class="n">message</span><span class="p">.</span>
<span class="p">[</span><span class="n">ERROR</span><span class="p">]:</span><span class="mi">10</span> <span class="o">-</span> <span class="n">Error</span> <span class="n">log</span> <span class="n">message</span><span class="p">.</span>
<span class="p">[</span><span class="n">WARNING</span><span class="p">]:</span><span class="mi">11</span> <span class="o">-</span> <span class="nb">Warning</span> <span class="n">log</span> <span class="n">message</span><span class="p">.</span>
<span class="p">[</span><span class="n">INFO</span><span class="p">]:</span><span class="mi">12</span> <span class="o">-</span> <span class="n">Info</span> <span class="n">log</span> <span class="n">message</span><span class="p">.</span>
<span class="p">[</span><span class="n">DEBUG</span><span class="p">]:</span><span class="mi">13</span> <span class="o">-</span> <span class="n">Debug</span> <span class="n">log</span> <span class="n">message</span><span class="p">.</span>
</code></pre></div></div>
<blockquote>
<p><i class="fab fa-python"> </i> <strong>Best Practice</strong> <br />
Define a custom log format for easier log parsing.</p>
</blockquote>
<blockquote>
<p><i class="fas fa-bolt"> </i> <strong>Note:</strong> <br />
Use the <code class="language-plaintext highlighter-rouge">LogRecord</code> <a href="https://docs.python.org/3/library/logging.html#logrecord-attributes">class’ attributes</a> to configure a custom format.</p>
</blockquote>
<p><code class="language-plaintext highlighter-rouge">basicConfig</code> will always be called by the root logger if no handlers are defined,
unless the parameter <code class="language-plaintext highlighter-rouge">force</code> is set to <code class="language-plaintext highlighter-rouge">True</code> (<code class="language-plaintext highlighter-rouge">logging.basicConfig(force=True)</code>).
By default <code class="language-plaintext highlighter-rouge">basicConfig</code> will configure the root logger to output logs to <code class="language-plaintext highlighter-rouge">stdout</code>,
with the default format of <code class="language-plaintext highlighter-rouge">{level}:{logger_name}:{message}</code>.</p>
<h2 id="custom-loggers">Custom Loggers</h2>
<p>Python’s logging library allow us to create custom loggers which will <em>always</em>
be children of the <code class="language-plaintext highlighter-rouge">root</code> logger.</p>
<p>We can create a new logger by using <a href="https://docs.python.org/3/library/logging.html#logging.getLogger"><code class="language-plaintext highlighter-rouge">logging.getLogger("mylogger")</code></a>.
This method only accepts one parameter, the <code class="language-plaintext highlighter-rouge">name</code> of the logger. <code class="language-plaintext highlighter-rouge">getLogger</code> is a
<em>get_or_create</em> method. We can use <code class="language-plaintext highlighter-rouge">getLogger</code> with the same <code class="language-plaintext highlighter-rouge">name</code>
value and we’ll be working with the same logger configuration regardless if
we’re doing this in a different class or module.</p>
<p>The logging library uses dot notation to
create hierarchies of loggers. Meaning, if we create three loggers with the names
<code class="language-plaintext highlighter-rouge">app</code>, <code class="language-plaintext highlighter-rouge">app.models</code> and <code class="language-plaintext highlighter-rouge">app.api</code> the parent logger will be <code class="language-plaintext highlighter-rouge">app</code> and,
<code class="language-plaintext highlighter-rouge">app.db</code> and <code class="language-plaintext highlighter-rouge">app.api</code> will be the children. Here’s a visual representation:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> + app <span class="c"># main logger</span>
|
+ app.api <span class="c"># api logger, child of "app" logger</span>
|
- app.api.routes <span class="c"># routes logger, child of "app" and "app.api" loggers</span>
|
- app.api.models <span class="c"># models logger, sibling of "app.api.routes" logger</span>
|
- app.utils <span class="c"># utils logger, sibling of "app.api" logger</span>
</code></pre></div></div>
<p>We can get a module’s dot
notation name from the global variable <code class="language-plaintext highlighter-rouge">__name__</code>. Using <code class="language-plaintext highlighter-rouge">__name__</code> to create
our custom loggers simplifies configuration and avoids collisions:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># project/api/utils.py
</span>
<span class="kn">import</span> <span class="nn">logging</span>
<span class="c1"># Use __name__ to create module level loggers
</span><span class="n">LOG</span> <span class="o">=</span> <span class="n">logging</span><span class="p">.</span><span class="n">getLogger</span><span class="p">(</span><span class="n">__name__</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">check_if_true</span><span class="p">(</span><span class="n">var</span><span class="p">):</span>
<span class="s">"""Check if variable is `True`.
:param bool var: Variable to check.
:return bool: True if `var` is truthy.
"""</span>
<span class="n">LOG</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="s">"This logger's name is 'project.api.utils'."</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">bool</span><span class="p">(</span><span class="n">var</span><span class="p">)</span>
</code></pre></div></div>
<blockquote>
<p><i class="fas fa-bolt"> </i> <strong>Note:</strong>
If you are defining loggers at the module level (like the example above)
is better to stick to global variable naming. Meaning, use <code class="language-plaintext highlighter-rouge">LOG</code> or <code class="language-plaintext highlighter-rouge">LOGGER</code>
instead of the lowercase version <code class="language-plaintext highlighter-rouge">log</code> or <code class="language-plaintext highlighter-rouge">logger</code>.</p>
</blockquote>
<blockquote>
<p><i class="fab fa-python"> </i> <strong>Best Practice</strong> <br />
Create loggers with custom names using <code class="language-plaintext highlighter-rouge">__name__</code> to avoid collision and for
granular configuration.</p>
</blockquote>
<p>When using a logger the message’s level is checked against the logger’s
<code class="language-plaintext highlighter-rouge">level</code> attribute if it’s the same or above then the log message is passed to the logger
that’s being used and every parent <strong>unless</strong> one of the logger in the hierarchy sets
<code class="language-plaintext highlighter-rouge">propagate</code> to <code class="language-plaintext highlighter-rouge">False</code> – <em>by default <code class="language-plaintext highlighter-rouge">propagate</code> is set to <code class="language-plaintext highlighter-rouge">True</code></em> –.</p>
<h2 id="how-to-configure-loggers-formatters-filters-and-handlers">How to configure loggers, formatters, filters and handlers</h2>
<p>We’ve talked about using <code class="language-plaintext highlighter-rouge">basicConfig</code> to configure a logger but there are <a href="https://docs.python.org/3/howto/logging.html#configuring-logging">other
ways</a> to configure loggers.
The recommended way of creating a logging configuration is using a
<a href="https://docs.python.org/3/library/logging.config.html#logging-config-dictschema"><code class="language-plaintext highlighter-rouge">dictConfig</code></a>. The examples in this article will
show three different variations (code, <code class="language-plaintext highlighter-rouge">fileConfig</code> and <code class="language-plaintext highlighter-rouge">dictConfig</code>),
feel free to use whatever is better for your project.</p>
<p>As we’ve learned before python’s logging
library already comes with some useful default values, which makes defining our own
formatters, filters and handlers optional. As a matter of fact when using a
dictionary to configure logging the only required key is <code class="language-plaintext highlighter-rouge">version</code>, and currently
the only valid value is <code class="language-plaintext highlighter-rouge">1</code>.</p>
<p>Whenever we use a logger (<code class="language-plaintext highlighter-rouge">LOG.debug("Debug log message.")</code>) the first thing that happens
is that a <code class="language-plaintext highlighter-rouge">LogRecord</code> object is created with our log message and other
<a href="https://docs.python.org/3/library/logging.html#logrecord-attributes">attributes</a>. This <code class="language-plaintext highlighter-rouge">LogRecord</code> instance is then passed to any <strong>filters</strong>
attached to the logger instance we used. If the filter does not reject the <code class="language-plaintext highlighter-rouge">LogRecord</code>
instance then the <code class="language-plaintext highlighter-rouge">LogRecord</code> is passed to the configured <strong>handlers</strong>.
If any of the configured <strong>handlers</strong> are enable for the level in the passed <code class="language-plaintext highlighter-rouge">LogRecord</code>
then the <strong>handlers</strong> apply any configured <strong>filters</strong> to the <code class="language-plaintext highlighter-rouge">LogRecord</code>.
Finally, if the <code class="language-plaintext highlighter-rouge">LogRecord</code> is not rejected by any of the <strong>filter</strong> the <code class="language-plaintext highlighter-rouge">LogRecord</code>
is emitted. A more detailed diagram can be seen in <a href="https://docs.python.org/3/howto/logging.html#logging-flow">python’s documentation</a>.</p>
<p>To simplify Python’s logging flow we can focus on what happens in a single logger:</p>
<h4 id="python-logging-flow-simplified">Python logging flow simplified</h4>
<figure class="img center">
<a href="/assets/images/Python_logging_flow_simplified-1.jpg">
<img src="/assets/images/Python_logging_flow_simplified-1.jpg" style="max-width:740px;" alt="Python logging flow simplified" class="img-responsive" />
<figcaption><em>Logger sounds like a cool frogger fork</em></figcaption>
</a>
</figure>
<p>Here’s a few important things to note:</p>
<ul>
<li>The diagram above is from the point of view of the logger used.</li>
<li>Filters, handlers and formatters are defined once and can
be used multiple times.</li>
<li><strong>Only</strong> the filters and formatters assigned to the parent’s
handler are applied to the <code class="language-plaintext highlighter-rouge">LogRecord</code> (this is the loop that says “Parent Loggers*”)</li>
</ul>
<h5 id="every-log-event-created-will-be-processed-except-in-the-following-cases">Every log event created will be processed <strong>except in the following cases</strong>:</h5>
<ol>
<li>The logger or the handler configured in the logger are not enabled
for the log level used.</li>
<li>A filter configured in the logger or the handler rejects
the log event.</li>
<li>A child logger has <code class="language-plaintext highlighter-rouge">propagate=False</code> causing events not to be passed
to any of the parent loggers.</li>
<li>Your are using a different logger or the logger is not a parent of the
one being used.</li>
</ol>
<h3 id="formatters">Formatters</h3>
<p>A <a href="https://docs.python.org/3/library/logging.html#formatter-objects">formatter object</a> transforms a
<code class="language-plaintext highlighter-rouge">LogRecord</code> instance into a human readable string or a string that will be consumed
by an external service. We can use any <a href="https://docs.python.org/3/library/logging.html#logrecord-attributes"><code class="language-plaintext highlighter-rouge">LogRecord</code> attribute</a>
or anything sent in the logging call as the <code class="language-plaintext highlighter-rouge">extra</code> parameter.</p>
<blockquote>
<p><i class="fas fa-bolt"> </i> <strong>Note:</strong>
Formatters can only be set to <strong>handlers</strong></p>
</blockquote>
<p>For example, we can create a formatter to show all the details of where and when
a log message happened:</p>
<h5 class="toggler" data-cls="fmts-dict-config" data-default="true">
Formatters definition using a dictionary
<i class="icon-show fas fa-angle-down"></i>
<i class="icon-hide fas fa-angle-up"></i>
</h5>
<div class="fmts-dict-config">
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">LOGGING_CONFIG</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">"version"</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s">"formatters"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"detailed"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"format"</span><span class="p">:</span> <span class="s">"[APP] %(levelname)s %(asctime)s %(module)s "</span>
<span class="s">"%(name)s.%(funcName)s:%(lineno)s: %(message)s"</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="s">"handlers"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"console"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"class"</span><span class="p">:</span> <span class="s">"logging.StreamHandler"</span><span class="p">,</span>
<span class="s">"formatter"</span><span class="p">:</span> <span class="s">"detailed"</span><span class="p">,</span>
<span class="s">"level"</span><span class="p">:</span> <span class="s">"INFO"</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div> </div>
<p>When using a dictionary to define a formatter we have to use the <code class="language-plaintext highlighter-rouge">"formatters"</code> key.
Any key inside the <code class="language-plaintext highlighter-rouge">"formatters"</code> object will become a formatter. Every inside the
formatter object will be sent as parameters when intializing the formatter instance.</p>
</div>
<h5 class="toggler" data-cls="fmts-code">
Formatters definition using code
<i class="icon-show fas fa-angle-down"></i>
<i class="icon-hide fas fa-angle-up"></i>
</h5>
<div class="fmts-code">
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">logging</span>
<span class="c1"># create Formatter
</span><span class="n">formatter</span> <span class="o">=</span> <span class="n">logging</span><span class="p">.</span><span class="n">Formatter</span><span class="p">(</span>
<span class="s">"[APP] %(levelname)s %(asctime)s %(module)s "</span>
<span class="s">"%(name)s.%(funcName)s:%(lineno)s: %(message)s"</span>
<span class="p">)</span>
<span class="n">stream_handler</span> <span class="o">=</span> <span class="n">logging</span><span class="p">.</span><span class="n">StreamHandler</span><span class="p">(</span><span class="n">sys</span><span class="p">.</span><span class="n">stdout</span><span class="p">)</span>
<span class="n">stream_handler</span><span class="p">.</span><span class="n">setFormatter</span><span class="p">(</span><span class="n">formatter</span><span class="p">)</span>
</code></pre></div> </div>
<p>Initializing a formatter in code is straight forward. We have to remember to assign it
to a handler by using the handler’s <code class="language-plaintext highlighter-rouge">setFormatter</code> method.</p>
</div>
<h5 class="toggler" data-cls="fmts-file">
Formatters definition using a file
<i class="icon-show fas fa-angle-down"></i>
<i class="icon-hide fas fa-angle-up"></i>
</h5>
<div class="fmts-file">
<div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[formatters]</span>
<span class="py">key</span><span class="p">=</span><span class="s">detailed</span>
<span class="nn">[handlers]</span>
<span class="py">key</span><span class="p">=</span><span class="s">console</span>
<span class="nn">[formatter_detailed]</span>
<span class="py">format</span><span class="p">=</span><span class="s">[APP] %(levelname)s %(asctime)s %(module)s %(name)s.%(funcName)s:%(lineno)s: %(message)s</span>
<span class="py">datefmt</span><span class="p">=</span>
<span class="py">class</span><span class="p">=</span><span class="s">logging.Formatter</span>
<span class="nn">[handler_console]</span>
<span class="py">class</span><span class="p">=</span><span class="s">StreamHandler</span>
<span class="py">level</span><span class="p">=</span><span class="s">DEBUG</span>
<span class="py">formatter</span><span class="p">=</span><span class="s">detailed</span>
<span class="py">args</span><span class="p">=</span><span class="s">(sys.stdout,)</span>
</code></pre></div> </div>
<p>In a file configuration we first define the keys of the object we will be referencing.
In this case we create a <code class="language-plaintext highlighter-rouge">detailed</code> formatter. We then configure this formatter by creating a
<code class="language-plaintext highlighter-rouge">formatters_<formatter_key></code> section, here will be <code class="language-plaintext highlighter-rouge">formatter_detailed</code>.
If <code class="language-plaintext highlighter-rouge">datefmt</code> is not defined the default ISO is used.</p>
</div>
<p>Whenever this formatter is used it will print the level, date and time,
module name, function name, line number and any string sent as parameter.
We can add other variables not present in <code class="language-plaintext highlighter-rouge">LogRecord</code>’s attributes by using
the <code class="language-plaintext highlighter-rouge">extra</code> attribute:</p>
<h5 class="toggler" data-cls="fmts-dict-extra" data-default="true">
Formatter with extra atributes using a dictionary
<i class="icon-show fas fa-angle-down"></i>
<i class="icon-hide fas fa-angle-up"></i>
</h5>
<div class="fmts-dict-extra">
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">LOGGING_CONFIG</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">"version"</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s">"formatters"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"short"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"format"</span><span class="p">:</span> <span class="s">"[APP] %(levelname)s %(asctime)s %(module)s "</span>
<span class="s">"%(name)s.%(funcName)s:%(lineno)s: "</span>
<span class="s">"[%(session_key)s:%(user_id)s] %(message)s"</span> <span class="c1"># add extra data
</span> <span class="p">}</span>
<span class="p">},</span>
<span class="c1">#[...]
</span><span class="p">}</span>
</code></pre></div> </div>
</div>
<h5 class="toggler" data-cls="fmts-code-extra">
Formatter with extra atributes using code
<i class="icon-show fas fa-angle-down"></i>
<i class="icon-hide fas fa-angle-up"></i>
</h5>
<div class="fmts-code-extra">
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">logging</span>
<span class="c1"># create Formatter
</span><span class="n">formatter</span> <span class="o">=</span> <span class="n">logging</span><span class="p">.</span><span class="n">Formatter</span><span class="p">(</span>
<span class="s">"[APP] %(levelname)s %(asctime)s %(module)s "</span>
<span class="s">"%(name)s.%(funcName)s:%(lineno)s: "</span>
<span class="s">"[%(session_key)s:%(user_id)s] %(message)s"</span> <span class="c1"># add extra data
</span><span class="p">)</span>
</code></pre></div> </div>
</div>
<h5 class="toggler" data-cls="fmts-file-extra">
Formatter with extra atributes using a dictionary
<i class="icon-show fas fa-angle-down"></i>
<i class="icon-hide fas fa-angle-up"></i>
</h5>
<div class="fmts-file-extra">
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="n">formatters</span><span class="p">]</span>
<span class="n">key</span><span class="o">=</span><span class="n">detailed</span>
<span class="p">[</span><span class="n">handlers</span><span class="p">]</span>
<span class="n">key</span><span class="o">=</span><span class="n">console</span>
<span class="p">[</span><span class="n">formatter_detailed</span><span class="p">]</span>
<span class="nb">format</span><span class="o">=</span><span class="p">[</span><span class="n">APP</span><span class="p">]</span> <span class="o">%</span><span class="p">(</span><span class="n">levelname</span><span class="p">)</span><span class="n">s</span> <span class="o">%</span><span class="p">(</span><span class="n">asctime</span><span class="p">)</span><span class="n">s</span> <span class="o">%</span><span class="p">(</span><span class="n">module</span><span class="p">)</span><span class="n">s</span> <span class="o">%</span><span class="p">(</span><span class="n">name</span><span class="p">)</span><span class="n">s</span><span class="p">.</span><span class="o">%</span><span class="p">(</span><span class="n">funcName</span><span class="p">)</span><span class="n">s</span><span class="p">:</span><span class="o">%</span><span class="p">(</span><span class="n">lineno</span><span class="p">)</span><span class="n">s</span><span class="p">:</span> <span class="p">[</span><span class="o">%</span><span class="p">(</span><span class="n">session_key</span><span class="p">)</span><span class="n">s</span><span class="p">:</span><span class="o">%</span><span class="p">(</span><span class="n">user_id</span><span class="p">)</span><span class="n">s</span><span class="p">]</span> <span class="o">%</span><span class="p">(</span><span class="n">message</span><span class="p">)</span><span class="n">s</span>
<span class="n">datefmt</span><span class="o">=</span>
<span class="n">class</span><span class="o">=</span><span class="n">logging</span><span class="p">.</span><span class="n">Formatter</span>
<span class="p">[</span><span class="n">handler_console</span><span class="p">]</span>
<span class="n">class</span><span class="o">=</span><span class="n">StreamHandler</span>
<span class="n">level</span><span class="o">=</span><span class="n">DEBUG</span>
<span class="n">formatter</span><span class="o">=</span><span class="n">detailed</span>
<span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">sys</span><span class="p">.</span><span class="n">stdout</span><span class="p">,)</span>
</code></pre></div> </div>
</div>
<p>And to send that extra data we can do it like this:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># app.api.UserManager.list_users
</span>
<span class="n">LOG</span><span class="p">.</span><span class="n">info</span><span class="p">(</span>
<span class="s">"Listing users"</span><span class="p">,</span>
<span class="n">extra</span><span class="o">=</span><span class="p">{</span><span class="s">"session_key"</span><span class="p">:</span> <span class="n">session_key</span><span class="p">,</span> <span class="s">"user_id"</span><span class="p">:</span> <span class="n">user_id</span><span class="p">}</span>
<span class="p">)</span>
</code></pre></div></div>
<p>The output would be:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>APP] INFO 2020-11-07 20:47:00,123 user_manager app.api.user_name.list_users:15 <span class="o">[</span>123abcd:api_usr] Listing <span class="nb">users</span>
</code></pre></div></div>
<p>The downside of referencing variables sent via the <code class="language-plaintext highlighter-rouge">extra</code> parameter
in a format is that if the variable is not passed the log event is
not going to be logged because the string cannot be created.</p>
<blockquote>
<p><i class="fab fa-python"> </i> <strong>Best Practice</strong> <br />
Make sure to send every additional variable that the configured
formatter references if using the <code class="language-plaintext highlighter-rouge">extra</code> parameter</p>
</blockquote>
<h3 id="filters">Filters</h3>
<p>Filters are very interesting. They can be used both at the logger level
or at the handler level, they can be used to stop certain log events from
being logged and they can also be used to inject additional context into
a <code class="language-plaintext highlighter-rouge">LogRecord</code> instance which will be, eventually, logged.</p>
<p>The <code class="language-plaintext highlighter-rouge">Filter</code> class in Python’s <code class="language-plaintext highlighter-rouge">logging</code> library filters <code class="language-plaintext highlighter-rouge">LogRecords</code> by
logger name. The filter will allow any <code class="language-plaintext highlighter-rouge">LogRecord</code> coming from the logger name
configured in the filter and any of its children.</p>
<p>For instance, if we have these loggers configured:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> + app.models
| |
| - app.models.users
|
+ app.views
|
- app.views.users
|
- app.views.products
</code></pre></div></div>
<p>And we define the filter config like this:</p>
<h5 class="toggler" data-cls="filters-dict" data-default="true">
Filters definition using a dictionary
<i class="icon-show fas fa-angle-down"></i>
<i class="icon-hide fas fa-angle-up"></i>
</h5>
<div class="filters-dict">
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">LOGGING_CONFIG</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">"version"</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s">"filter"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"views"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"name"</span><span class="p">:</span> <span class="s">"app.views"</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="s">"handlers"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"console"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"class"</span><span class="p">:</span> <span class="s">"logging.StreamHandler"</span><span class="p">,</span>
<span class="s">"level"</span><span class="p">:</span> <span class="s">"INFO"</span><span class="p">,</span>
<span class="s">"filter"</span><span class="p">:</span> <span class="s">"views"</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="s">"loggers"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"app"</span><span class="p">:</span> <span class="p">{</span> <span class="s">"handlers"</span><span class="p">:</span> <span class="p">[</span><span class="s">"console"</span><span class="p">]</span> <span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div> </div>
<p>We use the <code class="language-plaintext highlighter-rouge">"filter"</code> key in a dictionary configuration to define any filters.
Each key inside the <code class="language-plaintext highlighter-rouge">"filter"</code> object will become a filter we can later reference.
Every key and value inside the filter object we create will be sent as parameters
when intializing the filter.</p>
</div>
<h5 class="toggler" data-cls="filters-code">
Filters definition using code
<i class="icon-show fas fa-angle-down"></i>
<i class="icon-hide fas fa-angle-up"></i>
</h5>
<div class="filters-code">
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">logging</span>
<span class="n">views_filter</span> <span class="o">=</span> <span class="n">logging</span><span class="p">.</span><span class="n">Filter</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s">"views"</span><span class="p">)</span>
<span class="n">console_handler</span><span class="p">.</span><span class="n">addFilter</span><span class="p">(</span><span class="n">views_filter</span><span class="p">)</span>
<span class="n">console_logger</span><span class="p">.</span><span class="n">addHandler</span><span class="p">(</span><span class="n">console_handler</span><span class="p">)</span>
</code></pre></div> </div>
<p>Initializing a filter using code is straight forward.
We have to remember to use the handler’s <code class="language-plaintext highlighter-rouge">addFilter</code> so a handler
can use the filter.</p>
</div>
<h5 class="toggler" data-cls="filters-file">
Filters definition using a file
<i class="icon-show fas fa-angle-down"></i>
<i class="icon-hide fas fa-angle-up"></i>
</h5>
<div class="filters-file">
<div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[filters]</span>
<span class="py">key</span><span class="p">=</span><span class="s">views</span>
<span class="nn">[handlers]</span>
<span class="py">key</span><span class="p">=</span><span class="s">console</span>
<span class="nn">[filter_views]</span>
<span class="py">name</span><span class="p">=</span><span class="s">views</span>
<span class="nn">[handler_console]</span>
<span class="py">class</span><span class="p">=</span><span class="s">StreamHandler</span>
<span class="py">level</span><span class="p">=</span><span class="s">DEBUG</span>
<span class="py">filters</span><span class="p">=</span><span class="s">views</span>
<span class="py">args</span><span class="p">=</span><span class="s">(sys.stdout,)</span>
</code></pre></div> </div>
<p>When using a file we have to first define the filter keys that we are going to reference.
To configure a filter we have to create a section with the name <code class="language-plaintext highlighter-rouge">filter_<name_of_filter></code>.
In this case <code class="language-plaintext highlighter-rouge">filter_views</code>.</p>
</div>
<p>The previous configuration will <strong>only</strong> allow <code class="language-plaintext highlighter-rouge">LogRecord</code> coming from the
<code class="language-plaintext highlighter-rouge">app.views</code>, <code class="language-plaintext highlighter-rouge">app.views.users</code> and <code class="language-plaintext highlighter-rouge">app.views.products</code>
loggers.</p>
<p>When you set a filter to a specific logger the filter will <strong>only</strong> be used
when calling that logger directly and <strong>not</strong> when a descendant of said logger
is used. For example, if we had set the filter in the previous example to the
<code class="language-plaintext highlighter-rouge">app.view</code> logger instead of the <code class="language-plaintext highlighter-rouge">app</code> logger handler. The filter will not
reject any <code class="language-plaintext highlighter-rouge">LogRecord</code> coming from <code class="language-plaintext highlighter-rouge">app.models</code> loggers simply because when
using the logger <code class="language-plaintext highlighter-rouge">app.models</code>, or any of it’s childrens, the filter will not be called.</p>
<p>Let’s see how can we create custom
filters to prevent <code class="language-plaintext highlighter-rouge">LogRecords</code> to be processed based on more complicated conditions
and/or to add more data to a <code class="language-plaintext highlighter-rouge">LogRecord</code>.</p>
<p><strong>Since Python 3.2</strong> you can use any callable that accepts a <code class="language-plaintext highlighter-rouge">record</code> parameter</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">custom_filter</span><span class="p">(</span><span class="n">record</span><span class="p">):</span>
<span class="s">"""Filter out records that passes a specific key argument"""</span>
<span class="c1"># We can access the log event's argument via record.args
</span> <span class="k">return</span> <span class="n">record</span><span class="p">.</span><span class="n">args</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"instrumentation"</span><span class="p">)</span> <span class="o">==</span> <span class="s">"console"</span>
</code></pre></div></div>
<p>The way to configure these custom classes/callables using a dictionary or a file
is by using the special keyword <code class="language-plaintext highlighter-rouge">()</code>. Whenever Python’s logging config sees
<code class="language-plaintext highlighter-rouge">()</code> it will create an instance of the class (dot notation has to be used).</p>
<blockquote>
<p><i class="fas fa-bolt"> </i> <strong>Note:</strong>
When using a dictionary to configure logging you can use the <code class="language-plaintext highlighter-rouge">()</code>
keyword to configure custom handlers or filters</p>
</blockquote>
<p>Another interesting thing about filters is that they see virtually every <code class="language-plaintext highlighter-rouge">LogRecord</code>
that <em>might</em> be logged. This makes filters a great place to further customize <code class="language-plaintext highlighter-rouge">LogRecords</code>.
This is called “adding context”.</p>
<h5 id="custom-filter-example">Custom filter example</h5>
<p>The following custom filter will apply a mask to every password passed to a <code class="language-plaintext highlighter-rouge">LogRecord</code>
<strong>if</strong> the <code class="language-plaintext highlighter-rouge">pwd</code> or <code class="language-plaintext highlighter-rouge">password</code> key is used.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># module: app.logging.filters
</span>
<span class="k">def</span> <span class="nf">pwd_mask_filter</span><span class="p">(</span><span class="n">record</span><span class="p">)</span>
<span class="s">"""A factory function that will return a filter callable to use."""</span>
<span class="k">def</span> <span class="nf">filter</span><span class="p">(</span><span class="n">record</span><span class="p">):</span>
<span class="c1"># a Logger cord instance holds all it's arguments in record.args
</span> <span class="k">def</span> <span class="nf">mask_pwd</span><span class="p">(</span><span class="n">record</span><span class="p">):</span>
<span class="k">return</span> <span class="s">'*'</span> <span class="o">*</span> <span class="mi">20</span>
<span class="k">if</span> <span class="n">record</span><span class="p">.</span><span class="n">args</span><span class="p">.</span><span class="n">has_key</span><span class="p">(</span><span class="s">"pwd"</span><span class="p">):</span>
<span class="n">record</span><span class="p">.</span><span class="n">args</span><span class="p">[</span><span class="s">"pwd"</span><span class="p">]</span> <span class="o">=</span> <span class="n">mask_pwd</span><span class="p">()</span>
<span class="k">elif</span> <span class="n">record</span><span class="p">.</span><span class="n">args</span><span class="p">.</span><span class="n">has_key</span><span class="p">(</span><span class="s">"password"</span><span class="p">):</span>
<span class="n">record</span><span class="p">.</span><span class="n">args</span><span class="p">[</span><span class="s">"pwd"</span><span class="p">]</span> <span class="o">=</span> <span class="n">mask_pwd</span><span class="p">()</span>
<span class="k">return</span> <span class="nb">filter</span>
</code></pre></div></div>
<h5 class="toggler" data-cls="filters-custom-dict" data-default="true">
Custom filter configuration example using a dictionary
<i class="icon-show fas fa-angle-down"></i>
<i class="icon-hide fas fa-angle-up"></i>
</h5>
<div class="filters-custom-dict">
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">LOGGING_CONFIG</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">"version"</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s">"formatters"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"detailed"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"format"</span><span class="p">:</span> <span class="s">"[APP] %(levelname)s %(asctime)s %(module)s "</span>
<span class="s">"%(name)s.%(funcName)s:%(lineno)s: %(message)s"</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="s">"filters"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"mask_pwd"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"()"</span><span class="p">:</span> <span class="s">"app.logging.filters.mask_pwd"</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="s">"handlers"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"console"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"class"</span><span class="p">:</span> <span class="s">"logging.StreamHandler"</span><span class="p">,</span>
<span class="s">"formatter"</span><span class="p">:</span> <span class="s">"detailed"</span><span class="p">,</span>
<span class="s">"level"</span><span class="p">:</span> <span class="s">"INFO"</span><span class="p">,</span>
<span class="s">"filters: ["</span><span class="n">mask_pwd</span><span class="s">"]
}
}
}
</span></code></pre></div> </div>
<p>When using <code class="language-plaintext highlighter-rouge">()</code> in a dictionary configuration the referenced module will be imported
and instantiated. In this case the factory function <code class="language-plaintext highlighter-rouge">mask_pwd</code> will be called and
the actual function that handles filtering will be returned.</p>
</div>
<h5 class="toggler" data-cls="filters-custom-code">
Custom filter configuration example using code
<i class="icon-show fas fa-angle-down"></i>
<i class="icon-hide fas fa-angle-up"></i>
</h5>
<div class="filters-custom-code">
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">from</span> <span class="nn">filters</span> <span class="kn">import</span> <span class="n">mask_pwd</span>
<span class="n">stream_handler</span> <span class="o">=</span> <span class="n">logging</span><span class="p">.</span><span class="n">StreamHandler</span><span class="p">(</span><span class="n">sys</span><span class="p">.</span><span class="n">stdout</span><span class="p">)</span>
<span class="n">stream_handler</span><span class="p">.</span><span class="n">addFilter</span><span class="p">(</span><span class="n">mask_pwd</span><span class="p">())</span>
</code></pre></div> </div>
<p>Since we define <code class="language-plaintext highlighter-rouge">mask_pwd</code> as a factory function we have to instantiate it.
This way the handler is using the filter function returned by <code class="language-plaintext highlighter-rouge">mask_pwd</code>.</p>
</div>
<h5 class="toggler" data-cls="filters-custom-file">
Custom filter configuration example using a file
<i class="icon-show fas fa-angle-down"></i>
<i class="icon-hide fas fa-angle-up"></i>
</h5>
<div class="filters-custom-file">
<blockquote>
<p><i class="fas fa-bolt"> </i> <strong>Note:</strong> <br />
We cannot configure filters when using file configuration</p>
</blockquote>
</div>
<h3 id="handlers">Handlers</h3>
<p>Handlers are objects that implement how formatters and filters are used. if we remember the flow diagram we can see
that whenever we use a logger the handlers of all the parent loggers are going to be called recursivley.
This is where the entire picture comes together and we can take a look at a complete loggin configuration
using everything else we’ve talked about.</p>
<p>Python offers a list of very useful <a href="https://docs.python.org/3/howto/logging.html#useful-handlers">handlers</a>. Handler classes have a very simple API and in most
cases we will only use these classes to add a formatter, zero or more filters and setting the logging level.</p>
<blockquote>
<p><i class="fas fa-bolt"> </i> <strong>Note:</strong> <br />
Only <strong>one</strong> formatter can be set to a handler.</p>
</blockquote>
<p>For instance, the following configuration makes sure that every log is printed to the standard output, uses a filter
to select a different handler depending if the application is running in debug mode or not and it uses a different
format for each different environment (based on the debug flag).</p>
<h4 id="full-logging-configuration-example">Full logging configuration example</h4>
<h5 class="toggler" data-cls="full-config-dict" data-default="true">
Configuration using a dict
<i class="icon-show fas fa-angle-down"></i>
<i class="icon-hide fas fa-angle-up"></i>
</h5>
<div class="full-config-dict">
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">LOGGING</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">"version"</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s">"disable_existing_loggers"</span><span class="p">:</span> <span class="bp">False</span><span class="p">,</span>
<span class="s">"formatters"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"long"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"format"</span><span class="p">:</span> <span class="s">"[APP] {levelname} {asctime} {module} "</span>
<span class="s">"{name}.{funcName}:{lineno:d}: {message}"</span><span class="p">,</span>
<span class="s">"style"</span><span class="p">:</span> <span class="s">"{"</span>
<span class="p">},</span>
<span class="s">"short"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"format"</span><span class="p">:</span> <span class="s">"[APP] {levelname} [{asctime}] {message}"</span><span class="p">,</span>
<span class="s">"style"</span><span class="p">:</span> <span class="s">"{"</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="s">"filters"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"debug_true"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"()"</span><span class="p">:</span> <span class="s">"filters.require_debug_true_filter"</span>
<span class="p">},</span>
<span class="s">"debug_false"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"()"</span><span class="p">:</span> <span class="s">"filters.require_debug_false_filter"</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="s">"handlers"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"console"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"level"</span><span class="p">:</span> <span class="s">"INFO"</span><span class="p">,</span>
<span class="s">"class"</span><span class="p">:</span> <span class="s">"logging.StreamHandler"</span><span class="p">,</span>
<span class="s">"formatter"</span><span class="p">:</span> <span class="s">"short"</span><span class="p">,</span>
<span class="s">"filters"</span><span class="p">:</span> <span class="p">[</span><span class="s">"debug_false"</span><span class="p">]</span>
<span class="p">},</span>
<span class="s">"console_debug"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"level"</span><span class="p">:</span> <span class="s">"DEBUG"</span><span class="p">,</span>
<span class="s">"class"</span><span class="p">:</span> <span class="s">"logging.StreamHandler"</span><span class="p">,</span>
<span class="s">"formatter"</span><span class="p">:</span> <span class="s">"long"</span><span class="p">,</span>
<span class="s">"filters"</span><span class="p">:</span> <span class="p">[</span><span class="s">"debug_true"</span><span class="p">]</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="s">"loggers"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"external_library"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"handlers"</span><span class="p">:</span> <span class="p">[</span><span class="s">"console"</span><span class="p">],</span>
<span class="s">"level"</span><span class="p">:</span> <span class="s">"INFO"</span>
<span class="p">},</span>
<span class="s">"app"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"handlers"</span><span class="p">:</span> <span class="p">[</span><span class="s">"console"</span><span class="p">,</span> <span class="s">"console_debug"</span><span class="p">],</span>
<span class="s">"level"</span><span class="p">:</span> <span class="s">"DEBUG"</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div> </div>
</div>
<h5 class="toggler" data-cls="full-config-code">
Configuration using code
<i class="icon-show fas fa-angle-down"></i>
<i class="icon-hide fas fa-angle-up"></i>
</h5>
<div class="full-config-code">
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">import</span> <span class="nn">app.settings</span>
<span class="k">def</span> <span class="nf">require_debug_true_filter</span><span class="p">():</span>
<span class="s">"""Only process records when DEBUG is True"""</span>
<span class="k">return</span> <span class="n">app</span><span class="p">.</span><span class="n">settings</span><span class="p">.</span><span class="n">DEBUG</span>
<span class="k">def</span> <span class="nf">require_debug_false_filter</span><span class="p">():</span>
<span class="s">"""Only process records when DEBUG is False"""</span>
<span class="k">return</span> <span class="err">!</span><span class="n">app</span><span class="p">.</span><span class="n">settings</span><span class="p">.</span><span class="n">DEBUG</span>
<span class="c1"># in app's entry point
</span>
<span class="c1"># create logger
</span><span class="n">app_logger</span> <span class="o">=</span> <span class="n">logging</span><span class="p">.</span><span class="n">getLogger</span><span class="p">(</span><span class="s">"app"</span><span class="p">)</span>
<span class="n">app_logger</span><span class="p">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="p">.</span><span class="n">DEBUG</span><span class="p">)</span>
<span class="n">external_library_logger</span> <span class="o">=</span> <span class="n">logging</span><span class="p">.</span><span class="n">getLogger</span><span class="p">(</span><span class="s">"external_library"</span><span class="p">)</span>
<span class="n">external_library_logger</span><span class="p">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="p">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="c1"># long formatter
</span><span class="n">long_fmt</span> <span class="o">=</span> <span class="n">logging</span><span class="p">.</span><span class="n">Formatter</span><span class="p">(</span>
<span class="s">"[APP] {levelname} {asctime} {module} {name}.{funcName}:{lineno:d}: {message}"</span><span class="p">,</span>
<span class="n">style</span><span class="o">=</span><span class="s">"{"</span>
<span class="p">)</span>
<span class="c1"># short formatter
</span><span class="n">short_fmt</span> <span class="o">=</span> <span class="n">logging</span><span class="p">.</span><span class="n">Formatter</span><span class="p">(</span>
<span class="s">"[APP] {levelname} [{asctime}] {message}"</span><span class="p">,</span>
<span class="n">style</span><span class="o">=</span><span class="s">"{"</span>
<span class="p">)</span>
<span class="c1"># console handler config
</span><span class="n">console_handler</span> <span class="o">=</span> <span class="n">logging</span><span class="p">.</span><span class="n">StreamHandler</span><span class="p">()</span>
<span class="n">console_handler</span><span class="p">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="p">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="n">console_handler</span><span class="p">.</span><span class="n">setFormatter</span><span class="p">(</span><span class="n">short_fmt</span><span class="p">)</span>
<span class="n">console_handler</span><span class="p">.</span><span class="n">addFilter</span><span class="p">(</span><span class="n">require_debug_false_filter</span><span class="p">)</span>
<span class="c1"># console debug handler config
</span><span class="n">console_debug_handler</span> <span class="o">=</span> <span class="n">logging</span><span class="p">.</span><span class="n">StreamHandler</span><span class="p">()</span>
<span class="n">console_debug_handler</span><span class="p">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="p">.</span><span class="n">DEBUG</span><span class="p">)</span>
<span class="n">console_debug_handler</span><span class="p">.</span><span class="n">setFormatter</span><span class="p">(</span><span class="n">long_fmt</span><span class="p">)</span>
<span class="n">console_debug_handler</span><span class="p">.</span><span class="n">addFilter</span><span class="p">(</span><span class="n">require_debug_true_filter</span><span class="p">)</span>
<span class="n">app_logger</span><span class="p">.</span><span class="n">addHandler</span><span class="p">(</span><span class="n">console_handler</span><span class="p">)</span>
<span class="n">app_logger</span><span class="p">.</span><span class="n">addHandler</span><span class="p">(</span><span class="n">console_debug_handler</span><span class="p">)</span>
<span class="n">external_library_logger</span><span class="p">.</span><span class="n">addHandler</span><span class="p">(</span><span class="n">console_handler</span><span class="p">)</span>
</code></pre></div> </div>
</div>
<h5 class="toggler" data-cls="full-config-file">
Configuration using a file
<i class="icon-show fas fa-angle-down"></i>
<i class="icon-hide fas fa-angle-up"></i>
</h5>
<div class="full-config-file">
<div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[loggers]</span>
<span class="py">keys</span><span class="p">=</span><span class="s">root,external_library,app</span>
<span class="nn">[handlers]</span>
<span class="py">keys</span><span class="p">=</span><span class="s">console,console_debug</span>
<span class="nn">[formatters]</span>
<span class="py">keys</span><span class="p">=</span><span class="s">long,short</span>
<span class="nn">[formatter_long]</span>
<span class="py">format</span><span class="p">=</span><span class="s">[APP] {levelname} {asctime} {module} {name}.{funcName}:{lineno:d}: {message}</span>
<span class="py">style</span><span class="p">=</span><span class="s">{</span>
<span class="py">datefmt</span><span class="p">=</span>
<span class="py">class</span><span class="p">=</span><span class="s">logging.Formatter</span>
<span class="nn">[formatter_short]</span>
<span class="py">format</span><span class="p">=</span><span class="s">[APP] {levelname} [{asctime}] {message}</span>
<span class="py">style</span><span class="p">=</span><span class="s">{</span>
<span class="py">datefmt</span><span class="p">=</span>
<span class="py">class</span><span class="p">=</span><span class="s">logging.Formatter</span>
<span class="nn">[handler_console]</span>
<span class="py">class</span><span class="p">=</span><span class="s">StreamHandler</span>
<span class="py">level</span><span class="p">=</span><span class="s">INFO</span>
<span class="py">formatter</span><span class="p">=</span><span class="s">short</span>
<span class="py">args</span><span class="p">=</span><span class="s">(sys.stdout,)</span>
<span class="nn">[handler_console_debug]</span>
<span class="py">class</span><span class="p">=</span><span class="s">StreamHandler</span>
<span class="py">level</span><span class="p">=</span><span class="s">DEBUG</span>
<span class="py">formatter</span><span class="p">=</span><span class="s">long</span>
<span class="py">args</span><span class="p">=</span><span class="s">(sys.stdout,)</span>
<span class="nn">[logger_root]</span>
<span class="py">level</span><span class="p">=</span><span class="s">NOTSET</span>
<span class="py">handlers</span><span class="p">=</span><span class="s">console</span>
<span class="nn">[logger_external_library]</span>
<span class="py">level</span><span class="p">=</span><span class="s">INFO</span>
<span class="py">handlers</span><span class="p">=</span><span class="s">console</span>
<span class="py">propagate</span><span class="p">=</span><span class="s">1</span>
<span class="py">qualname</span><span class="p">=</span><span class="s">external_library</span>
<span class="nn">[logger_app]</span>
<span class="py">level</span><span class="p">=</span><span class="s">DEBUG</span>
<span class="py">handlers</span><span class="p">=</span><span class="s">console,console_debug</span>
<span class="py">qualname</span><span class="p">=</span><span class="s">app</span>
</code></pre></div> </div>
</div>
<p>When using a dictionary configuration <code class="language-plaintext highlighter-rouge">disable_existing_loggers</code> is set to <code class="language-plaintext highlighter-rouge">True</code> by default.
What this does is that it will grab all the loggers created before the configuration is applied
and disable them so they cannot process any log events. This is specially important when
creating loggers at the module level.</p>
<p>Say we have a <code class="language-plaintext highlighter-rouge">app.models.client</code> module in which we create a logger like this:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># app.models.client
</span>
<span class="kn">import</span> <span class="nn">logging</span>
<span class="n">LOG</span> <span class="o">=</span> <span class="n">logging</span><span class="p">.</span><span class="n">getLogger</span><span class="p">(</span><span class="n">__name__</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Client</span><span class="p">:</span>
<span class="c1"># ...
</span></code></pre></div></div>
<p>And then in our application’s entry point we import our models and configure our logging:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># app.__main__
</span>
<span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">from</span> <span class="nn">app.settings.logging</span> <span class="kn">import</span> <span class="n">LOGGING_CONFIG</span> <span class="c1"># A dictionary
</span><span class="kn">from</span> <span class="nn">app.models</span> <span class="kn">import</span> <span class="n">Client</span>
<span class="n">logging</span><span class="p">.</span><span class="n">config</span><span class="p">.</span><span class="n">dictConfig</span><span class="p">(</span><span class="n">LOGGING_CONFIG</span><span class="p">)</span>
</code></pre></div></div>
<p>In this case the logger created in <code class="language-plaintext highlighter-rouge">app.models.client</code> will not work unless <code class="language-plaintext highlighter-rouge">disable_existing_loggers</code>.
Because when we import the <code class="language-plaintext highlighter-rouge">Client</code> class from <code class="language-plaintext highlighter-rouge">app.models</code> we are initializing the logger used
in <code class="language-plaintext highlighter-rouge">app.models.client</code>. Then, after the logger is already initialized we apply our configuration, making
any existing loggers disabled.</p>
<p>When definig a handler using a dictionary, any key that is not <code class="language-plaintext highlighter-rouge">class</code>,
<code class="language-plaintext highlighter-rouge">level</code>, <code class="language-plaintext highlighter-rouge">formatter</code> or <code class="language-plaintext highlighter-rouge">filters</code> will be passed as a parameter to
the handler’s class specified for instantiation.</p>
<p>Filters cannot be used when using a file to configure logging. One more reason
why is better to use a dictionary for configuration.</p>
<h2 id="things-to-lookout-for-when-configuring-your-loggers">Things to lookout for when configuring your loggers</h2>
<ul>
<li>Filters can be set both to Loggers and Handlers. If a filter is set at the logger level
it will only be used when that specific logger is used and not when any of its descendants are used.</li>
<li>File config is still support purely for backwards compatibility. Avoid using file config.</li>
<li><code class="language-plaintext highlighter-rouge">()</code> can be used in config dictionaries to specify a custom class/callable to use as
formatter, filter or handler. It’s assumed a factory is referenced when using <code class="language-plaintext highlighter-rouge">()</code> to allow for complete
initialization control.</li>
<li>Go over the <a href="#every-log-event-created-will-be-processed-except-in-the-following-cases">list of cases</a> when
a log event is not processed if you don’t see what you expect in your logs.</li>
<li>Make sure to set <code class="language-plaintext highlighter-rouge">disable_existing_loggers</code> to <code class="language-plaintext highlighter-rouge">False</code> when using dict config and creating loggers
at the module level.</li>
</ul>
<h2 id="recommendations-for-better-logging">Recommendations for better logging</h2>
<ul>
<li>Take some time to plan how your logs should look like and where are they going to be stored.</li>
<li>Make sure your logs are easily parsable both visually and programmatically, like using <code class="language-plaintext highlighter-rouge">grep</code>.</li>
<li>Most of the time you don’t need to set a filter at the logger level.</li>
<li>Filters can be used to add extra context to log events. Make sure any custom filters are fast.
In case your filters do external calls consider using a <a href="https://docs.python.org/3/howto/logging-cookbook.html#subclassing-queuehandler-a-zeromq-example">queue handler</a></li>
<li>Log more than errors. Make sure whenever you see an error in the logs you can trace back
the state of the data in the logs as well.</li>
<li>If you are developing a library, always set a <a href="https://docs.python.org/3/library/logging.handlers.html#logging.NullHandler"><code class="language-plaintext highlighter-rouge">NullHandler</code> handler</a> and let the user
define the logging configuration.</li>
<li>Use the handlers already offered by Python’s logging library unless is a very special case.
Most of the time you can hand off the logs and then process them with a more specialized tool.
For example, send logs to <code class="language-plaintext highlighter-rouge">syslog</code> using <a href="https://docs.python.org/3/library/logging.handlers.html"><code class="language-plaintext highlighter-rouge">syslogHandler</code> handler</a>
and then use <code class="language-plaintext highlighter-rouge">rsyslog</code> to maintain your logs.</li>
<li>Create a new logger using the module’s name to avoid collisions, <code class="language-plaintext highlighter-rouge">logging.getLogger(__name__)</code></li>
</ul>
<blockquote>
<p><i class="fas fa-bolt"> </i> <strong>Note:</strong> <br />
If you want to use the test the configuration above or just play around with different configs,
checkout the <a href="https://github.com/rmcomplexity/intro-to-python-logging">complementary repo to this article</a></p>
</blockquote>rmcomplexityLogging is one of the best ways to keep track of what is going on inside your code while it is running. Python comes with a very powerful logging library but with great power... things start to get a bit complicated.Tales From The Keyboard: Getting To Know Docker2018-08-14T20:10:00+00:002018-08-14T20:10:00+00:00https://rmcomplexity.com/article/2018/08/14/tales-from-the-keyboard-getting-to-know-docker<p>At work, we use docker for local development and production. When I first
started in this job (3+ years ago), I had barely heard about Docker. I have grown
to love/hate Docker, although the hate part is not so much about how Docker
solves problems but more about how we are using Docker. The love part is because
I do find containers interesting, however I also think containers can get
over-used and processes get unnecessarily over-complicated. In this Tale
we’ll visit some struggles I (and other team members) had when first getting
to know Docker.</p>
<p>First, some context. Here at the <a href="https://tacc.utexas.edu">Texas Advanced Computing Center</a> (TACC)
we build Science Gateways, amongst other amazing things. Science Gateways can
get very complicated, for the sake of simplicity let’s think about Science
Gateways as a distributed web application. We use Mysql or Postgres for
database, Django for the back-end, AngularJS (currently upgrading to something
else, haven’t fully decided what) for the front-end, and we run all of this in
containers. There are other services we use but this gives enough information
to get the picture of the dev environment we’ll be talking about.</p>
<h2 id="how-is-docker-running-in-your-computer">How Is Docker Running In Your Computer?</h2>
<p>The projects we work here at TACC are, what we call, <strong>highly distributed</strong>.
We use the term <strong>highly distributed</strong> because these projects use multiple
services build by different teams. Some of these services are exposed
via REST APIs and some others are HPC resources. For instance, TACC’s main
storage resource is called <a href="https://www.tacc.utexas.edu/systems/corral">Corral3</a>. For specific type of data
we have to do a <a href="https://en.wikipedia.org/wiki/Network_File_System">Network File System (NFS)</a> mount to have direct
access from the host where we have Python/Django applications running.
NFS mounts are common to use specially when you want a storage system that’s robust.
In production this is quite simple to set up using <a href="https://www.centos.org/docs/5/html/5.1/Deployment_Guide/s2-nfs-fstab.html">fstab</a> and in our
development environment (using docker) it’s easy to simulate since the host
is the one who mounts the NFS and then docker sees the mount as a regular
folder. The development cycle runs smoothly until some serious debugging is
necessary. I had to do an NFS mount directly in my laptop to figure out
this one bug. I’m currently using a Mac Book Pro and had to use <a href="https://osxfuse.github.io/">FUSE</a>
and do an <a href="https://github.com/osxfuse/osxfuse/wiki/SSHFS">SSHFS</a> mount. The permissions on the mount were correct.
I then fired up <code class="language-plaintext highlighter-rouge">docker-compose</code> and waited
for everything to run, then I saw this error:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ERROR: <span class="k">for</span> <service name> Cannot start service <service name>:
b<span class="s1">'Mounts denied:
The path /path/to/mount
is not shared from OS X and is not known to Docker.
You can configure shared paths from Docker -> Preferences... ->
File Sharing.
See https://docs.docker.com/docker-for-mac/osxfs/#namespaces
for more info.'</span>
</code></pre></div></div>
<p>One good thing is that the error is very explicit and it tells you right there
where to get more information to fix it. It is still odd to get this error
since I’m running docker for mac and it feels native. Here’s the catch: <strong>it
feels and sounds native, but it is not native</strong>. <a href="https://docs.docker.com/docker-for-mac/docker-toolbox/#the-docker-for-mac-environment">Docker describes</a>
how Docker for Mac differs from Docker Toolbox. They do describe
it as a native application but only because it uses <a href="https://github.com/moby/hyperkit">Hyperkit</a>,
although this means there is still a VM between OSX and Docker. It is simpler
to use, but we have to keep this structure in mind. After looking into how Docker
for Mac runs it is easier to understand <strong>why</strong> we cannot mount folders
into our Docker containers <em>a la willy-nilly</em>. The fix is to let the Docker VM
know about the folder(s) we want to mount. We configure the paths the Docker VM
can see in the Docker settings. This is called <strong>File Sharing</strong>.</p>
<figure class="img center">
<a href="/assets/images/getting_to_know_docker/docker-file-sharing.png">
<img src="/assets/images/getting_to_know_docker/docker-file-sharing.png" alt="Docker Settings Share Files" class="img-responsive" />
</a>
<figcaption>
<em>
I usually mount my <code>$HOME/projects</code> folder to save some time.
</em>
</figcaption>
</figure>
<p>Once this is set up one can mount any path and sub-path in the previously shown
list <a href="https://docs.docker.com/storage/volumes/#choose-the--v-or---mount-flag">using <code class="language-plaintext highlighter-rouge">-v</code></a> or <a href="https://docs.docker.com/compose/compose-file/#volumes"><code class="language-plaintext highlighter-rouge">volume</code> in a <code class="language-plaintext highlighter-rouge">docker-compose.yml</code> file</a>.</p>
<p>Another thing that threw me off was when we started using
<a href="https://www.elastic.co/">elasticsearch</a>. Elasticsearch maps indices into memory using a
specific type of file system. This memory mapping allows elasticsearch to be
really fast. The trick here is that Linux will
restrict memory mapping to only a portion of the available virtual memory
address space. The restriction is necessary mainly to protect other processes
to not hog all the virtual memory address space. On Linux operating systems
this is easily fixed with one command <code class="language-plaintext highlighter-rouge">sysctl -w vm.max_map_count=262144</code>.
There is more information on <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html">elasticsearch’s site</a>.
Once again, we were able to set up the correct <code class="language-plaintext highlighter-rouge">max_map_count</code> in our prod
environment without any hassles. An interesting thing happened when I was
first trying out elasticsearch in development. After setting up the
elasticsearch container and starting up the project locally we saw this error:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ERROR: bootstrap checks failed max virtual memory areas
vm.max_map_count <span class="o">[</span>65530] likely too low, increase to at least <span class="o">[</span>262144]
</code></pre></div></div>
<p>The previous error is expected if the virtual memory is too low and the
elasticsearch docs are handy when solving this. The <code class="language-plaintext highlighter-rouge">sysctl</code> command to
increase the <code class="language-plaintext highlighter-rouge">max_map_count</code> needs to be run on the host environment.
Naturally I tried running the command on OSX only to get this error in return:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sysctl: unknown oid <span class="s1">'vm.max_map_count'</span>
</code></pre></div></div>
<p>“How curious”, I thought. I then remembered that Docker for Mac does not
actually run on OSX instead it runs on a VM. This means the memory
Elasticsearch is using is actually the memory the Docker VM has access to.
Usually incrementing the memory through Docker’s settings is enough:</p>
<figure class="img center">
<a href="/assets/images/getting_to_know_docker/docker-vm-advanced.png">
<img src="/assets/images/getting_to_know_docker/docker-vm-advanced.png" alt="Docker VM Advanced Settings" class="img-responsive" />
</a>
<figcaption>
<em>
You can modify other VM settings in this window.
</em>
</figcaption>
</figure>
<p>If that is not enough, there is a way to actually drop into a <code class="language-plaintext highlighter-rouge">tty</code> on the
Docker’s VM (which is the actual host of your containers). You can run this
command:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty
</code></pre></div></div>
<p>If you’re using Docker for Windows and running the regular Linux daemon
you can also drop into a shell within the tiny Linux VM
<a href="https://github.com/justincormack/nsenter1">using nsenter</a>.</p>
<h2 id="dabbling-in-docker-compose">Dabbling in <code class="language-plaintext highlighter-rouge">docker-compose</code></h2>
<p><code class="language-plaintext highlighter-rouge">docker-compose</code> is <a href="https://docs.docker.com/compose/">defined as</a>: <em>“a tool for
defining and running multi-container Docker applications.”</em>. <code class="language-plaintext highlighter-rouge">docker-compose</code>
can be used in different ways other than running multiple containers. I’ve seen
projects where it’s used as a way to run management commands. It is common to
see a <code class="language-plaintext highlighter-rouge">docker-compose.yml</code> file on a project which defines the entire
environment of said project.</p>
<p>Soon after starting to use <code class="language-plaintext highlighter-rouge">docker-compose</code> in a
project I realized I had to update the file in different ways to test things
locally. Usually these are only tests and I don’t necessarily want to track
this changes with <code class="language-plaintext highlighter-rouge">git</code>. Luckily I realized <code class="language-plaintext highlighter-rouge">docker-compose</code> can merge multiple
files. By default <code class="language-plaintext highlighter-rouge">docker-compose</code> will look for two files:
<code class="language-plaintext highlighter-rouge">docker-compose.yml</code> and <code class="language-plaintext highlighter-rouge">docker-compose.override.yml</code>. I tend to add the
<code class="language-plaintext highlighter-rouge">.override</code> file to <code class="language-plaintext highlighter-rouge">.gitignore</code> that way I can override different values locally
without worrying messing up everybody else’s setup. This allows you to have a
<code class="language-plaintext highlighter-rouge">docker-compose.yml</code> file such as:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">django</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">project/portal:local</span>
<span class="na">links</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">potgresql:postgresql</span>
<span class="pi">-</span> <span class="s">memcached:memcached</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">./../src:/srv/www/project</span>
<span class="pi">-</span> <span class="s">/srv/www/project/static:/srv/www/project/static</span>
<span class="pi">-</span> <span class="s">/srv/www/project/media:/srv/www/project/media</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">8000:8000</span>
<span class="na">command</span><span class="pi">:</span> <span class="s">newrelic-admin run-program /usr/local/bin/uwsgi --ini /srv/www/project/wsgi.py</span>
</code></pre></div></div>
<p>Which can then be overridden with a <code class="language-plaintext highlighter-rouge">docker-compose.override.yml</code> file such as:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">django</span><span class="pi">:</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">./../src:/srv/www/project</span>
<span class="pi">-</span> <span class="s">static:/srv/www/project/static</span>
<span class="pi">-</span> <span class="s">media:/srv/www/project/media</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">8001:8000</span>
<span class="na">command</span><span class="pi">:</span> <span class="s">/srv/www/project/bin/run-dev.sh</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="na">static</span><span class="pi">:</span>
<span class="na">media</span><span class="pi">:</span>
</code></pre></div></div>
<p>This would result in a <code class="language-plaintext highlighter-rouge">docker-compose</code> config file such as:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">django</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">project/portal:local</span>
<span class="na">links</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">potgresql:postgresql</span>
<span class="pi">-</span> <span class="s">memcached:memcached</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">./../src:/srv/www/project</span>
<span class="pi">-</span> <span class="s">static:/srv/www/project/static</span>
<span class="pi">-</span> <span class="s">media:/srv/www/project/media</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">8000:8000</span>
<span class="pi">-</span> <span class="s">8001:8001</span>
<span class="na">command</span><span class="pi">:</span> <span class="s">/srv/www/project/bin/run-dev.sh</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="na">static</span><span class="pi">:</span>
<span class="na">media</span><span class="pi">:</span>
</code></pre></div></div>
<p>As we can see not every value gets simply overridden after merging.
<code class="language-plaintext highlighter-rouge">docker-compose</code> treats fields with the next policy:</p>
<ul>
<li><strong>Single-value</strong>: (e.g. <code class="language-plaintext highlighter-rouge">image</code>, <code class="language-plaintext highlighter-rouge">command</code> or <code class="language-plaintext highlighter-rouge">mem_limit</code>)
The value from the <code class="language-plaintext highlighter-rouge">.override</code> file will be used.</li>
<li><strong>Multi-value</strong>: These fields are treated as arrays or maps and depending on the type
of field <code class="language-plaintext highlighter-rouge">docker-compose</code> will act differently:
<ul>
<li><code class="language-plaintext highlighter-rouge">ports</code>, <code class="language-plaintext highlighter-rouge">expose</code>, <code class="language-plaintext highlighter-rouge">external_links</code>, <code class="language-plaintext highlighter-rouge">dns</code>, <code class="language-plaintext highlighter-rouge">dns_search</code> and <code class="language-plaintext highlighter-rouge">tmpfs</code> are
concatenated. Meaning we <strong>can never completly override the value set
on the first file, we can only add more values to the array or map</strong>.
I recommend being conservative with the value used on the first file.</li>
<li><code class="language-plaintext highlighter-rouge">environment</code> and <code class="language-plaintext highlighter-rouge">labels</code> are merged together.
Meaning, we can add more values to the array or map <strong>and we can override
previously set values</strong>. The trick here is that the values used in this
array or map are in the form of <code class="language-plaintext highlighter-rouge">UNIQUE_STRING:VALUE</code>. <code class="language-plaintext highlighter-rouge">docker-compose</code> will
use the left side of the string as uniqueness and will override values
based on that.
<blockquote>
<p><strong>Note:</strong> <code class="language-plaintext highlighter-rouge">environment</code> value can be an array or a map.</p>
</blockquote>
</li>
<li><code class="language-plaintext highlighter-rouge">volumes</code> and <code class="language-plaintext highlighter-rouge">devices</code> are <em>also</em> merged together.
Meaning, we can add more values to the array <strong>and we can override
previously set values</strong>. The difference here is that <code class="language-plaintext highlighter-rouge">docker-compose</code>
will use the right side of the string as uniqueness. <code class="language-plaintext highlighter-rouge">docker-compose</code>
will see every value as <code class="language-plaintext highlighter-rouge">VALUE:UNIQUE_STRING</code>. This makes sense because
the right side of the value is unique for the container we are creating.</li>
</ul>
</li>
</ul>
<p>We can also use other file names and as many as we want taking the previous
rules into consideration. You can read more about overriding configuration in
<a href="https://docs.docker.com/compose/extends/#adding-and-overriding-configuration">Docker’s documentation</a>.
The way to do this is to concatenate them using <code class="language-plaintext highlighter-rouge">-f</code> flag such as the next example.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker-compose <span class="nt">-f</span> docker-compose.local-dev.yml <span class="nt">-f</span> docker-compose.local-dev.override.yml <span class="nt">-f</span> docker-compose.local-dev.shenanigans.yml
</code></pre></div></div>
<p>I’ve also realized this tends to get confusing and I often loose track of what
I’m overriding and what not. The simple way to check how <code class="language-plaintext highlighter-rouge">docker-compose</code> will
end up being configured is to use:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker-compose config
</code></pre></div></div>
<p>The previous command will print the entire configuration used.</p>
<h3 id="bringing-up-and-down-the-house">Bringing <code class="language-plaintext highlighter-rouge">up</code> and <code class="language-plaintext highlighter-rouge">down</code> the house</h3>
<p>When we first started using Docker and <code class="language-plaintext highlighter-rouge">docker-compose</code> in our projects I found
running and stopping an entire project was a bit confusing. Let me clarify this:
it wasn’t confusing because I wasn’t sure how to check if everything was running
or not, it was confusing because it wasn’t clear to me what was the correct way
to bring up or down all the containers in a project.</p>
<p>First, Let’s talk about <strong>projects</strong>. A <code class="language-plaintext highlighter-rouge">docker-compose.yml</code>
file describes a set of <em>services</em>, and each of these services corresponds to a
container. <code class="language-plaintext highlighter-rouge">docker-compose</code> uses the concept of <strong>projects</strong> to somewhat group
together the containers described in the configuration. In reality a
<strong>project</strong> uses a project name to manage different container resources.</p>
<blockquote>
<p>A <code class="language-plaintext highlighter-rouge">docker-compose</code> project name can be defined by the configuration’s root folder,
by using the <code class="language-plaintext highlighter-rouge">-p</code> argument or the <code class="language-plaintext highlighter-rouge">COMPOSE_PROJECT_NAME</code> environment variable.</p>
</blockquote>
<p>A project name is prepended to any object’s name defined in the configuration:</p>
<ul>
<li>Prepended to every service name. If one of the service’s name is <code class="language-plaintext highlighter-rouge">django</code>, the
container created will be named <code class="language-plaintext highlighter-rouge">PROJECT_NAME_django_1</code> <em>unless
<code class="language-plaintext highlighter-rouge">container_name</code> is defined in the service definition</em>.</li>
<li>When <a href="https://docs.docker.com/compose/compose-file/#volume-configuration-reference">defining <code class="language-plaintext highlighter-rouge">volume</code>s</a> in reality the volume’s name
is prepended with <code class="language-plaintext highlighter-rouge">PROJECT_NAME</code>. We can verify this by using <code class="language-plaintext highlighter-rouge">docker volume ls</code>.</li>
<li><code class="language-plaintext highlighter-rouge">PROJECT_NAME</code> is also prepended to any <a href="https://docs.docker.com/compose/compose-file/#volumes">network defined</a>.
We can also verify this by using <code class="language-plaintext highlighter-rouge">docker network ls</code>.</li>
</ul>
<p>Now that we have clarified <code class="language-plaintext highlighter-rouge">docker-compose</code> uses <code class="language-plaintext highlighter-rouge">docker-compose.yml</code> file and
<strong>projects</strong> to group together service, volumes and networks let’s talk about
how do we initialize and destroy these objects.</p>
<p>The easiest way to start up everything within a project is to use <code class="language-plaintext highlighter-rouge">$ docker-compose up</code>.
This will download or build any images which need to be downloaded or build, create
every container, network and volume defined in the configuration file, and run
all the services defined. <code class="language-plaintext highlighter-rouge">docker-compose up</code> is pretty handy and after creating and
running everything it will continue printing the output from every container to
<code class="language-plaintext highlighter-rouge">stdout</code>. If you want to run everything in the background you can use the <code class="language-plaintext highlighter-rouge">-d</code>
flag to run it as a daemon and keep it in the background.</p>
<p>After using <code class="language-plaintext highlighter-rouge">up</code> I realized there’s also <code class="language-plaintext highlighter-rouge">start</code>, <code class="language-plaintext highlighter-rouge">run</code> and <code class="language-plaintext highlighter-rouge">create</code>. This is
a bit confusing since those verbs could potentially be synonyms. I later realized
<code class="language-plaintext highlighter-rouge">create</code> is deprecated and should not be used. <code class="language-plaintext highlighter-rouge">run</code> is used for when you want
to execute a one-off command inside a container. The container will be created,
the command will be executed inside the container and the container will then be
stopped. I recommend to use <code class="language-plaintext highlighter-rouge">--rm</code> to make sure the container is deleted after
the command is run and exits. You can read more about <code class="language-plaintext highlighter-rouge">run</code> in the
<a href="https://docs.docker.com/compose/reference/run/">Docker’s docs</a>. <code class="language-plaintext highlighter-rouge">start</code> is also useful, it will only
run containers for the services defined in the configuration. I recommend
to use <code class="language-plaintext highlighter-rouge">start</code> <strong>only</strong> if you need to use <code class="language-plaintext highlighter-rouge">stop</code> to temporarily stop services
running.</p>
<blockquote>
<p><strong>Note:</strong> <code class="language-plaintext highlighter-rouge">docker-compose start</code> does not create containers, it will only
start containers for services that are already created.</p>
</blockquote>
<p>By now we can safely say most of the time we’ll be using <code class="language-plaintext highlighter-rouge">docker-compose up</code>
to start our services since it’s the most compact way to create and run
everything necessary. Sometimes we encounter errors when bringing everything
up, when this happens I find it helpful to go step by step:</p>
<ol>
<li>Pull necessary images:
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nv">$ </span>docker-compose pull
</code></pre></div> </div>
</li>
<li>Build necessary images:
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nv">$ </span>docker-compose build
</code></pre></div> </div>
</li>
<li>Create necessary networks and volumes:
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nv">$ </span>docker-compose up <span class="nt">--no-build</span> <span class="nt">--no-start</span>
</code></pre></div> </div>
</li>
<li>Run services:
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nv">$ </span>docker-compose start
</code></pre></div> </div>
</li>
</ol>
<blockquote>
<p><strong>Note:</strong> The previous list is not the actual steps taken by docker,
but it is a simple way to simulate what happens.</p>
</blockquote>
<blockquote>
<p><strong>Note:</strong> <code class="language-plaintext highlighter-rouge">docker-compose build</code> will only work correctly
if the <a href="https://docs.docker.com/compose/compose-file/#build">build configuration</a> is defined in the <code class="language-plaintext highlighter-rouge">docker-compose.yml</code>
file.</p>
</blockquote>
<p>Most of the time errors appear when building images, but I find it
helpful to know everything that happens when running <code class="language-plaintext highlighter-rouge">docker-compose up</code>.</p>
<p>Let’s take a look how can we bring everything down. I quickly realized
when using <code class="language-plaintext highlighter-rouge">$ docker-compose up</code> I was able to hit Ctrl+C and compose will
automatically stop every service. Sometimes this doesn’t work correctly,
I believe it has to do with how
<a href="https://medium.com/@gchudnov/trapping-signals-in-docker-containers-7a57fdda7d86">docker processes handle signals</a>. When this happens it is
necessary to use <code class="language-plaintext highlighter-rouge">$ docker-compose down</code>, which is pretty simple and compact.
After a while, one is (almost) bound to work with multiple projects. Every
once in a while I encounter an error or a bug that might be related to
docker and not necessarily to the code I’m working on. When something like
this happens I usually first delete every container created for a project.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker-compose <span class="nb">rm</span> <span class="nt">-f</span>
</code></pre></div></div>
<p>Something similar can be done using <code class="language-plaintext highlighter-rouge">up</code> in this form:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker-compose up <span class="nt">--force-recreate</span>
</code></pre></div></div>
<p>Which recreates every container for every service in the configuration.</p>
<p>Sometimes I have to delete everything from a project to make sure every
service is correctly configured and initialized. This is done by
removing every container with the command shown above and then:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker-compose down <span class="nt">--rmi</span><span class="o">=</span>all <span class="nt">-v</span> <span class="nt">--remove-orphans</span>
</code></pre></div></div>
<p>The previous command will remove <em>all</em> images for a project (<code class="language-plaintext highlighter-rouge">--rmi=all</code>),
every volume created (<code class="language-plaintext highlighter-rouge">-v</code>) and every orphan container left (<code class="language-plaintext highlighter-rouge">--remove-orphans</code>).
Next time I use <code class="language-plaintext highlighter-rouge">docker-compose up</code> everything for that project will be created again.</p>
<p>We can use <code class="language-plaintext highlighter-rouge">docker-compose down</code>, <code class="language-plaintext highlighter-rouge">docker-compose stop</code> or <code class="language-plaintext highlighter-rouge">docker-compose kill</code> to stop
the services. The difference between <code class="language-plaintext highlighter-rouge">down</code> and <code class="language-plaintext highlighter-rouge">stop</code> is that <code class="language-plaintext highlighter-rouge">down</code> will delete
any containers and networks created by default, if needed you can specify to delete
volumes as stated above. <code class="language-plaintext highlighter-rouge">stop</code> will only stop the containers, you can re-start the services
by using <code class="language-plaintext highlighter-rouge">docker-compose start</code>, this is better suited for a temporary stop on services.
If everything else fails you can force the services to be stopped by using <code class="language-plaintext highlighter-rouge">kill</code>.
I strongly recommend to only use <code class="language-plaintext highlighter-rouge">kill</code> in your local development because some containers
can end up in a false state when using <code class="language-plaintext highlighter-rouge">kill</code>. If this happens and you have to run <code class="language-plaintext highlighter-rouge">kill</code> in
production I recommend to use <code class="language-plaintext highlighter-rouge">docker-compose rm -f</code> to remove all containers and then
bring everything up. This has been really helpful to me a lot of times.</p>
<p>These are some things I learned when I first started using docker.
As with any technology, docker can be a breeze to use until it is not.
I hope this will be useful to other developers.
The <strong>Tales from the Keyboard</strong> series describes the things me (and other team-members)
learned when using different technologies.</p>rmcomplexity'Tales From The Keyboard' is a series based on real issues and lessons learned. This time we'll take a look at DockerFirst (Few) Tech (or Python) Conference(s)2018-08-04T18:45:00+00:002018-08-04T18:45:00+00:00https://rmcomplexity.com/article/2018/08/04/first-few-tech-or-python-conferences<p>Although this article comes quite late since PyCon 2018 ended back in May and this post was originally
written to describe my first PyCon. I was hesitant to publish this article
since I never got around to edit it. I decided to publish it but focusing more on the things I think
every newcomer should try at PyCon or any other tech (or python related) conference.</p>
<p>PyCon 2018 has come to an end. A bittersweet feeling still lingers with me after a few days of returning to my usual routine.
Thankfully, I was able to interact with a lot of interesting people. The venue’s air was always filled with ideas.
Navigating from conversation to conversation, planting curious seeds in any attentive mind nearby.
Authors of widely used libraries and frameworks were being humanized by all the interactions between said authors and
the rest of the community. Little by little GitHub handles transformed into approachable peers.
A welcoming sense of belonging filled the halls.
One can truly appreciate the great effort from the organizers which is poured into this conference.
An event made by the python community for the python community.</p>
<p>This feeling of community is not unique to PyCon, it is a constant in every python related conference I’ve been to.
In reality, it has much less to do with the people organizing the conference and more with who and how
you approach your peers.</p>
<h2 id="key-notes">Key Notes</h2>
<p>To this day I am not sure how any conference select their keynotes.
I assume the conference chairs get together and take into consideration anyone who, in the past year, has done anything noticeable enough.
Noticeable enough is probably measured by the impact in the community as a whole.
This is only an assumption, which is as good as any assumption from any other carbon based person.
I can only imagine how hard this exercise might be.</p>
<p>The <a href="https://www.youtube.com/watch?v=ITksU31c1WY">first PyCon 2018 keynote</a> by <a href="https://hacks.mozilla.org/author/dcallahanmozilla-com/"><strong>Dan Callahan</strong></a> (<a href="https://twitter.com/callahad">@callahad</a>) was
particularly interesting to me since Dan works for Mozilla, and he was talking about how the web is a platform
and is the most popular platform. Javascript (JS) is native to the web and JS is a very popular language.
Dan was presenting <a href="https://webassembly.org/">WebAssembly</a> as an alternative to use JS on the web. <a href="https://webassembly.org/">WebAssembly</a>
allows for development of web applications in other languages. The idea is that we can bring python into the web.</p>
<blockquote>
<p>
<i class="fas fa-quote-left fa-2x"> </i>
Why are we programming if not to serve people and solve human problems?
<i class="fas fa-quote-right fa-2x"> </i>
</p>
<cite>-- Dan Callahan, PyCon 2018 Keynote</cite>
</blockquote>
<p>Not every keynote is about introducing new technologies. <strong>Ying Li</strong>’s <a href="https://twitter.com/cyli">@cyli</a>) <a href="https://youtu.be/VJ0vibC_Hl0?t=35m6s">keynote</a>
was very interesting with a very different format than <strong>Dan Callahan</strong>’s. <strong>Ying Li</strong> talked about
security and how to make sure our applications are secure. I highly recommend watching <a href="https://youtu.be/VJ0vibC_Hl0?t=35m6s">Ying Li’s keynote video</a>.
I remember one of the first keynotes I saw live was <a href="https://www.aeracode.org/"><strong>Andrew Godwin</strong></a>’s keynote at DjangoCon US 2016.
He was presenting some advanced techniques with <a href="https://channels.readthedocs.io/en/latest">Django Channels</a>, which back then was on the 1st version.
<strong>Andrew</strong>’s keynote wasn’t the first keynote I saw, but I didn’t really pay that much attention to the keynotes at
the other two tech conferences I went before DjangoCon US 2016. I wasn’t really sure what keynotes were about.
The good thing is that now we can re-watch a lot of keynotes in Youtube. I’m catching up.</p>
<p>If the conference you attend posts the keynotes on Youtube you should try to attend to the keynotes live.
It’s always interesting to see things happen live, you can pick up some pointers for future talks.
Another thing I like about attending keynotes is that most of the conference will surely attend, one can
meet a lot of interesting people before, after or even while the keynote is happening.</p>
<h2 id="talks">Talks</h2>
<p>A great reason to attend a conference is its lineup. A lot of conferences are able to invest in a good
audio-visual package, and they are able to record every presentation at the conference. This is great
since one can go back and re-watch interesting talks. Sometimes this is taken as if it is not that good of an idea
to actually attend the presentation. Let me tell you that is not so black and white.
The fact that talks are recorded give the attendees some freedom to do more things at a conference, which
we’ll talk about later. Watching a talk live has its own rewards. The main upside I see about attending
talks is that, most of the time, you get to ask questions to the speaker. Some speakers prefer to not
use time from their talk on questions from the public, instead you can probably approach the speaker afterwards.</p>
<blockquote>
<p>
<i class="fas fa-quote-left fa-2x"> </i>
Attending conferences can feel overwhelming. It's OK if you feel overwhelmed, <br />
try to look for someone who can help you.
<i class="fas fa-quote-right fa-2x"> </i>
</p>
<cite>-- Josue B. Coronel</cite>
</blockquote>
<p>Experiencing a conference in person has great advantages it can also get overwhelming. <strong>Remember that it is
OK to feel overwhelmed while at a conference</strong>. I tend to feel pressured to attend every talk mainly because I’ve
been able to get my employer to pay for the conferences I’ve attended. The fact that somebody else is paying for my
ticket makes me feel it’s part of my job to participate in as many things as possible in a conference.
While this might be partially true, we also have to remember to take care of ourselves. It’s OK to go to the hotel
room or Airbnb, or go to a silent room (if you have access to one), or maybe just go and grab some lunch outside of
the conference to avoid feeling too overwhelmed. Believe me, you will feel better and you will even retain more information.
There is a great <a href="https://zapier.com/engineering/introvert-conferences/">article by zapier</a> about attending a conference as an introvert.</p>
<h2 id="open-spaces">Open Spaces</h2>
<p>This year’s PyCon was the first time I attend PyCon and was the first time I experienced, firsthand,
what <a href="http://openspaceworld.org/wp2/">Open Spaces</a> are. I’ve been to different conferences with an Open Space track, but
for some reason in this year’s PyCon people were talking more about going to open spaces. I was not sure
what to expect of an open space and it seemed everyone was already acquainted with how the whole thing worked.
Let me summarize what open spaces are, at least to me: Open Spaces are a particular gathering format where the
content is created by whomever attends. I know <a href="https://2018.djangocon.us">DjangoCon US</a> has open spaces some years, which is awesome.
I found this year’s PyCon used a couple big grid boards where you could place a post-it (or card) with the topic
of an open space. These grid boards where right in the middle of the hallway so everyone noticed them.
I am hoping to participate in this year’s <a href="https://2018.djangocon.us">DjangoCon US</a>’s open space track.
Below is a picture of the grid boards used by this year’s PyCon.</p>
<figure class="img center">
<img src="/assets/images/open-spaces-pycon-schedule.jpg" alt="PyCon's Open Spaces Schedule" class="img-responsive" />
<figcaption><em>Look, it's self organizing!.</em></figcaption>
</figure>
<p>I soon realized that it’s a great way to have more personal conversations about a specific topic
with other conference attendees. The idea is to place a post it on the schedule, which anybody can do,
and propose a topic. I saw topics ranging from <code>pipenv</code> to <strong>chocolate</strong>. Any topic is OK.
After you’ve chosen a slot and a topic you show up at the reserved room and wait for people to come.
Then the open space starts. In the <a href="http://www.chriscorrigan.com/parkinglot/wp-content/uploads/2008/08/Original-Users-Guide-smaller.pdf">Open Space’s User Guide</a> mentions that an open space
starts when it has to start. This is a more informal way of gathering people and talk about stuff.
I went to a very interesting open space about Geographic Information Systems (GIS) and then to a
writing open space lead by the one and only <a href="http://mindviewllc.com/">Bruce Eckel</a>.</p>
<p>Attending to a few open spaces made me realize I wanted to participate more. I went on and looked
for a spot on one of those big grid boards. The slot was on the last day of the conference
and I thought this was better, that way I wouldn’t feel bad if people didn’t come.
I wrote on a piece of paper <strong>“Task Queues (Celery and such)”</strong>. I started telling some
new people I met at the conference about the Open Space I was organizing, and they all responded
with supporting words. The time had come and there were a good 15 or 20 people waiting in the room.
It was great!. I gave a small introduction about what the topic meant and about me. After my
introduction I invited everyone in the room to say their name and mention any topic about <strong>Task Queues</strong>
they needed help with or wanted to know more about. With this exercise the conversation started flowing.
I was able to take notes on the main topics we talked about, below you can see an image of these notes.</p>
<figure class="img center">
<img src="/assets/images/celery_open_space_notes.jpg" alt="Celery Open Space Notes" class="img-responsive" />
<figcaption><em>Graphologists have a field day with my handwriting.</em></figcaption>
</figure>
<p>If you are thinking about organizing an open space in your next conference I have three things to say
that helped me on my very first open space:</p>
<ol>
<li>Let the conversation flow but beware some people won’t know what specific acronyms or “lingo” means.
It’s ok to ask whomever is talking to explain what they mean for the rest of the group.</li>
<li>Try to take notes, either on a white board or a big piece of paper. Leave your details on top of wherever
you’re taking notes and make yourself available afterwards.</li>
<li>It’s OK to interrupt someone if you think the conversation is being monopolized into one topic.
There are more people and an open space is made by everyone.</li>
</ol>
<h2 id="the-hallway-track">The Hallway Track</h2>
<p>Finally, do not miss on <strong>the hallway track</strong>. The hallway track is everything that’s happening on
the hallways. There’s usually people meeting other people or great conversations starting on the hallways.
Remember to always approach people respectfully and your doors will open to stimulating conversations.
Approaching somebody could be as simple as asking them what event are they going to attend next,
or where are they from. Don’t feel bad if they don’t want to talk at that specific moment, maybe they’re feeling
overwhelmed. Remember to always: <em>have fun, learn stuff, meet people, and let yourself be inspired</em>.
Everything except for having fun is optional.</p>rmcomplexityThings I've been learning and discovering as I attend more tech conferences.What I learned from PyCon 2018 ProgCom2018-05-01T16:00:00+00:002018-05-01T16:00:00+00:00https://rmcomplexity.com/article/2018/05/01/pycon-2018-what-i-learned-as-progcom<p>Attending PyCon has been in my wishlist for a while now. This year I convinced myself to do everything possible to attend PyCon 2018.
I decided I had to earn it. I know, this sounds intense, it’s not. <em>First</em> of all there are no requirements to attend PyCon (other than buying a ticket).
<em>Second</em>, by <strong>“earn it”</strong> I meant to, at least, submit a talk. And I already had a few ideas.
<em>Third</em>, PyCon has a lot of opportunities to get involved, help out and meet some awesome people.
Coincidentally my criteria of <strong>“earn it”</strong> was to either submit a talk (as mentioned before), become a reviewer and/or volunteer for something.
I’ll expand on the first two on this article.</p>
<figure class="img center">
<img src="/assets/images/pycon2018.png" style="max-width:200px;" alt="PyCon 2018 logo" class="img-responsive" />
<figcaption><em>That snake is even more impressive in real life</em></figcaption>
</figure>
<p>Volunteering at PyCon was quickly out of the question because of work related reasons which I won’t mention in this article.
Mainly because these reasons are boring and not particularly aligned to the context of this article.
Since we are on the topic of <strong>Things I Won’t Be Talking About Here</strong>, I should mention I won’t be describing, in too much detail, PyCon’s reviewing process.
If you would like to know more about the actual reviewing process <strong>Ned Jackson Lovely</strong>
(the author of the ProgCom platform) <a href="http://www.njl.us/essays/pycon-process/">wrote about it</a> back in 2015
and <strong>Eric J. Ma</strong> <a href="http://ericmjl.com/blog/2018/2/6/pycon-program-committee-review/">wrote a wonderful article</a> about it, too. Eric was also a part of PyCon’s ProgCom this year (2018).</p>
<p>This is how, back in Dec 2017, I started writing down a few talk ideas.
<em>– I have never put together a tutorial. Naturally I’m hesitant about submitting a tutorial idea, maybe next year.–</em>
I had three ideas in mind. These were really <strong>raw</strong>. These ideas consisted of: a first draft title and, at least, two main topics.
I decided to use <strong>metaprogramming</strong> as my submission topic. Two main reasons were involved in this decision:</p>
<ol>
<li>I find metaprogramming, and its uses, very interesting.</li>
<li>I have used metaprogramming in different projects and I’ve seen some widely used libraries/frameworks use it successfully.</li>
</ol>
<p>I should say I am fully aware of what <a href="http://wiki.c2.com/?MetaClass">Tim Peters said about metaclasses</a>
(to this day, I have not found where this quote was first introduced).
To me metaprogramming is really interesting and useful. I agree more with <a href="https://blog.thumbtack.net/python-metaclasses-without-magic/">Dmitry Vakhrushev’s article</a>.
Also, times have changed and I believe there are more uses for metaprogramming/metaclasses than only 1% of the cases. The talk was not accepted.
This might start to look like a rant. Trust me, it is not. This is meant to be helpful to anybody who wants to submit a talk to PyCon
or any other conference for that matter. More on this later.</p>
<figure class="img center">
<img src="/assets/images/thought.jpg" alt="Thoughts image" style="max-width:200px;" class="imgr-responsive" />
<figcaption><em>Chalkboard clouds can't contain ideas. Let your ideas run free!</em></figcaption>
</figure>
<h2 id="becoming-part-of-the-program-committee-progcom">Becoming Part of The Program Committee (ProgCom).</h2>
<p>There was I, trying to put together a talk submission. Using all the tricks I knew to search on the vast sea of information that is the internet.
Trying to get a sense of <strong>how</strong> I should write the talk outline, introduction, etc…
I was lucky enough to stumble upon an <a href="https://pycon.blogspot.com/2012/07/i-want-you-for-pycon-program-commitee.html">article from 2012</a> written by <a href="https://jacobian.org/"><strong>Jacob Kaplan-Moss</strong></a>.
I am not entirely sure [yet] what <strong>Mr. Kaplan’s</strong> role was on PyCon 2012, but I am <em>very much grateful</em> for that article.
In this article <strong>Mr. Kaplan</strong> invites volunteers to subscribe to the <a href="https://mail.python.org/mailman/listinfo/pycon-pc">PyCon Program Committee mailing list</a>.
I subscribed and in a few weeks I received a call for action. The review process had begun.
The ProgCom mailing list is open to any member of the community. To some of you, this might look a bit strange.
To the contrary, most conferences’ review committee is integrated of members from the community.
This is great, because it means that PyCon (as many other conferences) is built by your own peers.</p>
<p>PyCon’s review process consist of two main stages. <em>First Stage</em> is, what we could call, a <em>light</em> filter.
There is a predefined checklist which all talks must adhere to.
The <em>Second Stage</em> consist of creating multiple groups based on topics drawn from the submitted talks.
Out of each group there is only one talk selected. This is done by a collective vote.
If a reviewer also submitted a talk, they will not be able to neither review their own talk nor vote on the group said talk is placed.
All of this is possible thanks to all the people who have contributed to PyCon’s ProgCom platform.</p>
<h2 id="the-art-of-peer-review">The Art of Peer-review</h2>
<p>There were many reviewers working together to build the best possible PyCon for everyone to enjoy.
ProgCom members came from different technical and social backgrounds from all over the world.
There was a constant communication between us all. This is very helpful because one has the opportunity to have conversations about topics
that one might not fully understand with people who have more expertise. We did not only benefit from a technical point of view.
There was also the opportunity to analyze the potential of a talk’s <strong>clearness, completeness, relevancy</strong> and pretty much any other
aspect you can think of.</p>
<figure class="img center">
<img src="/assets/images/peer-review.png" alt="Peer Review Image" style="max-width:200px;" class="img-responsive" />
<figcaption><em>In practice, people don't talk in "star grades"</em></figcaption>
</figure>
<p>Having this peer-review also does wonders for <em>diminishing bias</em>, since the conversation always involve
people from different backgrounds. This is one of the most important aspects of having diversity in a review panel.
We all are human beings and we all have biases. There is really not much to do about that, it is part of what makes us human.
But, by analyzing talks with people with different biases it becomes easier to <strong>not give into these biases.</strong></p>
<p>For me, these conversations is from where I draw the most benefits. I was able to learn a great deal about new topics and also
topics with which I was already very acquainted. <strong>Security, machine learning and CPython</strong> are some topics I was able to learn
about in the few weeks I was part of the ProgCom.
I would like to believe I also helped other committee members to learn a few new curiosities.</p>
<h2 id="advice-for-future-submissions">Advice For Future Submissions</h2>
<p>As I mentioned in the beginning, my talk was not accepted. There is nothing wrong on having a talk rejected from a conference.
Specially a conference as big as PyCon. I saw very good, well written, talks being rejected because, simply, there wasn’t enough
slots to place them. There were others talks which had to be grouped in highly competitive groups and decisions had to be made.</p>
<p>If you submitted a talk this year and it was not accepted, <strong>do not, I repeat, do not</strong> feel discouraged about it.
If you really believe in the topic, work on it a bit more and submit it again next year. Maybe submit it to your local python meetup.
You can even try to submit more than one <em>– I know I will be submitting multiple talks next year –</em>.</p>
<p>I also encourage more community members to volunteer for next year’s program committee, or any other volunteer opportunities for that matter.
This will give you a better perspective on some aspects of the conference.
And not only on PyCon but also try to get involved in other conferences. Reading other people’s talk submissions can give you new ideas
for your own submissions. Even volunteering for anything on-site can give you a better perspective of the needs of other attendees
which you might not be familiar with. And <strong>last, but not least</strong> you will most definitely will meet amazing people.</p>
<h2 id="by-the-community-for-the-community">By The Community, For The Community</h2>
<p>I wrote this article to show, from my perspective, how PyCon is build <strong>by the community, for the community</strong> and to, hopefully,
encourage others to participate in future conferences (not only PyCon).
Forming part of the program committee is only one way a community member could help.
There are many more volunteering opportunities. And there is a great deal to learn from a lot of very smart, interesting and thoughtful people.</p>
<p>I would also like to thank everyone that made, and keep making, PyCon possible. It is hard work and it is worth it.
Special thanks to <strong>Jason Myers</strong> who wore the hat of leader this year and did a great job at keeping us in line.</p>
<p>Her work will have been finished by tonight</p>rmcomplexityPyCon has many ways of allowing community members to participate. One of these opportunities is the program committee (ProgCom). This a summary about my experience participating in PyCon 2018 ProgCom.