A personal website with Wagtail ...

Captura de pantalla de 2022-03-25 07-27-38

After almost a decade, I decide to update my personal website using my own media ans opensource software. As always, a good opportunity to review concepts.

A little bit of history

About 1995 this was the look of my website, just very basic html, some animated gifs and a couple of cgis. Simple and light; both the networks and the systems of the time demanded it.

joslumar archieological web page

The NCSA Mosaic of the time couldn't do much more when it came to rendering nested tables, and that with the table support enabled; Yes, I had a choice on the menu for that. Recall that in 1996 an extension to the standard was published HTML2, defined in theRFC 1942. With netscape (navigator) things started to progress, and new html elements allowed for better layout without really separating the data from the layout, in addition to the support of animated gifs, which gave a little life to the pages whenever you don't get carried away, of course.

I can't say where my first site was hosted, with its resource ftp behind it, but at that time there was a certain freedom impossible to imagine now. At that time you accessed the network as you could, from college, from work, or through dial-up connections, which gave you 58kbps at most. Downloading an entire distribution of linux slackware on three hundred 1.44M floppy disks was quite an adventure. The appearance of the first flat rates allowed you to be connected for a large number of hours, but you could not use the phone because the connection was shared with the switched telephone network itself.

joslumar 20 century web

This is how it looked my page at the end of the last century. With the advent of ADSL connections, most operators used to have the fixed IP option, which made self-hosting even DNS much easier, through the use of small, low-power computers such as those based on the Intel Atom.

Since then there have been many reasons, which I am not going to go into, why this content has become static and obsolete, almost like an archaeological witness and the end point of an era to which we will never return: dialup connections, uucp, usenet, the news, the bbs, the Legion fanzine and Angel's Isla Tortuga... "larga vida para el jamón y el vino".

lynx-oldweb

The advantage of using only html, allowed a good display even in terminal-only browsers, such as the lynx, today practically impossible, even with a little care, with the use and abuse of javascript, css, DOM, ...

The renewal

The objective of the project is to have a website that can be a sharing point for projects and ideas, a basic astrophotography gallery and a portal to access online resources. One of the requirements was that it allow the localization (optional) of the pages created in various languages and that it be fairly usable on all types of devices; bravo for the @screen, flex and grid of the css. After giving it a lot of thought I decided on wagtail:

  • Based on django (python), and can be integrated into an environment with other existing django applications.
  • Very developer oriented, but once configured you don't need to touch anything to add content.
  • Customizable admin interface
  • Many contributions, extensions, active, and reasonably up-to-date
  • Internationalization and multilanguage options.
  • Light

To give it the functionality I wanted I added the puput module, which automatically gives you the blog functionality and, with extensions without touching the sources, the astrophotography gallery.

Key points

I am not going to go into this post to describe step by step how this site is made, but I will describe some key aspects or important points that I have found during the design and construction. There are plenty of quality tutorials on the net that cover everything from getting started with python and django, to designing and building a wagtail production environment:

StreamFields

This is an elegant way to generate complex presentation structures, with only one major drawback: migrations. When you change your data model you are forced to build a custom migration to be able to transform the old data structure to the new one, for example adding a new field. The standard wagtail migration procedure does not support conversion of these structures, and you have to do it by hand by generating a manual migration:

   manage.py makemigrations --empty yourappname

and conveniently edit the file it generates in the migrations/ directory of the affected application, to later launch the migration. The good news is that the streamfield data is represented by a standard json structure. These structures are easy to handle with python by representing the json through lists and dictionaries.

Below is a simple migration example to go from one type of block to another, which needs a new field, using recursion to search for and process the json structure.

   import json
   from importlib import import_module
   from django.db import migrations, models
   from django.core.serializers.json import DjangoJSONEncoder

   def migrate_block(obj):
      """Recursively fetch values from nested JSON."""
      def extract(obj):
         global match
         """Recursively search for values of key in JSON tree."""
         if isinstance(obj, dict):
            if 'type' in obj.keys() and obj['type'] == 'image':
             obj['type'] = 'image2'
             obj['value'] = {'image_type':'center','aImage': obj['value']}
            else:
               for (k, v) in obj.items():
                  if isinstance(v, (dict, list)):
                     extract(v)
         elif isinstance(obj, list):
             for item in obj:
                extract(item)
      extract(obj)

   def forwards(apps, schema_editor):
      AboutPage = import_module('home.models').AboutPage
      for page in AboutPage.objects.all():
         page_data = json.loads(page.to_json())
         content_data = json.loads(page_data['content'])
         migrate_block(content_data)
         page.content=json.dumps(content_data,cls=DjangoJSONEncoder)
         page.save()
         for revision in page.revisions.all():
            content_data = json.loads(revision.content_json)
            revision_data = content_data
            migrate_block(content_data)
            # StreamField data is stored in revision.content_json in a string field
            revision_data['content'] = json.dumps(content_data)
            revision.content_json = json.dumps(
                revision_data, cls = DjangoJSONEncoder)
            revision.save()

   class Migration(migrations.Migration):
      dependencies = [
         ('home', '0021_auto_20220218_1943'),
      ]
      operations = [
         migrations.RunPython(forwards, forwards),
      ]

Python package managment

To have well-controlled dependencies, versions in various environments, I recommend using a self-contained and independent installation of python. The pipenv utility makes maintenance of python modules much easier by making it completely independent of the system installation, ensuring that the necessary modules are present for the application and with the correct version. required.

Internationalization

No secrets, I use the wagtail-localize module following the documentation. At the time of building this site I had to use version 1.1rc1 looking for compatibility with the latest version of wagtail available. If so in your case, you would need to install it like this:

 pipenv install \
    -e git+https://github.com/wagtail/wagtail-localize.git@v1.1rc1#egg=wagtail-localize"

On the other hand, in the integration of the blog I have found a problem in a method of the EntryPageServe class of puput. Since it is not good to correct it directly in the sources, I decide to override the affected method in execution in this way. Notice the last line:

 from django.http import Http404, HttpResponse
 from wagtail.core import hooks
 from wagtail.core.models import Site
 from puput.utils import strip_prefix_and_ending_slash, import_model
 from django.views.decorators.csrf import ensure_csrf_cookie
 from django.utils.decorators import method_decorator
 from puput.views import EntryPageServe

 @method_decorator(ensure_csrf_cookie)
 def get(self, request, *args, **kwargs):
        site = Site.find_for_request(request)
        if not site:
            raise Http404
        if request.resolver_match.url_name == 'entry_page_serve_slug':
            splited_path = strip_prefix_and_ending_slash(request.path).split("/")
            path_components = splited_path[:-4] + splited_path[-1:]
        else:
            path_components = [strip_prefix_and_ending_slash(request.path).split('/')[-1]]
        #page, args, kwargs = site.root_page.specific.route(request, path_components) <<-----
        page, args, kwargs = site.root_page.localized.specific.route(request, path_components)
        for fn in hooks.get_hooks('before_serve_page'):
            result = fn(page, request, args, kwargs)
            if isinstance(result, HttpResponse):
                return result
        return page.serve(request, *args, **kwargs)

 setattr(EntryPageServe,'get',get)

Astrophotography gallery or how to extend the puput model

Fortunately, puput allows you to extend the page model very easily without having to rewrite a lot of code or mess with the original.

On the one hand I write my own blog post abstract class, adding a new field of type StreamBlock:

   class ContentBlock(StreamBlock):
      heading = blocks.CharBlock(form_classname="full title")
      paragraph = blocks.RichTextBlock()
      markdown = MarkdownBlock(icon="code")
      image = ImageChooserBlock()

   class MyEntryAbstract(EntryAbstract):
      body = RichTextField(verbose_name=_('body'),blank=True)
      content = StreamField(ContentBlock(),blank=True)
      EntryAbstract.content_panels = [
         MultiFieldPanel(
            [        
               FieldPanel('title', classname="title"),
               ImageChooserPanel('header_image'),
               FieldPanel('excerpt', classname="full"),
               FieldPanel('body', classname="full"),
               StreamFieldPanel('content'),
            ],       
            heading=_("Content")
         ),    
      ] + EntryAbstract.content_panels[1:]

      class Meta:
         abstract = True

We can forget, as the documentation indicates, adding the following lines in the properties file settings/base.py of the django environment where the web will run. This way the Entry class will inherit the methods of MyEntryAbstract:

    MIGRATION_MODULES = {'puput': 'blog.puput_migrations'}
    PUPUT_ENTRY_MODEL = 'blog.abstracts.MyEntryAbstract'

On the other hand, I write a new child class of the EntryPage of puput, inheriting the methods of the previous abstract class, as we have explained.

  class TakenDate(Orderable):
      taken_date =  models.DateField(blank=True);
      page = ParentalKey(
         'AstrophotoEntryPage',
         on_delete=models.CASCADE,
         related_name='taken_dates',
      )
      panels = [
         FieldPanel('taken_date')
      ]

   class TimeUnitsChoicesIterable(object):
      def __iter__(self):
         tagchoices = [('s','Seconds'),('ms','miliSeconds'),('fps','Frames per Second')]
         return tagchoices.__iter__()

   class TakenFrames(models.Model):
      filter =  models.CharField(max_length=20, blank=False)
      frames =  models.PositiveSmallIntegerField(blank=False)
      time =  models.PositiveSmallIntegerField(blank=False)
      time_unit =  models.CharField(max_length=3, blank=False, default="s")
      temperature =  models.IntegerField(blank=True,null=True)
      page = ParentalKey(
         'AstrophotoEntryPage',
         on_delete=models.CASCADE,
         related_name='taken_frames',
      )
      timeunit_select = forms.Select()
      timeunit_select.choices = TimeUnitsChoicesIterable()
      panels = [
         FieldPanel('filter'),
         FieldPanel('frames'),
         FieldPanel('time'),
         FieldPanel('time_unit',widget=timeunit_select),
         FieldPanel('temperature'),
      ]        

   class AstrophotoEntryPage(EntryPage):
      camera = models.CharField(max_length=80, blank=True)
      filters = models.CharField(max_length=200, blank=True)
      telescope = models.CharField(max_length=80, blank=True)
      mount = models.CharField(max_length=80, blank=True)
      guiding = models.CharField(max_length=200, blank=True)
      other = models.CharField(max_length=400, blank=True)
      software = models.CharField(max_length=200, blank=True)
      site = models.CharField(max_length=80, blank=True)
      content_panels = EntryPage.content_panels[:1] + [
         MultiFieldPanel([              
          InlinePanel('taken_dates', min_num=1, max_num=6, label="Dates"),
          InlinePanel('taken_frames', label="Frames"),
          FieldPanel('site'),
         ], heading="Adquisition Details"),
         MultiFieldPanel([
          FieldPanel('camera'),
          FieldPanel('filters'),
          FieldPanel('telescope'),
          FieldPanel('mount'),
          FieldPanel('guiding'),
          FieldPanel('other'),
          FieldPanel('software'),
         ], heading="Equipment"),
      ] + EntryPage.content_panels[1:]

   BlogPage.subpage_types.append(AstrophotoEntryPage)

This last line of code is essential, if I want to add this new type of post, in addition to the original one.

Carousel of blog posts with puput

One of the features I was looking for was to be able to display carousels of images and blog categories. To do this we will build several template tags blog/templatetags/blog_tags.py:

 from importlib import import_module
 from django.template import Library
 from django.urls import resolve
 from django.template.defaultfilters import urlencode
 from django.template.loader import render_to_string
 from django_social_share.templatetags.social_share import _build_url
 from el_pagination.templatetags.el_pagination_tags import show_pages, paginate
 from puput.utils import import_model
 from puput.models import Category, Tag
 from wagtail.core.models import Site, Locale
 import uuid

 register = Library()

 @register.inclusion_tag('puput/tags/my_entries_list.html', takes_context=True)
 def my_recent_entries(context, limit=None, page=None, request=None):
    blog_page_m = import_module('puput.models').BlogPage
    locale_id=context['page'].locale.id
    blog_page = blog_page_m.objects.filter(locale=locale_id).first()
    context['blog_page'] = blog_page
    context['carousel_id'] = str(uuid.uuid4())
    entries = blog_page.get_entries().exclude(header_image__isnull=True).order_by('-date')
    if limit:
        entries = entries[:limit]
    context['entries'] = entries
    return context

 @register.inclusion_tag('puput/tags/my_entries_list.html', takes_context=True)
 def my_entries_by_category(context, category, limit=None):
    blog_page_m = import_module('puput.models').BlogPage
    locale_id=context['page'].locale.id
    blog_page = blog_page_m.objects.filter(locale=locale_id).first()
    context['blog_page'] = blog_page
    context['carousel_id'] = str(uuid.uuid4())
    entries = blog_page.get_entries().filter(entry_categories__category__name=category)
    entries = entries.exclude(header_image__isnull=True).order_by('-date')
    if limit:
        entries = entries[:limit]
    context['entries'] = entries
    return context

 @register.inclusion_tag('puput/tags/my_entries_list.html', takes_context=True)
 def my_entries_by_tag(context, label, limit=None):
    blog_page_m = import_module('puput.models').BlogPage
    locale_id=context['page'].locale.id
    blog_page = blog_page_m.objects.filter(locale=locale_id).first()
    context['blog_page'] = blog_page
    context['carousel_id'] = str(uuid.uuid4())
    entries = blog_page.get_entries().filter(tags__name=label)
    entries = entries.exclude(header_image__isnull=True).order_by('-date')
    if limit:  
        entries = entries[:limit]
    context['entries'] = entries
    return context

with its corresponding template puput/tags/my_entries_list.html for the presentation:

 {% load i18n wagtailimages_tags puput_tags %}

 <div id="mycarousel_{{ carousel_id }}" class="carousel slide mycarousel-{{self.carousel_type}} {%if self.label%}blog-carousel-{{self.label}}{%endif%}" data-ride="carousel">
    <div class="carousel-inner">
        {% for entry in entries %}
            {% image entry.header_image fill-800x450 as carousel_img %}
            <div class="carousel-item {% if forloop.first %}active{% endif %}">
              <a href="{% entry_url entry blog_page %}">
                 <img src="{{ carousel_img.url }}" class="d-block w-100" alt="{{ carousel_img.alt }}">
              </a>
              <div class="carousel-caption">
                <h3>{{ entry.title }}</h3>
                <p>{{ entry.date|date:"SHORT_DATE_FORMAT" }}</p>
              </div>
            </div>
        {% endfor %}
    </div>
  <a class="carousel-control-prev" href="#mycarousel_{{ carousel_id }}" role="button" data-slide="prev">
    <span class="carousel-control-prev-icon"></span>
  </a>
  <a class="carousel-control-next" href="#mycarousel_{{ carousel_id }}" role="button"  data-slide="next">
    <span class="carousel-control-next-icon"></span>
  </a>
</div>

In this way, we can include the carousel in a normal template, looking like this:

   <section>
      <h3>{% trans "Blog Last Entries" %}</h3>
      {% my_recent_entries 10 %}
      <h3>Category Entries</h3>
      {% my_entries_by_category "Astronomy" 5 %}
      <h3>Tags Entries</h3>
      {% my_entries_by_tag "Planetary" 5 %}
   </section>

To finish, this would be a StructBlock with a carousel of latest blog posts,

 class CarouselBlogChoiceBlock(blocks.ChoiceBlock):
    choices = (
        ('all', 'Recent entries'),
        ('tagged', 'Recent tagged entries'),
        ('categories', 'Recent categories entries'),
    )
 class AlignChoiceBlock(blocks.ChoiceBlock):
    choices = (
        ('left', 'Float Left'),
        ('right', 'Float Right'),
        ('center', 'Center'),
    )
 class CarouselBlogBlock(StructBlock):
    carousel_type = AlignChoiceBlock(label='Carousel Type', default='center')
    carousel_blog_type = CarouselBlogChoiceBlock(label='Entry Filter', default='all')
    label = CharBlock(max_length=80)
    limit = IntegerBlock(min_value=5, max_value=20, default=5)
    class Meta:
        icon = "cog"
        template = "streamfieldblocks/carousel_blog_block.html"

leaving the template carousel_blog_block.html like this,

 {% load wagtailimages_tags wagtailcore_tags blog_tags %}
 {% if self.carousel_blog_type == 'all' %}
   {% my_recent_entries limit page request%}
 {% elif self.carousel_blog_type == 'categories' %}
   {% my_entries_by_category self.label self.limit %}
 {% elif self.carousel_blog_type == 'tagged' %}
   {% my_entries_by_tag self.label self.limit %}
 {% endif %}

LaTeX support for mathematical formulas and code coloring

Support for $\LaTeX$ was done using javascript thanks to the MathJax library, some configuration code in the base template

   <script type="text/x-mathjax-config">
   MathJax.Hub.Config({
      extensions: ["tex2jax.js"],
      jax: ["input/TeX", "output/HTML-CSS"],
      tex2jax: {
      inlineMath: [['$','$']],
      displayMath: [['$$','$$']] ,
      processEscapes: true
      }, 
      "HTML-CSS": { availableFonts: ["TeX"] }
   });
   </script>
   ...
   <script type="text/javascript" 
      src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/MathJax.js?config=TeX-AMS-MML-AM_CHTML">
   </script>

and some markdown integration code where we also add support for codehilite to give the code some color, mysite/md_converter/utils.py:

   import markdown

   def render_markdown(value):
   html = markdown.markdown(
      value,
      extensions=[
         'extra',
         'codehilite',
         'mysite.md_converter.mdx.mdx_mathjax',
      ],
      extension_configs={
         'codehilite': [
          ('guess_lang', False),
         ]
      },
      output_format='html5'
   )
   return html

This would be the extension code, based on python-markdown-mathjax and considering this stackoverflow thread, mysite/md_converter/mdx/mdx_mathjax.py:

   import markdown                      
   import html                          

   class MathJaxPattern(markdown.inlinepatterns.Pattern):
      def __init__(self, md):
         markdown.inlinepatterns.Pattern.__init__(
            self, r'(?<!\\)(\$\$?)(.+?)\2', md)

      def handleMatch(self, m):
         text = html.escape(m.group(2) + m.group(3) + m.group(2))
         return self.markdown.htmlStash.store(text)

   class MathJaxExtension(markdown.Extension):
        def __init__(self, **kwargs):
         super(MathJaxExtension, self).__init__(**kwargs)

      def extendMarkdown(self, md, md_globals):
         md.inlinePatterns.add('mathjax', MathJaxPattern(md), '<escape')

    def makeExtension(configs={}):
      return MathJaxExtension(**configs)

You should also add the following templatetag

   from django import template
   from mysite.md_converter.utils import render_markdown

   register = template.Library()

   @register.filter(name='markdown')
   def markdown(value):
      return render_markdown(value)

to use it as a filter in the main html templates:

  ...
      </div>
  {% elif iner_block.block_type == 'markdown' %}
      <div class="block-{{ iner_block.block_type }}">
          {{ iner_block.value|markdown|safe }}
      </div>
  {% elif iner_block.block_type == 'carousel_blog' %}
  ...

and once everything is configured, as an example, this would be the result:

\begin{equation} \begin{aligned} \frac{\partial \vec {\mathcal D}}{\partial t} & = \nabla \times \vec{\mathcal H} \\ \frac{\partial \vec {\mathcal B}}{\partial t} & = -\nabla \times \vec{\mathcal E} \\ \nabla \cdot \vec {\mathcal B} & = 0 \\ \nabla \cdot \vec {\mathcal D} & = 0 \end{aligned} \end{equation}

or this way, which I like better

\begin{equation} \begin{aligned} \nabla \times \vec{\mathcal B} & = \mu_0 \left( \vec {\mathcal J} + \varepsilon_0 \frac{\partial \vec {\mathcal E}}{\partial t} \right) \\ \nabla \times \vec{\mathcal E} & = - \frac{\partial \vec {\mathcal B}}{\partial t} \\ \nabla \cdot \vec {\mathcal B} & = 0 \\ \nabla \cdot \vec {\mathcal E} & = \frac{\rho}{\varepsilon_0} \end{aligned} \end{equation}

If we look at the generated html code we see an xml structure MathML, so that browsers that support it can render the formulas directly.

Cookies disclaimer

I agree Our site saves small pieces of text information (cookies) on your device in order to deliver better content and for statistical purposes. You can disable the usage of cookies by changing the settings of your browser. By browsing our website without changing the browser settings you grant us permission to store that information on your device.