Automated Installation of exa in WSL

Recently, I was pointed to exa by this tweet from Mathias.

The installation is pretty easy. Unless you are using Ubuntu (as WSL) as there is no package available. But then again, compiling it by yourself is pretty straightforward as well.

1. Download and install Rust for your platform.

2. Install libgit2 and cmake.

3. To download the latest version, run: git clone https://github.com/­ogham/exa.git

4. Run make install in the new directory to compile and install exa into /usr/local/bin.

As you might know, I am using Ansible to install all of my WSL instances. Eventually, things did turn out not so easy. However, two evenings later, I finished an Ansible role doing this fully automated.

Eventually, you have to set the variables for exa_dir and rust_dir to make this role working. That way, you not only get a great tool, but you also get it fully automated into your Ubuntu WSL.

Link: https://the.exa.website/
Gist: https://gist.github.com/aheil/387336a46938ff5c53ea51a1591f6ca5

Automation of the Home Automation

After we talked a lot about it in our podcast, I finally started with my “Automation of the Home Automation” project.

What I’ve done so far: Set up a Raspi 4 with Ubuntu Server, fully provisioned using Ansible. Also, it seems, the Kernel bug causing USB devices to fail on the 4 GB version of the Raspi 4 seams to be removed with the most recent binaries available.

Deployed MQTT, InfluxDB, Telegraf and Node-Red on Docker containers using Ansible.

Wrote my very first Node-Red flow to get data into the broker and the database:

The data is read from my EZcontrol XS1, which became surprisingly easy using Node-Red.

The EZcontrol XS1 integration for Home Assistant allows you to observe and control devices configured on the XS1 Gateway. Please have a look at the official docs for using this gateway.

While I was not sure about the available resources on the Raspberry, it seems there is plenty of space (RAM, CPU) left on the device.

On the other side, using Ansible I will be able to deploy services to other (more) nodes once necessary.

Unfortunately, docker-ce is not yet available for Ubuntu 19.10 Eoan. Therefore, adding the Docker repositories will fail.

A simple workaround is to add the repositories for disco instead.

The Ansible code to do so is quite simple:

- name: Add stabel repository 
  apt_repository: 
    repo: deb [arch=amd64] https://download.docker.com/linux/ubuntu "disco" stable
    state: present  

Also, I realized some issues with Ansible on Ubuntu running on the Raspi and Python. Therefore, I made sure Python 2.7 is entirely uninstalled. Instead, I made sure my inventory file is set to use Python 3 on the target system.

[raspi]
raspi4

[raspi:vars]
ansible_python_interpreter=/usr/bin/python3

There have been some other issues, I came along, especially because for various bits OI was looking for there are no ARM aarch64 releases available. Also, the official Docker images I used do not support Apline on ARM v8 / aarch64 as Alpine seems not to support this target architecture, yet.

More bits and bytes to be added soon…

Powerline Fonts in Windows WSL

Just in case you like modding your environment as I do, you might end up as well in a situation where you SSH into a remote server or using WSL requiring Powerline Fonts. And just in case, if you are wondering what Powerline is? Here you go:

Powerline is a statusline plugin for vim, and provides statuslines and prompts for several other applications, including zsh, bash, tmux, IPython, Awesome and Qtile. 

The problem arises if your host system is Windows and you are using SSH and WSL a lot as I do.

Stanley Meng provides a condensed instruction set on how to get Powerline fonts on your Windows system up and running.

Once you installed the fonts, you can select them e.g. for WSL:

Update (2019-11-18)
While I was looking for the new Cascadia Code font providing Powerline glyphs, I was pointed to Delugia, which in fact is Cascadia Code extended with Powerline glyphs.

Powerline: https://github.com/powerline/powerline
Powerline on Windows: https://medium.com/@slmeng/how-to-install-powerline-fonts-in-windows-b2eedecace58
Bash on Ubuntu on WSL: https://medium.com/@jrcharney/bash-on-ubuntu-on-windows-the-almost-complete-set-up-1dd3cb89b794
Powerline Fonts: https://github.com/powerline/fonts
Delugia Font: https://github.com/adam7/delugia-code/releases

Fixing USB Problems Ubuntu Server on Raspberry 4

When trying to rung Ubunto Server on a Raspberry 4 with 4 GB RAM, you might end up with the same problem as me: USB is not working at all.

The cause is a kernel bug affecting 4 GB version of the Raspberry 4 Model B only. 1 and 2 GB versions are working quite fine.

The issue has been identified and it’s been found to be a kernel bug with a solution in the works that being tested. In the meantime, you can access to your Raspberry Pi 4 4GB USB ports by limiting the memory to 3GB in /boot/firmware/usercfg.txt […]

To limit the available RAM to 3 GB just add

total_mem=3072

to the usercfg.txt.

Link: https://www.cnx-software.com/2019/11/04/raspberry-pi-4-4gb-models-usb-ports-dont-work-on-ubuntu-19-10/

Automatically upgrading Ubuntu on WSL to the latest release

As I use multiple Windows machines, all running the identical setup of WSL (Ubunto) I wanted to keep all these instances automatically updated.

I am using Ansible scripts to set up the WSL. While all the files are hosted on the file system (\mtn\c\…) I can throw away a WSL instance and create a new one within minutes.

While many packages are not available on Ubunto 18.04 LTS, I created a small Ansible task upgrading my WSL to the latest Ubuntu version available.

  1. Start your WSL instance the very first time and run sudo apt-get update
  2. Install ansible by running sudo apt-get install ansible
  3. Make sure the Ansible scripts are located on the host system (e.g. c:\dev\…)
  4. Start your WSL instance and change e.g. to /mnt/c/dev
  5. Run your Ansible playbook as sudo

You can check your dist version with lsb_release -a:

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 19.04
Release:        19.04
Codename:       disco

The Ansible tasks can be found here.

GitHub Gist: https://gist.github.com/aheil/6d48a77ac44c83e00666d3c49ffd7890

How to deal with GMAIL Undelivered Mail Returned to Sender

If you run your own mail server, you might end up quite frustrated because the Google relays do reject your mails.

Maybe your mails get rejected by the Google servers with the following message:

Our system has detected that this  message does not meet IPv6 sending guidelines regarding PTR records and authentication. Please review  https://support.google.com/mail/?p=IPv6AuthError for more information.

If you start reading your frustration level might even increase, due to the difficulty to deal with many of the requirements to get your mails to a Google server delivered. If you start digging into the various topics, you might end up even more frustrated as you have more questions as before.

At the very end, you just have to set up a few things to make your mails fully compliant, so even Google’s server do accept them. In addition you will get high quality mails – at least regarding the tech, content is entirely up to you.

I do run a setup of mail, Dovecot, Postfix and SpamAssasin on a Ubuntu server. Therefore, this article will cover the topic on these examples but should be able to be applied to almost any other system – as long as you replace tech X by tech Y.

Testing you Mail

The first issue is how to learn about your mails issues at all. To evaluate your mails, http://www.mail-tester.com turned out to be a perfect platform for me. As you only can evaluate three mails a day for free, you might throw in a few bucks if you in a hurry, or it might simply take some days to you to finish the tests.

How does it work? You get an unique mail address where you send your mail to. Once the mail was received any flaws in the mail will be displayed and explained (which definitely is a huge added value).

10/10 Score at mail-test.com

Believe me or not, I definitely haven’t had a score of 10/10 before.

SPF Record

First make sure, your DNS provides an SPF record. With no or little knowledge, I tried to create one by myself based on Google’s help page about SPF records. Did not work well at all.

I finally ended up with the following SPF entry for my DNS:

v=spf1 mx ip4:92.51.166.153 -all
  • a is just to indicate that there might be scripts sending mails
  • ip4:92.51.166.153 depicts the mails server we are talking about in the entry
  • -all means there only this mail server while the mx indicates that only domain’s MXs are allowed to send mail for the domain while all others are prohibited.

It took me a while to figure out the right (or at least something working) based in the SFP record syntax. So why do we do this? The idea behind SPF is simple, once you have published your policy, the receiving server can check if the mail is compliant with your policy. In my case, mails sent from scripts might be fine, as long as they come from my server. If someone sends a mail from server 154.354.32.2 this mail will be probably not sent by me, as it is not compliant with my policy. Google’s server do these checks, and that’s on of the reasons your mails are rejected by them.

Finally the SPF entries doe look like the following at my DNS:

DKIM – DomainKeys Identified Mail

Next, I started with DKIM. Again, if you start reading at dkim.org, you probably end up reading several IETF RFCs and a lot of question. Reading RFCs is always good and highly recommended by me, but actually did not solve my issue.

The good thing about Ubuntu is, there is an package called OpenDKIM which can be installed via

sudo apt-get install opendkim opendkim-tools

I got relatively lost, when i came to configure OpenDKIM and Postfix until I found the Postfix/DKIM site at the Ubuntu documentation.

First of all you need to add the domain to the /etc/opendkim.conf. You might want to exchange my domain aheil.de to your corresponding one.

Domain    aheil.de
KeyFile   /etc/postfix/dkim.key 
Selector   dkim

Now, I had to generate the key file.

opendkim-genkey -t -s dkim -d aheil.de

If you did it the same way as me, the file is probably located in the wrong directory. So move the file to the location provided above.

mv dkim.private /etc/postfix/dkim.key

Now I went to /etc/default/opendkim. Unlike as described in the Ubuntu documentation, I had to add the listening socket – which seems to be a standard one at Ubuntu.

SOCKET="inet:8891@localhost"

I then went back to /etc/opendkim.conf and added the same port right below the Selector entry.

Domain    aheil.de 
KeyFile   /etc/postfix/dkim.key 
Selector  dkim
Socket    inet:8891@localhost

Now, I went to the /etc/postfix/main.cf file and added the following entries:

milter_default_action = accept
milter_protocol = 2
smtpd_milters = inet:localhost:8891
non_smtpd_milters = inet:localhost:8891

For some reason, I now ended up with two DKIM entries in my outgoing mails. I had to add no_milters to the receive_override_options line in my /etc/postfix/master.cf. I had to fiddle a little bit with the master.cf until everything worked again.

-o receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_milters

Wherever you have created the keyfile, there will be an additional dkim.txt file. It will contain something like

dkim._domainkey IN TXT ( "v=DKIM1; k=rsa; t=y; "
 "p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCZFYWrNHupXvZHvWJAo6aGB6sXYf+kpUIJv+BcP6NFiF9GBy26oYZKt6/ngCXmNAIY9+yReY8UxG5GIm/QnInbXWxwDXyD0MHD8HrhHyVa6JVqTncexm29M5Bbp/u0JI4ToOQwIQqpgTr0t9ADP8i76pbelYQ5IQDOwJRJQ2N1iQIDAQAB" ) ; ----- DKIM key dkim for aheil.de

Creating the corresponding DNS entry out of this seemed to be the most challenging part. You need to create an TXT entry similr to dkim._domainkey.aheil.de, of course using your domain. and you simply need the follwowing text in there

v=DKIM1;
p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCZFYWrNHupXvZHvWJAo6aGB6sXYf+kpUIJv+BcP6NFiF9GBy26oYZKt6/ngCXmNAIY9+yReY8UxG5GIm/QnInbXWxwDXyD0MHD8HrhHyVa6JVqTncexm29M5Bbp/u0JI4ToOQwIQqpgTr0t9ADP8i76pbelYQ5IQDOwJRJQ2N1iQIDAQAB

No quotes, no brackets, no nothing beside the above. This took me quite a while especially as DNS replication is nothing that works in real time.

For me the entry does now look like the following.

Once this is done, start OpenDKIM and restart Postfix so the changes will apply.

sudo service opendkim start
sudo service postfix restat

So far what does this actually mean? When a mail server receives your mail, it now will be signed with a DKIM key. If you check your mail headers, the will look similar to the following one.

By checking the dkim._domainkey DNS entry created before, the mail receiver now can check if the mail signature is authentic. Any mail server not in possession of the private key will probably not be able to sign the mail though, and could be thought of a spam server.

DMARC – Domain-based Message Authentication, Reporting & Conformance

Finally, it is necessary to create a DMARC record. Once SPF and DKIM are setup as above, you proceed similar as with the DKIM DNS entry.

Again you create a TXT revord for your DNS. This time the record is named _dmarc.domain.TDL, e.g. _dmarc.aheil.de in my case.

I created a relatively simple record:

v=DMARC1; p=none

I.e. p=none: the I do not requests any specific action be taken on mail that fails DMARC authentication and alignment.

At the very end, it looks like this for my DNS:

Summary

To get Google accepting mails from your own mail server, only a few steps are necessary. Setting up SPF, DCIM and DKIM are basically all which might be needed if Google rejects mails from your handcrafted mail server. You might want polish your SPF and DMARC records to align your policies with your very personal goals. If you are interested further in e-mail architecture, the book from Kevin Thomas called Email Architecture, Design, and Implementations might be something for your.