Automated Housekeeping of your Twitter Footprint

As we discussed in our podcast, I am working on reducing my social media footprint. While I use Twitter quite a lot for communicating, it is almost not possible to find information on my Twitter feed once I am looking for it.

I was pointed to Amnesia, a small Python script using the Twitter API, deleting old tweets and likes. The goal is to run this housekeeping as automated as possible.


The setup is relatively simple:

  1. Use a Docker container hosting the script
  2. Run the script on a daily base
  3. Deploying the container automatically using Ansible

The Repository

Instead of cloning the repository locally and adding all files manually to the deployment process, we will clone the repository directly to be used by the container. Here we consider two aspects:

  1. We could clone the repository directly into the container. This will leave a minimal footprint on the host but would require to have Git in your container.
  2. We clone the repository to the file system of the host and bind it within the container.

The second aspect will be the source of the repository. If cloned directly from GitHub, it might appear convenient to clone the repository directly from the original master. That way you would always have an up to date version of the script. The drawback here is that you will probably not be aware of any (breaking) changes or malicious code introduced in the repository.

To avoid this, I created a fork of and will use the forked and reviewed repository at instead for the deployment.

The Twitter API

Do use the Amnesia script you will need to sign up for a Twitter developer account.

This process might take 10 to 15 minutes as Twitter introduced a verification and review process. However, when using the personal or happy developer path you will probably end up in an automated enrollment process.

Once enrolled and logged in at the Twitter developer page, under our apps detail page at Keys and tokens you will find the information needed for the Amnesia script later on.

You probably need to click Generate once to obtain a valid access token ad access token secret for your app.

Scheduling the Script

Julien Dubreuil shows how to manage Docker containers using crontab. The pattern provided there can be adapted relatively easily for our needs.


First of all, we generate a crontab file

* * 1 * * python /var/amnesia/ >> /var/log/cron/cron.log 2>&1

Make sure the file ends with a new line LF character (as required by POSIX systems). Otherwise, cron might end up in refusing to process your crontab file.

The crontab entry will cause the amnesia script to be executed once a day. Also, when executed all log entries will be added to the cron.log.

The second file we need is a shell script to be executed later in the container.

crond -L /var/log/cron/cron.log "$@" && tail -f /var/log/cron/cron.log

-L will cause crond to log into the file specified, $@ will pass additional parameters to the script (although we are not using any at the moment) and we will tail the log file so we see something in the container log.

The Docker Image

We will create a Docker image for the Amnesia script. Therefore, we have to prepare the Docker image, apply the setup steps for the script and set up the cron jobs specified before.

My first draft (probably not a perfect one) of the Dockerfile for the service looks currently like the following:

FROM alpine
RUN apk --no-cache add bash python2 py-virtualenv
RUN virtualenv venv
RUN source venv/bin/activate
COPY repo/requirements.txt /requirements.txt
RUN pip install -r /requirements.txt
COPY crontab /tmp/crontab
RUN cat /tmp/crontab >> /etc/crontabs/root
RUN chmod -v +x /
RUN mkdir -p /var/log/cron && touch /var/log/cron/cron.log
CMD ["/"]

First, we install the required packages (python2 and virtualenv) and go through the setup steps for the Amnesia script. Before running pip install I copy the requirements.txt file to be accessed in the container. Afterward all prerequisites for the script are installed. You don’t have to bother about the script itself, as it will be executed by the cron job specified above.

Similar to the requirements.txt file I copy our crontab file to be concatenated to the crontab file. To execute the script file specified before I copy this as well and make it executable.

After creating folders and files for logs the script can be started using the CMD command.

Ther might be better ways to set up the container, however, I have chosen a quite straightforward approach to get the first draft working quickly.

At this point, you could already build and run the container if you clone the Amnesia repository into a folder called repo.

The Secrets

Before we start creating the deployment scripts, we have to take care of our Twitter keys and secrets. I would not recommend to write them down in any configuration files in plain text. As we will use Ansible for the deployment, we can use Ansible Vault to encrypt the sensitive information on our local system. Keys will be then decrypted during the automated deployment process on the target system.

You end up with something similar to

> MacBook:$ ansible-vault encrypt_string h5fk43556jgGdfER4
> !vault |
>    $ANSIBLE_VAULT;1.1;AES256
>    54675433456436790545679876544567776u76535654563421000
>    38633933326565663731356524267653367632467654987654554
>    43072352520500023523523875275002035273528375235 

You’ll do this with your consumer_key, consumer_secret, access_token and access_token_secret and add this into the corresponding Ansible files.

consumer_key: !vault |

We will use these keys later in the Ansible tasks e.g. when replacing lines in the configuration script:

   line: "    'consumer_key': '{{ consumer_key }}',"

When I started with Ansible, it took me quite a while to figure out how to store the encrypted data. The easiest way however is, to copy the encrypted string with all the whitespaces directly into your configuration files.

Automated Deploying

The final step is now to create your ansible script. I am using the recommended directory layout for ansible roles. Therefore, all files we have generated so far are located in a folder called files. The variables with encrypted secrets are located in a file main.yml in the folder defaults while the main.yml containing the script we are creating now lives in a folder called tasks.

The overall structure for this project looks like the following:

The Ansible Script

Usually, I start by creating the directories I need on the server


- name: Make sure install directory is present
path: "{{ amnesia_install_dir }}"
state: directory
- amnesia

The directory I use here is stored as variable in the main.yml file in my defaults folder as shown above. Each role on the server gets usually its own directory where I store all uploaded data.

amnesia_install_dir: /opt/amnesia

Very important to keep in mind: Everything in this folder will be overwritten bythe scripts and is autoamtically deployed or generated. Consequently, you shouldn’t store any persistant data in this folder. If this is the case, I use a different location on the server. However, this is nit necessary for this particular role.

- name: Copy the required files
    src: files/
    dest: "{{ amnesia_install_dir }}"
    force: yes
    - amnesia

- name: Clone amnesia repo
    # Master GitHub Rpo:
    # We use our own fork instead to avoid breaking changes
    version: master
    dest: "{{ amnesia_install_dir }}/repo"
    clone: yes
    update: yes
    force: yes
    - amnesia

We now copy everything from the local files director to the server. That’s the reason I also provide a .dockerignore file in this directory. Everything on the server not needed within the Docker image context can be excluded in this file. This will reduce your image’s footprint. Again, there are other (and maybe better) ways to do so, but this worked for my use case quite well.

In the second step, the GitHub repository (remember, the forked one) is cloned onto the server. As the repository is cloned into a folder called repo. That’s the reason the requirements.txt file was copied from this location in the Dockerfile.

- name:  Copy files remote to remote
    src: "{{ amnesia_install_dir }}/repo/"
    dest: "{{ amnesia_install_dir }}/repo/"
    remote_src: yes
    - amnesia

#- name: turn on dry run
#  lineinfile:
#    path: "{{ amnesia_install_dir }}/repo/"
#    regex: "^\\s+'dry_run':"
#    line: "    'dry_run': True,"
#    state: present
#  tags:
#    - amnesia

Following the Amnesia instructions, we have to copy the file for further configuration. Once the file is copied you can modify the settings using the lineinfile task in Ansible.

In the next few tasks, we finish the configuration by setting all the secrets and the duration of tweets. Once the script is executed, all the secrets will be encrypted and deployed on the server automatically.

- name: set Twitter consumer_key 
    path: "{{ amnesia_install_dir }}/repo/"
    regex: "^\\s+'consumer_key':"
    line: "    'consumer_key': '{{ consumer_key }}',"
    state: present
    - amnesia

- name: set Twitter consumer_secret 
    path: "{{ amnesia_install_dir }}/repo/"
    regex: "^\\s+'consumer_secret':"
    line: "    'consumer_secret': '{{ consumer_secret }}',"
    state: present
    - amnesia

- name: set Twitter access_token 
    path: "{{ amnesia_install_dir }}/repo/"
    regex: "^\\s+'access_token':"
    line: "    'access_token': '{{ access_token }}',"
    state: present
    - amnesia

- name: set Twitter access_token_secret 
    path: "{{ amnesia_install_dir }}/repo/"
    regex: "^\\s+'access_token_secret':"
    line: "    'access_token_secret': '{{ access_token_secret }}',"
    state: present
    - amnesia

- name: set Twitter delete_after_days 
    path: "{{ amnesia_install_dir }}/repo/"
    regex: "^\\s+'delete_after_days':"
    line: "    'delete_after_days': 7,"
    state: present
    - amnesia


The very last task in the Ansible script is to create and start the containers.

- name: Start container
    project_src: "{{ amnesia_install_dir }}"
    build: yes
    remove_orphans: yes
    state: present
    restarted: yes
  register: output
    - amnesia

To make this task succeed we need a docker-compose.yml file as below:

---version: '3'
    container_name: amnesia
    build: .    
    restart: always    
    - ./repo:/var/amnesia

To make all the script above work we have to bin the directory repo on the host to /var/amnesia as this is used in the container several times to make the files accessible within the container. You can choose any other layout based on your preference, though.

Given you have a Ansible playbook and a host file, you now can deploy the service using

ansible-playbook playbook.yml -i hosts  --tags amnesia


It turned out to be one evening of work to get the container and script done. I had to do some research on some details as well but was able to base on some of my previous Ansible projects.

As long as the Twitter API is not changed (or the developer tokens have benn revoked), this is a quite nice solution to keep my Twitter footprint on a minimal level.

The usage of Docker and Ansible adds a not to undererstimated level of complexity to the overall solution. Without (i.e. you install and run the script on your host) the Amnesia script is installed within a few minutes. However, with the solution provided here, it can be deployed on any of your servers (locally, hosted, virtual etc.) again and again.

Proper Logwatch Configuration using Ansible

On my way setting up a proper monitoring for my server, I just installed Logwatch to receive a daily summary of the what happened recently on the machine. I will have a look into Prometheus, Grafana, Icinga etc. later. However, for now I just wanted a quick summary of the daily “what’s going on on the machine”. Eventually, I had to fix an occur No such file or directory error.

Therefore, I decided to use Logwatch as a lightweight solution to my needs.

Installation Script

The Ansible script to install Logwatch is straight forward:

- name: Install logwatch
name: logwatch
state: latest
- logwatch

- name: Create logwatch.conf file for customisations
path: /etc/logwatch/conf/logwatch.conf
state: touch
- logwatch

- name: E-Mail to
dest: /etc/logwatch/conf/logwatch.conf
regexp: "^MailTo ="
line: "MailTo = {{ logwatch_email }}"
state: present
- logwatch

- name: Set detail
dest: /etc/logwatch/conf/logwatch.conf
regexp: "^Detail ="
line: "Detail = {{ logwatch_detail }}"
state: present
- logwatch

Configuration & Troubleshooting

I basically set up two parameters, the e-mail as well as the detail level I want for the report. Important to know is the order Logwatch is applying your configuration settings. Following the recommendations, I did not change anything in the configuration file at


rather I decided to copy the file to


The reason is the order, logwatch is scanning for configuration parameters in the following order. Each step actually overwrites the previous one.

  • /usr/share/logwatch/default.conf/*
  • /etc/logwatch/conf/dist.conf/*
  • /etc/logwatch/conf/*
  • The script command line arguments

Eventually, I ended up in the following error:

/var/cache/logwatch No such file or directory at /usr/sbin/logwatch line 634.
run-parts: /etc/cron.daily/00logwatch exited with return code 2

To fix this, avoid copying the original configuration to one of the other places. I did this because I followed some recommendation I received. Instead, I now touch a new configuration file as well as setting the two parameters for MailTo= as well as Detail=. Both are s set using Ansible variables in my scripts. The additional configuration file now looks pretty boring, though:

MailTo =
Detail = Low

You also can provide these parameters when calling the script in the cron job: Using Ansible the modification would look like the following:

dest: /etc/cron.daily/00logwatch
regexp: "^/usr/sbin/logwatch"
line: "/usr/sbin/logwatch --output mail --mailto {{ logwatch_email }} --detail {{ logwath_detail }}"
state: present
create: yes

I decided to change the cron job call as one never can be safe from the file changing during package updates. The same should be valid for the configuration file at its origin place.


Setting up Logwatch using Ansible might cause strange “No file or directory”-errors during the cron job call. This can be avoided by applying additional configuration settings at appropriate configuration locations.

Personal DevOps #3

While most of the prerequisites are met for my automated server setup I came across some issues when I started with my very first Ansible playbooks.

First Ansible Playbooks

First of all, I wanted to start with a quite simple ping playbook, to ensure the servers are reachable by Ansible.

# Playbook to ping all hosts 
- hosts: all
  gather_facts: false
    - ping:

When I run this script I was immediately confronted with the very first error. I really love when such things happen. Nothing can motivate one more than immediate failures like the following.

FAILED! => {
"changed": false,
"module_stderr": "Shared connection to closed.\r\n",
"module_stdout": "/bin/sh: 1: /usr/bin/python: not found\r\n",
"msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
"rc": 127

As I started with a minimal Ubuntu 18.04 LTS installation, there is simply no Python 2 installed. However, to run the Ansible tasks on the node, Python is required. I made use of the raw task in Ansible to update the package lists as well as install the package python-minimal. In addition, I added the package python2.7-apt in this bootstraper as it is needed later on. Once Python has been installed the ping playbook worked without any problems.

# Bootstrap playbook to install python 2 and python-apt
# It checks first so no unecessary apt updates are performed
- hosts: all
  gather_facts: False
  - name: install python 2
    raw: test -e /usr/bin/python || (apt -y update && apt install -y python-minimal)
  - name: install python-apt 
    raw: test -e  /usr/lib/python2.7/dist-packages/apt || (apt install -y python2.7-apt)

For both packages, I test for the corresponding directories on the node to avoid unnecessary updates.

Note: When testing for a directory on the shell the following line became very handy:

> [ -e /usr/lib/python2.7/dist-packages/apt ] && echo "Found" || echo "Not found"

At a second step, I created a maintenance playbook to update and upgrade the packages on my node.

# Playbook to update Ubuntu packages 
- hosts: all
  gather_facts: false
  - name: update and upgrade apt packages
    become: true
      upgrade: yes
      update_cache: yes
      cache_valid_time: 86400

Before including the pyhton-apt package to the bootstraper, I got the following error when dry running the playbook.

fatal: []: FAILED! => {"changed": false, "msg": "python-apt must be installed to use check mode. If run normally this module can auto-install it."}


While this is not any rocket science for sure, I now have a few essential scripts to bring my server to a base level I can start working with.

Personal DevOps #2

Even though, I wanted to do this structured, beginning with this topic is quite a mess. You have to set up everything and pull all pieces together before one can start working properly.

Getting a Server

First of all, I picked up a new root server. I will install Ubuntu 18.04 LTS on this particular one. I found a great offer at Netcup, where I just ordered a new root server with unlimited traffic.

That way, I can start moving bit by bit from my old, handcrafted and home-brewed server to the new one instead of replacing the old server. Once everything works fine, I will migrate the data and change the DNS entries to the new server.

macOS as Ansible Control Machine

When starting with such a project, one should expect problems from the very first moment. For me, it started already when I wanted to install Ansible on macOX. Why I’ve chosen Ansible over Chef and Puppet will be covered later.

There is no native Ansible package for macOS available. Instead, you can use Homebrew to do so. Assuming Homebrew is already installed, the following command should do the job.

> brew install ansible

However, for me, it already ended in some errors:

xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools),
missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun

As most of the time the latest macOS version Mojave might be the reason, running

> xcode-select --install

the command line tools will be installed as part of Xcode. Once accomplished, the homebrew command should work like a charm. Eventually, Ansible is available on my local laptop which is now my control machine.

Setting up the SSH Access

Usually, I deactivate all password-based logins on a server and allow only RSA-key-based logins. To generate the key, you can follow the instruction on

To upload the key, ssh-copy-id is needed. However, once again this is not available on macOS and you have to install it using

> brew install ssh-copy-id

Now you can upload the key using

> ssh-copy-id -i ~/.ssh/key user@host

So far I have planned this was the only step necessary on the server. All future settings, including deactivating password only access should be set using Ansible scripts.

Setting up a Repository

As a (former) developer, I almost don’t do anything without putting my stuff into a revisioning system. For my Ansible scripts, I decided to set up a private GitHub repository. Although I use GitLab and Subversion at the moment at work as well as running a Subversion server at home, I meanwhile put almost everything int GitHub repositories. Therefore Github comes just in quite handy.

Why Ansible?

For automation, there are several frameworks. One major advantage of Ansible is the fact, only a single so-called control machine is needed. There is no further Ansible infrastructure needed. Once you have installed Ansible as described above you are ready to go. For Puppet, you need again to deal with a server and agents an eventually you stick with running daemons. While this is a feasible approach e.g. to manage my team’s 100 servers at work, this is an overkill for personal use. Same with Chef. That’s the reason, I decided to use Ansible as it seems a quite feasible approach for personal use with little overhead while being an approach with the least attack vectors.

If not being familiar with ansible, I recommend watching the following 14-minute video, which gives you a good overview of Ansible’s capabilities. You might want to skip the last four minutes as it deals with Red Hat’s Ansible Tower which targets enterprises.