Andres' Blog

Options to install R on a Raspberry Pi and other ARM systems

Sat, 11 Feb 2023 00:00:00 +0000

This started out as a section in some of my other posts but as installation options started to pile up it began to take too much space to be just a section in an article so I’ve decided to turn it into an article of its own.

The options described here are mostly applicable to the Raspberry Pi family of Single Board Computers but since most options also work on Ubuntu distributions they can also be applicable to other Linux ARM systems like other SBCs or even “full-size” ARM computers like Ampere Altra or AWS Graviton.

Install from the Operating System’s repositories

This is the easiest option, you can install R from the standard repositories for your OS, but those R versions are usually quite old, for example, the latest Raspberry Pi OS comes with R 4.2.2 and Ubuntu 22.04 comes with R 4.1.2.

To install R with this option you just need to run this command in your system terminal:

sudo apt install r-base

This is fine if you just want to try R or do simple stuff with it but if you are going to work on a serious project, chances are you want to start with an up-to-date version of R instead.

Compile R from source

Another option is to compile the latest R version yourself (4.3.1 at the moment of writing) from the source code files, this is a time-consuming task and it might feel a little intimidating for newcomers but it gives you the most flexibility when it comes to set compilation options, for example, you can choose options like compiling with BLAS and LAPACK to speed up matrix calculations.

You can compile R from source by running this script on a system terminal, it works for both Raspberry Pi OS and Ubuntu distributions:

sudo su
    
# Set the R version to install
R_VERSION=4.3.1

apt install -y g++ gcc gfortran libreadline-dev libx11-dev libxt-dev \
            libpng-dev libjpeg-dev libcairo2-dev xvfb \
            libbz2-dev libzstd-dev liblzma-dev libtiff5 \
            libssh-dev libgit2-dev libcurl4-openssl-dev \
            libblas-dev liblapack-dev libopenblas-base \
            zlib1g-dev openjdk-11-jdk \
            texinfo texlive texlive-fonts-extra \
            screen wget libpcre2-dev make cmake
cd /usr/local/src
wget https://cran.rstudio.com/src/base/R-${R_VERSION:0:1}/R-$R_VERSION.tar.gz
tar zxvf R-$R_VERSION.tar.gz
cd R-$R_VERSION
./configure --enable-R-shlib --enable-memory-profiling --with-blas --with-lapack #BLAS and LAPACK are optional
make
make install
cd ..
rm -rf R-$R_VERSION*
exit

Install R using rig

rig is an R installation manager that allows you to install multiple R versions side by side, it has recently added support for Linux ARM64 systems through the rstudio/r-builds GitHub repository, although, these builds are stated to be a community resource, they are not professionally supported by Posit. It currently supports only Ubuntu and Debian distributions (The latest Raspberry Pi OS version is based on Debian 12 so it is compatible too).

You can install rig in your system by running this command on a system terminal:

curl -Ls https://github.com/r-lib/rig/releases/download/latest/rig-linux-arm64-latest.tar.gz |
  sudo tar xz -C /usr/local

Then you can install the latest stable R version with these commands:

sudo rig add release # Or an specific version like 4.3.1
sudo rig default release

Optionally, you can manually download and install R from R-hub as explained on the rstudio/r-builds GitHub repository, but this is a more involved procedure only worthy if you are going to integrate this into your own infrastructure management workflow and you want to avoid the overhead of installing rig.

Install R from the R4Pi Project

And last but not least, my new favorite, you can install R from the R4Pi project through its deb repository, which means you will always get R version updates just by running sudo apt update && sudo apt upgrade. Currently supports the latest (Debian 12 based) Raspberry Pi OS version but support is 64-bits only. Also, there are plans to support Ubuntu 24.04 LTS arm64 in the near future.

To add the R4Pi deb repository to your deb source list and install R run these commands on a system terminal:

source /etc/os-release
curl -O  https://pkgs.r4pi.org/dl/${VERSION_CODENAME}/r4pi-repo-conf_0.0.1-1_all.deb
sudo dpkg -i  r4pi-repo-conf_0.0.1-1_all.deb
sudo apt update
sudo apt upgrade
sudo apt install r4pi
rm r4pi-repo-conf_0.0.1-1_all.deb

The main benefit of the R4Pi project is that it also provides R package repositories with precompiled binaries for a big selection of R packages which greatly reduces the time and effort required to install R packages on the Raspberry Pi, and by the way, you can also use the R4Pi package repository with all the previous installation options just by setting the “repos” option in a .Rprofile file.

local({r <- getOption("repos")
       r["R4Pi"] <- "https://pkgs.r4pi.org/bookworm"
       r["CRAN"] <- "https://cran.rstudio.com/" # Secondary repository for when no binary is available
       options(repos=r)})

Installing Quarto-CLI on Linux-arm64 systems

Fri, 09 Dec 2022 00:00:00 +0000

UPDATE:

This article is no longer relevant since the availability of official Quarto arm64 builds. Installation is pretty simple now:

# Download deb installer
wget https://github.com/quarto-dev/quarto-cli/releases/download/v1.3.433/quarto-1.3.433-linux-arm64.deb
# Install Quarto
sudo dpkg -i quarto-1.3.433-linux-arm64.deb

I’ll leave the old article down below for reference.

Quarto is Posit’s new open-source scientific and technical publishing system built on Pandoc that allows you to weave together narrative text and code to produce high-quality outputs including reports, presentations, websites, and more. It’s the successor of the highly successful, but R specific, R Markdown ecosystem, bringing all the benefits of R Markdown to other programming langues like Python, Julia and Observable.

Unfortunately, because of the lack of official arm64 binary sources for some of its dependencies (e.g. Deno, SASS, Pandoc), and as explained to me, their limited bandwidth to support not sufficiently popular architectures like Linux arm64. Quarto does not officially support the arm64 architecture, but that doesn’t mean we can’t make it work on it ourselves.

With the kind help of Carlos Scheidegger (@scheidegger@mastodon.social), a software engineer at Posit working on Quarto, I have managed to hack my way into installing Quarto on arm64 systems, and I’ll like to share how I did it with the R community.

Testing Platform

I have tried this on Raspberry Pi 3B+ and 4B running Raspberry Pi OS and Ubuntu 22.04 LT, and on an Oracle Ampere Altra VM running Ubuntu 22.04 LTS, so I’m pretty sure this should work on any arm64 system, provided that you can get suitable arm64 versions of Quarto’s dependencies on your specific Linux distribution. In all cases, I started with a clean and up-to-date install of the operating system so make sure you update yours as well before you start.

sudo apt update
sudo apt dist-upgrade

Installing Dependencies

First, we need a sufficiently modern arm64 version of Pandoc, and since the version that comes with the standard deb repositories for your OS is most likely too old, it is better if we download a modern version ourselves (this is very important since Quarto will fail with older Pandoc versions).

wget https://github.com/jgm/pandoc/releases/download/2.19.2/pandoc-2.19.2-1-arm64.deb
sudo dpkg -i pandoc-2.19.2-1-arm64.deb
rm pandoc-2.19.2-1-arm64.deb

Also, we need R and the rmarkdown R package, but since you are reading this article on an R related blog, I will assume you already have R installed in your system and you know how to install R packages, if you don’t, you can check this other post where I detail options to install R on a Pi. To clarify, I’m considering R as a dependency because I’m writing on an R related blog, but if you are going to use Quarto with some other programming language, R is not mandatory.

Installing Quarto

Since there are no arm64 binaries for Quarto, we are going to install it from source. Follow these steps from top to bottom without skipping any since they all depend on the previous one:

First, we need to clone the GitHub repository in a suitable location. Note that you can optionally clone a specific release version if you prefer by using the --branch option (e.g. --branch v1.3.47)

sudo su -

cd /usr/local/src

git clone --depth 1 https://github.com/quarto-dev/quarto-cli

cd quarto-cli

Next, we are going to trick Quarto into installing an arm64 version of Deno from an alternative source by modifying its configure.sh file before executing it.

sed -i 's/https:\/\/github.com\/denoland\/deno\/releases\/download/https:\/\/github.com\/LukeChannings\/deno-arm64\/releases\/download/' configure.sh
sed -i 's/DENOFILES=deno-x86_64-unknown-linux-gnu.zip/DENOFILES=deno-linux-arm64.zip/' package/scripts/common/utils.sh
sed -i 's/DENO_DIR=deno-x86_64-unknown-linux-gnu/DENO_DIR=deno-linux-arm64/' package/scripts/common/utils.sh

./configure.sh

Finally, we are going to replace the external amd64 binaries Quarto downloads during installation with arm64 ones.

mkdir package/dist/bin/tools/deno-x86_64-unknown-linux-gnu/
ln -s /usr/local/src/quarto-cli/package/dist/bin/tools/deno-linux-arm64/deno package/dist/bin/tools/deno-x86_64-unknown-linux-gnu/deno

rm package/dist/bin/tools/pandoc
ln -s /usr/bin/pandoc package/dist/bin/tools/pandoc

rm -rf package/dist/bin/tools/dart-sass/
wget https://github.com/sass/dart-sass/releases/download/1.57.1/dart-sass-1.57.1-linux-arm64.tar.gz
tar zxvf dart-sass-1.57.1-linux-arm64.tar.gz
rm dart-sass-1.57.1-linux-arm64.tar.gz
mv -f dart-sass/ package/dist/bin/tools/

If all went well, you now have a working installation of the Quarto Command Line Interface in your system, and you can start rendering qmd files from the terminal. To test if it actually works, let’s render a test document; create a test.qmd file with the following content:

---
title: "Test Document"
format: html
---

## Hello

```{r}
print("world")
```

And render the document from a system terminal with the following command:

quarto render test.qmd

If all goes well, you will get a test.html file in the current location.

I haven’t figured out how to enable the RStudio IDE integration with Quarto (RStudio’s experimental builds for arm64 come with Quarto integration features disabled). I have raised the question on the GitHub repository, and I’ll update this post if a solution or workaround arises.

Setting up your own Shiny and RStudio Server on a Raspberry Pi

Mon, 05 Sep 2022 00:00:00 +0000

This article was originally published back in 2018 and a lot has changed since then, so I have completely re-write it to take into account the most recent advancements in support for the arm64 architecture within the R ecosystem.

Back in 2018, I participated in a forum topic at Posit Community where @jladata was asking if a Raspberry Pi 3B+ could make it as a viable Shiny server. I actually use a Raspberry Pi for that very purpose so I decided to share with the R community, my experience setting up Shiny and Rstudio Server natively on a Raspberry Pi.

In those days there was almost zero support for ARM architectures (either x32 or x64) so most of the software had to be compiled from source on the Pi itself, which makes the installation process a titanic effort with lots and lots of waiting hours.

Nowadays, things have changed and we have better support for the arm64 architecture due to community efforts and even official RStudio support in the works. In this renewed article I’m going to guide you through the installation options we have currently available.

✏️ I also have a related article showing how to automate the installation process using Ansible.

Prepare The Raspberry Pi

First, you need a 64-bit operating system for your Pi (make sure it’s 64-bit since RStudio only has support for the arm64 architecture), these installation instructions work for both Raspberry Pi OS and Ubuntu, you can download either of those using the Raspberry Pi Imager software from their official site. Also, make sure to update your system libraries with sudo apt update && sudo apt dist-upgrade before you start.

Although not mandatory, I recommend performing the following actions using the sudo raspi-config application (In Ubuntu you need to install it first with sudo apt install raspi-config):

Expand the filesystem (Advanced Options / Expand Filesystem).
Configure your locale, and make sure to choose an UTF.8 option to avoid compilation issues (Localisation Options / Locale).
Disable “Predictable network interfaces” (Advanced Options / Network Interface Names).

Next, you need to set a static IP for the Pi so you can easily find it on your network.

For setting a static IP on Raspberry Pi OS, run these commands. This is a sample IP configuration for a wired connection, modify accordingly with your own needs:

# Sample static IP configuration:
sudo nmcli c mod 'Wired connection 1' ipv4.addresses 192.168.1.101/24 ipv4.method manual
sudo nmcli con mod 'Wired connection 1' ipv4.gateway 192.168.1.1
sudo nmcli con mod 'Wired connection 1' ipv4.dns "192.168.1.1"
sudo nmcli c down 'Wired connection 1' && sudo nmcli c up 'Wired connection 1'

And for setting the IP on Ubuntu you have to create this file, sudo nano /etc/netplan/01-netcfg.yaml,add this content (edit as needed) and apply the configuration with this command, sudo netplan apply.

network:
    version: 2
    renderer: networkd
    ethernets:
        eth0:
            dhcp4: false
            addresses:
                - 192.168.1.101/24
            nameservers:
                addresses: [8.8.8.8, 8.8.4.4, 192.168.1.1]
            routes:
                - to: default
                  via: 192.168.1.1

In case you have a low-RAM Raspberry Pi model, you have to add some swap memory for the installation process. For 1GB Pi models, 3GB of swap would be enough, you can do it with these commands:

sudo /bin/dd if=/dev/zero of=/var/swap.1 bs=1M count=3072
sudo chmod 600 /var/swap.1
sudo /sbin/mkswap /var/swap.1
sudo /sbin/swapon /var/swap.1
sudo sh -c 'echo "/var/swap.1 swap swap defaults 0 0 " >> /etc/fstab'

For preventing unnecessary use of swap memory and protect your SD card, open this file, sudo nano /etc/sysctl.conf, and add this line at the end:

vm.swappiness=10

Install R

You have several options for installing R on your Pi:

Install from the Operating System’s repositories (Not recommended)
Compile R from source
Install R using rig
Install R from the R4Pi Project

Since there are several options now I have moved this section to a separate article that you can consult here for more details.

Whenever possible (whit a compatible OS), I strongly recommend using the R4Pi package repository as your R packages source since it provides precompiled binaries that make installation easier and faster.

Install Shiny Server

Currently, shiny-server offers no ARM support so the only option for installing it on the Pi is to compile from source. There is hope this will change in the future as per the response on this GitHub Issue, but for the time being we are stuck with this.

First, you need to install the R package dependencies, if you are using a CRAN repository, this is going to take a while, but it is much quicker if you are using the R4Pi repository (available with R installed from R4Pi):

# Define your R package repository (You have to choose one of these)
# REPO="'http://cran.rstudio.com/'" # For OSes that are not compatible with R binaries build for RPi OS (Debian 12 based)
# REPO="c('https://pkgs.r4pi.org/noble', 'http://cran.rstudio.com/')" # For Ubuntu 24.04
REPO="c('https://pkgs.r4pi.org/bookworm', 'http://cran.rstudio.com/')" # For the latest RPi OS (Debian 12 based)

# Make sure the system dependencies are installed
sudo apt install libcairo2-dev libxt-dev git cmake pandoc

# Install required R packages as sudo
sudo su - -c "R -e \"install.packages('Rcpp', repos=$REPO)\""
sudo su - -c "R -e \"install.packages('later', repos=$REPO)\""
sudo su - -c "R -e \"install.packages('fs', repos=$REPO)\""
sudo su - -c "R -e \"install.packages('R6', repos=$REPO)\""
sudo su - -c "R -e \"install.packages('Cairo', repos=$REPO)\""
sudo su - -c "R -e \"install.packages('httpuv', repos=$REPO)\""
sudo su - -c "R -e \"install.packages('mime', repos=$REPO)\""
sudo su - -c "R -e \"install.packages('jsonlite', repos=$REPO)\""
sudo su - -c "R -e \"install.packages('digest', repos=$REPO)\""
sudo su - -c "R -e \"install.packages('htmltools', repos=$REPO)\""
sudo su - -c "R -e \"install.packages('xtable', repos=$REPO)\""
sudo su - -c "R -e \"install.packages('sourcetools', repos=$REPO)\""
sudo su - -c "R -e \"install.packages('shiny', repos=$REPO)\""
sudo su - -c "R -e \"install.packages('rmarkdown', repos=$REPO)\""

✏️ It’s necessary to install R packages as sudo for shiny-server to be able to use them, do not install these packages from the R console using your regular user.

Now, you can install shiny-server with these commands:

# Download the source code for the latest shiny-server release from GitHub
git clone --depth 1 --branch v1.5.23.1030 https://github.com/rstudio/shiny-server.git

# Compile the source code
cd shiny-server
DIR=`pwd`
PATH=$DIR/bin:$PATH
mkdir tmp
cd tmp
PYTHON=`which python`
sudo cmake -DCMAKE_INSTALL_PREFIX=/usr/local -DPYTHON="$PYTHON" ../
sudo make
mkdir ../build

# Install node
(cd .. && sudo ./external/node/install-node.sh)
(cd .. && ./bin/npm ci --omit-dev --python="${PYTHON}") 

# Install shiny-server
sudo make install

# Configure shiny-server
sudo mkdir -p /etc/shiny-server
sudo cp ../config/default.config /etc/shiny-server/shiny-server.conf
cd
sudo rm -rf shiny-server
sudo ln -s /usr/local/shiny-server/bin/shiny-server /usr/bin/shiny-server
sudo useradd -r -m shiny
sudo mkdir -p /var/log/shiny-server
sudo mkdir -p /srv/shiny-server
sudo mkdir -p /var/lib/shiny-server
sudo chown shiny /var/log/shiny-server

# Edit the shiny-server.service file
sudo nano /etc/systemd/system/shiny-server.service 
    # Paste the following content
    
    [Unit]
    Description=ShinyServer

    [Service]
    Type=simple
    ExecStart=/usr/bin/env bash -c 'exec /usr/local/shiny-server/bin/shiny-server >>     /var/log/shiny-server.log 2>&1'
    KillMode=process
    ExecReload=/usr/bin/env kill -HUP $MAINPID
    ExecStopPost=/usr/bin/env sleep 5
    Restart=on-failure
    RestartSec=1
    StartLimitInterval=45
    StartLimitBurst=3

    [Install]
    WantedBy=multi-user.target

# Edit the shiny-server logrotate file
sudo nano /etc/logrotate.d/shiny-server 
    # Paste the following content
    
    /var/log/shiny-server.log {
       rotate 12
       copytruncate
       compress
       missingok
       size 1M
    }

# Set permissions for the files
sudo chown root:root /etc/systemd/system/shiny-server.service
sudo chmod 644 /etc/systemd/system/shiny-server.service
sudo chown root:root /etc/logrotate.d/shiny-server
sudo chmod 644 /etc/logrotate.d/shiny-server

# Start the service
sudo systemctl daemon-reload
sudo systemctl enable shiny-server
sudo systemctl start shiny-server

# Create symlinks for pandoc and the sample apps
sudo ln -s -f /usr/bin/pandoc /usr/local/shiny-server/ext/pandoc/pandoc
sudo ln -s -f /usr/bin/pandoc-citeproc-preamble /usr/local/shiny-server/ext/pandoc/pandoc-citeproc
sudo ln -s /usr/local/shiny-server/samples/sample-apps /srv/shiny-server/sample-apps
sudo ln -s /usr/local/shiny-server/samples/welcome.html /srv/shiny-server/index.html

# Set proper user permissions, I'm assuming your user is "pi", change it if it isn't
sudo groupadd shiny-apps
sudo usermod -aG shiny-apps pi
sudo usermod -aG shiny-apps shiny
cd /srv/shiny-server
sudo chown -R pi:shiny-apps .
sudo chmod g+w .
sudo chmod g+s .
cd

Install Rstudio Server

In the past, the only way to install RStudio Server on a Pi was to compile from source and this was extremely time-consuming (took me 3 days the last time I tried on a Pi 3B+) and very prone to failure because the RStudio IDE was under active development.

Luckily, since then, RStudio has made available “experimental” builds of the IDE for arm64 (I wrote an article about it), they are compiled for Ubuntu 22, 24 LTS, RedHat 9 and 10. The Ubuntu 22 and 24 binaries are compatible with Raspberry Pi OS as well. You can check the available daily builds here. Choose a link accordingly to the operating system you are using, also, have in mind that these builds are meant for testing so they are likely to have bugs or be unstable, if you have problems installing the latest one available, try with an older version.

The following instructions use a link for Ubuntu 24 (Raspberry Pi OS compatible), remember to change the link if you want to use another OS or a more recent IDE version:

sudo apt install gdebi
wget https://s3.amazonaws.com/rstudio-ide-build/server/jammy/arm64/rstudio-server-2026.01.0-392-arm64.deb
sudo gdebi rstudio-server-2026.01.0-392-arm64.deb
sudo rm rstudio-server-2026.01.0-392-arm64.deb

✏️ I’ll update this section once the arm64 support becomes a stable feature for the RStudio IDE.

Install Additional Support Services (optional)

Nginx is a low-requirements web server, which comes in handy to self-host HTML files but also to use as a reverse proxy (more about this in the next section). You can install it with this command:

sudo apt install nginx

Also, for working with “bigger than memory” data sets (have in mind that the Pi doesn’t have much RAM), having an RDBMS comes in handy. I personally prefer PostgreSQL since it’s open-source and has impressive capabilities, you can install PostgreSQL with this command:

sudo apt install postgresql libpq-dev postgresql-client postgresql-client-common

Extra Steps (optional)

Make pretty URLs for Shiny and Rstudio Server (e.g. http://your-ip/shiny/your-app instead of http://your-ip:3838/your-app) by using Nginx as a reverse proxy, you only need to replace the content of this file sudo nano /etc/nginx/sites-enabled/default, with this:

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 80 default_server;
    listen [::]:80 default_server;
    
    root /var/www/html;
    
    index index.html index.htm;
    
    server_name _;
    
    location / {
        try_files $uri $uri/ =404;
    }
    
    rewrite ^/shiny$ $scheme://$http_host/shiny/ permanent;
    
    location /shiny/ {
        rewrite ^/shiny/(.*)$ /$1 break;
        proxy_set_header    Host $host;
        proxy_set_header    X-Real-IP $remote_addr;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    X-Forwarded-Proto $scheme;
        proxy_pass          http://localhost:3838;
        proxy_read_timeout  20d;
        proxy_buffering off;
        
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_http_version 1.1;
        
        proxy_redirect      / $scheme://$host/shiny/;
    }
    
    location /rstudio/ {
        rewrite ^/rstudio/(.*)$ /$1 break;
    
        proxy_pass http://localhost:8787;
    
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_read_timeout 20d;
        proxy_buffering off;
    
        proxy_set_header X-RStudio-Root-Path /rstudio;
    
        proxy_set_header Host $host:$server_port;
    }
}

On Raspberry Pi OS there is a weird issue where Nginx starts before RStudio Server is able to start, and the reverse proxy fails. A walk-around for this is to edit this file, sudo nano /etc/systemd/system/multi-user.target.wants/nginx.service, and replace the part that says After=network.target with After=network-online.target and reload the daemon with sudo systemctl daemon-reload. You can omit this if you are using Ubuntu.

Now you have to restart Nginx to apply the changes:

sudo systemctl restart nginx

If you want to have access to your server from remote origins (over the internet) you must follow these extra steps:

If you don’t have a static public IP:
- Open an account in a Dynamic DNS Service (e.g. https://www.noip.com)
- Configure Dynamic DNS in your router/modem, if you don’t have that option, you can install a DDNS client on your server.
- Optionally configure your personal domain name to point to your DDNS service
If you have a public IPv4, configure port forwarding in your router/modem:
- For SSH, SCP and SFPT open port TCP/UDP 22
- If you haven’t configured Nginx, open port 8787 for RStudio.
- If you haven’t configured Nginx, open port 3838 for Shiny server.
- For HTTP open port TCP 80 (also for using Nginx as a reverse proxy).
- For PostgreSQL open port TCP/UDP 5432
- Make sure you have “Block WAN traffic” disabled on your router/modem.

If you are interested, I have an article that provides an example on how to configure port forwarding at home.

Final Comments

Arm64 support in general is increasing rapidly, so very likely things are going to change in the near future. I’m going to try my best to keep this article up to date with the latest developments so you can leverage the benefits.

Performance is relatively poor, especially for plotting, but usable. Keep in mind that even though the Raspberry Pi has 4 cores they are not fast ones and most of the R packages don’t use parallel computing.

Take the RStudio IDE experimental support for arm64 architectures out for a spin

Mon, 22 Aug 2022 00:00:00 +0000

Apple’s bet on the arm architecture with their M1 and M2 processors has had an interesting side effect, developers of mainstream software are now forced to support the arm64 architecture and, since they are already supporting macOS for Apple Silicon processors anyways, supporting Linux arm64 distributions is no longer that much of a burden for them. The RStudio IDE development team is not strange to this circumstance and they have been working on general arm64 support, currently, they have made available experimental builds for the arm64 architecture (also known as aarch64) compiled for Ubuntu 20/22 LTS (the Ubuntu 22 build being compatible with the latest Raspberry Pi OS x64 based on Bookworm) and RedHat 9. I have decided to give them a try on my Raspberry Pi SBC, if you are interested, join me in the installation process.

Basic Set Up and Dependencies

The first thing you need is a Raspberry Pi (3/4/5) with the lastest Raspberry Pi x64 OS (Debian 12 based) or Ubuntu 20/22 LTS x64 installed and updated (or any other Linux arm64 system running a supported OS), make sure to update your system libraries before you start:

sudo apt update
sudo apt dist-upgrade

Then, the most important dependency for RStudio is, unsurprisingly, the R language. You have some options for installing R on arm systems, each one with its own pros and cons:

Install from the Operating System’s repositories
Compile R from source
Install R using rig
Install R from the R4Pi Project

Since there are several options now I have moved this section to a separate article that you can consult here for more details.

In all cases, I strongly recommend using the R4Pi package repository as your R packages source since it provides precompiled binaries that make installation easier and faster.

The latest experimental RStudio IDE builds (>= 2023.08.0 builds) for arm64 now come with Quarto and Pandoc bundled so there is no longer the need to install them separately and Quarto integration features are now fully supported on arm64 systems, although, if you want a newer version of any of these software you can download the corresponding .deb file and install it using the dpkg command.

You can download the latest Pandoc installer from here: https://github.com/jgm/pandoc/releases/
And the latest Quarto installer from here: https://github.com/quarto-dev/quarto-cli/releases

And the installation command would be something like this:

sudo dpkg -i <<file_name.deb>>

Lastly, for installing the RStudio IDE deb file, as you would do on an x86_64 system, you might want to install gdebi to handle package dependencies for you:

sudo apt install gdebi

Installing the RStudio IDE

Now we get to the point. There is no stable release of the RStudio IDE for arm64 but there are experimental daily builds available here, both for the server and desktop versions and compiled for Ubuntu 20/22 LTS and RedHat9. I’m going to try out the server version for Ubuntu 22 because it is compatible with Raspberry Pi OS x64 and I prefer to work with the IDE on my regular computer instead of directly on the Raspberry Pi, but you can install the desktop version the same way if you prefer, just change the link accordingly.

wget https://s3.amazonaws.com/rstudio-ide-build/server/jammy/arm64/rstudio-server-2023.12.0-daily-232-arm64.deb
sudo gdebi rstudio-server-2023.12.0-daily-232-arm64.deb

Once the installation process is completed, you get a working RStudio IDE installation running on your Raspberry Pi, if you have chosen to install the server version as I did, you can open an RStudio session by opening a browser on any device in your network and typing the proper address i.e. http:\\your_pi_ip_addres:8787

Final Comments

Have in mind that these are experimental builds of the development version of the RStudio IDE and they are likely to be unstable and buggy so do not trust this for any sort of critical job. You can follow along with the progress on this GitHub issue.

Performance on the Raspberry Pi is acceptable, within reason for under-powered systems like these SBCs, but it is pretty far from the performance you would get on more powerful systems like Apple Silicon or full-size arm64 computers if you have access to one of those (tested on an Ampere VM and works great). If you want to get the most performance out of your Pi, use Raspberry Pi OS Lite since it’s the lightest option.

I hope you found this article useful, I will be updating the content as things develop.

How to make your home Shiny or Rstudio Server accessible from the public internet

Mon, 16 Aug 2021 00:00:00 +0000

⚠️ Some assembly required! This project is going to require you to investigate the specifics of your network equipment on your own.

You have installed Shiny and/or RStudio server in your home computer and it works great as long as you are connected to your local network, however, you can’t access it from other locations, What can you do?

The solution is to expose your server to the public internet by forwarding incoming traffic to the corresponding port in your server. Now, I’m not going to lie to you telling you that this article is going to give you detailed instructions on how to do it with your specific router because it would be virtually impossible, each router brand (an even model) has a different management interface, I can only give you the required steps along with an example using the router I have at home, although, generally speaking, the procedure involves the same steps regardless of the specific router model you have, so with a little effort on your part you should be able to figure out how to make it work with your own router.

Prerequisites

I’m writing this article under the assumption that you already have Shiny and/or RStudio server properly installed on a computer at home (if you haven’t, you can check out the other articles in my blog for inspiration).
You need to have a public IP assigned to your router (it doesn’t matter if it changes dynamically), not everybody has one since some Internet Service Providers (ISPs) use a NAT3 configuration, which basically means that you share a public IP with a group of their clients and there will be no means to map an HTTP request to your home server, however, it is possible that you can get your NAT configuration changed if you call your ISP and request for a public IP so you might want to give it a try.
You need to know the user and password to access your router’s configuration interface, some ISPs let it as default and you only need to google the default password for your router’s model, but in most cases, they change it so you would need to give them a call and ask for your credentials.

Open the Required Ports in the Server

Specific ports for Shiny and/or RStudio server need to be accessible in your server to be able to connect to those services. Most Linux distributions come with all ports accessible by default but it is a common (and good) practice to install a firewall to restrict access to your server, if this is your case, you need to make sure that you have the required ports open.

For Shiny server the default port is 3838 and for RStudio server it is 8787 (this can be customized). If you have configured a reverse proxy with Nginx or Apache you can obviate those ports and only open port 80 for HTTP or port 443 for HTTPS. For example, the command to allow incoming traffic on port 80 using the ufw firewall in Ubuntu-based systems would be like this:

sudo ufw allow 80/tcp

And using firewalld in RHEL-based systems would be:

sudo firewall-cmd --permanent --zone=public --add-port=80/tcp
sudo firewall-cmd --reload

You could have other rules that could prevent traffic from going into your server from remote locations so you might want to inspect your firewall rules and look for anything restricting incoming traffic.

Configure Port Forwarding in your Router

Open a web browser on a device that is connected to your local network and type your router’s local IP, usually, it is something like 192.168.0.1 or 192.168.1.1. All routers have a web server running on port 80 by default so by typing their IP it would open a web page with the configuration interface.

In my particular case, I’m steel using a cheap Sagemcom HFC modem my ISP gave me (because I’m always going to set up my custom pfSense router very soon), so if I open a Chrome browser and type 192.168.0.1 this is what I get.

In your case, the user interface would look different and options might have slightly different names but they are basically the same.

Enter your credentials to get access to the configuration interface, which in my case looks like this.

Then, search among all the available tabs to find the “port forwarding” section, it is not going to be in the same place for you but most modern routers have this section.

Once in there add forwarding rules for Shiny and/or Rstudio server (You have to replace “your_sever_ip” with your actual server’s local IP) and apply changes.

Local Address	Start Port	End Port	External Address	Protocol	Description
your_sever_ip	3838	3838	0.0.0.0	TCP	Shiny-Server
your_sever_ip	8787	8787	0.0.0.0	TCP	RStudio-Server

If you are using a reverse proxy, you only need to make sure you have one of these.

Local Address	Start Port	End Port	External Address	Protocol	Description
your_sever_ip	80	80	0.0.0.0	TCP	HTTP
your_sever_ip	443	443	0.0.0.0	TCP	HTTPS

Also, make sure you don’t have a “Block WAN traffic” option enabled in your router, this will block all incoming traffic whatsoever.

After doing all these you should be able to access your shiny apps by typing your.public.ip:3838/your_app_name and/or your Rstudio Server by typing your.public.ip:8787 in any browser.

Optional Extra Steps

Setup a Dynamic DNS Service (DDNS)

Most residential internet services do not include a “static” public IP, which means that even if you have a public IP assigned at any given moment, it will not always be the same. A public IP is needed to access your server from the internet directly or to point your custom domain (web address) to it, so you can imagine how inconvenient would be having to figure out what the value is at any given moment.

To address this issue, you can use a Dynamic DNS Service (I like to use No-IP). What this service does, is to automatically update your domain (they can even give you a free subdomain if you want) to point towards your current public IP every time it changes, it does it by using a built-in DDNS agent in your router if available or if not, by using a service installed in your server that automatically reports the current IP to the DDNS service.

If your router has a compatible built-in DDNS agent, you just have to enter your credentials and you are ready to go.

If your router does not have a DDNS agent, you are going to have to install one in your server (provided by the DDNS provider). How to install it depends on what DDNS provider you are using and what operating system your server is running but surely you can find detailed instructions on the DDNS provider’s documentation page.

With this extra step, you will access your shiny apps with your_domain:3838/your_app_name and your RStudio server with your_domain:8787 which doesn’t require you to know your current public IP in advance.

Configure a Reverse Proxy

Up until the previous step we had solved the problem with the changing IP address but we still have awkward port numbers in the web addresses for our server, by configuring a reverse proxy we can get to use nicer addresses like your_domain/shiny/your_app_name and your_domain/rstudio.

To configure a reverse proxy, install Nginx

sudo apt install nginx

Edit Nginx config file: sudo nano /etc/nginx/sites-enabled/default, add the following lines before the line that reads server {

map $http_upgrade $connection_upgrade {
      default upgrade;
      ''      close;
    }

and the following lines right after the line that reads server_name _;

    rewrite ^/shiny$ $scheme://$http_host/shiny/ permanent;
    # This part is for Shiny server
    location /shiny/ {
        rewrite ^/shiny/(.*)$ /$1 break;
        proxy_pass http://localhost:3838;
        proxy_redirect / $scheme://$http_host/shiny/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_read_timeout 20d;
        proxy_buffering off;
    }
    # This part is for RStudio server
    location /rstudio {
       # Needed only for prefix of /rstudio
       rewrite ^/rstudio/(.*)$ /$1 break;

       # Use http here when ssl-enabled=0 is set in rserver.conf
       proxy_pass http://localhost:8787;
       proxy_http_version 1.1;

       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection $connection_upgrade;

       # Optionally, use an explicit hostname and omit the port if using 80/443
       proxy_set_header Host $host:$server_port;

       # Not needed if there's no prefix or www-root-path is set in rserver.conf
       proxy_set_header X-RStudio-Root-Path /rstudio;

       proxy_read_timeout 20d;
    }

Now you just have to restart Nginx to apply the changes and you are done.

sudo service nginx restart

Automatically installing Shiny and RStudio server on Amazon Web Services with Ansible

Sun, 28 Mar 2021 00:00:00 +0000

After writing a very similar post about installing Shiny and RStudio server on a Raspberry Pi using automation with Ansible, I realized that the same approach can be easily ported to work with cloud computing services as well, so, basically, this is going to be the Amazon Web Services (AWS from now on) version of that post.

✏️ Have in mind that this is a beginner-oriented project, it is not meant as a production tool, so if you are already familiar with Ansible you might want to simply download some roles from Ansible Galaxy and integrate them into your workflow instead of following this article.

In case you haven’t read the aforementioned post, I’m going to tell you what this is about. The installation and configuration process of an R-based data science server on AWS can be automated using a tool called Ansible, those in DevOps must be familiar with the power of Ansible but even if you’ve never heard of it, don’t worry, you don’t need to know Ansible in order to take advantage of its automation capabilities, I’m not an Ansible expert myself and I’ve been capable of accomplishing this project with just basic knowledge of the tool, so if I can do it, so can you. I’m going to guide you through the process assuming you have no prior experience with this tool and just some basic Linux terminal skills.

I have wrote some Ansible playbooks to automate most of the installation process but leaving some room for you to configure the installation to your liking without having to modify the playbooks (which will require you to know how to edit them) via a config file. Although, if you want to have a basic understanding of how this tool works, there is a nice Ansible 101 series on Jeff Geerling’s YouTube channel.

In case you are not familiar with AWS, it is a cloud services provider that among many other things, it provides access to virtual servers hosted on the cloud, so you don’t have to take care of a physical infrastructure yourself, also, to follow along with this article, there are some AWS specific terms that we need to define in simple terms:

EC2 Instance: It refers to an individual instance of a virtual server.
Amazon Machine Images (AMI): It provides the information required to launch an instance, like the Operating System, pre-installed libraries and configurations required for a server to work. Note that you can make your own AMIs.
Regions: Amazon EC2 is hosted in multiple locations world-wide. These locations are composed of Regions, and each Region is a separate geographic area.
Security Group: A security group acts as a virtual firewall for your instance to control inbound and outbound traffic.

Create an AWS Account and Security Credentials

In order to use AWS services you need to have an account, if you already have one, you can skip this part, if not, to create one go to https://aws.amazon.com/ and click on “Create an AWS Account”, fill in the required information and keep pressing “Continue”, you are going to be asked to provide your credit card information, but you are not going to be charged for using the products comprised on the “Free Tier” at least for a year.

Once your account has been created, go to the “AWS Management Console” and log in, once there, click on the search button, type “EC2” and select “EC2 Virtual Servers in the Cloud”.

On the upper right corner choose a suitable region, the geographically closer to you the better. Remember your choice since you have to set this exact same location in the playbooks later.

Then in the left menu, look for “Network & Security -> Key Pairs” and click on it.

On the right upper corner look for “Create Key Pair” and click on it.

You are going to see this dialog box, fill it this way, click on “Create Key Pair” and save the resulting file on this path, ~/.ssh/aws_server.pem, in your machine.

Set propper permissions for your key file, the .ssh directory permissions should be 700 (rwx______) and the private key (aws_server.pem) should be 600 (rw_______).

sudo chmod 700 ~/.ssh/
sudo chmod 600 ~/.ssh/aws_server.pem

Next, you need to create an access key, so you can have access to your account programmatically, go to your account menu and select “My Security Credentials”.

Then select “Create New Access Key” and save the file with your credentials

⚠️ Keep the downloaded file safe and at hand, you are going to need it later. Be careful not to share this file since it could enable other people to generate charges to your credit card.

Install Ansible

It is possible to install Ansible locally on the EC2 instance itself and run playbooks directly on it with connection: local but it wouldn’t be practical, because among other things it would require us to manually launch the EC2 instance, it is better to run the playbooks from a system other than your EC2 instance.

For installing Ansible, if you have a Unix based system (i.e. Linux, macOS) at your disposal, you are golden, installing Ansible is very simple.

# Install python3
sudo apt install python3 # On macOS use `brew install python3`
# Install pip3
sudo apt install python3-pip 
# Install Ansible version 9.5.1 (for reproducibility) with pip3 
sudo pip3 install ansible==9.5.1 --break-system-packages

If you are on Windows, sadly, there is no way to run Ansible natively, your best bet is to enable WSL (Windows Subsystem for Linux), install a Linux distribution from the Microsoft Store (I recommend Ubuntu), and execute the previous steps in your Linux VM.

A detailed guide for installing WSL can be found here and more information about installing Ansible can be found on the official documentation site.

Also, since we are going to be testing this with a low power EC2 instance (i.e. T2.Micro), some of these tasks are going to take a while and the SSH connection might get automatically closed due to inactivity, to prevent this situation, activate the sending of “keep-alive” packets to the server by editing the ssh_config file with sudo nano /etc/ssh/ssh_config and adding this two lines under Host *:

    ServerAliveInterval 300
    ServerAliveCountMax 2

⚠️ Is very important not to skip the previous step, otherwise, the ssh connection is going to silently fail while running long tasks in Ansible and you are going to be waiting pointlessly with your EC2 instance doing nothing in reality.

Download and Configure the Playbooks

The playbooks are on a public repository on GitHub, you can clone the repository with these commands:

# Install git if you don't have it already
sudo apt install git # On macOS use `brew install git`
# Clone the latest commit from the repository
git clone https://github.com/andresrcs/aws_r_server.git --depth 1

To start configuring the playbooks, you usually first need to define an “inventory” (a list of servers to connect to), but in this case, the provision_ec2_instance.yml playbook is going to create one automatically for you when you run it and the other playbooks are going to get an updated inventory from AWS automatically using the aws_ec2 plugging when needed, this is one of the advantages of working with well supported cloud computing services like AWS, but in order for the aws_ec2 plugging to do its magic, you need to provide it with your AWS credentials and define your AWS EC2 region, so edit the aws_r_server/inventories/aws_ec2.yml file, and fill your credentials and region on the respective variables. Remember that you already created these credentials on a previous step and you saved a .csv file containing them, also the AWS Region you set here has to be the same one you selected when you started creating your credentials.

---
plugin: aws_ec2

aws_access_key: 'your_key_goes_here'
aws_secret_key: 'your_secret_goes_here'

regions:
  - 'closest_aws_region' #  e.g. sa-east-1

hostnames:
  - 'ip-address' # Do not change this

keyed_groups:
  - key: tags.inventory_group

Now, if you want to change the default installation settings, you can do it by editing the variables on the aws_r_server/vars/config_vars.yml file, although, the default options are fine for most use case scenarios, including reasonable security settings to use on real-world applications. The only things you certainly need to change here are the AWS credentials, AWS region, AWS AMI (it changes constantly even for the same OS version) the personal user and password, and the password for the PostgreSQL main user.

✏️ This post has been written as an example of applying this approach with cloud computing services so for simplicity’s sake, I have made it work only with the Ubuntu Server 22.04 LTS AMI, so to be clear, it is not OS-agnostic. If you want to use it with an RHEL based AMI you would need to modify the playbooks considerably.

---
# AWS Config Vars ##############################################################

# AWS credentials
aws_access_key: 'your_key_goes_here'
aws_secret_key: 'your_secret_goes_here'

# General AWS configurations
aws_region: 'closest_aws_region' #  e.g. sa-east-1
aws_ec2_ami: 'ami-08ae71fd7f1449df1' # Ubuntu 22.04 LTS

# List of instances
instances:
  - name: rstudio
    group: aws
    security_group: ["default", "public_server"]
    ssh_key: aws_server
    type: 't2.micro'

# AWS firewall settings
security_groups:
  - name: public_server
    rules:
      - proto: tcp # http
        from_port: 80
        to_port: 80
        cidr_ip: 0.0.0.0/0
      - proto: tcp  # https
        from_port: 443
        to_port: 443
        cidr_ip: 0.0.0.0/0
      - proto: tcp  # SSH TCP
        from_port: 22
        to_port: 22
        cidr_ip: 0.0.0.0/0
      - proto: udp  # SSH UDP
        from_port: 22
        to_port: 22
        cidr_ip: 0.0.0.0/0
      - proto: tcp  # PostgreSQL
        from_port: 5432
        to_port: 5432
        cidr_ip: 0.0.0.0/0
    rules_egress: []

# System Configurations ########################################################

# Personal user
personal_user: 'your_linux_user'
personal_user_password: 'very_secure_password'

# Swap parameters
swap_file_path: /var/swap
# Use any of the following suffixes
# c=1
# w=2
# b=512
# kB=1000
# K=1024
# MB=1000*1000
# M=1024*1024
# xM=M
# GB=1000*1000*1000
# G=1024*1024*1024
swap_file_size: 3GB
swappiness: '10'

# Language and locale
language_pack: []                 #  e.g. language-pack-es-base
default_locale: 'en_US.UTF-8'     #  e.g. es_PE.UTF-8

# Security Configurations ######################################################

# Postgresql Password
postgres_password: 'very_secure_password'

# Access rules for Postgresql
postgresql_rules:
  - { contype: local, users: all, address: samehost, method: trust }
  - { contype: local, users: postgres, address: samehost, method: trust }
  - { contype: host, users: all, address: 0.0.0.0/0, method: password }
  - { contype: host, users: all, address: 127.0.0.1/32, method: md5 }
  - { contype: host, users: all, address: '::1/128', method: trust }

# Main Software Versions #######################################################

# Shiny-server version to install
shiny_server_version: '1.5.19.995'

# RStudio version to install
rstudio_version: '2022.07.1+554'
preview_version: false

Run the Playbooks

You could simply access the aws_r_server folder and run the main.yml playbook to install everything at once like this:

cd aws_r_server
ansible-playbook main.yml

But to make the installation process more flexible, I have divided the process into four individual playbooks:

ansible-playbook provision_ec2_instance.yml
ansible-playbook install_basic_services.yml
ansible-playbook install_shiny_server.yml
ansible-playbook install_rstudio_server.yml

⚠️ When provisioning the ec2 instance the AWS API sometimes fails to return the instance IP causing the playbook to fail. This is a random error not caused by the playbook, you can safely ignore it if you are running individual playbooks and the instance has already been provisioned.

If you decide that you only need Shiny server or RStudio server but not the other, or you already have installed the support services you are going to use, then you can run only the playbooks you actually need.

If you want to update something in the future, like RStudio, or Shiny server (I can’t guaranty this is always going to work out of the box), you can simply change the version in the config file and run the specific part of the playbook by taking advantage of the defined “tags”.

For example, this will only install the RStudio version defined in the config file and nothing more:

ansible-playbook install_rstudio_server.yml --tags "rstudio"

The available tags are:

install_basic_services.yml
- secure: Set security settings on the server
- swap: Add swap memory to the server
- nginx: Install and configure Nginx + PHP
- postgresql: Install and configure PostgreSQL
- r: Install R from the CRAN repository
install_shiny_server.yml
- shiny-server: Install shiny-server
- configure_shiny: Configure shiny-server
install_rstudio_server.yml
- rstudio: Install RStudio server
- configure_rstudio: Configure RStudio server

You can also go the other way and skip specific parts of the playbooks by using the --skip-tags option, for example, if you don’t need PostgreSQL, you can avoid installing it by running the playbook this way:

ansible-playbook install_basic_services.yml --skip-tags "postgresql"

After successfully running all the playbooks, you will have a fully functional installation ready to be used, so you could simply open an RStudio session at http://your_server_ip/rstudio/ and/or publish your Shiny apps in the /srv/shiny-server folder and access them at http://your_server_ip/shiny/your_app_name.

Final Notes

To finish I just want to make you aware that there are other options for getting RStudio and Shiny server on AWS that you might consider to be simpler, like using a premade AMI or containers, but they are not as flexible and customizable as defining your own infrastructure in code with Ansible. Obviously, the example in this article is my conception of a basic installation for an R-based data science server but you can use it as a starting point to custom tailor your own infrastructure and make your workflow more efficient as you become more proficient with Ansible.

Automatically installing Shiny and RStudio server on a Raspberry Pi with Ansible

Wed, 13 Jan 2021 00:00:00 +0000

I have a related post about manually installing Shiny and RStudio server on a Raspberry Pi but even with the latest advancements in software support for the ARM architecture, this process is still considerably time-consuming and requires a lot of copy/pasting and watching the terminal waiting for one task to be completed to be able to start with the next one, so, I decided to come up with a better solution, and that is to automate the process so it can be left unattended and come back later when it’s done.

To automate the installation and configuration process I decided to use Ansible, those in DevOps must be familiar with the power of Ansible but even if you’ve never heard of it, don’t worry, you don’t need to know Ansible in order to take advantage of its automation capabilities, I’m not an Ansible expert myself and I’ve been capable of accomplishing this project with just basic knowledge of the tool, so if I can do it, so can you. I’m going to guide you through the process assuming you have no prior experience with this tool and just some basic Linux terminal skills.

Ansible is an open-source software provisioning, configuration management, and application-deployment tool enabling infrastructure as code. It runs on many Unix-like systems, and can configure both Unix-like systems as well as Microsoft Windows. In layman’s terms, it lets you automate tasks in your infrastructure (One, or many Raspberry Pi SBCs at once in our case) by defining a set of steps to execute in YAML files called “playbooks”. I took one for the team and wrote some Ansible playbooks to automate most of the installation process but leaving some room for you to configure the installation to your liking via a config file, so you don’t need to modify the playbooks (which will require you to know how they work) . Although, if you want to have a basic understanding of how this tool works, there is a nice Ansible 101 series on Jeff Geerling’s YouTube channel and if you are the kind of people that prefer reading rather than seeing things, here is a nice Ansible Tutorial for Beginners.

✏️ Note that this has only been tested with the software versions specified in the config_vars.yml file, it is very likely that it will still work out of the box with newer software versions, but in some cases, it might require some modifications to the playbooks that I will made as soon as I notice the problem.

Prepare the Raspberry Pi

Even if most of the process can be automated with Ansible, we still need at least one working Raspberry Pi to configure, so, we need to start by flashing an Operating System (OS) onto an SD card (I recommend at least 32GB class 10 HC1). Have in mind that RStudio Server is compatible only with arm64 operating systems, so if you want to be able to install it, you will have to choose a 64-bit OS, the playbooks work with Raspberry Pi OS, and Ubuntu as well.

Download “Raspberry Pi Imager” from the official site and install it in your system, execute “Raspberry Pi Imager”, choose a suitable OS, choose the “SD Card” and press “Write”.

Once the process is done, you have a clean installation of an operating system on your micro SD card, insert the card into your Raspberry Pi, plug in a keyboard and a screen, and turn it on.

Log into the system (the default user and password are pi and raspberry for Raspberry Pi OS, and in Ubuntu is ubuntu for both user and password) and run an initial system update with these commands:

sudo apt update
sudo apt dist-upgrade

Next, you need to set a static IP for the Pi so you can easily find it on your network later on.

For setting a static IP on Raspberry Pi OS, run these commands. This is a sample IP configuration for a wired connection, modify accordingly with your own needs:

# Sample static IP configuration:
sudo nmcli c mod 'Wired connection 1' ipv4.addresses 192.168.3.101/24 ipv4.method manual
sudo nmcli con mod 'Wired connection 1' ipv4.gateway 192.168.3.1
sudo nmcli con mod 'Wired connection 1' ipv4.dns "192.168.3.1"
sudo nmcli c down 'Wired connection 1' && sudo nmcli c up 'Wired connection 1'

Then you have to stablich a new SSH connection with your Pi since the IP has changed.

And for setting the IP on Ubuntu, you have to create this file sudo nano /etc/netplan/01-netcfg.yaml, add the content below (edit as needed), and apply the configuration with sudo netplan apply.

network:
    version: 2
    renderer: networkd
    ethernets:
        eth0:
            dhcp4: false
            addresses:
                - 192.168.3.101/24
            nameservers:
                addresses: [8.8.8.8, 8.8.4.4, 192.168.3.1]
            routes:
                - to: default
                  via: 192.168.3.1

Then we need to set some basic configurations with raspi-config. Run sudo raspi-config, and perform the following tasks (In Ubuntu you need to install it first with sudo apt install raspi-config):

Optionally, change the password for the “pi” user (System Options –> Password)
Enable the SSH server (Interface Options –> SSH)
Set your locale settings (Localization Options)
Expand the filesystem to use the full capacity of your SD card (Advanced Options –> Expand Filesystem)
Disable Predictable Network Interface Names (Advanced Options –> Network Interface Names)

When you are done, exit the raspi-config tool and reboot your Pi.

If you don’t have one, you need to create an SSH key, even if it is possible to use Ansible by providing a password for your hosts interactively, a more convenient and secure way to do it is by using an SSH key pair. To create one run the following commands in the terminal:

# Create the .ssh folder
cd ~
mkdir .ssh
cd .ssh
# Create the authorized_keys file
touch authorized_keys

# Set proper access permissions
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

# Create the key pair
ssh-keygen
# Add the new key to the authorized_keys file
cat id_rsa.pub >> authorized_keys

# If you use Putty as your SSH client (Windows), generate a ppk key, otherwise, ignore these steps
sudo apt install putty-tools
puttygen id_rsa -o id_rsa.ppk

Copy the id_rsa and id_rsa.pub (and id_rsa.ppk if applicable) files to the machine you are going to use to execute the playbooks, I like to use sftp but you can do it the way you prefer (e.g. with a USB memory). After you have copied the key files, delete them from your Pi with rm ~/.ssh/id_rsa*. If you already have an SSH key, I trust you know how to add it to your OS at flashing time using the Raspberry Pi Imager software.

Now your Pi is ready to go, disconnect the keyboard and monitor and plug the Pi into your network (I recommend using a wired connection).

Install Ansible

In theory, you could install Ansible on the Raspberry Pi itself and run the playbooks directly on it with connection: local but it wouldn’t be practical, it is better to run the playbooks from a computer other than your Pi.

For installing Ansible, if you have a Unix based system (i.e. Linux, macOS) at your disposal, you are golden, installing Ansible is very simple:

# Install python3
sudo apt install python3 # On macOS use `brew install python3`
# Install pip3
sudo apt install python3-pip 
# Install Ansible version 9.5.1 (for reproducibility) with pip3 
sudo pip3 install ansible==9.5.1 --break-system-packages

A detailed guide for installing WSL can be found here and more information about installing Ansible can be found on the official documentation site.

Also, since we are going to be installing several packages from source on a low power SBC, some of these tasks are going to take a while and the SSH connection might get automatically closed due to inactivity, to prevent this situation, activate the sending of “keep-alive” packets to the server by editing the ssh_config file on the computer running Ansible, run sudo nano /etc/ssh/ssh_config and add these two lines under Host *:

    ServerAliveInterval 300
    ServerAliveCountMax 2

⚠️ Is very important not to skip the previous step, otherwise, the SSH connection migth silently fail while running long tasks in Ansible and you are going to be waiting pointlessly with your Raspberry Pi doing nothing in reality.

Download and Configure the Playbooks

The playbooks are on a public repository on GitHub, you can clone the repository with these commands:

# Install git if you don't have it already
sudo apt install git
# Clone the latest commit from the repository
git clone https://github.com/andresrcs/raspberry_pi_server.git --depth 1

Ansible doesn’t run playbooks in “world writable folders” so you need to set the raspberry_pi_server folder permissions to 755.

sudo chmod 755 raspberry_pi_server

Have in mind that on WSL it is necessary to remount the C drive in order to be able to change file permissions so you have to do this before changing permissions:

sudo umount /mnt/c 
sudo mount -t drvfs C: /mnt/c -o metadata

To configure the playbooks, you first need to define your “inventory”, this is a list of the hosts you are going to connect to, edit the raspberry_pi_server/inventory.ini file and add the IP of your Raspberry Pi(s):

[raspberries] # This represents a group of Raspberries
raspberry_01 ansible_host=192.168.18.101 # This is the hostname and IP for an individual Raspberry Pi
# You can add as many Raspberries as you need

Then set common variables for your [raspberries] group by editing the raspberry_pi_server/group_vars/raspberries.yml file, the most important things to check here are the path to your ssh key and the default user fot the operating system you are using.

---
ansible_user: pi # This is the default user for the operating system
ansible_become_method: sudo
ansible_python_interpreter: /usr/bin/python3
ansible_ssh_private_key_file: ~/.ssh/raspberrypi_key # The location of the ssh key in the computer runing Ansible.

Now, you can set installation options by editing the variables in the raspberry_pi_server/vars/config_vars.yml file, most of the already set options are fine for most use cases including reasonable security settings but at the bare minimum, you need to set your personal password for the PostgreSQL main user. You can also choose your preferred method for installing R, if you don’t want to use the default option, have in mind that using the R4Pi Project option is only possible with Raspberry Pi OS and it will always install the latest R version available.

# System Configurations ########################################################

# Swap parameters
swap_file_path: /var/swap.1
# Use any of the following suffixes
# c=1
# w=2
# b=512
# kB=1000
# K=1024
# MB=1000*1000
# M=1024*1024
# xM=M
# GB=1000*1000*1000
# G=1024*1024*1024
swap_file_size: 3GB
swappiness: '10'

# Install Nginx?
install_nginx: true

# Install PostgreSQL?
install_postgresql: true

# Security Configurations ######################################################

# Ports to open
exposed_ports:
  - { rule: 'allow', port: 22, proto: 'tcp' }     # ssh
  - { rule: 'allow', port: 22, proto: 'udp' }     # ssh
  - { rule: 'allow', port: 80, proto: 'tcp' }     # http
  - { rule: 'allow', port: 5432, proto: 'tcp' }   # Postgresql (ommit if Postgresql is not installed)
  - { rule: 'allow', port: 5432, proto: 'udp' }   # Postgresql (ommit if Postgresql is not installed)
  # - { rule: 'allow', port: 8787, proto: 'tcp' } # RStudio Server (uncomment if Nginx is not installed)
  # - { rule: 'allow', port: 3838, proto: 'tcp' } # Shiny Server (uncomment if Nginx is not installed)

# Password for the main PostgreSQL user (ignore if PostgreSQL is not installed)
postgres_password: 'very_secure_password'

# Access rules for Postgresql (ignore if PostgreSQL is not installed)
postgresql_rules:
  - { contype: local, users: postgres, address: samehost, method: trust }
  - { contype: local, users: all, address: samehost, method: trust }
  - { contype: hostssl, users: all, address: 0.0.0.0/0, method: scram-sha-256 }
  - { contype: host, users: all, address: 0.0.0.0/0, method: scram-sha-256 }
  - { contype: hostssl, users: all, address: '::0/0', method: scram-sha-256 }
  - { contype: host, users: all, address: '::0/0', method: scram-sha-256 }
  - { contype: host, users: all, address: 127.0.0.1/32, method: scram-sha-256 }
  - { contype: host, users: all, address: '::1/128', method: scram-sha-256 }

# Main Software Versions #######################################################

# R version to install
use_r4pi: true # If set to true the latest R version will be installed, this is only compatible with
                # Raspberry Pi OS.
use_rig: false # Don't forget to specify an R version in the following line
r_version: '' # (e.g. '4.3.3') Currently, you can only choose the R version if it's going to be
                   # installed with rig or compiled (with both 'use_r4pi' and 'use_rig' set as false).

# Shiny-server version to install
shiny_server_version: 'v1.5.21.1012'

# RStudio version to install
rstudio_version: '2024.04.0-daily+651'

Run the Playbooks

Under ideal conditions, like using the latest Raspberry Pi model with 8GB of RAM, booting from an SSD drive and connected to a stable network, you could simply access the raspberry_pi_server folder and run the site.yml playbook to install everything at once:

cd raspberry_pi_server
ansible-playbook site.yml

But under less than ideal conditions it is very likely tha you will encounter some random erros caused by overloading the Pi or unestable network connections, so, to mitigate this issue and to make the installation process more flexible, I have divided the process into three individual playbooks:

cd raspberry_pi_server
ansible-playbook install_basic_services.yml
ansible-playbook install_shiny_server.yml
ansible-playbook install_rstudio_server.yml

If you decide that you only need Shiny server or RStudio server but not the other, or you already have installed the support services you want, you can run only the playbooks you need.

If you want to update something in the future, like R, RStudio, or Shiny server (I can’t guaranty that is always going to work out of the box), you can simply change the version in the config file and run the specific part of the playbook by taking advantage of the defined “tags”.

For example, this will only install the R version defined in the config file and nothing more:

cd raspberry_pi_server
ansible-playbook install_basic_services.yml --tags "r"

The available tags are:

install_basic_services.yml
- secure: Set security settings on the server
- swap: Add swap memory to the server
- nginx: Install and configure Nginx + PHP
- postgresql: Install and configure PostgreSQL
- r: Install R from source
- disable_wifi: Disable Wifi and Bluetooth module
install_shiny_server.yml
- shiny-server: Install shiny-server from source
- configure_shiny: Configure shiny-server
install_rstudio_server.yml
- rstudio: Install RStudio server from source
- configure_rstudio: Configure RStudio server

cd raspberry_pi_server
ansible-playbook install_basic_services.yml --skip-tags "postgresql"

After successfully running all the playbooks, you will have a fully functional installation ready to be used, so you could simply open an RStudio session at http://your_server_ip/rstudio and/or publish your Shiny apps in the /srv/shiny-server folder and access them at http://your_server_ip/shiny/your_app_name.

⚠️ If the Nginx configuration hasn’t been set by the install_basic_services.yml playbook because you have omitted it, you have to configure the reverse proxy manually or access the services with their specific port, e.g. http://your_server_ip/3838/your_app_name, this would also require you to open the corresponding port in the firewall.

Final Notes

Even with the automation, installing natively is still going to take much longer than, for example, using containers, especially if you are using a low RAM Raspberry Pi SBC (< 4GB) but since it is automatized, you can simply let it running and come back later, and you will enjoy the simpler manageability of a native installation.

Also, be aware that the installation process is going to push your Raspberry Pi to its limits so you can expect some random errors, but that doesn’t mean it’s not going to work, just restart your Pi, rerun the problematic playbook and see if it works the second time. I have done my best to make this functional in the long run by setting specific software versions but it is still possible that a system library update might break the playbooks, if you find any issues, please go to the GitHub repository for the playbooks and file an issue report, or even better, if you know how to fix it, make a pull request.

As a final comment, I want to point out that this approach can be further extended accordingly to your own needs, for example, you can implement automatic restoration of your server including deployed content. For example, I have an additional playbook to restore the applications, services, and databases I normally run on my Raspberry Pi, so in the event of a catastrophic failure, I can get back to a working state in a short period of time and with little effort, compared with doing a manual restoration of my system.

Customizing your package-library location

Mon, 26 Oct 2020 00:00:00 +0000

For reasonably experienced R users this simple topic might not seem worthy of a blog post, so if you are not an R beginner, you may want to skip this post. Having that said, I have had to explain this task so many times on R community forums that writing one becomes mandatory to avoid typing the same text yet once again.

The reasons to customize your library path instead of going with the defaults could be varied, but for R beginners the most common one is to avoid package installation problems due to write permissions, non-ASCII characters on folder paths, cloud-synced folders, or network drives, which lead into error messages such as:

Warning in install.packages :
  'lib = "/path/to/your/library"' is not writable
Error in install.packages : unable to install packages

** byte-compile and prepare package for lazy loading
Error: unexpected symbol in "setwd('incomplete/path/to/your/library"
Execution halted

Very often, the solution for these issues is to set your package library somewhere else in your system, where you have proper permissions and there are no known R incompatibilities. The way to change a package library location is to manually set it on a startup file i.e. Rprofile.site and Renviron.site files for R-version level (located at R_HOME/etc/) or .Rprofile and .Renviron files, for user-level (located at your HOME folder) or project-level (located at the current working directory).

✏️ For more information about R startup files, you can read the following support article.

For Renviron type files, the corresponding environmental variable must be specified, for example, if I want to change the R library trees on a Linux system I would add a line like this to the Renviron.site file located at /usr/lib/R/etc/ (On this case the R_HOME environmental variable gets translated to /usr/lib/R/).

R_LIBS_SITE="/usr/lib/R/site-library:/usr/lib/R/library"

Being the first location from the left the default one, which gets used by install.packages() if the lib argument is not specified, and all of them (which are colon-separated) are scanned for packages at startup in order. Have in mind that only directories that exist at the time will be included.

For Rprofile type files, the .libPaths() function must be used, since these files must contain valid R code to be executed at startup, the equivalent R command to the previous example would be:

.libPaths(c("/usr/lib/R/site-library", "/usr/lib/R/library"))

Obviously, the folder paths are going to be different depending on your specific operating system and setup, but I hope you get the general idea that you have to specify a folder path that is suitable for your specific needs, for example, on a Windows system the equivalent would be:

.libPaths(c("C:/Program Files/R/R-4.0.3/library"))

By default, the default package-library is set at the user-level (HOME folder), which, in some cases, can cause the aforementioned issues. Now, this is opinionated advice, but for simple individual use, I find it more practical to set the default package-library at R-version level (R_HOME/library/ folder) and explicitly make use of other locations, selectively, and accordingly to specific use cases. Be aware that depending on your security settings and operating system, you might need to run your R session with “administrator” rights or from a user with “sudo” rights in order for this to work.

Alternatives for retrieving sensor data from Arduino compatible microcontrollers into R

Sun, 09 Feb 2020 00:00:00 +0000

Have you ever wanted to get your own real-world data using sensors? With the advent of the Open-Source Hardware movement, this has become more accessible for the general public, microcontrollers and sensors have become cheaper and easier to program with human-friendly languages like Arduino (C++) and MicroPython (Python), so now anybody can produce its own real-world sensorial data, but, there is still a problem, How do we get this data into R so we can analyze it?

Depending on your application’s requirements you could choose among a plethora of options. For example, if you only need data on batches and you can have easy physical access to your device, you could simply add an SD card module to your project and log the data to a .csv file on an SD card but, that wouldn’t be fun, isn’t it? A much cooler solution would be to retrieve data remotely and preferably wirelessly, don’t you think?. Let’s explore some options.

Serial Connection

If you have never thought about getting data from microcontrollers before, I bet the first thought that comes to your mind, is to somehow directly connect the microcontroller with R to get the data, but this is currently the most technically challenging option, it can be done using a serial connection between the microcontroller and your computer, but as far as I know, there is only one R package implementing this and only runs on POSIX-compatible systems (so no Windows).

I can’t test this approach myself since I don’t have a physical Linux or macOS machine to connect a microcontroller to, so for this case, I’m just going to link to a blog post by @haozhu233 (the package author) showcasing its functionality.

If your application requires real-time data acquisition or very low latency this might be your only option and you may want to dive deeper into this.

Streaming arduino sensor signals right into @rstudio! I wonder if there is really anything #rstats can’t do 😎😎😎 pic.twitter.com/nT6q6TT6xc
— Hao Zhu (@haozhu233) January 25, 2019

Writing to a SQL server

There are two ways of doing this, one is to directly connect the microcontroller to the database and the other is to use another service to pull data from the microcontroller and write it into the database.

The first option is harder to accomplish, as far as I know, there is only one Arduino library that provides a SQL connector for MySQL servers but, there are no already-made solutions for other SQL servers and although you could write your own connector for other Open Source SQL servers (like PostgreSQL), that would require a considerable amount of technical skills.

The second option is easier to implement, and my personal favorite.

The first step is to broadcast the data from the microcontroller, one way of doing this is to implement a simple web server on the microcontroller itself and serve the data on a suitable format like HTML, JSON or CSV.

This is a simple example of an Arduino Sketch for an ESP8266 board (Wemos D1 mini) and DS18B20 temperature probe sensors, that serves HTML and JSON outputs over a web server.

The second step is to pull the data out of the microcontroller and into a SQL server, there are a lot of approaches and languages you can use to implement this part but obviously we are going to use R. You can write an R script that connects to the microcontroller, retrieves the data and writes it into a SQL server, and schedule the script with a cron job.

#! /usr/bin/env Rscript

# DATA ADQUISITION##############################################################
raw_data <- rjson::fromJSON(file="http://192.168.0.102/reading.json") 
current_time <- as.character(Sys.time())

# PARAMETERS ###################################################################
current_location = "some_place"
point_names = c("probe0" = "measurement_point_0", 
                "probe1" = "measurement_point_1",
                "probe2" = "measurement_point_2")

# LIBRARIES ####################################################################
library(dplyr)

# DATA CLEANING ################################################################
tidy_data <- as_tibble(raw_data) %>%
    tidyr::gather(measurement_point, value) %>% 
    mutate(time = current_time,
           location = current_location,
           variable = stringr::str_extract(measurement_point, "^.+(?=-)"),
           measurement_point = stringr::str_extract(measurement_point, "(?<=-).+$"),
           measurement_point = stringr::str_replace_all(measurement_point, point_names)
           ) %>%
    select(time, location, variable, measurement_point, value)

# LOAD DATA ####################################################################
con <- dbConnect(drv = odbc::odbc(),
                 driver = 'PostgreSQL ANSI',
                 server = 'localhost',
                 database = 'sensors',
                 port = 5432,
                 uid = Sys.getenv('MY_UID'),
                 pwd = Sys.getenv('MY_PWD'),
                 encoding = 'utf8')

DBI::dbAppendTable(conn = con,
                   name = 'sensors_data',
                   value = tidy_data)

odbc::dbDisconnect(con)

unlink(x = "*.log", force = TRUE)

Now that you have the data into a SQL server, you are at one SQL query away of getting it into R

library(odbc)

con <- dbConnect(drv = odbc::odbc(),
                 driver = 'PostgreSQL ANSI',
                 server = 'localhost',
                 database = 'sensors',
                 port = 5432,
                 uid = Sys.getenv('MY_UID'),
                 pwd = Sys.getenv('MY_PWD'),
                 encoding = 'utf8')

query <- "
    SELECT * 
    FROM public.sensors_data
    "

raw_data <- dbGetQuery(
  conn = con,
  statement = query
)

dbDisconnect(con)

What makes this approach my personal favorite is that it is fairly easy to implement, very flexible and allows you to implement almost any security measure you might need.

Using the MQTT protocol

For more resource-constrained applications you can use a similar approach to the previous one, but with a different technology, the MQTT protocol is ideal for low power devices over slow and unreliable connections, moreover, this is also a way to get data from LoRa networks like the publicly available TTN (The Things Network) which provides an API over MQTT, giving you access to a low-power wide-area network for remote sensor deployment over greater distances.

This approach also allows you to write data to a SQL server like in the previous example but, a more interesting application for this would be to get instant status updates to show on dashboards or shiny apps. Consider this simple shiny app that receives a message from a simulated remote sensor and updates the color of a circle accordingly, you can update the temperature value of the simulated sensor by sending an MQTT message to the test topic from a system terminal on any computer with this command mosquitto_pub -h test.mosquitto.org -t simulated_sensor -q 1 -m 25.9 -r and see how the color changes.

✏️ In order for this to work you need to have the mosquitto client installed in your system.

sudo apt install mosquitto-clients

library(shiny)
library(ggplot2)
library(rmqtt)  # This package is just a simple wrapper for an MQTT client using
                # system() calls, you need to have the mosquitto client installed
                # in your system for this to work.

# Initialize Simulated Sensor
mqtt_topic_publish(
    topic = "simulated_sensor", 
    message_to_send = "24.9",
    host = "test.mosquitto.org",
    port = 1883,
    qos = 1, 
    retain_message = TRUE
)

ui <- fluidPage(
    titlePanel("Dashboard"),
    mainPanel(
        h2("Temperature Indicator"),
        plotOutput("status_indicator")
    )
)

server <- function(input, output, session) {
    
    get_color <- function() {
        message <- mqtt_topic_subscribe(
            topic = "simulated_sensor", 
            host = "test.mosquitto.org",
            port = 1883,
            qos = 1,
            intern = TRUE,
            num.messages = 1
        )
        
        if (as.numeric(message) > 25) {
            color = 'red'
        } else {
            color = 'green'
        }
        
        return(color)
    }
    
    pollData <- reactivePoll(100, session,
                             checkFunc = get_color,
                             valueFunc = get_color
    )
    
    output$status_indicator <- renderPlot({
        ggplot(data = data.frame(x = 1, y = 1), aes(x, y)) +
            geom_point(size = 50, color = pollData()) +
            scale_x_discrete() +
            scale_y_discrete() +
            theme_void() +
            theme(axis.title = element_blank())
    })
}

shinyApp(ui = ui, server = server)

If you are interested in testing this, here is a GitHub Gist with the Arduino Sketch for an ESP8266 board whit a DS18B20 temperature probe sensor that can update the values in the example app.

I know this doesn’t look very interesting right now but consider the implications. You could be getting instant updates in your dashboard from sensors deployed at kilometers downrange and you could even control remote actuators from your shiny app by sending an MQTT message from it with the click of a button!

The downside of this method is that you need to set an MQTT broker (you should not abuse the publicly available ones that are intended for testing), but it can be done for no or little money on a Raspberry Pi or an EC2 AWS instance.

Disclaimer

I want to make clear that this is by no means an exhaustive list, there are other alternatives and countless variations of the ones presented here so if none of these fitts your particular project’s needs, just keep looking, very likely someone has come up with a more suitable method.

✏️ If you want to see a detailed step by step example of any of these options, write it on the comments and I’ll try to make a blog post about it.

Keeping track of your actual internet speed over time, using R and Speedtest-CLI

Tue, 21 Jan 2020 00:00:00 +0000

I thought this only happened in my country (Perú), but it turns out it happens to people around the globe, you contract an Internet Service Plan that offers up to X Mbps, where X is an impressive value (relative to each country’s context), but the catch is in that word, Up to.

If it makes you feel better, it could be worse if you lived in another country, trust me, I know... pic.twitter.com/Rm3J2J2Cdz
— Andrés Castro Socolich (@Andresrcs) January 16, 2020

In reality, you end up getting far slower download speeds than advertised, and ISPs (Internet Service Providers) get away with this because they only test download speed with you once when they install the service (suspiciously it works great at that moment) and most people don’t bother to keep track of their download speeds over time because it would be impractical to manually perform speed tests every short period of time, luckily there is a better way, Ookla’s Speedtest service provides a Command Line Interface (CLI) that allows you to perform tests from your system terminal enabling you to set up automated scripts to collect connection performance data.

In the next lines, I’m going to show you how to do this from R and get the results in the form of nice plots you can tweet to your ISP to annoy them a little bit. 😆

Getting Your Setup Ready

We have to set up our working environment for this project. First, we need a machine that is constantly powered on and connected to the same internet connection you want to test so it can perform the tests and store the data, it could be any computer connected to your local network but the most practical (and cost-effective) solution I have found is using a Raspberry Pi SBC so I’m going to use one for this project.

Since you are reading an R related article I’m going to assume you already have your basic R environment set, so we are going to focus on the additional stuff, we need to install the Speedtest Command Line Interface (CLI), if you are in a Linux machine, you can do it with this commands on a system terminal:

✏️ There are installation instructions for other OSs on the Speedtest web site

sudo apt-get install curl
curl -s https://packagecloud.io/install/repositories/ookla/speedtest-cli/script.deb.sh | sudo bash
sudo apt-get install speedtest

✏️ Check if you have an older python version of the CLI installed in your system and make sure to uninstall it with this command sudo pip uninstall speedtes-cli, since it could cause conflicts with the newer official version. This happened to me and I didn’t notice 😳, thanks to @jasongrahn for the heads up.

Gathering Data

Now we can write an R script that retrieves the test output, parses the content and stores the data. In this example I’m going to store it into a PostgreSQL database but you could choose other options like another SQL server, Google Sheets, a CSV file, etc. I have called this script file speedtest_job.R.

#! /usr/bin/env Rscript

# Data adquisition
output <- read.csv(
  text = system(
    command = "speedtest -f csv --output-header --accept-license",
    intern = TRUE
  )
)

results <- data.frame(
  stringsAsFactors = FALSE,
  time = as.POSIXct(Sys.time()),
  ip = system("curl ifconfig.me", intern = TRUE), # This field is no longer provided on the new CLI version
  ping = output$latency,
  download = output$download * 7.629395E-6, # Convert to Mbps
  upload = output$upload * 7.629395E-6, # Convert to Mbps
  isp = "movistar" # This field is not provided on CSV format
)

# Data storing
con <- dbConnect(drv = odbc::odbc(),
                 driver = 'PostgreSQL ANSI',
                 server = 'localhost',
                 database = 'internet', # Name of the database on the sql server
                 port = 5432,
                 uid = Sys.getenv('MY_UID'),
                 pwd = Sys.getenv('MY_PWD'),
                 encoding = 'utf8')

DBI::dbAppendTable(
  conn = con,
  name = "speed_test", # Table name on the sql server
  value = results
)

odbc::dbDisconnect(conn = con)

# Cleaning log files
unlink(x = "*.log", force = TRUE)

Now we can schedule a cron job to run this script on regular intervals, I’m going to set it to run once every hour running this commands on a system terminal:

✏️ This can also be done from R using the cronR package.

env EDITOR=nano crontab -e
  # Add this line
  0 * * * * /usr/local/lib/R/bin/Rscript '/home/pi/speedtest_job.R' >> '/dev/null' 2>&1 # Change the file path as needed
sudo service cron reload

After letting some time to pass for registers to accumulate, we can fetch data from the server with a SQL query:

library(odbc)

con <- dbConnect(drv = odbc::odbc(),
                 driver = 'PostgreSQL ANSI',
                 server = Sys.getenv('MY_REMOTE'),
                 database = 'internet',
                 port = 5432,
                 uid = Sys.getenv('MY_UID'),
                 pwd = Sys.getenv('MY_PWD'),
                 encoding = 'utf8')

query <- "
    SELECT * 
    FROM public.speed_test 
    ORDER BY time
    "

raw_data <- dbGetQuery(
  conn = con,
  statement = query
)

dbDisconnect(con)

Visualizing the Data

Now that we have some data to work with we can start making some plots. Since this is not an article about ggplopt2 I’m not going to get much into details about this part, I’m just going to show you some interesting plots you can get about your internet connection speed.

In Figure 1 we can see how observed download speeds distribute, letting aside the fact that the advertised speed is already slow (for developed countries standards), this reveals a pattern that is very common, most of the time you get download speeds that are much slower than advertised by the ISP and even so slow that defaults the terms of your contract, like in my case (I actually used this image to file a complaint).

library(tidyverse)
library(xkcd)
library(lubridate)
library(tibbletime)
library(scales)

# These are my personal theme settings you can ignore them if you prefer
theme_set(
  theme_gray() +
    theme_xkcd() +
    theme(legend.position = "right",
          plot.title.position = "plot",
          axis.title.x = element_text(margin = margin(t = 10)),
          axis.title.y = element_text(margin = margin(r = 10)),
          axis.text.x = element_text(angle = 20, hjust = 1, vjust = 1),
          plot.margin = margin(10, 10, 10, 10),
          text = element_text(family = "Cloud Calligraphy"))
)

color <- c("Mode" = "#008B00", "Parameters" = "orange")

raw_data %>% 
  ggplot(aes(x = download)) +
  geom_histogram(binwidth = 0.1, fill = "#00B2EE") +
  geom_vline(aes(xintercept = 20, color = "Parameters"),  linetype = "dashed") +
  geom_vline(aes(xintercept = 8, color = "Parameters"),  linetype = "dashed") +
  stat_bin(geom = "vline",
           aes(xintercept = stat(ifelse(count == max(count), x, NA)),
               color = "Mode"),
           binwidth = 0.1) +
  annotate("text",
           x = c(8.4, 20.4),
           y = c(8, 8),
           label = c("Guaranteed Minimum Speed", "Advertised Speed"),
           family = "Cloud Calligraphy",
           size = 5,
           angle = 90) +
  stat_bin(geom = "label",
           aes(label = stat(ifelse(count == max(count), round(x, 1), NA))),
           binwidth = 0.1,
           family = "xkcd",
           color = "#008B00",
           vjust = -0.2) +
  labs(title = 'HISTOGRAM OF OBSERVED DOWNLOAD SPEEDS',
       subtitle ='ISP: MOVISTAR 20 Mbps', 
       x = 'Speed (Mbps)',
       y = 'Frequency',
       colour = '') +
  scale_x_continuous(breaks = seq(0, 22, by = 2),
                     limits = c(0, 23)) +
  scale_colour_manual(values = color) +
  coord_cartesian(clip = 'off') +
  NULL

Figure 1: Histogram of Observed Download Speeds

With figures 2 and 3, we can show how often the download speeds fall below the minimum guaranteed speed.


colors <- c("Mean Speed" = "blue",
            "Parameters" = "orange",
            "Fault" = "red",
            "Download Speed" = "#00B2EE")

plot_data <- raw_data %>%
  as_tbl_time(index = time) %>% 
  collapse_by('1 hour', side = 'start', clean = TRUE)

mean_speed <- plot_data %>% 
  pull(download) %>% 
  mean() %>%
  round(1)

plot_data %>% 
  ggplot(aes(x = time, y = download)) +
  geom_line(aes(color = "Download Speed")) +
  geom_point(data = plot_data %>% filter(download < 8),
             aes(color = "Fault")) +
  geom_point(data = plot_data %>% filter(download < 8),
             shape = 1,
             color = "red",
             size = 5) +
  geom_hline(aes(yintercept = 20, color = "Parameters"),  linetype = "dashed") +
  geom_hline(aes(yintercept = 8, color = "Parameters"),  linetype = "dashed") +
  geom_hline(aes(yintercept = mean_speed, color = "Mean Speed")) +
  annotate("text",
           x = as.POSIXct(c("2019-06-24 11:00:00 UTC", "2019-06-24 11:00:00 UTC")),
           y = c(7, 19),
           label = c("Guaranteed Minimum Speed", "Advertised Speed"),
           family = "Cloud Calligraphy",
           size=5) +
  geom_label(x = as.POSIXct("2019-08-01 11:00:00 UTC"),
             y = mean_speed,
             label = mean_speed,
             family = "xkcd",
             show.legend = FALSE,
             inherit.aes = FALSE,
             color = "blue")  +
  labs(title = 'OBSERVED DOWNLOAD SPEEDS',
       subtitle ='ISP: MOVISTAR 20 Mbps', 
       x = 'Date',
       y = 'Speed (Mbps)',
       color = 'Legend:') +
  scale_x_datetime(date_breaks = "5 days",
                   labels = label_date_short(), 
                   expand = expansion(c(0, 0.04))) +
  scale_y_continuous(breaks = seq(0, 22, by = 2), limits = c(0, 23)) +
  scale_colour_manual(values = colors) +
  coord_cartesian(clip = 'off') +
  NULL

Figure 2: Observed Download Speeds

faults <- plot_data %>%
  mutate(event_type = ifelse(download <= 8, 'fault', 'normal')) %>% 
  filter(event_type == 'fault') %>% 
  mutate(tbf = as.numeric(as.period(interval(lag(time), time),
                                    unit = 'seconds')) / (3600)) %>% 
  tail(-1)

faults %>% 
  ggplot(aes(x = '', y = tbf)) +
  geom_boxplot(fill = '#FF303094') +
  coord_flip() +
  geom_label(y = median(faults$tbf),
             label = paste(round(median(faults$tbf),1), 'h'),
             family = "xkcd",
             show.legend = FALSE,
             color = "blue") +
  labs(title = 'BOXPLOT OF TIME BETWEEN FAILURES',
       subtitle ='ISP: MOVISTAR 20 Mbps',
       x = '',
       y = 'Hours') +
  NULL

Figure 3: Boxplot of Time Between Failures

And with Figure 4 we can find out when the peak hours occur, so we can know at what time of the day is more likely for us to experience slow internet speeds.

hour_data <- plot_data %>%
    as_data_frame() %>% 
    mutate(time = format(time, "%H:%M"))

hour_data %>% 
  ggplot(aes(x = time, y = download)) +
  geom_boxplot(fill = "#00B2EE") +
  geom_point(data = hour_data %>% filter(download < 8),
             shape = 1,
             color = "red",
             size = 5) +
  geom_hline(aes(yintercept = 20), color = "orange",  linetype = "dashed") +
  geom_hline(aes(yintercept = 8), color = "orange",  linetype = "dashed") +
  annotate("text",
           x = c("04:00", "04:00"),
           y = c(7, 19),
           label = c("Guaranteed Minimum Speed", "Advertised Speed"),
           family = "Cloud Calligraphy",
           size = 5) +
  labs(title = 'BOXPLOT OF OBSERVED DOWNLOAD SPEEDS BY HOUR OF THE DAY',
       subtitle ='ISP: MOVISTAR 20 Mbps',
       x = 'Hour',
       y = 'Speed (Mbps)') +
  scale_y_continuous(breaks = seq(0, 22, by = 2),
                     limits = c(0, 22)) +
  coord_cartesian(clip = 'off') +
  NULL

Figure 4: Boxplot of Observer Download Speeds by Hour of the Day

Once you have data, plotting possibilities are only limited by your imagination, so I’m going to stop here, I hope you have enjoyed reading this article and you are motivated now to start monitoring your own internet speed. Have fun! see you soon!.

About

Sat, 18 Jan 2020 00:00:00 +0000

Andrés Castro Socolich

Hello! my name is Andrés I’m from Arequipa-Perú, I’m an Industrial Engineer with a specialization in Data Science and my favorite tool is R, mostly because of its friendly and supportive community, so I have made this blog to share my personal, data projects with the data science community.

Posts here are going to have a broad variety of topics but all are going to be data-driven, I hope you enjoy reading my posts and find them useful.