Clusterboard A64 Insidious Reset Problem: Solved

A bare-metal compute node may soft-lock, spin-lock, deadlock, overheat, encounter resource starvation, the Docker daemon goes away, systemd becomes unstable, and on. In these cases, a watchdog timer acting like a dead man’s switch is not updated, a timer reaches zero and the watchdog circuit restarts all the hardware. However, the clusterboard A64 SoCs have a WDT reset problem which we solve satisfyingly.

Embedded Linux Completely from Scratch

Each node of my cluster computer is nameless and stateless like an AWS Lambda, so the entire OS must reside in memory. Having explored minimal Debian, Alpine Linux, and even RancherOS, the most exciting conclusion is to learn to compile the embedded Linux kernel and bootloaders from scratch for ARM64 and learn how to network-boot bare-metal hardware over HTTP.

Set up an ODROID XU4 RAID Server with Cross-Compiled Docker Images for ARM

ODROID XU4s are awesome. They are 8-core, 2GHz ARM single-board computers (SBCs) with Gigabit Ethernet and USB 3.0 connections. They only have 2GB of DDR3 RAM, but when paired with a CloudShell2 case and a couple of HDDs (or SSDs), they become an impressive NAS, or better, dedicated network-activity drivers for other projects.

Puppet Several Headless Chrome Instances Behind Different VPNs in Docker with no IP Leaks

For my machine learning projects, I need data you just can’t buy. This requires SPA (Single-Page App) web data extraction involving multiple clicks and page scrolling that curl can’t handle. Headless Chrome puppeted by RDP (Remote Debug Protocol) is a brilliant solution for this. Here is how I orchestrated several headless Chrome instances across several VPNs in Docker.

Attempts at Supplying Efficient Logic-Level Voltage with no Decay to Sensitive Electronics

A power supply, when suddenly turned off, bleeds voltage slowly. Attached electronics experience a gradual voltage decline from 5V to 3.3V and eventually to zero. The problem is that microcontrollers and microprocessors don’t know how to behave with under-voltage. Their behavior and flash memory integrity is not defined. Flash memory can even be erased. Here I outline my attempts to achieve an efficient logic-level power supply.

Flash the AVR Atmel ATtiny Chips with a Modified CH341A ISP Programmer

I bricked a lot of ATtiny85 Digistump chips while I was building a custom bootloader. Here is how I unbricked them and flashed onto them bootloaders with less than $5 of hardware using an AliExpress CH341A BIOS programmer and some soldering. The problem is that the CH341A is not designed for ATtiny chips, but for flashing motherboard BIOS chips, so I had to hack it.

Build a Secure MFA Web-Browser-Based Remote Desktop Gateway on a Raspberry Pi

Let’s say I’m in a Starbucks or the airport (or both) and I want to connect to my Windows (or OSX or Linux) machine to check on things. Maybe I don’t want to take my primary computer with me on vacation; maybe I’ll just take a Chromebook. Behind a restrictive firewall all we have is port 80 and port 443 (no VNC or RDP allowed), so let’s make a secure web-based remote desktop gateway with a Raspberry Pi, Docker, and Cloudflare.

Random Useful Linux Commands

These useful Linux commands and snippets are a reference for myself, all in one place, so I can refer to them quickly during my workflow without searching for them again and again. These include networking, git, grep, journald, ss/netstat, process uptime, and more. More are added as I go.

Hack the Digispark BlinkStick and ATtiny85 Microcontroller into a FadeStick

My project this weekend was to fork both the BlinkStick C firmware and the Java API to make the Digispark USB hardware with the AVR ATtiny85 microprocessor do something never done before: execute color patterns on the microcontroller, not the host CPU. I outline how I failed many times, and how I eventually succeeded with links to my Github repos and pictures of my hardware hacks.

Big Data Backup to S3 Glacier via Java SDK with Spend Evaluation

Why use AWS Glacier for big data backup? It’s exceedingly inexpensive to archive data for disaster recovery on Glacier. AWS Glacier is only US$0.004 per GB/mo, and their SDK is beautiful. Here I outline a pricing matrix for cloud storage providers, and I take a look at the Java SDK for working with AWS Glacier to effectively archive 200GB a week.

Block Malicious and Obnoxious Ads and Scripts at the DNS Level with Docker

Ads are getting more and more aggressive. Some ads are even malicious. Some sites even load crypto-currency mining scripts in the background in JavaScript. Users have discovered that a lot of traffic is advertisement or tracking scripts, and putting a damaging strain on our mobile device data plans and batteries. Here I explain how to safely setup Pi-hole – a network-wide ad-traffic blocker – in a Docker container on an external Linux device as a hardware DNS server to block ads.

Profile Remote Java Apps with VisualVM or JMC

Sometimes remote Java apps leak memory or are killed by the OS. Let’s connect through an SSH tunnel to a remote JVM running on an embedded Ubuntu system and profile memory and CPU usage with free tools VisualVM and JStatD, or Java Mission Control. No firewall adjustments are needed. We’ll also set up JMX connections to allow remote heap dumps and garbage collection. Finally, I’ll explore the features of VisualVM.

Troubleshooting a Voltage Discrepancy

Breadboard power supplies cost less than a dollar on AliExpress. They are quite convenient for quickly powering and prototyping microprocessor circuits, Arduino projects with sketches, USB-powered prototypes, and on. The imagination is the limit. I spent the morning trying to figure out why my MB102 breadboard power supply was outputting only 3.5V, not the expected 5.0V.

Cluster Computing – Benchmarking Local Storage

Given a cluster computing rig of twenty-eight processors, each can have either a USB 2.0 or microSD local flash storage. Which type of flash and maker is the fastest? Make the wrong choice and the cluster is painfully slow. Not all microSD cards or USB drives are made the same, and interestingly random read and write speeds vary wildly. Here I test several storage configurations with striking benchmark results.

Experience with Inexpensive MicroSD Cards

For the cluster computing project I’m working on, I need 28 microSD cards. There was an AliExpress sale with good reviews, so I ordered a batch of 30 microSD cards, and at a great price point at the time. As long as the cards are Class 10 and work then we should be good, right? Results: Half are fake or defective. The rest are painfully slow. No refunds.

Cleaning Raw Candle Data for Time-Series Analysis

Problem: How to clean the raw OHLCV candle data from the broker for time series analysis? Suppose we have an autonomous program that prioritizes and continually downloads the latest minute and day candles, as well as periodically gets new symbols from the broker. The problem is that the candles are not guaranteed to be full-period […]

Acquiring Candle Data for Quantitative Financial Analysis Research

This would make a good interview question: There are about 120,000 public North American securities, bonds, rights, and index symbols. You have a paid API that can access all of them in OHLCV format if they are quotable. There are two critical API constraints: 15,000 calls per hour 20 calls per second Napkin math Minute […]

Automated Web Testing with PHP and Selenium

Things break. Just the other day through a series of seemingly unrelated events, a new Microsoft x509 certificate made its way into a security handshake process which went unnoticed until current single sign-on sessions began to expire. Had we also had automated security testing, we would have caught this one-off. I’ll explain how I set […]

Algorithm: Optimized PDF Web Page Print Layout

I’d like to share my efforts to prevent page breaks in the middle of paragraphs and maximize the use of page space when printing web pages to PDF. I’ll outline how this PHP+NodeJS+Chrome tool and algorithm accomplish this. The motivation is to prevent pictures from being cut off, cut halfway through, or from being pushed […]

GoAccess Web Log Analyzer Installation on CentOS

GoAccess web log analyzer is a beautiful tool to show real-time traffic and stats – including GeoIP information, bandwidth usage, and visitor time distributions – of my web projects and apps over and above Google’s Webmaster Tools and UA reporting. At a glance I can see current traffic and historic traffic just by adding a […]

Download YouTube 4K Videos with PHP and SlimerJS

Among friends let’s agree we’ll be privately caching videos and not permanently saving them, or we’ll be using them for Fair Use, and we’ll certainly not upload nor share these videos outside of the originating platform (e.g. Existing YouTube downloader scripts: YouTube-Downloader (does not work with videos using a cipher signature) YouTube video downloader […]

Running Xvfb on a RHEL Shared Host (without X)

This is how I compiled the Xorg Server for RHEL on a CentOS machine with modifications to create a portable Xvfb binary. Xvfb (X virtual framebuffer) is an in-memory display server for Linux and Unix-like OSes. It enables running graphical applications without a display such as running a headless browser (e.g. A full-blown Firefox instance […]

How to Get a LINE User MID

When getting started with LINE API messaging, you need to know the mid of a message recipient. It’s not his/her username. It’s a string that looks like ub8dbd4a12c322f6c0118883d839c55a4. LINE utilizes a callback URL that you can set for your trial LINE bot. At this endpoint you can place a script, shown below, which will report […]

Dropbox Direct Download Link

Sometimes I want to share a large file on my site without tying up bandwidth. If I don’t intend for the file to be downloaded often, I can offload the work to Dropbox and use PHP or htaccess to share a convenient URL. You can get the direct download link for a Dropbox […]

Mitigating AdWords Click Fraud

Every now and then there is an hours-long campaign of fraudulent AdWords-clicking from countries all over the world, ranging from Iran to Singapore, dedicated to clicking my cost-per-click Google ads in a vain attempt to exhaust a given daily budget early. My hat goes off to the chap for organizing the attack, or at least […]

Posted in: Problem StoriesTags: AdWords, Cloudflare, Google, GTM, Htaccess

Pokémon Go Scanner

The inspiration to make my own Pokémon Go scanner came from this great site (and Twitter feed). Try this site first before venturing out to make your own scanner. It’s a neat site, but unfortunately each scan is slow takes upwards of 20 seconds, and the failure rate is high. It’s strength comes from […]